[p3t] 'contract first' and OpenAPI

[Dave Morris]

Hiya,

Can we jump back a bit and just check what the aim of the pt3 team is.

As I understood it, the aim of the team was to evaluate how to make the 
IVOA services compatible with new tools and frameworks like OpenAPI, 
FastAPI and Spring. Identify the legacy issues that make it hard to 
implement our services using these tools, and make recommendations on 
how best to develop the next generation of our services.

I'm not sure we have the remit to design the next generation cone-search 
and UWS here. I'd have though that would be up to the DAL and GWS 
working groups to build on the recommendations that come from this team 
to help them develop the next generation of their services.

Anyway, I'd like to leave the details of the cone-search and UWS 
improvements for a moment and concentrate on the evaluating the new 
tools and processes part. In particular, this email is about how and why 
we want to use OpenAPI.

 From what I understand the current contribution from Russ uses Python 
and the FastAPI framework to generate the service description. The 
service is defined by the Python code and the OpenAPI definition is 
reverse engineered from that.

I'm not clear how this would be applicable in the context of the IVOA, 
where we are explicitly designing standards to work across a 
heterogeneous environment of different programming languages and 
frameworks.

Where OpenAPI descriptions may be able to help us develop interoperable 
specifications is if we work "contract first". Starting with the OpenAPI 
definition, included as part of the standard document, and then each 
group could use the machine readable definition to generate or validate 
their implementations.

Now that Russ has given us an initial OpenAPI description for 
cone-search, I'd like to see if we change direction of the process and 
generate our service implementations from the OpenAPI description.

With this in mind I would like to add a number of directory levels at 
the top of the project repository.

First, I would separate the specification from the implementations, 
placing the openapi.json contract in a separate 'ivoa' directory at the 
top of the project. Making it clear that this is the interoperable part 
that would be provided as part of the IVOA standard.

     cone-search
         ivoa
             openapi.json
         impl
            ....

A number of people have commented on the readability and maintainability 
of the JSON version of the OpenAPI description. In which case, perhaps 
it would be better to start with the YAML version.

     cone-search
         ivoa
             openapi.yaml
         impl
             ....

Then, within the implementations, I would add a separation based on 
languages and framework.

     cone-search
         ivoa
             openapi.yaml
         impl
             python
                 fastapi
                     ....
             java
                 spring
                     ....

This enables others to add implementations using their favourite 
frameworks and languages.

I would be willing to volunteer to develop a Java based implementation 
using the Spring framework.

It would be interesting to see if we could get volunteers to have a go 
at developing similar implementations using Rust or Go.

Note that the Python FastAPI implementation here would be the reverse of 
the one currently provided by Russ. It would start with the OpenAPI 
description and use the FastAPI framework to auto-generate the service 
implementation Python classes from the OpenAPI description.

To make this clearer we could add a further level of distinction between 
the forward process, starting with the OpenAPI definition and generating 
the implementation classes, and the reverse process, starting with the 
implementation classes and auto-generating the OpenAPI definition.

     cone-search
         ivoa
             openapi.yaml
         impl
             python
                 fastapi
                     forward
                         # OpenAPI -> auto-generated Python
                     reverse
                         # Python -> auto-generated OpenAPI (Russ's 
current code)
             java
                 spring
                     forward
                         # OpenAPI -> auto-generated Java
                     reverse
                         # Java -> auto-generated OpenAPI


Once we have at least 2 forward implementations that start "contract 
first" from the OpenAPI description and generate the implementation 
classes from it, then we would be able to evaluate if including the 
machine readable OpenAPI description of the service is a useful addition 
that helps in developing interoperable services.

It is interesting to note that Gregory also has cone-search 
implementation that is able to provide OpenAPI and Swagger descriptions 
of their service, but so far no one has said that they started with an 
OpenAPI description and generated their implementation from it.

If it turns out that everyone is happier developing their code using 
their existing tools, without auto-generating the classes from an 
OpenAPI description, then this would suggest that providing an OpenAPI 
definition as part of our standards might not be as useful as we first 
thought.

Historical note - the same thing happened with WSDL. Everyone said it 
was the "next best thing", but in the end, no one actually used it to 
generate code for their services.

The resulting code was just too fragile.

Looking forward to the discussion.

-- Dave

--------
Dave Morris
Research Software Engineer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------
AIMetrics: [{"name": "*","contribution": {"value": 0,"units": "%"}}]
--------