[p3t] 'contract first' and OpenAPI

[Dempsey, James (IM&T, Black Mountain)]

Hi Dave,

Thanks for raising this. This is a really important point.

We want to make recommendations to the WGs for how to make the protocols easier to implement for modern web tools. Some of this will be tidying up RESTful use to align with industry best practice. We’ve also said we will investigate OpenAPI as a formal definition of the protocol APIs.

Those changes are highly likely to break compatibility with the existing protocols. However, the overall design of the protocols and much of the consensus which has coalesced around those protocols is still highly valuable and we should not throw that out.

Redesigning the protocols from scratch, or coming up with new protocols, is not something we have the resources to do. We have three more monthly meetings plus a two day workshop. Coming up with an agreed set of best practices and a path forward is already an ambitious goal for this group in that timeframe!

The layout you’ve proposed for the repository looks good to me.

Also, I’d be happy to assist with the spring implementation if you want a hand Dave. I’ve used SwaggerCodeGen in the past to generate Java client libraries from OpenAPI specs with good results.

Cheers,
James Dempsey
Senior Developer  |  CSIRO
james.dempsey at csiro.au<mailto:james.dempsey at csiro.au>  |  02 6214 2912


From: p3t <p3t-bounces at ivoa.net> on behalf of Dave Morris via p3t <p3t at ivoa.net>
Date: Monday, 26 February 2024 at 3:48 pm
To: p3t at ivoa.net <p3t at ivoa.net>
Subject: [p3t] 'contract first' and OpenAPI
Hiya,

Can we jump back a bit and just check what the aim of the pt3 team is.

As I understood it, the aim of the team was to evaluate how to make the
IVOA services compatible with new tools and frameworks like OpenAPI,
FastAPI and Spring. Identify the legacy issues that make it hard to
implement our services using these tools, and make recommendations on
how best to develop the next generation of our services.

I'm not sure we have the remit to design the next generation cone-search
and UWS here. I'd have though that would be up to the DAL and GWS
working groups to build on the recommendations that come from this team
to help them develop the next generation of their services.

Anyway, I'd like to leave the details of the cone-search and UWS
improvements for a moment and concentrate on the evaluating the new
tools and processes part. In particular, this email is about how and why
we want to use OpenAPI.

 From what I understand the current contribution from Russ uses Python
and the FastAPI framework to generate the service description. The
service is defined by the Python code and the OpenAPI definition is
reverse engineered from that.

I'm not clear how this would be applicable in the context of the IVOA,
where we are explicitly designing standards to work across a
heterogeneous environment of different programming languages and
frameworks.

Where OpenAPI descriptions may be able to help us develop interoperable
specifications is if we work "contract first". Starting with the OpenAPI
definition, included as part of the standard document, and then each
group could use the machine readable definition to generate or validate
their implementations.

Now that Russ has given us an initial OpenAPI description for
cone-search, I'd like to see if we change direction of the process and
generate our service implementations from the OpenAPI description.

With this in mind I would like to add a number of directory levels at
the top of the project repository.

First, I would separate the specification from the implementations,
placing the openapi.json contract in a separate 'ivoa' directory at the
top of the project. Making it clear that this is the interoperable part
that would be provided as part of the IVOA standard.

     cone-search
         ivoa
             openapi.json
         impl
            ....

A number of people have commented on the readability and maintainability
of the JSON version of the OpenAPI description. In which case, perhaps
it would be better to start with the YAML version.

     cone-search
         ivoa
             openapi.yaml
         impl
             ....

Then, within the implementations, I would add a separation based on
languages and framework.

     cone-search
         ivoa
             openapi.yaml
         impl
             python
                 fastapi
                     ....
             java
                 spring
                     ....

This enables others to add implementations using their favourite
frameworks and languages.

I would be willing to volunteer to develop a Java based implementation
using the Spring framework.

It would be interesting to see if we could get volunteers to have a go
at developing similar implementations using Rust or Go.

Note that the Python FastAPI implementation here would be the reverse of
the one currently provided by Russ. It would start with the OpenAPI
description and use the FastAPI framework to auto-generate the service
implementation Python classes from the OpenAPI description.

To make this clearer we could add a further level of distinction between
the forward process, starting with the OpenAPI definition and generating
the implementation classes, and the reverse process, starting with the
implementation classes and auto-generating the OpenAPI definition.

     cone-search
         ivoa
             openapi.yaml
         impl
             python
                 fastapi
                     forward
                         # OpenAPI -> auto-generated Python
                     reverse
                         # Python -> auto-generated OpenAPI (Russ's
current code)
             java
                 spring
                     forward
                         # OpenAPI -> auto-generated Java
                     reverse
                         # Java -> auto-generated OpenAPI


Once we have at least 2 forward implementations that start "contract
first" from the OpenAPI description and generate the implementation
classes from it, then we would be able to evaluate if including the
machine readable OpenAPI description of the service is a useful addition
that helps in developing interoperable services.

It is interesting to note that Gregory also has cone-search
implementation that is able to provide OpenAPI and Swagger descriptions
of their service, but so far no one has said that they started with an
OpenAPI description and generated their implementation from it.

If it turns out that everyone is happier developing their code using
their existing tools, without auto-generating the classes from an
OpenAPI description, then this would suggest that providing an OpenAPI
definition as part of our standards might not be as useful as we first
thought.

Historical note - the same thing happened with WSDL. Everyone said it
was the "next best thing", but in the end, no one actually used it to
generate code for their services.

The resulting code was just too fragile.

Looking forward to the discussion.

-- Dave

--------
Dave Morris
Research Software Engineer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------
AIMetrics: [{"name": "*","contribution": {"value": 0,"units": "%"}}]
--------
--
p3t mailing list
p3t at ivoa.net
http://mail.ivoa.net/mailman/listinfo/p3t
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240226/66b00664/attachment-0001.htm>