[p3t] 'contract first' and OpenAPI

Mon Feb 26 17:56:25 CET 2024

Hi,

As others have pointed out, the OpenAPI specification I wrote for UWS in the Space Telescope repo below was written by hand, from the UWS standard. It was partly guided by the work done in FastAPI for our TAP service, but was written whole-cloth using the UWS document as the ultimate source of truth (which came with all the difficulties of parsing that document I have often complained about).

https://github.com/spacetelescope/vo-openapi

I like the repository structure you've given and I'd be happy to move this work to the IVOA repo, with the understanding that this is not the definitive version of the document, but maybe something just informative.

As you mentioned Dave, I did find writing it in JSON to be somewhat distasteful, and despite some of my gripes with the YAML syntax, it is much more readable in my opinion.

I would also strongly advocate for Dave's 'contract-first' approach. The following is mostly in response to your last few paragraphs, Dave-- your comment of everyone being happier with their existing tools and not using OpenAPI to generate code for their services (though I know you're not actually suggesting that) made me somewhat anxious that we might lose sight of what I see is real benefit of this work.

~~ Jumping on the soapbox here ~~

What I view as the most important product of all of this: a way to clear, concise, and (importantly) technical way define standards-- a 'thing' that describes the standard and nothing but the standard. You send this payload to that endpoint, and you'll get this response. All the wonderful benefits of using the OpenAPI tooling, generating code, validating, etc. are just icing on the cake.

When I think about the IVOA, I'm looking at it as just an engineer. With this group specifically, I'm putting myself in the shoes of an engineer, who just had the request to create a TAP service dropped on their lap, who has no prior knowledge that something like the IVOA even existed (which is exactly what happened to me). Consider also, and I'll say the uncomfortable part out loud, an engineer who might not actually care all much about VO standards, and just needs to write a service that passes validation before he clocks out and gets to stop thinking about it. Considering that scenario, that's where all the usefulness of OpenAPI documents (and this work more generally) will come out:

1. A small team could, yes, just opt to use the OpenAPI spec to generate a server, plug their machinery in and be done with it. They don't need even need to look too closely at the OpenAPI spec, except to know what the data is that they want to send.
2. Maybe there is no code generator for my particular language, but I can look at it and write my own server, and I know that it will be compliant with the standard. X endpoint, Y payload, Z response.
3. No ambiguity. No, "this other client interprets things this way, so we have to do it that way too." No, "This has been historically understood as..." No, "Well, the way I interpret this paragraph is.." and maybe my biggest gripe: "This behavior is fully described by (some other standard)."

~~ done with the preaching ~~

I know this isn't ~directly~ following the chain of the thread here, but it's what's been in my mind following the various threads thus far.

On 2/25/24, 11:47 PM, "p3t on behalf of Dave Morris via p3t" <p3t-bounces at ivoa.net <mailto:p3t-bounces at ivoa.net> on behalf of p3t at ivoa.net <mailto:p3t at ivoa.net>> wrote:

External Email - Use Caution

Hiya,

Can we jump back a bit and just check what the aim of the pt3 team is.

As I understood it, the aim of the team was to evaluate how to make the
IVOA services compatible with new tools and frameworks like OpenAPI,
FastAPI and Spring. Identify the legacy issues that make it hard to
implement our services using these tools, and make recommendations on
how best to develop the next generation of our services.

I'm not sure we have the remit to design the next generation cone-search
and UWS here. I'd have though that would be up to the DAL and GWS
working groups to build on the recommendations that come from this team
to help them develop the next generation of their services.

Anyway, I'd like to leave the details of the cone-search and UWS
improvements for a moment and concentrate on the evaluating the new
tools and processes part. In particular, this email is about how and why
we want to use OpenAPI.

From what I understand the current contribution from Russ uses Python
and the FastAPI framework to generate the service description. The
service is defined by the Python code and the OpenAPI definition is
reverse engineered from that.

I'm not clear how this would be applicable in the context of the IVOA,
where we are explicitly designing standards to work across a
heterogeneous environment of different programming languages and
frameworks.

Where OpenAPI descriptions may be able to help us develop interoperable
specifications is if we work "contract first". Starting with the OpenAPI
definition, included as part of the standard document, and then each
group could use the machine readable definition to generate or validate
their implementations.

Now that Russ has given us an initial OpenAPI description for
cone-search, I'd like to see if we change direction of the process and
generate our service implementations from the OpenAPI description.

With this in mind I would like to add a number of directory levels at
the top of the project repository.

First, I would separate the specification from the implementations,
placing the openapi.json contract in a separate 'ivoa' directory at the
top of the project. Making it clear that this is the interoperable part
that would be provided as part of the IVOA standard.

cone-search
ivoa
openapi.json
impl
....

A number of people have commented on the readability and maintainability
of the JSON version of the OpenAPI description. In which case, perhaps
it would be better to start with the YAML version.

cone-search
ivoa
openapi.yaml
impl
....

Then, within the implementations, I would add a separation based on
languages and framework.

cone-search
ivoa
openapi.yaml
impl
python
fastapi
....
java
spring
....

This enables others to add implementations using their favourite
frameworks and languages.

I would be willing to volunteer to develop a Java based implementation
using the Spring framework.

It would be interesting to see if we could get volunteers to have a go
at developing similar implementations using Rust or Go.

Note that the Python FastAPI implementation here would be the reverse of
the one currently provided by Russ. It would start with the OpenAPI
description and use the FastAPI framework to auto-generate the service
implementation Python classes from the OpenAPI description.

To make this clearer we could add a further level of distinction between
the forward process, starting with the OpenAPI definition and generating
the implementation classes, and the reverse process, starting with the
implementation classes and auto-generating the OpenAPI definition.

cone-search
ivoa
openapi.yaml
impl
python
fastapi
forward
# OpenAPI -> auto-generated Python
reverse
# Python -> auto-generated OpenAPI (Russ's
current code)
java
spring
forward
# OpenAPI -> auto-generated Java
reverse
# Java -> auto-generated OpenAPI

Once we have at least 2 forward implementations that start "contract
first" from the OpenAPI description and generate the implementation
classes from it, then we would be able to evaluate if including the
machine readable OpenAPI description of the service is a useful addition
that helps in developing interoperable services.

It is interesting to note that Gregory also has cone-search
implementation that is able to provide OpenAPI and Swagger descriptions
of their service, but so far no one has said that they started with an
OpenAPI description and generated their implementation from it.

If it turns out that everyone is happier developing their code using
their existing tools, without auto-generating the classes from an
OpenAPI description, then this would suggest that providing an OpenAPI
definition as part of our standards might not be as useful as we first
thought.

Historical note - the same thing happened with WSDL. Everyone said it
was the "next best thing", but in the end, no one actually used it to
generate code for their services.

The resulting code was just too fragile.

Looking forward to the discussion.

-- Dave

--------
Dave Morris
Research Software Engineer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------
AIMetrics: [{"name": "*","contribution": {"value": 0,"units": "%"}}]
--------
--
p3t mailing list
p3t at ivoa.net <mailto:p3t at ivoa.net>
https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!waPXMrWFO_L7ZtnVai8Vh29AiMJSnl4laf2lgvNv4prS7i10jsWrqaFFW1hSk8cbu2QoE7HGJ0o$ <https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!waPXMrWFO_L7ZtnVai8Vh29AiMJSnl4laf2lgvNv4prS7i10jsWrqaFFW1hSk8cbu2QoE7HGJ0o$>