[p3t] 'contract first' and OpenAPI

[Russ Allbery]

Dave Morris via p3t <p3t at ivoa.net> writes:

> As I understood it, the aim of the team was to evaluate how to make the
> IVOA services compatible with new tools and frameworks like OpenAPI,
> FastAPI and Spring. Identify the legacy issues that make it hard to
> implement our services using these tools, and make recommendations on
> how best to develop the next generation of our services.

This is also my understanding, but in my opinion the primary obstacle that
makes it hard to develop IVOA services with newer tools is the protocol.
The protocol redesign is therefore the part that I think is the most
important; without that, we won't get the benefits that we're hoping for.

My goal for the outcome of this process is two-fold:

1. Have a road map towards IVOA protocols that are friendly to a broader
   array of current development tools, and avoid the experience I had when
   implementing an IVOA service (as someone familiar with REST services
   but not at all familiar with IVOA services) of having to stop every
   paragraph or so and hand-code a workaround to something the IVOA
   protocol required that the development framework didn't expect.

2. Build an understanding of IVOA protocols that separates content from
   protocol serialization so that adapting to the *next* change to
   development tools is much easier.  If in five years everyone is now
   using gRPC and protobuf4 or something instead of REST and JSON, it
   should be relatively trivial to issue a gRPC and protobuf4 iteration of
   the IVOA protocols without having to revisit all the parts that are
   domain-specific to astronomy, and with a clean decoupling of the
   semantics from the protocol representation.

Having an OpenAPI protocol spec is a nice side effect of this process for
all of the reasons that Joshua said, and I'm sure that some folks will
want to autogenerate clients.  Personally, I'm uninterested in
autogenerating clients; for me, it's generally easier to write some
Pydantic models to the OpenAPI spec and then share them between the server
and client.  Other people's mileage will of course vary, but I want to
make sure we don't lose the forest for the trees.

The goal, from my perspective, is to have a protocol suite that's simple
and straightforward to implement for a developer who has experience with
web services but no experience with IVOA protocols.  The OpenAPI spec can
to some extent give us feedback on whether we're achieving that goal, but
I don't consider code generation to be the measure of success.  For
example, if we were able to code idiosyncratic behavior into the OpenAPI
protocol spec so that a code generator can handle it but in a way that
would still be a stumbling block for hand-coded implementations to the
spec, I think that would be a failure.

> Anyway, I'd like to leave the details of the cone-search and UWS
> improvements for a moment and concentrate on the evaluating the new
> tools and processes part. In particular, this email is about how and why
> we want to use OpenAPI.

I think there's a lot of work to be done and division of labor is great.
I'd be very happy to see other people working on the code generation part
and experimenting with that feature of OpenAPI.  I think it may make sense
to proceed in two tracks: I and other folks who care primarily about the
protocol can continue iterating on that design, and you and anyone else
who wants to experiment with features unlocked by OpenAPI can start
experimenting with generating clients.

I have no opinions about the code layout of the repository; I just started
with something simple to get going.  Please feel free to move things
around however makes sense.

> From what I understand the current contribution from Russ uses Python
> and the FastAPI framework to generate the service description. The
> service is defined by the Python code and the OpenAPI definition is
> reverse engineered from that.

> I'm not clear how this would be applicable in the context of the IVOA,
> where we are explicitly designing standards to work across a
> heterogeneous environment of different programming languages and
> frameworks.

I tried to explain this in my first message, but I think I may have
failed.  The only reason why I generated the initial OpenAPI spec from a
FastAPI server implementation is that writing the initial spec that way
was two or three times faster than working directly in OpenAPI *for me*.
In other words, this is just a prototyping tool.  That server
implementation is not a useful output of a standardization process; it's
just the source code for how I came up with a draft OpenAPI spec.

The intent is for an OpenAPI spec to be a first-class output, and that
prototype generator would probably be thrown away in the future.  It's
only there to be useful to me as a way to iterate quickly and regenerate
new versions of the spec in a couple of minutes when we decide to change
things.

-- 
Russ Allbery (eagle at eyrie.org)             <https://www.eyrie.org/~eagle/>