[p3t] 'contract first' and OpenAPI

[Dave Morris]

Yes, I agree.
More later when I find WiFi for laptop.


On 26 February 2024 16:56:38 Joshua Fraustro via p3t <p3t at ivoa.net> wrote:

> Hi,
>
> As others have pointed out, the OpenAPI specification I wrote for UWS in 
> the Space Telescope repo below was written by hand, from the UWS standard. 
> It was partly guided by the work done in FastAPI for our TAP service, but 
> was written whole-cloth using the UWS document as the ultimate source of 
> truth (which came with all the difficulties of parsing that document I have 
> often complained about).
>
> https://github.com/spacetelescope/vo-openapi
>
> I like the repository structure you've given and I'd be happy to move this 
> work to the IVOA repo, with the understanding that this is not the 
> definitive version of the document, but maybe something just informative.
>
> As you mentioned Dave, I did find writing it in JSON to be somewhat 
> distasteful, and despite some of my gripes with the YAML syntax, it is much 
> more readable in my opinion.
>
> I would also strongly advocate for Dave's 'contract-first' approach. The 
> following is mostly in response to your last few paragraphs, Dave-- your 
> comment of everyone being happier with their existing tools and not using 
> OpenAPI to generate code for their services (though I know you're not 
> actually suggesting that) made me somewhat anxious that we might lose sight 
> of what I see is real benefit of this work.
>
> ~~ Jumping on the soapbox here ~~
>
> What I view as the most important product of all of this: a way to clear, 
> concise, and (importantly) technical way define standards-- a 'thing' that 
> describes the standard and nothing but the standard. You send this payload 
> to that endpoint, and you'll get this response. All the wonderful benefits 
> of using the OpenAPI tooling, generating code, validating, etc. are just 
> icing on the cake.
>
> When I think about the IVOA, I'm looking at it as just an engineer. With 
> this group specifically, I'm putting myself in the shoes of an engineer, 
> who just had the request to create a TAP service dropped on their lap, who 
> has no prior knowledge that something like the IVOA even existed (which is 
> exactly what happened to me). Consider also, and I'll say the uncomfortable 
> part out loud, an engineer who might not actually care all much about VO 
> standards, and just needs to write a service that passes validation before 
> he clocks out and gets to stop thinking about it. Considering that 
> scenario, that's where all the usefulness of OpenAPI documents (and this 
> work more generally) will come out:
>
> 1. A small team could, yes, just opt to use the OpenAPI spec to generate a 
> server, plug their machinery in and be done with it. They don't need even 
> need to look too closely at the OpenAPI spec, except to know what the data 
> is that they want to send.
> 2. Maybe there is no code generator for my particular language, but I can 
> look at it and write my own server, and I know that it will be compliant 
> with the standard. X endpoint, Y payload, Z response.
> 3. No ambiguity. No, "this other client interprets things this way, so we 
> have to do it that way too." No, "This has been historically understood 
> as..." No, "Well, the way I interpret this paragraph is.." and maybe my 
> biggest gripe: "This behavior is fully described by (some other standard)."
>
> ~~ done with the preaching ~~
>
> I know this isn't ~directly~ following the chain of the thread here, but 
> it's what's been in my mind following the various threads thus far.
>
>
>
> ?On 2/25/24, 11:47 PM, "p3t on behalf of Dave Morris via p3t" 
> <p3t-bounces at ivoa.net <mailto:p3t-bounces at ivoa.net> on behalf of 
> p3t at ivoa.net <mailto:p3t at ivoa.net>> wrote:
>
>
> External Email - Use Caution
>
>
> Hiya,
>
>
> Can we jump back a bit and just check what the aim of the pt3 team is.
>
>
> As I understood it, the aim of the team was to evaluate how to make the
> IVOA services compatible with new tools and frameworks like OpenAPI,
> FastAPI and Spring. Identify the legacy issues that make it hard to
> implement our services using these tools, and make recommendations on
> how best to develop the next generation of our services.
>
>
> I'm not sure we have the remit to design the next generation cone-search
> and UWS here. I'd have though that would be up to the DAL and GWS
> working groups to build on the recommendations that come from this team
> to help them develop the next generation of their services.
>
>
> Anyway, I'd like to leave the details of the cone-search and UWS
> improvements for a moment and concentrate on the evaluating the new
> tools and processes part. In particular, this email is about how and why
> we want to use OpenAPI.
>
>
> From what I understand the current contribution from Russ uses Python
> and the FastAPI framework to generate the service description. The
> service is defined by the Python code and the OpenAPI definition is
> reverse engineered from that.
>
>
> I'm not clear how this would be applicable in the context of the IVOA,
> where we are explicitly designing standards to work across a
> heterogeneous environment of different programming languages and
> frameworks.
>
>
> Where OpenAPI descriptions may be able to help us develop interoperable
> specifications is if we work "contract first". Starting with the OpenAPI
> definition, included as part of the standard document, and then each
> group could use the machine readable definition to generate or validate
> their implementations.
>
>
> Now that Russ has given us an initial OpenAPI description for
> cone-search, I'd like to see if we change direction of the process and
> generate our service implementations from the OpenAPI description.
>
>
> With this in mind I would like to add a number of directory levels at
> the top of the project repository.
>
>
> First, I would separate the specification from the implementations,
> placing the openapi.json contract in a separate 'ivoa' directory at the
> top of the project. Making it clear that this is the interoperable part
> that would be provided as part of the IVOA standard.
>
>
> cone-search
> ivoa
> openapi.json
> impl
> ....
>
>
> A number of people have commented on the readability and maintainability
> of the JSON version of the OpenAPI description. In which case, perhaps
> it would be better to start with the YAML version.
>
>
> cone-search
> ivoa
> openapi.yaml
> impl
> ....
>
>
> Then, within the implementations, I would add a separation based on
> languages and framework.
>
>
> cone-search
> ivoa
> openapi.yaml
> impl
> python
> fastapi
> ....
> java
> spring
> ....
>
>
> This enables others to add implementations using their favourite
> frameworks and languages.
>
>
> I would be willing to volunteer to develop a Java based implementation
> using the Spring framework.
>
>
> It would be interesting to see if we could get volunteers to have a go
> at developing similar implementations using Rust or Go.
>
>
> Note that the Python FastAPI implementation here would be the reverse of
> the one currently provided by Russ. It would start with the OpenAPI
> description and use the FastAPI framework to auto-generate the service
> implementation Python classes from the OpenAPI description.
>
>
> To make this clearer we could add a further level of distinction between
> the forward process, starting with the OpenAPI definition and generating
> the implementation classes, and the reverse process, starting with the
> implementation classes and auto-generating the OpenAPI definition.
>
>
> cone-search
> ivoa
> openapi.yaml
> impl
> python
> fastapi
> forward
> # OpenAPI -> auto-generated Python
> reverse
> # Python -> auto-generated OpenAPI (Russ's
> current code)
> java
> spring
> forward
> # OpenAPI -> auto-generated Java
> reverse
> # Java -> auto-generated OpenAPI
>
>
>
>
> Once we have at least 2 forward implementations that start "contract
> first" from the OpenAPI description and generate the implementation
> classes from it, then we would be able to evaluate if including the
> machine readable OpenAPI description of the service is a useful addition
> that helps in developing interoperable services.
>
>
> It is interesting to note that Gregory also has cone-search
> implementation that is able to provide OpenAPI and Swagger descriptions
> of their service, but so far no one has said that they started with an
> OpenAPI description and generated their implementation from it.
>
>
> If it turns out that everyone is happier developing their code using
> their existing tools, without auto-generating the classes from an
> OpenAPI description, then this would suggest that providing an OpenAPI
> definition as part of our standards might not be as useful as we first
> thought.
>
>
> Historical note - the same thing happened with WSDL. Everyone said it
> was the "next best thing", but in the end, no one actually used it to
> generate code for their services.
>
>
> The resulting code was just too fragile.
>
>
> Looking forward to the discussion.
>
>
> -- Dave
>
>
> --------
> Dave Morris
> Research Software Engineer
> Wide Field Astronomy Unit
> Institute for Astronomy
> University of Edinburgh
> --------
> AIMetrics: [{"name": "*","contribution": {"value": 0,"units": "%"}}]
> --------
> --
> p3t mailing list
> p3t at ivoa.net <mailto:p3t at ivoa.net>
> https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!waPXMrWFO_L7ZtnVai8Vh29AiMJSnl4laf2lgvNv4prS7i10jsWrqaFFW1hSk8cbu2QoE7HGJ0o$ 
> <https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!waPXMrWFO_L7ZtnVai8Vh29AiMJSnl4laf2lgvNv4prS7i10jsWrqaFFW1hSk8cbu2QoE7HGJ0o$>
>
>
>
> --
> p3t mailing list
> p3t at ivoa.net
> http://mail.ivoa.net/mailman/listinfo/p3t



Sent with Aqua Mail for Android
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240226/e7202f87/attachment-0001.htm>