[p3t] Draft documents demonstrating standards layout

[Dave Morris]

Hi Russ,

This is a really good place to start the discussions from.

We should try to avoid referring to the overall goal as the 'JSON 
network protocol' - that is what P3T should be trying to avoid. If we 
bind too strongly to JSON we are in danger of repeating the mistakes of 
20 years ago when we bound our standards to XML, WSDL and SOAP.

I suggest we split the content over 4 documents.

1) IVOA Service protocol
A document containing generic definitions of what services, operations 
are etc. This is pretty close to what you have in 
https://sqr-091.lsst.io/, but replace the phrase 'web service' with 
simply 'Service'.

2) IVOA HTTP protocol
A document describing how the IVOA should use aspects of the HTTP 
protocol. This includes the formatting of HTTP GET parameters from 
https://sqr-092.lsst.io/, and how IVOA services should use the HTTP 
content negotiation process. This also gives us somewhere to talk about 
HTTP error codes, caching, cross-site-scripting, and things like the 
POST-redirect-GET pattern.

3) IVOA JSON serialisation
This is just the details of how IVOA services should map object data 
into a JSON serialisation. This is a stand-alone document, separate from 
how we use the HTTP transport protocol. Pretty much what you have in 
https://sqr-092.lsst.io/, but without the HTTP parts. It should be 
explicitly clear what is inherited from the existing JSON standard and 
what is IVOA specific.

4) IVOA WebService protocol
A specific sub-type of the [IVOA Service Protocol] for IVOA service HTTP 
based WebServices.

An [IVOA WebService]
* MUST format HTTP request parameters as defined in [IVOA HTTP 
protocol].
* MUST use HTTP headers for content negotiation as defined in [IVOA HTTP 
protocol].
* MAY specify [IVOA JSON serialisation] as the default serialisation for 
objects in requests and responses.
* MAY include other serialisations for objects in requests and 
responses.

A particular IVOA standard, e.g. SODA, MAY specify a minimal set of 
content types. We already do this when we say TAP requires VOTable, but 
we should use the HTTP content type headers for this rather than a 
separate FORMAT parameter.

* Standards being developed in 2024 MAY include JSON in the minimum set.
* Standards being developed in 2024 MAY specify JSON as the default.

* Standards defined in 2034 MAY specify a different minimum set.
* Standards defined in 2034 MAY specify a different default.

This is not a huge technical change from what we have, but it does 
change the way we think about it.

Separating the data model, transport, and serialization.

* The main part of an IVOA Service standard defines the data model.
* A sub-section of the standard refers to [IVOA WebService protocol] as 
an implementation.
* The [IVOA WebService protocol] says a WebService MUST use content 
negotiation for HTTP requests.
* The [IVOA WebService protocol] says a WebService SHOULD specify JSON 
as the default content type serialisation.

End result is the same, just a few more lines in the OpenAPI spec, but 
it breaks the hard coded reliance on one serialization format.

For example, the main part of the SODA Standard would describe the 
operations and data model in an abstract way, without referring to a 
transport protocol or serialization format.

The SODA Standard would include a section on [Implementations], with a 
sub-section for the [WebService] implementation, which describes how to 
implement the SODA service using the [IVOA WebService] protocol.

As part of this, the SODA Standard can define a minimum set of content 
type serialisations that a [SODA WebService] MUST support.
In 2024, this minimum set would include JSON as the default.

End result, Rubin et al get what they want; the 2024 version of SODA 
requires JSON, BUT we do not build it in from the start. Change one 
single line in the SODA Standard and a future version could require 
[not_invented_yet] as the default serialization.

---- ----

One of the issues with the IVOA standards raised in the P3T team remit 
was the extent of the document cross referencing. To address the 
complaint that "to understand one standard you need to refer to at least 
three other documents each of which refer to three more" which makes it 
hard to get started.

As I write this email I am aware that we are in danger of making this 
worse, adding yet more documents for Services, WebService, JSON, HTTP 
etc ...

I still think this separating things out like this is the right way to 
go, but we should look at ways of mitigating the steep learning curve. 
Perhaps a section at the start of each document that lists the other 
documents needed and what they are needed for.


"
This document refers to the following set of documents and standards:
* [IVOA WebService protocol] for how to implement this standard as a 
WebService.
* [IVOA HTTP protocol] for HTTP specific details of the WebService 
methods.
* [IVOA JSON serialization] for object serialization.
* [IVOA VOTable standard] for data serialization.
"

---- ----

To make things worse, I hope to be able to set aside some time soon to 
contribute yet another document describing how to handle error messages.


Hope this is useful,
-- Dave


--------
Dave Morris
Research Software Engineer
UK SKA Regional Centre
Department of Physics and Astronomy
University of Manchester
--------
AIMetrics: []
--------


On 2024-08-24 01:43, Russ Allbery via p3t wrote:
> Hi all,
> 
> I have finished a very preliminary first draft of some documents that
> hopefully demonstrate what I had in mind for how to organize standards
> documents, and that also start fleshing out the specification for a 
> JSON
> protocol.
> 
> Some caveats:
> 
> The main point of these documents is to demonstrate a possible way of
> structuring standards and offer an example of the level of detail that 
> I
> think documents should provide.  These documents therefore include a 
> lot
> of specific specification details.  Those details are very much up for
> debate and I suspect many of them are incorrect.  At this stage I think
> the exact content is less important than the structure and the "vibes."
> My goal is to help create an intuitive feel for what a specification 
> could
> look like under this organization scheme.
> 
> I invented a new type system for these documents and did not attempt to
> reconcile it with the existing IVOA type system.  This is only due to 
> lack
> of time, not because I don't think that should be done.  There are a 
> few
> new types that don't quite fit into the IVOA type model, but most if 
> not
> all of the primitive types here should be replaced with the standard 
> IVOA
> types.  The point was merely to demonstrate the pattern of the main
> document defining a set of primitive types, the service document using
> them, and the network protocol document describing an encoding for 
> them.
> 
> This is very incomplete even relative to the deliverables I've been
> working on.  Missing are the UWS specification and how to write a 
> service
> document that uses UWS, the OpenAPI schema, and multiple other things.
> This is just a starting point to discuss.  I may be able to add some 
> more
> Monday morning before the meeting.
> 
> These documents are all published via our internal tech note system 
> only
> because that was fast and I didn't have to learn anything new and I 
> wanted
> to focus on getting content down to talk about.  All of this can be 
> lifted
> into the proper IVOA documentation system when we start hammering out
> actual documents.
> 
> With those caveats in mind, here are the three documents I've started:
> 
> https://sqr-091.lsst.io/
>     The framework document that lays out the standards structure, 
> defines
>     data types, provides data models that are shared between standards
>     such as structured error messages, and says what the other 
> documents
>     should contain.  Please don't put any weight on the structured 
> error
>     specification here; it's just a half-baked iteration on the model 
> from
>     our previous discussion, included as a placeholder.
> 
> https://sqr-092.lsst.io/
>     The JSON network protocol encoding.  This is where we get to the 
> heart
>     of the new work.  I tried to write the separation between this
>     document and the other documents with both an HTTP XML and a gRPC
>     specification in mind and put the things that would vary in this
>     document.  It's also a starting point for a proper JSON 
> serialization
>     specification.
> 
> https://sqr-093.lsst.io/
>     A web service specification for a very simple SODA-style cutout
>     service.  This is not a great example service.  I picked this one 
> just
>     because I was already very familiar with it and could write it
>     quickly, and because I'm currently working on that part of our 
> service
>     stack.
> 
> --
> Russ Allbery (eagle at eyrie.org)             
> <https://www.eyrie.org/~eagle/>