[p3t] A thought on DataLink in a JSON-based protocol

[Paul Harrison]

I had had a similarish idea a while back https://github.com/ivoa-std/DataLink/issues/53 <https://github.com/ivoa-std/DataLink/issues/53> - I feel that an openAPI description would be way too low level for this description, however. You will see that I even made the heretical suggestion that perhaps the service description did not need to be embedded within VOTable – that removes a lot of problems… 

On 31/05/2024, 18:54, "p3t" <p3t-bounces at ivoa.net> wrote: 


Russ Allbery via p3t <p3t at ivoa.net> writes:

> This does not (at all) solve the problem of how to then embed these
> things in VOTables, since OpenAPI does not (so far as I know) have an
> XML representation for a schema. I think only JSON and YAML are
> defined. So that's another obvious drawback that I'm not sure how to
> approach.

I went for a walk and thought about this some more, and I think I have an
idea. This will at least make it obvious what sort of complexity would be
involved in going this route.

Currently service descriptors define two separate things: a simple schema
or standard protocol reference for an associated service, and a set of
instructions for how to set parameters to that service from either static
values or columns in the VOTable.

I have no desire to specify an XML serialization of an OpenAPI schema.
That doesn't sound like a good time. But what if we separated those two
things? A service descriptor would then be a link to a schema
(specifically, in the OpenAPI context, to an operation specified by that
schema), and a set of instructions for how to set input values. Those
instructions would be an identifier for the input parameter, which would
be a string whose meaning is determined by the type of schema, and the
value, which similar to today would either be a static value or a
reference to a column in the associated VOTable.

We would continue to define an XML serialization of this service
descriptor, as well as a JSON serialization (and any future protocol
serialization). The JSON version of a service descriptor would look
something like this:

{
"schema": {
"type": "openapi",
"url": "https://urldefense.com/v3/__https://example.com/service/openapi.json*/__;Iw!!PDiH4ENfjr2_Jw!GOfLj4fKgYpG37s2WU2gWq0g1PobVXoGBwr1uwVMPYm3D9X5i6ZgUOn473wMPKiWLHkJP36iHMEqPGfGIQ4$ [example[.]com]<json-pointer>"
},
"inputParams": [
{
"parameter": "<json-pointer>",
"value": "static-value"
},
{
"body": "<json-pointer? json-path? see below>",
"ref": "primaryID"
}
]
}

(This is very incomplete; more fields would be needed.)

The end of the URL is a JSON-Pointer to the operation in the schema. This
means that in the case of web services that publish their own OpenAPI
schemas, the service descriptor can just point to the OpenAPI schema
published by the service with the appropriate JSON-Pointer to reference
the specific intended operation. (An "operation" is basically a path plus
an HTTP verb.)

Input parameters then reference either an OpenAPI parameter or a field in
the body of the request. Referencing parameters is easy, since they're
right there in the schema so this can just be another JSON-Pointer
relative to the operation. The body is a bit harder, since (to support
content negotiation) it's defined in the OpenAPI schema as a content type
mapped to a schema, and then the schema defines the nested structure. I'm
not sure if we want a JSON-Pointer to the schema definition or (for JSON
network serializations) a JSON path within the body.

Advantages:

1. We're leaning into OpenAPI already, at least for now, and we can tag
the type of schema to give us the flexibility to use other types of
schemas in the future (asyncAPI or whatever).

2. Being able to point directly to the OpenAPI schema provided by the
service is a nice reduction in duplicated information. Right now, our
DataLink services have to first write the service, which creates an
OpenAPI schema, and then separately also define the same information in
a DataLink snippet, which is annoying and involves two sources of data
that can get out of sync.

3. This gives us the ability to handle any HTTP verb and any format that
can be defined by OpenAPI, or any similar specification standard we
adopt in the future. This is way more power than the current service
descriptor syntax has.

Disadvantages:

1. Parsing the semantics of an entire OpenAPI schema is way work than
clients are going to want to do. I think we would need to be able to
define a really narrow subset of information in the schema that clients
would need to look at and allow clients to ignore everything else, and
we're still paying some complexity cost because there probably aren't
great OpenAPI schema client libraries for doing this sort of operation.
And we do want the client to parse the schema in order to get things
like a list of allowable values, minimum and maximum values, etc. Most
of the JSON Schema libraries I'm aware of are focused on validation of
input data, not in parsing the schema for semantic information that one
might want to use to, for instance, build an HTML form.

2. Related, even finding the schema is going to be complicated,
particularly for a request body. I'm not sure if a pointer into the
schema is the right approach, because the schema can be a reference to
a different part of the OpenAPI schema, which in turn can reference
other parts of the OpenAPI schema.

2. I'm not sure how good the library support for JSON-Pointers is. It
does look like there are a few good Python libraries out there.

4. If we want the specification of the service to continue to contain IVOA
type information (and I think we do), we will need to define an OpenAPI
extension that carries that information. I suspect we'll want to do
that anyway, because I suspect we'll want to add IVOA type information
to all of our schemas, but I'm not sure how hard that's going to be.

-- 
Russ Allbery (eagle at eyrie.org) <https://urldefense.com/v3/__https://www.eyrie.org/*eagle/__;fg!!PDiH4ENfjr2_Jw!GOfLj4fKgYpG37s2WU2gWq0g1PobVXoGBwr1uwVMPYm3D9X5i6ZgUOn473wMPKiWLHkJP36iHMEqJDz6o3k$ [eyrie[.]org]>
-- 
p3t mailing list
p3t at ivoa.net
https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!PDiH4ENfjr2_Jw!GOfLj4fKgYpG37s2WU2gWq0g1PobVXoGBwr1uwVMPYm3D9X5i6ZgUOn473wMPKiWLHkJP36iHMEqPB9p7Rk$ <https://urldefense.com/v3/__http:/mail.ivoa.net/mailman/listinfo/p3t__;!!PDiH4ENfjr2_Jw!GOfLj4fKgYpG37s2WU2gWq0g1PobVXoGBwr1uwVMPYm3D9X5i6ZgUOn473wMPKiWLHkJP36iHMEqPB9p7Rk$> [mail[.]ivoa[.]net] 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240603/f7f6555a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5983 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240603/f7f6555a/attachment-0001.bin>