[p3t] Draft documents demonstrating standards layout

Thu Aug 29 20:07:38 CEST 2024

My assertion is that UWS is just an interface to manipulate a small data model that describes jobs, and other IVOA standards can be though of in the same way. (in fact you can think of very complex things like Kubernetes in the same fashion)

I have replied in-line below to some of your specific points

> On 29 Aug 2024, at 17:11, Russ Allbery <eagle at eyrie.org> wrote:
> 
> This Message Is From a New External Sender
> You have not previously corresponded with this sender. Please exercise caution when opening links or attachments included in this message.
> Paul Harrison <paul.harrison at manchester.ac.uk <mailto:paul.harrison at manchester.ac.uk>> writes:
> > On 26 Aug 2024, at 06:18, Dave Morris via p3t <p3t at ivoa.net <mailto:p3t at ivoa.net>> wrote:
> 
> >> We should try to avoid referring to the overall goal as the 'JSON
> >> network protocol' - that is what P3T should be trying to avoid. If we
> >> bind too strongly to JSON we are in danger of repeating the mistakes of
> >> 20 years ago when we bound our standards to XML, WSDL and SOAP.
> 
> > Near the start of these conversations, I tried to point out that the
> > IVOA already has a standard way of describing the “schema” part of these
> > interface definitions - i.e VO-DML - if the data models associated with
> > a service are described in VO-DML, then in another 20 years time when
> > some other serialisation format is fashionable, then the VO-DML tooling
> > can just be altered to emit the schema for that serialisation, without
> > having to re-author the data model source, and obviously the more data
> > models there are the more advantageous this is.
> 
> Right, this was one of the sources of existing data types that this work
> needed to be reconciled with.  (The others are UCD, DALI, and VOTable.
> There are probably others that I'm missing.)  I only hadn't looked at that
> because I didn't have time yet.  I wanted to get a raw framework out
> first, and this was one of the details I wanted to go back and revisit.
> 
> I've taken a brief look this morning.  There's a lot of material in the
> VO-DML standard, so I'll have to go back to it later for a longer look.
> I had a few quick questions based on that initial look (and apologies if
> these are clearly answered already and I just missed them):
Yes the standard itself is a bit dense - however, there is some “standard tooling” that goes with it for creating and manipulating 
the models that does have the start of more “cookbook” style documentation,
but that is still lacking.https://ivoa.github.io/vo-dml/
> 
> 1. The existing document covers a lot of the ground that I think we'll
>    need (boolean, integer, real, datetime, string, enum) and a few things
>    I hadn't started thinking about (complex), but the primitive types by
>    design don't include valid ranges or precision information.  It looks
>    like ranges are handled with a Constraint element (and maybe precision
>    as well?).  Are there already data types for integers and reals defined
>    that conform to the sorts of constraints we'll need for interoperable
>    wire protocols (IEEE 64-bit floating point numbers, for example, which
>    impose multiple restrictions on both range and precision)?
I think that it was a design decision not to really go down to the actual binary serialisation of the base types in the definition (generally I think that string serialisation was all that was considered) - in principle that can be specified for binary serialisations where appropriate too though.

Constraints in VO-DML are in general rather under specified (they are pretty much allowed to be free from at the moment), but the concept is in VO-DML.
> 
> 2. The definition of a string needs to define what a "character" is for
>    interoperable protocols.  How would we go about doing that?  Is that an
>    update to VO-DAL, or a specialisation of the primitive type defined
>    there?  Similarly, datetime doesn't specify precision or time standard,
>    which we would need to nail down.
https://github.com/ivoa/vo-dml/issues/37 and I am sure that we can agree on a (set of) character representations that are supported across many OSs and computer languages.
> 
> 3. The VO-DML specification makes reference to a lot of useful composite
>    types like SkyCoordinate, but they're not in the HTML page linked from
>    the VO-DML 1.0 document page.  Is there a registry somewhere else of
>    composite types like that which have already been defined?

Models have been historically been the responsibility of the DM WG https://wiki.ivoa.net/twiki/bin/view/IVOA/IvoaDataModel and you can get an idea of what domains have been worked on - early models do not have a VO-DML representation. Note that the models have concentrated on describing data rather than necessarily things that appear in service interfaces - (though there is some cross-over) The official coordinate model is https://github.com/ivoa-std/CoordinateDM

> 
> 4. The VO-DML document is extremely XML-heavy.  It looked like every new
>    type defined in that framework is accompanied by an XML schema.  Is
>    this required for new types?  I think we only have as a goal that
>    implementers need not know XML, not that specification authors need not
>    know XML, but the latter would be a desirable property from my
>    perspective since I think it will be limiting when we try to find
>    enough resources to write down good specifications.
I agree that it all looks XMLy and I have thought that makes it difficult for people to write models for quite a while (about 10yrs!) - so I proposed a domain specific language that compiles into VO-DML https://www.ivoa.net/documents/Notes/VODSL/index.html - this is now integrated into the VO-DML tooling https://ivoa.github.io/vo-dml/modelling/VODSL/ - although VODSL is not “standard”, it is only there to compile to the XML form, all other transformations such as schema and code generation are done from the XML form, but you do not have to deal with XML at all to create models.
> 
> I think having a shared data type registry is an excellent idea, and I'd
> like to use it heavily.  I would be inclined to register a reusable data
> type with a name as soon as we have a need for the same data type in two
> different standards, so ideally we should make it very easy to register
> new ones, which if I understand correctly is part of the goal of the
> VO-DML work.
> 
> What I'm hoping for as output is something like an IANA registry page that
> includes all of the registered data types and their specifications in
> human-readable HTML, and without requiring publication of a standards
> revision in order to add a new data type.  If I understand VO-DML
> correctly, that's in close alignment with its goals, and maybe I'm just
> missing where that registry page is currently?
The publishing of the models on the IVOA site could be improved in my opinion, but on a model that I am developing that has fully embraced this way of working I have published to github pages to show the possibilities - 

documentation https://ivoa.github.io/ProposalDM/generated/proposal/Observation/ from this snippet of VODSL https://github.com/ivoa/ProposalDM/blob/aa70a46a60a80a29becfc5de2799479741c79bce/src/main/vodsl/proposaldm.vodsl#L266

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240829/4259d281/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2893 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/p3t/attachments/20240829/4259d281/attachment-0001.p7s>