Fwd: MIVOT: fully qualified attribute names

Fri Feb 17 17:18:27 CET 2023

-------- Message transféré --------
Sujet : 	Re: MIVOT: fully qualified attribute names
Date : 	Fri, 17 Feb 2023 17:12:33 +0100
De : 	BONNAREL FRANCOIS <francois.bonnarel at astro.unistra.fr>
Pour : 	Laurent Michel <laurent.michel at astro.unistra.fr>, dm at ivoa.net

Hi Markus, Laurent, all,
Le 16/02/2023 à 18:28, Markus Demleitner a écrit :
> Dear Laurent,
>
> On Thu, Feb 16, 2023 at 04:40:32PM +0100, Laurent Michel wrote:
>> We could continue playing ping-pong, but we reached a point where
>> there is no longer great benefits to expect by doing so.
> Hm, I don't know about that -- at least I'm learning quite a bit, and
> I think figuring things out in this way will help many others, too.
>
> First, it made me notice another (perhaps small) snag:
>
>> - all needed VODML files
> These are a bit hairy to find at this point. Admittedly, they *are*
> on https://ivoa.net/xml/, but they're intermingled with all kinds of
> XSDs, and it's really hard to see what's ancient and (excuse me)
> stillborn and what's fresh and current. My suggestion would be: Have an
> extra section for VO-DML files in the Schema repo so they are not
> drowned in the legacy of more or less rusty XSD from the DM group.
> And perhaps use a slightly different table pattern so there are links
> to both the .vo-dml.xml and the generated HTML documentation.

Yes this has been pointed out in the TCG a couple of years ago. But no 
decision was taken.

Anyway Vo-dml.xml are not xsd schema. they are xml DOCUMENTS, following 
the vodml xsd schema.

vo-dml xml documents could be attachedd to the fron page of each data 
model specification.

> I'd suggest "Formal VO-DML models" as a section header, perhaps with
> VO-DML being a link to the spec.
>
>> At this point, the annotator knows that it must build an instance 
>> with a role given by the model; 2 options:
>> taking the NAME (Markus option) —> @dmrole=error
>> taking the VODML-ID prefixed with the model name (MIVOT) —> 
>> @dmrole=meas:Measure.error
> Ah! See, this is what I had missed -- sorry about that. The
> attribute names are not computed, they're taken from the VO-DML
> files, more specifically, their vodml-id elements (the existence of
> which I had forgotten since when I had contributed to VO-DML).
> Still, for all I can tell, the syntax of the dmrole attribute is not
> specified in MIVOT yet (at least searching for vodml-id just turned
> up
>
> element_model.tex:All models that define vodml-ids used in the 
> annotation must be declared.
>
> for me). Perhaps you should put, somewhere above Table 2, some
> language like:
>
> The values of both dmrole and dmtype are constructed by taking
> the vodml-id of the respective VO-DML elements and prefixing it with
> the canonical short name of the model and a colon.
>
> I can't *promise* that would have prevented me from having asked for
> a clarification, but I'd claim there's a good chance.
Yes dm-roles are the vodml-id of classes and attributes. It's unambiguous
>
>> with the roles ala MIVOT , the client selecting
>> “.//INSTANCE[@dmrole=meas:Measure.error]: is sure of what it gets
>> with the roles ala Markus , the client selecting
>> “.//INSTANCE[@dmrole=error]: will return anything playing any role
>> named error, either from the meas:Measure class or any other. This
>> is the price for using non-unique identifiers.
> I note in passing that no client probably ever could learn anything
> useful when it does this in either case, so that's a weak argument.
> But then: While I'm still not particularly happy with the choice of
> vodml-id, and I still think there is no good reason for that choice,
> given the dmrole values are at least well-defined, it is certainly
> not a blocker any more.
>
>> We can add a section in the spec relating the process above. I
>> admit that would help.
> A non-normative appendix should indeed say where to find models and
> where to find the vodml-ids.
>
>> I will be less open-minded about the requirement of assigning the
>> RFC ending with the release of software pieces operating all steps
>> of the model mapping process with a good test coverage furthermore;
>> very new in the IVOA landscape!
> Well, the role string-computing function is no longer needed, because
> it's now clear where the role strings come from. So scratch that.
>
> What's not new in the VO is the demonstration of two (or, when there
> are producers and consumers, three) interoperating implementations.
>
> I *am* trying to provide one on the server side (that's why I'm
> here), but so far failed, partly because I couldn't figure out how to
> use the existing models to annotate even the basic aspects of the
> Gaia source catalogue (linking positions and proper motions, linking
> scalar errors, defining photometries; note that I'm not asking for
> annotation of the correlations any more -- sigh!).
>
> If I can't do that, I'd say it indicates *some* sort of problem:
> Perhaps our models are still insufficient -- but then there's no
> particular hurry to pass MIVOT (because we can't do anything useful
> with it until the models are there).
>
> Another possibility is that I'm too dumb or too grumpy --
Markus !!!
> in that
> case someone else should jump in and provide a... well, "second"
> implementation.

It is true that we don't have all the data models to annotate all our 
VOTables

The missing level is made of integrating datamodels such as Mango, Cube 
or maybe also TimeSeries, Spectrum (as specialisation of Cube or Mango)

But at least we have PhoTDM, Meas;, Coords which allow to demonstrate 
some annotation capability with MIVOT.

We have plenty of working examples for that.

My view on this :

     It's possible to generate xml components for each class from 
vo-dml-xml documents . Laurent's code does it.

     Now integrating annotation in any VO service framework, Dachs or 
whatever, requires new software development. That's unavoidable, with 
MiVOT or with an alternative which doesn't exist.

     One way to do it with MIVOT is to add these xml components in 
specific tables of the relational schema of the database (or the TAP 
schema in case of a TAP service)

    Then the pieces of code which build the VOtable responses can be 
modified to use on the fly these components to build the appropriate 
annotation for each VOTable response.

> I'm using quotes here because very frankly what I have seen so far as
> implementation examples is either outdated and does not correspond to
> the spec as present, or it strongly looks like hand-crafted examples.
> Whether or not the hand-crafted is right: For "implemenation", I'd
> strong prefer something that exists in the wild (i.e., in an actual
> data centre) and is consumed by something that is not completely
> contrived.

We have to break the deadlock loop.

Validating an annotation mechanism only when we have a fully integrated 
datamodel versus specify an annotation mechanism validated by partial 
models and then go forward by building   more complete datamodels

> only
>
> Remember my suggestion for an annotation syntax back in the Workshop
> days, <https://github.com/msdemlei/astropy>? You know, that *would*
> have worked as a PR against astropy, and the things it consumed where
> generated by a machine from an abstract representation in an actual
> data centre. It's that sort of thing I'd really like to see for
> something as wide-reaching and ambitious as MIVOT.
>
> But, really, at this point I'd already be content with just something
> coming out of an actual data centre (like mine) that some of your
> client-side code can consume. But that's really the minimum in terms
> of what I'll take as a "reference implementation" (as required by
> Stdproc forever).
>
>> The purpose of MIVOT is to map data on models serialised in VODML.
>> All the implementations we provide show that this requirement is
>> achieved even if we could dream about tools making the job easier.
> That's the second point: I have no idea just *what part* of MIVOT is
> actually exercised by the examples mentioned on the RFC page. There
> is simply *soo* much material in MIVOT that I feel unable to decide
> what is and what is not covered by implementation examples. To
> cite the (to me) scariest part of the spec: There is a 5-way
> definition by cases for COLLECTION in mostly very dense language, and
> I freely admit I can't understand most of it.
>
> I'd trust it a lot more if there were a non-author implementor who
> had looked at it and found that it at least does not contradict
> anything else in the spec. I expect I'd have to spend days working
> out what that is and writing test cases and all -- and sorry, I'll
> not spend that time on something I think we should think about in
> version 1.4 or version 1.5 of MIVOT, after we know *a lot* more.
>
>> Actually I’ve code generating MIVOT instances from VODML
>> (github:ivoa/modelinstanceinvot-code
>> branch:feature/instance-generator), i’m using it to build the
>> snippets I was talking about i my previous mail. This code is not
>> ready to be published, I would be happy to do it but I’ve just no
>> time to complete the job; but anyway, this must not block the REC.
> But on the other hand, no urget use case is actually waiting for this
> to become REC (or is one?), and given we're betting our coordinate
> annotation on it, we can at least wait with REC until we successfully
> exchange a 6-parameter astrometric solution with it and do an epoch
> propagation, with one client and two services implemented by three
> different people. I'm volunteering to be one on the server side, but
> I'll need help making sense of MCT.

Photometry at least can strongly benefit from MIVOT

Cheers

François

>
> Even then I'd feel *a lot* more comfortable if MIVOT 1.0 just
> contained whatever is necessary for that exercise, because I am
> almost 100% sure anything not exercised in that way will contain lots
> of contradictions and errors that will bug us later (that's
> invariably been the case for unused parts of specs that I have
> written).
>
> But if you really cannot be moved to postpone the complex stuff until
> we better understand what we're doing, I'd not stand in the way once
> we have the epoch propagation use case properly demonstrated. But
> that, I feel, is not asking too much.
>
> Thanks,
>
> Markus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20230217/4e265592/attachment.htm>