Time Series Cube DM - IVOA Note

[Markus Demleitner]

Hi Omar,

On Thu, Feb 23, 2017 at 09:47:30AM -0500, Laurino, Omar wrote:
> On Thu, Feb 23, 2017 at 4:46 AM, Markus Demleitner <
> msdemlei at ari.uni-heidelberg.de> wrote:
> 
> > Now, my appeal is to have the same property across DMs: We should
> > build things so that a document can be valid NDCube-1.0 regardless of
> > whether it uses Dataset-1.0 or Dataset-2.4 or both, or even none
> > of them.  The point of my previous mail was that that's feasible;
> >
> 
> The problem with this is that it would depend on how *exactly* the
> importing model imports and uses the imported model.

Exactly -- this is why I'm making all the noise here.  I'm
advertising a model in which we minimize dependencies, and I argue
that's the big dividend of all the pain we've gone into with VO-DML
and proper annotation.

> If a TimeSeries *is* a Dataset (the example is arbitrary), then it is
> important to know what version of a Dataset it is, because Dataset-1.0 and
> Dataset-2.0 may be, by design, not compatible. Your point is still relevant

Right.  And I'm arguing that we should keep inheritance and embedding
at an absolute minimum, using it only when technically necessary.  In
your example, I don't believe there actually should be a type
TimeSeries (what would it convey?).  There's Dataset (and that would
have an attribute dataproduct_type="timeseries" as in obscore).

Time series clients would then know it's something for them.  They'd
then look for annotation they know.  Typically, they'll look for an
NDCube annotation, but that's independent of the fact that we have
time series here.  And in particular, things remain time series
whether there's NDCube-1.0 or NDCube-2.4 annotation in there, or
both.

Finally, the time series will contain a time, so we need STC
annotation, and there are dependent axis that could have STC
annotations (for time series over positions), or photometry, or may
have spigot annotation.  Neither NDCube nor Dataset need to know
anything about the details here.

> for minor increments, though. Depending on the type of parser, it might not
> really matter if a TimeSeries is expressed in terms of Dataset-1.0 or
> Dataset-1.2, because by design a Dataset-1.2 valid annotation is also a
> valid Dataset-1.0 annotation, and vice versa. A validator would still be
> able to validate the instance, for example.

Well, I'd argue that by definition, Dataset-1.2 and Dataset-1.0 must
be interchangable (with 1.2 being able to express a bit more, of
course).  That, I claim, is the definition of a minor update, and
that's what we're trying to reflect on the schema side (RFC still
open: http://wiki.ivoa.net/twiki/bin/view/IVOA/XMLVersRFC).

> In any case an importing VODML model MUST say what models it is importing,
> otherwise it cannot refer to their elements, not even to state that
> "something" has "such and such" datatype, e.g. that a ts:TimeSeries is a

True.  Which is why this should be avoided, except for a well-defined
set of well-thought-out, rarely changing basic classes.  Well, of
course, there's the other application, where individual institutions
take an IVOA data model and adapt it to their needs.

Even then multiple imports should, by extension, be strongly
discouraged, so the Academy of Sciences of Paraphernalia would
separately extend NDCube to paraph-ndcube: and STC to paraph-stc: if
they absolutely have to.

As I said a few mails back, four points with three connections each
result in a completely rigid tetrahedron in which you can't move
anything any more.

> Now, the designers of the importing model can be wise and reduce the number

s/can/should/

> of references to an imported model if the coupling between the models is
> inherently loose. This might involve using object composition over type
> inheritance, if this makes sense from a modeling point of view. Still, if
> they want to enable compatibility with a new major version of a model, they
> need to create a new xml descriptor that describes the relationship with
> the new model. It could be trivial to do so, but this depends on the

Uh -- you lost me at the part with the compatibility with new
major versions of a model.  Can you try to come up with an example?

> Most likely, if we wanted to enable this (new) requirement we would need to
> make some tweaks to the VODML spec, and possibly to the mapping spec. One
> might say that TimeSeriesDM imports the Dataset type from any version of
> DatasetDM, and the binding is done at serialization time, i.e. when a

No, I don't think anything needs to change in the draft specs to
accomodate this.  There should simply be as few imports as
possible, and as I said above I'm pretty sure there's no need for
imports of DMs like DatasetDM at all.  STC *might* be a slightly
different matter if we keep a Characterisation DM, but even there I'd
advocate (until someone shows me an example where that paradigm
breaks) there char: should only talk about generic "limits", and the
actual physical annotation of these limits can be kept out of char: and
just live in STC, phot:, or spigot:, keeping char: nicely generic.

> However, I could see a model (probably not DatasetDM) removing a type
> between major increments, possibly leaving importing models with dangling
> references to non-existent types, e.g. ad absurdum DatasetDM-2.0 might
> remove the Dataset type altogether, so TimeSeries would be left with a
> dangling reference to a Dataset type that does not exist anymore.

Yes, obviously, such references cannot go across major versions;
that's what major versions are about: They break compatibility, and
the big art is to make that possible without breaking clients.  I
think VO-DML and out annotation plans have all it takes to pull off
this trick.

Whether references to and within DMs contain minor versions or not is
orthogonal to this discussion, but since we've touched it I take the
liberty of mentioning that I've  argued in
http://mail.ivoa.net/pipermail/dm/2017-January/005458.html they
should not.

Just to be sure we're on the same page: Within one instance, since
we've agreed on a fixed mapping from prefixes to minor DM versions,
you can only "use" one minor version (i.e., it's impossible (and
would be silly) to have annotations for both NDCube-1.0 and
NDCube-1.2).  Agreed?

> The only way I can see this working at this point is by removing the link
> altogether. It could be that TimeSeries is just an additional layer of
> metadata that can be added to a file on top of the existing ones. So
> something might be annotated as being both a Dataset (of any version) and a

Yes! Except, as I said: 

TL;DR: I don't think we need the notion of TimeSeries except as
a term in the values of dataproduct_type at all: it's just a Dataset,
and NDCube; and if you look, you'll see that there's just one
independent axis that happens to be a time in STC).

          -- Markus