Time Series Cube DM - IVOA Note

[Mark Taylor]

Hi DM.

I am reluctant to wade in here, because I've really only skim-read
the thread up till now, and I'm generally not well-informed about
data models.  So, if you want to dismiss my opinions as half-baked,
I won't be offended.  But since apps and ops are being mentioned,
I'll give my reactions.

On Tue, 21 Mar 2017, CresitelloDittmar, Mark wrote:

> On Tue, Mar 21, 2017 at 9:35 AM, Markus Demleitner <
> msdemlei at ari.uni-heidelberg.de> wrote:
> 
> > I think here we're getting to the bottom of what we're trying to work
> > out here: *why* do you want to say this?  What I'm trying to argue in
> > my parallel mail
> > http://mail.ivoa.net/pipermail/dm/2017-March/005492.html (look for
> > "For illustration") is that an object about you'd say such things
> > isn't what's actually useful for clients.  These, rather, need
> > annotation topical for what they're trying to do (data structure for
> > a cube plotter, axis/frame metadata for data merging component,
> > dataset metadata for an ingestor or a bibliography component).

>From a client point of view: I tend to agree.  See below for some
more explanation.
 
> I consider the validation requirement a pretty important one..
>   * an application like IRIS to verify that the product being read is
> compatible with the code expectations
>   * folks like 'Operations' to check that a data provider is producing what
> they say they are

With my Ops hat on, I'd say that kind of thing is nice to have, but
I wouldn't want it to drive DM architecture decisions if there are
pressing arguments in the other direction.

> > In other words: I'm proposing to abandon the hope that "This dataset
> > is valid" will be a statement useful beyond management and
> > beancounting.  Instead, I hope we'll see "This dataset has valid
> > STC-1, STC-2, photometry-1, Dataset-1, and NDCube-1 annotations",
> > which tells concrete software if whatever annotation(s) it needs are
> > all right.
> >
> 
> I think we need more input from the client/Applications side... to me, this
> feels like an interoperability nightmare (though I think you argue the
> opposite).  An application would need to check that the instance contains
> valid annotation for every component that it uses, rather than just knowing
> it is OK by seeing it is an NDCube-1 instance.

To address this question from an application author's point of view,
I would tend to approach this in the way that Markus is suggesting,
i.e. think about the details I need for what I want to do (plot,
calculate, output, whatever) and if the pieces I need for the job
at hand are in place, ignore (never even notice) errors or inconsistencies
relating to parts of a DM that are not relevant to what I'm trying to do.
That's what I do with e.g. VOTable; look for the elements I want to use,
and if they're in place work with them, rather than validate the whole
thing against a schema and refuse to look at it if the validation fails.

Some reasons why:

   - Gives you a chance to work with partly broken data instances
     (there are often a lot out there) - though I acknowledge there
     is a downside, in that broken instances have less incentive to
     get fixed

   - Gives current versions of the software a chance to work
     unmodified with future versions of data models.
     Even if the software is still maintained in parallel with
     ongoing DM developments, it means people using older software
     versions stand a chance of using current services.

   - Validation against a schema or VO-DML specification(?) or
     whatever probably does not do all the validation you need to
     do for a particular job.  I haven't followed the VO-DML story
     very carefully, but to pursue the XSD analogy, just because
     an XML document validates against the VOTable XSD doesn't mean
     it's usable, there are lots of things that can go wrong that
     do not trigger validation failure.  So it's typically necessary
     to do an amount of extra validation by hand in any case, and it's
     usually not that much extra effort to put in checks against
     DM expectations at that stage.
     It is of course extremely useful to have a rigorous but
     human-readable, DM specification to hand at that stage so
     you know what ought to be legal.

Other people may have different approaches or requirements.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/