Time Series Cube DM - IVOA Note

[Laurino, Omar]

Markus,

(apologies in advance if I misinterpreted anything)

On Wed, Mar 22, 2017 at 9:20 AM, Markus Demleitner <
msdemlei at ari.uni-heidelberg.de> wrote:

> With a "God annotation", it would have to, in such a situation,
> entirely reject the dataset even though it could do quite a bit with
> it based on the annotation it understands.  Clients tend to try to
> avoid that situation and will try to hack around it, which kind of
> defeats the purpose of validation in the first place.
>

this could be a straw man argument, though. All the mapping proposals we
have made since the beginning of this effort had this as one of the main
requirements: if you are aware of a model, say STC-2.1, you should be able
to find instances of such model in any valid (and even a lot of invalid)
VOTables, even if you ignore all the other models. You'll find the model
definition in the VO-DML preamble in the VOTable and you'll say: OK, there
are some annotations here I can understand. You can't find such model
definition? You can still look for STC-2.x model identifiers, maybe you'll
find them even if the document wouldn't validate.

There is no God model in VO-DML. If anything, exactly the opposite, you
have building blocks that interoperate nicely with each other, as long as
they make sense as models, you don't fall into anti-patterns, etc... you
get the idea.

What you can't avoid in some cases is to tightly couple two models, the
same way you tightly couple, just to make an example, an application to
some version(s) of Python, working around any incompatible differences. Can
you assure me that the current DaCHS release will be compatible with all
the versions of Python from now to eternity? Did everybody's JAVA code
evolve seamlessly when commons-lang3 was releases as a major, incompatible
replacement for commons-lang?

You have a valid point in my opinion when you say that such tight coupling
should be reduced as much as possible. Decoupling DatasetDM and CubeDM, for
instance, might work pretty well. I am skeptical that you can do the same
with  any model using STC, or any other basic building block model.

VO-DML shines over previous annotation mechanisms because the coupling
among models is much less tight. Now you can say: CubeDM-1.x import
STC-2.x, and then use STC-2.x identifiers. An STC2-aware application will
be able to find all the STC-2.x annotations and ignore any knowledge of any
other model, including CubeDM. However, a Cube1-aware application will not
have a hard time figuring out which version of STC you are using... as far
as it's concerned it could be plenty, but it will only focus on STC-1.x. If
CubeDM-2 comes up with significant changes, and an application wants to
support both versions, some complexity is inevitable. Such cases will
always be complex, whatever the annotation.

The dark side of what you are suggesting, I agree with Mark, is that
decoupling everything sounds like an interoperability nightmare. While it
could be useful to have multiple DM annotations, relying on such mechanism
by default would lead to applications having to incorporate a lot of
business logic to handle cases where annotations are mixed, because they
are unpredictable by design. In other terms, I think it would work well for
applications supporting low-level building block models like STC, but
really bad for applications of complex, higher level objects like Cubes or
Time Series. We are after something that would work equally well at both
levels.

The main argument you have brought up for decoupling DMs completely has
been, in my opinion, that tight coupling would make the DMs evolve very
slowly. First of all, I would think that's a plus. We are now in a hot
transitional period with a lot of models under review or constructions, but
we can't simply pop new baseline models (or major revisions of them) all
the time. Major releases should be baked only when really necessary to add
new value (e.g. STC2) and new models should be baked only to cover new
domains (Source, Cube) or to provide site-specific extensions to baseline,
standard models (e.g. SdssSource, ChandraCube).

But I don't think your point is valid to begin with: we now have a
framework that allows us to swiftly and neatly produce new models, at least
from a technical point of view. Whether or not the IVOA processes that lead
to a new model or a revision of an existing one are as swift is a whole
different story, and as I argued above I would beware of too much swiftness
in that respect.

By the way, I agree we need a model for complex quantities. Inside or
outside STC I don't think I care too much, as long as they are defined
independently of specific domains for a broad range of usages.

I really think we could use some concrete examples, because an image (or an
XML document) is worth 1000 words (probably 1500 in XML ;-) ). I'll try to
come up with a couple of examples in a not-too-distant future.

The references to your "native" elements don't strike me as particularly
interoperable, unless they are formalized so they can work with different
formats and proper mappings. One thing is to say that "the type dali:Point
is mapped to a such and such PARAM in VOTable", a different one (that I
don't like) is to say "there is something over there that I call <point>,
but you don't necessarily know what it is, except that it might be a DALI
point, but again, not sure".

I believe we agree on the general requirements, though, right? models and
instances (annotations) should be as loosely coupled as possible.
Applications that are aware of some model should find the relevant
annotations in any file, while ignoring all the other models. We should
have a framework that allows for agile definition of new models or
revisions of old models, although we won't pop a new model every three
weeks. And the goal is, above all, interoperability among models,
applications, and services.

Omar.

-- 
Omar Laurino
Smithsonian Astrophysical Observatory
Harvard-Smithsonian Center for Astrophysics
100 Acorn Park Dr. R-377 MS-81
02140 Cambridge, MA
(617) 495-7227
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20170322/a1644be6/attachment-0001.html>