ImageDM: what does extension mean?

[Laurino, Omar]

Hi Ray,

Of course, there is more to "extending" than just stating that you are
extending ;)

However, the answer to your question depends pretty much on how formal we
want the "extension" mechanism to be expressed in the document.

Informally, it could be enough if the document stated it would inherit the
concepts defined in ObsCore, while defining only new concepts. There is
also the possibility of overriding concepts in ObsCore in a reasonable way,
i.e. by only restricting their semantics (e.g. acceptable units in ImageDM
are a subset of the units acceptable in ObsCore).

While there may be satisfactory informal approaches, we have worked a great
deal to come up with a solution that works in a wide range of possible
cases (including embarrassingly-naive clients).

So, I believe that if we want to do something formal, we should leverage
the work done with VODML, which addresses exactly this case in a formal and
consistent way.

The first step is to produce meaningful UML diagrams using a well-defined
subset of UML, i.e. a profile (which perfectly maps to VODML, of course).
Drawing random boxes with random lines and interchangeable names is not an
option. Describing Data Models as lists of UTYPEs is not an option either.

The way I see it, we should express ObsCore in a VODML description file.
ImageDM can then *extend* classes defined in ObsCore by adding attributes
or restricting a field's semantics. Of Course, ImageDM can also introduce
its own classes.

ImageDM is a DM, so it should only be concerned with the DM stuff, not with
DAL stuff. This seems obvious, but in practice it is easy to cross the
demarcation line, and we can see this even in the current standards.

In practical terms, a DAL document (SIAv2?) could define the UTYPEs as we
have been discussing over the past months, i.e. as shortcuts into the Data
Model, so that these UTYPEs can be used in DAL responses by annotating
FIELDs and allowing naive clients to directly access the information they
are seeking. Actual serializations (or annotations) of the data files could
be described in terms of the Data Model using the UTYPEs defined in the
VODML description, in a more expressive and portable way.

Does this answer your question (Whether or not you like the answer is a
different issue ;) )? This is all high level, we can go into the low level
if you think we need to.

A note on what Doug suggests, or rather what I understand he is suggesting.

First: a Data Model is not a list of strings. Not in this universe, not in
any parallel universe. Secondly, creating thousands of new UTYPEs each time
you want to simply include a Data Model as a component of a different Data
Model does not help clientsand their developers. Quite the opposite. The
Current Usages document convincingly shows that the current UTYPEs do not
work for clients, unless the client is a data discovery tool, which
probably uses a couple of UTYPEs out of the thousands defined, in a context
that is already constrained by the protocol (you can't look for SSA UTYPEs
in a SIA response). So, in order to address Matthew's point we need to be
smarter.

In this case being smarter means, imho:
  - Decouple the Data Model from the serialization. Let's do what people in
the 21st century does.
  - Use something that makes it easier for clients to provide useful
functionality to the user. Statically linking tens of thousands of fixed
strings to Object Oriented Class fields is not smart, and it's quite error
prone. Let's do the right thing now. Ten years from now somebody might have
this conversation:
  - Hey, why didn't we do things right in 2013?
  - Well, for historical reasons. Ten years ago we decided to do things
this way.
  - Can we change them now?
  - No, this is how we have done things in the past, we shouldn't change.
  - But so why didn't we do things right in 2013?

Cheers,

Omar.


On Tue, Oct 29, 2013 at 1:11 PM, Douglas Tody <dtody at nrao.edu> wrote:

> On Tue, 29 Oct 2013, CresitelloDittmar, Mark wrote:
>
>  The composite models (Image, Spectral) should 'refer' to the component
>> models.
>> They can include summary or semantic details, but should defer to the
>> original
>> for definition and detail.  This reduces the 'bloat' in models and these
>> small variations
>> we are struggling with due to migrations and 'simplifications'.
>>
>> For the ObsCore extension, there are some details at the top level which
>> need to be resolved (see the 'Convergence' thread).  But this question
>> brings
>> up a good point w.r.t. serialization.  Serializations include the model
>> 'prefix'.
>> For ObsCore serializations, this is 'obs'?, other models will have a
>> different
>> prefix.  Optimally, an ObsCore aware application should be able to ignore
>> the
>> prefix and recognize the content.  This would probably work, but only
>> because
>> it is a TOP level model.
>>
>> This is similar to discussions which went on for lower level objects,
>> where the
>> practice of combining utype strings make it impossible for a generic
>> utility
>> to identify the axes (for example) within a model an plot them.
>>
>
> Things can get very complicated for a client application if they have to
> be prepared for any version of any data model (noted by the DM prefix)
> cropping up in a dataset instance.  That is what it means to open up the
> DM namespace in a fundamental way in a top level composite model like
> Observation, Image, Spectrum, etc.  Using a global "obs:" or "im:" for
> the top level model, and defining this model to explicitly include
> specified versions of the required referenced component models,
> constrains what is defined as part of the standard top level model while
> allowing re-use of standard component models.  It is still possible by
> extension to include instances of other models, but anything required by
> the specific DM application i.e. top level model (obstap, imagedm, etc.)
> is explicilty integrated into the composite model.
>
> This is enough to have a workable scheme, as we do now.  The next step
> might be to, *external to a DM instance*, formally define the data
> model, which would allow us to do things like note the relationship to
> other models in a machine-readable fashion.  This would allow the more
> advanced DM manipulations without complicating basic usage.  It is what
> I have always hoped the new formalized Utype mechanism would permit, and
> would be backwards compatible with our current models, addressing the
> issue Matthew mentions.  Hopefully something like this can come along in
> the future, but it is out of scope for what we are doing to get
> something usable for image/cube access this fall.
>
>         - Doug




-- 
Omar Laurino
Smithsonian Astrophysical Observatory
Harvard-Smithsonian Center for Astrophysics
100 Acorn Park Dr. R-377 MS-81
02140 Cambridge, MA
(617) 495-7227
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ivoa.net/pipermail/dm/attachments/20131029/b4bb9bb7/attachment-0001.html>