Revised ImageDM-ObsCore architecture

[Douglas Tody]

I am not the only one proposing this; the suggestion originates mainly
with Mireille and Francois, the lead authors for ObsCore / ObservationDM
and Char/Char2.  But I support this path if we really want to integrate
Image/Spectral and ObsCore.

All that is being proposed here is a minor update to ObsCore to improve
compatibility with more recent work done within DM on other models such
as Char2 and STC.  Someone not intimately familiar with the standard
would likely not even notice the difference.  We can prototype this as
part of SIAV2 development, and update the ObsCore/ObsTAP standard to a
1.1 version over the course of the next year.  Otherwise the opportunity
will pass, and we will propagate known issues with ObsCore into the new
standards and all our implementations for image/cube access etc.
Frankly I could probably live with either approach so long as we get on
with it, but I sympathize with those trying to fix known issues.

I suggest we get back to finishing up specifying the proposed revisions
and then we will be in a better position to evaluate the aproach.  At
this point we are delayed another day getting the details of the revised
data model posted.

 	- Doug


On Tue, 12 Nov 2013, Laurino, Omar wrote:

> Doug,
>
> This was a very down-to-earth discussion about a concrete problem we are
> trying to solve. I hope it can get back to that.
>
> I guess your last comment settles it: before changing current standards,
> let's think twice.
>
> I believe we can proceed with the current version of ObsCore, but of course
> I would like to hear what Jesus and the others have to say: as far as I am
> concerned we need more compelling reasons to create a new major release of
> ObsCore than the ones you are offering.
>
> And even if we did consider a new major release of ObsCore, that should be
> a thourough process, not a rushed last-minute call.
>
> Omar.
> On Nov 12, 2013 8:39 PM, "Douglas Tody" <dtody at nrao.edu> wrote:
>
>> I agree that the WG Chairs and the TCG ultimately need to sign-off on
>> any decision about the eventual architecture and standards.  The author
>> teams are merely doing a detailed analysis and recommendation for how to
>> proceed.  It will be interesting to see where those most directly
>> involved stand on all this several days from now as the discussion is
>> just getting down to the key details.  Meanwhile the world will not wait
>> for us; we have 4-5 FTEs at least scheduled for Cube work over the next
>> year and we urgently need to get on with development for real projects,
>> with some expectation that the standards will not completely change out
>> from under us while this is going on.  Some sort of interim
>> update/solution is not an option at this point.
>>
>>         - Doug
>>
>>
>> On Tue, 12 Nov 2013, Matthew Graham wrote:
>>
>>  Hi,
>>>
>>> There is a process here that we follow so that we can coordinate the
>>> international effort and prevent people going off and doing what they
>>> believe is needed on their own. The work path of the IVOA WGs is not
>>> determined by the individual members of the group but by the WG chairs. If
>>> ObsCore needs updating then this has to ultimately come from the WG Chair
>>> or Deputy and be on the IVOA Roadmap. I would therefore urge the ImageDM
>>> authors to discuss this with the DM Chairs at their earliest convenience.
>>>
>>>         Cheers,
>>>
>>>         Matthew
>>>
>>>
>>>
>>>
>>> On Nov 12, 2013, at 4:51 PM, Douglas Tody wrote:
>>>
>>>  Hi Omar -
>>>>
>>>> This is degerating into a meta-discussion and we don't have time for
>>>> that, so I won't respond in detail.
>>>>
>>>> The agreement coming out of the interop was that Image and Spectral
>>>> should (one way or the other) be an extension of ObsCore, to avoid
>>>> duplication/inconsistency and enhance interoperability, e.g., between
>>>> ObsTAP and SIA-based components.
>>>>
>>>> There are two ways to do this: either we leave ObsCore unchanged and
>>>> layer ImageDM/SpectralDM, or we update ObsCore as needed to reflect more
>>>> recent DM work (e.g., Char2), incorporate any missing
>>>> generic/Observation metadata, and limit Image/Spectral to what is
>>>> specific to each class of data.  My original tendency was to do the
>>>> former but the consensus of the ImageDM co-authors (see discussion
>>>> archived on the ImageDM TWiki page), which includes most of the original
>>>> ObsCore main authors, was the latter.  Yes there is an impact on
>>>> existing implementations, but on the other hand Image and Spectral are
>>>> major developments and an update to ObsCore may be timely in any case as
>>>> it has been a while.
>>>>
>>>> The Utype names are based directly upon data model class names, however
>>>> these evolve hence we get inconsistencies.  There are also choices to be
>>>> make regarding how to expose an element as either a reference value
>>>> (required for simple use cases), or something more complicated and fully
>>>> detailed.  This is not a simple autogenerated data model issue but
>>>> requires some human judgment.  Futrher integrating independently
>>>> developed and use-cased models like Observation/Char and STC is not
>>>> trivial.
>>>>
>>>> While a data model is more than just a set of Utypes and their
>>>> shortnames (e.g. DBMS table column names), these are essential to our
>>>> applications and archives, and greatly impact the user community outside
>>>> our little VO club.  Annoying as it may be, we need to pin this stuff
>>>> down to support real-world applications.
>>>>
>>>> We *will*, one way or the other, have a stable version of this to
>>>> support critical Cube project development and uptake by the community,
>>>> while these debates rage for the next 6-24 months.
>>>>
>>>>         - Doug
>>>>
>>>>
>>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>>
>>>>  Hi Doug,
>>>>>
>>>>> I am not suggesting that this is all about Utypes or that we don't need
>>>>>
>>>>>> a formal class hierarchy as the basis for the data model.
>>>>>>
>>>>>
>>>>>
>>>>> Well, you explicitly wrote this:
>>>>>
>>>>>  The consensus earlier is that an update to ObsCore *is* required, as
>>>>>> there are issues with a number of the Utypes in that version of the
>>>>>> model.
>>>>>>
>>>>>
>>>>>
>>>>> There is no such consensus. And you name UTYPEs as the main reason. So,
>>>>> it
>>>>> looks
>>>>> like it's all about UTYPEs according to you.
>>>>>
>>>>>
>>>>>  I suggest
>>>>>> that what I posted over the weekend is a valid, even simple, data model
>>>>>> architecture basing Image on Observation as we agreed to do in the
>>>>>> interop.  Have you actually look at this?
>>>>>>
>>>>>>
>>>>> Yes, I did, and in my opinion it was a regression from what Mark was
>>>>> trying
>>>>> to achieve. Hence my comment.
>>>>>
>>>>>
>>>>>
>>>>>> While a data model is more than just Utypes, we do have a problem with
>>>>>> Utypes in the current ObsCore/ImageDM/SpectralDM/Char2.  The problem
>>>>>> is
>>>>>> relatively minor (e.g., "Calib" vs "Calibration", Coord as an array
>>>>>> value vs separated .C1, and .C2 attributes, and the like)
>>>>>>
>>>>>
>>>>>
>>>>> I won't comment this anymore. This whole effort is not about nice names.
>>>>>
>>>>> I already have to come up with a name for my son, I think that's enough
>>>>> without having to choose between Calib and Calibration :)
>>>>>
>>>>>
>>>>> however it
>>>>>
>>>>>> will require an update to ObsCore synchronized with Image/Spectral to
>>>>>> fix the problem.  Otherwise we will have to propagate stuff from
>>>>>> ObsCore
>>>>>> 1.0 which the authors now agree is invalid.
>>>>>>
>>>>>>
>>>>>>  First, there is no such thing as an ObsCore update in the Roadmap. If
>>>>> I am
>>>>> mistaken, the TCG chairs will prove me wrong.
>>>>>
>>>>> Second, you don't need to change the ObsCore UTYPEs in order to have a
>>>>> sound Data Model for ImageDM, so your statement is false: it is not
>>>>> required.
>>>>>
>>>>> Third, the DM WG is trying to adopt a mechanism that allows to "fix the
>>>>> problem" without changing Data Models every time you don't like a
>>>>> string.
>>>>>
>>>>> Fourth, if you change ObsCore, then you need to consider the impact on
>>>>> all
>>>>> the existing implementations. Since you are not solving any actual
>>>>> problems
>>>>> by turning "Calib" into "Calibration", you are not presenting any
>>>>> compelling reasons to update the ObsCore DM.
>>>>>
>>>>> Plus, the consensus was to build on top of ObsCore, not to change it.
>>>>>
>>>>>
>>>>>
>>>>>  I don't have any problem with proceeding with (or at least
>>>>>> investigating) a more rigorous approach based upon VO-DML - what Gerard
>>>>>> posted earlier was quite promising.  However I doubt very much if we
>>>>>> can
>>>>>> complete that this week, and we need a stable update to the ImageDM by
>>>>>> the end of the week (or at least end of Nov at the latest) to support
>>>>>> cube project development.
>>>>>>
>>>>>>
>>>>> If you don't try, you won't ever know. Also, it was agreed that we would
>>>>> provide such descriptions anyway. Whether they describe your concepts or
>>>>> Mark's, that does not make any difference.
>>>>>
>>>>> Also, Mark's suggestion, which is simpler and does not require changing
>>>>> current recommendations, was already in the right direction. Again, if
>>>>> you
>>>>> have points that do not involve changing strings, let's consider them
>>>>> together.
>>>>>
>>>>> Anyway, I am surprised that you consider being rigorous as an option you
>>>>> can procrastinate forever, and not a professional requirement. We need
>>>>> to
>>>>> be rigorous, otherwise we are just playing games and wasting time.
>>>>>
>>>>>
>>>>>
>>>>>> The main thing is that the data model *architecture* must not change
>>>>>> greatly.
>>>>>>
>>>>>
>>>>>
>>>>> Which is exactly what Mark is doing. Changing ObsCore means changing the
>>>>> architecture. And would require a major release, since backward
>>>>> compatibility would be broken. That's a big change you are suggesting.
>>>>>
>>>>>
>>>>>  Hence if we really want to integrate Image/Spectral and
>>>>>> Observation (Image formally extending Observation as agreed in the
>>>>>> interop) then we need a usable first cut at that *now*.
>>>>>>
>>>>>
>>>>>
>>>>> Again, Mark's suggestion is *now*. But it is also simpler, and doesn't
>>>>> require changes to the architecture.
>>>>>
>>>>>
>>>>>  But the IVOA consensus to move in this direction was very
>>>>>> clear.
>>>>>>
>>>>>>
>>>>> I don't see anything in the IVOA roadmap that involves reopening the REC
>>>>> process for ObsCore. So, I am not sure where your statement, in the
>>>>> name of
>>>>> the IVOA, comes from.
>>>>>
>>>>> Omar.
>>>>>
>>>>>
>>>>>
>>>>>>        - Doug
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>>>>
>>>>>>  The consensus earlier is that an update to ObsCore *is* required, as
>>>>>>>
>>>>>>>> there are issues with a number of the Utypes in that version of the
>>>>>>>> model.
>>>>>>>>
>>>>>>>>
>>>>>>> No, no, and no :)
>>>>>>>
>>>>>>> You fix the model if you need to fix the model. And models are not
>>>>>>> list of
>>>>>>> UTYPEs.
>>>>>>> Ergo, you don't fix a model if you don't like the UTYPEs. And we don't
>>>>>>> "model by UTYPEs" anymore.
>>>>>>> That's the consensus.
>>>>>>>
>>>>>>> On the other hand, if what you are saying is true, you are suggesting
>>>>>>> that
>>>>>>> we *need* to redefine almost all of the IVOA Data Models in order to
>>>>>>> create
>>>>>>> SIAPv2, for one reason or the other.
>>>>>>>
>>>>>>> I hope this is not true (and I am sure this was not the consensus,
>>>>>>> anyway).
>>>>>>> But if it is, then let's get it really right this time. Which means:
>>>>>>> we
>>>>>>> don't "model by UTYPEs".
>>>>>>>
>>>>>>>
>>>>>>> Then the ImageDM
>>>>>>>
>>>>>>>> architecture is much simpler, as most of it is just ObsCore and Char.
>>>>>>>>
>>>>>>>>
>>>>>>>>  This is a non-sequitur. Assuming that by architecture you refer to
>>>>>>> the
>>>>>>> class diagram, the class diagram does not need to be simple: it needs
>>>>>>> to
>>>>>>> meet the requirements in the most efficient way, which is different.
>>>>>>>
>>>>>>> Also, Mark's diagram doesn't seem complicated, so you cannot use
>>>>>>> Occam's
>>>>>>> razor in this case.
>>>>>>>
>>>>>>> Actually, Mark's diagram doesn't require to review ObsCore, ergo it is
>>>>>>> simpler.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  This should be more clear in the revised DM spreadsheet; I hope to
>>>>>>>> post
>>>>>>>> that later today.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> I don't think anybody cares about the list of UTYPEs, at this point,
>>>>>>> as
>>>>>>> Mireille suggested.
>>>>>>>
>>>>>>> Let's figure out how the model needs to look like, then I am sure we
>>>>>>> will
>>>>>>> find a solution for the UTYPEs (which, as you have always been
>>>>>>> adamant,
>>>>>>> are
>>>>>>> not to be parsed).
>>>>>>> Unless you want to "model by UTYPES", which is something that the DM
>>>>>>> WG
>>>>>>> doesn't do anymore.
>>>>>>>
>>>>>>> There is a difference in what you and Mark are suggesting. Unless you
>>>>>>> provide some actual arguments showing why your approach is better
>>>>>>> (arguments that would have nothing to do with UTYPEs), I suggest that
>>>>>>> we
>>>>>>> adopt Mark's approach, which is simpler and is backed up by sound data
>>>>>>> modeling reasoning.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I need to review your new figures but I think the obvious class names
>>>>>>>
>>>>>>>> are Observation, Characterisation, Image, etc.  Mireille suggested
>>>>>>>> earlier that the "Obs.x" element in Observation (containing
>>>>>>>> dataProductType etc.) could be replaced by the more general
>>>>>>>> Dataset.x.
>>>>>>>>
>>>>>>>>
>>>>>>>> How can a name be more general than another?
>>>>>>>>
>>>>>>>
>>>>>>> "What's in a name? That which we call a rose by any other name would
>>>>>>> smell
>>>>>>> as sweet."
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Omar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Omar Laurino
>>>>>>> Smithsonian Astrophysical Observatory
>>>>>>> Harvard-Smithsonian Center for Astrophysics
>>>>>>> 100 Acorn Park Dr. R-377 MS-81
>>>>>>> 02140 Cambridge, MA
>>>>>>> (617) 495-7227
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Omar Laurino
>>>>> Smithsonian Astrophysical Observatory
>>>>> Harvard-Smithsonian Center for Astrophysics
>>>>> 100 Acorn Park Dr. R-377 MS-81
>>>>> 02140 Cambridge, MA
>>>>> (617) 495-7227
>>>>>
>>>>>
>>>>
>>>
>>>
>