Revised ImageDM-ObsCore architecture

[Laurino, Omar]

Doug,

This was a very down-to-earth discussion about a concrete problem we are
trying to solve. I hope it can get back to that.

I guess your last comment settles it: before changing current standards,
let's think twice.

I believe we can proceed with the current version of ObsCore, but of course
I would like to hear what Jesus and the others have to say: as far as I am
concerned we need more compelling reasons to create a new major release of
ObsCore than the ones you are offering.

And even if we did consider a new major release of ObsCore, that should be
a thourough process, not a rushed last-minute call.

Omar.
On Nov 12, 2013 8:39 PM, "Douglas Tody" <dtody at nrao.edu> wrote:

> I agree that the WG Chairs and the TCG ultimately need to sign-off on
> any decision about the eventual architecture and standards.  The author
> teams are merely doing a detailed analysis and recommendation for how to
> proceed.  It will be interesting to see where those most directly
> involved stand on all this several days from now as the discussion is
> just getting down to the key details.  Meanwhile the world will not wait
> for us; we have 4-5 FTEs at least scheduled for Cube work over the next
> year and we urgently need to get on with development for real projects,
> with some expectation that the standards will not completely change out
> from under us while this is going on.  Some sort of interim
> update/solution is not an option at this point.
>
>         - Doug
>
>
> On Tue, 12 Nov 2013, Matthew Graham wrote:
>
>  Hi,
>>
>> There is a process here that we follow so that we can coordinate the
>> international effort and prevent people going off and doing what they
>> believe is needed on their own. The work path of the IVOA WGs is not
>> determined by the individual members of the group but by the WG chairs. If
>> ObsCore needs updating then this has to ultimately come from the WG Chair
>> or Deputy and be on the IVOA Roadmap. I would therefore urge the ImageDM
>> authors to discuss this with the DM Chairs at their earliest convenience.
>>
>>         Cheers,
>>
>>         Matthew
>>
>>
>>
>>
>> On Nov 12, 2013, at 4:51 PM, Douglas Tody wrote:
>>
>>  Hi Omar -
>>>
>>> This is degerating into a meta-discussion and we don't have time for
>>> that, so I won't respond in detail.
>>>
>>> The agreement coming out of the interop was that Image and Spectral
>>> should (one way or the other) be an extension of ObsCore, to avoid
>>> duplication/inconsistency and enhance interoperability, e.g., between
>>> ObsTAP and SIA-based components.
>>>
>>> There are two ways to do this: either we leave ObsCore unchanged and
>>> layer ImageDM/SpectralDM, or we update ObsCore as needed to reflect more
>>> recent DM work (e.g., Char2), incorporate any missing
>>> generic/Observation metadata, and limit Image/Spectral to what is
>>> specific to each class of data.  My original tendency was to do the
>>> former but the consensus of the ImageDM co-authors (see discussion
>>> archived on the ImageDM TWiki page), which includes most of the original
>>> ObsCore main authors, was the latter.  Yes there is an impact on
>>> existing implementations, but on the other hand Image and Spectral are
>>> major developments and an update to ObsCore may be timely in any case as
>>> it has been a while.
>>>
>>> The Utype names are based directly upon data model class names, however
>>> these evolve hence we get inconsistencies.  There are also choices to be
>>> make regarding how to expose an element as either a reference value
>>> (required for simple use cases), or something more complicated and fully
>>> detailed.  This is not a simple autogenerated data model issue but
>>> requires some human judgment.  Futrher integrating independently
>>> developed and use-cased models like Observation/Char and STC is not
>>> trivial.
>>>
>>> While a data model is more than just a set of Utypes and their
>>> shortnames (e.g. DBMS table column names), these are essential to our
>>> applications and archives, and greatly impact the user community outside
>>> our little VO club.  Annoying as it may be, we need to pin this stuff
>>> down to support real-world applications.
>>>
>>> We *will*, one way or the other, have a stable version of this to
>>> support critical Cube project development and uptake by the community,
>>> while these debates rage for the next 6-24 months.
>>>
>>>         - Doug
>>>
>>>
>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>
>>>  Hi Doug,
>>>>
>>>> I am not suggesting that this is all about Utypes or that we don't need
>>>>
>>>>> a formal class hierarchy as the basis for the data model.
>>>>>
>>>>
>>>>
>>>> Well, you explicitly wrote this:
>>>>
>>>>  The consensus earlier is that an update to ObsCore *is* required, as
>>>>> there are issues with a number of the Utypes in that version of the
>>>>> model.
>>>>>
>>>>
>>>>
>>>> There is no such consensus. And you name UTYPEs as the main reason. So,
>>>> it
>>>> looks
>>>> like it's all about UTYPEs according to you.
>>>>
>>>>
>>>>  I suggest
>>>>> that what I posted over the weekend is a valid, even simple, data model
>>>>> architecture basing Image on Observation as we agreed to do in the
>>>>> interop.  Have you actually look at this?
>>>>>
>>>>>
>>>> Yes, I did, and in my opinion it was a regression from what Mark was
>>>> trying
>>>> to achieve. Hence my comment.
>>>>
>>>>
>>>>
>>>>> While a data model is more than just Utypes, we do have a problem with
>>>>> Utypes in the current ObsCore/ImageDM/SpectralDM/Char2.  The problem
>>>>> is
>>>>> relatively minor (e.g., "Calib" vs "Calibration", Coord as an array
>>>>> value vs separated .C1, and .C2 attributes, and the like)
>>>>>
>>>>
>>>>
>>>> I won't comment this anymore. This whole effort is not about nice names.
>>>>
>>>> I already have to come up with a name for my son, I think that's enough
>>>> without having to choose between Calib and Calibration :)
>>>>
>>>>
>>>> however it
>>>>
>>>>> will require an update to ObsCore synchronized with Image/Spectral to
>>>>> fix the problem.  Otherwise we will have to propagate stuff from
>>>>> ObsCore
>>>>> 1.0 which the authors now agree is invalid.
>>>>>
>>>>>
>>>>>  First, there is no such thing as an ObsCore update in the Roadmap. If
>>>> I am
>>>> mistaken, the TCG chairs will prove me wrong.
>>>>
>>>> Second, you don't need to change the ObsCore UTYPEs in order to have a
>>>> sound Data Model for ImageDM, so your statement is false: it is not
>>>> required.
>>>>
>>>> Third, the DM WG is trying to adopt a mechanism that allows to "fix the
>>>> problem" without changing Data Models every time you don't like a
>>>> string.
>>>>
>>>> Fourth, if you change ObsCore, then you need to consider the impact on
>>>> all
>>>> the existing implementations. Since you are not solving any actual
>>>> problems
>>>> by turning "Calib" into "Calibration", you are not presenting any
>>>> compelling reasons to update the ObsCore DM.
>>>>
>>>> Plus, the consensus was to build on top of ObsCore, not to change it.
>>>>
>>>>
>>>>
>>>>  I don't have any problem with proceeding with (or at least
>>>>> investigating) a more rigorous approach based upon VO-DML - what Gerard
>>>>> posted earlier was quite promising.  However I doubt very much if we
>>>>> can
>>>>> complete that this week, and we need a stable update to the ImageDM by
>>>>> the end of the week (or at least end of Nov at the latest) to support
>>>>> cube project development.
>>>>>
>>>>>
>>>> If you don't try, you won't ever know. Also, it was agreed that we would
>>>> provide such descriptions anyway. Whether they describe your concepts or
>>>> Mark's, that does not make any difference.
>>>>
>>>> Also, Mark's suggestion, which is simpler and does not require changing
>>>> current recommendations, was already in the right direction. Again, if
>>>> you
>>>> have points that do not involve changing strings, let's consider them
>>>> together.
>>>>
>>>> Anyway, I am surprised that you consider being rigorous as an option you
>>>> can procrastinate forever, and not a professional requirement. We need
>>>> to
>>>> be rigorous, otherwise we are just playing games and wasting time.
>>>>
>>>>
>>>>
>>>>> The main thing is that the data model *architecture* must not change
>>>>> greatly.
>>>>>
>>>>
>>>>
>>>> Which is exactly what Mark is doing. Changing ObsCore means changing the
>>>> architecture. And would require a major release, since backward
>>>> compatibility would be broken. That's a big change you are suggesting.
>>>>
>>>>
>>>>  Hence if we really want to integrate Image/Spectral and
>>>>> Observation (Image formally extending Observation as agreed in the
>>>>> interop) then we need a usable first cut at that *now*.
>>>>>
>>>>
>>>>
>>>> Again, Mark's suggestion is *now*. But it is also simpler, and doesn't
>>>> require changes to the architecture.
>>>>
>>>>
>>>>  But the IVOA consensus to move in this direction was very
>>>>> clear.
>>>>>
>>>>>
>>>> I don't see anything in the IVOA roadmap that involves reopening the REC
>>>> process for ObsCore. So, I am not sure where your statement, in the
>>>> name of
>>>> the IVOA, comes from.
>>>>
>>>> Omar.
>>>>
>>>>
>>>>
>>>>>        - Doug
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>>>
>>>>>  The consensus earlier is that an update to ObsCore *is* required, as
>>>>>>
>>>>>>> there are issues with a number of the Utypes in that version of the
>>>>>>> model.
>>>>>>>
>>>>>>>
>>>>>> No, no, and no :)
>>>>>>
>>>>>> You fix the model if you need to fix the model. And models are not
>>>>>> list of
>>>>>> UTYPEs.
>>>>>> Ergo, you don't fix a model if you don't like the UTYPEs. And we don't
>>>>>> "model by UTYPEs" anymore.
>>>>>> That's the consensus.
>>>>>>
>>>>>> On the other hand, if what you are saying is true, you are suggesting
>>>>>> that
>>>>>> we *need* to redefine almost all of the IVOA Data Models in order to
>>>>>> create
>>>>>> SIAPv2, for one reason or the other.
>>>>>>
>>>>>> I hope this is not true (and I am sure this was not the consensus,
>>>>>> anyway).
>>>>>> But if it is, then let's get it really right this time. Which means:
>>>>>> we
>>>>>> don't "model by UTYPEs".
>>>>>>
>>>>>>
>>>>>> Then the ImageDM
>>>>>>
>>>>>>> architecture is much simpler, as most of it is just ObsCore and Char.
>>>>>>>
>>>>>>>
>>>>>>>  This is a non-sequitur. Assuming that by architecture you refer to
>>>>>> the
>>>>>> class diagram, the class diagram does not need to be simple: it needs
>>>>>> to
>>>>>> meet the requirements in the most efficient way, which is different.
>>>>>>
>>>>>> Also, Mark's diagram doesn't seem complicated, so you cannot use
>>>>>> Occam's
>>>>>> razor in this case.
>>>>>>
>>>>>> Actually, Mark's diagram doesn't require to review ObsCore, ergo it is
>>>>>> simpler.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  This should be more clear in the revised DM spreadsheet; I hope to
>>>>>>> post
>>>>>>> that later today.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I don't think anybody cares about the list of UTYPEs, at this point,
>>>>>> as
>>>>>> Mireille suggested.
>>>>>>
>>>>>> Let's figure out how the model needs to look like, then I am sure we
>>>>>> will
>>>>>> find a solution for the UTYPEs (which, as you have always been
>>>>>> adamant,
>>>>>> are
>>>>>> not to be parsed).
>>>>>> Unless you want to "model by UTYPES", which is something that the DM
>>>>>> WG
>>>>>> doesn't do anymore.
>>>>>>
>>>>>> There is a difference in what you and Mark are suggesting. Unless you
>>>>>> provide some actual arguments showing why your approach is better
>>>>>> (arguments that would have nothing to do with UTYPEs), I suggest that
>>>>>> we
>>>>>> adopt Mark's approach, which is simpler and is backed up by sound data
>>>>>> modeling reasoning.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I need to review your new figures but I think the obvious class names
>>>>>>
>>>>>>> are Observation, Characterisation, Image, etc.  Mireille suggested
>>>>>>> earlier that the "Obs.x" element in Observation (containing
>>>>>>> dataProductType etc.) could be replaced by the more general
>>>>>>> Dataset.x.
>>>>>>>
>>>>>>>
>>>>>>> How can a name be more general than another?
>>>>>>>
>>>>>>
>>>>>> "What's in a name? That which we call a rose by any other name would
>>>>>> smell
>>>>>> as sweet."
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Omar.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Omar Laurino
>>>>>> Smithsonian Astrophysical Observatory
>>>>>> Harvard-Smithsonian Center for Astrophysics
>>>>>> 100 Acorn Park Dr. R-377 MS-81
>>>>>> 02140 Cambridge, MA
>>>>>> (617) 495-7227
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Omar Laurino
>>>> Smithsonian Astrophysical Observatory
>>>> Harvard-Smithsonian Center for Astrophysics
>>>> 100 Acorn Park Dr. R-377 MS-81
>>>> 02140 Cambridge, MA
>>>> (617) 495-7227
>>>>
>>>>
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ivoa.net/pipermail/dm/attachments/20131112/23b53095/attachment-0001.html>