Revised ImageDM-ObsCore architecture

[François Bonnarel]

Hi all,
      Just my two words on this discussion. Doug is perfectly right on 
all that.
      The question is not about utypes per se.
      It's about model reuse and specifically how Obscore reused Char 
and how Char reuses Stc.

     1 ) Char 1 (see recommandation page 21, 22) reuses STc class 
instances in Coverage.location , coverage.bounds and coverage.support
      Actually the idea is that *****.Location.Coord is an instance of 
stc AstroCoords. *****.Support.Limits is an instance of stc 
AstroCoordArea type.

     2 ) When we built ObsCore, Mireille, me and some others had 
allready started the Char2 effort which was (still is, see preliminary 
Working draft)
              a ) a cleaning effort on Char 1. (little inconsistencies 
here and there, most often at the low level)
              b ) an extension towards characterisation variability and 
complex data
         So under the contradictory pressure of cleaning up Char and 
providing utype strings for Obscore mimicking the structure we made some 
small mistakes were done. At that time 2009/2010 nobody was 
reconsidering that utypes should follow the path along the structure 
made of classes, attributes and associations roles.

     3 ) currently, naming the attributes by path-utypes is a convenient 
way to do, fairly understandable by looking to the UML. (at least in 
ImageDM). But the real question is about: what are the  attributes we 
need. What is tehir type. simple ? or borrowed to another model ?
How does this organize in classes. What are the relationships between 
this classes ? Everybody agrees this conceptualisation has to be 
represented in UML first. Representations, utypes will come later

Cheers
François



Le 13/11/2013 05:32, Douglas Tody a écrit :
> I am not the only one proposing this; the suggestion originates mainly
> with Mireille and Francois, the lead authors for ObsCore / ObservationDM
> and Char/Char2.  But I support this path if we really want to integrate
> Image/Spectral and ObsCore.
>
> All that is being proposed here is a minor update to ObsCore to improve
> compatibility with more recent work done within DM on other models such
> as Char2 and STC.  Someone not intimately familiar with the standard
> would likely not even notice the difference.  We can prototype this as
> part of SIAV2 development, and update the ObsCore/ObsTAP standard to a
> 1.1 version over the course of the next year.  Otherwise the opportunity
> will pass, and we will propagate known issues with ObsCore into the new
> standards and all our implementations for image/cube access etc.
> Frankly I could probably live with either approach so long as we get on
> with it, but I sympathize with those trying to fix known issues.
>
> I suggest we get back to finishing up specifying the proposed revisions
> and then we will be in a better position to evaluate the aproach.  At
> this point we are delayed another day getting the details of the revised
> data model posted.
>
>     - Doug
>
>
> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>
>> Doug,
>>
>> This was a very down-to-earth discussion about a concrete problem we are
>> trying to solve. I hope it can get back to that.
>>
>> I guess your last comment settles it: before changing current standards,
>> let's think twice.
>>
>> I believe we can proceed with the current version of ObsCore, but of 
>> course
>> I would like to hear what Jesus and the others have to say: as far as 
>> I am
>> concerned we need more compelling reasons to create a new major 
>> release of
>> ObsCore than the ones you are offering.
>>
>> And even if we did consider a new major release of ObsCore, that 
>> should be
>> a thourough process, not a rushed last-minute call.
>>
>> Omar.
>> On Nov 12, 2013 8:39 PM, "Douglas Tody" <dtody at nrao.edu> wrote:
>>
>>> I agree that the WG Chairs and the TCG ultimately need to sign-off on
>>> any decision about the eventual architecture and standards.  The author
>>> teams are merely doing a detailed analysis and recommendation for 
>>> how to
>>> proceed.  It will be interesting to see where those most directly
>>> involved stand on all this several days from now as the discussion is
>>> just getting down to the key details.  Meanwhile the world will not 
>>> wait
>>> for us; we have 4-5 FTEs at least scheduled for Cube work over the next
>>> year and we urgently need to get on with development for real projects,
>>> with some expectation that the standards will not completely change out
>>> from under us while this is going on.  Some sort of interim
>>> update/solution is not an option at this point.
>>>
>>>         - Doug
>>>
>>>
>>> On Tue, 12 Nov 2013, Matthew Graham wrote:
>>>
>>>  Hi,
>>>>
>>>> There is a process here that we follow so that we can coordinate the
>>>> international effort and prevent people going off and doing what they
>>>> believe is needed on their own. The work path of the IVOA WGs is not
>>>> determined by the individual members of the group but by the WG 
>>>> chairs. If
>>>> ObsCore needs updating then this has to ultimately come from the WG 
>>>> Chair
>>>> or Deputy and be on the IVOA Roadmap. I would therefore urge the 
>>>> ImageDM
>>>> authors to discuss this with the DM Chairs at their earliest 
>>>> convenience.
>>>>
>>>>         Cheers,
>>>>
>>>>         Matthew
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 12, 2013, at 4:51 PM, Douglas Tody wrote:
>>>>
>>>>  Hi Omar -
>>>>>
>>>>> This is degerating into a meta-discussion and we don't have time for
>>>>> that, so I won't respond in detail.
>>>>>
>>>>> The agreement coming out of the interop was that Image and Spectral
>>>>> should (one way or the other) be an extension of ObsCore, to avoid
>>>>> duplication/inconsistency and enhance interoperability, e.g., between
>>>>> ObsTAP and SIA-based components.
>>>>>
>>>>> There are two ways to do this: either we leave ObsCore unchanged and
>>>>> layer ImageDM/SpectralDM, or we update ObsCore as needed to 
>>>>> reflect more
>>>>> recent DM work (e.g., Char2), incorporate any missing
>>>>> generic/Observation metadata, and limit Image/Spectral to what is
>>>>> specific to each class of data.  My original tendency was to do the
>>>>> former but the consensus of the ImageDM co-authors (see discussion
>>>>> archived on the ImageDM TWiki page), which includes most of the 
>>>>> original
>>>>> ObsCore main authors, was the latter.  Yes there is an impact on
>>>>> existing implementations, but on the other hand Image and Spectral 
>>>>> are
>>>>> major developments and an update to ObsCore may be timely in any 
>>>>> case as
>>>>> it has been a while.
>>>>>
>>>>> The Utype names are based directly upon data model class names, 
>>>>> however
>>>>> these evolve hence we get inconsistencies.  There are also choices 
>>>>> to be
>>>>> make regarding how to expose an element as either a reference value
>>>>> (required for simple use cases), or something more complicated and 
>>>>> fully
>>>>> detailed.  This is not a simple autogenerated data model issue but
>>>>> requires some human judgment.  Futrher integrating independently
>>>>> developed and use-cased models like Observation/Char and STC is not
>>>>> trivial.
>>>>>
>>>>> While a data model is more than just a set of Utypes and their
>>>>> shortnames (e.g. DBMS table column names), these are essential to our
>>>>> applications and archives, and greatly impact the user community 
>>>>> outside
>>>>> our little VO club.  Annoying as it may be, we need to pin this stuff
>>>>> down to support real-world applications.
>>>>>
>>>>> We *will*, one way or the other, have a stable version of this to
>>>>> support critical Cube project development and uptake by the 
>>>>> community,
>>>>> while these debates rage for the next 6-24 months.
>>>>>
>>>>>         - Doug
>>>>>
>>>>>
>>>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>>>
>>>>>  Hi Doug,
>>>>>>
>>>>>> I am not suggesting that this is all about Utypes or that we 
>>>>>> don't need
>>>>>>
>>>>>>> a formal class hierarchy as the basis for the data model.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Well, you explicitly wrote this:
>>>>>>
>>>>>>  The consensus earlier is that an update to ObsCore *is* 
>>>>>> required, as
>>>>>>> there are issues with a number of the Utypes in that version of the
>>>>>>> model.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> There is no such consensus. And you name UTYPEs as the main 
>>>>>> reason. So,
>>>>>> it
>>>>>> looks
>>>>>> like it's all about UTYPEs according to you.
>>>>>>
>>>>>>
>>>>>>  I suggest
>>>>>>> that what I posted over the weekend is a valid, even simple, 
>>>>>>> data model
>>>>>>> architecture basing Image on Observation as we agreed to do in the
>>>>>>> interop.  Have you actually look at this?
>>>>>>>
>>>>>>>
>>>>>> Yes, I did, and in my opinion it was a regression from what Mark was
>>>>>> trying
>>>>>> to achieve. Hence my comment.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> While a data model is more than just Utypes, we do have a 
>>>>>>> problem with
>>>>>>> Utypes in the current ObsCore/ImageDM/SpectralDM/Char2.  The 
>>>>>>> problem
>>>>>>> is
>>>>>>> relatively minor (e.g., "Calib" vs "Calibration", Coord as an array
>>>>>>> value vs separated .C1, and .C2 attributes, and the like)
>>>>>>>
>>>>>>
>>>>>>
>>>>>> I won't comment this anymore. This whole effort is not about nice 
>>>>>> names.
>>>>>>
>>>>>> I already have to come up with a name for my son, I think that's 
>>>>>> enough
>>>>>> without having to choose between Calib and Calibration :)
>>>>>>
>>>>>>
>>>>>> however it
>>>>>>
>>>>>>> will require an update to ObsCore synchronized with 
>>>>>>> Image/Spectral to
>>>>>>> fix the problem.  Otherwise we will have to propagate stuff from
>>>>>>> ObsCore
>>>>>>> 1.0 which the authors now agree is invalid.
>>>>>>>
>>>>>>>
>>>>>>>  First, there is no such thing as an ObsCore update in the 
>>>>>>> Roadmap. If
>>>>>> I am
>>>>>> mistaken, the TCG chairs will prove me wrong.
>>>>>>
>>>>>> Second, you don't need to change the ObsCore UTYPEs in order to 
>>>>>> have a
>>>>>> sound Data Model for ImageDM, so your statement is false: it is not
>>>>>> required.
>>>>>>
>>>>>> Third, the DM WG is trying to adopt a mechanism that allows to 
>>>>>> "fix the
>>>>>> problem" without changing Data Models every time you don't like a
>>>>>> string.
>>>>>>
>>>>>> Fourth, if you change ObsCore, then you need to consider the 
>>>>>> impact on
>>>>>> all
>>>>>> the existing implementations. Since you are not solving any actual
>>>>>> problems
>>>>>> by turning "Calib" into "Calibration", you are not presenting any
>>>>>> compelling reasons to update the ObsCore DM.
>>>>>>
>>>>>> Plus, the consensus was to build on top of ObsCore, not to change 
>>>>>> it.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  I don't have any problem with proceeding with (or at least
>>>>>>> investigating) a more rigorous approach based upon VO-DML - what 
>>>>>>> Gerard
>>>>>>> posted earlier was quite promising.  However I doubt very much 
>>>>>>> if we
>>>>>>> can
>>>>>>> complete that this week, and we need a stable update to the 
>>>>>>> ImageDM by
>>>>>>> the end of the week (or at least end of Nov at the latest) to 
>>>>>>> support
>>>>>>> cube project development.
>>>>>>>
>>>>>>>
>>>>>> If you don't try, you won't ever know. Also, it was agreed that 
>>>>>> we would
>>>>>> provide such descriptions anyway. Whether they describe your 
>>>>>> concepts or
>>>>>> Mark's, that does not make any difference.
>>>>>>
>>>>>> Also, Mark's suggestion, which is simpler and does not require 
>>>>>> changing
>>>>>> current recommendations, was already in the right direction. 
>>>>>> Again, if
>>>>>> you
>>>>>> have points that do not involve changing strings, let's consider 
>>>>>> them
>>>>>> together.
>>>>>>
>>>>>> Anyway, I am surprised that you consider being rigorous as an 
>>>>>> option you
>>>>>> can procrastinate forever, and not a professional requirement. We 
>>>>>> need
>>>>>> to
>>>>>> be rigorous, otherwise we are just playing games and wasting time.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> The main thing is that the data model *architecture* must not 
>>>>>>> change
>>>>>>> greatly.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Which is exactly what Mark is doing. Changing ObsCore means 
>>>>>> changing the
>>>>>> architecture. And would require a major release, since backward
>>>>>> compatibility would be broken. That's a big change you are 
>>>>>> suggesting.
>>>>>>
>>>>>>
>>>>>>  Hence if we really want to integrate Image/Spectral and
>>>>>>> Observation (Image formally extending Observation as agreed in the
>>>>>>> interop) then we need a usable first cut at that *now*.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Again, Mark's suggestion is *now*. But it is also simpler, and 
>>>>>> doesn't
>>>>>> require changes to the architecture.
>>>>>>
>>>>>>
>>>>>>  But the IVOA consensus to move in this direction was very
>>>>>>> clear.
>>>>>>>
>>>>>>>
>>>>>> I don't see anything in the IVOA roadmap that involves reopening 
>>>>>> the REC
>>>>>> process for ObsCore. So, I am not sure where your statement, in the
>>>>>> name of
>>>>>> the IVOA, comes from.
>>>>>>
>>>>>> Omar.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>        - Doug
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>>>>>>
>>>>>>>  The consensus earlier is that an update to ObsCore *is* 
>>>>>>> required, as
>>>>>>>>
>>>>>>>>> there are issues with a number of the Utypes in that version 
>>>>>>>>> of the
>>>>>>>>> model.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> No, no, and no :)
>>>>>>>>
>>>>>>>> You fix the model if you need to fix the model. And models are not
>>>>>>>> list of
>>>>>>>> UTYPEs.
>>>>>>>> Ergo, you don't fix a model if you don't like the UTYPEs. And 
>>>>>>>> we don't
>>>>>>>> "model by UTYPEs" anymore.
>>>>>>>> That's the consensus.
>>>>>>>>
>>>>>>>> On the other hand, if what you are saying is true, you are 
>>>>>>>> suggesting
>>>>>>>> that
>>>>>>>> we *need* to redefine almost all of the IVOA Data Models in 
>>>>>>>> order to
>>>>>>>> create
>>>>>>>> SIAPv2, for one reason or the other.
>>>>>>>>
>>>>>>>> I hope this is not true (and I am sure this was not the consensus,
>>>>>>>> anyway).
>>>>>>>> But if it is, then let's get it really right this time. Which 
>>>>>>>> means:
>>>>>>>> we
>>>>>>>> don't "model by UTYPEs".
>>>>>>>>
>>>>>>>>
>>>>>>>> Then the ImageDM
>>>>>>>>
>>>>>>>>> architecture is much simpler, as most of it is just ObsCore 
>>>>>>>>> and Char.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  This is a non-sequitur. Assuming that by architecture you 
>>>>>>>>> refer to
>>>>>>>> the
>>>>>>>> class diagram, the class diagram does not need to be simple: it 
>>>>>>>> needs
>>>>>>>> to
>>>>>>>> meet the requirements in the most efficient way, which is 
>>>>>>>> different.
>>>>>>>>
>>>>>>>> Also, Mark's diagram doesn't seem complicated, so you cannot use
>>>>>>>> Occam's
>>>>>>>> razor in this case.
>>>>>>>>
>>>>>>>> Actually, Mark's diagram doesn't require to review ObsCore, 
>>>>>>>> ergo it is
>>>>>>>> simpler.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  This should be more clear in the revised DM spreadsheet; I 
>>>>>>>> hope to
>>>>>>>>> post
>>>>>>>>> that later today.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't think anybody cares about the list of UTYPEs, at this 
>>>>>>>> point,
>>>>>>>> as
>>>>>>>> Mireille suggested.
>>>>>>>>
>>>>>>>> Let's figure out how the model needs to look like, then I am 
>>>>>>>> sure we
>>>>>>>> will
>>>>>>>> find a solution for the UTYPEs (which, as you have always been
>>>>>>>> adamant,
>>>>>>>> are
>>>>>>>> not to be parsed).
>>>>>>>> Unless you want to "model by UTYPES", which is something that 
>>>>>>>> the DM
>>>>>>>> WG
>>>>>>>> doesn't do anymore.
>>>>>>>>
>>>>>>>> There is a difference in what you and Mark are suggesting. 
>>>>>>>> Unless you
>>>>>>>> provide some actual arguments showing why your approach is better
>>>>>>>> (arguments that would have nothing to do with UTYPEs), I 
>>>>>>>> suggest that
>>>>>>>> we
>>>>>>>> adopt Mark's approach, which is simpler and is backed up by 
>>>>>>>> sound data
>>>>>>>> modeling reasoning.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I need to review your new figures but I think the obvious class 
>>>>>>>> names
>>>>>>>>
>>>>>>>>> are Observation, Characterisation, Image, etc.  Mireille 
>>>>>>>>> suggested
>>>>>>>>> earlier that the "Obs.x" element in Observation (containing
>>>>>>>>> dataProductType etc.) could be replaced by the more general
>>>>>>>>> Dataset.x.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> How can a name be more general than another?
>>>>>>>>>
>>>>>>>>
>>>>>>>> "What's in a name? That which we call a rose by any other name 
>>>>>>>> would
>>>>>>>> smell
>>>>>>>> as sweet."
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Omar.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Omar Laurino
>>>>>>>> Smithsonian Astrophysical Observatory
>>>>>>>> Harvard-Smithsonian Center for Astrophysics
>>>>>>>> 100 Acorn Park Dr. R-377 MS-81
>>>>>>>> 02140 Cambridge, MA
>>>>>>>> (617) 495-7227
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Omar Laurino
>>>>>> Smithsonian Astrophysical Observatory
>>>>>> Harvard-Smithsonian Center for Astrophysics
>>>>>> 100 Acorn Park Dr. R-377 MS-81
>>>>>> 02140 Cambridge, MA
>>>>>> (617) 495-7227
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>