Revised ImageDM-ObsCore architecture

[Matthew Graham]

Hi,

There is a process here that we follow so that we can coordinate the international effort and prevent people going off and doing what they believe is needed on their own. The work path of the IVOA WGs is not determined by the individual members of the group but by the WG chairs. If ObsCore needs updating then this has to ultimately come from the WG Chair or Deputy and be on the IVOA Roadmap. I would therefore urge the ImageDM authors to discuss this with the DM Chairs at their earliest convenience. 

	Cheers,

	Matthew




On Nov 12, 2013, at 4:51 PM, Douglas Tody wrote:

> Hi Omar -
> 
> This is degerating into a meta-discussion and we don't have time for
> that, so I won't respond in detail.
> 
> The agreement coming out of the interop was that Image and Spectral
> should (one way or the other) be an extension of ObsCore, to avoid
> duplication/inconsistency and enhance interoperability, e.g., between
> ObsTAP and SIA-based components.
> 
> There are two ways to do this: either we leave ObsCore unchanged and
> layer ImageDM/SpectralDM, or we update ObsCore as needed to reflect more
> recent DM work (e.g., Char2), incorporate any missing
> generic/Observation metadata, and limit Image/Spectral to what is
> specific to each class of data.  My original tendency was to do the
> former but the consensus of the ImageDM co-authors (see discussion
> archived on the ImageDM TWiki page), which includes most of the original
> ObsCore main authors, was the latter.  Yes there is an impact on
> existing implementations, but on the other hand Image and Spectral are
> major developments and an update to ObsCore may be timely in any case as
> it has been a while.
> 
> The Utype names are based directly upon data model class names, however
> these evolve hence we get inconsistencies.  There are also choices to be
> make regarding how to expose an element as either a reference value
> (required for simple use cases), or something more complicated and fully
> detailed.  This is not a simple autogenerated data model issue but
> requires some human judgment.  Futrher integrating independently
> developed and use-cased models like Observation/Char and STC is not
> trivial.
> 
> While a data model is more than just a set of Utypes and their
> shortnames (e.g. DBMS table column names), these are essential to our
> applications and archives, and greatly impact the user community outside
> our little VO club.  Annoying as it may be, we need to pin this stuff
> down to support real-world applications.
> 
> We *will*, one way or the other, have a stable version of this to
> support critical Cube project development and uptake by the community,
> while these debates rage for the next 6-24 months.
> 
> 	- Doug
> 
> 
> On Tue, 12 Nov 2013, Laurino, Omar wrote:
> 
>> Hi Doug,
>> 
>> I am not suggesting that this is all about Utypes or that we don't need
>>> a formal class hierarchy as the basis for the data model.
>> 
>> 
>> Well, you explicitly wrote this:
>> 
>>> The consensus earlier is that an update to ObsCore *is* required, as
>>> there are issues with a number of the Utypes in that version of the
>>> model.
>> 
>> 
>> There is no such consensus. And you name UTYPEs as the main reason. So, it
>> looks
>> like it's all about UTYPEs according to you.
>> 
>> 
>>> I suggest
>>> that what I posted over the weekend is a valid, even simple, data model
>>> architecture basing Image on Observation as we agreed to do in the
>>> interop.  Have you actually look at this?
>>> 
>> 
>> Yes, I did, and in my opinion it was a regression from what Mark was trying
>> to achieve. Hence my comment.
>> 
>> 
>>> 
>>> While a data model is more than just Utypes, we do have a problem with
>>> Utypes in the current ObsCore/ImageDM/SpectralDM/Char2.  The problem is
>>> relatively minor (e.g., "Calib" vs "Calibration", Coord as an array
>>> value vs separated .C1, and .C2 attributes, and the like)
>> 
>> 
>> I won't comment this anymore. This whole effort is not about nice names.
>> 
>> I already have to come up with a name for my son, I think that's enough
>> without having to choose between Calib and Calibration :)
>> 
>> 
>> however it
>>> will require an update to ObsCore synchronized with Image/Spectral to
>>> fix the problem.  Otherwise we will have to propagate stuff from ObsCore
>>> 1.0 which the authors now agree is invalid.
>>> 
>>> 
>> First, there is no such thing as an ObsCore update in the Roadmap. If I am
>> mistaken, the TCG chairs will prove me wrong.
>> 
>> Second, you don't need to change the ObsCore UTYPEs in order to have a
>> sound Data Model for ImageDM, so your statement is false: it is not
>> required.
>> 
>> Third, the DM WG is trying to adopt a mechanism that allows to "fix the
>> problem" without changing Data Models every time you don't like a string.
>> 
>> Fourth, if you change ObsCore, then you need to consider the impact on all
>> the existing implementations. Since you are not solving any actual problems
>> by turning "Calib" into "Calibration", you are not presenting any
>> compelling reasons to update the ObsCore DM.
>> 
>> Plus, the consensus was to build on top of ObsCore, not to change it.
>> 
>> 
>> 
>>> I don't have any problem with proceeding with (or at least
>>> investigating) a more rigorous approach based upon VO-DML - what Gerard
>>> posted earlier was quite promising.  However I doubt very much if we can
>>> complete that this week, and we need a stable update to the ImageDM by
>>> the end of the week (or at least end of Nov at the latest) to support
>>> cube project development.
>>> 
>> 
>> If you don't try, you won't ever know. Also, it was agreed that we would
>> provide such descriptions anyway. Whether they describe your concepts or
>> Mark's, that does not make any difference.
>> 
>> Also, Mark's suggestion, which is simpler and does not require changing
>> current recommendations, was already in the right direction. Again, if you
>> have points that do not involve changing strings, let's consider them
>> together.
>> 
>> Anyway, I am surprised that you consider being rigorous as an option you
>> can procrastinate forever, and not a professional requirement. We need to
>> be rigorous, otherwise we are just playing games and wasting time.
>> 
>> 
>>> 
>>> The main thing is that the data model *architecture* must not change
>>> greatly.
>> 
>> 
>> Which is exactly what Mark is doing. Changing ObsCore means changing the
>> architecture. And would require a major release, since backward
>> compatibility would be broken. That's a big change you are suggesting.
>> 
>> 
>>> Hence if we really want to integrate Image/Spectral and
>>> Observation (Image formally extending Observation as agreed in the
>>> interop) then we need a usable first cut at that *now*.
>> 
>> 
>> Again, Mark's suggestion is *now*. But it is also simpler, and doesn't
>> require changes to the architecture.
>> 
>> 
>>> But the IVOA consensus to move in this direction was very
>>> clear.
>>> 
>> 
>> I don't see anything in the IVOA roadmap that involves reopening the REC
>> process for ObsCore. So, I am not sure where your statement, in the name of
>> the IVOA, comes from.
>> 
>> Omar.
>> 
>> 
>>> 
>>>        - Doug
>>> 
>>> 
>>> 
>>> On Tue, 12 Nov 2013, Laurino, Omar wrote:
>>> 
>>>> The consensus earlier is that an update to ObsCore *is* required, as
>>>>> there are issues with a number of the Utypes in that version of the
>>>>> model.
>>>>> 
>>>> 
>>>> No, no, and no :)
>>>> 
>>>> You fix the model if you need to fix the model. And models are not list of
>>>> UTYPEs.
>>>> Ergo, you don't fix a model if you don't like the UTYPEs. And we don't
>>>> "model by UTYPEs" anymore.
>>>> That's the consensus.
>>>> 
>>>> On the other hand, if what you are saying is true, you are suggesting that
>>>> we *need* to redefine almost all of the IVOA Data Models in order to
>>>> create
>>>> SIAPv2, for one reason or the other.
>>>> 
>>>> I hope this is not true (and I am sure this was not the consensus,
>>>> anyway).
>>>> But if it is, then let's get it really right this time. Which means: we
>>>> don't "model by UTYPEs".
>>>> 
>>>> 
>>>> Then the ImageDM
>>>>> architecture is much simpler, as most of it is just ObsCore and Char.
>>>>> 
>>>>> 
>>>> This is a non-sequitur. Assuming that by architecture you refer to the
>>>> class diagram, the class diagram does not need to be simple: it needs to
>>>> meet the requirements in the most efficient way, which is different.
>>>> 
>>>> Also, Mark's diagram doesn't seem complicated, so you cannot use Occam's
>>>> razor in this case.
>>>> 
>>>> Actually, Mark's diagram doesn't require to review ObsCore, ergo it is
>>>> simpler.
>>>> 
>>>> 
>>>> 
>>>>> This should be more clear in the revised DM spreadsheet; I hope to post
>>>>> that later today.
>>>>> 
>>>> 
>>>> 
>>>> I don't think anybody cares about the list of UTYPEs, at this point, as
>>>> Mireille suggested.
>>>> 
>>>> Let's figure out how the model needs to look like, then I am sure we will
>>>> find a solution for the UTYPEs (which, as you have always been adamant,
>>>> are
>>>> not to be parsed).
>>>> Unless you want to "model by UTYPES", which is something that the DM WG
>>>> doesn't do anymore.
>>>> 
>>>> There is a difference in what you and Mark are suggesting. Unless you
>>>> provide some actual arguments showing why your approach is better
>>>> (arguments that would have nothing to do with UTYPEs), I suggest that we
>>>> adopt Mark's approach, which is simpler and is backed up by sound data
>>>> modeling reasoning.
>>>> 
>>>> 
>>>> 
>>>> I need to review your new figures but I think the obvious class names
>>>>> are Observation, Characterisation, Image, etc.  Mireille suggested
>>>>> earlier that the "Obs.x" element in Observation (containing
>>>>> dataProductType etc.) could be replaced by the more general Dataset.x.
>>>>> 
>>>>> 
>>>>> How can a name be more general than another?
>>>> 
>>>> "What's in a name? That which we call a rose by any other name would smell
>>>> as sweet."
>>>> 
>>>> Cheers,
>>>> 
>>>> Omar.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Omar Laurino
>>>> Smithsonian Astrophysical Observatory
>>>> Harvard-Smithsonian Center for Astrophysics
>>>> 100 Acorn Park Dr. R-377 MS-81
>>>> 02140 Cambridge, MA
>>>> (617) 495-7227
>>>> 
>>>> 
>> 
>> 
>> --
>> Omar Laurino
>> Smithsonian Astrophysical Observatory
>> Harvard-Smithsonian Center for Astrophysics
>> 100 Acorn Park Dr. R-377 MS-81
>> 02140 Cambridge, MA
>> (617) 495-7227
>> 
>