SED Data Model working draft (on behalf of D. Tody)

[Douglas Tody]

Hi Mark, Omar -

Thanks for the comments - this is the kind of discussion we need to have
to finally bring this together.

I haven't had time to read all this carefully yet, and need to switch
back to SIAV2 and Image soon.  A few quick comments however.

Much of this discussion concerns the aggregate SED and segments.  In
this most recent SEDDM it is proposed that a segment of the aggregate
SED is an externally viable, self contained dataset, which is merely
stored in the serialized aggregate SED, with the votable or FITS MEF
serving as a container (pointing to an actual external dataset with a
URI is also permitted and might be desirable for large datasets).
Usually this externally viable dataset could indeed be an object derived
from SpectralDM and compliant with the VO models, but if we merely have
a simple container of segments this does not have to be the case.  Since
the aggregate SED is used for SED building or editing it could be
desirable to include or reference native datasets which have not been
converted to a VO model.

If instead we want to require that segments of the aggregate SED must be
derived from and compatible with the SpectralDM then the model might
indeed want to be changed as you both describe.  Even in this case
however it is possible to merely provide a container for aggregating
externally viable, standalone datasets, giving us the flexibility to
reference external data in some native format.

** So the issue to be decided here is whether we permit the aggregate
SED to reference native datasets or restrict it to only data classes
derived from SDM **

The uniform SED of course, is fully transformed into VO and compliant
with SDM and there is no such issue.

>  "Uniform (or rebinned)"  I think implies a synonym relationship, while the 
> description that follows shows that they have a different level of 
> processing.  Also, it says that rebinned handles overlaps.  Does that mean 
> that Uniform does NOT.. so the single Uniform sequence can have overlapping 
> data?

Yes, it is possible for segments in the uniform SED to overlap, e.g., a
spectrum may overlap another spectrum or photometry point from another
observation (segment).  Rebinning further normalizes the data to combine
any overlapped regions, which may or may not be desirable.  In both
cases we have a uniform SED however.  (This business of uniform vs
rebinning originated with Jonathan).

> Section 3.5.1:
>  + Utypes containing SED?  Isn't this contrary to the changes we did to 
> remove the 'Spectrum' node from Spectral utypes?

We decided earlier that we would have a core SpectralDM defining classes
which are common to all the models - Spectrum, SED, TimeSeries, etc.  So
Dataset, Char, DataID, etc. Utypes do not start with "Spectrum.",
"SED." and so on, and are the same for all classes of data derived from
the model.  We would then extend the core model by adding a class which
is for stuff which really is specific to the particular class of data
and not part of the generic core, inherited model.  Hence for TimeSeries
for example, all the usual SDM stuff is there unchanged, but we add a
new class TimeSeries which contains metadata specific to time series
data, e.g., information about the period or periods, trend removal,
whatever.  For SED we have a similar construct.  Most of the SDM is
inherited unchanged, the Utypes are generic, but stuff which is really
specific to SED, e.g., nSegments, is put into a SED.xx element.

 	- Doug




On Tue, 23 Oct 2012, Mark Cresitello-Dittmar wrote:

> Doug..
> This does look like we're converging.
>
> Comments ensue...
>
> general:
>  I still think the term "SED" is being used in two different contexts.. when 
> discussing a 'segment' of the SED and for the SED itself.
>  This feels kind of like the 'Segment' discussion in Spectral. I'm going to 
> introduce a term "UniformSPS" below.
>
> Section 2:
>  "no attempt is made at this stage to reconcile the data in the different 
> segments"
>   One requirement we got for Iris is that the segments of an SED must be 
> "compatible" quantities, but not homogenous.  Is this generally true, (so 
> that this should be clarified)? or only relevant for certain use-cases?
>
>  "Uniform (or rebinned)"  I think implies a synonym relationship, while the 
> description that follows shows that they have a different level of 
> processing.  Also, it says that rebinned handles overlaps.  Does that mean 
> that Uniform does NOT.. so the single Uniform sequence can have overlapping 
> data?
>
> Section 3.1/Figure 2:
>  I think this maps to Spectral better if you shift it up a level... that 
> document already contains a container for a list of BaseSPS objects. 
> Spectral:Section 2, pg 11 - "Spectral" object.
>  -----------------------------
>   SED = Extension of Spectral
>     + Adds UniformSPS
>     + UniformSPS is an extension of BaseSPS as described by your SED node 
> (without the 0:n Segments)
>     This way SED contains a UniformSPS plus a list of BaseSPS, which you may 
> want to restrict to Spectrum, PhotometryPoint, TimeSeries as you do in Figure 
> 3.
>
>   A Uniform SED = SED with UniformSPS including DATA, and empty list.
>   Aggregate SED = SED with UniformSPS with no DATA (or possibly with), and 
> list content
>
>   The Uniform SED could be interpreted by Spectral aware software by 
> extracting the UniformSPS
>  ----------------------------
>
>  Aggregate SED - Bullet points:
>    + aggregate composed of actual BaseSPS extensions OR references to them..
>       - so you want to define a Segment object which contains EITHER a URI 
> or an instance of a BaseSPS extension [Spec, PP, TS]
>          Segment
>             * ref:idRef
>             * sequence:BaseSPS[restrict Spec, PP, TS]
>
>         that makes my suggestion above less useful.  However, I'm not a fan 
> of defining an extension of a thing which contains that thing.  I prefer 
> defining a container for it.  If it isn't Spectral, but an SED container 
> which holds a UniformSPS plus list of Segments, that's fine, but I'd then 
> vote for removing the "Spectral" object from the Spectral model, cuz if we 
> don't use it here.. where will it be?
>
> I see you're trying to make a stronger connection between the SED and the 
> UniformSPS than I.  If that is to support the above, recall that the 
> UniformSPS allows for Data.TimeAxis and Data.SpatialAxis, which Spectrum does 
> not.. it will also need to have a unique Dataset.Type value which is not 
> "Spectrum"... so for Spectrum savvy software to use it, they will have to 
> either be ignoring the Spectrum requirements or make special allowance.  In 
> either case, I'm not sure why SED itself must be a BaseSPS extension to allow 
> that.
>
> Section 3.3: Namespace
>  This ties in with the xmlns comment on the Spectral model (as you note on 
> pg 11)
>  This seems at first to be against convention...  I'm trying to see how 
> things go across the division.
>  If we apply the normal Utype inheritance/extension rules (like we did in 
> Spectral for the PhotometryFilter elements)
>   (presumably SED is not part of the utypes)
>   sed:Dataset
>   sed:Dataset.DataModel
>   sed:Segment
>   sed:Segment.ref
>   sed:Segment.Spectrum
>   spec:Segment.Spectrum.Dataset.DataModel
> spec:Segment.Spectrum.CoordSys.FluxFrame.PhotCal.PhotometryFilter...
>
>  I don't think that is what you mean.  Do you want to specify that this 
> model does NOT use the defacto UType extension convention, but defines each 
> segment instance as it's own base using that namespace?
>
>   sed:Dataset
>   sed:Dataset.DataModel
>   sed:Segment
>   sed:Segment.ref
>   sed:Segment.Spectrum
>   spec:Dataset.DataModel
>   spec:CoordSys.FluxFrame.PhotCal.PhotometryFilter...
>
>  That doesn't look quite right either, I don't see why that would be OK here 
> and not for the Char or Photometry elements of Spectral.
>  But this goes heavy into UType land..
>
> Section 3.5.1:
>  + Utypes containing SED?  Isn't this contrary to the changes we did to 
> remove the 'Spectrum' node from Spectral utypes?
>  + these fields provide information which is readily determined from the 
> data product, so are you thinking of these as query metadata?
>  + isn't this similar to the discussions on Spectral that created the 
> DataSet object?  would these items fit under an extension of Dataset?
>
> Section 3.5.2:
>  + Defines extension to Spectral:Data for the SED node (what I called the 
> UniformSPS above).  thats fine
>  + "Any attribute of the SpectralDM may be included as a data column in the 
> uniform SED."
>     That is a mighty powerful statement.. I think that will need some 
> clarification.
>     (and makes for a mighty big UType list )
>
>
> We'll worry about serialization later.
>
>
>