Time Series Cube DM - IVOA Note
CresitelloDittmar, Mark
mdittmar at cfa.harvard.edu
Thu Mar 16 21:58:58 CET 2017
Markus,
Sorry for my earlier frustration.. I have no issue with implementation
feedback and am enjoying this interchange with you and Jiri.. it is
invaluable.
first, w.r.t. to Quantity.. that is included in the 'ivoa' model described
in the vo-dml document.
The description thereof has been moved to the normative part of the spec
(Section 5)
and the vo-dml representation and schema are to be provided as per other
models.
I started this response by extracting the corresponding elements from the
SparseCube example I have generated.
see:
https://volute.g-vo.org/svn/trunk/projects/dm/CubeDM-1.0/examples/chandra_events_annotated.vot
I will attach this, but when I got to the STC+Quantity stuff.. I started
realizing there were more questions,
so I'll put those inline below:
It's important to note that the vo-dml annotation syntax is isolated from
the VOTable, so the Group/Param
elements are NOT tagged with vodml-type/role.. but I think the relation can
be seen.
The closest analogy is annotating where it copies the Param info, rather
than refer to the existing Param, so I'll show that.
On Thu, Mar 16, 2017 at 7:03 AM, Markus Demleitner <
msdemlei at ari.uni-heidelberg.de> wrote:
>
> Now, here's how I'd like such an annotation to look like. I'm using
> vodml-type and -role as attributes rather than elements here for
> readability, and I'm interspersing the comments; please allow me to
> indulge in improvising class and attribute names for the time being.
> I hope if they don't match current models they should at least
> readily map to them:
>
> sure..
> ================= Dataset annotation =======================
>
<snip> Dataset stuff has good agreement
> ================== Cube annotation =========================
>
> <GROUP vodml-type="ndcube:Cube">
> <!-- No reason to have an extra type for time series; that's
> already defined in ds:Dataset.dataproductType and unlikely to
> be of relevance to a cube-only client (e.g., a plot program)
> anyway. -->
>
This particular Instance is a TimeSeriesCube from the diagrams, so the
vodml-type would be <model>:TimeSeriesCube.
I don't think this can be the generic SparseCube, as the axes of that
object are essentially equivalent.
I think this is the big distinction between the SparseCube and the
specialized views such as TimeSeries, Spectrum, etc..
there is no dominant (independent) axis..
<FIELDref vodml-role="independent-axis" ref="obs_date"/>
> <!-- that's it; a client just counts
> *[@vodml-role="independent-axis" and knows the number of dimensions
> in the cube. All additional annotation is on the FIELD itself. -->
>
> <FIELDref vodml-role="dependent-axis" ref="FLX"/>
> <FIELDref vodml-role="dependent-axis" ref="MAG"/>
> <!-- that's it; a single reference defines a "value" in this cube,
> and any further annotation is on the field itself, where non-cube
> clients can also use it. -->
> </GROUP> <!-- I don't think much further metadata is needed here -->
>
>
Comparing with the Note diagram, the dependent/independent axes have 2
components
1) field (presumably the 'value')
2) error
The annotation above does not say whether "obs_date" is the 'value' or
'error' of the independent axis.
<similar for "FLX" and "MAG">
============== STC+Quantity annotation =====================
>
> <GROUP vodml-type="stc:Time">
> <!-- the one place STC metadata is collected -->
> <FIELDref vodml-role="value" ref="obs_date"/>
> <!-- note how we "amend" metadata on obs_date here; by co-reference
> with the ndcube:Cube annotation, obs_date is *both* an independent
> axis *and* a time. -->
>
I do not believe one can "amend" metadata on objects. This instance is
entirely independent of the above instance of "independent-axis". They
both use the same column (obs_date) for the 'value', but they are not the
same thing.
This is defining an 'Absolute Time coordinate'.. a value along the 'time'
axis in a particular TimeFrame. The value is coming from a FIELD, so
there are many of them, but who/what is using this Time 'column'? ie: This
GROUP must be assigned to a role with multiplicity greater than one. (eg.
in a CubeAxis.field in your diagrams or NDPoint.axis.observable in mine.
> <PARAM vodml-role="timescale" value="TT"/>
> <PARAM vodml-role="timeformat" value="MJD"/>
> <!-- timeformat is an invention; STC 1.0 uses classes to
> distinguish between JD, MJD, "ISO". -->
> <PARAM vodml-role="referencePosition" value="BARYCENTER"/>
> ...
>
Translating this.. This is Frame information in the Time domain.. and would
be bundled within a grouping identified with the appropriate vodml-type.
Otherwise, there is no way of telling which object this 'timescale' role
belongs to. This Group reverse-engineers to an Object with spec:
Time
+value:RealQuantity[*]
+timescale:TimeScale
+timeformat:string
+referencePosition:StdReferencePosition
Maybe I'm interpreting it too literally. My point is that the structure
must be represented in the annotation.. a Type has content which are
assigned Roles, this is 1 level deep. One cannot assume that 'timescale'
maps to "TimeFrame.timescale".
> </GROUP>
>
> <GROUP vodml-type="ivoa:Quantity">
> <!-- all measurements (can) have errors, min/max vals, etc, so
> there's no point separately modelling this in cube, stc,
> photometry, etc.; let's have ivoa:Quantity for that. -->
> <FIELDref vodml-role="value" ref="obs_date"/>
> <FIELDref vodml-role="standard-deviation" ref="err_time"/>
> <PARAM name="minimum" value="56493.339"/>
> <PARAM name="maximum" value="56498.341"/>
> <!-- which also does much of char:, without introducing 1000s of
> utypes -->
> </GROUP>
>
>
OK.. so this maps more to the 'coords' model content (this is NOT in
cube). Specifically, this would be 'coords:domain.temporal.Time' .
I see here ref="obs_date" is used again... is this supposed to
'cross-reference' to the stc:Time object above?, so that this
Quantity.value IS an stc:Time object with value and associated Frame, which
is also cross-referenced to the independent axis of the Cube? How does the
user know if this Quantity or the stc:Time element is the cube "independent
axis"?
The ivoa Quantity is an atomic physical value. It is used in both the
value and error elements of a 'measurement'.
The coords model defines a pattern which associates the coordinate value
with associated errors. (the pattern maps to your assertion that 'there is
no point separately modeling this in cube, stc, photometry, etc.'). The
pattern establishes the framework for all Coordinates (measurements) and
each domain implementation must follow the pattern, adding any
specialization needed. We are starting with a simple Error model, but as
things develop, it can grow to accommodate more complex things (like
probability distribution errors). All domain implementations should follow
this pattern.
I'm not sure what you mean by min/max here
+ a single 'measurement' can have a min/max uncertainty (error)
+ a group of 'measurements' can have a min/max range (char)
+ an axis can have a min/max range in the domain (axis.domainMin/Max in
the coordsys model)
The identification of the 'error' as 'standard-deviation' is interesting..
this implies the value is a sampling and the error is calculated by the
spread in the sampling.
The current error model is simpler than that, it allows for specifying a
symmetrical error type, but not HOW that value was determined (standard
deviation).
> ============ Photometry+Quantity annotation ==================
>
> <snip>
>
> <GROUP vodml-type="ivoa:Quantity">
> <!-- it's exactly the same thing as above for obs_date; if
> a client understands Quantity, it'll need no extra code, no extra
> utypes to interpret this as well as the obs_date annotation. -->
> <FIELDref vodml-role="value" ref="FLX"/>
>
> <FIELDref vodml-role="standard-deviation" ref="FLXERR"/>
> <!-- here, and not in a custom thing within ndcube or somewhere
> else, is the connection made between FLX and FLXERR; that way, a
> Quanitity-knowing client can figure out the error, not just one
> for NDCube -->
> <PARAM name="minimum"...
> </GROUP>
>
So, the theme seems to be that you don't want the various model elements
annotated within the annotation of another model?
If you look at my example, a user who understands Time measurements
"coords:domain.temporal.Time" can find
them by scanning for INSTANCE with that dmtype.. even if it resides within
an object they are not familiar with.
What one cannot do is find all coords (your Quantity here) because the
underlying pattern is not actualized. We are not assuming that all
measurements are modeled the same, and have the same error model. The
dimensionality of some domains also hinders that because it requires
multiple 'values' (not a problem) with cross-correlated errors (again,
different error model than simple 1D domains).
Ah! that may not be entirely true. In my examples, I short-cut a relation.
cube:Observable has reference to coords:DerivedCoordinate
I short-cut it because this is the aggregation relation, and I figured one
would know that cube:Observable is a coords:DerivedCoordinate.
But going through your example, I see an important tag would be missing
that would allow a coords savvy, but non-cube savvy user to FIND all
"coords:DerivedCoordinate" types. (This type is bound to follow the coords
pattern.. so if a user has implemented the pattern, they should be able to
handle at least some content of any implementations)
Hmm.. Nope.. that wouldn't do it (sorry for the train of thought
rambling). Even with that relation, the element at the target of the
reference would be of the implemented type, not the base type. (so, it
would show "coords:domain.temporal.Time", but not
"coords:DerivedCoordinate"). One would have to interrogate the model
definition to see that Time extends DerivedCoordinate.
> ============= Field declarations
> <FIELD ID="dateObs" name="dateObs"/> << obs_date
> <FIELD ID="FLX" name="FLX"/>
> <FIELD ID="FLXERR" name="FLXERR"/>
>
>
>
Wow.. that's a lot to take in..
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20170316/24edec8c/attachment-0001.html>
-------------- next part --------------
================= Dataset annotation =======================
<INSTANCE dmtype="cube:SparseCubeDataset">
<ATTRIBUTE dmrole="ds:dataset.Dataset.dataProductType">
<LITERAL dmtype="ds:dataset.DataProductType" value="EVENT"/>
</ATTRIBUTE>
<COMPOSITION dmrole="ds:dataset.Dataset.curation">
<INSTANCE dmtype="ds:dataset.Curation">
<ATTRIBUTE dmrole="ds:dataset.Curation.publisherDID">
<LITERAL dmtype="ivoa:anyURI" value="ADS/Sa.CXO#obs/01838"/>
</ATTRIBUTE>
</INSTANCE>
</COMPOSITION>
<COMPOSITION dmrole="ds:experiment.ObsDataset.target">
<INSTANCE dmtype="ds:experiment.AstroTarget">
<REFERENCE dmrole="ds:experiment.BaseTarget.position">
<!-- NOTE in my example, this is not actually a reference, but to conserve space
and match with Marcus' outline, I've converted it. -->
<IDREF>_targetPosition</IDREF>
</REFERENCE>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
<INSTANCE ID="_targetPosition" dmtype="coords:domain.spatial.Position2D">
<ATTRIBUTE dmrole="coords:domain.spatial.Position2D.coord">
<INSTANCE dmtype="coordsys:domain.spatial.SpatialCoord">
<ATTRIBUTE dmrole="coordsys:domain.spatial.SpatialCoord.loc">
<LITERAL dmtype="ivoa:RealQuantity" value="54.3"/>
</ATTRIBUTE>
<REFERENCE dmrole="coordsys:domain.spatial.SpatialCoord.axis">
<IDREF>_01017W9piipgxj8L2</IDREF>
</REFERENCE>
</INSTANCE>
<INSTANCE dmtype="coordsys:domain.spatial.SpatialCoord">
<ATTRIBUTE dmrole="coordsys:domain.spatial.SpatialCoord.loc">
<LITERAL dmtype="ivoa:RealQuantity" value="-12"/>
</ATTRIBUTE>
<REFERENCE dmrole="coordsys:domain.spatial.SpatialCoord.axis">
<IDREF>_01017cwvI7yS93aI2</IDREF>
</REFERENCE>
</INSTANCE>
</ATTRIBUTE>
</INSTANCE>
================== Cube+Coords annotation =========================
<!-- Here I could try to separate out the coord from cube annotation
to match your outline better, but would have to devise the
reference. We identify/annotate a column (FIELD) as
"the value of a <domain> coordinate along an <axis>"
If there were another column with the associated Error.. that
column "time_err" is the symmetric error associated with the
<domain> coordinate value. A field value will map to a particular
element of a Coordinate.
->
<INSTANCE ID="_A9OKA8Pw7UvuTGbF2" dmtype="cube:SparseCube">
<COMPOSITION dmrole="cube:SparseCube.data">
<EXTINSTANCES>_AK9CaKnrQwqiiyvn3</EXTINSTANCES>
<!-- instances of this object are in the TEMPLATES area (which point to fields) ->
</COMPOSITION>
</INSTANCE>
<TEMPLATES tableref="_table1">
<INSTANCE ID="_AK9CaKnrQwqiiyvn3" dmtype="cube:NDPoint">
<COMPOSITION dmrole="cube:NDPoint.axis">
<!-- This is a Template for the NDPoint.. particular values are
associated with FIELDs (COLUMN). I show 2 data axes 'time' and 'pha'.
In the generic SparseCube, there is no distinction between
dependent and independent.. all axes are equivalent. -->
<INSTANCE dmtype="cube:DataAxis">
<COMPOSITION dmrole="cube:DataAxis.observable">
<INSTANCE dmtype="cube:Observable">
<ATTRIBUTE dmrole="coords:domain.temporal.Time.coord">
<INSTANCE dmtype="coordsys:domain.temporal.TimeCoord">
<ATTRIBUTE dmrole="coordsys:domain.temporal.TimeOffset.loc">
<COLUMN ref="_col-time" dmtype="ivoa:RealQuantity"/>
</ATTRIBUTE>
<REFERENCE dmrole="coordsys:domain.temporal.TimeCoord.axis">
<IDREF>_0118WLLnSEBj0p462</IDREF>
</REFERENCE>
</INSTANCE>
</ATTRIBUTE>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
<INSTANCE dmtype="cube:DataAxis">
<COMPOSITION dmrole="cube:DataAxis.observable">
<INSTANCE dmtype="cube:Observable">
<ATTRIBUTE dmrole="coords:domain.spectral.DerivedSpectralCoordinate.coord">
<INSTANCE dmtype="coordsys:domain.spectral.SpectralCoord">
<ATTRIBUTE dmrole="coordsys:domain.spectral.SpectralCoord.loc">
<COLUMN ref="_col-pha" dmtype="ivoa:RealQuantity"/>
</ATTRIBUTE>
<REFERENCE dmrole="coordsys:domain.spectral.SpectralCoord.axis">
<IDREF>_0129ge70cAi4MErU2</IDREF>
</REFERENCE>
</INSTANCE>
</ATTRIBUTE>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
</TEMPLATES>
============== Coordsys annotation =====================
<!-- The bit that is missing above is the frame specs to
which the various coordinates are associated.
These are domain implementations of the Frame pattern
which is defined in the Coordsys model. -->
<INSTANCE ID="_0117pYWsEJmSbhJP2" dmtype="coordsys:domain.temporal.TimeFrame">
<ATTRIBUTE dmrole="coordsys:domain.temporal.TimeFrame.timescale">
<LITERAL dmtype="coordsys:domain.temporal.TimeScale" value="TT"/>
</ATTRIBUTE>
<ATTRIBUTE dmrole="coordsys:domain.temporal.TimeFrame.refPosition">
<INSTANCE dmtype="coordsys:domain.spatial.StdSpatialLocation">
<ATTRIBUTE dmrole="coordsys:domain.spatial.StdSpatialLocation.coord">
<LITERAL dmtype="coordsys:domain.spatial.StdRefPosition" value="TOPOCENTER"/>
</ATTRIBUTE>
</INSTANCE>
</ATTRIBUTE>
<ATTRIBUTE dmrole="coordsys:domain.temporal.TimeFrame.offset">
<INSTANCE dmtype="coordsys:domain.temporal.TimeOffset">
<ATTRIBUTE dmrole="coordsys:domain.temporal.TimeOffset.loc">
<LITERAL dmtype="ivoa:RealQuantity" value="50814.0"/>
</ATTRIBUTE>
</INSTANCE>
</ATTRIBUTE>
<COMPOSITION dmrole="coordsys:domain.temporal.TimeFrame.coordSpace">
<INSTANCE dmtype="coordsys:domain.temporal.CoordSpace1D">
<COMPOSITION dmrole="coordsys:domain.temporal.CoordSpace.axis">
<INSTANCE ID="_0118WLLnSEBj0p462" dmtype="coordsys:domain.temporal.CoordAxis">
<ATTRIBUTE dmrole="coordsys:domain.temporal.CoordAxis.name">
<LITERAL dmtype="ivoa:string" value="time"/>
</ATTRIBUTE>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
<INSTANCE ID="_01290NTXcYmRii442" dmtype="coordsys:domain.spectral.SpectralFrame">
<ATTRIBUTE dmrole="coordsys:domain.spectral.SpectralFrame.refPosition">
<INSTANCE dmtype="coordsys:domain.spatial.StdSpatialLocation">
<ATTRIBUTE dmrole="coordsys:domain.spatial.StdSpatialLocation.coord">
<LITERAL dmtype="coordsys:domain.spatial.StdRefPosition" value="TOPOCENTER"/>
</ATTRIBUTE>
</INSTANCE>
</ATTRIBUTE>
<COMPOSITION dmrole="coordsys:domain.spectral.SpectralFrame.coordSpace">
<INSTANCE dmtype="coordsys:domain.spectral.CoordSpace1D">
<COMPOSITION dmrole="coordsys:domain.spectral.CoordSpace.axis">
<INSTANCE ID="_0129ge70cAi4MErU2" dmtype="coordsys:domain.spectral.CoordAxis">
<ATTRIBUTE dmrole="coordsys:domain.spectral.CoordAxis.name">
<LITERAL dmtype="ivoa:string" value="pha"/>
</ATTRIBUTE>
<ATTRIBUTE dmrole="coordsys:domain.spectral.CoordAxis.domainMin">
<LITERAL dmtype="ivoa:RealQuantity" unit="adu" value="0"/>
</ATTRIBUTE>
<ATTRIBUTE dmrole="coordsys:domain.spectral.CoordAxis.domainMax">
<LITERAL dmtype="ivoa:RealQuantity" unit="adu" value="36855"/>
</ATTRIBUTE>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
</COMPOSITION>
</INSTANCE>
More information about the dm
mailing list