3D data model

[Jonathan McDowell]

3D spectral data model

(Some of this repeats comments made by Doug, Brian and Francois,
but I thought I'd give my take.)

Igor's started an interesting discussion on 3D data in the VO. I note
that all of Igor's examples are essentially spatial-spectral cubes; the
question we need to address is to what extent this needs a different
treatment from other 3D data: 
  -spatial-temporal cubes (movies) are an obvious case;
  -spatial-polarization cubes are a little further away. Then there are 
 - redshift surveys, which to the extent that redshift is a
   spectral coordinate fit in the discussion, but also are not so far away
   from:
 - 3D spatial cubes in which two axes are RA, Dec and the third coordinate is 
   distance, and then to:
 - 3D cartesian cubes of X,Y,Z which at the moment are mostly confined to 
   theoretical simulations (or planetary probe data?). 

Even within the list of Igor's examples we can distinguish an important
difference between continuous and discrete axes in the spatial and spectral
directions.
 - integral field spectroscopy, radio line imaging, X-ray CCD, etc are true continuous
   spatial-spectral cubes.
 - In multi-object spectroscopy (eg fibers), the spatial axis is replaced by a 1-D list of
   objects - it's a 2D dataset of object-number versus spectral coordinate, even
   if you arrange it differently. You can also think of it as a discontinuous
   spatial coordinate.
 - In multi-line imaging, it's the spectral coordinate that is discontinuous.

The Quantity model addresses the basic problem of multidimensional data
and coordinates. The STC model and the pieces of the 1D SED model
address most of the rest of the metadata we'll need and the Observation
model gives the context.

So Igor and others, I'd ask you to take these existing pieces of work into
account as you construct a model. You should decide which of the above
cases are going to be in scope for your particular model, and how
it fits in the the existing framework.

> From: Francois Bonnarel <bonnarel at alinda.u-strasbg.fr>
> Message-Id: <200502150959.KAA13600 at alinda.u-strasbg.fr>
> To: dal at ivoa.net, dm at ivoa.net
> Subject: 3D data model
> Doug, Igor, Philippe, Giovanni,

I agree with the comments of Francois, and welcome the proposals
that Igor gather the consensus of various 3D data users, and that
he participate in the Characterization discussion.

>      I am just afraid that adding new subgroups will make the whole
> Working group unmanageable and I would really like to have Jonathan's
> advice about that.

I think that it's great that the 3D folks should discuss their metadata
issues specifically, but they should look at the work done in
Observation, Quantity, STC and Characterization and see if they can fit
within that framework. If not, that needs to feed back to those
proto-standards, they will have to be changed. But you're right we
shouldn't have separate standards for very similar problems, the whole
point of DM is to avoid that.


Regan noted:
> The Data Format Description Language group of the 
> Global Grid Forum is making progress on building 
> XML-based representations of the structure of 
> arbitrary binary data sets. 

Yes, and we should look at this when we discuss serializing
the Observation model and its 3D spatial-spectral instance.
However, DFDL won't address what astronomical metadata needs
to be added to the datasets to make them interoperate scientifically,
which is the first thing we need to agree on. The data structure
of the binary data array is pretty trivial really.

  - Jonathan