Handling data cubes in VO

[Francois Bonnarel]

Dear all,
      I post this on both DM and DAL for people missing one of these, but 
please answer on DAL only to avoid cross-posting.
     
      I come back to this discussion on Data cube with a little bit more details
than in my mail of December 24th.

 1)   From the discussion so far I retain the first point that datacubes are a 
category of data set where there is an obvious need for several distinct data 
access:

      full retrieval
      "generalized" cutout (1, 2 or 3D)
      resampling/reprojection 

      one could also have: previews, continuum map, velocity field, etc...

     Actually for me this need is not specific to Data cubes but also
exist for other kind of data (even 2D images, where we could also have
previews, full retrieval, cutouts and resampled "views" of the same data set)


  2 ) This plurality of accesses has to be provided not only by the client
software (in that case it would be called processing, visualisation and 
not simply access) but by the server. One main reason for that is the 
volume of the data sets, which requires some clever visualisation of the 
data for selection before full download. So we are not speaking further
on manipulation visualisation tools (where, eg, the Quantity classes
mentionned by Brian could play a role) but only on the DAL layer.

  3 ) As we want to build such a server in the VO frame, Doug's conclusion
which I agree with, is that our server will be some kind of
new generation SIA server (mainly because SIA is dealing for services with 
regular sampling in all directions, and with some WCS information available).

  4 ) In order to build that kind of service we need an ExtendedDataAccess
model , that is a model of the various ways the same dataset can be accessed. 
This model doesn't have to be specific to 3D data. It can be
seen as an extension of the Access Package as defined in SSA.

  5 ) Having this model, we will then use it to define a set of
SIA query parameters allowing to request for these specific views.
That's the essence  of Doug's proposal, if I understand well.

  5 bis ) My proposal is to use this datamodel to define new acces
metadata in the Query response. These acces metadata will describe
the various access modes and visualisation with the associated AccessRef
which can be in some cases an URL, in some others an URL template* with
variable parameters  or a pointer to a WEB service.


  6 ) discussion:  For the first query the "Acces and View" query parameters 
cannot generally be given, BECAUSE their range or possible values depend from 
the characterisation and other metadata given in the coming response.
      Doug's solution for this (see 5)is to use the response of the first query 
to generate a second query and then be able to proceed with the retrieval. 
     I think it's fine and will be OK in some cases.
     In 5 bis , I propose to add the possibility that these access methods will 
be described in the query response, and allow to directly retrieve the specific
 view of the observation in the second step.
     I think it's more flexible because the various access modes can be 
different from an observation to another one. 
     This also applies to 2D images where a cutout retrieval can be directly 
generated as soon as the FOV of the Observation (which is part of 
characterisation level 3) and an URL template is given in the query response

7 ) CGPS CADC/CDS prototype.
   First I must say: it is a prototype from the point of view of the
IVOA DAL protocols, but it is an OPERATIONAL SERVICE from the CADC CGPS
archive point of View. 
See  http://www1.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/cgps/navigate.html

    As you can see it is an archive browser and it is not intended for
full analysis but for pre-analysis or to help the astronomer choosing
the data set he needs.

      To describe very shortly the technology used behind the Aladin
metadatree for that:
     There is a query response which is designed exactly like in SIA 1.0
for all the CGPS observations, including the cubes...
     Then comes an extension RESOURCE, in the sense of the SIA extensions
which I described several times in the IVOA DAL meetings on behalf of a
group of people (eg http://www.ivoa.net/twiki/bin/view/IVOA/InterOpOct2005DAL ).
     This extension describes, for each observation, the various acces modes
to the observations given in the first section. There are several access records
for  EACH observation described in the main section.
     To do that we defined for CGPS CADC servive a primitive "AccessAndViews"
model.
     Nothing prevent us to move to the next version of SIA, and to modify
the Access datamodel to a standard one.

     Presently there are only three access modes available:
       - Full retrieval, for download in the WEB broaser
       - 2D preview for loading in Aladin
       - 2D slicer also for loading in Aladin (the range of possible values
being inferred from the query responde metadata)

     But nothing prevent us to add new modes, such as spectra (to 
be given to a collaborative SpecView or VOSPEC) or subcubes (to be
downloaded with help of the viewer, or, why not, given to a 3D tool
in the future), as soon as the server provides this new mode.
    I think this approach can also be tested with other servers than
the CGPS/CADC one.

     I have nothing against Doug's solution, but I would not like the
philosophy of what we have implemented being discarded as "NonVOStandard"
without being considered and discussed.

8 ) conclusion:
   For me what is needed for SIA 1.0 :
     - define a simple but clean DataAccessAndViews or ExtendedAccess model
(I will give you my view on that later)

     - Use this model to define both
        a ) new query parameters as was proposed by Doug
        b ) New Description metadata, to be included in either
main or Extension SIA Query reponse as I proposed.

     I am confident that these two approaches do not oppose but are
complementary as soon as we use the SAME MODEL to define them.

Regards
François

* like the velocity plane browser in our CGPS tool.


=====================================================================
Francois   Bonnarel               Observatoire Astronomique de Strasbourg
CDS (Centre de donnees          11, rue de l'Universite
astronomiques de Strasbourg)    F--67000 Strasbourg (France)

Tel: +33-(0)3 90 24 24 11       WWW: http://cdsweb.u-strasbg.fr/people/fb.html
Fax: +33-(0)3 90 24 24 25       E-mail: bonnarel at astro.u-strasbg.fr
---------------------------------------------------------------------