Updated ImageDM and SIAV2 drafts

[Douglas Tody]

On Sat, 28 Sep 2013, Slava Kitaeff wrote:

> I'd like to provide a quick feedback
> on http://wiki.ivoa.net/internal/IVOA/ImageDM/WD-ImageDM-20130812.pdf.
> 
> The references to FITS in the document are quite numerous. Some of the statements,
> e.g. on FITS good performance, can be challenged. I'm raising this because the
> radioastronomy community is rapidly moving away from FITS being faced with its
> limitations.

I agree that FITS runs into performance problems for large datasets.
I suspect the comment about performance you are referring to was just
stating that representing the data as a n-D array can have major
advantages for performance, as opposed to for example trying to work
directly on a visibility dataset.  We meant n-D arrays in general, not
FITS in particular.

> I'd question if a reference to a file format is necessary in a data model
> at all. Shouldn't the data model describe the data, and then a necessary format be
> produced as required if the server or client are equipped with a capability - FITS
> (uncompressed, compressed with H-transform, GZIP_1, GZIP_2, RICE: 5 options for
> encoding), JPEG, PNG, JPEG2000 (two options for encoding with many parameters), HDF5
> (several options for encoding) etc?

The approach is to define the data model abstractly, independent of
serialization.  We just give some examples of serializations that we
expect to be used for cube data in astronomy, for which mapping from the
ImageDM are likely to be required.  Other serializations are certainly
possible.  When we produce actual data services like SIAV2 it will also
be necessary to say what is mandatory to support to return data to a
client.  FITS would probably be the mandatory output format, but more
advanced services may support other serializations (JPEG2000, HDF5,
whatever).

Speaking of JPEG2000, accessData (SIAV2) can support streaming of data
back to the client application.  So, given a sufficiently powerful
service, it will be possible to access a portion or view of a large cube
and stream JPEG2000 encoded data back to the client, allowing advanced
capabilities such as progressive display in an interactive client.  The
same service might return the data subset in FITS format if that is what
the client wants back.

 	- Doug


> Regards,
> Slava
> 
> 
> 
> 
> ----------------------------------------------------------
> Dr Slava Kitaeff
> 
> Research Associate Professor
> International Centre for Radio Astronomy Research
> The University of Western Australia
> 
> Ph. +61 8 6488 7744
> 
> 
> 
> 
> On 12/09/2013, at 4:33 AM, Douglas Tody <dtody at nrao.edu> wrote:
>
>       The Image data model (ImageDM) and associated image access protocol
>       (SIAV2) documents have been updated on the IVOA TWiki (updated versions
>       were actually uploaded several weeks ago to support author group
>       discussions, but were not announced at the time).
>
>       The ImageDM doc was largely written from scratch and replaces the
>       placeholder version from the May interop.  The SIAV2 draft has also been
>       largely rewritten to make use of the closely linked ImageDM and other
>       recent DAL developments such as DALI.  Full support is provided for
>       multidimensional image data including large data cubes and sparse cubes.
>
>       The drafts live on these pages:
>
>           http://wiki.ivoa.net/twiki/bin/view/IVOA/ImageDM
>           http://wiki.ivoa.net/twiki/bin/view/IVOA/SiaInterface
>
>       Neither document is complete yet, although the ImageDM draft comes close
>       so far as the core model is concerned.  The main thing to be updated at
>       this point is the data access model for slicing and dicing, computing
>       moments,etc., of large image cubes.  We hope to update one or both
>       documents prior to the interop to reflect author group discussions and
>       prototyping over the past month or so.
>
>       These drafts are also the basis for the cube project data access (SIAV2)
>       prototype being developed by VAO, as Ray mentioned on the list a few
>       days ago.  The VAO SIAV2 prototype includes test multidimensional image
>       datasets for ALMA, JWST, JVLA (and legacy VLA), JCMT, Keck (Osiris), and
>       several cube datasets from NED.  See the following for details and
>       status; http://wiki.ivoa.net/twiki/bin/view/IVOA/SIA2VAOPrototype
> 
> 
> 
>