relative fluxes

[Doug Tody]

On Mon, 29 Jun 2009, Randy Thompson wrote:
>
>  - More importantly, in my opinion it will take more work to generate
>    the needed metadata than to reformat the FITS files. That work will
>    be required whether the metadata is to be added as FITS keywords or
>    just written to the VOTable response.

This is certainly the case.  While it would be easy to do something
simple with these spectra, producing decent metadata could be difficult,
especially in the case of the small, older collections we are mainly
talking about here.  Someone would have to be willing to this work.

> By the way, I don't think the FITS format described in the SDM was
> selected because it was a common FITS format. It was choosen because
> it was a logical format for storing, and describing, spectral data,
> and in theory,  could be expanded to support SEDs, light curves,
> echelle spectra, etc.

The binary table FITS format specified by SDM is fairly widely used,
and powerful enough to accurately deal with a variety of spectra
(as is VOTable of course).  Using a simple 1D image for spectra
on the other hand has serious limitations, e.g., one cannot have
an associated error vector, there is no way to deal with variable
resolution, a background vector cannot be provided, etc.  As Petr
says this use of a 1D image probably originated with IRAF, but IRAF
moved on to more complex formats such as multispec format (or FITS
bintable for many IRAF-based projects) maybe 20 years ago to be able
to deal with these limitations.  If we had had SDM 20 years ago we
would have used that in IRAF instead.

Due to the serious limitations of a 1D FITS image as a spectral format
I don't think we should encourage this in the VO era.  It can be
supported anyway already via native pass-through in a "query compliant"
service, although I do not think this is very satisfactory.  If the
issue is client code then this could be supported as an optional
output format; CSV/TSV for example is already also supported for
simple clients.

If we look at the issue of making it easier to capture data in this
format from small data providers the first thing we would have to do
is take a more careful look at the sorts of use-cases Petr mentions.
How many of these small spectral archives are really worth trying
to get into the VO, given issues such as questionable or nonexistent
metadata?  If there are enough such cases where it would be worthwhile,
I might agree that some step should be taken.  But I still do not
personally know of many such cases, at least from the major ground
based observatories.  Are these groups even operating a basic archive
yet where such spectra are available?  If not, nothing we can do is
likely to solve the problem.  We need a broader inventory of sites
which have such data and which are willing and able to do at least
some work to publish their spectra.

If there were enough such sites to warrant some solution then I still
think giving them a ready to use framework ("SSA in a box") that can
directly ingest the FITS files and handle all the intricacies of SSA
might be a simpler solution for a small site with limited resources
(although metadata could still be an issue).  Another alternative
might be to upload the spectra to a spectral archive service, in
which case the smaller site does not even have to operate an archive.

In summary a more concrete use case is needed.  If all we are trying
to do here is help ESO and SAO or a few major sites like that, then
we probably have the resources to just rework the spectral archives
in question as they are interfaced to the VO.  If the issue is the
small data provider then there are other important issues which would
have to be considered as well.

 	- Doug