relative fluxes

[Petr Skoda]

before I finished the ideas, your comments came - so I will just repeat
in other words my feelings:


> especially in the case of the small, older collections we are mainly
> talking about here.  Someone would have to be willing to this work.

I know several people who are eager but it has to be relatively easy


>> By the way, I don't think the FITS format described in the SDM was
>> selected because it was a common FITS format.
No, I think as well it is not so common (except satellites)

It was choosen because
>> it was a logical format for storing and describing,
Yes it is a natural model for general spectrum

>> and in theory,  could be expanded to support SEDs, light curves,
>> echelle spectra, etc.
sure - the DAL effort is based on this idea and the SDM 
will probably become the base for general data set description (am I right 
Doug ?)



> The binary table FITS format specified by SDM is fairly widely used,

In some specific domain - to be honest I haven't met yet in my list of 
optical ground based collections the bintable fits as the calibrated 
scientific result (probably partly due to the simple reason:
The most of ground optical astronomers do not know what it is exactly 
bintable FITS and they would not be able to display it ;-)
No joking - my personal survey among colleagues from variable star 
research - some are aware of it however - (answer of one colleague who 
works with Galex, EUVE and SDSS data : "it is a strange kind of FITS, 
(not the proper one ;-) used by satellites - the only way you can do with 
it is to use IRAF TABLES tcopy to print to ascii table and you can use
rspectext to convert in proper (1D FITS) IRAF file  to work with splot"

other just came to me - I have spectra in FITS but it is corrupted - IRAF 
will not show it !

......


> (as is VOTable of course).  Using a simple 1D image for spectra
> on the other hand has serious limitations, e.g., one cannot have
> an associated error vector, there is no way to deal with variable
> resolution, a background vector cannot be provided, etc.

Absolutely - it is a one of problems with echelle reduction - the merging 
of orders and rebining in wavelenght is incorrect and introduces errors !
All the effort of UVES, HARPS, FEROS, HEROS .... pipelines is in vain 
converted to this format. However, it is the final science result and 
people want it ! Very few people care about sigma errors . They want to 
see the spectrum shape, the background should be substracted already.

I am afraid that it would be nice if the astronomers were educated by VO 
standards to properly work with spectra (and realize the issues of 
background, sigma errors etc -  but even IRAF has the support for this for 
very short time an only in specific packages (e.g. gemini), not in common 
ones recommended in tutorials...
Not talking about MIDAS FITS convertor (from MIDAS tables) ...

As Petr
> says this use of a 1D image probably originated with IRAF, but IRAF
> moved on to more complex formats such as multispec format
yes the result of reduction is multispec with 4 bands - but many 
reductions are finished by scopy to onedspec or the extensions are 
eliminated ...

> bintable for many IRAF-based projects
  - well all space based AND perhaps Gemini

>
> Due to the serious limitations of a 1D FITS image as a spectral format
> I don't think we should encourage this in the VO era.

Not encourage but tollerate it in clients (maybe in some transition period 
before the new SDM and SxAP for GDS (general data sets)  is approved.
But we have to be able to  show some substantial amount of SSA services 
with various set of spectra to be consumed in ALL VO clients...

I am still doubting about the sense of current spectra services (except 
the role of visualisation - the serious analysis still has a lot of 
problems including the limitation of SDM as implementes...

> supported anyway already via native pass-through in a "query compliant"
> service,
Astronomers do want to get a list of spectra - they want to see them in VO 
tools

> issue is client code then this could be supported as an optional
> output format; CSV/TSV for example is already also supported for
> simple clients.

I do not understand the difference in supported as optional output format 
(case of CSV),
and the explicit  list in table in chapter 4.1.1.5 od SSAP 1.04.
>

> How many of these small spectral archives are really worth trying
> to get into the VO, given issues such as questionable or nonexistent
> metadata?

The key question is - what is proper VO  metadata ? The full char, 
provenance and curation, all STC ?
Show me service supporting all of this !
I bet 99.999 of spectra comming from SSA services do not have proper 
metadata (including the latest ESO SSA) I do not know about space project 
what it is required.
But I can tell all the observatories producing the spectra have enough 
metadata described in FITS headers all elsewhere in Logs, to be able to do 
their science analysis. And I am afraid we will ever have the whole 
provenance and characterisation in all data sets as required bymodels in 
preparation. It would require formal description of whole pipeline, all 
information from telescope (including the meteo - e.g. the humidity 
influencing telluric lines....) and much more else (Igor could list for 
several more pages .....)

And still not talking about IFU spectra, multigrating and multichip 
echelles etc ...

But despite this lack of metadata - one look at spectrum shape will tell 
to experienced astronomer more than reading pages of metadata .... ;-)


>If there are enough such cases where it would be worthwhile,

every information counts ! We want to pretend that VO is a heritage of 
mankind where the new kind of research is allowed by the technology.
And that implies that ALL of the astronomical knowledge should be there.
Do we more care about proper curation and provenance of Almagest tables or 
the content ?
What is the border between VO as presented to getting funds (unique, 
federalization, all archives .... bridging the Digital divide) and the VO 
as the just another version of large (space) mission archives  - like 
extended MAST.

And even with the space missions - will we have Spitzer, Chandra, Corrot, 
Kepler, Herschel, Planck data in VO ? In what format when the consortia 
have been developing different non-VO compatible archives ?


> I might agree that some step should be taken.  But I still do not
> personally know of many such cases, at least from the major ground
> based observatories.


> Are these groups even operating a basic archive
> yet where such spectra are available?
Some kind of it for internal purposes based on some public RDBMS 
(postgresql, Mysql) and Apache is quite common but the data are here as it 
is. Those who are not showing data might be convinced by promissing them 
the VO access will make their facility more famous ....

An still there is a way of using VO tools (if enough enhanced for 
particular type of analysis) for the work with proprietary data in local 
environment instead of legacy tools.



> likely to solve the problem.  We need a broader inventory of sites
> which have such data and which are willing and able to do at least
> some work to publish their spectra.

To convince some of them, a clear advantage of VO spectra access has to be 
demonstrated - and after my experience with SSA servers and spec tools it 
is still not massive scaled science (hardly several spectra can be handled 
at once, not hundreds.... And the lack of "prenormalisation" and 
basention of absolute flux calibration is critical 
to display different spectra in one image. (I am preparing some critical 
review of SSA from the point of view of a stellar astronomer).

So again the VO client development is important ..

And still there is a lot of spectra in ascii tables in Vizier that should 
be SSA accessed - I am not sure about the TAP but it seems not support 
spectra in its current version...
One example for all:
http://web.pd.astro.it/adsd

here is the real science content - even if lacking some metadata.


> If there were enough such sites to warrant some solution then I still
> think giving them a ready to use framework ("SSA in a box") that can
> directly ingest the FITS files and handle all the intricacies of SSA
> might be a simpler solution for a small site with limited resources

Yes I have noted this in previous email - e.g SAADA is near this goal.


> might be to upload the spectra to a spectral archive service, in
> which case the smaller site does not even have to operate an archive.

have a look at the BeSS and its content (if it is still accesible)

> In summary a more concrete use case is needed.  If all we are trying
> to do here is help ESO and SAO

But once this is solved at ESO ... there will be the example of correctly 
prepared spectra archive that may be used as study case for the smaller 
providers.


> small data provider then there are other important issues which would
> have to be considered as well.

Exactly ! I tried to put some issues for consideration.
Now I am done with all the experience and feelings of small data providers.

This is now turn of Alberto as a representant of the richest and most 
powerful astronomical company in the ground based instrumentation ;-)

Petr

*************************************************************************
*  Petr Skoda                         Phone : +420-323-649201, ext. 361 * 
*  Stellar Department                         +420-323-620361           *
*  Astronomical Institute AS CR       Fax   : +420-323-620250           *
*  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
*  Czech Republic                                                       *
*************************************************************************