relative fluxes

[Douglas Tody]

Hi Petr -

Thanks, that helps a lot.  There are many ways I could respond to this.
I agree we would like to make it easier for small data providers to
publish data and we would like to help get simple spectra into the
VO, provided minimal quality concerns are met.  In the age of big
surveys however, we have massive spectral data collections which are
uniformly processed, with good metadata, and many more on the way.
I have to think that supporting these should be our top priority.
A simpler solution (toolkit or upload tool) may be sufficient to deal
with spectra from small legacy data providers.

The other side of the problem is client spectral tools, which I agree
is a major issue not yet adequately addressed.  In the case of a big
system like IRAF this can be solved by adding SSA support, which is
already planned.  Specview, SPLAT, etc. already have some support and
it is likely to get better as more data comes online.  We have only
been able to properly register SSA services for a few months now,
plus resources are limited and folks have other things to do, so it
is to be expected that client integration will take a while longer.

>>> and in theory,  could be expanded to support SEDs, light curves,
>>> echelle spectra, etc.

SSA can do all these other cases, with minor evolution

> sure - the DAL effort is based on this idea and the SDM will probably become 
> the base for general data set description (am I right Doug ?)

Not the SDM per se, but the Generic Dataset or Observation data
model upon which most of it is based.  By definition this applies to
all data.  The SDM inherits this but in addition addresses general
spectrophotometric data, including spectra/timeSeries/SED.

> The key question is - what is proper VO  metadata ? The full char, provenance 
> and curation, all STC ?
> Show me service supporting all of this !

Of course there aren't any services that provide everything, but
neither is it required to provide all of this metadata.  SSA specifies
a minimum recommended complement of metadata ("should" provide),
which is a subset of the full model, intended to reflect what is
most important for typical analysis scenarios.  The reality is that
effective analysis does not require more than a typical subset of
the full model.  Most analysis gets by with even less than that.
Nonetheless the full model should be more comprehensive, without
becoming overly complex, so that we have a standard way to specify
the metadata in cases where it is useful for the data in question.

> want it ! Very few people care about sigma errors . They want to see the 
> spectrum shape, the background should be substracted already.

I agree that often this is all that is needed, e.g., to identify the
object type.  Depends upon what one is doing.  There are folks who
refuse to take any data seriously unless error information is provided:
this was a major criticism of IRAF back when those 1D-Image spectra
were being produced.

 	- Doug





On Tue, 30 Jun 2009, Petr Skoda wrote:

>
> before I finished the ideas, your comments came - so I will just repeat
> in other words my feelings:
>
>
>> especially in the case of the small, older collections we are mainly
>> talking about here.  Someone would have to be willing to this work.
>
> I know several people who are eager but it has to be relatively easy
>
>
>>> By the way, I don't think the FITS format described in the SDM was
>>> selected because it was a common FITS format.
> No, I think as well it is not so common (except satellites)
>
> It was choosen because
>>> it was a logical format for storing and describing,
> Yes it is a natural model for general spectrum
>
>>> and in theory,  could be expanded to support SEDs, light curves,
>>> echelle spectra, etc.
> sure - the DAL effort is based on this idea and the SDM will probably become 
> the base for general data set description (am I right Doug ?)
>
>
>
>> The binary table FITS format specified by SDM is fairly widely used,
>
> In some specific domain - to be honest I haven't met yet in my list of 
> optical ground based collections the bintable fits as the calibrated 
> scientific result (probably partly due to the simple reason:
> The most of ground optical astronomers do not know what it is exactly 
> bintable FITS and they would not be able to display it ;-)
> No joking - my personal survey among colleagues from variable star research - 
> some are aware of it however - (answer of one colleague who works with Galex, 
> EUVE and SDSS data : "it is a strange kind of FITS, (not the proper one ;-) 
> used by satellites - the only way you can do with it is to use IRAF TABLES 
> tcopy to print to ascii table and you can use
> rspectext to convert in proper (1D FITS) IRAF file  to work with splot"
>
> other just came to me - I have spectra in FITS but it is corrupted - IRAF 
> will not show it !
>
> ......
>
>
>> (as is VOTable of course).  Using a simple 1D image for spectra
>> on the other hand has serious limitations, e.g., one cannot have
>> an associated error vector, there is no way to deal with variable
>> resolution, a background vector cannot be provided, etc.
>
> Absolutely - it is a one of problems with echelle reduction - the merging of 
> orders and rebining in wavelenght is incorrect and introduces errors !
> All the effort of UVES, HARPS, FEROS, HEROS .... pipelines is in vain 
> converted to this format. However, it is the final science result and people 
> want it ! Very few people care about sigma errors . They want to see the 
> spectrum shape, the background should be substracted already.
>
> I am afraid that it would be nice if the astronomers were educated by VO 
> standards to properly work with spectra (and realize the issues of 
> background, sigma errors etc -  but even IRAF has the support for this for 
> very short time an only in specific packages (e.g. gemini), not in common 
> ones recommended in tutorials...
> Not talking about MIDAS FITS convertor (from MIDAS tables) ...
>
> As Petr
>> says this use of a 1D image probably originated with IRAF, but IRAF
>> moved on to more complex formats such as multispec format
> yes the result of reduction is multispec with 4 bands - but many reductions 
> are finished by scopy to onedspec or the extensions are eliminated ...
>
>> bintable for many IRAF-based projects
> - well all space based AND perhaps Gemini
>
>> 
>> Due to the serious limitations of a 1D FITS image as a spectral format
>> I don't think we should encourage this in the VO era.
>
> Not encourage but tollerate it in clients (maybe in some transition period 
> before the new SDM and SxAP for GDS (general data sets)  is approved.
> But we have to be able to  show some substantial amount of SSA services with 
> various set of spectra to be consumed in ALL VO clients...
>
> I am still doubting about the sense of current spectra services (except the 
> role of visualisation - the serious analysis still has a lot of problems 
> including the limitation of SDM as implementes...
>
>> supported anyway already via native pass-through in a "query compliant"
>> service,
> Astronomers do want to get a list of spectra - they want to see them in VO 
> tools
>
>> issue is client code then this could be supported as an optional
>> output format; CSV/TSV for example is already also supported for
>> simple clients.
>
> I do not understand the difference in supported as optional output format 
> (case of CSV),
> and the explicit  list in table in chapter 4.1.1.5 od SSAP 1.04.
>> 
>
>> How many of these small spectral archives are really worth trying
>> to get into the VO, given issues such as questionable or nonexistent
>> metadata?
>
> The key question is - what is proper VO  metadata ? The full char, provenance 
> and curation, all STC ?
> Show me service supporting all of this !
> I bet 99.999 of spectra comming from SSA services do not have proper metadata 
> (including the latest ESO SSA) I do not know about space project what it is 
> required.
> But I can tell all the observatories producing the spectra have enough 
> metadata described in FITS headers all elsewhere in Logs, to be able to do 
> their science analysis. And I am afraid we will ever have the whole 
> provenance and characterisation in all data sets as required bymodels in 
> preparation. It would require formal description of whole pipeline, all 
> information from telescope (including the meteo - e.g. the humidity 
> influencing telluric lines....) and much more else (Igor could list for 
> several more pages .....)
>
> And still not talking about IFU spectra, multigrating and multichip echelles 
> etc ...
>
> But despite this lack of metadata - one look at spectrum shape will tell to 
> experienced astronomer more than reading pages of metadata .... ;-)
>
>
>> If there are enough such cases where it would be worthwhile,
>
> every information counts ! We want to pretend that VO is a heritage of 
> mankind where the new kind of research is allowed by the technology.
> And that implies that ALL of the astronomical knowledge should be there.
> Do we more care about proper curation and provenance of Almagest tables or 
> the content ?
> What is the border between VO as presented to getting funds (unique, 
> federalization, all archives .... bridging the Digital divide) and the VO as 
> the just another version of large (space) mission archives  - like extended 
> MAST.
>
> And even with the space missions - will we have Spitzer, Chandra, Corrot, 
> Kepler, Herschel, Planck data in VO ? In what format when the consortia have 
> been developing different non-VO compatible archives ?
>
>
>> I might agree that some step should be taken.  But I still do not
>> personally know of many such cases, at least from the major ground
>> based observatories.
>
>
>> Are these groups even operating a basic archive
>> yet where such spectra are available?
> Some kind of it for internal purposes based on some public RDBMS (postgresql, 
> Mysql) and Apache is quite common but the data are here as it is. Those who 
> are not showing data might be convinced by promissing them the VO access will 
> make their facility more famous ....
>
> An still there is a way of using VO tools (if enough enhanced for particular 
> type of analysis) for the work with proprietary data in local environment 
> instead of legacy tools.
>
>
>
>> likely to solve the problem.  We need a broader inventory of sites
>> which have such data and which are willing and able to do at least
>> some work to publish their spectra.
>
> To convince some of them, a clear advantage of VO spectra access has to be 
> demonstrated - and after my experience with SSA servers and spec tools it is 
> still not massive scaled science (hardly several spectra can be handled at 
> once, not hundreds.... And the lack of "prenormalisation" and basention of 
> absolute flux calibration is critical to display different spectra in one 
> image. (I am preparing some critical review of SSA from the point of view of 
> a stellar astronomer).
>
> So again the VO client development is important ..
>
> And still there is a lot of spectra in ascii tables in Vizier that should be 
> SSA accessed - I am not sure about the TAP but it seems not support spectra 
> in its current version...
> One example for all:
> http://web.pd.astro.it/adsd
>
> here is the real science content - even if lacking some metadata.
>
>
>> If there were enough such sites to warrant some solution then I still
>> think giving them a ready to use framework ("SSA in a box") that can
>> directly ingest the FITS files and handle all the intricacies of SSA
>> might be a simpler solution for a small site with limited resources
>
> Yes I have noted this in previous email - e.g SAADA is near this goal.
>
>
>> might be to upload the spectra to a spectral archive service, in
>> which case the smaller site does not even have to operate an archive.
>
> have a look at the BeSS and its content (if it is still accesible)
>
>> In summary a more concrete use case is needed.  If all we are trying
>> to do here is help ESO and SAO
>
> But once this is solved at ESO ... there will be the example of correctly 
> prepared spectra archive that may be used as study case for the smaller 
> providers.
>
>
>> small data provider then there are other important issues which would
>> have to be considered as well.
>
> Exactly ! I tried to put some issues for consideration.
> Now I am done with all the experience and feelings of small data providers.
>
> This is now turn of Alberto as a representant of the richest and most 
> powerful astronomical company in the ground based instrumentation ;-)
>
> Petr
>
> *************************************************************************
> *  Petr Skoda                         Phone : +420-323-649201, ext. 361 * * 
> Stellar Department                         +420-323-620361           *
> *  Astronomical Institute AS CR       Fax   : +420-323-620250           *
> *  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
> *  Czech Republic                                                       *
> *************************************************************************
>