general comments to SSAP and SDM from outside

[Nicholas A Walton]

Dear Petr

you make some very valid points below. I would like to fully concur with 
you - the publication of 'normalised' spectrum through the VO would be 
very useful - and as you note - for many purposes, normalised spectra 
are the 'science' product - they need never be flux calibrated in a 
formal sense - as the output measurements are usually just equivalent 
widths.

Thus, the SSAP spec should allow for the publication of normalised 
spectra, and as you note - if this were/is the case - it should make it 
a lot easier to get ground based spectra out through the VO.

Yours, Nic Walton

Petr Skoda wrote:
>
> Dear colleagues,
>
> I decided to enter IVOA DAL after having discussion with some of you 
> during both ESAC workshops in March and June and after reading the 
> SSAP1.01 and DM documents and mailing lists archive concerning it. I 
> am not sure about sending the copy of this to DM list, but hope that 
> this will be read by DM people as well (I am not sure if the spectrum 
> DM is part of DM group or if it is rather a business of DAL -- feeling 
> some fuzziness of the competences). So I am sending my general 
> comments here to DAL group and further detailed comments to both 
> documents will be sent soon in respective list.
>
>
>
> I try to understand all the reasons leading to the Spectrum data model 
> and SSAP as it proposed, but after discussion with many of you, I feel 
> need to give my impression of the overall effort connected with SDM 
> and SSAP from outside - as a representant of a VO-aware scientific 
> community having experience with the stellar optical spectroscopy and 
> knowing well the data reduction procedures of most world optical 
> spectrographs (both echelle and single order).
>
> I have a feeling that all the spectral part of a VO is being developed 
> in a "space-instruments-centric" manner. Of course, it is clear the 
> development is driven by requirements of large (mostly space-based) 
> projects. Unfortunately, the developers are surrounded by scientists 
> working only with particular type of data and from here comes a false 
> feeling that most of a spectra should have similar properties (to 
> those produced by satellites) and what is different is "obsolete" or 
> "incomplete" and thus not worth of proper handling in VO. (e.g. 1D 
> FITS in image format)
>
> Maybe I am wrong, but my current understanding of SSA and SDM is that: 
> the only one good spectra format is VOTable and FITS with binary 
> tables. The spectrum without absolute flux calibration in physical 
> units (like W cm-2 s-1 A-1) is considered not "fully science-ready" 
> because of missing Flux-axis information etc ... When I asked some of 
> you how to display continuum normalized spectra, I was told exactly 
> this - as the protocol pre 1.0 required to represent SCALEQ and DIMEQ 
> in units, I was told the units for flux should be "n/a" .....
>
>
> I think that normalization does not deserve such a neglection: Let me 
> comment on this a little: I think it is again space-centric view: the 
> flux calibration for ground-based spectra is quite rare - only for 
> certain studies (e.g spectral classification) is done - mostly on 
> low-resolution spectra - morever it is extremely difficult to do it 
> precisely (changing extinction, seeing ...).
>
> The major part of ground based optical spectra from middle and high 
> resolution spectrographs is uncalibrated (in some counts or data 
> numbers - after the extraction by pipeline) and most people make the 
> science on them after continuum normalization - look at the examples 
> of many graphs in ApJ or A&A !
>
> Moreover all the modern high resolution spectrographs are echelle - 
> here the separate orders are published normalized. If the trials are 
> made to merge the orders in one long spectrum (with very uncertain 
> results due to complexity of behaviour of blaze function) the final 
> spectrum is in artificial units like counts or is nomalized again. I 
> have not yet see flux-calibrated spectrum from GROUND-BASED echelle.
>
> All abundance studies, asteroseismology, multiple stars disentangling, 
> time evolution of envelopes, circumstellar shells or stellar winds 
> needs normalized spectra. The RV measurement - either direct or by 
> cross-correlation needs well normalized spectrum as well (at least for 
> processing by classical programs like fxcor)
> Even the properly calculated synthetic spectra (in absolute flux) are 
> usualy  normalized to continuum to allow the comparison with 
> observations (and sometimes the continuum is corrected using the 
> synthetic model)
>
> I think the ROUGH spectral classification of unknown objects is the 
> only application for absolute flux calibrated spectra. (especially in 
> extragalactic research). Low resolution spectra are used for this. But 
> even the topics like detailed spectral classification or analysis of 
> chemical composition (if done properly) are comparing continuum 
> normalized synthetic and observed spectra (e.g. by Chi^2 fits).
> The problem how to fit the physical continuum is another story 
> (depending on experience and physical nature of object - but in 
> general it has to be done somehow to allow most of the analysis named 
> above).
>
>
> The space-based research works mostly with SEDs - OK the collection of 
> various spectral regions is a nice demonstration of VO power (and most 
> often mentioned in VO propaganda;-) so the non-VO aware astronomer can 
> get the feeling that all the VO effort is just done for collection of 
> very rough multispectral data and its main result is the SED !
>
>
> I think the continuum - normalized spectra should be regarded as fully 
> science ready both in observation and theory and proper support should 
> be given to them in VO tools and protocols. (I will comment more on 
> this in next mails) Concerning DAL business - one of SSA results can 
> be (in addition to VOTable) the FITS. The tricky part is what FITS (it 
> is implied binary tables FITS - but it is not seen immediately in both 
> DM and SSAP proposals.
>
>> From further reading of previous documents and general history of
>  SSAP I got the feeling that all versions except the binary tables are 
> considered not worth of support and let to the benevolence of the 
> client to support it and display - but rather it is wanted to let it 
> go through as the NATIVE format to some external legacy application. I 
> understand this - every project can have special format of data - for 
> most complicated instruments the bintable FITS were used.
>
> But again most of optical ground spectra are not in bintables but in 
> 1D images! The answer is following: When someone is asked to reduce 
> spectra (if not having pipeline derived from some space-project) he 
> takes either MIDAS or IRAF tasks to produce extracted spectra(in 
> onedspec or echelle format). As the IRAF is still dominating the 
> world, the results of MIDAS reduction has to be finally converted to 
> FITS readable by IRAF splot or spectool as the most powerful spectra 
> analysis tools. On the other hand MIDAS can read IRAF-produced FITS if 
> rebinned and the non-linear WCS (using WAT keywords - unique for IRAF) 
> is replaced by simple CDELT1, CRVAL1.
>
>
> So from both sides we end with 1D FITS image format (perhaps with 
> several more spectra for variance etc ..) Echelle spectra have to be 
> expanded into separate files for each echelle order (again with 
> CDELT1, CRVAL1), otherwise they are not interchangeable with other 
> tools. Very seldom are the spectra analyzed in form of wavelength-flux 
> ASCII table and sure not in VOTable format - but all the middle and 
> high resolution spectroscopy I have seen (and discussed at different 
> stellar conferences) is usually conducted in 1D FITS where only CDELT1 
> and CRVAL1 and NAXIS=1 is present -- it is a common denominator of all 
> formats used for spectra exchange.
> Many one-man customized tools for spectrum analysis are even looking 
> for only these keywords in header directly (a lot of experienced 
> scientists I know are not even aware of FITSIO library)
>
>
> So I think that the requirement for binary table FITS should be 
> complemented by 1D FITS image in the simple form using CRVAL,CDELT1. I 
> know a lot of professional astronomers who understand the FITS format 
> only in this form (never heard about binary tables) - and I think the 
> most astronomers can work with such a format and know how to display 
> such a spectra or have converters to their favourite tools. (BTW even 
> amateur spectroscopic SW or commercial SW for laboratory spectroscopy 
> provides 1D FITS with only few keywords and CRVAl1, CDELT1... after 
> extraction, too)
>
> The objections against rebinning (loosing the accuracy) are clearly 
> understandable, but quietly ignored (and proved not to be critical). 
> Many people would immediately benefit from the VO tools supporting 
> (esp. displaying) and servers delivering spectra in such a format. 
> After all the metadata should clearly describe what the axes mean in 
> what units and what is a precision and treatment of the data (e.g. by 
> the pipeline).
>
> The problem with serialization is here as well (I need to know the 
> size of each axis...) but to be honest - what is a longest spectrum in 
> pixels or sigle echelle order? - only several thousand! (today 4600 as 
> largest CCD chip - the mosaicing of chips for coverage of one whole 
> spectral (echelle) order is nonsense - such a spectra should be 
> divided to two parts as a complex object) So the keeping of several 
> thousand of numbers in a memory should not be serious serialization 
> problem.
>
> The effort of making the standards overwhelming everything conceivable 
> is appreciated, the purity of the model is understandable - it is 
> clear that the UCD should be devised for every physical variable... 
> But the question is what is important for scientists today and who are 
> the scientists for whom the VO is build for. The VO should give ASAP 
> something not only for SED investigators and galaxy classifiers but it 
> should attract all the "classical" stellar astronomers as well. Not 
> only as consumers but as providers as well.
>
> There is a wealth of spectra being produced by spectrographs on 
> smaller telescopes all over the world and reduced mostly in IRAF (or 
> MIDAS). Despite the problem of data privacy (another story ...) there 
> is a will to publish even small collection of spectra in VO - but 
> there is a lack of tools and tutorials how to do this easily.
>
> BTW - if you look at list of 21 SSA services in VO Registry most of 
> them are space data. Only 3 ground based archives are there - some 
> services are just retooling another server data (Elodie data in BeSS) 
> or are just project data from several days of observation elsewhere. 
> Most of the spectra from ground are served by a  single server tool 
> (Pleinpot by P. Prugniel)
>
> I am one of scientists with SW development abilities who has some 
> knowledge of VO (and tries to understand it more deeply) - but current 
> VO spectra publishing tools (as presented at June workshop) are not 
> helping too much to publish easily small collection of IRAF reduced 1D 
> spectra - BTW I was not able to find any tool allowing easily the 
> conversion of 1D fits to binary tables (e.g. the TABLES package tasks 
> did not accept such result of IRAF *ms* spectra produced by apall tasks).
>
> In the examples in appendices of both documents there is a lot of 
> stuff but no simple recommendations how to use the documents for 
> building simple SSA 1.01-compliant service of a bunch of normalized 1D 
> FITS, what might be the main goal of many people from general 
> astronomical community.
>
> I am pretty sure that some tutorial showing a minimal required params 
> and the example values for such a simple spectra is required to 
> attract more spectra publishers from smaller observatories and 
> amateurs as well ..
>
>
> I didn't mean this like the critics of VO effort and I do not want my 
> objections to delay the approval of documents - but I want just to 
> emphasize the practical needs of ground-based astronomical community. 
> I have presented more inputs for further VO development in my March 
> presentation (and if somebody is interested I would be glad to discuss 
> the potential benefits of VO approach for everyday astronomical work.
>
>
> Best regards,
>
> Petr Skoda
>
> *************************************************************************
> *  Petr Skoda                         Phone : +420-323-649201, ext. 
> 361 * *  Stellar Department                         
> +420-323-620361           *
> *  Astronomical Institute AS CR       Fax   : +420-323-620250           *
> *  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
> *  Czech Republic                             pskoda at mbox.cesnet.cz     *
> *************************************************************************

-- 
Dr N. A. Walton                  
(AstroGrid Project Scientist	       http://www.astrogrid.org)   
(Euro-VO VOTC Project Scientist	       http://www.euro-vo.org)
Institute of Astronomy          Tel:   +44 1223 337503
University of Cambridge         Fax:   +44 1223 337523
Madingley Road                  WWW:   http://www.ast.cam.ac.uk/~naw
Cambridge, CB3 0HA              email: naw at ast.cam.ac.uk