UCD problem in SSA/SpectrumDM

[Douglas Tody]

Hi Petr -

On Fri, 20 Nov 2009, Petr Skoda wrote:

>> I also think that a minor revision is warranted - mostly minor fixes
>> (we have only a couple so far including Alberto's) and clarifications
>> to text that was found to be insufficiently precise during the
>> implementation phase.  We have been saving some of these up for the
>> next doc update.
>
> If the work on small changes starts, I would like to add more clarification 
> to the concept of Flux calibration (e.g. the keyword NORMALIZED seems to 
> bring some ambiguity (e.g. with respect to spectral synthesis models).
>
> And maybe the required concepts of spectra cutout, rebining and normalization 
> could be introduced here without changing the DM and protocol - just by some 
> explanatory add-ons.

Ok, some clarification would be fine here, and this has been a point of
confusion I guess.

> It would be good to start the discussion about the concept of the data 
> collection as one service versus a buch a separate services (e.g. in ESO case 
> - is it better to put all spectra from different spectrographs on one server 
> with one URL ? or use several service URLs?
>
> It is very difficult then to separate them as there is not even optional) 
> query parameter to instrument or instrument mode and I think the collection 
> is not what it could solve the problem .... Simply this requires little 
> discussion and some practical rules ...

This is a perennial problem; I am not sure we can solve it in a minor
document revision.  COLLECTION is a good start as it abstracts the
issue to the scienfific type of data being searched/accessed.

In general I think it is better to classify data by the scientific
characteristics (collection, instrument) rather than by the origin
(ESO).  Just putting up one service for all data at a site is too
simplistic.

> And what is an absolute must is a case of photometric series - e.g. the COROT 
> archive in SSA is an example of service that is urgently needed for science 
> work (and so cannot wait for proper definition in protocol, although is using 
> the concept of Data model).

> But the SSA clients are not properly understandig what to do with the data.
>
> As was said the science should be the main driver of IVOA and so we should 
> reflect this need ASAP.

I don't understand the problem Petr - can you expand upon this?
SSA can support time series of spectra for example, if we just search
based upon time.  True photometry data would be better handled by a
TimeSeries service.

> Simply this minor revison (if it will be released) could correct many 
> misunderstandings by more explanatory material (and correct errors) but will 
> not introduce any new concepts - just may describe what is meant (and hint 
> how to implement) the vaguely said concepts (e.g. Virtual data in sec.2 ) and 
> extend the part of GetData (the section 5 is too sparse)  and AcRef.

Well we can improve matters I expect, but we will never completely
eliminate confusion and misunderstanding.  Ultimately some actual human
interaction may be needed, e.g. a support discussion list/newsgroup.

> And just a science case confirming the need of postprocessing:
>
> It is clearly seen that the spectrum cutout has to be supported on a 
> reasonable high dispersion spectra

> For  example - the  UVES or HARPS spectra inside the ESO network takes tens 
> of sec to download - I can imagine how to get hundereds of them (e.g. for 
> asteroseismology studies)

Yes, this is important for services that service high resolution
spectra.

> But you are still not able to display them on the screen - so usually must be 
> zoomed - and the client may do it for a longer time than to download it 
> repeatedly from server who will send only short pieces (not talking about 
> memory limitations).

This would appear to be a client app issue.  It is similar to
the multi-resolution problem for images.  Ultimately we need a
multi-resolution solution, plus cutouts for very small regions such
as individual spectral lines.

> Simply I think that if we want the astronomers to use VO tools we have to 
> solve the issues of fast response and large data volumes by clever design of 
> particular service thinking about the practical impact on a user feeling. And 
> my experience form practical usage of SSA services even on a high speed 
> network is awfull  - you can show 10-20 spectra overploted after some time 
> but not hundreds or thousands as would be expected from "petabyte science 
> surveys"
>
> So I strongly support the idea of minor SSA revison instead of waiting for 
> SSA2 introducing fancy concepts of features required by few services (e.g. 3D 
> spactroscopy, IFU and so on ). I am sure that by proper hints and examples 
> most of the useful things may be done already now - but is has to be said 
> explicitly how.

SSA already gets us much of the way towards providing these
capabilities - it already provides more than any application has yet
used.  Ultimately we need more work on the client application side
before it is worthwhile to evolve the service capability.  The main
problem here is simply that these days we have such limited resources
available for advanced analysis software development.  I think this
is more of an issue currently than what we have available for VO
development.

 	- Doug