DataLink issues

Fri Oct 5 07:50:19 PDT 2012

Le 26/09/2012 13:05, Petr Skoda a écrit :
> Hi all
>
> I was waiting to see more discussion about datalink ongoing (in particular I expected to see comments by Markus Demleitner ;-)
> but as it seems to be converging to some extra-big beast I think the pragmatic view is necessary:
I'll do my best.

>
> The question which is not still answered is - what is datalink needed or useful for ? Not what it could do in principal !

The answer is rather simple (from my point of view): Datalink is a mechanism allowing to get extra-data attached to one specific 
datafile. This extra data can either be a simple file or the result of some online processing.
>
> I still do not see clear boundary between DAL (specific, detailed..) service andand discovery service of alternative
> representation - which is what DL is presented as.
Getting ancillary data (housekeeping, calibration, preview...) with DAL services makes no sense since it is attached to a 
specific product files and, right now, DAL protocols are unable to handle such logical links.
>
> In May I was hoping DL could solve my (still pending) wish of postprocessing spectra (cutout, normalize, resolution degradation
> etc ...) in a clear and well formalized way. After practical experience with that I have found a catch in data format as a
> primary source of complexity. And as I remember the end of special session about DL was a general acceptance of DL as a "easily
> clickable" fork of accessing static data.
Easily clickable dos not mean that you get data at the first click. That means that "something happens" as Doug said.
>
> That's why we decided with Markus to prototype the SSA getData operation instead of relying on DL to solve it in universal
> "S*APv2" way on GDS level, which is imho still too ambitious taking into account current achievements in practical
> implementations of IVOA protocols in tools.
>
> But now I see that the tendency of making DL the all-purpose "one-link-to-rule-them-all" standard.
>
> So lets take it in simple case of SSA analogy, which is still field I understand well (I hope;-): sorry for diverging to SSA:
>
> In SSA we have couple of parameters (PQL like ;-) which in all current implementations (except mine) means a restriction of
> selection - i.e giving the BAND, TIME and FORMAT I restrict the space of possible answers only to data set containing the given
> spectral range, exposed in given epoch and expressed in given format . In my effort I am trying to introduce the postprocessing
> using the parameters as a control set of params according which the postprocessing works - e.g. I command the service to cut the
> region I want using BAND or if giving FLUXCALIB I force the data to be on-the-fly normalized. To do this I need two services -
> one interprets the params as control params the other makes just the selection as was the original intention.
>
> But the catch is in FORMAT. All data in our case are in NATIVE (1D FITS) and the on-the-fly processing is done when asking for
> FITS, VOTABLE or COMPLIANT. But the postprocessing can work only on the bintable FITS (for which we have simple cutouts in
> wavelengths - not pixels as the WCS may be non-linear - OTOH we cannotgive the number of pixels of output for cutout operation
> in metadata) . This means that the postprocessing implies the conversion of format and it is difficult to state the metadata for
> spectral coverage or for dagta size just after basic querydata operation - as the exact cutout may be done on pixels which may
> represent different wavelengths than requested in BAND - and say after rebinning the amount of data is resulting from algorithm
> that was not yet run after queryData - only the getData will know it when finished.
>
> So in general, the problem of virtual data sets is we cannot describe them in metadata before the real operation is run.
>
> Please notice I use as well parameter FORMAT as a operation with unknown results ;-) But to be able to work with it in intuitive
> manner - we have to pretend the existence of all files in different format (having multiple times the number of rows in a
> database) and the SSA keyword will restrict the number of returned rows (whan asking for COMPLIANT it returns only VOT and
> bintable fits as applic.fits , when asking for ALL or ANY it gives as well the image/fits , if asking NATIVE gives me only that
> one). So I have to be sure I am able to do the data conversion always and pretend I have it in a database.
>
> Here you see the complications with simple protocol like SSA (although so far it is the best way how to do some real
> spectroscopic analysis instead of just displaying the static data)
>
> And suppose the ambitions of DL : accesing complex data sets, process them using given parameters, access more specific metadata
> according to the nature of data set, even UWS service for computational intensive tasks ......  !!!
>
> I think that it would be nightmare to implement the clients to work with it. Beware that we need to state the parameters for DL,
> formalize the semantics of them and they would need to be able to be sent deeper to the underlying DAL service or even UWS param
> description.
>
> All the complexity would follow from the begining - suppose the DL on a spectrum with given PUBID returns the existence of
> service that allows its cutout and rebinning. How I will pass the params to it ? Will the client open special window with form
> ?  What if there is another link to image of the source and the serice allows the cutout , rotation and rebinning of image or
> other conversions ....

I suggested in my previous Email a way to make DL services self-describing (I like your special popup window). As the number of 
treatments possibly run by DL is very huge (not to say infinite), we can not expect to design  a generic protocol embracing all 
of them. Furthermore, downloading attached files to process them locally is often not realistic (very large files) so, all that 
remains for us is to encapsulate the definition  of the parameters which are specific to that DL service in a container which is 
part of the DL standard. This way of doing things looks reasonable to me.

Bye
Laurent

>
> *************************************************************************
> *  Petr Skoda                         Phone : +420-323-649201, ext. 361 *
> *  Stellar Department                         +420-323-620361           *
> *  Astronomical Institute AS CR       Fax   : +420-323-620250           *
> *  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
> *  Czech Republic                                                       *
> *************************************************************************

-- 

---- Laurent MICHEL              Tel  (33 0) 3 68 85 24 37
      Observatoire de Strasbourg  Fax  (33 0) 3 68 85 24 32
      11 Rue de l'Universite      Mail laurent.michel at astro.unistra.fr
      67000 Strasbourg (France)   Web  http://astro.u-strasbg.fr/~michel
---
-------------- next part --------------
A non-text attachment was scrubbed...
Name: laurent_michel.vcf
Type: text/x-vcard
Size: 391 bytes
Desc: not available
URL: <http://www.ivoa.net/pipermail/dal/attachments/20121005/60906011/attachment.vcf>