[obs-tap]:updates on the Proposed recommendation
Arnold Rots
arots at head.cfa.harvard.edu
Mon Aug 1 14:20:02 PDT 2011
Ah, so you *don't* want to separate Data Discovery and Data Link?
Only when it becomes too complicated.
The problem is that "too complicated" depends on the context and that
is dangerous.
My assertion is that the level of Data Linking in the PR has already
made things too complicated in some cases, and that therefore (but
also as a matter of good standards design principles) the right thing
to do is to separate them completely.
Cheers,
- Arnold
François Bonnarel wrote:
[ Charset ISO-8859-1 unsupported, converting... ]
> Well Arnold, I think
> that for many full retrieval or simple access the access stuff in ObsTap
> is sufficient for many people and is usefull immediatly...
> DataLink allows some more fine tuning and for complex data
> it is an optimization for coupling global discovery of complex data
> set/ clever access to parts of it
> using ObstAp alone works but will be less elegant...
> Best regards
> Fran?ois
> Le 01/08/2011 15:53, Arnold Rots a ?crit :
> > Francois,
> >
> > Nothing is going on underground.
> > I have shared our experiences in implementing ObsTAP with some local
> > members of the TCG. I made it clear that the PR can be implemented,
> > but that there are problems.
> >
> > But if I understand your argument below, we are in full agreement.
> > You want to separate Data Link from Data Discovery and that is
> > precisely what I was arguing; my complaint was that there are Data
> > Link elements in the ObsTAP Data Discovery protocol that are causing a
> > problem.
> > Specifically: the access_* elements belong in Data Link, not in Data
> > Discovery, and with them removed the data types available can be
> > enumerated in a single record.
> > So, the example I gave (the responses to a Data Discovery query and a
> > Data Linking query) are in full agreement with what you are
> > advocating, as far as I can tell.
> >
> > Is there still an issue, then?
> >
> > Cheers,
> >
> > - Arnold
> >
> >
> > Francois Bonnarel wrote:
> >> Hi Arnold, all dm people,
> >>
> >> Let me go back to this, because apparently, this discussion is going on
> >> underground
> >>
> >> First come back to the very beginning of the ObsTap effort...
> >> It was a strong commitment from the comitee to build something fast
> >> reusing tAP protocol and observation/charac data model for
> >> data discovery covering most of the needs...
> >> From the very beginning also, it was obvious that Data links
> >> and virtual access data could not and will not be covered by Obstap
> >> The DataLink method or service concept has been around in various DAL notes
> >> since years now. As far as I am concerned I made presentations in the
> >> last three
> >> Interop meetings (Victoria, Nara and Napoli, see eg the latter:
> >> http://www.ivoa.net/internal/IVOA/DAL-InteropMay2011/DataLink.pdf )
> >>
> >> This concept is there, because you cannot imagine providing both Data
> >> Discovery
> >> and complex linkage features (or linkage for complex data structure) in
> >> one step
> >> and a SINGLE table, (single table required by the TAP-ADQL protocol as
> >> all may remember)
> >> So ObsTap is there for DataDiscovery... the only thing you can imagine
> >> to provide access to the
> >> various Data sets in an observation is to duplicate the observation raws
> >> until you reach full
> >> discovery of all observation-related products as was allready
> >> explained... This is verbose
> >> and works . So now how can DataLink work in the future ? see below on
> >> your use case ...
> >> Data Link is now in the roodmap of the DAL working group and an IVOA
> >> note is in preparation as a
> >> very first drafting effort of this new "protocol".... The note will be
> >> available within 3 weeks or so..
> >>
> >> Arnold Rots a e'crit :
> >>> This is becoming unwieldy.
> >>> Trying to make X-ray data (and I suspect the same is true for aperture
> >>> synthesis data) fit into something that is designed with optical
> >>> images in mind is reminiscent of round pegs and square holes.
> >>>
> >>> Service providers are free to define subtypes and titles, but you are
> >>> saying that if they don't follow rules that are not spelled out,
> >>> things won't work as envisaged.
> >>> Also, if I understand the argument correctly, if data discovery
> >>> software is to be helpful at all, it needs to be able to extract some
> >>> information from the title field - but that is intended for human
> >>> consumption.
> >>>
> >>> If I see this, it looks like I need to generate at least eight records
> >>> for a single observation, some containing a mix of levels, and all
> >>> duplicating pretty much the same metadata.
> >>>
> >>> This is not going to make it attractive to provide ObsTAP services.
> >>>
> >>>
> >>> Maybe I should do what you did and provide an example of how I thought
> >>> it should have worked.
> >>>
> >>> Here is how I would envisage data discovery of Chandra data to work:
> >>> A single record per Obsid that provides the observational metadata and:
> >>> ObsId
> >>> 12345
> >>> Dataset Identifier
> >>> ivo://ADS/Sa.CXO#obs/12345
> >>> Data Types available
> >>> Package
> >>> Event list
> >>> Image
> >>> Calibration level
> >>> 2
> >>> Title
> >>> Chandra/ACIS ObsId 12345
> >>>
> >>>
> >> DataLink is a method or a service allowing to retrieve a table
> >> describing links between observations
> >> identified by their obsid and any kind of data retrieval ... Obsid known
> >> from an ObsTap discovery
> >> phase can be directly used for interrogating such a service of course..
> >> (and by the way in the case the Obstap service is a TAP-PQL service the
> >> DataLink table could be attached with the main obstap table in the same
> >> query response because the single table requirement is no more there in
> >> that case)
> >> But it is a qualified link which means that the semantic or type of the
> >> link is given in one field
> >> of the table, while the nature of the access is given in another field :
> >> this can tell us if it is a simple
> >> retrieval , an SIA Query service ans SSA AccesData method, etc ...
> >> So in your use case we will get three different links for the same
> >> Observation (obsid) .. the types
> >> (or semantic) will be Package, event list and image and the Access
> >> nature could be respectivly : retrieval
> >> retrieval and SIA query (for example)
> >> In addition the "Access" package (group of access fields in the table)
> >> is proposed to be extended
> >> beyond the traditional "reference" and "format" to describe which part
> >> of a complex "file" is to be retrieved
> >> ( path in a directory/tar file, extension in MEF file, table name in a
> >> VOTABLE, etc ...) .. A proposal
> >> for such an extended access package is described in the
> >> chaaracterisation 2 draft at the moment...
> >>
> >> Best regards
> >> Franc,ois
> >>> Then a data access protocol that allows querying the archive using any
> >>> of the above in a where clause, with either ObsId or DID required, and
> >>> returning:
> >>> ObsId DataType Contents Level Format URL
> >>> -----------------------------------------------------------
> >>> 12345 Pkg_1 evt,img 2 tar http://...
> >>> 12345 Pkg_2 evt,img 1 tar http://...
> >>> 12345 Pkg_12 evt,img 2,1 tar http://...
> >>> 12345 evt evt 2 fits-bin http://...
> >>> 12345 evt evt 1 fits-bin http://...
> >>> 12345 img img 2 fits http://...
> >>> 12345 img img 2 jpg http://...
> >>> 12345 img img 2 fits http://...
> >>> 12345 img img 2 jpg http://...
> >>> This is an example where the client specified ObsId or DID, but no
> >>> data type or format.
> >>>
> >>> Never mind the terms and abbreviations I used - you get the picture.
> >>>
> >>> Cheers,
> >>>
> >>> - Arnold
> >>>
> >>>
> >>> Douglas Tody wrote:
> >>>
> >>>> More precisely what you might have is something like (display in a wide view):
> >>>>
> >>>> ObsId Type Subtype Level Format Title
> >>>> ----------------------------------------------------------------------------------------------------------
> >>>> 123 event chandra.hrc.pkg 1 application/x-tar-gzip Chandra ACS-XYZ observation package (event,refimage)
> >>>> 123 image chandra.hrc.refimage 2 image/fits Chandra ACS-XYZ reference image
> >>>> 123 image chandra.hrc.preview 2 image/jpeg Chandra ACS-XYZ preview image
> >>>> 345 event rosat.foo.pkg 1 application/x-tar-gzip ROSAT whatever observation package (xxx)
> >>>>
> >>>> and so forth. The subtype could in principle be more generic but will
> >>>> likely be instrument-specific for a level 1 observation.
> >>>>
> >>>> The Title should concisely describe the data product, e.g., origin,
> >>>> instrument, ID, what it is (observation package, calibration, standard
> >>>> view, etc.). The title string is what one normally wants to output on a
> >>>> displayed image or plot to identify to a human the data being shown.
> >>>> You can put whatever you want in there to describe the data product so
> >>>> long as it is concise (one line of text).
> >>>>
> >>>> - Doug
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, 11 Jul 2011, Douglas Tody wrote:
> >>>>
> >>>>
> >>>>> On Thu, 7 Jul 2011, Arnold Rots wrote:
> >>>>>
> >>>>>
> >>>>>> Aside from what I reported in a previous message, quoted below, there
> >>>>>> are more discrepancies between Table 5 and Tables 6 and 7:
> >>>>>>
> >>>>>> obs_creator_did is missing from Table 7
> >>>>>> o_units in Table 5 should be o_unit
> >>>>>> pol_states is missing from Table 6
> >>>>>> facility_name and instrument_name are spelled differently;
> >>>>>> even though required, they show up in Table 7, rather than 6
> >>>>>> em_unit is missing from Table 5
> >>>>>> o_stat_error is missing from Table 7
> >>>>>>
> >>>>>> Also, note the comment I made on MJD in use case 1.6
> >>>>>> and on the uselessness of bib_reference because of its murky
> >>>>>> definition
> >>>>>>
> >>>>>> I still lament the fact that the data access functionality is
> >>>>>> compromising the self-consistency and usefulness of the data discovery
> >>>>>> function, but decided for our tarred packages to use:
> >>>>>> dataproduct_type = NULL
> >>>>>> dataproduct_subtype = package:event,image
> >>>>>> access_format = application/x-tar
> >>>>>> As far as I can tell, this is within the specifications.
> >>>>>>
> >>>>> Well we don't specify what the subtypes you provide for your archive
> >>>>> should be so I suppose you could get away with this, but this example is
> >>>>> not at all what we had in mind. The subtype should be the science type
> >>>>> of the specific data product, *not* details about the content of the
> >>>>> data product. I would expect the type to be "event" (meaning "event
> >>>>> data" not "event list") and the subtype to be something more like
> >>>>> "chandra.hrc.package", "chandra.hrc.refimage (or "rosat.XX" etc.).
> >>>>>
> >>>>> Note subtypes are supposed to be fixed strings so that one can search
> >>>>> the local archive for a particular type of data product; if you try to
> >>>>> describe what is included in a particular data product then such
> >>>>> selection won't be possible. So for example a client will do a generic
> >>>>> query to see what subtypes Chandra defines, and then they can pose a
> >>>>> more specific query to get a certain type of Chandra-specific data
> >>>>> product. Likewise for ALMA etc.
> >>>>>
> >>>>> Note you also have obs.title where you can provide a short description
> >>>>> of the data product and for this you can provide whatever you want.
> >>>>>
> >>>>> - Doug
> >>>>>
> >>>>>
> >>> --------------------------------------------------------------------------
> >>> Arnold H. Rots Chandra X-ray Science Center
> >>> Smithsonian Astrophysical Observatory tel: +1 617 496 7701
> >>> 60 Garden Street, MS 67 fax: +1 617 495 7356
> >>> Cambridge, MA 02138 arots at head.cfa.harvard.edu
> >>> USA http://hea-www.harvard.edu/~arots/
> >>> --------------------------------------------------------------------------
> >>>
> >>>
> >>
> >> --
> >> =====================================================================
> >> Franc,ois Bonnarel Observatoire Astronomique de Strasbourg
> >> CDS (Centre de donne'es 11, rue de l'Universite'
> >> astronomiques de Strasbourg) F--67000 Strasbourg (France)
> >>
> >> Tel: +33-(0)3 68 85 24 11 WWW: http://cdsweb.u-strasbg.fr/people/fb.html
> >> Fax: +33-(0)3 68 85 24 25 E-mail: francois.bonnarel at astro.unistra.fr
> >> ---------------------------------------------------------------------
> >>
> > --------------------------------------------------------------------------
> > Arnold H. Rots Chandra X-ray Science Center
> > Smithsonian Astrophysical Observatory tel: +1 617 496 7701
> > 60 Garden Street, MS 67 fax: +1 617 495 7356
> > Cambridge, MA 02138 arots at head.cfa.harvard.edu
> > USA http://hea-www.harvard.edu/~arots/
> > --------------------------------------------------------------------------
>
--------------------------------------------------------------------------
Arnold H. Rots Chandra X-ray Science Center
Smithsonian Astrophysical Observatory tel: +1 617 496 7701
60 Garden Street, MS 67 fax: +1 617 495 7356
Cambridge, MA 02138 arots at head.cfa.harvard.edu
USA http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------
More information about the dm
mailing list