Obscore dataproduct_type question
Laurent MICHEL
laurent.michel at astro.unistra.fr
Wed May 11 16:02:55 CEST 2016
Hello Dough,
Le 09/05/2016 01:19, Douglas Tody a écrit :
> These are good points. Note, dataproduct_type is coarse grain, e.g.,
> at this level we say just "image" not what type of image. "Catalog"
> as I suggested earlier is also too fine grained. So if we add support
> for table data the value for dataproduct_type should be "table".
To me, "table" is too coarse. Event lists or time series can be seen as
tables either. The word we are looking for is related to tables
containing sky objects related to one observation. According to this i
continue to support the "sourcelist" option.
Laurent
> While not a mandatory parameter, dataproduct_subtype would be perfect
> for specifying the data-provider defined type of the table, e.g.,
> "sourcelist" or "catalog" (this is intentionally not a controlled
> vocabulary, rather it is site or domain specific, controlled by the
> user). A UCD might be useful for specifying the table type as well,
> although this is perhaps beyond the scope of a UCD since it is not
> a quantity. - Doug
>
>
> On Wed, 20 Apr 2016, Laurent Michel wrote:
>
>> Hello,
>>
>>
>> I'm likely guided by my experience with XMM data but this is a
>> valuable use case anyway.
>> The data bulk released with one XMM observation contains various data
>> types. To make it simple, we can find there event lists, images,
>> spectra, time series and source lists possibly in various formats
>> (FITS, PDF..)
>> All these data sets can take place in an Obscore table except the
>> source list. This exception has no justification since the source list
>> is a scientific product issued from the processing of a particular
>> observation. There is even one stronger reason for pushing the source
>> lists in Obscore which is that source level products (e.g. time
>> series, spectra) are attached to individual detections which are
>> nothing else than source list rows. So hiding the source list would
>> limit the possibilities of the data discovery.
>> I guess that this point of view can easily be extended to other
>> missions of instruments.
>>
>> Moreover, for catalogues which are table compilations (e.g. XMM
>> catalogue) or large sky coverage surveys (e.g. 2MASS), most of the
>> characterisation axes parameters are not relevant. So Obscore
>> description does not bring much. The Dataset DM or the source DM are
>> better suited for his kind of datasets.
>>
>> The right position of the cursor between publishing or not a table in
>> Obscore is left to the appreciation of the data publisher.
>> But an appropriate vocabulary may help for this.
>>
>> As far as vocabulary is concerned, I would prefer to speak about a
>> 'source list' instead of a 'catalogue' which is too generic for a list
>> of sources extracted from one specific observation.
>>
>> Assuming that most of the Obscore axe values can be set from the
>> parameters of the related observation, I suggest to allow
>> dataproduct_type='sourcelist' in Obscore1.1.
>>
>>
>> Laurent
>>
>> Le 15/04/2016 19:45, Douglas Tody a écrit :
>>> I have always thought that ObsCore should include "catalog" as a
>>> dataproduct type. This was opposed in the first version due to a desire
>>> by the Exec to keep things as simple as possible, but I do not think it
>>> would have complicated things or delayed the release.
>>>
>>> A catalog is a valid data product with calib_level 3 or higher. It is
>>> not that much different than some other high level, derived data
>>> products such as a dither stack. These higher level data products are
>>> often the data products one most wants to find for analysis.
>>>
>>> While many catalogs are derived from multiple observations or even
>>> collections, it is possible for example to perform object detection on a
>>> single observation, and include the result in the set of data products
>>> published for the observation, all sharing the same obs_id. Catalogs
>>> can even have valid metadata giving the spatial, spectral, and time
>>> coverage for the catalog. So, +1 for me too. - Doug
>>>
>>>
>>> On Fri, 15 Apr 2016, Patrick Dowler wrote:
>>>
>>>> Guilty as charged :-)
>>>>
>>>> Our underlying data model (CAOM2) has catalog as a valid type and I
>>>> recall some of us trying to get catalog into the vocabulary back
>>>> before 1.0.... I can easily fix this because ObsCore is a view on
>>>> CAOM2, with the cost being that people can't find all the products via
>>>> ObsCore. But for ASKAP, if they implement ObsCore directly then they
>>>> would be left with no way to provide discovery of catalog products.
>>>>
>>>> The DM rationale is that ObsCore is a list of products (it is a
>>>> flattened view of Observation+Product where there is a 1..* relation).
>>>> As such, there are certainly other kinds of things that can be created
>>>> from data (besides just more data). Do such things belong in ObsCore?
>>>> I obviously think they do so IMO we should add to the vocabulary and
>>>> I've either been meaning to request it or forgot about that little
>>>> non-compliance.
>>>>
>>>> The alternative that I see if that access to the catalog would be a
>>>> link in a DataLink response with semantics="derivation"... and then
>>>> probably augment the DataLink vocabulary to be able to ay catalog...
>>>> and then still not be able to discover them via a data discovery
>>>> query. DataLink *is* awesome and all, but that doesn't seem so great.
>>>>
>>>> So: can we add "catalog" to the ObsCore-1.1 vocabulary for
>>>> datapoduct_type? +1 from me
>>>>
>>>>
>>>>
>>>> On 15 April 2016 at 05:21, Mark Taylor <M.B.Taylor at bristol.ac.uk>
>>>> wrote:
>>>>> James,
>>>>>
>>>>> it looks like you're not the only one to hit this. If I use the
>>>>> ObsTAP service at http://www.cadc.hia.nrc.gc.ca/tap to run:
>>>>>
>>>>> select distinct top 100
>>>>> dataproduct_type, obs_collection
>>>>> from ivoa.obscore
>>>>> where dataproduct_type not in
>>>>> ('image', 'cube', 'spectrum', 'sed',
>>>>> 'timeseries', 'visibility', 'event')
>>>>>
>>>>> I get
>>>>>
>>>>> +------------------+----------------+
>>>>> | dataproduct_type | obs_collection |
>>>>> +------------------+----------------+
>>>>> | catalog | APASS |
>>>>> | catalog | CFHTTERAPIX |
>>>>> | catalog | JCMT |
>>>>> +------------------+----------------+
>>>>>
>>>>> (note this is one of the tests run by taplint, currently failing at
>>>>> CADC).
>>>>>
>>>>> Mark
>>>>>
>>>>> On Fri, 15 Apr 2016, James.Dempsey at csiro.au wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In ObsCore v1.0 and 1.1 the dataproduct_type field has a defined
>>>>>> list of values. We currently advertise our ASKAP catalogue
>>>>>> data products in our ObsCore table, but have to leave the
>>>>>> dataproduct_type field blank for these records as none of the
>>>>>> possible types match.
>>>>>>
>>>>>> Has the inclusion of a catalogue type been considered before? If
>>>>>> not, would it be possible to include in ObsCore v1.1?
>>>>>>
>>>>>> Cheers,
>>>>>> James Dempsey
>>>>>>
>>>>>
>>>>> --
>>>>> Mark Taylor Astronomical Programmer Physics, Bristol
>>>>> University, UK
>>>>> m.b.taylor at bris.ac.uk +44-117-9288776
>>>>> http://www.star.bris.ac.uk/~mbt/
>>>>
>>>>
>>>>
>>>> --
>>>> Patrick Dowler
>>>> Canadian Astronomy Data Centre
>>>> Victoria, BC, Canada
>>>>
>>
>> --
>> jesuischarlie/Tunis/Paris/Bruxelles
>>
>> Laurent Michel
>> SSC XMM-Newton
>> Tél : +33 (0)3 68 85 24 37
>> Fax : +33 (0)3 )3 68 85 24 32
>> laurent.michel at astro.unistra.fr <mailto:laurent.michel at astro.unistra.fr>
>> Université de Strasbourg <http://www.unistra.fr>
>> Observatoire Astronomique
>> 11 Rue de l'Université
>> F - 67200 Strasbourg
>> http://amwdb.u-strasbg.fr/HighEnergy/spip.php?rubrique34
>>
--
---- Laurent MICHEL Tel (33 0) 3 68 85 24 37
Observatoire de Strasbourg Fax (33 0) 3 68 85 24 32
11 Rue de l'Universite Mail laurent.michel at astro.unistra.fr
67000 Strasbourg (France) Web http://astro.u-strasbg.fr/~michel
---
More information about the dm
mailing list