Obscore dataproduct_type question

Wed Apr 20 15:03:50 CEST 2016

Hello,

I'm likely guided by my experience with XMM data but this is a valuable use case anyway.
The data bulk released with one XMM observation contains various data types. To make it simple, we can find there event lists, 
images, spectra, time series and source lists possibly in various formats (FITS, PDF..)
All these data sets can take place in an Obscore table except the source list. This exception has no justification since the 
source list is a scientific product issued from the processing of a particular observation. There is even one stronger reason 
for pushing the source lists in Obscore which is that source level products (e.g. time series, spectra) are attached to 
individual detections which are nothing else than source list rows. So hiding the source list would limit the possibilities of 
the data discovery.
I guess that this point of view can easily be extended to other missions of instruments.

Moreover, for catalogues which are table compilations (e.g. XMM catalogue) or large sky coverage surveys (e.g. 2MASS), most of 
the characterisation axes parameters are not relevant. So Obscore description does not bring much. The Dataset DM or the source 
DM are better suited for his kind of datasets.

The right position of the cursor between publishing or not a table in Obscore is left to the appreciation of the data publisher.
But an appropriate vocabulary may help for this.

As far as vocabulary is concerned, I would prefer to speak about a 'source list' instead of a 'catalogue' which is too generic 
for a list of sources extracted from one specific observation.

Assuming that most of the Obscore axe values can be set from the parameters of the related observation, I suggest to allow 
dataproduct_type='sourcelist' in Obscore1.1.

Laurent

Le 15/04/2016 19:45, Douglas Tody a écrit :
> I have always thought that ObsCore should include "catalog" as a
> dataproduct type.  This was opposed in the first version due to a desire
> by the Exec to keep things as simple as possible, but I do not think it
> would have complicated things or delayed the release.
>
> A catalog is a valid data product with calib_level 3 or higher.  It is
> not that much different than some other high level, derived data
> products such as a dither stack.  These higher level data products are
> often the data products one most wants to find for analysis.
>
> While many catalogs are derived from multiple observations or even
> collections, it is possible for example to perform object detection on a
> single observation, and include the result in the set of data products
> published for the observation, all sharing the same obs_id.  Catalogs
> can even have valid metadata giving the spatial, spectral, and time
> coverage for the catalog.  So, +1 for me too.    - Doug
>
>
> On Fri, 15 Apr 2016, Patrick Dowler wrote:
>
>> Guilty as charged :-)
>>
>> Our underlying data model (CAOM2) has catalog as a valid type and I
>> recall some of us trying to get catalog into the vocabulary back
>> before 1.0.... I can easily fix this because ObsCore is a view on
>> CAOM2, with the cost being that people can't find all the products via
>> ObsCore. But for ASKAP, if they implement ObsCore directly then they
>> would be left with no way to provide discovery of catalog products.
>>
>> The DM rationale is that ObsCore is a list of products (it is a
>> flattened view of Observation+Product where there is a 1..* relation).
>> As such, there are certainly other kinds of things that can be created
>> from data (besides just more data). Do such things belong in ObsCore?
>> I obviously think they do so IMO we should add to the vocabulary and
>> I've either been meaning to request it or forgot about that little
>> non-compliance.
>>
>> The alternative that I see if that access to the catalog would be a
>> link in a DataLink response with semantics="derivation"... and then
>> probably augment the DataLink vocabulary to be able to ay catalog...
>> and then still not be able to discover them via a data discovery
>> query. DataLink *is* awesome and all, but that doesn't seem so great.
>>
>> So: can we add "catalog" to the ObsCore-1.1 vocabulary for
>> datapoduct_type? +1 from me
>>
>>
>>
>> On 15 April 2016 at 05:21, Mark Taylor <M.B.Taylor at bristol.ac.uk> wrote:
>>> James,
>>>
>>> it looks like you're not the only one to hit this.  If I use the
>>> ObsTAP service at http://www.cadc.hia.nrc.gc.ca/tap to run:
>>>
>>>    select distinct top 100
>>>           dataproduct_type, obs_collection
>>>    from ivoa.obscore
>>>    where dataproduct_type not in
>>>        ('image', 'cube', 'spectrum', 'sed',
>>>         'timeseries', 'visibility', 'event')
>>>
>>> I get
>>>
>>>    +------------------+----------------+
>>>    | dataproduct_type | obs_collection |
>>>    +------------------+----------------+
>>>    | catalog          | APASS          |
>>>    | catalog          | CFHTTERAPIX    |
>>>    | catalog          | JCMT           |
>>>    +------------------+----------------+
>>>
>>> (note this is one of the tests run by taplint, currently failing at CADC).
>>>
>>> Mark
>>>
>>> On Fri, 15 Apr 2016, James.Dempsey at csiro.au wrote:
>>>
>>>> Hi,
>>>>
>>>> In ObsCore v1.0 and 1.1 the dataproduct_type field has a defined list of values. We currently advertise our ASKAP catalogue
>>>> data products in our ObsCore table, but have to leave the dataproduct_type field blank for these records as none of the
>>>> possible types match.
>>>>
>>>> Has the inclusion of a catalogue type been considered before? If not, would it be possible to include in ObsCore v1.1?
>>>>
>>>> Cheers,
>>>> James Dempsey
>>>>
>>>
>>> --
>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>>> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>>
>>
>>
>> --
>> Patrick Dowler
>> Canadian Astronomy Data Centre
>> Victoria, BC, Canada
>>

-- 
jesuischarlie/Tunis/Paris/Bruxelles

Laurent Michel
SSC XMM-Newton
Tél : +33 (0)3 68 85 24 37
Fax : +33 (0)3 )3 68 85 24 32
laurent.michel at astro.unistra.fr <mailto:laurent.michel at astro.unistra.fr>
Université de Strasbourg <http://www.unistra.fr>
Observatoire Astronomique
11 Rue de l'Université
F - 67200 Strasbourg
http://amwdb.u-strasbg.fr/HighEnergy/spip.php?rubrique34