[HEIG -Obscore extension ] Distributing observation data sets together with instrumental response functions
Dr. Ian N. Evans
ievans at cfa.harvard.edu
Fri May 23 19:42:53 CEST 2025
Hi Mireille,
> On May 23, 2025, at 08:46, Mireille Louys via dm <dm at ivoa.net> wrote:
>
> Hi HEIG ,
>
> Since a while we have been discussing about the way Obscore can be used for the distribution and discovery of high energy data sets
>
> in the context of multi-wavelength data discovery in the VO.
>
> Obscore describes how data products are sampled along the main physical axes involved in astronomical data :
>
> spatial, spectral, temporal, polarimetric.
>
> Coverage, resolution and sampling properties are mandatory in the Obscore table to represent a dataset.
> this means we need to have s_ra, s_dec, em_min, em_max , t_min, t_max, s_region, s_resolution, etc. filled with representative values .
>
> I think this rule does not fully apply for describing instrument response function as data sets .
>
There is already a similar case described in the ObsCore document. For dataproduct_type = “measurements” the document states "A few mandatory keywords for the axes description may be non-applicable for such a data product. In this case the coverage on spatial, energy, time, and polarization may inherit the values from the ObsCore description of its progenitor.”
Practically, however, response-functions (as well as advanced data products) will often have their own coverage, resolution, and sampling properties in terms of spatial, spectral, temporal, and polarimetric physical axes on the sky that they apply to, even if the products themselves do not include all (or in some cases any) of those axes. I sent an example ObsCore entry for a PSF response-function a couple of days ago that demonstrates this.
> The response functions belong to the instrumental/calibration information , and probably are stored in Calibration DB .
> I think this is not homogeneous in content with the observation datasets representing flux variation on the sky,
> so they should not be considered as Obscore data product type.
>
I don't agree with this suggestion. If only datasets “representing flux variation on the sky” should be included in ObsCore, then that also means that many types of “advanced data products” would also not qualify for inclusion in ObsCore (even if they refer to a constrained spatial/spectral/temporal location such as the region surrounding a detected source).
I think the IVOA works because it tries to be encompassing. But from the perspective of a science end-user, why would I want to use IVOA interfaces for data discovery if I know I have to use another interface to ensure I can find all of the data products in the high-energy waveband? Conversely, from the perspective of a HEA data provider, why would I want to put the effort into supporting an IVOA interface if it can’t serve all of my data products (knowing that I’m going to have to provide a second interface that can)?
In principle, one could propose an alternative IVOA-supported data discovery interface that doesn’t have the same restrictions that you are suggesting for ObsCore. But practically, we know how long it takes for anything to become a reality in the IVOA, and so this approach doesn’t seem terribly realistic. But why would any end-user or data provider want to use two interfaces to circumvent what seems to be an artificial constraint?
> Currently from our HEIG discussions , we have identified two ways to distribute data sets in Obscore :
>
> Include the data together with their instrument response details into a tarball.
> this can be represented by a data product type like eventlist-bundle, spectrum-bundle, etc . depending of the axes covered in the data
> Distribute the data set as a datalink entry which contains a list of links to the dataset , e.g an event-list and to the various instrumental response files, arf rmf, psf, energy dispersion , etc.
> accessible through the datalink table .
> for instance , reusing examples given by Laurent Michel
> spectrum bundle
>
> a) If this spectrum bundle is exposed as an ObsCore entry in a data link, we should have something like this:
>
> Obscore row:
> -----------
> - dataproduct_type=spectrum-bundle (tbd)
> - access_format=application/x-votable+xml;content=datalink
>
> Datalink response:
> -----------------
> - link #1
> - semantics=#this
> - content_qualifier=spectrum-bundle (TBD)
> - content_type=application/tar+gzip
> - description="spectrum file + preview + ARF + RMF + Background spectrum"
>
>
> b) If the spectrum is exposed in a separate row:
>
> Obscore row:
> -----------
> - product_type=spectrum
> - access_format=application/x-votable+xml;content=datalink
>
> Datalink response:
> -----------------
> - link #1
> - semantics=#this
> - content_qualifier=spectrum
> - content_type=application/fits
> - description="spectrum file"
> - link #2
> - semantics=#package
> - content_qualifier=spectrum-bundle (TBD)
> - content_type=application/tar+gzip
> - description="spectrum file + preview + ARF + RMF + Background spectrum"
>
> if we define the terms hierarchy as a response vocabulary like ivoa.net/rdf/instrument-response
> Instrument_response
> arf
> rmf
> energy-disp
> background
> etc.
>
> from Bruno's example we can have :
> Obscore row:
> -----------
> - product_type=spectrum-bundle
> - access_format=application/x-votable+xml;content=datalink
>
> Datalink response:
> -----------------
> - link #1
> - semantics=#this
> - content_qualifier=event-list
> - content_type=application/fits
> - description="event file"
> - link #2
> - semantics=#arf
> - content_qualifier=#arf
> - content_type=application/x-fits-ogip (to be added in table 1 of ObsCore)
> - description="ARF "
> - link #3
> - semantics=#rmf
> - content_qualifier=#rmf
> - content_type=application/x-fits-ogip (to be added in table 1 of ObsCore)
> - description="RMF"
> - link #4
> - semantics=#background
> - content_qualifier=image ( if it is a map)
> - content_type=application/x-fits-ogip (to be added in table 1 of ObsCore)
> - description="Background"
>
> doing so , we can discover a data set based on the Obscore + heig extension columns ,
> and in a second step access the response files stored in the calibration part of an archive through the links .
>
While these approaches are certainly useful in cases where the end user is interested in retrieving these products together with the event-list (and in many cases the user may want to do just that), that restriction is also a limitation. For example, our experience with the Chandra Source Catalog (that includes ~90 million data products) is that scientists who are doing advanced analysis of large samples (i.e., they are not interested in just one or two observations) typically want to discover and retrieve the data products independently from the associated event-lists.
> Comments, examples, ideas?
>
> Mireille
>
> --
> --
> Mireille Louys, MCF (Assistant Professor)
> Centre de données Astronomiques (CDS) Equipe Images, ICube
> Observatoire de Strasbourg Telecom Physique Strasbourg
> 11, rue de l' Université 300, Bd Sebastien Brandt CS 10413
> F-67000 Strasbourg F-67412 Illkirch Cedex
Thanks,
—Ian
—
Dr. Ian Evans
Astrophysicist
Chandra X-ray Center
Center for Astrophysics | Harvard & Smithsonian
Office: (617) 496 7846 | Cell: (617) 699 5152
60 Garden Street | MS 81 | Cambridge, MA 02138


<http://cfa.harvard.edu/>cfa.harvard.edu <http://cfa.harvard.edu/> | Facebook <http://cfa.harvard.edu/facebook> | Twitter <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube> | Newsletter <http://cfa.harvard.edu/newsletter>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20250523/fb63670e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-2.png
Type: image/png
Size: 581 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20250523/fb63670e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 21717 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20250523/fb63670e/attachment-0003.png>
More information about the dm
mailing list