[Heig] Post running meeting thoughts

[Dr. Ian N. Evans]

Dear Francois,

I consider the arf, rmf, and psf to be response-function data products.  Background images and pixel masks are not response-function data products - they are determined directly from the observation event list similarly to a total counts image.  Bad pixel is a region data product, but it’s something of a gray area since it’s a combination of known bad pixel regions plus bad pixel regions derived directly from the observation event list.

For the Chandra Source Catalog (CSC) prototype, at least initially we plan to expose all of the data products directly to demonstrate that the extension provides the flexibility that we need.  However in production, we likely would not expose all of the data products individually but rather combine some of them with the event lists as event bundles (at least for the individual observation full-field data product set).  We would want to expose the individual observation event lists individually, but might choose for example to construct an event bundle that exposes (at least) the event list, bad pixel regions, aspect histogram, and possible aspect solution as a bundle since there is very little use for the latter 3 types of data product without the event list.

While tying associated and derived data products to an event list in an event bundle seems sensible for individual observations, our experience is that this isn’t appropriate for the CSC advanced data products.  Since CSC 2.0 was released we have had millions of catalog data product downloads and surveyed our user base as to data product usage.

The typical usage patterns for the CSC advanced data products are different from the typical usage patterns for individual X-ray observation data.

For the latter the user typically downloads the event list and ancillary data products (such as responses or other data products that can be used to build responses) as a set, and then performs data analysis steps directly on the event list using the ancillary data products, often after applying spatial/spectral/temporal filters to the data.  Event bundles facilitate this usage.

For the CSC advanced data products the usage patterns are quite different.  Many (most) of these advanced data products are derived from multiple (in some cases hundreds) observations.  Typically the users aren’t interested in performing data analysis steps on the event lists themselves, and often aren’t interested in knowing which observation(s) they are derived from (at least not from the perspective of having to perform a data query).  They just want (e.g.) all the spectra (or light curves, or photometry MPDFs, or ...) in a certain region of the sky, or in a given time range, etc.  And given the data volume that’s all they want.  Maybe they’ll come back later and ask for a subset of additional data products after they’ve performed some preliminary analyses on those data products, but they don’t want those up front.

Based on these usage patterns, I think we will likely want to expose the remaining CSC data products individually.

Thanks,
—Ian

> On Jan 27, 2026, at 09:52, BONNAREL FRANCOIS gmail via heig <heig at ivoa.net> wrote:
> 
> Dear all,
> 
> After the meeting last week, I was still thinking about what the Chandra prototype could look like
> 
> For the Paris HESS prototype, I get the idea since a couple of years now.
> 
> Trying to understand what the CSC data products could be I came back to Ian's Malta interop presentation.
> 
> I copy/paste here one of the slides where some of these products are described.
> 
> Before trying to define dataproduct_type vocabulary terms for those products I am wondering if we really need to expose all this data directly in
> 
> an ObsTAP service.
> 
> For example background images, psf, pixel mask, bad pixel regions, ARF belong to the "response functions" category if I'm not mistaking.
> 
> They probably are attached to a photon event list or an image or ....
> 
> Including all this in the main ObsCore table will overload it very heterogeneously. Some of these response functions will be similar to what we get in other domains (psf) some will be very different and specific to Xray.
> 
> I understood that the spatial, spectral, time characterization of these specific products could be borrowed from the observation they are associated with. It's ok but is that useful ?
> 
> For accessing these response functions I can imagine 4 solutions which all will have the  advantage to let the OBsTAp service be focused on measurements obtained from the sky at whatever calib level.
> 
>     1 ) the photon event list and response functions are gathered together in the same tar or archive file (or MEF) which is typed as an event-bundle. Direct access to this bundle from Obstap access_url is then easy. It's the client task to figure out what to do with the content of the bundle.
> 
>    2 ) the various response material is kept as a set of individual products. All are associated to an event list or an image or a spectrum. In that case ObsTAP point to a datalink response which lists all these different products. The semantics FIELD writes calibration or response function.  Content_qalifier FIELD writes the very nature of the product.
> 
>    3 ) the DataLink reponse content may be organized as a TAP table. It's then possible to query at the same time the  ObsTAP table and the DataLink-like table by a join on ObsCore/obs_publisher_did-DataLink/ID
> 
>   4 ) if we need a more detailed description of the response products to help discover and select them we could imagine creating a specific "response product" table following a specific datamodel as proposed by Mireille in her Gorlitz presentation. This will allow to attach specific eg :
> 
>       - time range to a psf or
> 
>       - specific release date and description to an arf or a bad pixel map
> 
>       -....
> 
>       Natural join on obs_publisher_did in both tables will allow to query those table at the same time with selection criteria from both.
> 
>    Cheers
> 
> François
> 
>  
> 
> 
> 
> 
> 
> <ulnvpjC4zXtPT0zn.png>
> 
> -- 
> heig mailing list
> heig at ivoa.net
> https://www.google.com/url?q=http://mail.ivoa.net/mailman/listinfo/heig&source=gmail-imap&ust=1770130364000000&usg=AOvVaw1NuTRL3a6Ib8NM2g8f0TM8

—

Dr. Ian Evans
Astrophysicist
Chandra X-ray Center
Center for Astrophysics | Harvard & Smithsonian

Office: (617) 496 7846 | Cell: (617) 699 5152
60 Garden Street | MS 81 | Cambridge, MA 02138



 


 <http://cfa.harvard.edu/>cfa.harvard.edu <http://cfa.harvard.edu/> | Facebook <http://cfa.harvard.edu/facebook> | Twitter <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube> | Newsletter <http://cfa.harvard.edu/newsletter>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260204/14a493fb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-2.png
Type: image/png
Size: 581 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260204/14a493fb/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 21717 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260204/14a493fb/attachment-0003.png>