[Heig] HEIG Obscore Note - ready for final review- part Uses-cases
Dr. Ian N. Evans
ievans at cfa.harvard.edu
Thu Aug 27 23:33:42 CEST 2026
Hi Mireille,
I provided more comments/answers in-line below. Again, let me know if you need more information.
> On Aug 24, 2026, at 13:28, Mireille Louys via heig <heig at ivoa.net> wrote:
>
> Dear HEIG ,
>
> Here is another set of remarks for the Usecase part :
> UseCases.tex :
>
> Uses-cases assumptions on vocabularies :
> We have proposed to distinguish 3 categories of datasets in this note.
> Observation datasets, typically in HE : event-lists, and response function datasets, and advanced datasets.
>
> In agreement with Semantics WG we have 3 vocabularies :
> http://www.ivoa.net/rdf/product-type <https://www.google.com/url?q=http://www.ivoa.net/rdf/product-type&source=gmail-imap&ust=1788197300000000&usg=AOvVaw2WknCLh7tuoS87FRHGlr-b> used for data products served in ivoa.obscore table
> http://www.ivoa.net/rdf/response-type <https://www.google.com/url?q=http://www.ivoa.net/rdf/response-type&source=gmail-imap&ust=1788197300000000&usg=AOvVaw2-sAy71YaWUZm-sWrBMkqz> to expose response data products
> The draft for an advanced data product vocabulary is discussed here:
> https://github.com/ivoa-std/VEPs/pull/21/changes/c4195e1975341881d8fda22cb851ff11b5b74c1f#top
> <https://www.google.com/url?q=https://github.com/ivoa-std/VEPs/pull/21/changes/c4195e1975341881d8fda22cb851ff11b5b74c1f%23top&source=gmail-imap&ust=1788197300000000&usg=AOvVaw1gLazGoIpzZXOMa9sfvfB0>
> Note that the Radio astronomical community also requires the definition of new dataproduct types for their science ready datasets as « advanced dataset ».
> See https://www.sciencedirect.com/science/article/pii/S2213133725000769 <https://www.google.com/url?q=https://www.sciencedirect.com/science/article/pii/S2213133725000769&source=gmail-imap&ust=1788197300000000&usg=AOvVaw1dZtJ_v3t4MK4QSapa6s9c>
>
> The note naturally insists to distinguish between the kinds of datasets,
> due to their role in the science interpretation process.
> But to serve all these datasets by an ObsTAP service we need to define
> an implementation strategy compatible to all HEIG projects and stable for future data archives.
>
> Currently the Uses-case Appendix A shows 3 strategies:
>
>
> Assumption 1: dataproduct_type in ivoa.obscore belongs to any of the 3 vocabularies product-type,
> advanced_data_products, response data products:
> All products are distributed with the ivoa.obscore table .
> Chandra A.1.8, A.2.1,A.2.3, A2.4 , CTAO: A1.9, SWGO: A1.3
>
Minor correction here - A2.4 is a CTAO use case example.
> Pb: If you search event-list or light-curves in a region, with response functions,
> you get all datasets together and need to sort which psf matches which light curve for instance.
>
> You can use GROUP by obs_id for instance, but still there may be several light curves involved in the region,
> so how to match light_curves and psf files properly if the psf varies in the field of view?
>
Well for Chandra, this could be done by data product naming (although that wouldn’t be recommended). All of our data products include sufficient header metadata that these connections can be made straightforwardly. In practice, we find that when people do queries on catalog data products they typically fall into one of two categories: (1) looking for information on a single catalog source, in which case the query constraints are sufficiently tight that only a single object is identified (in several of the examples in my demos at the Interop, the position constraint was something like CONTAINS(POINT(s_ra, s_dec, CIRCLE(ra, dec, 0.0005)) = 1) which would be a highly constrained search radius (in this case 1.8 arcsec); (2) looking for data products for multiple catalog sources in a region or that satisfy other search criteria - in this case one might expect a (possibly very) large number of data products to be returned. Our experience in the latter case is that people tend to write analysis scripts that do the typically repetitive work of further analyzing these data sets, in which case the scripts do the work of matching the datasets using the header metadata.
More generally I think this depends in part on how good a job the archive has done in providing metadata to make such matching easy. My experience is that archives that know they have this concern usually do a good job of making the task manageable by their end users.
One can always add an additional index column(s) to ObsCore to make these data product associations explicit if needed (as suggested in Section 6).
>
> Assumption 2: data_product_subtype in ivoa.obscore provides the type of response function
> and "response-function" is added to product_type vocabulary.
> All products are distributed with the ivoa.obscore table .
> Km3net A1.10, A1.13
> Chandra
> Pb: dataproduct_subtype is not standard in ObsCore and left open for archives’ specific needs .
> No distinctions between response and maps ( excess maps , significance maps, etc)
> also proposed in the dataproduct_subtype category.
>
We proposed several dataproduct_subtype values for dataproduct_type “response-function”, and these correspond to the standard set of response function types that we defined in Section 5. We also suggested that standardizing some dataproduct_subtypes such as “exposure-map” etc. (a subtype of “image”) would likely be a good idea since some of these concepts are standardized across HEA, although I admit we didn’t take this further in the document.
I can think of “background-map”, “exposure-map”, “instrument-map”, “probability-map”, “sensitivity-map”, “significance-map” to name a few, and I suspect that most of these would apply across multiple wavebands and not just HEA. I can think of several similarly “likely-to-be-commonly-used” dataproduct_subtypes for other dataproduct_types such as region (e.g., “contours”, “extended-source-region”, “field-of-view”) or “pha” for a pulse-height “spectrum” etc.
Even if these were to be standardized (which I encourage pursuing), I still think individual archives need the ability to define their own dataproduct_subtypes, as there are many data product types that won't be common across multiple projects. We have dataproduct_subtypes such as “source-list” and “bayesian-blocks” still under the “measurements” dataproduct_type for example (pending the definition of an alternative dataproduct_type that represents generic tabular data) and while the former might reasonably be standardized, the latter might remain a Chandra archive specific advanced data product.
> Assumption 3: separate the 3 kinds of products and implement one table for each kind
>
> ivoa.obscore uses data product_type in product-type vocabulary
> ivoa.response uses data product_type in response-type vocabulary
> ivoa.adp uses data product_type in advanced-dataproduct-type vocabulary
> The ObsCore data model shows the links between the various dataset categories.
> See Fig class-diagram-ObscoreExtensiondesign_for HE_interop2025v1 presented in Görlitz in
> EN-ObscoreExtensiondesign_for_HE_interop2025v1.pdf here <https://www.google.com/url?q=https://wiki.ivoa.net/internal/IVOA/InterOpNov2025DM/ObscoreExtensiondesignForHE-interop2025-Louys.pdf&source=gmail-imap&ust=1788197300000000&usg=AOvVaw3BlH7m8Tm291vJDdYlC1P0>
>
> Data discovery can then query each table separately, or join them to discover the links between these 3 types of products.
> Here is an example:
> "Provide me all advanced datasets, light curves and spectra for one region in the sky."
>
> Select obs_publisher_did, dataproduct_type, adp_publisher_did, adp_product_type
> FROM ivoa.obscore as c
> JOIN ivoa.obscore-adp as adp
> ON adp.obs_publisher_id = c.obs_publisher_id
> WHERE c.dataproduct_type LIKE ‘%light-curve%’
> OR c.dataproduct_type LIKE ‘%spectrum%’
> AND (position constraint)
>
I don’t understand this example. It appears that the same obs_publisher_did appears in both the obscore and adp tables, which means (because obs_published_did is unique for each data product) that the data product must have a record in each table. In which case, since dataproduct_type is mandatory, it is going to have the advanced data product dataproduct_type in ObsCore, as well as all of the other ObsCore attributes. Which is what we need. I can see that if there are additional attributes that belong only to advanced data products that the adp table could then include additional columns that are not in the ObsCore table. However, that’s not the case for the examples we showed in the Chandra demo. Maybe I'm missing something that’s described in the linked presentation, but the IVOA wiki is still down so I can’t see it.
>
> Loose criteria in queries
> Select * is used on all uses cases and may deliver a very wide table
> with empty values in many columns or redundancy between a response
> taken as an ObsCore record and the related observation.
> It is recommended in ObsTAP to keep the parameter list short like in the example above,
> and to use the result table for a finer query afterwards.
>
I don't have a problem with “SELECT *” in the use cases because for most of the use cases the returned properties aren’t all that relevant for the use case, which mostly focuses on the WHERE clause. I do agree that “SELECT *” may return a very wide table and that may not be desirable in practice. I do note that we are following the lead of the draft radio extension here in using “SELECT *”.
>
> Relationship between response functions and the ivoa.obscore-hea table
>
> Many of our use cases show systematic JOIN between ivoa.obscore and ivoa.obscore-hea
> tables when searching for response function.
> This may lead to ambiguities as for instance, I suppose ev_xel is a property for an event-list but not for a psf or edisp response function.
> Is this right ?
> Any example of the interest of joining response and ivoa.obscore-hea, to provide another use-case?
>
I think the JOIN with ivoa.obscore_hea is largely needed in the use cases so that the queries can be performed in energy units, or to query based on messenger, xxx_mode, or t_intervals. There are some use cases that don’t need the JOIN at all for the query (but may want the JOIN so that they get some of the properties recorded in the ivoa.obscore_hea table returned).
ev_xel probably only makes sense for event lists (it could conceivably be useful information in a few cases for data products extracted from an event-list, for example an image that is constructed from all of the events in the event-list, but this seems to be a stretch to me) so I think the way it is defined “the number of events in an hea-event-list (effectively, the length of the “events” axis in the hea-event-lists’s table)” is correct and that means that ev_xel is meaningful only for an event list (in the same way that pol_xel is really only meaningful for a polarized dataset).
I think at this point there is an example of joining response and obscore tables in Appendix B, and given that further thought is needed here I don't think there should be a use case in Appendix A (yet). However, the example in Appendix B does not JOIN ivoa.obscore_hea, so this could be added to the existing example. (Since ivoa.obscore_hea is strictly an extension, I don’t think there would be a meaningful example with a JOIN that includes ivoa.obscore_hea but does not include ivoa.obscore.)
Where the response table idea needs additional thought is that most response functions really come from the detection-chain, i.e., the telescope and the instrument/detector, and for ground-based facilities the atmosphere. As such, they typically don’t depend on sky coordinates such as RA, Dec, but do depend on telescope and instrument information such as off-axis and azimuthal angle, instrument mode etc., telescope altitude and azimuth for ground-based facilities, atmospheric conditions during the observation, and the spectrum, spatial distribution, and time-variability of the source being observed. For some facilities, it may be possible to generate response functions and make a set available to the end user, either that (in a very few cases) may be independent of the individual observations, or that depend on the individual observations but differ based on the source model or the analysis mode etc. IMHO, the latter would be an ideal example for a separate response table. More often there will be response functions that are specific to an individual observation and so can be directly DataLinked to the observation data (and so can be linked to RA, Dec through the observation). Historically (starting with the X-ray) the responses have been separable, i.e., the psf, redistribution matrix, and effective area can be computed independently and subsequently combined. This is also possible for some lower energy gamma ray instruments. However, at higher energies and/or using different types of detectors, the responses are often not separable like this. COSI is an example, where a single 7-dimensional response is used. For Chandra, we don’t event provide response functions (other than psfs) to end-users - because the telescope dithers on the sky (and so the X-ray source moves on the detector during the observation) the user’s selection of events becomes critical for computing the correct responses. So instead we provide the users tools to create the appropriate responses and the ancillary data products needed to do so. Spectral response function data products included in the Chandra Source Catalog are specifically tied to the selection of events and the spectral model used in the catalog data processing pipelines.
>
> More exploration is probably needed to set up working examples for these uses cases.
>
> I plan to have a pull request ready on our github by wednesday, with some proposal to update the text.
>
> Thanks, Mireille
>
>
> Le 12/08/2026 à 11:08 PM, Janet Evans via heig a écrit :
>> Dear HEIG,
>>
>> Apologies for the long gap in communication. Here is a summary of some of the things that have happened since our last HEIG meeting.
>>
>> o IVOA Interop in Strasbourg
>> Great meeting for HEIG with interest in the Obscore extension, a plenary to present the HE Obscore extension doc details and the demos prepared by this group.
>> o Semantics meeting at Interop
>> Several of us met with the Semantics WG to settle several vocabulary keywords - it was a good meeting
>> o Further Obscore Note input and final review/scrub
>> There is a new section 6 along with a vocabulary update and scrub for typos/etc.
>>
>> It’s now your turn to review the document as we prepare to turn it over to the TCG/Exec for Endorsed Note status.
>> Here’s the schedule we’re working to:
>> Wed, Aug 12 — Janet sends notice of final doc to HEIG for review (typos/minor edits, no new content)
>> Tue, Aug 25 — End of HEIG review/feedback
>> Thu, Aug 27 — Janet sends note to TCG with request for review/endorsed status
>>
>> The document is available in GitHub (https://github.com/ivoa/HighEnergyObsCoreExt?tab=readme-ov-file <https://www.google.com/url?q=https://github.com/ivoa/HighEnergyObsCoreExt?tab%3Dreadme-ov-file&source=gmail-imap&ust=1788197300000000&usg=AOvVaw3qIs8TSU5Q58_qfXEUYDt5>); I’ve also appended the PDF below.
>>
>> Best regards,
>> -janet
>>
>>
>> Janet Evans | Software Development Manager | Chandra X-ray Center
>> Center for Astrophysics | Harvard & Smithsonian
>> Office: (617) 495-7160
>> 60 Garden Street | MS 81 | Cambridge, MA 02138
>>
>>
>>
>>
>>
> --
> --
> Mireille Louys, MCF (Assistant Professor)
> Centre de données Astronomiques (CDS) Equipe Images, ICube
> Observatoire de Strasbourg Telecom Physique Strasbourg
> 11, rue de l' Université 300, Bd Sebastien Brandt CS 10413
> F-67000 Strasbourg F-67412 Illkirch Cedex
> --
> heig mailing list
> heig at ivoa.net
> https://www.google.com/url?q=http://mail.ivoa.net/mailman/listinfo/heig&source=gmail-imap&ust=1788197300000000&usg=AOvVaw1AZuD1b8PQ6wmeJbaaS4ch
Cheers,
—Ian
—
Dr. Ian Evans
Astrophysicist
Chandra X-ray Center
Center for Astrophysics | Harvard & Smithsonian
Office: (617) 496 7846 | Cell: (617) 699 5152
60 Garden Street | MS 81 | Cambridge, MA 02138


<http://cfa.harvard.edu/>cfa.harvard.edu <http://cfa.harvard.edu/> | Facebook <http://cfa.harvard.edu/facebook> | Twitter <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube> | Newsletter <http://cfa.harvard.edu/newsletter>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260827/b139ac36/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-2.png
Type: image/png
Size: 581 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260827/b139ac36/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 21717 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/heig/attachments/20260827/b139ac36/attachment-0003.png>
More information about the heig
mailing list