[EXTERNAL] [BULK] Re: [Heig] vocabulary update: proposal for dataproduct_type update for high energy data : event-list definition and event-bundle
Jaffe, Tess (GSFC-6601)
tess.jaffe at nasa.gov
Wed Apr 30 15:48:29 CEST 2025
Hi all,
Possibly dumb question:
If an ObsCore table lists an event-bundle as a separate row with its own product_type, and the access_url follows best practice specifying a datalink that will return the bundle, what should the DataLink result include as #this? We are actively putting this together now at HEASARC. If the product type is simply a spectrum, our datalink result has the spectrum file as #this and the response matrices, background, etc. as related products in the same result table. If the product itself is a bundle, what is the #this? Do we have to provide a tarball or something? Or are there multiple #this with different dataproduct_subtypes? The latter doesn't sound right to me.
Cheers,
Tess
PS We have a visible test ObsTAP service (https:// heasarc.gsfc.nasa.gov/xamin_test/vo/tap<https://heasarc.gsfc.nasa.gov/xamin_test/vo/tap>) but it has not yet been "datalink-ized" so the access_urls are currently direct to the files. Our production SIA service is now using datalink, if you want to see how we are currently doing it. Try
https:// heasarc.gsfc.nasa.gov/xamin/vo/sia?table=chanmaster&POS=CIRCLE+202.469575+47.1952583+0.<https://heasarc.gsfc.nasa.gov/xamin/vo/sia?table=chanmaster&POS=CIRCLE+202.469575+47.1952583+0.>
and you get access_url like
<TD>https://heasarc.gsfc.nasa.gov/xamin/vo/datalink?datalink_key&id=ivo://nasa.heasarc/chanmaster?6786/chandra.obs.img.cntr.fits</TD>
which gets you
<TD>https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/6//6786/primary/acisf06786N003_cntr_img2.fits.gz</TD>
Comments welcome.
________________________________
From: heig <heig-bounces at ivoa.net> on behalf of Bruno Khelifi via heig <heig at ivoa.net>
Sent: Wednesday, April 30, 2025 4:11 AM
To: semantics at ivoa.net <semantics at ivoa.net>; heig at ivoa.net <heig at ivoa.net>
Subject: [EXTERNAL] [BULK] Re: [Heig] vocabulary update: proposal for dataproduct_type update for high energy data : event-list definition and event-bundle
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hello Markus,
Le 30/04/2025 à 08:57, Markus Demleitner via heig a écrit :
Hi Bruno,
On Tue, Apr 29, 2025 at 03:27:35PM +0200, Bruno Khelifi via semantics wrote:
Le 25/04/2025 à 09:29, Markus Demleitner via heig a écrit :
(1) used-in: I really, *really* would like to see actual, published
data here (always, in all VEPs; it's a pain if we go into all the
[...]
used-in: dataset ivo://csc.harvard.edu/scsr2?some-obs-id onhttp://cda.cfa.harvard.edu/csctap
In the VHE domain, CTAO, KM3NeT, SWGO wish to publish their current and
future data in VO. The current running experiments MAGIC and HESS are
working toward the publication of their legacy data into the VO.
And this is also in this context that such extensions are of interest for
us.
Is anything already online in obscore or via datalink? It would
really be great if there was something out there when the VEP
officially goes up?
Not yet. We are preparing a HEIG note to present our needs and their justifications. Mathieu Servillat, Ian Evans and al. will make some R&D to be sure that it works well... even if DataLink is well secured;)
(2) Relationship: That's an operational field, i.e., I need to create
[...]
user side: If I'm looking for #event-bundle, do I want to see
#event-list, too? If I'm looking for #event-list, do I want to see
#event-bundle, too? Whatever ought to encompass the other is the
wider term.
event-bundle is e.g. event-list + response (a set of IRFs)
(+provenance+readme+etc)
In this context, one would need to use DataLink to access to all files
Well... but does
#event-list #skos:broader #event-bundle
hold or is it the other way round?
As a simple physicist, I do not know that means #skos:broader. In words, an event-list is contained into an event-bundle, because event-bundle=event-list+IRFs+provenance+etc
First, our IRFs are not just accessory, they are mandatory for HE (photons
and neutrinos) to make physics (for data analysis specialists:
I don't doubt this, I was just wondering if, from a *product-type*
point of view there is a sufficient point in telling #event-bundle
and #event-list apart: Do you ever want to discover one or the other?
Would it perhaps be necessary to tell the two apart for the purpose
of choosing an appropriate SAMP client?
A good answer to that would also help answering the relationship
question.
As I said, having event-list w/o IRFs is so poor and does not allow to make science. One needs to discover the event-bundle, and as consequence define properly its contents, ie event-list and response.
In addition, one has use cases where these IRFs can be used without
event-list: the case of simulations. Also, one can have one set of IRFs for
many observations. So a data producer could use a dedicated entry in ObsCore
for that.
Ah-ha! But this would mean that you would need IRFs as an
independent product type, no? How else would you annotate these
stand-alone IRFs?
The UCs to use IRFs by its own and to be able to discover them are:
- make source simulations for a given instrument
- compute sensitivity of an instrument
- make predictive performance curves for an instrument
- share (large) IRFs valid for a wide range of observations (in the spirit of the CALDB of X-rays) - very useful for IceCube, KM3NeT, HAWC, SWGO (and maybe AUGER, etc)
This is quite important already, I think. I have to think a bit more to check if the list is complete.
If that's solid reasoning (and frankly, I only have a fairly vague
idea of what I'm talking about here), then we could introduce some
concept of *#ancillary into product-type (which feels a bit wrong
because we have that concept in datalink/core, too, but I think it's
justifiable), and then *#irf as a narrower term. #event-bundle would
then be narrower than both #irf and #event-list.
ancillary is so vague that it can contains everything... This is a vague stuff, a container somehow. Here, we propose to describe correctly some of its possible elements: response, ARF, RMF, PSF, Edisp, Aeff, Bkg(=background). All these elements are critical for us and they deserve a proper description such that the HE experiments (from X-rays to PeV, in photons and neutrino). Indeed, there are specific elements, like a phot.mag.distMod, phys.ionizParam.coll or src.calib.guideStar. Each astronomical domain has its own peculiarities, applicable only for its own data releases. Narrow elements are common in IVOA. Are these elements too narrow? We are just covering 12 orders of magnitude in the electro-magnetic spectrum, which is not so narrow;)
The consequence would be that searches for #event-list yield
#event-bundle, too, but searches for #event-bundle wouldn't. Does
that sound right to you? If so, could you try to write a definition
for *#irf (and perhaps come up with a less cryptic label)? I think
we should amend the #event-bundle VEP with that on top, if this is
what we're going to do.
The data producers could have the freedom to organise their releases as they wish. The 2 UCs associated to the data release organisation are:
- a list of event-bundle (using DataLink to access to all downloadable files) - typical case for Imaging Atmospheric Cherenkov telescopes
- or separate entries for the event-list and the response - a possible case for Water Cherenkov Detectors and neutrinos (or cosmic-rays experiments)
- or dedicated entries about the response that is "representative" of an instrument (UC: proposal preparation, estimation of instrument performance for a given science use case, instrument sensitivity - Zenodo is great, but all the IVOA technology permits such important use cases)
Maybe one can discuss about that this afternoon!
Thank you for your feedback,
Bruno
Thanks,
Markus
--
Bruno Khelifi
Physicist at CNRS (laboratory APC, Paris)
Phone: +33.1.57.27.61.58 - Fax: +33.1.57.27.60.71
APC, IN2P3/CNRS - Universite de Paris Cite
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/semantics/attachments/20250430/ff83b055/attachment-0001.htm>
More information about the semantics
mailing list