<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">
<div class="moz-cite-prefix">Hi Ian,</div>
<div class="moz-cite-prefix"> I think when Mireille talked of 3
categories she was talking of</div>
<div class="moz-cite-prefix"> 1 ) response functions</div>
<div class="moz-cite-prefix"> 2 ) the new category of products
such as pdf , draws, and regions that you initially called
" advanced data products"</div>
<div class="moz-cite-prefix"> 3 ) the data products currently
covered by the ivoa : dataproduct_type vocabulary extension of
initial ObsCore vocabulary.<br>
</div>
<div class="moz-cite-prefix"> So she was not exactly doing the
same distinction that you are doing below because some of the
products in 3 may indeed been produced by "inverse methods" or
sophisticated analysis of raw data. <br>
</div>
<div class="moz-cite-prefix">Actually, thinking more about it I
realize that up to now data products in 3 ) were always
containing measurements initiated by a messenger of the sky and
not quantities derived from analysis of them such as
"distances", metallicities or MCM simulations of any physical
quantity, such as some of the pdf, draws and regions included in
the current definition in the proposed endorsed note <br>
</div>
<div class="moz-cite-prefix"> So our proposal is to still
distinguish the 3 categories by using different vocabularies
even if terms belonging to any of them could be used in the
dataproduct_type column of ObscOre. This is consistent with the
tables in the draft and could be reflected in the lain text by
very little changes<br>
</div>
This is now more a semantics issue than ObsCore strictly
speaking, because beside this we admit that the 3 categories of
data could be exposed via ObscOre under some conditions. </div>
<div class="moz-cite-prefix">Cheers</div>
<div class="moz-cite-prefix">François</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">PS : by the way I congratulate you and
Janet for your retirement. As Mireille is also retiring today and
Catherine and I retired officially two years ago, this note
definitely exposes a high density of end-career people !<br>
</div>
<div class="moz-cite-prefix">Le 27/08/2026 à 17:33, Dr. Ian N. Evans
via heig a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:E57A06B0-8A4F-4F19-9149-2DEDB746D8C3@cfa.harvard.edu">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Hi Mireille,
<div><br>
</div>
<div>I think this is an area where we might disagree ;-)</div>
<div><br>
</div>
<div>In my view, these “advanced” data product types can play
roles in all of the 3 categories (by which I assume you are
meaning data processing, data selection, and data analysis) and
so segregation by role is difficult.</div>
<div><br>
</div>
<div>Historically we used to acquire raw images, spectra, etc. at
the telescope and there would be a well-defined set of steps to
convert these raw data (for example, DN in an image pixel in a
camera exposure through a filter) into physical quantities (flux
in a pixel with known sky world coordinate edges and a defined
spectral bandpass and start and stop time). And so we would
consider the steps (bias subtraction, dark subtraction, flat
fielding, etc.) to take the former data and produce the later as
“data reduction” steps and could create a data processing
pipeline that performs those steps. Anything after that
required knowledge of the goals of the science study and would
be termed “data analysis” since it was in general performed by a
human. I think most historical archival data (particularly in
the optical) consist of either the raw data or the output of the
data processing pipeline with just these steps applied.</div>
<div><br>
</div>
<div>However, as data have become more complex and therefore more
difficult for individual astronomers to perform what might have
previously been termed data analysis, we are seeing more of
these steps being performed automatically in the data processing
pipelines. The increased importance of time domain astronomy,
where rapid analysis of data (e.g., photometry, spectra,
positions etc. for detected astronomical sources) is needed to
trigger alerts has also contributed significant momentum to this
transition. So now our data processing pipelines do far more
than simply converting from raw data to “calibrated” data. They
also perform “data selection” steps identifying the
“interesting” data regions and perform various “data analysis”
functions such as computing draws and pdfs and selecting between
different models for the data based on statistical analyses.
This is particularly the case for HEA where data “calibration”
doesn’t work the same way as in the optical and some elements of
“data analysis” (such as selecting appropriate source models)
are required to produce calibrated spectra and fluxes from raw
measurements. X-ray astronomy is leading the way here largely
because the instrumentation now has spatial, spectral, and
temporal resolution that rivals ground-based optical telescopes,
with effective areas and detection efficiency high enough to
gather significant numbers of counts for X-ray bright sources.
Additionally, the detailed set of steps needed to produce
scientifically optimal and robust data analysis products is
becoming increasingly complicated and difficult for the typical
multi-wavelength astronomer who is not an expert in the specific
telescope/instrument data to perform. These scientists are not
interested in becoming experts in the nuances of foibles of
every instrument whose data they use, but are primarily
interested in the final results for their science. So there is
also a strong push to further automate these steps. This is
different from 2-3 decades ago, where observers went to the
telescope and worked directly with the instrumentation, and is a
result of the growth of, and access to, archival data. This
lack of detailed exposure to the instrumentation is not
necessarily a good thing, but highlights the need for the domain
experts to foster the data analysis needs of the broader
community.</div>
<div><br>
</div>
<div>I see this merging of what were previously separate data
processing, data selection, and data analysis steps to
accelerate even more rapidly as AI enhances our capabilities in
these areas. And as I have previously mentioned, I expect that
in some cases the intermediate data products may never be
returned from the observatory. This is the future of
observational astronomy, and I don’t think segregating data
products into these separate roles is really appropriate under
these circumstances.</div>
<div><br>
</div>
<div>With regard to data levels or calibration levels, I do agree
that each project will likely have data products that fall into
multiple classifications. What I don’t agree with is that our
“advanced data products” will always have a specific calibration
level. This is similar to an image data product, where an image
may be raw instrumental data, in internal or a standard format,
or calibrated, science ready data, or enhanced such as mosaicked
data, or even an analysis product such as a heat map. So if an
image can be calib_level 0, 1, 2, 3, or 4 we should be careful
not to assume that an advanced data product such as draws or pdf
or region must automatically have calib_level = 4. A region
data product could, for example, identify the region of a
detector in detector coordinates that has a specific property
such as region that is coated with a filter material (e.g., an
Al coating). I’m not saying such a data product would be
queryable in ObsCore - and maybe it’s OK because calib_level is
an ObsCore-specific attribute - but these data product types may
have a wider range of applicability than just a single data
product level.</div>
<div><br>
</div>
<div>In your example - comparing pdfs between two different data
collections - one would need to ensure that the pdfs represent
the same quantities that are similarly calibrated in the same or
interconvertible units (e.g., PSF-fraction corrected aperture
photometry of the same region of the sky in, say, optical
magnitude with a known flux reference and X-ray energy flux).</div>
<div><br>
</div>
<div>Capturing how to handle all this appropriately and generally
in the next ObsCore update is going to be both important and
somewhat tricky.</div>
<div><br>
</div>
<div>For now, let’s keep “advanced data products” in the HEA
extension document.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>—Ian</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Aug 26, 2026, at 13:20, Mireille Louys
<a class="moz-txt-link-rfc2396E" href="mailto:mireille.louys@unistra.fr"><mireille.louys@unistra.fr></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Ian, Hi HEIG readers </p>
<p>Thanks for your clarifications and examples.</p>
<p>Still we have written the note to sort out all the
differences between the various data sets , <br>
and highlighted the benefice of the statistical
validation that pdf and draws may bring into the
game. <br>
I find it strange to mixe them back again , as we had
in the very beginning in Nov 2025. <br>
<br>
The users need to segregate between the various kinds
because the 3 categories do not play the same role<br>
in science interpretation as far as I understand. <br>
This is also why data levels are defined f.i in CTAO. </p>
<p>If I want to compare pdf or draws between 2 different
collections , the data represented need to be fully
calibrated, so <br>
they have calib_level >3 at least in Obscore sense
, and can be named advanced data products as in
Obscore 1.1 </p>
<p>The term is not statisfying , but the distinction is
, so we can keep this term for this note and will
brainstorm later to find a better one <br>
for the main Obscore extension WD ? <br>
<br>
This is my suggestion. <br>
Best wishes , Mireille<br>
<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">Le 25/08/2026 à 9:22 PM,
Dr. Ian N. Evans a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:C5894CB4-60CC-41F3-9A56-88B81A504437@cfa.harvard.edu">
<blockquote type="cite">
<div>
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
This question will need to be adressed for
Obscore 1.2 , but for this Note we can keep
"advanced data products" <br>
with the caveat that clarification is needed. </p>
</div>
</div>
</blockquote>
<div>Shall we just drop the term altogether in the
document and just call them all “data products”?</div>
</blockquote>
<pre class="moz-signature" cols="72">--
--
Mireille Louys, MCF (Assistant Professor)
Centre de données Astronomiques (CDS) Equipe Images, ICube
Observatoire de Strasbourg Telecom Physique Strasbourg
11, rue de l' Université 300, Bd Sebastien Brandt CS 10413
F-67000 Strasbourg F-67412 Illkirch Cedex</pre>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<meta charset="UTF-8">
<div
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Arial; font-size: 9.5pt; font-style: normal; font-variant-caps: normal; font-weight: 700; letter-spacing: normal; text-transform: none; white-space: pre-wrap; word-spacing: 0px; text-decoration: none; -webkit-text-stroke-width: 0px; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;">—</span></div>
<div
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Arial; font-size: 9.5pt; font-style: normal; font-variant-caps: normal; font-weight: 700; letter-spacing: normal; text-transform: none; white-space: pre-wrap; word-spacing: 0px; text-decoration: none; -webkit-text-stroke-width: 0px; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline;">
Dr. Ian Evans</span></div>
<div
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><font
face="Arial"><span
style="font-size: 12.666666984558105px; white-space: pre-wrap;"><b>Astrophysicist</b></span></font></div>
<div
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><font
face="Arial"><span
style="font-size: 12.666666984558105px; white-space: pre-wrap;"><b>Chandra X-ray Center</b></span></font></div>
<div
style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: -webkit-standard; line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="font-size: 9.5pt; font-family: Arial; background-color: rgb(255, 255, 255); font-weight: 700; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Center for Astrophysics | Harvard & Smithsonian</span></div>
<div
style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: -webkit-standard; line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="font-size: 9.5pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">
</span></div>
<div
style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: -webkit-standard; line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="font-size: 9.5pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Office: (617) 496 7846 | Cell: (617) 699 5152</span></div>
<div
style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: -webkit-standard; line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;"><span
style="font-family: Arial; font-size: 9.5pt; white-space: pre-wrap;">60 Garden Street | MS 81 | Cambridge, MA 02138</span></div>
<br class="Apple-interchange-newline"
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span
style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span><span><span><span><span><span><span><span><span><span><span><span><span
style="color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-weight: normal; font-style: normal;"><span><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span><img
alt="PastedGraphic-2.png"
src="cid:part1.LAwrENsk.MB20jfu9@gmail.com" class=""
width="263"></span>
<meta charset="UTF-8">
<span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;"><br
class="Apple-interchange-newline">
<span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"><br>
</span></span>
<div
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;"><span
style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"> </span><span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span><span><span><span><span><span><span><span><span><span><span><span
style="color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-weight: normal; font-style: normal;"><span><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></div>
<span><img alt="PastedGraphic-3.png"
src="cid:part2.bUv0PCTh.F0b2rRlD@gmail.com" class=""
width="111"></span>
<meta charset="UTF-8">
<u
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><span
style="font-size: 9.5pt; line-height: 14.566667556762695px; font-family: Arial, sans-serif;"
lang="EN"><a href="http://cfa.harvard.edu/"
moz-do-not-send="true"><br
class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</a></span></u><font
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;"
size="2" face="Arial"><u
style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
style="line-height: 14.566667556762695px;" lang="EN"><a
href="http://cfa.harvard.edu/" moz-do-not-send="true">cfa.harvard.edu</a></span></u><span
style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-height: 14.566667556762695px;"
lang="EN"> | <u><a href="http://cfa.harvard.edu/facebook"
moz-do-not-send="true">Facebook</a></u> | <u><a
href="http://cfa.harvard.edu/twitter"
moz-do-not-send="true">Twitter</a></u> | <u><a
href="http://cfa.harvard.edu/youtube"
moz-do-not-send="true">YouTube</a></u></span><span
style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-height: 14.566667556762695px;"
lang="EN"> | <u><a
href="http://cfa.harvard.edu/newsletter"
moz-do-not-send="true">Newsletter</a></u></span></font>
</div>
<br>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
</blockquote>
<p><br>
</p>
</body>
</html>