<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I just want to comment on the two competing interpretations of #calibration (and children) and how one would use them in datalink.<br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Suppose you have two data products from ObsCore: raw data at calib_level 1 and calibrated data at calib_level 2. With the #calibration that could be applied interpretation, the datalink(s) provided could be something like:<br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">rawID #this {url to raw fits file}</div><div class="gmail_default" style="font-size:small">rawID #dark {url to a suitable dark frame}</div><div class="gmail_default" style="font-size:small">rawID #flat {url to a suitable flat field}</div><div class="gmail_default" style="font-size:small">rawID #derivation {url to datalinks for calibID}</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">calibID #this {url to calibrated fits file}</div><div class="gmail_default" style="font-size:small">calibID #progenitor {url to datalinks for rawID}</div><div class="gmail_default" style="font-size:small"></div><div class="gmail_default" style="font-size:small"></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">With this set of links, clients that find the rawID can find out about the derivation or they can chose to download #this, #bias, and #flat and then do the subtraction and division: those rawID links are "actionable". Clients that find the calibID can navigate to the progenitor to look at the calibration files associated. Caveat: this doesn't say those were the actual calibration files used -- those could be the default recommended calibration files -- so it is a weaker statement. Knowing those were actually applied to create "calibID #this" requires provenance.... for that maybe we really want something like:<br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">calibID #provenance {url to provenance metadata eg instance of ProvDM}</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">On the other hand, with the #calibration already applied interpretation, you would have links like this:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><div class="gmail_default" style="font-size:small">rawID #this {url to raw fits file}</div><div class="gmail_default" style="font-size:small">rawID #derivation {url to datalinks for calibID}</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">calibID #this {url to calibrated fits file}</div><div class="gmail_default" style="font-size:small"><div class="gmail_default" style="font-size:small">calibID #bias {url to the bias frame}</div><div class="gmail_default" style="font-size:small">calibID #flat {url to the flat field}</div></div><div class="gmail_default" style="font-size:small">calibID #progenitor {url to datalinks for rawID}</div></div><div class="gmail_default" style="font-size:small"><br></div><div style="font-size:small" class="gmail_default">So, someone with the calibID could examine the calibration files and in principle someone with the rawID could navigate to the derivation and find the calibration files. In this case the interpretation of calibration (and children) is a little stronger and you could infer that they were the ones actually used to produce "calibID #this" and you could use those links to recalibrate "rawID #this". <br><br><b>And here is the big BUT:</b> #calibration already applied is only useful if you actually have the calibrated data! A data provider with only raw data (yeah, that is still a thing) has no way to tell users how to calibrate "rawID #this".<br></div><div style="font-size:small" class="gmail_default"><b><br></b></div><div style="font-size:small" class="gmail_default"><b>So, there are two use cases here: assess quality by looking at calibration files already applied vs perform calibration of raw data. You really want to do the latter in the case where calibrated data doesn't exist, which means only one of the above interpretations works.</b></div><div style="font-size:small" class="gmail_default"><br></div><div style="font-size:small" class="gmail_default">Aside 1: the top-level concept of #auxiliary seems to me to indicate "resources needed to interpret #this" (error, noise, weights are in there) and I think if calibID above could have some #auxiliary links for some of the things we've discussed... I don't think calibration terms belong in there in either interpretation)</div><div style="font-size:small" class="gmail_default"><br></div><div style="font-size:small" class="gmail_default">Aside 2: in the above, the cross linking with #progenitor and #derivation are intended to mean "calibID #this to rawID #this" and not to specify that "calibID #this" was created from all of the rawID links. That is, I do agree the #progenitor is for the "science data" and a #progenitor link to another set of links just means that "rawID" is the progenitor. I'm not actually sure that's the best way to present a link to other links... it is just an example. <br></div><br><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div>--<br></div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 16 Jun 2021 at 03:48, BONNAREL FRANCOIS <<a href="mailto:francois.bonnarel@astro.unistra.fr">francois.bonnarel@astro.unistra.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
1 ) The current definition of #calibration (and child elements) is <br>
unambiguous I think. They currently read "resource used to calibrate <br>
the primary data" , "used to subtract the detector offset level" (bias), <br>
"used to subtract the accumulated detector dark current" (dark), "used <br>
to calibrate variations in detector sensitivity" (flat)<br>
To me this looks unambiguous and means that the link's target HAS been <br>
used to calibrate this. And I think the use-case for that is quality <br>
checking as Mireille an Paul already enhanced it.<br>
<br>
<br>
</blockquote></div></div>