VEP-006: Discussion summary

[Patrick Dowler]

I just want to comment on the two competing interpretations of #calibration
(and children) and how one would use them in datalink.

Suppose you have two data products from ObsCore: raw data at calib_level 1
and calibrated data at calib_level 2. With the #calibration that could be
applied interpretation, the datalink(s) provided could be something like:

rawID #this {url to raw fits file}
rawID #dark {url to a suitable dark frame}
rawID #flat {url to a suitable flat field}
rawID #derivation {url to datalinks for calibID}

calibID #this {url to calibrated fits file}
calibID #progenitor {url to datalinks for rawID}

With this set of links, clients that find the rawID can find out about the
derivation or they can chose to download #this, #bias, and #flat and then
do the subtraction and division: those rawID links are "actionable".
Clients that find the calibID can navigate to the progenitor to look at the
calibration files associated. Caveat: this doesn't say those were the
actual calibration files used -- those could be the default recommended
calibration files -- so it is a weaker statement. Knowing those were
actually applied to create "calibID #this" requires provenance.... for that
maybe we really want something like:

calibID #provenance {url to provenance metadata eg instance of ProvDM}

On the other hand, with the #calibration already applied interpretation,
you would have links like this:

rawID #this {url to raw fits file}
rawID #derivation {url to datalinks for calibID}

calibID #this {url to calibrated fits file}
calibID #bias {url to the bias frame}
calibID #flat {url to the flat field}
calibID #progenitor {url to datalinks for rawID}

So, someone with the calibID could examine the calibration files and in
principle someone with the rawID could navigate to the derivation and find
the calibration files. In this case the interpretation of calibration (and
children) is a little stronger and you could infer that they were the ones
actually used to produce "calibID #this" and you could use those links to
recalibrate "rawID #this".

*And here is the big BUT:* #calibration already applied is only useful if
you actually have the calibrated data! A data provider with only raw data
(yeah, that is still a thing) has no way to tell users how to calibrate
"rawID #this".

*So, there are two use cases here: assess quality by looking at calibration
files already applied vs perform calibration of raw data. You really want
to do the latter in the case where calibrated data doesn't exist, which
means only one of the above interpretations works.*

Aside 1: the top-level concept of #auxiliary seems to me to indicate
"resources needed to interpret #this" (error, noise, weights are in there)
and I think if calibID above could have some #auxiliary links for some of
the things we've discussed... I don't think calibration terms belong in
there in either interpretation)

Aside 2: in the above, the cross linking with #progenitor and #derivation
are intended to mean "calibID #this to rawID #this" and not to specify that
"calibID #this" was created from all of the rawID links. That is, I do
agree the #progenitor is for the "science data" and a #progenitor link to
another set of links just means that "rawID" is the progenitor. I'm not
actually sure that's the best way to present a link to other links... it is
just an example.

--
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada


On Wed, 16 Jun 2021 at 03:48, BONNAREL FRANCOIS <
francois.bonnarel at astro.unistra.fr> wrote:

> 1 ) The current definition of #calibration (and child elements) is
> unambiguous I think.  They currently read "resource used to calibrate
> the primary data" , "used to subtract the detector offset level" (bias),
> "used to subtract the accumulated detector dark current" (dark), "used
> to calibrate variations in detector sensitivity" (flat)
> To me this looks unambiguous and means that the link's target HAS been
> used to calibrate this. And I think the use-case for that is quality
> checking as Mireille an Paul already enhanced it.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/semantics/attachments/20210616/7c8d0104/attachment-0001.html>