New ProvenanceDM working draft released, part I

[Ole Streicher]

Hi Markus,

On 17.10.2017 09:25, Markus Demleitner wrote:
> On Mon, Oct 16, 2017 at 10:50:13PM +0200, Kristin Riebe wrote:
>> Well, I think that 'wasDerivedFrom' is meant to be used to just give you the
>> main track, i.e. the main progenitors. So I expect a wasDerivedFrom
>> relationship only to those input files of the generating activity that are
>> the main inputs. E.g. if an image is corrected using a dark frame, then the
>> image was derived from the raw image, not from the dark frame. But the raw
>> image and dark frame are both inputs.
> 
> Well, but how exactly are they different?  When using Provenance,
> 
> * if it's about debugging, a problem might equally well result from
>   an issue in the dark frame or the raw image (or perhaps an
>   interesting interaction between border cases in both).
> * if it's about dependency modeling, the output will have to be
>   re-made whether it's the dark frame or the raw image that's
>   changed.
> * if it's about giving credit, I'd argue if a dark frame (or, say, a
>   superflat) is done with enough deliberation that credit is given on
>   it in the first place, then this should be preserved in further
>   products, too.

To give you a real-world use case, which is kind-of debugging: Someone
detects an "interesting structure" on a science-ready exposure, and to
be sure he wants to process the raw image with his own, alternative
pipeline (which may or may not need the same kind of calibration). Then
he has to find out "which is *the* raw image that I need to process?",
and the answer is wasDerivedFrom (maybe recursively).

This is also something that may be already provided by the metadata: the
standard ESO files have some provenance information in the header, and
they distinguish between "RAW" and "CALIB" input files. Just as an
example one that I currently work with:

ESO PRO REC1 ID = 'muse_scibasic'
ESO PRO REC1 DRS ID = 'cpl-6.5.1'
ESO PRO REC1 PIPE ID = 'muse/1.0'
ESO PRO REC1 RAW1 NAME = 'R_AST_RA05_02.301116.fits'
ESO PRO REC1 RAW1 CATG = 'OBJECT  '
ESO PRO DATANCOM =  1
ESO PRO REC1 CAL1 NAME = 'badpix_table.fits'
ESO PRO REC1 CAL1 CATG = 'BADPIX_TABLE'
ESO PRO REC1 CAL2 NAME = [...]

Best regards

Ole