about #calibration (VEP-006) : ----> IMPORTANT for DataLInk EXTENDED USAGE

[BONNAREL FRANCOIS]

Hi Markus, all
Le 09/10/2021 à 07:43, Markus Demleitner a écrit :
> Dear François,

I'm really disappointed that nobody else get inside  this discussion.

Semantics terms in DataLink have to be thought with a wide perspective 
if we want to extend the usage

But anyway, although I think it's useless to answer you I still do it.

> On Fri, Oct 08, 2021 at 07:06:31PM +0200, BONNAREL FRANCOIS wrote:
>> Le 07/10/2021 à 15:24, Markus Demleitner a écrit :
>>> Based on this, could you then explain as clearly and concisely as you
>>> can why VEP-006 impedes that use case?
>> A user discovers a calibrated image (HST, ESO, etc...) . With DataLink
>> (#this or #preview) she has a look to the image and want to see how the
>> uncalibrated data and the flat field looked like to understand some of the
>> features. DataLink provides  a link to the #progenitor and also (by some
>> record the semantics of which cannot be anymore "calibration or #flat) to
>> the flat field, etc... used to calibrate this progenitor.
> ...but for this use case there is no need to distinguish between what
> you call a progenitor (i.e., non-calibration part of provenance) and
> calibration files applied.  Right?

Of course it's needed to make this distinction. Even to obtain the right 
caption for the display.

Not to speak about possible reprocessing

> Plus: A client can already do that, no?  If you think not: What do
> you see missing?
What would be the semantics term able to drive that? Progenitor alone  
is not : this, at least, as been discussed extensively (see below 
references)
>> Client software is intended to display all these images (science and
>> calibration) together for checking and comparison. Moreover an advanced
>> version could poropose some kind of reprocessing of progenitor.
> Not that that has any relationship to VEP-006 at all, but we have
> provenance for a detailed description of how the various pieces of
> the provenance chain play together; we certainly do not want to
> re-model that in the datalink vocabulary.  It's been compicated
> enough to do that modelling once.

Of course it is a very interesting use case of DataLink to provide a 
link towards a full (or last step) ivoa provenance record.

What #calibration-applied provides is a kind of  "poor-lady" provenance 
which only links used datasets without any insight on the activity and 
agents involved

DataLink in itself has a poor but efficient way to characterize 
relationship between #this item and the target of the link


>> Where VEP-006 "impedes" that is by letting the "already applied" use case
>> orphan.  We have no more terms to qualify the calibration files used this
>> way. With the new definition we can only apply the calibration files to the
>> discovered image itself (#this in DataLink), and not to the progenitor
> As stated multiple times, VEP-006 is entirely unconcerned with this
> problem.
>
> First, we don't have any such links now, so nothing is orphaned at
> this point.  And I maintain it would be prudent to wait until we
> actually have such links, as the motivation of people publishing such
> links will inform us if what we think is sensible behaviour (the
> "pragmatics") actually is in the view of the data providers.
>
> Second, the current #progenitor is clear that if there were any
> "Calibration applied" links, they would be covered by its concept; see
> its description: "data resources that were used to create this
> dataset (e.g. input raw data)".  You may not like the concept or its
> label, but we have VEP-009 to discuss that.

Let's go back to VEP-009

Some references

Paul Harrison May the 5th

> The description of #progenitor in the RDF is just "data resources that were used to create this dataset (e.g. input raw data)”, and this combined with the disjointness of #calibration leads me to interpret #progenitor as being “science data” progenitor. I don’t see anything written inhttps://www.ivoa.net/documents/DataLink/20150617/REC-DataLink-1.0-20150617.html  that makes this interpretation invalid.

Mireille , March the 23rd

> One more reason :
> #progenitor should be reserved to designate the data in transformation through various steps within a pipeline.
> this applies to the data stream...
> calibration, configuration, parameter sets have a distinct nature with respect to the data processing.
> The two categories should not be mixed, in my view.

Stephane Erard March the 25th

> The use cas I have in mind is a spectral parameter map built from many spectral cubes, but I think the conclusion is identical to Mireille’s example.
> In any case, I would certainly reserve #progenitor to identify calibrated products used to build a derived product. If this also links to calibration files, it will become difficult to identify the building pieces - at this stage, I’m no longer interested in details of calibration.

Not to speak about the  solution Pat proposed me in a private email (see 
my email last monday for details). I have some, concerns about it but 
this is the part I fooly agree with

Recursive usage of DataLink to provide both science data and 
calibration-used data

#progenitor link followed by #this link to get science data

#progenitor link followed by #calibration to get calibration data 
associated to these rawr science data

The consequence of this is that #progenitor itself are science data


BY the way, ESO ObsTAP service is using #progenitor exclusively for 
science data in a rawer status.

> The relevant point here is: VEP-006 in no way predetermines whether
> we do nothing at all about "Calibration applied" (full disclosure: I
> think that's the right choice), whether we create a child of
> #progenitor (perhaps after fixing its label) or whether we change its
> meaning and create a sibling of it.  Or whether we do something
> entriely different, depending on what the pragmatics turns out to be.
>
> If you disagree on this assessment: How would VEP-006 influence this
> deliberation?
VEP-006 is not proposing new terms it's changing the definition of old 
terms in a sense that calibration-applied is now forbidden.
> So, again, please let's not mix up all these different discussions.
> We will never get anywhere in semantics if we do.
>
> Everyone but François: Do you, as François alluded to, still have
> concerns with VEP-006?.
>
> If not, François, can you at least agree to: "I think VEP-006 is
> wrong, but I'll not veto it"?

Exactly this : if nobody interested I give up. But I think we will 
encounter consistency issues in a near future if we don't discuss the 
consequences of this major change of definition for #calibration.

Regards

François

>              -- Markus


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20211011/1c506996/attachment.html>