about #calibration (VEP-006) : ----> IMPORTANT for DataLInk EXTENDED USAGE

Mon Oct 11 10:11:08 CEST 2021

Forgot to say that I had my astronomer's hat firmly on my head when I 
made the comment.

Francoise

Le 11/10/2021 à 10:09, Francoise Genova a écrit :
> Le 11/10/2021 à 09:58, BONNAREL FRANCOIS a écrit :
>>
>>
>> Hi Markus, all
>> Le 09/10/2021 à 07:43, Markus Demleitner a écrit :
>>> Dear François,
>>
>> I'm really disappointed that nobody else get inside  this discussion.
>>
>> Semantics terms in DataLink have to be thought with a wide 
>> perspective if we want to extend the usage
>>
>> But anyway, although I think it's useless to answer you I still do it.
>>
>>> On Fri, Oct 08, 2021 at 07:06:31PM +0200, BONNAREL FRANCOIS wrote:
>>>> Le 07/10/2021 à 15:24, Markus Demleitner a écrit :
>>>>> Based on this, could you then explain as clearly and concisely as you
>>>>> can why VEP-006 impedes that use case?
>>>> A user discovers a calibrated image (HST, ESO, etc...) . With DataLink
>>>> (#this or #preview) she has a look to the image and want to see how the
>>>> uncalibrated data and the flat field looked like to understand some of the
>>>> features. DataLink provides  a link to the #progenitor and also (by some
>>>> record the semantics of which cannot be anymore "calibration or #flat) to
>>>> the flat field, etc... used to calibrate this progenitor.
>>> ...but for this use case there is no need to distinguish between what
>>> you call a progenitor (i.e., non-calibration part of provenance) and
>>> calibration files applied.  Right?
>>
>> Of course it's needed to make this distinction. Even to obtain the 
>> right caption for the display.
>>
>> Not to speak about possible reprocessing
>>
> I did not follow the discussion until now but I fully agree with 
> Francois on that (and we did not discuss the point beforehand)
>
> Francoise
>
>>> Plus: A client can already do that, no?  If you think not: What do
>>> you see missing?
>> What would be the semantics term able to drive that? Progenitor 
>> alone  is not : this, at least, as been discussed extensively (see 
>> below references)
>>>> Client software is intended to display all these images (science and
>>>> calibration) together for checking and comparison. Moreover an advanced
>>>> version could poropose some kind of reprocessing of progenitor.
>>> Not that that has any relationship to VEP-006 at all, but we have
>>> provenance for a detailed description of how the various pieces of
>>> the provenance chain play together; we certainly do not want to
>>> re-model that in the datalink vocabulary.  It's been compicated
>>> enough to do that modelling once.
>>
>> Of course it is a very interesting use case of DataLink to provide a 
>> link towards a full (or last step) ivoa provenance record.
>>
>> What #calibration-applied provides is a kind of "poor-lady" 
>> provenance which only links used datasets without any insight on the 
>> activity and agents involved
>>
>> DataLink in itself has a poor but efficient way to characterize 
>> relationship between #this item and the target of the link
>>
>>
>>>> Where VEP-006 "impedes" that is by letting the "already applied" use case
>>>> orphan.  We have no more terms to qualify the calibration files used this
>>>> way. With the new definition we can only apply the calibration files to the
>>>> discovered image itself (#this in DataLink), and not to the progenitor
>>> As stated multiple times, VEP-006 is entirely unconcerned with this
>>> problem.
>>>
>>> First, we don't have any such links now, so nothing is orphaned at
>>> this point.  And I maintain it would be prudent to wait until we
>>> actually have such links, as the motivation of people publishing such
>>> links will inform us if what we think is sensible behaviour (the
>>> "pragmatics") actually is in the view of the data providers.
>>>
>>> Second, the current #progenitor is clear that if there were any
>>> "Calibration applied" links, they would be covered by its concept; see
>>> its description: "data resources that were used to create this
>>> dataset (e.g. input raw data)".  You may not like the concept or its
>>> label, but we have VEP-009 to discuss that.
>>
>> Let's go back to VEP-009
>>
>> Some references
>>
>> Paul Harrison May the 5th
>>
>>> The description of #progenitor in the RDF is just "data resources that were used to create this dataset (e.g. input raw data)”, and this combined with the disjointness of #calibration leads me to interpret #progenitor as being “science data” progenitor. I don’t see anything written inhttps://www.ivoa.net/documents/DataLink/20150617/REC-DataLink-1.0-20150617.html  that makes this interpretation invalid.
>>
>> Mireille , March the 23rd
>>
>>> One more reason :
>>> #progenitor should be reserved to designate the data in transformation through various steps within a pipeline.
>>> this applies to the data stream...
>>> calibration, configuration, parameter sets have a distinct nature with respect to the data processing.
>>> The two categories should not be mixed, in my view.
>>
>> Stephane Erard March the 25th
>>
>>> The use cas I have in mind is a spectral parameter map built from many spectral cubes, but I think the conclusion is identical to Mireille’s example.
>>> In any case, I would certainly reserve #progenitor to identify calibrated products used to build a derived product. If this also links to calibration files, it will become difficult to identify the building pieces - at this stage, I’m no longer interested in details of calibration.
>>
>> Not to speak about the  solution Pat proposed me in a private email 
>> (see my email last monday for details). I have some, concerns about 
>> it but this is the part I fooly agree with
>>
>> Recursive usage of DataLink to provide both science data and 
>> calibration-used data
>>
>> #progenitor link followed by #this link to get science data
>>
>> #progenitor link followed by #calibration to get calibration data 
>> associated to these rawr science data
>>
>> The consequence of this is that #progenitor itself are science data
>>
>>
>> BY the way, ESO ObsTAP service is using #progenitor exclusively for 
>> science data in a rawer status.
>>
>>> The relevant point here is: VEP-006 in no way predetermines whether
>>> we do nothing at all about "Calibration applied" (full disclosure: I
>>> think that's the right choice), whether we create a child of
>>> #progenitor (perhaps after fixing its label) or whether we change its
>>> meaning and create a sibling of it.  Or whether we do something
>>> entriely different, depending on what the pragmatics turns out to be.
>>>
>>> If you disagree on this assessment: How would VEP-006 influence this
>>> deliberation?
>> VEP-006 is not proposing new terms it's changing the definition of 
>> old terms in a sense that calibration-applied is now forbidden.
>>> So, again, please let's not mix up all these different discussions.
>>> We will never get anywhere in semantics if we do.
>>>
>>> Everyone but François: Do you, as François alluded to, still have
>>> concerns with VEP-006?.
>>>
>>> If not, François, can you at least agree to: "I think VEP-006 is
>>> wrong, but I'll not veto it"?
>>
>> Exactly this : if nobody interested I give up. But I think we will 
>> encounter consistency issues in a near future if we don't discuss the 
>> consequences of this major change of definition for #calibration.
>>
>> Regards
>>
>> François
>>
>>>              -- Markus
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20211011/4c9fa0fc/attachment.html>