about #calibration (VEP-006) : ----> IMPORTANT for DataLInk EXTENDED USAGE

Mon Oct 11 14:07:01 CEST 2021

Dear Francois, Markus, 

Although I don’t have a solid knowledge on how datalink is implemented or how it is used by applications, from my point of view, it looks like the term as is currently proposed would block the usage of calibration to a unidirectional usage: applicable. I think that applied should also be taken into account. It looks to me that a not very sophisticated app would then just display for visual inspection, while a more sophisticated app could do something fancier...   

Cheers,
Ada

> On 11 Oct 2021, at 09:44, BONNAREL FRANCOIS <francois.bonnarel at astro.unistra.fr> wrote:
> 
> Hi Markus, all
> Le 09/10/2021 à 07:43, Markus Demleitner a écrit :
>> Dear François,
> I'm really disappointed that nobody else get inside  this discussion. 
> Semantics terms in DataLink have to be thought with a wide perspective if we want to extend the usage
> 
> But anyway, although I think it's useless to answer you I still do it.
>> On Fri, Oct 08, 2021 at 07:06:31PM +0200, BONNAREL FRANCOIS wrote:
>>> Le 07/10/2021 à 15:24, Markus Demleitner a écrit :
>>>> Based on this, could you then explain as clearly and concisely as you
>>>> can why VEP-006 impedes that use case?
>>> A user discovers a calibrated image (HST, ESO, etc...) . With DataLink
>>> (#this or #preview) she has a look to the image and want to see how the
>>> uncalibrated data and the flat field looked like to understand some of the
>>> features. DataLink provides  a link to the #progenitor and also (by some
>>> record the semantics of which cannot be anymore "calibration or #flat) to
>>> the flat field, etc... used to calibrate this progenitor.
>> ...but for this use case there is no need to distinguish between what
>> you call a progenitor (i.e., non-calibration part of provenance) and
>> calibration files applied.  Right?
> Of course it's needed to make this distinction. Even to obtain the right caption for the display. 
> Not to speak about possible reprocessing
>> 
>> Plus: A client can already do that, no?  If you think not: What do
>> you see missing?
> What would be the semantics term able to drive that? Progenitor alone  is not : this, at least, as been discussed extensively (see below references)
>> 
>>> Client software is intended to display all these images (science and
>>> calibration) together for checking and comparison. Moreover an advanced
>>> version could poropose some kind of reprocessing of progenitor.
>> Not that that has any relationship to VEP-006 at all, but we have
>> provenance for a detailed description of how the various pieces of
>> the provenance chain play together; we certainly do not want to
>> re-model that in the datalink vocabulary.  It's been compicated
>> enough to do that modelling once.
> Of course it is a very interesting use case of DataLink to provide a link towards a full (or last step) ivoa provenance record.
> 
> What #calibration-applied provides is a kind of  "poor-lady" provenance which only links used datasets without any insight on the activity and agents involved
> 
> DataLink in itself has a poor but efficient way to characterize relationship between #this item and the target of the link 
> 
>> 
>>> Where VEP-006 "impedes" that is by letting the "already applied" use case
>>> orphan.  We have no more terms to qualify the calibration files used this
>>> way. With the new definition we can only apply the calibration files to the
>>> discovered image itself (#this in DataLink), and not to the progenitor
>> As stated multiple times, VEP-006 is entirely unconcerned with this
>> problem.
>> 
>> First, we don't have any such links now, so nothing is orphaned at
>> this point.  And I maintain it would be prudent to wait until we
>> actually have such links, as the motivation of people publishing such
>> links will inform us if what we think is sensible behaviour (the
>> "pragmatics") actually is in the view of the data providers.
>> 
>> Second, the current #progenitor is clear that if there were any
>> "Calibration applied" links, they would be covered by its concept; see
>> its description: "data resources that were used to create this
>> dataset (e.g. input raw data)".  You may not like the concept or its
>> label, but we have VEP-009 to discuss that.  
> Let's go back to VEP-009
> 
> Some references 
> Paul Harrison May the 5th
> 
> 
>> The description of #progenitor in the RDF is just "data resources that were used to create this dataset (e.g. input raw data)”, and this combined with the disjointness of #calibration leads me to interpret #progenitor as being “science data” progenitor. I don’t see anything written in https://www.ivoa.net/documents/DataLink/20150617/REC-DataLink-1.0-20150617.html <https://www.ivoa.net/documents/DataLink/20150617/REC-DataLink-1.0-20150617.html> that makes this interpretation invalid.
> 
> Mireille , March the 23rd
> 
> 
>> One more reason : 
>> #progenitor should be reserved to designate the data in transformation through various steps within a pipeline.
>> this applies to the data stream...
>> calibration, configuration, parameter sets have a distinct nature with respect to the data processing.
>> The two categories should not be mixed, in my view. 
> 
> Stephane Erard March the 25th
> 
> 
>> The use cas I have in mind is a spectral parameter map built from many spectral cubes, but I think the conclusion is identical to Mireille’s example. 
>> In any case, I would certainly reserve #progenitor to identify calibrated products used to build a derived product. If this also links to calibration files, it will become difficult to identify the building pieces - at this stage, I’m no longer interested in details of calibration.
> 
> Not to speak about the  solution Pat proposed me in a private email (see my email last monday for details). I have some, concerns about it but this is the part I fooly agree with
> Recursive usage of DataLink to provide both science data and calibration-used data
> 
> #progenitor link followed by #this link to get science data
> 
> #progenitor link followed by #calibration to get calibration data associated to these rawr science data
> The consequence of this is that #progenitor itself are science data
> 
> 
> BY the way, ESO ObsTAP service is using #progenitor exclusively for science data in a rawer status.
>> 
>> The relevant point here is: VEP-006 in no way predetermines whether
>> we do nothing at all about "Calibration applied" (full disclosure: I
>> think that's the right choice), whether we create a child of
>> #progenitor (perhaps after fixing its label) or whether we change its
>> meaning and create a sibling of it.  Or whether we do something
>> entriely different, depending on what the pragmatics turns out to be.
>> 
>> If you disagree on this assessment: How would VEP-006 influence this
>> deliberation?
> VEP-006 is not proposing new terms it's changing the definition of old terms in a sense that calibration-applied is now forbidden.
>> 
>> 
>> So, again, please let's not mix up all these different discussions.
>> We will never get anywhere in semantics if we do.
>> 
>> Everyone but François: Do you, as François alluded to, still have
>> concerns with VEP-006?.
>> 
>> If not, François, can you at least agree to: "I think VEP-006 is
>> wrong, but I'll not veto it"?
> Exactly this : if nobody interested I give up. But I think we will encounter consistency issues in a near future if we don't discuss the consequences of this major change of definition for #calibration.
> 
> Regards
> 
> François
>> 
>>             -- Markus
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20211011/06a6fa06/attachment-0001.html>