about #calibration (VEP-006) : ----> IMPORTANT for DataLInk EXTENDED USAGE

Mon Oct 11 15:42:37 CEST 2021

Hi,

I don't want to add confusion to this discussion, but I don't see a situation where the DL client really needs to know if a 
calibration is applicable or already applied.
It should be assumed that the user, as a good scientist, knows the calibration level of the searched #this. If he/she doesn't 
he/she will get it by others means that infering on the calibration semantic.
For me, saying that the calibration is 'relevant-for' should be enough.
If it is not, if one need to specify the tense, then it looks reasonable to support both possible directions (applied and 
applicable)

LM

Translated with www.DeepL.com/Translator (free version)

Le 11/10/2021 à 14:07, ada nebot a écrit :
> Dear Francois, Markus,
> 
> Although I don’t have a solid knowledge on how datalink is implemented or how it is used by applications, from my point of view, 
> it looks like the term as is currently proposed would block the usage of calibration to a unidirectional usage: applicable. I 
> think that applied should also be taken into account. It looks to me that a not very sophisticated app would then just display 
> for visual inspection, while a more sophisticated app could do something fancier...
> 
> Cheers,
> Ada
> 
>> On 11 Oct 2021, at 09:44, BONNAREL FRANCOIS <francois.bonnarel at astro.unistra.fr <mailto:francois.bonnarel at astro.unistra.fr>> 
>> wrote:
>>
>> Hi Markus, all
>> Le 09/10/2021 à 07:43, Markus Demleitner a écrit :
>>> Dear François,
>>
>> I'm really disappointed that nobody else get inside  this discussion.
>>
>> Semantics terms in DataLink have to be thought with a wide perspective if we want to extend the usage
>>
>> But anyway, although I think it's useless to answer you I still do it.
>>
>>> On Fri, Oct 08, 2021 at 07:06:31PM +0200, BONNAREL FRANCOIS wrote:
>>>> Le 07/10/2021 à 15:24, Markus Demleitner a écrit :
>>>>> Based on this, could you then explain as clearly and concisely as you
>>>>> can why VEP-006 impedes that use case?
>>>> A user discovers a calibrated image (HST, ESO, etc...) . With DataLink
>>>> (#this or #preview) she has a look to the image and want to see how the
>>>> uncalibrated data and the flat field looked like to understand some of the
>>>> features. DataLink provides  a link to the #progenitor and also (by some
>>>> record the semantics of which cannot be anymore "calibration or #flat) to
>>>> the flat field, etc... used to calibrate this progenitor.
>>> ...but for this use case there is no need to distinguish between what
>>> you call a progenitor (i.e., non-calibration part of provenance) and
>>> calibration files applied.  Right?
>>
>> Of course it's needed to make this distinction. Even to obtain the right caption for the display.
>>
>> Not to speak about possible reprocessing
>>
>>> Plus: A client can already do that, no?  If you think not: What do
>>> you see missing?
>> What would be the semantics term able to drive that? Progenitor alone  is not : this, at least, as been discussed extensively 
>> (see below references)
>>>> Client software is intended to display all these images (science and
>>>> calibration) together for checking and comparison. Moreover an advanced
>>>> version could poropose some kind of reprocessing of progenitor.
>>> Not that that has any relationship to VEP-006 at all, but we have
>>> provenance for a detailed description of how the various pieces of
>>> the provenance chain play together; we certainly do not want to
>>> re-model that in the datalink vocabulary.  It's been compicated
>>> enough to do that modelling once.
>>
>> Of course it is a very interesting use case of DataLink to provide a link towards a full (or last step) ivoa provenance record.
>>
>> What #calibration-applied provides is a kind of  "poor-lady" provenance which only links used datasets without any insight on 
>> the activity and agents involved
>>
>> DataLink in itself has a poor but efficient way to characterize relationship between #this item and the target of the link
>>
>>
>>>> Where VEP-006 "impedes" that is by letting the "already applied" use case
>>>> orphan.  We have no more terms to qualify the calibration files used this
>>>> way. With the new definition we can only apply the calibration files to the
>>>> discovered image itself (#this in DataLink), and not to the progenitor
>>> As stated multiple times, VEP-006 is entirely unconcerned with this
>>> problem.
>>>
>>> First, we don't have any such links now, so nothing is orphaned at
>>> this point.  And I maintain it would be prudent to wait until we
>>> actually have such links, as the motivation of people publishing such
>>> links will inform us if what we think is sensible behaviour (the
>>> "pragmatics") actually is in the view of the data providers.
>>>
>>> Second, the current #progenitor is clear that if there were any
>>> "Calibration applied" links, they would be covered by its concept; see
>>> its description: "data resources that were used to create this
>>> dataset (e.g. input raw data)".  You may not like the concept or its
>>> label, but we have VEP-009 to discuss that.
>>
>> Let's go back to VEP-009
>>
>> Some references
>>
>> Paul Harrison May the 5th
>>
>>
>>> The description of #progenitor in the RDF is just "data resources that were used to create this dataset (e.g. input raw data)”, and this combined with the disjointness of #calibration leads me to interpret #progenitor as being “science data” progenitor. I don’t see anything written inhttps://www.ivoa.net/documents/DataLink/20150617/REC-DataLink-1.0-20150617.html  that makes this interpretation invalid.
>>
>> Mireille , March the 23rd
>>
>>
>>> One more reason :
>>> #progenitor should be reserved to designate the data in transformation through various steps within a pipeline.
>>> this applies to the data stream...
>>> calibration, configuration, parameter sets have a distinct nature with respect to the data processing.
>>> The two categories should not be mixed, in my view.
>>
>> Stephane Erard March the 25th
>>
>>
>>> The use cas I have in mind is a spectral parameter map built from many spectral cubes, but I think the conclusion is identical to Mireille’s example.
>>> In any case, I would certainly reserve #progenitor to identify calibrated products used to build a derived product. If this also links to calibration files, it will become difficult to identify the building pieces - at this stage, I’m no longer interested in details of calibration.
>>
>> Not to speak about the  solution Pat proposed me in a private email (see my email last monday for details). I have some, 
>> concerns about it but this is the part I fooly agree with
>>
>> Recursive usage of DataLink to provide both science data and calibration-used data
>>
>> #progenitor link followed by #this link to get science data
>>
>> #progenitor link followed by #calibration to get calibration data associated to these rawr science data
>>
>> The consequence of this is that #progenitor itself are science data
>>
>>
>> BY the way, ESO ObsTAP service is using #progenitor exclusively for science data in a rawer status.
>>
>>> The relevant point here is: VEP-006 in no way predetermines whether
>>> we do nothing at all about "Calibration applied" (full disclosure: I
>>> think that's the right choice), whether we create a child of
>>> #progenitor (perhaps after fixing its label) or whether we change its
>>> meaning and create a sibling of it.  Or whether we do something
>>> entriely different, depending on what the pragmatics turns out to be.
>>>
>>> If you disagree on this assessment: How would VEP-006 influence this
>>> deliberation?
>> VEP-006 is not proposing new terms it's changing the definition of old terms in a sense that calibration-applied is now forbidden.
>>> So, again, please let's not mix up all these different discussions.
>>> We will never get anywhere in semantics if we do.
>>>
>>> Everyone but François: Do you, as François alluded to, still have
>>> concerns with VEP-006?.
>>>
>>> If not, François, can you at least agree to: "I think VEP-006 is
>>> wrong, but I'll not veto it"?
>>
>> Exactly this : if nobody interested I give up. But I think we will encounter consistency issues in a near future if we don't 
>> discuss the consequences of this major change of definition for #calibration.
>>
>> Regards
>>
>> François
>>
>>>              -- Markus
>>
>>
> 

--
English version: https://www.deepl.com/translator
-- 
jesuischarlie/Tunis/Paris/Bruxelles/Berlin

Laurent Michel
SSC XMM-Newton
Tél : +33 (0)3 68 85 24 37
Fax : +33 (0)3 )3 68 85 24 32
Université de Strasbourg <http://www.unistra.fr>
Observatoire Astronomique
11 Rue de l'Université
F - 67200 Strasbourg