-Moving forwards <--Re: about #calibration (VEP-006) : ----> IMPORTANT for DataLInk EXTENDED USAGE

BONNAREL FRANCOIS francois.bonnarel at astro.unistra.fr
Thu Oct 14 10:47:48 CEST 2021


Hi Pat, all
Le 13/10/2021 à 20:26, Patrick Dowler a écrit :

>
> I agree with the intent of VEP-006 that #calibration and it's child 
> properties should cover the "use this raw data" use case
> and that the current ambiguity will become an issue of not resolved.

As you know I disagree that the previous definition was ambiguous (tense 
was Ok). But anyway we didn't have a working implementation yet in the 
VO landscape as far as we know.

So as I told yesterday during TCG I do not oppose to adoption of VEP-006 
to tackle this "applicable" use case.

>
> I understand the "already applied" use case and that some providers 
> will want to provide links with the product. If that use case is 
> "debugging" or "QA" by a human, then the existing #auxiliary and a 
> decent description should suffice to support that. Providers can start 
> there and see what shortcomings they run into... I completely agree 
> with Markus that when someone actually tries to do this that we'll 
> find out whether the use case really is QA or something else and we 
> can't solve it until we find out what something else is. So, is
>
> #auxiliary description="flat field used to calibrate this image"
>
> sufficient to prototype the other use case(s)?
>
I don't think so. With my provenance co-author hat and my radioastronomy 
interests and examples I don't think it's a good idea to mix all those 
things in auxiliray. at  very first argument, there is a possibility of 
selection of links using the semantics terms and their hierarchy which 
we will lose by doing that.

As I said yesterday I will propose a new VEP for 
calibration-already-applied and this will be an opportunity to discuss 
these things further. And I'm sure an implementation will come soon. We 
can create a directory on dal-use-cases git repository to push these use 
cases.

By the way #progenitor definition-fix has to be further discussed. this 
is VEP-009.

Cheers

François


> --
> Patrick Dowler
> Canadian Astronomy Data Centre
> Victoria, BC, Canada
>
>
> On Wed, 13 Oct 2021 at 08:48, BONNAREL FRANCOIS 
> <francois.bonnarel at astro.unistra.fr 
> <mailto:francois.bonnarel at astro.unistra.fr>> wrote:
>
>     Hi all, Markus
>     As I said, I am ready to let VEP-006 go because it's trying to
>     solve a
>     real use case if nobody's reluctant to it in the TCG.
>     But I am confident that the other use case (applied) will get out
>     very
>     soon, if its not already somewhere in some DataLink response and not
>     noticed by our "radars"
>     (there is no possibility to check which terms are used in DataLink
>     services or capabilities at the registry level)
>     So I will have warned : by not trying to find a consistent and global
>     solution now we may encounter problems in a near future
>     But However I cannot keep silent when I read some inconsistences
>     in the
>     rationale. see below
>     Le 11/10/2021 à 14:19, Markus Demleitner a écrit :
>     > François,
>     >
>     > On Mon, Oct 11, 2021 at 09:58:42AM +0200, BONNAREL FRANCOIS wrote:
>     >>> On Fri, Oct 08, 2021 at 07:06:31PM +0200, BONNAREL FRANCOIS wrote:
>     >>>> Le 07/10/2021 à 15:24, Markus Demleitner a écrit :
>     >>>>> Based on this, could you then explain as clearly and
>     concisely as you
>     >>>>> can why VEP-006 impedes that use case?
>     >>>> A user discovers a calibrated image (HST, ESO, etc...) . With
>     DataLink
>     >>>> (#this or #preview) she has a look to the image and want to
>     see how the
>     >>>> uncalibrated data and the flat field looked like to
>     understand some of the
>     >>>> features. DataLink provides  a link to the #progenitor and
>     also (by some
>     >>>> record the semantics of which cannot be anymore "calibration
>     or #flat) to
>     >>>> the flat field, etc... used to calibrate this progenitor.
>     >>> ...but for this use case there is no need to distinguish
>     between what
>     >>> you call a progenitor (i.e., non-calibration part of
>     provenance) and
>     >>> calibration files applied.  Right?
>     >> Of course it's needed to make this distinction. Even to obtain
>     the right
>     >> caption for the display.
>     > A datalink client will obviously take the caption from datalink's
>     > description field, no?  I frankly cannot see what role the semantics
>     > field could have in this.
>     >
>     > What else are you thinking of?  As I said, it helps to use the "A
>     > user wants... the computer does... using..." template when stating
>     > such things so other people can follow.
>     >
>     >> Not to speak about possible reprocessing
>     > I think we all agree that datalink metadata is *far* too weak to
>     > support this; I suspect even full provenance will not usually let a
>     > computer work out a reprocessing chain by itself.  You know,
>     workflow
>     > engines are the complex beasts they are for a reason. So, datalink
>     > may help selecting artefacts mentioned in a provenance instance, but
>     > for that, a "Part-of-Provenance" concept is enough. Agreed?
>
>     About the two answers above
>
>     If the "description" display is enough for "applied" why is it not
>     the
>     case for "applicable" (VEP-006 definition for #calibration) ?
>
>     What should the client do more in the case of applicable than for
>
>     You have anyway no idea in general what kind of process you will
>     really
>     apply with these calibration data  ?
>
>     You write that reprocessing is too complex for datalink in the
>     case of
>     "already applied" but I imagine it's excatly the same for applicable.
>
>     So if "description" are enough why don't we follow Laurent, Ada
>     and many
>     other before to relax the defintion and allow both use cases ?
>
>     It is not my prefered solution as you know but it would be more
>     consistent with what you are writing there.
>
>     Cheers
>
>     François
>
>     >>> Plus: A client can already do that, no?  If you think not: What do
>     >>> you see missing?
>     >> What would be the semantics term able to drive that? Progenitor
>     alone  is
>     >> not : this, at least, as been discussed extensively (see below
>     references)
>     > I do not see why it would not be.  "A user wants to debug a data
>     > product.  The computer takes all #progenitor links and displays them
>     > together with their descriptions, offering to download them for
>     > inspection or possible use with a full description of the
>     provenance."
>     >
>     >>>> Client software is intended to display all these images
>     (science and
>     >>>> calibration) together for checking and comparison. Moreover
>     an advanced
>     >>>> version could poropose some kind of reprocessing of progenitor.
>     >>> Not that that has any relationship to VEP-006 at all, but we have
>     >>> provenance for a detailed description of how the various pieces of
>     >>> the provenance chain play together; we certainly do not want to
>     >>> re-model that in the datalink vocabulary.  It's been compicated
>     >>> enough to do that modelling once.
>     >> Of course it is a very interesting use case of DataLink to
>     provide a link
>     >> towards a full (or last step) ivoa provenance record.
>     > Yes.  But that doesn't mean we have to re-build provenance in
>     > datalink.  On the contrary: we can have a nice, clean separation of
>     > concerns, where datalink says how to get things and provenance says
>     > how they fit together.
>     >
>     >> What #calibration-applied provides is a kind of "poor-lady"
>     provenance
>     >> which only links used datasets without any insight on the
>     activity and
>     >> agents involved
>     >>
>     >> DataLink in itself has a poor but efficient way to characterize
>     relationship
>     >> between #this item and the target of the link
>     > Yes -- it's enough to filter links, which is what we want in
>     datalink
>     > semantics.  And VEP-006 plus the current state does exactly this.
>     >
>     >>> Second, the current #progenitor is clear that if there were any
>     >>> "Calibration applied" links, they would be covered by its
>     concept; see
>     >>> its description: "data resources that were used to create this
>     >>> dataset (e.g. input raw data)".  You may not like the concept
>     or its
>     >>> label, but we have VEP-009 to discuss that.
>     >> Let's go back to VEP-009
>     > Sure, but can we *please* do that outside of the VEP-006 discussion?
>     > I'm very sure we're not yet proficient enough in this kind of
>     > discussion that we can have multiple of them at the same time.  And
>     > I think you still have not argued why VEP-006 and VEP-009 could not
>     > be treated separately, i.e., how my elaboration of how we still have
>     > all reasonable options even after accepting VEP-006.
>     >
>     >> Some references
>     >>
>     >> Paul Harrison May the 5th
>     >>
>     >> Mireille , March the 23rd
>     >>
>     >> Stephane Erard March the 25th
>     >>
>     > All these persons have been at meetings in the meantime, and (at
>     > least that's what I took away from these meetings) they were
>     > satisified that their concerns were taken into account in the
>     current
>     > form of VEP-006.
>     >
>     > Paul, Mireille, and Stéphane: If I'm misrepresenting you, please
>     > correct me.
>     >
>     >> Not to speak about the  solution Pat proposed me in a private
>     email (see my
>     >> email last monday for details). I have some, concerns about it
>     but this is
>     >> the part I fooly agree with
>     >>
>     >> Recursive usage of DataLink to provide both science data and
>     >> calibration-used data
>     >>
>     >> #progenitor link followed by #this link to get science data
>     >>
>     >> #progenitor link followed by #calibration to get calibration
>     data associated
>     >> to these rawr science data
>     > While I don't believe this belongs into a discussion of VEP-006,
>     this
>     > is one reason why I'm rather skeptical of your VEP-009: With current
>     > #progenitor, the link from the reduced to the raw datalink document
>     > would reasonably be #progenitor.  With VEP-009, that is quite
>     > certainly no longer the case (unless your definition of "science
>     > data" took a surprising turn later).  But let's discuss that with
>     > VEP-009.
>     >
>     >> The consequence of this is that #progenitor itself are science data
>     > No, in that case it would be a mix of "science data" and all
>     kinds of
>     > other things that went into the reduction.  Which, mind you, is
>     fine,
>     > and I think it's the way such things should be done. It's just not
>     > within the concept your description in VEP-009 seems to try to
>     > define.
>     >
>     >>> If you disagree on this assessment: How would VEP-006
>     influence this
>     >>> deliberation?
>     >> VEP-006 is not proposing new terms it's changing the definition
>     of old terms
>     >> in a sense that calibration-applied is now forbidden.
>     > Well, the concept "pre-VEP-006-#calibration minus
>     > post-VEP-006-#calibration" apparently is not populated in the
>     current
>     > VO, and as I argued in in two mails back, when members of that
>     > concept come around, all the options are still around whether or not
>     > we accept VEP-006; so, again, I don't see why we can't at least get
>     > VEP-006 off the table.
>     >
>     >
>     >>> If not, François, can you at least agree to: "I think VEP-006 is
>     >>> wrong, but I'll not veto it"?
>     >> Exactly this : if nobody interested I give up. But I think we
>     will encounter
>     >> consistency issues in a near future if we don't discuss the
>     consequences of
>     >> this major change of definition for #calibration.
>     > Can you speculate what consistency issues you expect to see? 
>     Because
>     > you see, the only reason VEP-006 is there is that without it there's
>     > the problem that current #progenitor and #calibration have a
>     nonempty
>     > intersection and a nonempty difference, which is really bad in a
>     > formal vocabulary of this sort.
>     >
>     > Now, since the only point of the VEP is to fix an inconsistency, it
>     > would defeat the purpose if new ones came up.
>     >
>     >
>     > Anyway, now that Fançoise has chimed in -- what do we do?
>     >
>     > For me, it would still be helpful to see what problem you François,
>     > are trying to solve or you, Françoise, see as well.  And please try
>     > to be as concrete as possible and to limit things to VEP-006. 
>     And if
>     > you feel that is *reayll* impossible, at least make a strong and
>     > reproducible case for why we have to solve the question of the
>     > "Part-of-Provenance" concept together with it.
>     >
>     >                -- Markus
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/semantics/attachments/20211014/c2eb46ec/attachment-0001.html>


More information about the semantics mailing list