Obscore: obs_id not null requirement

[Markus Demleitner]

Dear DAL,

On Tue, Mar 08, 2022 at 12:15:10PM +0100, Mireille LOUYS wrote:
> suppose 2 different service curators expose HST reprocessed data :
> they would have different /obs_publisher_did/ , minted by each curator , but
> could refer to the same /obs_id/ in the data provider collection.
> 
> when the ObsCore spec was discussed , we had in mind this would be easy to
> fill in the /obs_id/  in any case
> , because the data provider can always use the same id for both fields :
> /obs_publisher_did /, and /obs_id/ for the first round of published data.

Yeah, well, this is not about the ease of putting something in there,
this is about the requirement that it MUST NOT be NULL, which is
something that needs to be checked, with a check that is fairly
expensive.

What I'm trying to argue in the blog post is that we need a better
reason to require something than that it's easy to fill:  It must
actually enable something that is impossible without the requirement.


> I do not see today how I could interpret obs_id = NULL unambiguously :
> - missing data ? I don't know the origin of this data ?
> - primary data ? but we have calib_level to mention it more precisely , etc

Well, it would simply be "not given".  Which, given the use cases
"grouping" and "cross-service matching" would just mean "this data
set doesn't belong to a group" and "I don't know what other services
will call the underlying observation".

I give you the cross-service identification is a valid and credible
use case (a bit along the lines of the SSAP creatorDID).  But
requiring obs_id to be not NULL doesn't really help it: If people are
not diligent enough to put in the cross-service obs_id when they
could put in NULLs, they probably won't do it now either and they'll
just put in something (like the publisher DID).

So, as with the grouping use case, it does not seem to me that the
cross-service use case justifies the non-NULL requirement.

> What would this mean provided it is part of the mandatory keywords of
> ObsTAP?

I think nothing -- there are quite a few columns in ivoa.obscore that
are rather regularly NULL.

           -- Markus