[Ops] Obscore: obs_id not null requirement

[Mireille LOUYS]

Hi all ,
We also have another use case where /obs_id/ represents the observation 
identifier from which a data product comes from.

when applying Obscore to discover radio visibility data , we provide 
access to visibility chunks , having a unique /obs_publisher_did/
and they share the same /obs_id/ .
it is a way to identify when a set of data products share the same 
'progenitor' observation, even if this one is not accessible by the service.

suppose 2 different service curators expose HST reprocessed data :
they would have different /obs_publisher_did/ , minted by each curator , 
but could refer to the same /obs_id/ in the data provider collection.

when the ObsCore spec was discussed , we had in mind this would be easy 
to fill in the /obs_id/  in any case
, because the data provider can always use the same id for both fields : 
/obs_publisher_did /, and /obs_id/ for the first round of published data.

I do not see today how I could interpret obs_id = NULL unambiguously :
- missing data ? I don't know the origin of this data ?
- primary data ? but we have calib_level to mention it more precisely , 
etc ...

What would this mean provided it is part of the mandatory keywords of 
ObsTAP?

my 2 cents,
best wishes , Mireille



Le 07/03/2022 à 21:41, Patrick Dowler a écrit :
>
> ObsCore can have multiple result rows from the same "observation", eg 
> with different calib_level, and iirc the purpose of obs_id is that 
> that set of rows would have the same obs_id and thus users would know 
> they are different "products" of the same observation (aka same 
> photons messengers).
>
> In practice, obs_id could be null if that were not the case, eg if 
> every record was a different observation. I expect it could be OK for 
> some records to have null and others have a value that occurred 1+ 
> times in the table even within a single OsbCore as obs_id is a 
> specific kind of "part of this group of records" and null(s) would 
> mean "not part of a group".
>
> Having said that, this has a clear meaning to me because our ObsCore 
> impl is a view on CAOM and CAOM is normalised wrt. this obs_id concept 
> (where ObsCore is denormalised), but I expect it is not so simple to 
> specify use of nulls here and clearly deal with other variations of 
> underlying models.
>
> --
> Patrick Dowler
> Canadian Astronomy Data Centre
> Victoria, BC, Canada
>
>
> On Mon, 7 Mar 2022 at 09:56, Markus Demleitner 
> <msdemlei at ari.uni-heidelberg.de> wrote:
>
>     Dear DAL, dear Ops,
>
>     [Ops: I'm cc-ing you because I'm linking to a praise of validation,
>     while the thread leading up to this is perhaps not very relevant to
>     you; discussion on obs_id should probably happen on DAL exclusively]
>
>     On Fri, Mar 04, 2022 at 11:43:29AM +0000, Mark Taylor wrote:
>     > I'm not generally in favour of MUST rather than SHOULD constraints
>     > for non-NULL column values, on the grounds that data producers
>     > who don't have sensible values to put in there may end up giving
>     > useless or nonsense values just to comply with the standards,
>     > which doesn't provide value to anybody.
>     >
>     > However, it may be that the best answer to Markus's problem is
>     > for me to drop that taplint query, which could be reasonable.
>     > The intention is that taplint doesn't place unreasonable demands
>     > on the services that it validates, and I'm happy to take advice
>     > in cases where that's not happening.
>
>     When I tried to compose an answer to this this morning, I quickly
>     realised it was turning into a philosophical piece.  Which I then
>     turned into a blog post this afternoon:
>     <https://blog.g-vo.org/requirements-and-validators.html>
>
>     The TL;DR would be: Thanks, but no thanks.
>
>     Let's figure out whether we really have to require obs_id to be
>     non-NULL.  If we find a reason for that, I'll bite the bullet and
>     make the taplint query fast.  If we don't, let's drop the requirement
>     (I volunteer for writing an erratum, which should be quick and easy
>     after we've done the thinking here).
>
>     Thanks,
>
>                  Markus
>

-- 
--
Mireille Louys,  MCF (Associate Professor)
Centre de données CDS		IPSEO, Images, Laboratoire Icube
Observatoire de Strasbourg	Telecom Physique Strasbourg
11 rue de l'Université		300, Bd Sebastien Brandt CS 10413
F- 67000-STRASBOURG		F-67412 ILLKIRCH Cedex
Tel: +33 3 68 85 24 34
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/ops/attachments/20220308/33c24107/attachment.html>