[Radioig] Obscore extension for radio: review, implementation point (3)

[Markus Demleitner]

Dear DM, Dear Radio,

On Fri, Dec 22, 2023 at 04:11:53PM +0100, BONNAREL FRANCOIS wrote:
> The basic idea with the ObsCore extensions is the same than the one we
> enhanced by creating some Optional fields in the original ObsCore. Not
> forcing anything but ALLOWING to add details in order to better tackle some
> specific needs.

Uh... let's be *very* careful with language in the vicinity of
"allowing" and "optional".

If the obscore extensions are supposed to work for global discovery
(i.e., one query is executable on all compliant services), then all
fields people may write constraints against (and that means: by
default all of them) need to be mandatory for the ivoa.obs_radio
table.

Without such a requirement, an all-VO query would first have to work
out where some column is available and then decide whether to
re-write the query and drop some constraint or whether to skip the
service in question.  That would be painful for client writers and
mystifying for their users.

Ceterum censeo Optional Features Are A Bane.

So: either we have f_min/_max or we don't.

> parallel the Euro-VO Asterics project (+ESCAPE) held several sessions around
> this, the first thing we heard from many/many radio astronomers and
> potential users was :
>
> "Hahh, gosh .... Wavelengths !!!"

And right they were: We should have used energies, which not only
(like frequencies) are independent of the medium but also work for
massive messengers.

But we didn't, and we can't sensibly fix that by adding extra
columns.  I, for one, would totally be in favour of planning a
transition to energies over a few versions of obscore, but that's
nothing an extension could do.

> In practice I imagine many radio archives start by storing their metadata in
> frequencies and transform them into wavelengths using functions, views or
> whatever to be consistent with ObsCore.

For ingestion, I claim it really doesn't matter; the ingestion rules
are written once, and very quickly on top.

No, the question is: Can we make writing obscore queries more
pleasant across the electromagnetic spectrum?  I can see how
f_min/_max help a bit there, but it's just *a bit* (because everyone
still has their non-Hz native units, including wavelengths ("21 cm",
"submillimeter")).  And *to me* the massive denormalisation is too
high a price for what little it buys.

Anyway, if you *really* can't find it in yourself to simply drop the
two columns, at least say something like: "Non-NULL f_min MUST be
equal to c/em_min and f_max MUST be equal to c/em_max, with
c=299792458 m/s; implementations are advised to ensure this by using,
for instance, views."

But don't you agree that, written like this, this definitely looks
like a bad idea?  I notice in passing that in this way, it might be
that

  em_min between 1e-2 and 2e-2

is fast but the -- according to the above stipulation -- equivalent

  f_max between 14989622900.0 and 29979245800.0

is not. That's because the query planner may very well not be smart
enough to see it could use an index on e_min when the query it sees
after expanding the view statement is against 14989622900.0/em_min[1]

> Nobody is forced to use a radio extension. But if people are in the spirit
> of using it then this has to be easy and readable for them.

Well, but we certainly would like them to use it if the have radio
data, right?

           -- Markus

[1] I just tried on postgres 13, and "pmra*10 between 1000 and 1200"
indeed doesn't use an index on pmra; I vaguely remember that wasn't
just a technical problem, but there was something conceptual when
pmra may be NULL.