SDM evolution roadmap

[Markus Demleitner]

Hi Mark,

On Mon, Mar 20, 2023 at 02:22:10PM -0400, CresitelloDittmar, Mark wrote:
> The UTypes and groupings look good for the most part.. a couple notes:
>       ** secondary comment that it'd be easier to read overall if the
> PARAMref-s had the UTYPEs, but the PARAMs did not.  But they did help me
> find the match!.

Well, I'll do whatever you (or preferably the spec) tell me :-).
It's easy to remove the utypes from the PARAMrefs.  Should I?

>    o a few PARAMs with UTypes are not included in their respective GROUP.
>       - "spec:Spectrum.Target.Class"
>       - "spec:Spectrum.Char.FluxAxis.Ucd"
>       - "spec:Spectrum.Curation.Publisher"

These are all easy, and I've moved them.

>       - "spec:Spectrum.Data.SpectralAxis.order"

This one is hard, because I can't (easily) put it in the general
SDM-making code.  How important is it to have it in the groups?  Is
this a validity issue?

>   o two errors
>       - "spec:Spectrum.Target.Redshift" is in Target GROUP, but also in
> Derived GROUP.
>         These are different (model-wise), and the Derived value would have
> UType "spec:Spectrum.Derived.Redshift.Value"

Ah.  Yes, that makes sense.  Dropped it from Derived.

>       - "spec:Spectrum.Length" has no value, but the DATA binary stream
> definitely contains rows.

As long as it's a NULL (rather than a 0 or some other concrete
length): is that a validity issue?  This is a bit of a special case
in the echelle spectra.  You see, normally these PARAMs are filled
from SSA metadata in DaCHS, but for the orders I don't have the
length in there, which is why this is NULL.

I could do some special and extra code, but that's unattractive as
long as nobody bothers to look there anyway (and I think clients
bothering can just look into the data stream, somewhat more easily
than I can, even).

So, as long as I'm not lying and claim a bad length: is that really an error?

> Other comments:
>   o most of the PARAMs have no value... so while they are there.. they
> aren't terribly useful.

Yeah; I'd suppress them if I then didn't have to tear down the
references, too.  Morale: If you're doing things like these groups,
have a strong reason; references aren't free at all.

>   o UCD discrepancies (no comment on legality/correctness, just differences
> from the spec.)
>      - "spec:Spectrum.Curation.PublisherDID":  missing specified UCD

Well, the UCD given in SpectrumDM is UCD-invalid. Another case for my
point that we should stop requiring specific UCDs for columns and
params in standards.

I now went for meta.ref.ivoid from
<https://wiki.ivoa.net/twiki/bin/view/IVOA/ObsCore-1_1-Erratum-1>;
this could be a good opportunity to update this UCD in SpectrumDM,
too (or, even better, make it explicit that the UCDs given in the
example are good patterns but not normative).

>      - "spec:Spectrum.Char.SpatialAxis.Coverage.Bounds.Extent":
> spec="instr.fov"  file="phys.angSize;instr.fov"

Ah... instr.fov is a again invalid.  See
<https://wiki.ivoa.net/twiki/bin/view/IVOA/SSA-1_1-Err-2>.  I'll not
make this UCD-invalid -- perhaps update the spec?

>      - "spec:Spectrum.Length": missing UCD

But that's the case in the spec, too?  And putting meta.number here
or so indeed doesn't seem helpful.

>      - "spec:Spectrum.Char.FluxAxis.Accuracy.StatError": UCD is missing a
> bit at end. spec has "em.*" which would presumably be "em.wl" in this
> dataset but file just has "em"

The ";em" is fine *UCD-wise*.  Whether it's helpful is another
question, and given the "must be one of" on p. 15 of the 1.1
document, I'm in violation of the current spec here (I frankly don't
remember whether I did that on purpose all these years ago).

It wouldn't be hard to fix this for this particular case, but in
DaCHS I'd need to parameterise it so it won't lie when people publish
spectra over frequency or energy, and that's an actual complication
*to my adopters* that I frankly would only like to resign into if
someone promises me they'll write code inspecting these UCDs one of
these days (and then look for wl, freq, or energy).

So, does someone?  If not, can we relax the "must" language?

Thanks for the review,

           Markus