[meas] RFC comment - MD #8,9,10,11

[CresitelloDittmar, Mark]

On Mon, Sep 30, 2019 at 8:18 AM Markus Demleitner <
msdemlei at ari.uni-heidelberg.de> wrote:

> Hi DM,
>
> On Sat, Sep 21, 2019 at 10:44:15AM -0400, CresitelloDittmar, Mark wrote:
> > 8) would like to remove Bounds?D, Ellipse, Ellipsoid, and
> CovarianceMatrix
> > <
> https://wiki.ivoa.net/twiki/bin/edit/IVOA/CovarianceMatrix?topicparent=IVOA.MeasRFC;nowysiwyg=0
> >
> >
> > These are in the model as basic forms of multi-dimensional error
> > types which do occur in our data.  I admit that the specific
> > products that I'm working with do use them, so they could be
>
> should that be "do *not* use them"?
>

Yes.. that should be 'do not use them', but the example suite has covered
them all.

<snip>


> > If the CovarianceMatrix representation is wrong.. it should be
> > corrected or removed.  This isn't really in my wheelhouse, but I
> > thought I looked into it enough to have it properly represented.
> > If not, I'd need someone to provide the corrected model.
>
> "Wrong" is a harsh word -- you're just storing quite a few values
> twice (25% in 2x2, 33% in 3x3, asymptotically 50% as the matrix size
> grows).  As I said, you could fix that by just keeping the upper
> right triangle of the matrices, as in:


> SymmetricMatrix2x2:
>   m11
>   m12
>   m22
>
> SymmetricMatrix3x3:
>   m11
>   m12
>   m13
>   m22
>   m23
>   m33
>

I'd like to point out that this option is an additive change to the model
improving something which is 'inefficient'.  ie: the model can work, as is,
but the CovarianceMatrix could be improved with refined elements.


> -- but frankly, I'd still consider this a rather cumbersome notation,
> in particular since we'd have to define one class per vector space
> dimension (and at least 6D -- space and derivatives -- seem rather
> obvious to me).
>

As a DataType, the matrix could also be modeled as a more general 'cell'
array
CovarianceMatrix3x3
  o cell:MatrixCell[9]

MatrixCell
  o m: integer
  o n: integer
  o value: real

which kind-of makes it easier to define various dimensionalities.  But, as
a DataType, it cannot be open (or really shouldn't be).
CovarianceMatrix
  o M
  o N
  o cell: MatrixCell[*]
which is how the ObjectType Matrix is modeled in the Transform model.


This extra notation is particularly annoying because we already have
> a way to express all sorts of arrays: VOTable's PARAM/@arraysize and
> PARAM/@value (ok, I give you that for strings that... has room for
> improvement).  Hence, I'd frankly say: "how the matrix is described
> (in terms of its size and content) is up to the serialisation
> format."  And then put any constraints (e.g., "a covariance matrix
> must be a symmetric matrix with as many rows as the vector it is an
> error for") into human-readable text.  Such things aren't easily
> expressible in modelling languages, but we shouldn't uglify our
> models too far in order to coax validators into validating things
> beyond their complexity class.
>
> > could model it as a list of cells, but would still need 2 types with
> fixed
> > lengths (#cells).  That is an easy switch, but more bulky in direct
> > serializations.
>
> Well, I, for one, am aiming for VOTable, and I'd very much hope that
> an array would be a VOTable array then (similar concerns apply for
> FITS tables).
>
> Questions like these are why I'm so sure we need to wait for the
> actual serialisation rules.  I do expect that people will only care
> if they see that, right now, they'd probably write 9 (ok, 6,
> exploiting the symmetry) params with humonguous labels rather than
> something like
>
>   <PARAM name="covMat" datatype="double" arraysize="3x3" value=
>     "0.1  0    0.01
>     0     0.2  0.5
>     0.01  0.5  0.3"/>
>
> (say).
>

IMO this could, maybe should, be fine.  Annotate this Param as a
CovarianceMatrix3x3
and your done.  This puts the responsibility of packing/unpacking the array
properly on the
client/provider though.  And the way they know HOW to pack/unpack is by
looking at the
model and seeing the order in which the cells are spec'd.

This is entirely a serialization/annotation point.


> 10) points out that Gaussian, and other distributions, are missing, but
> > also seems to indicate that we don't want to get into these at this
> point.
> >
> > So, we're not adding distributions yet.. please.
>
> But I'd say we have to say that we're aware of this deficiency and
> say something about the limitations ensuing from that.  It would also
> make the model more useful if we said in Symmetrical's description
> something like "Annotators should see that value-radius ..
> value+radius covers about 70% of the value's distribution (`1
> sigma')" and analogously in Asymmetrical "The interval value-minus ..
> value+plus should cover about 70% of the value's distribution."
>

I'm happy to add clarification text to any of the descriptions, but I'd
rather keep it very open..  "typically a 1-sigma level distribution", kind
of thing.. specific numbers vary depending on usage.  If confidence
is an important concept here, which seems to be the case, then we
should define a case which includes it and incorporate it into the model
(1.1)

<snip>


> > Anyway.. if the correlation isn't done right in this model, it is
> probably
> > best to skip it until we have a concrete example to work.  Perhaps with
> the
> > catalogue/Source properties thread which is getting input from Gaia (I
> > think).
>
> Right -- which is why I'd so much like to see that as a use case.
> In short, that would be:
>
>   The Gaia satellite observes ra, dec, parallax, rmra, pmdec, and
>   radial velocity, and photometry for a large number of stars.  The
>   reduction correlates the first five of these values.  A client
>   wants to work out the resulting covariance matrix without having to
>   know about the specific Gaia data model.
>
>
I'd like to see that too.
As I see the next stage, is to define such usage cases, which someone
has volunteered to implement, and provide feedback/new requirements to the
model.

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dm/attachments/20191001/2765bd1b/attachment.html>