<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 30, 2019 at 8:18 AM Markus Demleitner <<a href="mailto:msdemlei@ari.uni-heidelberg.de">msdemlei@ari.uni-heidelberg.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi DM,<br>
<br>
On Sat, Sep 21, 2019 at 10:44:15AM -0400, CresitelloDittmar, Mark wrote:<br>
> 8) would like to remove Bounds?D, Ellipse, Ellipsoid, and CovarianceMatrix<br>
> <<a href="https://wiki.ivoa.net/twiki/bin/edit/IVOA/CovarianceMatrix?topicparent=IVOA.MeasRFC;nowysiwyg=0" rel="noreferrer" target="_blank">https://wiki.ivoa.net/twiki/bin/edit/IVOA/CovarianceMatrix?topicparent=IVOA.MeasRFC;nowysiwyg=0</a>><br>
> <br>
> These are in the model as basic forms of multi-dimensional error<br>
> types which do occur in our data. I admit that the specific<br>
> products that I'm working with do use them, so they could be<br>
<br>
should that be "do *not* use them"?<br></blockquote><div><br></div><div>Yes.. that should be 'do not use them', but the example suite has covered them all.</div><div> </div><div><snip></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
> If the CovarianceMatrix representation is wrong.. it should be<br>
> corrected or removed. This isn't really in my wheelhouse, but I<br>
> thought I looked into it enough to have it properly represented.<br>
> If not, I'd need someone to provide the corrected model.<br>
<br>
"Wrong" is a harsh word -- you're just storing quite a few values<br>
twice (25% in 2x2, 33% in 3x3, asymptotically 50% as the matrix size<br>
grows). As I said, you could fix that by just keeping the upper<br>
right triangle of the matrices, as in:</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
SymmetricMatrix2x2:<br>
m11<br>
m12<br>
m22<br>
<br>
SymmetricMatrix3x3:<br>
m11<br>
m12<br>
m13<br>
m22<br>
m23<br>
m33<br></blockquote><div><br></div><div>I'd like to point out that this option is an additive change to the model<br></div><div>improving something which is 'inefficient'. ie: the model can work, as is,</div><div>but the CovarianceMatrix could be improved with refined elements.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">-- but frankly, I'd still consider this a rather cumbersome notation,<br>
in particular since we'd have to define one class per vector space<br>
dimension (and at least 6D -- space and derivatives -- seem rather<br>
obvious to me).<br></blockquote><div><br></div><div>As a DataType, the matrix could also be modeled as a more general 'cell' array</div><div>CovarianceMatrix3x3</div><div> o cell:MatrixCell[9]</div><div><br></div><div>MatrixCell</div><div> o m: integer</div><div> o n: integer</div><div> o value: real</div><div><br></div><div>which kind-of makes it easier to define various dimensionalities. But, as</div><div>a DataType, it cannot be open (or really shouldn't be).</div><div>CovarianceMatrix</div><div> o M</div><div> o N</div><div> o cell: MatrixCell[*] </div><div>which is how the ObjectType Matrix is modeled in the Transform model.</div><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">This extra notation is particularly annoying because we already have<br>
a way to express all sorts of arrays: VOTable's PARAM/@arraysize and<br>
PARAM/@value (ok, I give you that for strings that... has room for<br>
improvement). Hence, I'd frankly say: "how the matrix is described<br>
(in terms of its size and content) is up to the serialisation<br>
format." And then put any constraints (e.g., "a covariance matrix<br>
must be a symmetric matrix with as many rows as the vector it is an<br>
error for") into human-readable text. Such things aren't easily<br>
expressible in modelling languages, but we shouldn't uglify our<br>
models too far in order to coax validators into validating things<br>
beyond their complexity class.<br>
<br>
> could model it as a list of cells, but would still need 2 types with fixed<br>
> lengths (#cells). That is an easy switch, but more bulky in direct<br>
> serializations.<br>
<br>
Well, I, for one, am aiming for VOTable, and I'd very much hope that<br>
an array would be a VOTable array then (similar concerns apply for<br>
FITS tables).<br>
<br>
Questions like these are why I'm so sure we need to wait for the<br>
actual serialisation rules. I do expect that people will only care<br>
if they see that, right now, they'd probably write 9 (ok, 6,<br>
exploiting the symmetry) params with humonguous labels rather than<br>
something like<br>
<br>
<PARAM name="covMat" datatype="double" arraysize="3x3" value=<br>
"0.1 0 0.01 <br>
0 0.2 0.5<br>
0.01 0.5 0.3"/><br>
<br>
(say).<br></blockquote><div><br></div><div>IMO this could, maybe should, be fine. Annotate this Param as a CovarianceMatrix3x3</div><div>and your done. This puts the responsibility of packing/unpacking the array properly on the </div><div>client/provider though. And the way they know HOW to pack/unpack is by looking at the </div><div>model and seeing the order in which the cells are spec'd.</div><div><br></div><div>This is entirely a serialization/annotation point.</div><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">> 10) points out that Gaussian, and other distributions, are missing, but<br>
> also seems to indicate that we don't want to get into these at this point.<br>
> <br>
> So, we're not adding distributions yet.. please.<br>
<br>
But I'd say we have to say that we're aware of this deficiency and<br>
say something about the limitations ensuing from that. It would also<br>
make the model more useful if we said in Symmetrical's description<br>
something like "Annotators should see that value-radius ..<br>
value+radius covers about 70% of the value's distribution (`1<br>
sigma')" and analogously in Asymmetrical "The interval value-minus ..<br>
value+plus should cover about 70% of the value's distribution."<br></blockquote><div><br></div><div>I'm happy to add clarification text to any of the descriptions, but I'd </div><div>rather keep it very open.. "typically a 1-sigma level distribution", kind</div><div>of thing.. specific numbers vary depending on usage. If confidence </div><div>is an important concept here, which seems to be the case, then we </div><div>should define a case which includes it and incorporate it into the model (1.1)</div><div> </div><div><snip></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
> Anyway.. if the correlation isn't done right in this model, it is probably<br>
> best to skip it until we have a concrete example to work. Perhaps with the<br>
> catalogue/Source properties thread which is getting input from Gaia (I<br>
> think).<br>
<br>
Right -- which is why I'd so much like to see that as a use case.<br>
In short, that would be:<br>
<br>
The Gaia satellite observes ra, dec, parallax, rmra, pmdec, and<br>
radial velocity, and photometry for a large number of stars. The<br>
reduction correlates the first five of these values. A client<br>
wants to work out the resulting covariance matrix without having to<br>
know about the specific Gaia data model.<br>
<br></blockquote><div><br></div><div>I'd like to see that too.</div><div>As I see the next stage, is to define such usage cases, which someone</div><div>has volunteered to implement, and provide feedback/new requirements to the model.</div><div><br></div><div>Mark</div><div> </div></div></div>