<div dir="ltr"><div><div>My responses also in line.<br><br></div>Cheers,<br><br></div> - Arnold<br><div><div><div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-------------------------------------------------------------------------------------------------------------<br>Arnold H. Rots Chandra X-ray Science Center<br>Smithsonian Astrophysical Observatory tel: +1 617 496 7701<br>60 Garden Street, MS 67 fax: +1 617 495 7356<br>Cambridge, MA 02138 <a href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a><br>USA <a href="http://hea-www.harvard.edu/~arots/" target="_blank">http://hea-www.harvard.edu/~arots/</a><br>--------------------------------------------------------------------------------------------------------------<br><br></div></div></div>
<br><div class="gmail_quote">On Sun, Sep 3, 2017 at 8:48 AM, Jiří Nádvorník <span dir="ltr"><<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><p class="MsoNormal"><span>Hi Mark,<u></u><u></u></span></p><p class="MsoNormal"><span><u></u> <u></u></span></p><p class="MsoNormal"><span>Answers/thoughts inline.<u></u><u></u></span></p><p class="MsoNormal"><span><u></u> <u></u></span></p><p class="MsoNormal"><span>Cheers,<u></u><u></u></span></p><p class="MsoNormal"><span><u></u> <u></u></span></p><p class="MsoNormal"><span>Jiri<u></u><u></u></span></p><p class="MsoNormal"><span><u></u> <u></u></span></p><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b>From:</b> CresitelloDittmar, Mark [mailto:<a href="mailto:mdittmar@cfa.harvard.edu" target="_blank">mdittmar@cfa.harvard.<wbr>edu</a>] <br><b>Sent:</b> Thursday, August 31, 2017 10:22 PM<br><b>To:</b> Jiří Nádvorník <<a href="mailto:nadvornik.ji@gmail.com" target="_blank">nadvornik.ji@gmail.com</a>><br><b>Cc:</b> François Bonnarel <<a href="mailto:francois.bonnarel@astro.unistra.fr" target="_blank">francois.bonnarel@astro.<wbr>unistra.fr</a>>; Mireille Louys <<a href="mailto:mireille.louys@unistra.fr" target="_blank">mireille.louys@unistra.fr</a>>; Data Models mailing list <<a href="mailto:dm@ivoa.net" target="_blank">dm@ivoa.net</a>>; <a href="mailto:voevent@ivoa.net" target="_blank">voevent@ivoa.net</a>; <a href="mailto:dal@ivoa.net" target="_blank">dal@ivoa.net</a><span class=""><br><b>Subject:</b> Re: [timeDomain: model for Time series] discussion on Timeseries Data model Note / ND-point .... or not ??? !!!<u></u><u></u></span></p></div></div><p class="MsoNormal"><u></u> <u></u></p><div><div><div><div><div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt">All,<u></u><u></u></p></div><span class=""><p class="MsoNormal">I am finally able to devote some time to the VO efforts, and have just been reading through these discussions.<u></u><u></u></p></span></div><span class=""><p class="MsoNormal" style="margin-bottom:12.0pt">Maybe someone could summarize for me what the current state is. I would like to get back to iterating through example serializations as that seemed to make things more 'concrete' and provided important feedback about what is reasonable and usable.<u></u><u></u></p></span></div><span class=""><div><p class="MsoNormal">Anyway. My impression about the discussion...<u></u><u></u></p></div><div><p class="MsoNormal">1) NDPoint.<u></u><u></u></p></div><p class="MsoNormal" style="margin-bottom:12.0pt"> One of the differences between Jiri's approach in the Note, and the Cube model is that my model has this element which, as Francois noted, is primarily to associate a set of Observables as being related. The NDPoint collects data axes (time, pos, energy, mag ). Each instance of NDPoint is a set of values which describe a single 'event'. I believe the column-based approach looses this important relation. Recall, the primary use case for this is an Event List.<u></u><u></u></p></span><p class="MsoNormal" style="margin-bottom:12.0pt"><b><i>[[Jiri Nadvornik]] I’d just like to point out that the difference </i></b><b><i><span lang="EN-US">here is that in the TSCube model we collect the data axes in the SparseCube class directly (time, pos, energy, mag) because we don’t find useful to store axis metadata on the level of each Observable, but rather on the whole Dataset.</span></i></b></p></div></div></div></div></div></div></div></blockquote><div>I am not sure there is a difference here and it may actually be a serialization matter. Presumably, the individual items in the different arrays(?) in Jiri's case are associated with each other by index. In other words, it is a kind of "table" model of the data - which is, when serialized, indististinguishable from an STC serialization in a table, where the coordinate values are put in table columns and the axis metadata in the table header associated with the appropriate columns. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><u></u><u></u></span></p></div><p class="MsoNormal">2) Errors<u></u><u></u></p></div><span class=""><p class="MsoNormal"> The Observables in the cube model are STC CoordMeasure instances, which include the measured value, associated error, and frame reference. The error modeling to date is assuming a 1-1 association of the value to its error (eg Value +/- Error ). The modeling of errors in STC2 is very extensible and targets the most common error types. It is very compatible to have a separate effort take place to model other sorts of errors ( statistical distributions ) and extend the base STC-2 error class.<u></u><u></u></p></span><p class="MsoNormal"><b><i>[[Jiri Nadvornik]] I agree that the Error class in STC2 is very extensible, but I would go even one step further here because when we have „statistical distribution“ for Error, what’s the Value then? I’d suppose it is just some Expectation or Mean value, which is already part of that statistical distribution.</i></b></p></div></div></div></div></div></blockquote><div><br></div><div>You can do that in any way you want and that makes sense, since the Error class is an empty prototype. That means that you can derive an Error class that defines a certain type of distribution with appropriate parameters, or even an enumerated distributino function.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div><p class="MsoNormal"><u></u><u></u></p></div><div><span class=""><p class="MsoNormal"><br>If the relation is a bit more broad, and the 'measure' itself is a statistical distribution kind of thing, then what you are describing is not an Observable. This sort of thing I expected might come up with simulated data (when we think about bringing SimDM and Cube more in line with each other.. (they are quite compatible)). If this is the case, we we should define a different kind of DataAxis which better facilitates the statistical/mathematical nature of the measurement.<u></u><u></u></p></span><p class="MsoNormal"><b><i>[[Jiri Nadvornik]] I don’t agree completely on the first statement here – I think that even an Observable ‚measure‘ can be represented with a statistical distribution. Typically, a point spread function will have a gaussian distribution where the Flux Value+Error would be just Mean+Sigma of that statistical distribution.</i></b></p></div></div></div></div></div></blockquote><div><br></div><div>But here we are talking about Resolution, not Error/Uncertainty. Anyway, the same argument applies.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div><p class="MsoNormal"><b><i><u></u><u></u></i></b></p></div><div><div><div><div><span class=""><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">If the data can not be described as a series of 'events'<u></u><u></u></p></div><div><p class="MsoNormal"> "at time T1, we measured these items"<u></u><u></u></p></div><div><p class="MsoNormal"> "this source, has these proprerties"<u></u><u></u></p></div></span><div><span class=""><p class="MsoNormal" style="margin-bottom:12.0pt">then the data is not a SparseCube. I'm under the impression that most time series do fit this description, but if there are some which do not, we should evaluate an example to see if it needs a different sort of DataProduct.<u></u><u></u></p></span><p class="MsoNormal" style="margin-bottom:12.0pt"><b><i>[[Jiri Nadvornik]] I still think we can fit into this ‚event‘ description with all our use cases. I think in every case the Time Axis points can be actually represented by a single Value, rather than needing some kind of distribution here.</i></b></p></div></div></div></div></div></div></div></div></div></blockquote><div>No, any time stamp is characterized by the digital accuracy of the clock used (and usually represents a truncated value within a bin corresponding to the lowest clock bit), by the errors inherent in the clock (how stable, how correct is the long-term clock rate, including relativistic effects if necessary), and clock readout errors (was there a lag). So, at least conceptually, one cannot make the argument that just a time stamp provides sufficient information on absolute time.<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div><div><div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><b><i><u></u><u></u></i></b></p><p class="MsoNormal" style="margin-bottom:12.0pt"><b><i>Does anyone have a time series use case where we have uncertainty when exactly was the ‚measurement‘ taken?</i></b></p></div></div></div></div></div></div></div></div></div></blockquote><div>HEA event files contain two error measures for time (systematic, or long term, and short term), as well as the size of the time stamp bins (the accuracy of the clock). <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="CS"><div class="m_8888738718730781714WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div><div><div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><u></u><u></u></p></div><div><p class="MsoNormal">Mark<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p><div><div><p class="MsoNormal"><u></u><u></u></p></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></div></div></div></div>