<div dir="ltr"><div><div><div><div><div><div>All,<br><br></div>I do hope we get to spend some quality time on this topic soon (in Sydney?).<br></div>I find myself agreeing with different parts of different people's statements!<br><br></div>+ I see Arnold's point about the event list being pixelated.<br></div><div> I've been thinking along the same lines in the past couple days, so my <br> earlier classification of the event list in the 'non-pixelated' group may have<br> been in error. <br> The events are measured by a detector with finite pixel size and integration time,<br></div><div> and these are, at least for Chandra, recorded in the event list (ccd_id, chipx, chipy)<br></div><br></div><div>+ But I not convinced that #events is the proper value for any <domain>_dim field in ObsCore1.1<br><br></div><div>+ The element/descriptions of the ObsCore1.1 *_dim fields are directed at the <br></div><div> regularly sampled pixel array (Simple Image array) use case. My earlier comments was <br></div><div> simply that the document should be specific about the values to place provide for this case<br></div><div> AND what values to put for every other case (which I grouped as non-pixelated).<br></div><div> So, the language may need to be tweaked, but this statement should be true.<br><br></div><div>+ We will need to have a review of how these fields should be populated for each flavor <br></div><div> of image/cube covered by the NDCube model.<br></div><div><br>Mark<br><br></div><div><br></div><div><br><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 9, 2015 at 2:14 PM, Arnold Rots <span dir="ltr"><<a href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div>I beg to differ.<br></div>Pixels (or voxels) are coordinate bins for which we provide the value of some dependent variable.<br></div>That is precisely what we do for photon events, too, except that we enumerate all the pixels for<br></div>which we have a measurement, rather than provide the dependent variable value in an array<br></div>where the position in the array implies what the coordinate bins are. And that is another way<br></div>of describing what I called a sparse cube.<br><br></div>For instance, for Chandra data we provide the time stamp of the time bin (which does have a<br></div>definite finite size), the CCD pixel coordinates, and the quantized energy of the photons.<br></div>To me, that is a perfect example of a list of enumerated pixels.<br></div>Whenever we measure things and express them digitally, we are effectively binning, either<br></div>because we truncate, or because we round the value. And that means that we are projecting<br></div>our information on a pixel grid.<br><br></div>Let me be clear: I am not trying to make things more complicated; rather, I prefer to define<br></div>our concepts in such a way that in a generalization as much as possible is covered.<br></div>That does not mean that all possible special cases need to be implemented, but that we<br></div>keep the top level definition in mind when we implement, so that we won't find ourselves<br></div>later in a situation where we either can't accommodate an important use case, or have to<br></div>go to extreme lengths to fit a round peg into a square hole.<br><br></div>Cheers,<br><br></div> - Arnold<br></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><span class="">-------------------------------------------------------------------------------------------------------------<br>Arnold H. Rots Chandra X-ray Science Center<br>Smithsonian Astrophysical Observatory tel: <a href="tel:%2B1%20617%20496%207701" value="+16174967701" target="_blank">+1 617 496 7701</a><br>60 Garden Street, MS 67 fax: <a href="tel:%2B1%20617%20495%207356" value="+16174957356" target="_blank">+1 617 495 7356</a><br>Cambridge, MA 02138 <a href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a><br></span>USA <a href="http://hea-www.harvard.edu/~arots/" target="_blank">http://hea-www.harvard.edu/~arots/</a><br>--------------------------------------------------------------------------------------------------------------<br><br></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Fri, Oct 9, 2015 at 6:40 AM, François Bonnarel <span dir="ltr"><<a href="mailto:francois.bonnarel@astro.unistra.fr" target="_blank">francois.bonnarel@astro.unistra.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
The discussion to decide if an "event list" is a sparse cube because it is actually also a set of measurements or something radically different because it is made of photons is subtle.<br>
In any case, i think at the discovery purpose it is not really easy and usefull to code the number of events in one of the "dim" columns we have.<br>
Adding a new one to do so ? Or waiting for a model to describe this. I clearly prefer the second solution<br>
So "event list" in the dataproduct-type and NULL (or some other undetermined value) in the dim fields should be enough at this stage<br>
Cheers<span><font color="#888888"><br>
François</font></span><div><div><br>
Le 08/10/2015 20:53, Louys Mireille a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear Tom, dear list members,<br>
<br>
I think I also support your point of view.<br>
<br>
Pixels or let say numerical elements along one characterized axis<br>
of a science ready data product are the result of a sampling step in the detectors, followed eventually by a projection in a<br>
specified coordinate system.<br>
An event , as far as is the physical phenomenon that is converted/projected into a grid element.<br>
<br>
There are scientific features needed for a safe and rich interpretation of the content of data set, that are not covered in Obscore.<br>
for instance :<br>
- the signal to noise ratio in flux,<br>
- the number on non null channels in a spectra<br>
- the detection limit for a source in an image<br>
- correlation factors between spaxels in a cube ,<br>
- the proportion of good time intervals in a CTA observation,<br>
- the spatial density of detected sources in an image or cube ( crowding factor- probably there should be a better english name for that, sorry )<br>
etc<br>
...<br>
all this information is closer to the astronomers concerns for advanced dataset selection, but is context dependent.<br>
I think the number of events in an event list belongs to this category.<br>
<br>
This could be interesting to explore further use-cases for these specialized physical features about the data content.<br>
<br>
ObsCore should keep its general properties for global discovery , and CubeDM could tackle the 'interpretation profile' for various kinds of observation.<br>
<br>
Best regards , Mireille<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Le 08/10/2015 19:45, Tom McGlynn (NASA/GSFC Code 660.1) a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I confess that I find this a bit weird. E.g., consider a telescope with several coaligned photon-counting detectors (so multiple<br>
detectors are looking at the same or overlapping fields of view).<br>
<br>
After I combine detectors I can have sets of multiple photons with identical energy, time, and position. The photons may have been detected in different detectors originally, but the scientist has thrown that information away as irrelevant. The whole idea of a grid to me is that we only see each grid element once, but now we have the same grid element showing up multiple times. To me a grid value intrinsically<br>
represents some kind of integration over the grid area/volume. Events typically represent discrete photons. While there may be errors associated with the measurement of the characteristics of the photon (and Heisenberg suggests that there will be!) a photon is a different class of beast than a grid element. It's sort of like saying that when we are taking a census of people we should treat them as a grid of times and places of birth. We could do so but seeing the grid as the underlying structure seems wrong...<br>
<br>
Regards,<br>
Tom McGlyn<br>
<br>
Laurent Michel wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Arnold<br>
<br>
<br>
I agree with your remark. Event lists can be classified sparse cube. However, we do not have in Obscore a column giving a the product class (pixelated or not, sparse cube). We just have a product_type column with a limited vocabulary. There is no field giving explicitly the product class which could help to interpret correctly the content of the S-T-E axes.<br>
<br>
For now the point is to decide whether I) it's better to complete Obscore 1.1 with just the *dim and the extended list of product_type or II) whether we have to target a better description of these new categories of data products (new in term of Obscore) .<br>
In my opinion, the option I) is safer until the event lists are better taken into account and modelled in the VO.<br>
<br>
Bye<br>
Laurent<br>
<br>
Le 05/10/2015 18:48, Arnold Rots a écrit :<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I don't think event lists should be classified as /non-pixelated/.<br>
If we want to model data consistently, they should be classified as /pixel lists/:<br>
each event is a pixel; we just end up with a sparse hypercube.<br>
If we were to add one more parameter: the total number of pixels in<br>
the data product, that would solve the problem. For solid cubes it<br>
would be the product of the *dims, for event lists the number of events.<br>
<br>
- Arnold<br>
<br>
------------------------------------------------------------------------------------------------------------- <br>
Arnold H. Rots Chandra X-ray Science Center<br>
Smithsonian Astrophysical Observatory tel: <a href="tel:%2B1%20617%20496%207701" value="+16174967701" target="_blank">+1 617 496 7701</a><br>
60 Garden Street, MS 67 fax: <a href="tel:%2B1%20617%20495%207356" value="+16174957356" target="_blank">+1 617 495 7356</a><br>
Cambridge, MA 02138 <a href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a> <mailto:<a href="mailto:arots@cfa.harvard.edu" target="_blank">arots@cfa.harvard.edu</a>><br>
USA <a href="http://hea-www.harvard.edu/~arots/" rel="noreferrer" target="_blank">http://hea-www.harvard.edu/~arots/</a><br>
-------------------------------------------------------------------------------------------------------------- <br>
<br>
<br>
On Mon, Oct 5, 2015 at 11:00 AM, Laurent Michel <<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a> <mailto:<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a>>> wrote:<br>
<br>
Hello,<br>
<br>
A consensus seems to be taking shape about the model evolution for pixelated data:<br>
+ Adding several fields to provide the dimensionality in each domain. (s_dim, em_dim, etc) as described in the draft<br>
+ These are added as 'required' fields.. so must be populated for all records<br>
+ These fields provide for instance the size of each image axis.<br>
<br>
These new fields can however not be used to provide a robust representation the dimension of non pixelated data (e.g. number<br>
of event in an event list)<br>
<br>
The proposal to complete the version would be:<br>
1) To keep the new *dim* fields<br>
2) To support the EventList and other non pixelated as product_type<br>
3) To clearly specify that the *dim* columns must be set with NaN for non pixelated data.<br>
4) To provide some use cases about discovering non pixelated data with ObsTap<br>
<br>
That would quiclky allow us to start the RFC period for ObsCore without having to wait on a agreement about how to model non<br>
pixelated data<br>
<br>
<br>
Laurent<br>
<br>
--<br>
jesuischarlie<br>
<br>
Laurent Michel<br>
SSC XMM-Newton<br>
Tél : <a href="tel:%2B33%20%280%293%2068%2085%2024%2037" value="+33368852437" target="_blank">+33 (0)3 68 85 24 37</a> <tel:%2B33%20%280%293%2068%2085%2024%2037><br>
Fax : +33 (0)3 )3 68 85 24 32<br>
<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a> <mailto:<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a>> <mailto:<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a><br>
<mailto:<a href="mailto:laurent.michel@astro.unistra.fr" target="_blank">laurent.michel@astro.unistra.fr</a>>><br>
Université de Strasbourg <<a href="http://www.unistra.fr" rel="noreferrer" target="_blank">http://www.unistra.fr</a>><br>
Observatoire Astronomique<br>
11 Rue de l'Université<br>
F - 67200 Strasbourg<br>
<a href="http://amwdb.u-strasbg.fr/HighEnergy/spip.php?rubrique34" rel="noreferrer" target="_blank">http://amwdb.u-strasbg.fr/HighEnergy/spip.php?rubrique34</a><br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
</blockquote>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>