Support for data containing NaN values

[Mark Taylor]

Regarding -inf, I don't think that there is any special VO compliance
issue here, but I would question the idea of using -inf to flag bad
values, unless perhaps the data is such that absence of data can be
presumed to imply a large negative value.  While -inf is better than,
say, 9999, it is likely to cause problems when processing data to
do something like assess its range or calculate statistics.  
In the absence of a way of flagging a true null value, NaN would
seem like a less problematic choice.

Just my $0.02.

Mark

On Mon, 26 Sep 2011, Randy Thompson wrote:

> Thanks for the replies. The question initially came up in testing a
> VAO tool which will convert various input file formats to "compliant"
> VOTable or FITS files. If NaNs are included in the input files the
> values are passed unchanged. It sounds like this does not violate
> any VO standards, and other VO applications should be expected to
> handle them in data contained in either FITS or VOTable format.
> 
> We are also archiving FITS files from a project using -inf values
> to flag bad data points and wondered if any further processing would
> be required before these files would be considered "VO compliant".
> 
> Randy
> 
> 
> On 9/26/11 8:33 AM, "Tom McGlynn" <Thomas.A.McGlynn at nasa.gov> wrote:
> 
> >I'm not sure that Mark's comments really addresses the Randy's
> >question.  Suppose we have an original dataset, O, in some non-VO
> >format and a VO serialization of this dataset, V.  Both O and V may
> >contain NaNs.   As Mark points out a NaN in V is the recommended
> >representation for a null value. So in any context in which null
> >values are distinct from NaNs, VOTables cannot distinguish them, i.e.,
> >a NaN in the VOTable does not in general mean that there was a NaN in
> >the original data.  So if you wish to preserve NaNs VOTables are not
> >currently a safe way to do so.
> >
> >As alluded to, this particular issue has been discussed before and
> >some thoughts have been collected at
> >http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues.
> >
> >	Tom
> >
> >Mark Taylor wrote:
> >> Mike,
> >>
> >> in fact VOTable does permit NaN values in (single or double) floating
> >> point data; I think this is a consequence of the design decision
> >> that the VOTable model for data is to be as close as possible to
> >> that of FITS.  Furthermore, NaN is how VOTable recommends to represent
> >> NULL values in floating point data (again, following FITS) - whether
> >> that's a good idea or not is a question that has been debated elsewhere,
> >> but that's what VOTable section 6 says
> >> 
> >>(http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC4
> >>1)
> >>
> >> Mark
> >>
> >> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote:
> >>
> >>> I took the question differently:  If VO allows FITS  data, and
> >>> FITS data allows NaN, then apps should of course allow
> >>> for this.  OTOH, if the question is whether "VO data" as is
> >>> serialized in a VOTable allows NaN values the I think the
> >>> answer is 'no' (but I'd have to check).  There are similar issues
> >>> with how NULL values are handled, but again it depends on
> >>> whether it is in the serialized VOTable or the end data product
> >>> being accessed.  The DAL protocols don't say anything about
> >>> NaN/NULL beyond how they might be serialized, FITS is FITS
> >>> and if that's what the app retrieves in the end and then that is
> >>> the standard to follow when interpreting the data.  Was that
> >>> your question?
> >>>
> >>> My $0.02,
> >>> -Mike
> >>>
> >>>
> >>>
> >>> On Mon, Sep 26, 2011 at 2:06 AM, Mark
> >>>Taylor<m.b.taylor at bristol.ac.uk>wrote:
> >>>
> >>>> On Thu, 22 Sep 2011, Randy Thompson wrote:
> >>>>
> >>>>> As a general question, does data containing NaN values
> >>>>> violate any VO standards or protocols,and if not, should VO
> >>>>> applications be expected to accept them as input?
> >>>>
> >>>> the question is rather broad ("data" can take many forms), but on
> >>>> the whole the answer is that most standards and software in the VO
> >>>> should and do behave sensibly in the presence of NaN-valued floating
> >>>> point values.
> >>>>
> >>>> --
> >>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University,
> >>>>UK
> >>>> m.b.taylor at bris.ac.uk +44-117-928-8776
> >>>>http://www.star.bris.ac.uk/~mbt/
> >>>>
> >>>
> >>
> >> --
> >> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> >> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/
> >
> 
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/