Support for data containing NaN values

[Douglas Tody]

A couple of points:

     o   The only place to consider using NaN is in data arrays; it
 	is not supported (currently at least) for any VO metadata or for
 	RDBMS mappings (common for tables containing metadata), as FO
 	notes.

     o   It is generally unwise to use NaN or Inf or -Inf etc. in data
 	arrays; this will likely cause problems with software.  This is
 	not really a VO issue; VO doesn't care so long as the data
 	format (FITS or whatever) supports it.  However it will cause
 	problems with general client software unless the software
 	intended to consume the data is very constrained.  A better
 	approach, both from reliability and efficiency issues, is to
 	carry along a separate mask to explicitly note the status of
 	individual pixels or data values (this could even be a valid
 	place to use STC!).  Then general s/w muddles along ok, but
 	more sophisticated software can support the nuances.

So my suggestion would be to use a predefined value a la FITS, which is
hopefully ignored by most naive processing software (but mapped to
whatever by more advanced software), and a mask if one really wants to
do it right.  Of course, we do not yet have fully defined standards for
the use of masks, e.g., for pixel/data value quality, uncertainties,
etc.

     - Doug


On Mon, 26 Sep 2011, Randy Thompson wrote:

> Thanks for the replies. The question initially came up in testing a
> VAO tool which will convert various input file formats to "compliant"
> VOTable or FITS files. If NaNs are included in the input files the
> values are passed unchanged. It sounds like this does not violate
> any VO standards, and other VO applications should be expected to
> handle them in data contained in either FITS or VOTable format.
>
> We are also archiving FITS files from a project using -inf values
> to flag bad data points and wondered if any further processing would
> be required before these files would be considered "VO compliant".
>
> Randy
>
>
> On 9/26/11 8:33 AM, "Tom McGlynn" <Thomas.A.McGlynn at nasa.gov> wrote:
>
>> I'm not sure that Mark's comments really addresses the Randy's
>> question.  Suppose we have an original dataset, O, in some non-VO
>> format and a VO serialization of this dataset, V.  Both O and V may
>> contain NaNs.   As Mark points out a NaN in V is the recommended
>> representation for a null value. So in any context in which null
>> values are distinct from NaNs, VOTables cannot distinguish them, i.e.,
>> a NaN in the VOTable does not in general mean that there was a NaN in
>> the original data.  So if you wish to preserve NaNs VOTables are not
>> currently a safe way to do so.
>>
>> As alluded to, this particular issue has been discussed before and
>> some thoughts have been collected at
>> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues.
>>
>> 	Tom
>>
>> Mark Taylor wrote:
>>> Mike,
>>>
>>> in fact VOTable does permit NaN values in (single or double) floating
>>> point data; I think this is a consequence of the design decision
>>> that the VOTable model for data is to be as close as possible to
>>> that of FITS.  Furthermore, NaN is how VOTable recommends to represent
>>> NULL values in floating point data (again, following FITS) - whether
>>> that's a good idea or not is a question that has been debated elsewhere,
>>> but that's what VOTable section 6 says
>>>
>>> (http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC4
>>> 1)
>>>
>>> Mark
>>>
>>> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote:
>>>
>>>> I took the question differently:  If VO allows FITS  data, and
>>>> FITS data allows NaN, then apps should of course allow
>>>> for this.  OTOH, if the question is whether "VO data" as is
>>>> serialized in a VOTable allows NaN values the I think the
>>>> answer is 'no' (but I'd have to check).  There are similar issues
>>>> with how NULL values are handled, but again it depends on
>>>> whether it is in the serialized VOTable or the end data product
>>>> being accessed.  The DAL protocols don't say anything about
>>>> NaN/NULL beyond how they might be serialized, FITS is FITS
>>>> and if that's what the app retrieves in the end and then that is
>>>> the standard to follow when interpreting the data.  Was that
>>>> your question?
>>>>
>>>> My $0.02,
>>>> -Mike
>>>>
>>>>
>>>>
>>>> On Mon, Sep 26, 2011 at 2:06 AM, Mark
>>>> Taylor<m.b.taylor at bristol.ac.uk>wrote:
>>>>
>>>>> On Thu, 22 Sep 2011, Randy Thompson wrote:
>>>>>
>>>>>> As a general question, does data containing NaN values
>>>>>> violate any VO standards or protocols,and if not, should VO
>>>>>> applications be expected to accept them as input?
>>>>>
>>>>> the question is rather broad ("data" can take many forms), but on
>>>>> the whole the answer is that most standards and software in the VO
>>>>> should and do behave sensibly in the presence of NaN-valued floating
>>>>> point values.
>>>>>
>>>>> --
>>>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University,
>>>>> UK
>>>>> m.b.taylor at bris.ac.uk +44-117-928-8776
>>>>> http://www.star.bris.ac.uk/~mbt/
>>>>>
>>>>
>>>
>>> --
>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/
>>
>
>