Support for data containing NaN values

[Randy Thompson]

Thanks for the replies. The question initially came up in testing a
VAO tool which will convert various input file formats to "compliant"
VOTable or FITS files. If NaNs are included in the input files the
values are passed unchanged. It sounds like this does not violate
any VO standards, and other VO applications should be expected to
handle them in data contained in either FITS or VOTable format.

We are also archiving FITS files from a project using -inf values
to flag bad data points and wondered if any further processing would
be required before these files would be considered "VO compliant".

Randy


On 9/26/11 8:33 AM, "Tom McGlynn" <Thomas.A.McGlynn at nasa.gov> wrote:

>I'm not sure that Mark's comments really addresses the Randy's
>question.  Suppose we have an original dataset, O, in some non-VO
>format and a VO serialization of this dataset, V.  Both O and V may
>contain NaNs.   As Mark points out a NaN in V is the recommended
>representation for a null value. So in any context in which null
>values are distinct from NaNs, VOTables cannot distinguish them, i.e.,
>a NaN in the VOTable does not in general mean that there was a NaN in
>the original data.  So if you wish to preserve NaNs VOTables are not
>currently a safe way to do so.
>
>As alluded to, this particular issue has been discussed before and
>some thoughts have been collected at
>http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues.
>
>	Tom
>
>Mark Taylor wrote:
>> Mike,
>>
>> in fact VOTable does permit NaN values in (single or double) floating
>> point data; I think this is a consequence of the design decision
>> that the VOTable model for data is to be as close as possible to
>> that of FITS.  Furthermore, NaN is how VOTable recommends to represent
>> NULL values in floating point data (again, following FITS) - whether
>> that's a good idea or not is a question that has been debated elsewhere,
>> but that's what VOTable section 6 says
>> 
>>(http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#ToC4
>>1)
>>
>> Mark
>>
>> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote:
>>
>>> I took the question differently:  If VO allows FITS  data, and
>>> FITS data allows NaN, then apps should of course allow
>>> for this.  OTOH, if the question is whether "VO data" as is
>>> serialized in a VOTable allows NaN values the I think the
>>> answer is 'no' (but I'd have to check).  There are similar issues
>>> with how NULL values are handled, but again it depends on
>>> whether it is in the serialized VOTable or the end data product
>>> being accessed.  The DAL protocols don't say anything about
>>> NaN/NULL beyond how they might be serialized, FITS is FITS
>>> and if that's what the app retrieves in the end and then that is
>>> the standard to follow when interpreting the data.  Was that
>>> your question?
>>>
>>> My $0.02,
>>> -Mike
>>>
>>>
>>>
>>> On Mon, Sep 26, 2011 at 2:06 AM, Mark
>>>Taylor<m.b.taylor at bristol.ac.uk>wrote:
>>>
>>>> On Thu, 22 Sep 2011, Randy Thompson wrote:
>>>>
>>>>> As a general question, does data containing NaN values
>>>>> violate any VO standards or protocols,and if not, should VO
>>>>> applications be expected to accept them as input?
>>>>
>>>> the question is rather broad ("data" can take many forms), but on
>>>> the whole the answer is that most standards and software in the VO
>>>> should and do behave sensibly in the presence of NaN-valued floating
>>>> point values.
>>>>
>>>> --
>>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University,
>>>>UK
>>>> m.b.taylor at bris.ac.uk +44-117-928-8776
>>>>http://www.star.bris.ac.uk/~mbt/
>>>>
>>>
>>
>> --
>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/
>