Support for data containing NaN values

[Randy Thompson]

Hi Doug,
  Thanks for the reply. I agree the special IEEE data values are
best avoided and we can recommend projects do this, but if it doesn't
violate any standard we can only make recommendations.

Randy



On 10/2/11 6:58 PM, "Douglas Tody" <dtody at nrao.edu> wrote:

>A couple of points:
>
>     o   The only place to consider using NaN is in data arrays; it
> 	is not supported (currently at least) for any VO metadata or for
> 	RDBMS mappings (common for tables containing metadata), as FO
> 	notes.
>
>     o   It is generally unwise to use NaN or Inf or -Inf etc. in data
> 	arrays; this will likely cause problems with software.  This is
> 	not really a VO issue; VO doesn't care so long as the data
> 	format (FITS or whatever) supports it.  However it will cause
> 	problems with general client software unless the software
> 	intended to consume the data is very constrained.  A better
> 	approach, both from reliability and efficiency issues, is to
> 	carry along a separate mask to explicitly note the status of
> 	individual pixels or data values (this could even be a valid
> 	place to use STC!).  Then general s/w muddles along ok, but
> 	more sophisticated software can support the nuances.
>
>So my suggestion would be to use a predefined value a la FITS, which is
>hopefully ignored by most naive processing software (but mapped to
>whatever by more advanced software), and a mask if one really wants to
>do it right.  Of course, we do not yet have fully defined standards for
>the use of masks, e.g., for pixel/data value quality, uncertainties,
>etc.
>
>     - Doug
>
>
>On Mon, 26 Sep 2011, Randy Thompson wrote:
>
>> Thanks for the replies. The question initially came up in testing a
>> VAO tool which will convert various input file formats to "compliant"
>> VOTable or FITS files. If NaNs are included in the input files the
>> values are passed unchanged. It sounds like this does not violate
>> any VO standards, and other VO applications should be expected to
>> handle them in data contained in either FITS or VOTable format.
>>
>> We are also archiving FITS files from a project using -inf values
>> to flag bad data points and wondered if any further processing would
>> be required before these files would be considered "VO compliant".
>>
>> Randy
>>
>>
>> On 9/26/11 8:33 AM, "Tom McGlynn" <Thomas.A.McGlynn at nasa.gov> wrote:
>>
>>> I'm not sure that Mark's comments really addresses the Randy's
>>> question.  Suppose we have an original dataset, O, in some non-VO
>>> format and a VO serialization of this dataset, V.  Both O and V may
>>> contain NaNs.   As Mark points out a NaN in V is the recommended
>>> representation for a null value. So in any context in which null
>>> values are distinct from NaNs, VOTables cannot distinguish them, i.e.,
>>> a NaN in the VOTable does not in general mean that there was a NaN in
>>> the original data.  So if you wish to preserve NaNs VOTables are not
>>> currently a safe way to do so.
>>>
>>> As alluded to, this particular issue has been discussed before and
>>> some thoughts have been collected at
>>> http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/VOTableIssues.
>>>
>>> 	Tom
>>>
>>> Mark Taylor wrote:
>>>> Mike,
>>>>
>>>> in fact VOTable does permit NaN values in (single or double) floating
>>>> point data; I think this is a consequence of the design decision
>>>> that the VOTable model for data is to be as close as possible to
>>>> that of FITS.  Furthermore, NaN is how VOTable recommends to represent
>>>> NULL values in floating point data (again, following FITS) - whether
>>>> that's a good idea or not is a question that has been debated
>>>>elsewhere,
>>>> but that's what VOTable section 6 says
>>>>
>>>> 
>>>>(http://www.ivoa.net/Documents/VOTable/20091130/REC-VOTable-1.2.html#To
>>>>C4
>>>> 1)
>>>>
>>>> Mark
>>>>
>>>> On Mon, 26 Sep 2011, Mike Fitzpatrick wrote:
>>>>
>>>>> I took the question differently:  If VO allows FITS  data, and
>>>>> FITS data allows NaN, then apps should of course allow
>>>>> for this.  OTOH, if the question is whether "VO data" as is
>>>>> serialized in a VOTable allows NaN values the I think the
>>>>> answer is 'no' (but I'd have to check).  There are similar issues
>>>>> with how NULL values are handled, but again it depends on
>>>>> whether it is in the serialized VOTable or the end data product
>>>>> being accessed.  The DAL protocols don't say anything about
>>>>> NaN/NULL beyond how they might be serialized, FITS is FITS
>>>>> and if that's what the app retrieves in the end and then that is
>>>>> the standard to follow when interpreting the data.  Was that
>>>>> your question?
>>>>>
>>>>> My $0.02,
>>>>> -Mike
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 26, 2011 at 2:06 AM, Mark
>>>>> Taylor<m.b.taylor at bristol.ac.uk>wrote:
>>>>>
>>>>>> On Thu, 22 Sep 2011, Randy Thompson wrote:
>>>>>>
>>>>>>> As a general question, does data containing NaN values
>>>>>>> violate any VO standards or protocols,and if not, should VO
>>>>>>> applications be expected to accept them as input?
>>>>>>
>>>>>> the question is rather broad ("data" can take many forms), but on
>>>>>> the whole the answer is that most standards and software in the VO
>>>>>> should and do behave sensibly in the presence of NaN-valued floating
>>>>>> point values.
>>>>>>
>>>>>> --
>>>>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University,
>>>>>> UK
>>>>>> m.b.taylor at bris.ac.uk +44-117-928-8776
>>>>>> http://www.star.bris.ac.uk/~mbt/
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University,
>>>>UK
>>>> m.b.taylor at bris.ac.uk +44-117-928-8776
>>>>http://www.star.bris.ac.uk/~mbt/
>>>
>>
>>