Fwd: new table format proposed by astropy dev community.

[Patrick Dowler]

-------- Forwarded Message --------
From: 	Kavelaars, JJ <JJ.Kavelaars at nrc-cnrc.gc.ca>


Interested in table file formats?  There is some discussion of this
happening in astropy community, perhaps they would benefit from
insightful comment, see discussion at the github link below...


APE-6 : https://github.com/astropy/astropy-APEs/pull/7

Abstract
--------

Data tables in astronomical analysis frequently contain additional
metadata beyond just the column names and data values.  Common
attributes include the numerical data type, the physical unit, and a
longer textual description of the column content.  These attributes can
be represented in binary formats such as FITS, but the available options
for a text-only format are inadequate.

APE6 proposes to fill this void by specification of a standard for the
interchange of tabular data in a text-only format.  The format handles
the key issue of serializing column specifications and table metadata by
using a YAML-encoded data structure.  The actual tabular data are stored
in a standard character separated values (CSV) format, giving
compatibility with a wide variety of non-specialized CSV table readers.

Using YAML makes it extremely easy for applications *and humans* to read
both the standardized data format elements (e.g. column name, type,
description) as well as complex metadata structures.  YAML also lends
itself to simple table modifications by humans in a plain text editor.

The reference Python implementation in ``astropy.io.ascii`` is
relatively straightforward and will provide a significant benefit of
allowing text serialization of most astropy Table objects, persistent
storage, and subsequent interchange with other users.

Although developed in the context of Astropy, there is nothing
astronomy-specific in this format and it could be useful for other
domains or languages.  However, promoting that is beyond the scope of
this APE.