Gaia Archive arrays implementation

[Marco Molinaro]

Dear Juan Carlos, dear all,

just a quick reply.

Have you considered working directly on TAP-1.1?
It's in WD status now, but I was about to push it to PR this week and it
should solve at least the "size" issue in columns instead of adding the
array_dim and array_type columns (that you can always leave there, of
course).

Apart from that, accessing array internal elements is not yet covered,
that's true, so I'm glad GAIA proposes its own implementation as a use case
in that direction.

Cheers,
     Marco

2017-08-21 8:46 GMT+02:00 Juan Carlos Segovia <juan.carlos.segovia at sciops.
esa.int>:

> Dear All,
>
> This mail is to explain how arrays are going to be handled in the Gaia
> archive TAP service for DR2.
>
> We can consider to modify the approach in the future if TAP finally
> supports this data type but, in our view, the current approach is already
> providing the basic support required.
>
> In Gaia we are going to work with arrays of multiple dimensions. It means
> we need something like (VOTable syntax) 'AxBxC...[x*]' (e.g. '2x3x*') as
> the size of an array column.
>
> In the current stable specification (1.0), TAP_SCHEMA.columns.size is an
> integer. This value cannot be mapped to the required needs.
>
> Also, the current stable specification assigns to TAP_SCHEMA.columns.size,
> for numeric types with value > 1, a database type VARBINARY.
>
> We tried to be compatible with the current specification. So we have
> decided to assign VARBINARY type for all numeric array types. Note that as
> the data type VARBINARY is assigned, you loose the original data type.
>
> In order to be compatible, we have added two new columns to
> TAP_SCHEMA.columns table:
>
> array_dim (dimensions)
> array_type (numeric data type)
>
> So, for instance, for an integer array of 4x5, we have the following
> TAP_SCHEMA.column row (only involved columns are shown):
>
> datatype: 'VARBINARY'
> size: null
> array_dim: '4x5'
> array_type: 'int'
>
> (There are some problems with chars, so we finally decided to avoid char
> arrays.)
>
> Please, find the attached document with a basic mapping for our types.
> Only integer, short, char and unsignedByte are shown, but all numeric types
> (short, int, long, float, double) are handled in the same way. Please, take
> in mind that the last column, database data type, can be ignored as this is
> an implementation detail.
>
> The implementation we have done is quite similar to the specified in
> VOTable 1.1 working draft (main difference: VARBINARY for arrays data type).
>
> When retrieving data, the VOTable will contain the right array
> specification as they are extracted from the two extra columns: array_dim
> and array_type.
>
> In order to access an specific element (and until the ADQL syntax allows a
> direct access) we have implemented a database function named
> get_array_double_element (only for doubles) and can be used as follows
> (more information at http://gea.esac.esa.int/archive-help/index.html,
> section "ADQL syntax"):
>
>
> SELECT get_double_array_element('{{1.0,2.6},{0.8,0.1}}','[1][2]') from
> public.dual
>
>
> Best regards,
> Juan-Carlos Segovia.
>
> --
> Serco for ESA - European Space Agency
>
> Juan-Carlos Segovia Serrato
>
> ESAC Science Data Centre
> Data and Engineering Division
> Operations Department, Directorate of Science
> European Space Astronomy Centre (ESAC)
> European Space Agency (ESA)
>
> email: juan.carlos.segovia at sciops.esa.int
> Phone: (34)-91-8131-175 - 70175 internal
> Fax:   (34)-91-8131-308
>
> European Space Astronomy Centre (ESAC)
> Camino Bajo del Castillo s/n
> Urb. Villafranca del Castillo
> 28692 Villanueva de la Cañada, Madrid, Spain.
>
>
> This message and any attachments are intended for the use of the addressee
> or addressees only.
> The unauthorised disclosure, use, dissemination or copying (either in
> whole or in part) of its
> content is not permitted.
> If you received this message in error, please notify the sender and delete
> it from your system.
> Emails can be altered and their integrity cannot be guaranteed by the
> sender.
>
> Please consider the environment before printing this email.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20170821/a44b1171/attachment.html>