Gaia Archive arrays implementation

[Juan Carlos Segovia]

Dear Marco,

thanks for your comments.

Yes, we have considered TAP 1.1, but, as you can imagine, we do not want 
to use a working draft for an operational software.

Once it is released, we might schedule the implementation of TAP 1.1. in 
our archive.

Thanks,
cheers,
Juan Carlos.


On 08/21/2017 09:08 AM, Marco Molinaro wrote:
> Dear Juan Carlos, dear all,
>
> just a quick reply.
>
> Have you considered working directly on TAP-1.1?
> It's in WD status now, but I was about to push it to PR this week and 
> it should solve at least the "size" issue in columns instead of adding 
> the array_dim and array_type columns (that you can always leave there, 
> of course).
>
> Apart from that, accessing array internal elements is not yet covered, 
> that's true, so I'm glad GAIA proposes its own implementation as a use 
> case in that direction.
>
> Cheers,
>      Marco
>
> 2017-08-21 8:46 GMT+02:00 Juan Carlos Segovia 
> <juan.carlos.segovia at sciops.esa.int 
> <mailto:juan.carlos.segovia at sciops.esa.int>>:
>
>     Dear All,
>
>     This mail is to explain how arrays are going to be handled in the
>     Gaia archive TAP service for DR2.
>
>     We can consider to modify the approach in the future if TAP
>     finally supports this data type but, in our view, the current
>     approach is already providing the basic support required.
>
>     In Gaia we are going to work with arrays of multiple dimensions.
>     It means we need something like (VOTable syntax) 'AxBxC...[x*]'
>     (e.g. '2x3x*') as the size of an array column.
>
>     In the current stable specification (1.0), TAP_SCHEMA.columns.size
>     is an integer. This value cannot be mapped to the required needs.
>
>     Also, the current stable specification assigns to
>     TAP_SCHEMA.columns.size, for numeric types with value > 1, a
>     database type VARBINARY.
>
>     We tried to be compatible with the current specification. So we
>     have decided to assign VARBINARY type for all numeric array types.
>     Note that as the data type VARBINARY is assigned, you loose the
>     original data type.
>
>     In order to be compatible, we have added two new columns to
>     TAP_SCHEMA.columns table:
>
>     array_dim (dimensions)
>     array_type (numeric data type)
>
>     So, for instance, for an integer array of 4x5, we have the
>     following TAP_SCHEMA.column row (only involved columns are shown):
>
>     datatype: 'VARBINARY'
>     size: null
>     array_dim: '4x5'
>     array_type: 'int'
>
>     (There are some problems with chars, so we finally decided to
>     avoid char arrays.)
>
>     Please, find the attached document with a basic mapping for our
>     types. Only integer, short, char and unsignedByte are shown, but
>     all numeric types (short, int, long, float, double) are handled in
>     the same way. Please, take in mind that the last column, database
>     data type, can be ignored as this is an implementation detail.
>
>     The implementation we have done is quite similar to the specified
>     in VOTable 1.1 working draft (main difference: VARBINARY for
>     arrays data type).
>
>     When retrieving data, the VOTable will contain the right array
>     specification as they are extracted from the two extra columns:
>     array_dim and array_type.
>
>     In order to access an specific element (and until the ADQL syntax
>     allows a direct access) we have implemented a database function
>     named get_array_double_element (only for doubles) and can be used
>     as follows (more information at
>     http://gea.esac.esa.int/archive-help/index.html
>     <http://gea.esac.esa.int/archive-help/index.html>, section "ADQL
>     syntax"):
>
>
>     SELECT get_double_array_element('{{1.0,2.6},{0.8,0.1}}','[1][2]')
>     from public.dual
>
>
>     Best regards,
>     Juan-Carlos Segovia.
>
>     -- 
>     Serco for ESA - European Space Agency
>
>     Juan-Carlos Segovia Serrato
>
>     ESAC Science Data Centre
>     Data and Engineering Division
>     Operations Department, Directorate of Science
>     European Space Astronomy Centre (ESAC)
>     European Space Agency (ESA)
>
>     email: juan.carlos.segovia at sciops.esa.int
>     <mailto:juan.carlos.segovia at sciops.esa.int>
>     Phone: (34)-91-8131-175 - 70175 internal
>     Fax:   (34)-91-8131-308
>
>     European Space Astronomy Centre (ESAC)
>     Camino Bajo del Castillo s/n
>     Urb. Villafranca del Castillo
>     28692 Villanueva de la Cañada, Madrid, Spain.
>
>
>     This message and any attachments are intended for the use of the
>     addressee or addressees only.
>     The unauthorised disclosure, use, dissemination or copying (either
>     in whole or in part) of its
>     content is not permitted.
>     If you received this message in error, please notify the sender
>     and delete it from your system.
>     Emails can be altered and their integrity cannot be guaranteed by
>     the sender.
>
>     Please consider the environment before printing this email.
>
>
>


-- 
Serco for ESA - European Space Agency

Juan-Carlos Segovia Serrato

ESAC Science Data Centre
Data and Engineering Division
Operations Department, Directorate of Science
European Space Astronomy Centre (ESAC)
European Space Agency (ESA)

email: juan.carlos.segovia at sciops.esa.int
Phone: (34)-91-8131-175 - 70175 internal
Fax:   (34)-91-8131-308

European Space Astronomy Centre (ESAC)
Camino Bajo del Castillo s/n
Urb. Villafranca del Castillo
28692 Villanueva de la Cañada, Madrid, Spain.


This message and any attachments are intended for the use of the addressee or addressees only.
The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its
content is not permitted.
If you received this message in error, please notify the sender and delete it from your system.
Emails can be altered and their integrity cannot be guaranteed by the sender.

Please consider the environment before printing this email.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/dal/attachments/20170822/f3806ae7/attachment.html>