TAP issues

[Douglas Tody]

On Wed, 4 Feb 2009, Markus Demleitner wrote:

> (1) The TAP_SCHEMA tables have one field utype -- what happens if,
> say, a table corresponds to more than one data model?  Do you
> actually want to exclude that?

I think it is wise to avoid over complicating things, so yes it is
probably best to not try to do something like alias or union data
elements across models.  Nonetheless what you describe should still be
possible using relational techniques.  Basically, two different data
entities could reference a common table, the latter having a fixed
UTYPE describing only what is in that one table.  This is probably
a better approach for a relational facility like TAP in any case.

There are still some details to work out with how UTYPE is used,
both in schemas and in queries.  Without it all we have is a text
description of the semantic meaning of a table so it at least provides
a mechanism (optional of course) for an externally defined data model
to define its mapping onto DBMS tables.

> (2) 2.4.11 Range-list parameters -- I'm not sure I'm happy that we
> start yet another syntax for purposes like that. I trust you had
> good reasons to disregard ASU.
>
> However, I think the ASU experience at least tells us that mixing
> values and operators in a rather haphazard way (i.e., without
> rigorously defining what literals look like) is asking for trouble.
> We have to deal with funky values (pairs, strings), and if we start
> defining syntax on them, we should do it properly from the start (or,
> preferably, avoid having to do so).
>
> At least for enumerations, the standard HTTP URL syntax offers a way
> around "embedded syntax" -- just give the same argument multiple
> times, quite like forms to with selections or checkboxes.  Presto, no
> (extra) trouble with escaping.
>
> Similarly for ranges, I'd prefer to get the "operator" out of band,
> e.g., by defining "magic" names (there are reserved parameter names
> anyway, why not reserve everything starting with _OPR_ and then say
> _OPR_MAX_<colname> or similar?).
>
> The only alternative that I think will not come back and haunt us
> later is to strictly define literals, e.g., using SQL syntax.

A number of things were looked at before proposing this syntax,
including those used at CDS (ASU), IRSA, and HEASARC (there was an
email summarizing this analysis long ago).  The main reason range
list syntax is used is that this is the standard parameter syntax for
ranges used in DAL interfaces - it is simple, and was motivated by
ISO8601 time syntax, which uses "/" to express time ranges.  We have
already adopted ISO8601 so it is nice to be consistent.  If one has
already implemented other DAL services one will already have code to
do all the parameter handling.  It would not be good to have every
data access interface do these things differently unless there is a
very good reason to do so.

There has been some discussion of devising a more elaborate syntax
just for TAP, but of course we already have ADQL if we want to do more
complex things involving full expression syntax.  So the motivation
is to keep the param query expressions quite simple.

This syntax has already been implemented in existing TAP prototypes
at IRSA and STScI and the experience indicates that it can be done
rigorously, although more prototyping and real world experimentation
is needed.  While limited, it is adequate to express many simple
select-type filtering queries on individual astronomical catalogs
(the most common thing users do), which is all it is intended for.

Regarding code reuse, within NVO we have the IRSA and STScI prototypes,
however the IRSA one is just a translation layer over the existing
local archive services so may not be that useful elsewhere; the STScI
one is written in C# and could be useful if you use that language.
We plan to provide a generic TAP interface which can easily be adapted
to local data within the DALServer framework, but have not done it yet,
although cone search is in there now, and it can do all the standard
parameter handling including range lists.

 	- Doug