Arrays in TAP_SCHEMA

[Mark Taylor]

On Thu, 1 Jun 2017, Markus Demleitner wrote:

> Hi Pat,
> 
> On Wed, May 31, 2017 at 10:54:10AM -0700, Patrick Dowler wrote:
> > I agree with the goal of supporting array-typed columns, I think 1-d
> > arrays are OK now, and I think this is within the realm of the TAP-1.1
> > update. It would make TAPType in VODataService obsolete since we could
> > just use VOTableType.
> 
> I'm all for that.
> 
> > And I think this is actually a pretty small change and I'm almost
> > there in my prototype.
> > 
> > Thoughts?
> 
> But then why not just take over the VOTable type system (or at least
> arraysize) to TAP_SCHEMA, too?  I'm noticing just now, but if
> something in the vicinity of the VO is called arraysize, as an
> implementor I'd be somewhat sore if it contained something different
> than what is in VOTable's arraysize syntax (I'd be ok with dropping the
> number-plus-star feature, like arraysize="7*").
> 
> It does all we can possibly need at this point, and since dealing
> with it is part of VOTable, people already have code to deal with
> strings like these.

I basically like this idea: a TAP-1.1 arraysize column metadata
item that mirrors the VOTable arraysize attribute.
It would allow services to provide metadata
that they are currently unable to provide[*]
from TAP_SCHEMA (though they can say it with /tables),
namely that certain columns have an array type, along with 
additional, fairly flexible, array dimension information.
It's backwardly compatible, since TAP 1.0 services already 
can and do supply such array-valued columns, it's just that 
there is no proper way to tell clients via TAP_SCHEMA that 
that's what they'll get.  I think it should be string-typed,
and re-use the VOTable syntax, which is also currently used
by VODataService TableDataType's arraysize attribute.
We then have consistency between the arraysize metadata name
and content between VODataService (/tables), TAP_SCHEMA and VOTable.

TAP clients which don't want to parse VOTable arraysize syntax
can either ignore this metadata value completely, or make a partial
attempt to parse it (e.g. work with it if it just matches
the pattern [0-9]+).  In other cases they are at their liberty
to fail to understand the detailed content of the table columns
in question, and be no worse off than they are now.
So as far as that goes, it could be introduced in TAP 1.1.

[*: In particular topcat currently doesn't report anything about
    dimensionality of columns, because as Gregory points out the
    TAP_SCHEMA.columns size doesn't tell you this, and neither
    does anything else in TAP_SCHEMA.  This is a disappointing
    missing feature from the user's point of view]

> For 1.1, I don't think we can switch datatype to VOTable types to
> complete the transition (or can we?).  But arraysize is new anyway,
> and I can't see many reasons to keep the semantics of 1.0 size for
> it.

Unfortunately, I do see a problem with this.  The recently-accepted
TAP-1.0 Erratum 4 (http://wiki.ivoa.net/twiki/bin/view/IVOA/TAP-1_0-Err-4)
concerning the size column from TAP_SCHEMA.columns, says this:

   To avoid the reserved keyword collision, in the next major version of 
   TAP, this column will be called arraysize, in next minor revision(s)
   the name of the column will be kept for back-compatibility reasons.

This seems to say what I had always informally understood, that the
TAP-1.1 arraysize column in tap_schema.columns was going to
have the same syntax and semantics as the TAP-1.0 size column, but just
a different column name to avoid the reserved word collision.
That is rather at odds with the suggestion in this thread that
arraysize gets VOTable-arraysize-like type/syntax/semantics/capabilities.
Given my arguments above, that is a bit of a shame.

Assuming we do want a VOTable-arraysize like column in TAP_SCHEMA.columns
I can think of a few, fairly nasty, options:

 - Ignore the size==arraysize implication of the erratum?
 - Issue an erratum to the erratum?
 - Come up with a name other than arraysize for the VOTABLE-arraysize 
       column in TAP_SCHEMA.columns, leaving arraysize to do what
       size used to do?

though maybe I'm missing something.

Mark

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/