<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Pat,<br>
<br>
<br>
On 05/31/2017 07:54 PM, Patrick Dowler wrote:<br>
</div>
<blockquote
cite="mid:CAFK8nrqahPmG_1-dYr79ngkz6S9P2Jc-4vPA2fMJOJsOsg1k3g@mail.gmail.com"
type="cite">
<pre wrap="">The last thing I am trying to include in TAP-1.1 is the addition of
xtype in the tap_schema, which would *almost* make the type system the
same -- arraysize being more limited in TAP-1.1 than in
VOTable/VODataService.</pre>
</blockquote>
<br>
<br>
The addition of xtype may duplicate the TAPType (e.g. POINT,
TIMESTAMP), but if you say we are going to deprecate TAPType and
rather use VOTable-ish types, it is all good for me: I am ok with
the addition of xtype.<br>
<br>
<br>
<blockquote
cite="mid:CAFK8nrqahPmG_1-dYr79ngkz6S9P2Jc-4vPA2fMJOJsOsg1k3g@mail.gmail.com"
type="cite">
<pre wrap="">The first thing to accomplish was to remove "size" and shift services
and clients to something else: arraysize.</pre>
</blockquote>
<br>
Ok<br>
<br>
<blockquote
cite="mid:CAFK8nrqahPmG_1-dYr79ngkz6S9P2Jc-4vPA2fMJOJsOsg1k3g@mail.gmail.com"
type="cite">
<pre wrap="">The intent is to support array types in TAP (I already do it in my
service), at least as something you can select but maybe not do
anything with in other parts of the query. I think you can do that now
without breaking clients (for 1-D arrays) and I do that in my
implementation.
The tap_schema in TAP-1.1 says arraysize is an integer so supports 1-d
array only.</pre>
</blockquote>
<br>
<br>
Is there another version of the draft document for TAP 1.1 than:<br>
<br>
<a
href="http://www.ivoa.net/documents/TAP/20160428/WD-TAP-1.1-20160428.html">http://www.ivoa.net/documents/TAP/20160428/WD-TAP-1.1-20160428.html</a><br>
<br>
If not, it is written nowhere that "arraysize" is a 1-D array
length.....or at least it is not written in 4.3:<br>
<br>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<blockquote>
<pre><tt><span style="color: rgb(0, 0, 0); font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;">The arraysize column gives the length of variable length datatypes, for example varchar(256); this arraysize does not map exactly to the VOTable arraysize attribute because the latter can specify the size and shape of a multi-dimensional array as well as the variable size. The ßize" column is retained for backwards compatiblity to TAP-1.0 and must contain the same value as arraysize.</span></tt></pre>
<span style="color: rgb(0, 0, 0); font-family: sans-serif;
font-size: medium; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"></span></blockquote>
<span style="color: rgb(0, 0, 0); font-family: sans-serif;
font-size: medium; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial; display: inline
!important; float: none;"><br>
However, if there is another version of the TAP 1.1 document,
could you tell me where I can take a look on it?<br>
<br>
And if guarantee that the TAP 1.1's arraysize is clearly the
length of arrays (and not the length of variable-length datatypes
as "size" does), I am happy with that and I can start adapting my
TAP-Library in that direction.<br>
</span><br>
<br>
<blockquote
cite="mid:CAFK8nrqahPmG_1-dYr79ngkz6S9P2Jc-4vPA2fMJOJsOsg1k3g@mail.gmail.com"
type="cite">
<pre wrap="">We could change that to (allow) array-of-integer to
support multi-dimensional arrays and in most cases services would
continue to return a single number. I'd have to work out how drastic
that change looked but my feeling is that services could declare that
column to be length 1 or longer so existing services would not need to
be changed and services that wanted to expose multi-dim arrays would
make the change. Clients would have to be flexible.
That still doesn't allow for variable length and I have found that I
need to know variable length in my implementation and was going to add
a boolean column to tap_schema.columns for that. I had not intended to
push that into the standard as I'm not sure anyone else needs to know,
but the move from "adql" xtypes to VOTable datatype + DALI xtype
leaves that out of the tap_schema (except when implied by tings like
xtype="polygon").
In my protype of adding xtype to tap_schema and using tap_schema as
the definitive source of all metadata (impl design), I ended up
adding a DataType construct that looks like this:
DataType
- String datatype
- Integer arraysize
- Boolean varsize
- String xtype
If this was extended to
DataType
- String datatype
- Integer[] arraysize
- Boolean varsize
- String xtype
and we added varsize (name TBD) to tap_schema.columns then it would
match features of VOTable. We alrady nominally accepted that adding
xtype was OK for TAP-1.1 so I think the same would apply to varsize...
and I think we can specfy tap_schema.columns.arraysize so that it is
length 1 (TAP-1.0 compat with "size") or longer and then other columns
that are 1- or multi-d arrays could be described. Services that don't
support arrays would simply have arraysize that was just like "size"
in TAP-1.0.</pre>
</blockquote>
<br>
<br>
For 1-D array, I agree that making arraysize an INTEGER makes
perfectly sense, and that would be enough for TAP-1.1. But if later
in TAP-1.2 (or TAP-2.0) we want to support multi-dim arrays, we
would have to change the type of arraysize (into VARCHAR or
INTEGER[] as you suggest), which would imply a relatively important
transition for TAP clients...another one already after TAP 1.1. For
that reason, I think it would be better to make arraysize like in
VOTable (particularly if we tend into that direction in the future:
VOTable-ish types): char(*) or VARCHAR.<br>
<br>
If we agree to have a VARCHAR arraysize in TAP_SCHEMA.columns, we
can still say that in TAP-1.1, we only support 1-D array (and yes,
it would be just an integer into a String which is ugly, I agree).
And then it TAP-1.2 or 2.0, we would still have the possibility to
choose a better string syntax for multi-dim arrays (another syntax
than the VOTable one...probably simpler). Moreover, having a VARCHAR
arraysize would allow variable length arrays like in VOTable: with a
'*' for instance. I think that it is a good compromise that works
for TAP-1.1 but still allows an easy and flexible evolution for
later versions of the TAP protocol.<br>
<br>
Anyway, I agree that it makes the datatypes not as safe as you
suggest, but I think that having more and more columns just to
define a datatype starts to become too complicated for so few
benefit....but it is just my anticipated point of view on that
topic, so maybe I am wrong.<br>
<br>
<br>
About the variable length datatypes (e.g. CHAR, VARCHAR, ....), I
think it depends of the datatype system we choose. If we want to
keep TAPType, "size" could still be use for this purpose, but if we
want to get rid of "size", why not encoding the length of these
special datatypes directly in the datatypes like in databases:
CHAR(n), VARCHAR(n), .... The "datatype" column being a VARCHAR,
such thing is still possible, but that of course means that TAP
clients would have to change a bit the way they parse datatypes
(e.g. startsWith(...) instead of equals(...)). Otherwise, if we want
to choose a VOTable-ish datatype system, arraysize is fairly enough
for variable length datatypes, because it is already how it is done
in VOTable....am I wrong?<br>
<br>
<br>
Grégory<br>
<br>
<br>
<blockquote
cite="mid:CAFK8nrqahPmG_1-dYr79ngkz6S9P2Jc-4vPA2fMJOJsOsg1k3g@mail.gmail.com"
type="cite">
<pre wrap="">I agree with the goal of supporting array-typed columns, I think 1-d
arrays are OK now, and I think this is within the realm of the TAP-1.1
update. It would make TAPType in VODataService obsolete since we could
just use VOTableType.
And I think this is actually a pretty small change and I'm almost
there in my prototype.
Thoughts?
Pat
On 31 May 2017 at 08:08, alberto micol <a class="moz-txt-link-rfc2396E" href="mailto:amicol.ivoa@googlemail.com"><amicol.ivoa@googlemail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">My use case regarding arraysize=“2" in TAP is here described.
SSA 1.1 requires, in the VOTable output, a single field for the spatial
location,
expressed as:
<FIELD ID="SpatialLocation" name="SpatialLocation" datatype=“double"
ucd="pos.eq" utype="ssa:Char.SpatialAxis.Coverage.Location.Value"
arraysize="2" unit=“deg”>
At ESO we are implementing SSA on top of TAP.
I therefore created the spatial location column in the TAP_SCHEMA.columns
table declaring size=2,
but TAP does not translate this information into the (naively) expected
arraysize=“2”,
for the reasons that Gregory explained.
Hence, we are stack with the development of the ESO SSAP...
unless some smart DAL person comes up with a solution!
Many thanks,
Alberto
On 31 May 2017, at 16:34, Grégory Mantelet <a class="moz-txt-link-rfc2396E" href="mailto:gmantele@ari.uni-heidelberg.de"><gmantele@ari.uni-heidelberg.de></a>
wrote:
Dear DAL members,
Sorry to come back again with the "array" topic, but I have more and more
requests for having arrays in my TAP-Library (and, personally, I will also
need that quite soon) but I do not know how to proceed. Even though nothing
formally forbids it, there is actually no possibility to declare arrays in
TAP_SCHEMA...so in a way it is kind of preventing/forbidding the usage of
arrays if nobody can really know that a column is an array.
I have searched in TAP-1.0, the coming TAP-1.1 and in VODataService in the
hope to find something leading us toward a solution. Here is what I found
and my related questions:
## In TAP 1.0
In REC-TAP-1.0, two columns of TAP_SCHEMA.columns let specify the type of a
published column, defined as follows:
- datatype - "ADQL datatype as in section 2.5"
- size - "length of variable length datatypes"
With the following additional description:
"Data types and how they map to VOTable datatypes are described in
section 2.5
above. The “size” gives the length of variable length datatypes, for
example
varchar(256); this size does not map to the VOTable arraysize
attribute when the
latter specifies the size and shape of a multi-dimensional array."
As written here, "size" does not aim to tell whether the value is a scalar
or an array ; it is just the N in CHAR(N), VARCHAR(N), BINARY(N) and
VARBINARY(N).
## In TAP 1.1
In WD-TAP-1.1, in addition of the above two columns, "arraysize" has been
added. So the datatype descriptive columns are now:
- datatype - ?? (the description disappeared in this WD)
- "size" - ?? (idem)
- arraysize - ?? (idem)
With the following additional description:
"The arraysize column gives the length of variable length datatypes,
for
example varchar(256); this arraysize does not map exactly to the
VOTable
arraysize attribute because the latter can specify the size and
shape of a
multi-dimensional array as well as the variable size.
[...]
In the next major version of TAP, the "size" column
will be removed."
So, even in TAP-1.1 there will be no way to add information about arrays.
==> Furthermore, though I can understand the reason why "size" should be
deprecated (collision with an ADQL reserved keyword....by the way, will we
still have reserved keywords with the PEG grammar for ADQL?), is it really a
good idea to call a column "arraysize" if it is not about an array?
==> And then, why having the same name as in VOTable if it does not do the
same?
## In VODataService 1.1
In REC-VODataService-1.1 (used to describe published columns in TAP's entry
point '/tables'), the datatype of a column can be expressed using two types
of type:
- VOTableType (e.g. <dataType xsi:type="vs:VOTableType"
arraysize="*"> char </dataType>)
- TAPType (e.g. <dataType xsi:type="vs:TAPType" size="8" >
CHAR </dataType>)
According to the XML schema of VODataService-1.1, TAPType is the only one
that can have a "size" attribute defined as described in TAP 1.0 (i.e. "The
length of the variable-length data type."). Ok, that makes sense since it is
only something coming from TAP.
==> By the way, is it also planned to deprecate "size" from VODataService as
in TAP-1.1?
However, both VOTableType and TAPType can have an "arraysize" attribute
defined as described in VOTable (i.e. an ArrayShape = " An expression of a
the shape of a multi-dimensional array of the form LxNxM... where each value
between gives the integer length of the array along a dimension. An asterisk
(*) as the last dimension of the shape indicates that the length of the last
axis is variable or undetermined.").
So, here, we have a completely different definition of "arraysize" than in
WD-TAP-1.1.
==> Is there a mistake here? If yes, which standard has to be updated:
VODataService or TAP? And in which direction?
## To conclude,
==> considering these three documents and knowing that TAP-1.1 is still in
WD, how can we declare arrays in TAP_SCHEMA (and /tables result)?
I personally like to have something consistent and so I would go for
re-defining the new column "arraysize" as in VODataService and VOTable.
==> But does it make sense to combine this VOTable piece of information with
the datatypes of TAP (i.e. the so-called TAPType like VARCHAR, BIGINT, BLOB,
...)? If not, what other alternative(s) do we have?
Cheers,
Grégory
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>