<div dir="ltr"><div>Dear Dave, dear all,</div><div><br></div><div>(long email - beware)</div><div><br></div><div>Summarizing and adding my perspective out of Dave's mail.</div><div>(please Dave check if I'm missing or misunderstanding something from your mail).</div><div><br></div><div>--- Two examples reported</div><div><br></div><div>A) example from Vizier:</div><div><span class="" style="white-space:pre"> </span>vbig.J/other/PZ/29.1/table </div><div><span class="" style="white-space:pre"> </span>OR </div><div><span class="" style="white-space:pre"> </span><TR></div><div> <span class="" style="white-space:pre"> </span><TD>vbig</TD></div><div> <span class="" style="white-space:pre"> </span><TD>J/other/PZ/29.1/table</TD></div><div><span class="" style="white-space:pre"> </span></TR></div><div>B) example from GAVO:</div><div><span class="" style="white-space:pre"> </span>twomass.data</div><div><span class="" style="white-space:pre"> </span>OR</div><div><span class="" style="white-space:pre"> </span><TR></div><div> <span class="" style="white-space:pre"> </span><TD>twomass</TD></div><div> <span class="" style="white-space:pre"> </span><TD>twomass.data</TD></div><div><span class="" style="white-space:pre"> </span></TR></div><div><br></div><div>--- Three interpretations outlined</div><div><br></div><div>1) VODataService-1.1 (fully qualified name table)</div><div>A -> vbig | J/other/PZ/29 | 1/table</div><div>B -> N/A | twomass | data</div><div><br></div><div>proposed: quote (") all the SQL elements</div><div>It seems not to solve the _catalog_ use case, both I wouldn't take into account catalogs in service descriptions.</div><div><br></div><div>2) TAP-1.0 (name to be used in queries, may/may not be qualified)</div><div>A -> conflict if considering the . delimiter in the table_name value because we have a schema name and a table in that schema that seems to belong to another schema.</div><div>B -> seems ok if one considers the "common" interpretation of the TAP_SCHEMA fields</div><div><br></div><div>proposed: define quoting.</div><div>Catalogs seem again out of the scope.</div><div><br></div><div>3) Human</div><div>Human inspection resolves by common sense, but machine driven processes cannot rely on common sense.</div><div><br></div><div>Now, the mail from Gilles and this reply from Dave are moving from the discussion on TAP-1.1 we had in Banff. It is clear that the tables endpoint in TAP requires upgrading because it is currently not a scalable capability as it needs to be (as Vizier use case clearly shows).</div><div>The Vizier use case seems, however, to push further than the simple scalability the issue with current tables endpoint in TAP, given the use of the full available characters set for table namings.</div><div><br></div><div>My opinion is that quoting the schema and table names plus defining what we mean by qualified names should work, but I have some concerns.</div><div><br></div><div>First one is about catalogs, touched by Dave's mail: they're currently mentioned nowhere, introducing them now may create confusion. Can we just consider the DB tree starting at schemas only?</div><div><br></div><div>We risk a "fully quoted" scenario. This may be mainly a client-side problem, but I think it makes the queries less neat and readable.</div><div><br></div><div>If I remember correctly from Banff, there is the idea to have the tables endpoint as a ReST tree. If so, how can we cope with quoted parts in the URLs? Do we encode them preserving the schema/table[/column] tree? or will it be up to the server to provide asymmetric trees if it makes use of "/" in schema or table names? </div><div><br></div><div>All of these, I think, have an impact on the path we should follow to solve the scalability issue of the tables endpoint, it was already pointed out that a VOSI update may be a solution, Dave's suggestion do point in a VODataService revision? Or maybe can we consider rewording in VODataService at an errata level?</div><div><br></div><div>As for Gilles suggestion to add a url element to the tables, I'd prefer leveraging on name&title coupling to build the ReST tree, without the need to add further elements in the existing schema.</div><div><br></div><div>All above it's just my personal view,</div><div>Cheers,</div><div> Marco</div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-10-16 16:58 GMT+02:00 Dave Morris <span dir="ltr"><<a href="mailto:dave.morris@metagrid.co.uk" target="_blank">dave.morris@metagrid.co.uk</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2014-10-16 10:39, LANDAIS Gilles (OBS) wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Due to the important volumetry (~25,000 table , ~ 300,000 columns),<br>
the web resource /tables of TAPVizieR provides the schema without the<br>
columns descriptions.<br>
This output enables web applications like TAPHandle to work with<br>
TAPVizieR with a reasonable size of VOTable.<br>
Currently, TAPVizieR provides a non-standard REST URL to get the full<br>
table description (with columns). The output URL uses the same XML<br>
schema than the standard resource /tables (VODataService/1.1).<br>
<br>
Example:<br>
<a href="http://tapvizier.u-strasbg.fr/TAPVizieR/tap/tables" target="_blank">http://tapvizier.u-strasbg.fr/<u></u>TAPVizieR/tap/tables</a><br>
<a href="http://tapvizier.u-strasbg.fr/TAPVizieR/tap/tables/II/246/out" target="_blank">http://tapvizier.u-strasbg.fr/<u></u>TAPVizieR/tap/tables/II/246/<u></u>out</a><br>
<br>
</blockquote>
<br>
Publishing the metadata for the TAPVizieR service highlights some gaps in the current VODataService and TAP_SCHEMA specifications that will need to be clarified before this service can interoperate with other TAP services in the VO.<br>
<br>
This is not the fault of the TAPVizieR service, this is due to some omissions in the current VODataService and TAP_SCHEMA specifications which are not precise enough to handle the TAPVizieR metadata.<br>
<br>
----<br>
<br>
The /tables endpoint<br>
<br>
<a href="http://tapvizier.u-strasbg.fr/TAPVizieR/tap/tables" target="_blank">http://tapvizier.u-strasbg.fr/<u></u>TAPVizieR/tap/tables</a><br>
<br>
lists 477 instances of table names with two dots but no quotes.<br>
<br>
For example :<br>
<br>
<name>vbig.J/other/PZ/29.1/<u></u>table</name><br>
<br>
----<br>
<br>
Section 3.3 of the VODataService-1.1 specification defines the <name> element as containing :<br>
<br>
"A fully qualified name for the table."<br>
<br>
"This name should include all catalog or schema<br>
prefixes needed to sufficiently uniquely<br>
distinguish it in a query to the table."<br>
<br>
However the VODataService-1.1 specification does not describe how to handle a table name that includes non-delimiter dots in it.<br>
<br>
----<br>
<br>
Based on a literal reading of the text in the VODataService-1.1 specification<br>
<br>
"A fully qualified name for the table."<br>
<br>
Implies that a /tables result containing<br>
<br>
<name>vbig.J/other/PZ/29.1/<u></u>table</name><br>
<br>
refers to<br>
<br>
a catalog called<br>
<br>
'vbig'<br>
<br>
a schema called<br>
<br>
'J/other/PZ/29'<br>
<br>
a table called<br>
<br>
'1/table'<br>
<br>
whereas a human interpreter may guess based on context that this actually refers to<br>
<br>
a schema called<br>
<br>
'vbig'<br>
<br>
a table called<br>
<br>
'J/other/PZ/29.1/table'<br>
<br>
----<br>
<br>
The current VODataService-1.1 specification needs to be updated to describe how the /tables output should use quotes to wrap names that contain non-delimiter dots or other characters outside the basic set of alphanumeric characters.<br>
<br>
----<br>
<br>
In this example the schema and table names should probably be wrapped in double quotes to indicate which dot is part of the table name and which is the delimiter between schema and table.<br>
<br>
<name>"vbig"."J/other/PZ/29.1/<u></u>table"</name><br>
<br>
----<br>
<br>
The same table metadata is also available from the TAPVizieR TAP service<br>
<br>
<a href="http://tapvizier.u-strasbg.fr/TAPVizieR/" target="_blank">http://tapvizier.u-strasbg.fr/<u></u>TAPVizieR/</a><br>
<br>
via a TAP_SCHEMA query<br>
<br>
"SELECT schema_name, table_name FROM TAP_SCHEMA.tables"<br>
<br>
which returns a VOTable containing<br>
<br>
<TR><br>
<TD>vbig</TD><br>
<TD>J/other/PZ/29.1/table</TD><br>
</TR><br>
<br>
----<br>
<br>
Section 2.6 of the TAP-1.0 specification defines the table_name column as<br>
<br>
"table name as it should be used in queries"<br>
<br>
The text below this adds a bit more detail to the definition, but it is still less specific about qualifying the table name than the equivalent text in the VODataService-1.1 specification<br>
<br>
"The value of the table_name should be<br>
the string that is recommended for use<br>
in querying the table; it may or may not<br>
be qualified by schema and catalog name(s)<br>
depending on the implementation requirements."<br>
<br>
Given the current definition of 'may or may not be qualified', the table name in this example could be interpreted as<br>
<br>
a schema called<br>
<br>
'J/other/PZ/29'<br>
<br>
a table called<br>
<br>
'1/table'<br>
<br>
or as<br>
<br>
a table called<br>
<br>
'J/other/PZ/29.1/table'<br>
<br>
>From context we can guess that this does in fact represent the unqualified table name containing a non-delimiter dot.<br>
<br>
But this is a *guess*, and is not covered by the rules for representing qualified or unqualified names that may or may not contain non-delimiter dots.<br>
<br>
----<br>
<br>
The current TAP-1.0 specification needs to be updated to describe in how the metadata in the TAP_SCHEMA tables should use quotes to wrap names that contain non-delimiter dots or other characters outside the basic set of alphanumeric characters.<br>
<br>
----<br>
<br>
In this example the table name in the table_name column should probably be wrapped in double quotes to indicate that the dot is part of the table name and not a delimiter between schema and table.<br>
<br>
<TD>"J/other/PZ/29.1/table"</<u></u>TD><br>
<br>
----<br>
<br>
For comparison, sending the same TAP_SCHEMA query to the Gavo TAP servicve<br>
<br>
<a href="http://dc.zah.uni-heidelberg.de/__system__/adql/query/form" target="_blank">http://dc.zah.uni-heidelberg.<u></u>de/__system__/adql/query/form</a><br>
<br>
"SELECT schema_name, table_name FROM TAP_SCHEMA.tables"<br>
<br>
returns a VOTable containing<br>
<br>
<TR><br>
<TD>twomass</TD><br>
<TD>twomass.data</TD><br>
</TR><br>
<br>
If we apply the same parsing rules that we used for the TAPVizieR results, then this could refer to<br>
<br>
a schema called<br>
<br>
'twomass'<br>
<br>
and a table called<br>
<br>
'twomass.data'<br>
<br>
or this could refer to<br>
<br>
a schema called<br>
<br>
'twomass'<br>
<br>
and a table called<br>
<br>
'data'<br>
<br>
Applying the same set of parsing rules that were needed to interpret the TAPVizieR TAP_SCHEMA results to the Gavo TAP_SCHEMA results mean that the table names in the Gavo TAP_SCHEMA results may be open to misinterpretation.<br>
<br>
Note - there is nothing in any of the specifications that says that we cannot have combinations of catalogs, schemas, tables and columns with the same names.<br>
<br>
Just because the table name 'twomass.data' starts with the same sub-string as the schema name 'twomass' does not by itself mean that 'twomass.data' is the qualified table name including the parent schema name and delimited by a dot, rather than a table name which just happens to start with the same sub-string as the parent schema name and contain a non-delimiting dot.<br>
<br>
----<br>
<br>
We could simplify the parsing rules by defining both the schema name and table name as always unqualified, removing the need for using quotes within the metadata.<br>
<br>
<TR><br>
<TD>vbig</TD><br>
<TD>J/other/PZ/29.1/table</TD><br>
</TR><br>
<br>
and<br>
<br>
<TR><br>
<TD>twomass</TD><br>
<TD>data</TD><br>
</TR><br>
<br>
Note - in order to use the fully unqualified schema name we would have to add a separate column/element to the metadata to contain the catalog name.<br>
<br>
----<br>
<br>
We could simplify the parsing rules by making the table names always fully qualified and always wrap all the names in quotes.<br>
<br>
<TR><br>
<TD>"vbig"</TD><br>
<TD>"vbig"."J/other/PZ/29.1/<u></u>table"</TD><br>
</TR><br>
<br>
and<br>
<br>
<TR><br>
<TD>"twomass"</TD><br>
<TD>"twomass"."data"</TD><br>
</TR><br>
<br>
Note - the schema name also needs to be quoted because schema names may be qualified with a catalog name and both the schema and catalog names may themselves contain non-delimiter dots or other non alphanumeric characters.<br>
<br>
----<br>
<br>
We could try to define a more complex set of conditional rules which work for both the Gavo and TAPVizieR metadata and are compatible with the existing service and client implementations.<br>
<br>
----<br>
<br>
What do you think ?<br>
Anyone like to have a go at defining the rules for qualified/unqualified quoted/unquoted names ?<br>
<br>
--------<br>
Dave Morris<br>
Software Developer<br>
Wide Field Astronomy Unit<br>
Institute for Astronomy<br>
University of Edinburgh<br>
--------<br>
<br>
<br>
</blockquote></div><br></div>