prototype: scalable VOSI-tables-1.1

[Mark Taylor]

Pat et al.,

FYI, I've implemented a GUI client for this proposed protocol.
It works, though I still have my reservations about the 
<service-url>/tables/<schema-name>/<table-name> URL scheme
for the resources.

The TOPCAT TAP client now has a pluggable metadata acquisition
component that can pursue various strategies to get the table
metadata, currently:

   - standard /tables endpoint
   - multiple TAP_SCHEMA queries
   - VizieR-flavour non-standard multi-level /tables endpoint
   - CADC-flavour non-standard multi-level /tables endpoint

By default it picks one it thinks will do the best job ("Auto").

Both the VizieR and CADC /tables endpoints pull all schemas and tables
at first, and then get per-table content only as required.

If you want to see it working, you can find a pre-release version
here:

   ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/topcat-full_tap.jar

To get the TAP window, use the "Table Access Protocol (TAP) Query"
menu item in the top-level "VO" menu, or the equivalent toolbar button.
The "TAP|Metadata Reading" menu item in that window allows you to
choose which metadata acquisition strategy is in use (changing the
value triggers a metadata reload).  You can run with -verbose
or -verbose -verbose to get a bit more information about what
queries are going on under the hood.

Mark

On Mon, 4 May 2015, Patrick Dowler wrote:

> 
> I have implemented a prototype VOSI-tables-1.1 resource to deal with the
> issues that came up in the TAP discussion in Banff: some services have many
> tables and many columns and the top-level VOSI-tables document can be very
> large.
> 
> 
> The basic aproach is to define a RESTful resource tree following the
> VODataService model: tableset, schema, table. All the example URLs below work
> but note that the table names are fully qualified because that is how it works
> in TAP right now. There are separate discussions on that so it isn't really
> relevant here.
> 
> The VOSI-tables-1.0 resource returns a <tableset> document, so the "tables"
> resource name really means "the tableset". This remains unchanged:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables
> 
> Any schema name can be used as a child; this returns a <schema> document:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA
> 
> Any table name found in a schema can be used as a child; this returns a
> <table> document:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA/TAP_SCHEMA.tables
> 
> There is no REST binding to get a single <column> within  a <table> as I don't
> see the use case for that. The abvoe by itself is nice but doesn't solve the
> scalability problem. For that we need to ask for less than "all the details"
> of VOSI-tables-1.0 using the "detail" parameter (name taken from the VOSpace
> spec as it seems to be the same sort of thing: ask for a certain amount of
> detail. The default (as the above
> URLs show) is to get all the details below the resource. The detail parameter
> can taken two values: schema, table.
> 
> To get a <tableset> document with a list of <schema> only:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables?detail=schema
> 
> To get a <tableset> docuemnt with <schema> and <table> (no <column>):
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables?detail=table
> 
> Somewhat useless*: get a <schema> document without the <table>(s):
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA?detail=schema
> 
> To get a <schema> doccument with only the list of <table> (no <column>):
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA?detail=table
> 
> To get a <table> document (a single table):
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA/TAP_SCHEMA.tables
> 
> Somewhat useless*: get a <table> document without the <column>(s):
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA/TAP_SCHEMA.tables?detail=table
> 
> * it was simpler to implement this response than to come up with some logic
> for denying such a request; it can check the existence of the schema or table.
> 
> Given the size of the documents when detail=schema or detail=table is used
> (very small), I don't see an obvious need for a pagination mechanism. VOSpace
> has such a feature so we could model something on that but I'd like to see the
> practical need.
> 
> These would all return a 404 response code:
> 
> testSchemaNotFound:
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/no_such_schema
> 
> testTableNotFound:
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA/no_such_table
> 
> testExtraPathComponentNotFound (trying to get a <column>):
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/TAP_SCHEMA/tables/table_name
> 
> The base resource complies to VOSI-tables-1.0 so 1.0 clients could use a
> VOSI-tables-1.1 resource. The only outstanding issue is how would a service
> force clients to use 1.1 and forbid them from getting the whole <tableset>
> document from the base resource? That is already an issue and right now I
> think the only compliant choice (for TAPvizier, for example) would be to
> respond with a 403 "Forbidden": only authorized callers can get the whole
> tableset in full detail. This is already how some services handle things like
> a GET to a UWS job-list. I think any oter kind of response to indicate that
> some of the requests are not supported would mess wit backwards compatibility.
> Making the default "detail" not be "all" would likewise trod on compatibility.
> Obviously, the tap capabilities would express that the VOSI-tables resource
> implemented 1.0 or 1.1 and I don't see any obvious reason to require a
> specific one in TAP-1.1 (at least).
> 
> The change to the VOSI-tables XSD is to add <schema> and <table> as valid
> document root elements with types taken from VODataService-1.1 xsd.
> 
> 
> 
> -- 
> 
> Patrick Dowler
> Canadian Astronomy Data Centre
> National Research Council Canada
> 5071 West Saanich Road
> Victoria, BC V9E 2E7
> 
> 250-363-0044 (office) 250-363-0045 (fax)
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/