VOSI 1.1 initial working draft

[Mark Taylor]

Brian, Pat and GWS,

I have implemented a client for the scalable tableset endpoint as
proposed in the VOSI-1.1-20160129 WD.  

If you like, you can try it out using this topcat pre-release:

   ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/topcat-full_vosi11.jar

Open the TAP window and point it at Pat's implementation as advertised
at http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap (or just select
the "CADC Table Query (TAP) Service" from the list of TAP services),
To see this option at work, in the TAP window, use the
TAP|Metadata Acquisition submenu, and select one of

    VOSI11-1step (without detail=min)
    VOSI11-2step (with detail=min)

(changing the option re-acquires the metadata).
If you run topcat with the -verbose flag it will log to stderr
what HTTP connections it's actually making.

This service implementation seems to work as described in the WD.
So, I think the proposal is implementable, at least for this service
(which has well-behaved table names).

However, I have a few comments/suggestions on the text
(all concerning section 3.4, "Table Metadata"):

Table name specification:
   When requesting table details (columns & foreign keys) the table
   name is appended as a child resource of the tables endpoint,
   i.e. following a "/" after the ".../tables" part of the URL.
   Since this is only one level of hierarchy, I think it would be
   more natural to specify it as a parameter value, i.e.:

      http://example.net/srv/tables?table=ivoa.ObsCore

   rather than
     
      http://example.net/srv/tables/ivoa.ObsCore

   especially since lots of TAPVizieR tables contain '/' characters.
   If you don't agree, that's OK.  But either way I'd like to have
   some explicit text and an example for how to cope with table names
   that have weird characters in, as many (especially from TAPVizieR) do,
   to make sure that I know how to ask for details for, e.g. 
   "J/A+A/364/712/table6".

303 Redirect:
   I can sort of see the point of issuing a 303 redirect to the
   ?detail=min endpoint in the case of refusing the full dataset;
   it gives clients a way to determine whether they have been
   fobbed off with column-less metadata when that's not what they
   asked for.  But I'm not sure how useful it is.  At least in my
   libraries, grubbing around for HTTP response codes is fiddly to do,
   and when implementing it I didn't bother, I just assumes that if
   there are no columns it's because the service is refusing
   to issue them.

   In particular the proposed 303 mechanism does not, unlike the
   403 Forbidden code suggested in an earlier version of this proposal
   (http://mail.ivoa.net/pipermail/grid/2015-May/002728.html),
   do much to address the problem of backward compatibility,
   since many clients won't even notice that they didn't get a 200
   (java.net.HttpURLConnection.getFollowRedirects() defaults to true,
   and I'd guess similar things in other languages, though not curl).

   So I'd be inclined to suggest dropping this part of the proposal,
   since it complicates matters without adding much.

Detail specification:
   As proposed, there are the following endpoints:

       tap_url/tables?detail=min  - service must not include detail
       tap_url/tables             - service decides whether to include detail

   In some cases, it's not clear whether it's a good idea for the service
   to supply or withold the detail, and it could be helpful for both
   ends if the client could supply a hint.
   E.g. if the client is going to grab all the detail straight away
   in any case (by a load of unconditional per-table queries), then
   a service with a large-but-not-huge number of tables isn't helping
   anybody by forcing detail=min.

   I would suggest instead:

      tap_url/tables?detail=min  - service must not include detail
      tap_url/tables?detail=max  - service decides, but client would like detail
      tap_url/tables             - service decides, client has no preference
   
   The service remains free to implement detail=max in the same way
   as for no detail argument.

   Having written that ... maybe it's not such a great idea, since it's
   not intuitively obvious that detail=max should be able to decay to
   detail=min, maybe it should fail instead.  If that all sounds too
   complicated, maybe just leave it as is.  It's a thought anyway.

QName formatting:
   A couple of schema types come out wrong in the formatted text;
   the XML document root types are quoted in the LaTeX as:

      {http://www.ivoa.net/xml/VODataService/v1.1}TableSet
      {http://www.ivoa.net/xml/VODataService/v1.1}Table

   and those curly brackets are effectively ignored by the LaTeX
   processing.  I think the curly brackets should be escaped using
   preceding backslashes in the LaTeX source, or you might want to
   apply some different formatting to identify those strings as QNames.

typo:
   foreigh->foreign

Mark


On Tue, 2 Feb 2016, Brian Major wrote:

> Dear GWS,
> 
> An initial working draft of version 1.1 of the IVOA Support Interfaces
> (VOSI) document has been posted:
> 
> <goog_999565820>
> http://www.ivoa.net/documents/VOSI/20160129/
> 
> The aim of this version is to offer a more scalable mechanism for viewing
> table metadata.  As per the agreements reached in Sydney, the document
> describes how this can be achieved by:
> 
>     - Allowing services to implement and enforce a detail=min parameter on
> the /tables endpoint on which on a subset of table metadata (which does not
> include columns) is returned.
>     - By allowing clients to view the full metadata of a single table on
> the /tables/<table-name> endpoint.
> 
> Pat Dowler, along with editing this version of the document (thanks!), has
> prototyped the functionality into the CADC TAP service.  Below is the
> tables endpoint:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables
> 
> You can see how the request above enforces the minimum view by issuing a
> redirect with the detail=min parameter included:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables?detail=min
> 
> Clients can get full details on single tables by adding the table name to
> the path.  For example:
> 
> http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/tables/ivoa.ObsCore
> 
> 
> For your convenience, a view of the diff on the source .tex file can be
> seen here:
> 
> https://volute.g-vo.org/viewvc/volute/trunk/projects/grid/VOSI/VOSI.tex?r1=3233&r2=3232&view=diff&diff_format=f
> 
> 
> Cheers,
> Brian
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/