VOSpace 2.1 and view=data

[Dave Morris]

Hi Brian,

In answer to your specific questions :

On 2016-06-03 20:21, Brian Major wrote:
> 
> - Should VOSpace users require a smart client that understands transfer
> negotiations, or should simple HTTP clients such as curl and wget work?
>  (For downloads only--uploads and server to server transfers are 
> certainly
> too complex.)

Most of the IVOA specifications define technical protocols that enable 
applications and services to interoperate, and end users normally don't 
interact directly with the IVOA protocols.

Technically, it is possible to do service endpoint discovery, table 
metadata discovery, ADQL query building and async TAP requests using 
shell script to glue together curl requests. However, we normally 
recommend applications like TopCat and Aladin to our end users because 
they hide a lot of the technical details and enable the scientists to 
concentrate on the science.

Same with services like Amazon S3, GoogleDrive and DropBox, it is 
possible to drive them using shell script and curl requests, but normal 
users are encouraged to download the app and click on the nice friendly 
buttons.

Why would we want to treat VOSpace differently?

The raw protocol itself was not intended for end users, it was designed 
to handle transport negotiation between applications and services.

Follow up question :
Should we be providing command line tools for our users and Python and 
Java libraries for app developers to use - totally 100% yes.

> - Should VOSpace support view=data on the /nodes resource?

I don't think this needs to be part of the core VOSpace specification - 
see below.

> - Is there an alternate way of offering a shareable download URL?  (By
> shareable I mean a URL that you can give collaborators and will work 
> for as
> long as the node exists.)

I think we should be able to meet all of the requirements for a 
shareable URL without changing anything in the core specification by 
defining a transport protocol which would do everything we need.

The VOSpace client would request a PullFrom transfer using the public 
share protocol and the VOSpace server would reply with an endpoint URL 
that end users can use to access the node content with a simple HTTP GET 
request. Users can email the URL to others or publish it in a paper, 
anyone who has the URL can download the node content.

Underneath, the VOSpace server is doing something like retrieving the 
node content from storage, formatting it according to the selected view 
and placing it somewhere where it can be published by a webserver, but 
the details of exactly how that is done is implementation specific. From 
outside, what happens is the client asks for a 'publicshare' transfer, 
and the server responds with a URL. No changes needed to the VOSpace 
specification.

Actually, that isn't quite true. Perhaps we do need to add something to 
the VOSpace specification.

The original intention was that it should be easy for people to create 
and register new transport protocols for VOSpace, which is why the 
identifier for transport protocols is defined as anyURI. In theory, this 
enables anyone to define and use a new transport protocol simply by 
creating a new URI.

In the VOSpace specification we defined a list of standard transport 
protocols and their official URIs. In retrospect this was probably a 
mistake. The next section in the document goes on to describe custom 
protocols, but the examples given are very theoretical and the damage is 
already done, most readers get as far as the list of standard transport 
protocols and don't realise they are allowed to add new ones.

So - to start the process going, I hereby define a new VOSpace transfer 
protocol.
http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpacePublicShare

If this does what we need, then I propose we replace the theoretical 
examples in the 'Custom protocols' section (3.5.4) with a real world 
example of using a wiki page to define a transport protocol. Note that 
the URL of the wiki page *is* the URI for referring to the protocol in 
the VOSpace message.

     curl "https://server.example.com/vospace/synctrans
         ?TARGET=vos://example.com~vospace/mydata1
         &DIRECTION=pullFromVoSpace
         
&PROTOCOL=http://wiki.ivoa.net/twiki/bin/view/IVOA/VOSpacePublicShare

The response should contain a simple HTTP GET endpoint URL that you can 
mail to your friends or publish in a paper.

     
<vos:endpoint>http://public.example.com/001237995/mydata1</vos:endpoint>

Defining a new transfer protocol really is that easy. Just that (AFAIK) 
no one has actually done it before.


Cheers,
Dave

--------
Dave Morris
Software Developer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------