VOSpace 2.1 issues

[Dave Morris]

On 2014-09-11 18:39, Matthew Graham wrote:
> 
> (2) There is also no specific route for an error in a capability
> execution to be reported back to a user. For example, suppose a node
> has the capability that a VOTable uploaded to it will automatically be
> converted into a database table somewhere. In theory, the data
> transfer should complete only when the capability has successfully
> executed rather than just when the byte transfer component has
> completed. However, if there is an issue with converting the VOTable
> to a database table then the transfer should indicate that it failed
> but with what error message? Internal server issue? We would like
> there to be an additional error message for capability issues.
> 

Apologies, I was concentrating on the first part of your message and 
missed the second part.

Technically, what you describe is not a node capability, but I agree we 
need better tools for reporting errors.

----

Node capabilities is way of making additional services and operations 
available on a per node basis.

Basically, a capability is just a URI identifying the type of service 
and an endpoint URL for invoking it.

In the example of a node that imports VOTable files into database 
tables, the node capabilities could include the TAP service endpoint for 
querying the database.

In the example of a node that provides SIA access to a collection of 
images, the node capabilities could include the SIA service endpoint for 
querying the collection.

In the example of a node containing a Zip archive, the node capabilities 
could include a service endpoint with a method for unpacking the 
archive.

In each case, the capability is invoked explicitly by the client, in a 
separate call to the capability endpoint, and the details of any errors 
resulting from the operation would be returned in the response to that 
call.

----

This behaviour is similar to the right click 'OpenWith' options on a 
desktop file system.

In the Zip file example, the user would drag/drop the zip file into a 
directory, and then right click and select the 'Extract contents' option 
to unpack it.

Two separate operations, move and unpack, each with their own separate 
error reporting mechanisms.

----

In your example of the VOTable import, the transfer and import are 
happening in a single operation.

There is a clear use case for this - if the VOTable is large then it 
would make the system much more responsive if the data is imported into 
the database as it arrives, rather than staging the VOTable file in the 
filesystem and then importing it into the database after the transfer 
has completed.

----

In this case, the special capability would be applied to the node before 
the transfer is invoked.

1) Create a new (empty) node
2) Invoke special capability to associate the node with a database
3) Transfer data into the node, importing the data into the database as 
it arrives

In fact, the special capability does not need to be applied directly to 
the node itself. The special capability might be applied to the parent 
container at a different time, which then applies to all the child nodes 
after that.

1) Create a new container node
2) Invoke special capability to associate the container with a database
....
3) Create a new node within the container
4) Transfer data into the node, importing the data into the database as 
it arrives

In both cases, the reading and interpreting of the VOTable occurs during 
the transfer, not when invoking the special capability.

So we need better error reporting on the transfers .... which are 
represented as UWS jobs.

----

In the current UWS specification :
http://www.ivoa.net/documents/UWS/20140527/WD-UWS-1.1-20140527.html

2.2.2.5. Error

"When an error occurs in a job the UWS must signal this at a minimum by 
setting the PHASE to error. In addition the <errorSummary> element, 
giving a brief summary of the error, should be included within the <job> 
element."

"The UWS may include a more detailed error message for example an 
execution log or stack trace by providing such a resource at the 
/{jobs}/(jobid)/error URI. It is the responsibility of the implementing 
service to specify the form that such an error message may take."

----

So our UWS job detail would contain :

     <errorSummary type='fatal' hasDetail='true'>
         <message>Some text goes here</message>
     </errorSummary>

And we have the option of providing something VOSpace specific at the 
job error URL.

     /{jobs}/(jobid)/error

----

In the current VOSpace specification :
http://www.ivoa.net/documents/VOSpace/20140805/WD-VOSpace-2.1-20140805.html

We do have a similar block of text that mirrors what the UWS spec says.

5.4.1.3 Faults
5.4.2.3 Faults
5.4.3.3 Faults
5.4.4.3 Faults

"For all faults (using the UWS mode), the service shall set the PHASE to 
"ERROR" in the Job representation. The <errorSummary> element in the Job 
representation shall be set to the appropriate value for the fault type 
and the appropriate fault representation (see section 5.5) provided at 
the error URI:http://rest-endpoint/transfers/(jobid)/error."

----

The current VOSpace specification does define a couple of values for the 
<errorSummary> that might apply to an error that occurs while importing 
a VOTable into a database table.

5.4.2.3 Faults
"Data format does not match the requested View"     "Invalid Data"      
"InvalidData"

5.4.4.3 Faults
"Data transfer does not complete"                   "Transfer Failed"   
"TransferFailed"

However, the current specification avoids describing any details about 
what the error resource provided at the UWS error URI should contain.

----

Having looked at the current VOSpace specification I think it probably 
needs a new look at the various 'faults' sections that describe the 
current error reporting mechanisms.

As part of the process, we should look at describing a way for a VOSpace 
service to include details of different transfer protocol errors and 
data validation errors in the resource at the UWS error URI.

     /{jobs}/(jobid)/error


Hope this helps,
Dave

--------
Dave Morris
Software Developer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------