Question about VOSpace and cold storage

[Brian Major]

Hi Sonia,

On Thu, Dec 10, 2020 at 5:34 AM Zorba, Sonia <sonia.zorba at inaf.it> wrote:

> However the job requires to specify a protocol to use, but in this case it
> is an internal operation, so we thought to use a custom protocol with no
> endpoints, for example:
>
> <vos:transfer xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0"
> version="2.1">
>   <vos:target>vos://example.com!vospace/mydata1</vos:target>
>   <vos:direction>pullToVoSpace</vos:direction>
>   <vos:protocol uri="urn:tape-recall"></vos:protocol>
> </vos:transfer>
>
> In the beginning we thought to perform this operation as part of the
> asynchronous download (pullFromVoSpace), but in that case the job had to
> stay in the pending phase until the files were copied to the disk, with no
> possibility to check the status of the copy. Rethinking about it, it was
> clear that this copy operation is conceptually different from the download,
> so it has to be handled as a separate job, that is preparatory for the
> download. In that case using the pullToVoSpace operation seemed more
> appropriate.
>
> Do you think that this is the correct approach to use? Did someone build
> something similar?
>

Our implementations of VOSpace at CADC don't yet implement pullToVoSpace.
So, I can't speak from experience, but I believe it is absolutely the
correct vospace 'direction' to use to implement your tape import
requirement.

Regarding the 'phase' in the asynchronous pullFromVoSpace operation:  the
specification is probably not clear enough about how the jobs phases
reflect transfer negotiation and the resulting byte transfer.  With our
'vault' implementation (the one based on distributed object storage) we
took a different approach:  our jobs are in the EXECUTING phase during byte
transfer, and then are set to COMPLETE or ERROR when that's done.  However,
this is problematic as the service handling the byte transfer must make a
callback to vospace to set the final phase.  This introduced an undesirable
coupling between vospace and the service handling the byte transfer.  Also,
since that callback can fail, we need to monitor and correct those failures
in an out-of-band process.  If we get a chance to refactor this we will
probably choose to set the phase to COMPLETE or ERROR immediately after
transfer negotiation is complete, and consider the ensuing byte transfer an
operation that is outside the scope of the VOSpace protocol.

Cheers,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20201211/354b9b28/attachment.html>