VOstore -- context document?

[Guy Rixon]

Roy,

the main AstroGrid usage is simple. In an executing job, each job step is a
filter that maps zero or more inputs to zero or more outputs. Where the inputs
and outputs are clumps of data, the sources and sinks are VOStores. Thus, a
DAL service is a filter that maps zero input stores to one or more outputs. An
image-mosiacking service maps multiple input stores to one output store. A
visualiser, were it possible to run one from a workflow, would map one or more
input stores to zero output stores.

We're using the stores mainly as pipes between job-steps. Using stores instead
of real pipes (e.g. direct socket connections between services) satisfies some
of our design goals:

 - The system is tolerant of network outages.

 - Everything runs asynchronously; job-steps can be queued by
   services without upsetting other services that are waiting for the data.

 - Jobs that fail to conplete can be restarted from intermediate data products
   without having to start from the beginning.

 - In workflows where a few data items out of a flock have problems (e.g. the
   few FITS files with incomplete headers), the evidence of the problem stays
   around so that the users can check back.

 - There's an audit trail of intermediate data products by which a workflow
   can be checked.

 - The final products of a workflow are held in the system until the user
   calls for them.

The secondary use is as a place to keep user-accessible, private records of
what processing was done; the AstroGrid workflow executor keeps a log of what
was attempted and what happened. This, in principle, makes for better science,
as the results can be checked and the experiments reproduced.

I can write notes that discuss the other points in your email if you wish (but
not tonight).

Cheers,
Guy

On Thu, 5 May 2005, Roy Williams wrote:

> Wil, Dave, Guy
>
> I have been reading the VOStore document that you guys are authoring,
> and I look forward eagerly to the next version.
> http://www.ivoa.net/internal/IVOA/IvoaGridAndWebServices/vostore0.15.pdf
>
> So far it seems to be a specification of services and methods, but
> without much "real-world context". Perhaps there could be a second
> document, as part of the VO Architecture series of documents? If we are
> lucky, the technical and the explanatory documents would synchronize......
>
> By real-world context, I mean topics like this:
>
> -- use cases, what is this thing for
>
> -- scope and limits, what is it no for
>
> -- do customers see any vostore explicitly or is it all hidden in the
> backend?
>
> -- if the former, how do our customers leverage their knowledge of file
> systems and web browsers to make them comforatble and trusting of vostore
>
> -- what is the model of the data objects? Blobs only? Blobs and Tables?
> Mime-typed objects? Collections? Virtual data?
>
> -- how they (or their proxy) gets a certificate, and what motivates them
> to do so
>
> -- on how vostore relates to srb, webdav, etc, and why the VO is making
> something new rather than adopting something that already exists
>
> -- how vostore translates references from those other systems to vostore
> references
>
> -- on what it takes to turn a web server into a vostore server (so
> customer data can be exposed to VO services)
>
> -- on how to ensure that the metadata content matches the data content,
> while keeping the efficiency of indirect references
>
> -- how to use vostore in a workflow
>
> In particular, there are some threads from the NVO TechWG that have
> discussed some of these topics, see below (Be sure to click and repeat
> on "Next Message" for each, there is a lively conversation, not just a
> single message)
>
> I would like to see any material from Astrogrid or any other VO that
> discusses any of the "context" matters above, especially anything that
> might be the beginnings of an architecture/context document.
>
> Roy
>
> --------------------
>
> what is vostore and what is vospace
> http://www.us-vo.org/pipermail/techwg/2005-February/000728.html
>
> vostore as global file system (and no more)
> http://www.us-vo.org/pipermail/techwg/2005-February/000781.html
>
> suggested use cases for vostore
> http://www.us-vo.org/pipermail/techwg/2005-March/000814.html
>
> vostore metadata and organization
> http://www.us-vo.org/pipermail/techwg/2005-March/000830.html
>

Guy Rixon 				        gtr at ast.cam.ac.uk
Institute of Astronomy   	                Tel: +44-1223-337542
Madingley Road, Cambridge, UK, CB3 0HA		Fax: +44-1223-337523