Workflow

[Guy Rixon]

Roy,

I think there are two sets of standards involved with workflow: the contract
with the workers and the contract with the bosses.

Presume that we have some kind of agent programme that orchestrates the
workflow. The agent is instructed by some other software (UI/application)
and calls on various services to do the steps of the workflow.

There may be some standard for (aspects of) how the workflow agent calls the
workers services. In AstroGrid, this is part of CEA and in IVOA the
asynchronous-activities standard is relevant.

There may also be a standard for how the UI/application expresses a desired
workflow to the workflow agent. This is where BPEL etc apply.

We _should_ have standards for how workflow agents talk to worker services. We
_need not_ have standards for how applications talk to workflow agents (but it
would be useful if we did). They are separate issues. We need to tackle the
agent-to-worker end first, as there will be many more workers than agents.

One problem with stadnardizing the application-to-agent interface is that
there are many kinds of agents. The most-basic just executes the workflow as a
script. More-advanced agents have more autonomy and more scope. It's very hard
to build a common standard for instructing different kinds of agent, and we
don't want to generate multiple standards.

Regards,
Guy

On Wed, 19 Jan 2005, Roy Williams wrote:

> I think there are two sorts of standard that we should be thinking about:
>
> (1) How workflow components interact with their service container, and
> (2) How a client interacts with an asynchronous service.
>
> In more detail
>
> (1) I recall the situation with message-passing and parallel computers about 15 years
> ago: each vendor had their own API to their specific messaging system, and
> consequently the development of parallel computing was held back, because nobody want
> to invest the time making code that might become obsolete. The development of the
> standard MPI really made parallel computing fly because every code would run on every
> machine. I think we are now in a similar situation with workflow systems. Each
> workflow vendor has their own API for how the component interatcs with the framework:
> Kepler, Chimera/Pegasus, Astrogrid/CEA, Globus4, DAGMan, Viper, Opticon/ESO, WSRF,
> etc etc. Personally, I would find it very difficult to devote serious effort to
> building components without knowing which will survive. Of course, the real problem
> may be that I simply lack enough understanding of the subtleties to see that these
> are all completely different animals -- in that case I would like my ignorance
> banished.
>
> (2) When I interact with the batch queue on my cluster, I use the Unix commands qsub,
> qstat, qdel to submit, monitor, and kill jobs. I would like an analogous, standard,
> way to use an asynchronous web service, so that a single client code can interact
> with different services. I guess the conversation includes getting a sessionID,
> specifying parameters and inputs, starting up the job, then either the server
> notifies the client, or the client montors the service, and finally there is fetching
> a result. I would guess this is all in the WS-Something specification. And I seem to
> recall Guy Rixon posting something like this a year or so ago.
>
> Roy
>
> --------
> California Institute of Technology
> roy at caltech.edu
> 626 395 3670
>

Guy Rixon 				        gtr at ast.cam.ac.uk
Institute of Astronomy   	                Tel: +44-1223-337542
Madingley Road, Cambridge, UK, CB3 0HA		Fax: +44-1223-337523