R: Announcement about Workflow future plans

[Douglas Tody]

Hi Andr? Jose, all -

As Pepi mentions a lot of us already have data processing pipelines,
which are a type of workflow.  This raises the question, what is the
expected scope of the proposed VO workflow initiative?  It seems to me
there are at least several distinct areas this might address:

     o   Distributed data analysis workflows.  In this case the user or
         a client defines and executes a distributed workflow which
         invokes services on multiple remote sites via the VO
         infrastructure.  In this case the workflow would be entirely
         in VO-space, driving simpler services at the individual sites.

     o   Driving data processing pipelines and reprocessing from VO.
         In this case we have a traditional data processing pipeline and
         the remote user or client software invokes a job to do some
         pipeline reprocessing, e.g., to custom reprocess an instrumental
         dataset to produce a new image, cube, etc.  The "workflow" in
         this case runs at a single site, and VO is used to drive the job
         remotely (SSO, UWS) and manage the results (VOSpace, VO data
         services).

     o   Traditional data processing pipelines, e.g., instrumental or
         survey data processing pipelines which produce higher level
         data products.  At present there are many variants of these
         and they have little or no direct connection to VO, aside from
         possibly producing VO-compliant data or being optionally driven
         from VO.

The first of these is the first thing that comes to mind (to me at
least) when one mentions "workflow" in connection with VO.  The second
integrates the traditional data processing pipelines we already have
with VO to allow VO users to do on-the-fly reprocessing to generate data
products which can be analyzed with VO (we are very interested in this
use case for custom reprocessing of observatory data for example).  The
third currently has little connection with VO and it is not clear how
much we need VO mechanisms at this level, although there are some things
which come to mind (VO compliant data and metadata, modeling provenance,
etc.).

So my question is thus which of these areas would a VO workflow
initiative want to address?  What are the relevant priorities?  Do we
still have a fairly clear distinction between instrument-specific data
processing and (possibly distributed/multi-wavelength) data analysis,
where workflows are concerned?  More pragmatically, do we want to try to
get the folks at our respective institutions currently responsible for
conventional data processing pipelines involved, or is this expected to
be a higher level more VO-specific workflow mechanism?

 	- Doug




On Tue, 5 Jul 2011, Pepi Fabbiano wrote:

> Dear Peppe and Andre',
>
> There is a considerable experience in workflows (we call them pipelines) at 
> the CXC, although we realize that  the model will have to change somewhat to 
> adapt to the increasing data rates. It would be good to include Janet Evans 
> and Ian Evans in the debate. I'll cc them.
>
> Cheers. -pepi
>
>
>
> Giuseppe Longo wrote:
>> Dear André 
>> I believe this is a great and timely idea. I am very interested in
>> contributing (and I also volunteer Massimo Brescia who has a strong
>> Interest in it) in several aspects of the task. How do you intend to 
>> proceed?
>> Peppe
>> 
>> -----Messaggio originale-----
>> Da: interop-bounces at ivoa.net [mailto:interop-bounces at ivoa.net] Per conto di
>> Andre Schaaff
>> Inviato: venerdì 1 luglio 2011 15:31
>> A: interop at ivoa.net
>> Oggetto: Announcement about Workflow future plans
>> 
>> 
>> Dear all,
>> 
>> As you know we will soon be facing a new generation of facilities and
>> archives dealing with huge amounts of data (ALMA, LSST, panSTARRS, LOFAR,
>> SKA pathfinders,..) where scientific Workflows will play an important role
>> in the working methodology of astronomers.  A detailed analysis about the
>> state of the art of workflows in the frame of the VO involves languages,
>> design tools, execution engines, use cases, etc. A major topic is also the
>> preservation of the workflows and the capability to replay a workflow
>> several years after its design and implementation. Several talks concerning 
>> these issues have been presented during the past
>> IVOA Interop meetings (see references below).
>> 
>> In order to undertake this task within our community we think that as a
>> first step a Note should be written. Participation is welcome, in
>> particular, we would like to collect experiences (including use cases,
>> tools, etc.), references, remarks, etc. We plan the Note to be published
>> around September and discussed at the Pune Interop meeting. We should then
>> decide on starting a working draft describing those aspects and possibly in
>> a new Interest Group on Workflows.
>> 
>> Best regards,
>> 
>> 
>> André Schaaff and Jose Enrique Ruiz
>> 
>> 
>> References (not exhaustive)
>> 
>> Some presentations in the VO
>> 
>> Naples DCP session,
>> http://www.ivoa.net/internal/IVOA/InteropMay2011DCP/Wf4EverNaples.pdf
>> Nara DCP session,
>> http://www.ivoa.net/internal/IVOA/InterOpDec2010DCP/Wf4Ever.pdf
>> Garching GWS session,
>> http://www.ivoa.net/internal/IVOA/InterOpNov2009GWS/Garching-GWSWorkflow.pdf
>> Baltimore Application session,
>> http://www.ivoa.net/internal/IVOA/InterOpOct2008Applications/GWS-DM-REG-3010
>> 08.pdf
>> Trieste GWS session,
>> http://www.ivoa.net/internal/IVOA/InterOpMay2008GridAndWebServices/GWS-Chara
>> c-Workflow-20May08.pdf
>> 
>> Some projects with a link to the VO
>> 
>> http://www.france-ov.org/twiki/bin/view/GROUPEStravail/Workflow
>> http://amiga.iaa.es/p/212-web-services-based-scientific-workflows.htm
>> http://www.wf4ever-project.org/web/guest/home
>> 
>> Euro-VO DCA Theory and Grid Workshops
>> 
>> http://sait.oat.ts.astro.it/MSAIt800209/PDF/559.pdf
>> http://sait.oat.ts.astro.it/MSAIt800209/PDF/574.pdf
>> http://sait.oat.ts.astro.it/MSAIt800209/PDF/578.pdf
>> 
>> ADASS
>> 
>> ADASS London,
>> http://www.adass.org:8080/Conferences/2007/Venue/talks/session4a/adass2007-s
>> chaaff-final.pdf
>> ADASS London, http://adsabs.harvard.edu/abs/2008ASPC..394..273J
>> 
>>
>> 
>
>