ExecutionBroker and prior art.

[Salgado, Jesus]

Hi Paul,

Just to say that the coupling is by the definition of the ExecutionWorker in the ExecutionBroker that overlaps UWS. I think it is basically what you said and what I think it could be good to decouple although there are some problems that Dave can explain.

Yes, we can have a chat on this with Dave. As you know, the work on the ExecutionBroker is being developed in the context of the SRCNet so we can organize a meeting within one of the SRCNet architecture discussions to get your feedback accordingly in the spec.

Cheers,
Jesus

From: Paul Harrison <paul.harrison at manchester.ac.uk>
Date: Friday 30 August 2024 at 16:11
To: Jesus Salgado <jesus.salgado at skao.int>
Cc: Grid and Web Services WG <grid at ivoa.net>
Subject: Re: ExecutionBroker and prior art.




On 26 Aug 2024, at 11:34, Salgado, Jesus <jesus.salgado at skao.int> wrote:

Hi Paul,

I'm not entirely certain that the functionality of ExecutionBroker aligns precisely with the domain of UWS. ExecutionBroker is designed to assess the complexity of a specific execution on a particular science platform (or node) by considering execution requirements, available resources, permissions, data proximity, and more.

As I've discussed with Dave several times, there is a degree of coupling between the execution interface and ExecutionBroker so the execution interface defined in UWS is partially redefined within ExecutionBroker (Dave could provide more specific details). Due to the time-limited nature of "execution offers," complete decoupling is not feasible in his design. However, this is something I would be in favour of mitigate by a better decoupling if possible.

I am not sure what the coupling is really - that needs to be explained.



Rather than adding new functionality to UWS, I prefer to explore a more decoupled approach. This would involve using ExecutionBroker to identify available resources for an execution and employing UWS (potentially with updates in UWS 2.0) as the execution interface to execute the offer. This decoupling would result in clearer interfaces and simpler implementation, as we wouldn't need a monolithic service.

I am not suggesting adding the whole of the ExecutionPlanner interface to UWS - what I am saying is potentially add whatever is missing to achieve whatever the necessary coupling is.



It's true that there is some overlap in the definition of execution parameters and resource data models (e.g., CPUs, GPUs, memory). We might be able to reuse previous work (e.g. CEA) in this area. In fact, I also advocate for a separate data model that both UWS and ExecutionBroker can utilize for characterization.

I am all in favour of creating data models nowadays!



Let's discuss this further with Dave.

Of course the execution planning part is only of limited value if there is some sort of workflow runner that can make decisions based on what it gets back - it probably was not as obvious from the IVOA note, but the Astrogrid system did have a workflow system JES - described in this ADASS poster https://github.com/Javastro/astrogrid-legacy/blob/main/astrogrid/applications/docs/ADASS2004/PosterA3.pdf - in the end we did not attempt to bring any of the workflow part to the IVOA as we though it would be too difficult to get agreement, so only is the actual job execution interface - which got renamed UWS. However, I still feel that to properly design an execution planner there does need to be a simultaneous consideration of what will be the caller of the executionBroker.

Paul

The SKA Observatory is an inter-governmental organisation and the successor of SKA Organisation, a private limited company by guarantee registered in England and Wales with registered number 07881918, with a registered office of Jodrell Bank Observatory, Lower Withington, Macclesfield, Cheshire, England, SK11 9FT.


This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please inform the sender, and immediately and permanently delete the email. Do not use, copy or disclose the information contained in this message or in any attachment.


This email has been scanned for viruses and malware, and may have been automatically archived, by Mimecast Ltd. Although SKA Observatory and SKA Organisation have taken reasonable precautions to ensure no viruses are present in this email, neither SKA Observatory nor SKA Organisation accept responsibility for any loss or damage sustained as a result of computer viruses and the recipient must ensure that the email (and attachments) are virus free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20240902/26741bc5/attachment.htm>