ExecutionBroker and prior art.

[Paul Harrison]

> On 26 Aug 2024, at 11:34, Salgado, Jesus <jesus.salgado at skao.int> wrote:
> 
> Hi Paul,
> 
> I'm not entirely certain that the functionality of ExecutionBroker aligns precisely with the domain of UWS. ExecutionBroker is designed to assess the complexity of a specific execution on a particular science platform (or node) by considering execution requirements, available resources, permissions, data proximity, and more.
> 
> As I've discussed with Dave several times, there is a degree of coupling between the execution interface and ExecutionBroker so the execution interface defined in UWS is partially redefined within ExecutionBroker (Dave could provide more specific details). Due to the time-limited nature of "execution offers," complete decoupling is not feasible in his design. However, this is something I would be in favour of mitigate by a better decoupling if possible.

I am not sure what the coupling is really - that needs to be explained.

> 
> Rather than adding new functionality to UWS, I prefer to explore a more decoupled approach. This would involve using ExecutionBroker to identify available resources for an execution and employing UWS (potentially with updates in UWS 2.0) as the execution interface to execute the offer. This decoupling would result in clearer interfaces and simpler implementation, as we wouldn't need a monolithic service.

I am not suggesting adding the whole of the ExecutionPlanner interface to UWS - what I am saying is potentially add whatever is missing to achieve whatever the necessary coupling is.

> 
> It's true that there is some overlap in the definition of execution parameters and resource data models (e.g., CPUs, GPUs, memory). We might be able to reuse previous work (e.g. CEA) in this area. In fact, I also advocate for a separate data model that both UWS and ExecutionBroker can utilize for characterization.

I am all in favour of creating data models nowadays!

> 
> Let's discuss this further with Dave.
> 
Of course the execution planning part is only of limited value if there is some sort of workflow runner that can make decisions based on what it gets back - it probably was not as obvious from the IVOA note, but the Astrogrid system did have a workflow system JES - described in this ADASS poster https://github.com/Javastro/astrogrid-legacy/blob/main/astrogrid/applications/docs/ADASS2004/PosterA3.pdf - in the end we did not attempt to bring any of the workflow part to the IVOA as we though it would be too difficult to get agreement, so only is the actual job execution interface - which got renamed UWS. However, I still feel that to properly design an execution planner there does need to be a simultaneous consideration of what will be the caller of the executionBroker.

Paul




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20240830/61f1dbd5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2893 bytes
Desc: not available
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20240830/61f1dbd5/attachment.p7s>