Minutes of 20/09/23 IVOA GWS Meeting - Science Platforms Federated Execution

Fri Sep 22 15:47:29 CEST 2023

Hi Francois,

This is a very good question. We were discussing in which level we could involve the IVOA registry in the process (it was not mentioned the other day) and, in principle, the proposed Execution Planner approach could have a dedicated extension in the registry to allow registration (exactly as you are pointing out).

If the approach changes and we try to have realtime information of the status of the science platform (like, e.g. available CPUs in one particular instant), the IVOA registry could be not the preferred option due to latencies on the updates. This Information System approach (a service that is compiling information of the status of all the available science platform nodes) was discussed in the ambit of the SRCNet but it does not look the best approach for the IVOA and, also, the IVOA registry could not be a good candidate to gather realtime info like the one required for an information system.

In summary, to have the Execution Planner services (as currently defined) registered looks to be a good solution so, if there are not significant changes, we would need to coordinate with the registry WG this point whenever needed.

Thanks,
Jesus

[SKAO Horizontal Dividing Rule]

[SKAO Logo]

[SKAO Vertical Dividing Rule]

Jesús Salgado
SKA Regional Centre Architect
jesus.salgado at skao.int<mailto:jesus.salgado at skao.int>
www.skao.int<https://www.skao.int/>
SKA Observatory
Jodrell Bank, Lower Withington,
Macclesfield, SK11 9FT, UK

From: BONNAREL FRANCOIS <francois.bonnarel at astro.unistra.fr>
Date: Friday, 22 September 2023 at 12:49
To: Jesus Salgado <jesus.salgado at skao.int>, "grid at ivoa.net" <grid at ivoa.net>
Cc: src-olive <src-olive at skao.int>, src-orange <src-orange at skao.int>, src-magenta <src-magenta at skao.int>, "Bertocco, Sara" <sara.bertocco at inaf.it>, Dave Morris <dave.morris at metagrid.co.uk>
Subject: Re: Minutes of 20/09/23 IVOA GWS Meeting - Science Platforms Federated Execution

Dear Jesus, Dave and Sara,
Thank you for your minutes and email.
I'm sorry i was not able to attend.
I have  a question : how do we register computing services in the VO registry.
Is there a capability "execution planner interface" to be added to some of the services ? Or can we imagine one in a ComputerServiceRegistryExtension maybe ?
Something which could read : "well, this service has an execution planner interface. So go to this endpoint and it will give you the details."
Or do we integrate the execution planner interface metadata in the registry metadata for corresponding services ?
I think the document answers in the direction of the first solution.
Right ?
Cheers
François
Le 21/09/2023 à 12:25, Salgado, Jesus a écrit :
Dear all,

Thanks for the big attendance and the productive discussion during yesterday’s GWS meeting on Federated Execution across astronomical science platforms kick-off meeting.

Minutes of the meeting are at:
https://wiki.ivoa.net/twiki/bin/view/IVOA/GWSTelecon20230920

It was presented a summary of the issues to be address and Dave Morris presented the status of the Execution Planner, the first standard we want to promote within the team.

As this was a kick-off session, we have identified some points that could be further discussed in the mailing list or during next interop session in Tucson.

  1.  Are we replicating work already solved by the Grid community?
Our initial answer is that the grid (or later cloud communities) has some agreed standards that we need to study to introduce concepts but that are not usually software stack agnostics. In the same way we defined UWS (what is a job scheduler when many job schedulers are present in many other contexts), the abstraction layer of the Execution Planner could be implemented with different software stacks below. In any case, of course we need to take attention to other standard ways to solve these problems.

  1.  Status of Execution Planner and missing points?
Note is quite complete. It must be addressed the data model to characterise capabilities of the platforms and characterisation of the workflows (in the general sense). This data model could be part of the Execution Planner standard (not a separate document) and, also, it should address data location (this is an open issue)

  1.  Why not to create a layer on top of orchestrators? (Most of science platforms already have, e.g. a Kubernetes cluster)
Although it is true that some technologies are quite standard for most of the science platforms, technologies could change, and older platforms could have different set-ups. This is why IVOA protocols are usually software stack agnostics so they could be implemented by all. In any case, it is important to know the level of heterogeneity of the different partners to identify if some things could be simplified assuming that compatible technologies.

  1.  Could IVOA endorse or accept S3 as a data transfer protocol?
Not clear answer. Interesting to be discussed in the mailing list.

  1.  Next steps?
The more important step is to identify who is willing to help on the prototyping as, in this particular case, the prototyping will clarify a lot about possible missing points on the execution planner definition.

Best Regards,
Jesus Salgado, Sara Bertocco and Dave Morris

[SKAO Horizontal Dividing Rule]

[SKAO Logo]

[SKAO Vertical Dividing Rule]

Jesús Salgado
SKA Regional Centre Architect
jesus.salgado at skao.int<mailto:jesus.salgado at skao.int>
www.skao.int<https://www.skao.int/>
SKA Observatory
Jodrell Bank, Lower Withington,
Macclesfield, SK11 9FT, UK

The SKA Observatory is an inter-governmental organisation and the successor of SKA Organisation, a private limited company by guarantee registered in England and Wales with registered number 07881918, with a registered office of Jodrell Bank, Lower Withington, Macclesfield, Cheshire, England, SK11 9FT.

This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please inform the sender, and immediately and permanently delete the email. Do not use, copy or disclose the information contained in this message or in any attachment.

This email has been scanned for viruses and malware, and may have been automatically archived, by Mimecast Ltd. Although SKA Observatory and SKA Organisation have taken reasonable precautions to ensure no viruses are present in this email, neither SKA Observatory nor SKA Organisation accept responsibility for any loss or damage sustained as a result of computer viruses and the recipient must ensure that the email (and attachments) are virus free.

The SKA Observatory is an inter-governmental organisation and the successor of SKA Organisation, a private limited company by guarantee registered in England and Wales with registered number 07881918, with a registered office of Jodrell Bank Observatory, Lower Withington, Macclesfield, Cheshire, England, SK11 9FT.

This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please inform the sender, and immediately and permanently delete the email. Do not use, copy or disclose the information contained in this message or in any attachment.

This email has been scanned for viruses and malware, and may have been automatically archived, by Mimecast Ltd. Although SKA Observatory and SKA Organisation have taken reasonable precautions to ensure no viruses are present in this email, neither SKA Observatory nor SKA Organisation accept responsibility for any loss or damage sustained as a result of computer viruses and the recipient must ensure that the email (and attachments) are virus free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2274 bytes
Desc: image001.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0006.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 7129 bytes
Desc: image002.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0007.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 1650 bytes
Desc: image003.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0008.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.jpg
Type: image/jpeg
Size: 2275 bytes
Desc: image004.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0009.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 7130 bytes
Desc: image005.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0010.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.jpg
Type: image/jpeg
Size: 1651 bytes
Desc: image006.jpg
URL: <http://mail.ivoa.net/pipermail/grid/attachments/20230922/1a2b9f59/attachment-0011.jpg>