[p3t] P3T mtg - Mon, Jun 24 @20 UTC

[Joshua Fraustro]

> I'm not sure if there is a good Python queuing system that supports an async interface but
> can run sync jobs with stop and timeout support.

We're currently using celery w/ redis at MAST for job queues/execution and have had good results, though I'm sure it doesn't fit your particular Rubin-specific quirks.

> * A Pydantic model for the job parameters with an alternate constructor
> that takes a list of UWS job parameters that match the semantics of the
> current XML protocol.

I developed a package for MAST called vo-models [https://vo-models.readthedocs.io/latest/] that provides Pydantic models for IVOA service protocols with 1-to-1 mappings to their XML representations via pydantic-xml. It supports round-tripping from an XML payload->pydantic model->dict/json->XML. We currently use it for our TAP/UWS service. I've written models for UWS & VOSI (Availability/Tables) with drafts for VODataservice and VOResource on the way. Much like you've described, when implementing it into our services, we have to sub-class the UWS Parameters model with whatever we expect to be specific for that service.

Maybe you might find it useful to skim over, maybe not! It comes with all the benefits of Pydantic/FastAPI models I'm sure you're aware of. I was (and still am) quite keen on covering as many IVOA standards as possible with the package, but I admit a small part of me hoped the efforts of the P3T would obviate the need for the package in the long term.

On 6/14/24, 10:22 PM, "p3t on behalf of Russ Allbery via p3t" <p3t-bounces at ivoa.net <mailto:p3t-bounces at ivoa.net> on behalf of p3t at ivoa.net <mailto:p3t at ivoa.net>> wrote:


External Email - Use Caution


"Dubois-Felsmann, Gregory P." <gpdf at ipac.caltech.edu <mailto:gpdf at ipac.caltech.edu>> writes:


> For those of us who aren't familiar with the specific Rubin SODA
> service, can you clarify whether your refactored service will include an
> async/UWS endpoint? It's implicit in what you wrote, I think, but it
> wasn't stated directly. (You did say you'd write a UWS spec.)


Oh, yes, sorry: our SODA service supports both sync and async, with sync
implemented on top of an async UWS job.


This is a bit in the weeds and probably only of interest to FastAPI Python
developers, but the current design that I think I have made work is that
the service provides only:


* A Pydantic model for the job parameters with an alternate constructor
that takes a list of UWS job parameters that match the semantics of the
current XML protocol.


* A model (which may be the same or different) for communicating the
parameters to a backend worker, and a method to do the conversion.


* FastAPI dependencies, one for POST and one for GET, that specify the
input parameters to the job in the normal FastAPI way (for good OpenAPI
spec generation) and convert them into lists of UWS parameters.


* A backend worker function that does the actual work of the service,
however it chooses to do so.


* An implementation of /availability and /capabilities if needed.


The UWS library then provides everything else: route handlers for both
sync and async plus all of the UWS interface, a library to track UWS job
status, all of the glue to dispatch jobs via the queuing library that we
use (arq currently) and collect their results, error handling, input
validation, etc. The library is very opinionated and therefore might not
be that useful outside of Rubin, since it does some rather strange things
to handle some Rubin-specific issues (such as allowing the backend worker
to run on an entirely different Docker image than all of the other service
components).


(Unfortunately the architecture that we're currently using has the very
serious limitation that it doesn't support stopping or timing out the
backend worker once it's started unless the backend worker is written as
async. This turned out to be a serious limitation in Python with the
combination of sync code and arq that I wasn't expecting. I'm not sure if
there is a good Python queuing system that supports an async interface but
can run sync jobs with stop and timeout support. I have some ideas for
how to solve this problem specifically at Rubin, but they're very
Rubin-centric.)


Work in progress is at <https://urldefense.com/v3/__https://github.com/lsst-sqre/vo-cutouts__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!0uoiwNAGnpQ9mANsUXRS254p8u24VkQTYMF-eLClc0X6z6jp6Fv9YHcPZm0019jtfDLMAc7P56Y$ <https://urldefense.com/v3/__https://github.com/lsst-sqre/vo-cutouts__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!0uoiwNAGnpQ9mANsUXRS254p8u24VkQTYMF-eLClc0X6z6jp6Fv9YHcPZm0019jtfDLMAc7P56Y$> >. The
work on the model API is not yet complete; that's the last major piece I
need to separate. The UWS library has not yet been extracted into a
reusable library and can be found in the src/vocutouts/uws directory for
the time being. Lifting it into a reusable library is the first thing I
will be working on when I get back from vacation. The UWS library has
some protocol compliance bugs that I still need to fix.


--
Russ Allbery (eagle at eyrie.org <mailto:eagle at eyrie.org>) <https://urldefense.com/v3/__https://www.eyrie.org/ <https://urldefense.com/v3/__https://www.eyrie.org/>*eagle/__;fg!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!0uoiwNAGnpQ9mANsUXRS254p8u24VkQTYMF-eLClc0X6z6jp6Fv9YHcPZm0019jtfDLMMJc1Qcg$ >
--
p3t mailing list
p3t at ivoa.net <mailto:p3t at ivoa.net>
https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!0uoiwNAGnpQ9mANsUXRS254p8u24VkQTYMF-eLClc0X6z6jp6Fv9YHcPZm0019jtfDLMSs-lbV8$ <https://urldefense.com/v3/__http://mail.ivoa.net/mailman/listinfo/p3t__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!0uoiwNAGnpQ9mANsUXRS254p8u24VkQTYMF-eLClc0X6z6jp6Fv9YHcPZm0019jtfDLMSs-lbV8$>