TAPRegExt: sync/async preference

[Patrick Dowler]

I know a little more about what Rubin is doing in their TAP service and I'm
not sure static declarations in TAPRegExt can/will really capture the subtle
nature of the service behaviour.

In a nutshell: some sync queries are executed immediately/normally and return
results. Other queries are always put into a queue and executed in async mode
under the hood. We/they are also trying to figure out how the sync request can
block (until the job actually finishes) and probably how to tell
clients to keep retrying
until that happens.

note: specifically in this case, tap_schema queries execute in the
simple/direct/normal
way and queries on the other content are performed in async mode.

So this is not just a "preferred mode" but really a behaviour of the
implementation that
is currently opaque to the client. With the current state of the TAP
API, I don't think
a solution where it remains opaque is going to work very well. I can
see roughly two
kinds of solutions:

1. We more carefully define TAP-sync patterns to make blocking and retry
a part of the API so clients can be written to deal with it successfully
2. We more carefully define TAP-sync to be able to respond with "this
request is actually
executing in async mode, here is the UWS job URL"

I haven't thought either of these through all that much, but my gut
feeling is that from a
standard/clarity point of view, option 2 is simpler and option 1 could
be quite complex.

I think a feature-rich client like topcat and pyvo could also deal
with option 2 pretty easily, but that
quickly becomes a lot less true for simple/custom scripts or basic
command-line usage of curl, etc.

--
Maybe higher level questions to think about:
- does this kind of thing make TAP-sync a second class endpoint? it
would be simpler to implement
but worse for a lot of uses
- is this sync vs async more general that TAP? are there other
services/requests where the server
wants/needs to choose how it will execute?

--
Patrick Dowler
Canadian Astronomy Data Centre
Victoria, BC, Canada

On Mon, 7 Jul 2025 at 04:44, Mark Taylor via dal <dal at ivoa.net> wrote:
>
> Markus et al.,
>
> [I have followed up to the dal list only and not to registry
> to reduce noise and confusion; DAL anyway owns TAPRegExt as
> well as TAP]
>
> On Mon, 7 Jul 2025, Markus Demleitner via registry wrote:
>
> > Basically, Rubin would like to be able to tell TAP clients: "Prefer
> > async mode on this service".  I can totally see how, say, TOPCAT,
> > could pre-set the Mode combo box based on that, and that's enough to
> > convince me it's a sensible feature.
>
> Before offering an opinion, I'd like to understand better exactly
> what Rubin or other services might mean by "prefer async mode",
> or why they might want to express that.  Does it really mean
> "sync is deprecated here, and although we implement it because
> it's mandatory in the spec there's some reason why it's really
> a bad idea to use it for this service", or just "sync imposes
> limits that mean long-running queries may not complete"
> (which is anyway generally understood for sync)?
>
> For instance when topcat first hits a TAP service one thing it
> does is issue the following synchronous query:
>
>    SELECT COUNT(*) AS nrow FROM TAP_SCHEMA.columns
>
> so it can figure out whether it's a big or small service to
> inform some later decisions about metadata acquisition.
> The strong expectation is that this query (and some other
> under-the-hood ones that it makes) is going to execute fast,
> so the additional overhead on the client side of making an
> asychronous query is an unnecessary burden.
>
> If we implement some way for services to say "prefer async mode"
> does it mean that such quickie queries must/should be submitted
> async?  On the other hand, if a service says "prefer sync mode",
> does it mean that a query I know is going to exceed a normal
> HTTP timeout must/should be submitted sync?
>
> Mark
>
> --
> Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK
> m.b.taylor at bristol.ac.uk          https://www.star.bristol.ac.uk/mbt/