<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear DAL,</p>
<p><br>
</p>
<p>This is a very interesting discussion. While I don’t have a
strong opinion on the async vs sync mode debate, I am particularly
interested in the topic of defining different limits, such as
execution time and output size, depending on the type of "mode" (async
or sync mode) and whether the user is authenticated or anonymous.</p>
<p><br>
</p>
<p>At CEFCA/OAJ, we already apply different output limits depending
on the execution mode, but currently there is no standard way to
communicate this to clients. Additionally, in the future, we would
also like to define different limits based on the user’s
authentication status. So having a standard mechanism to declare
these variations would therefore be very helpful for us.</p>
<p><br>
</p>
<p>In this context, I believe Markus’s proposal of introducing
"limits" blocks could be a very good starting point to address
this.</p>
<p><br>
</p>
<p data-start="1107" data-end="1189">Thank you for your efforts and
for considering this point. <br>
</p>
<p data-start="1107" data-end="1189"><br>
</p>
<p data-start="1107" data-end="1189">Best regards,</p>
<p data-start="1107" data-end="1189"><br>
</p>
<p data-start="1107" data-end="1189">Tamara<br>
</p>
<p data-start="1191" data-end="1222"><br>
</p>
<p></p>
<div class="moz-cite-prefix">El 17/7/25 a las 20:27, Stelios
Voutsinas via dal escribió:<br>
</div>
<blockquote type="cite"
cite="mid:6a7e7897b24b49bcb8c6675d213e3f45@lsst.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div id="divtagdefaultwrapper"
style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"
dir="ltr">
<div>Dear DAL,</div>
<div><br>
Some thoughts on this interesting discussion:<br>
<br>
</div>
<div>I do like the idea of Pat's "option 2" and Paul's
504-with-job-URL approach. </div>
<div><br>
</div>
<div>For services with resource constraints, I think this
addresses a real operational problem.</div>
<div>In our case for example, we run sync queries asynchronously
underneath in an event-based architecture, but supporting sync
forces us to keep HTTP connections open, tying up connection
pools and memory for what are fundamentally async operations.
The continue-as-async pattern would let us be honest about our
architecture by starting the query immediately and then
releasing the connection if it doesn't complete quickly versus
maintaining expensive synchronous operations that don't scale
with user load.</div>
<div><br>
</div>
<div><br>
</div>
<div>On the question of why not just use async?</div>
<div>I think there are good reasons clients may choose to
default to sync beyond just simplicity, since sync is
stateless and perhaps slightly resource-efficient for clients,
i.e one HTTP request, one response, done <span
style="font-size: 12pt;">versus creating jobs, polling
loops, cleanup logic and maintaining state across multiple
requests. </span></div>
<div><br>
</div>
<div>If the majority of the queries complete within the sync
timeout limits for most services sync's resource efficiency
makes it a reasonable choice to default.</div>
<div>In fact as far as I can see pyvo (search method) and Topcat
default to sync, but then users hit service timeouts on
queries that could complete if given more time.</div>
<div><br>
</div>
<div>Also I'm not sure if I would classify this as a breaking
change. This actually improves the failure mode rather than
breaking it from my perspective. </div>
<div><br>
</div>
<div>Currently: sync timeout -> cryptic error -> user
frustration (?). </div>
<div>With continue-as-async: sync timeout -> clear "query
running, check here" message. </div>
<div><br>
</div>
<div>For "dumb" clients, the user would just get a VOTable back
with a message indicating that the job is still executing, and
a link to the async job instead of a 504 or cryptic message,
whereas smart clients can auto-switch to polling. The success
cases remain identical, so I'm not sure if this would be
breaking anything.</div>
<div><br>
</div>
<div>In terms of the implementation complexity, I do think this
would mainly benefit services that run sync jobs as async in
the background, and for those I think the implementation seems
straightforward.</div>
<div>For other services which would require architectural
changes to support this, I think the optional nature of this
allows them to choose whether to spend the effort implementing
it. Services could also check whether the user has polled the
async endpoint to make the decision whether to stop the query
in this case as well if there is concern about computational
load of keeping the query running in this case.</div>
<div><br>
</div>
<div><br>
</div>
<div>The fundamental issue I think is that sync mode is
structurally limited by HTTP timeouts and connection pools,
but potentially can be seen as a good starting point for both
users and client libraries. At the same time there is always
the question of how visible the TAP execution mode
(async/sync) should be to the user vs simplifying the process
and number of steps they need to go through get the results
they are interested in.</div>
<div><br>
</div>
<div><br>
</div>
<div>I do see an argument for why not just improve the error
messages pointing users to async. But I still see a couple
limitations that are not addressed if we go that route, which
is that the UX experience still isn't great for users of a
service that has very low sync limits. They'd have to remember
to switch to async mode each time they try to use the client,
and in some cases if they are unaware of the protocols it may
not be obvious what they have to do. Also we double the
computational load, since each query is essentially run twice.
Perhaps this is where Markus' idea comes in of advertising
limits and a preference for run mode. Where this gets tricky
from my point of view is for TAP_SCHEMA queries as mentioned,
where we know they will complete quickly and thus they follow
a different execution path where sync makes sense.</div>
<div><br>
</div>
<div><br>
</div>
<div>In general for our service both "continue-as-async" and the
"preference-in-the-capabilities" approach would help, assuming
clients like pyvo/topcat would implement changes to either
follow the redirect or switch the execution mode accordingly,
slightly leaning towards Option 2 because of the flexibility
and the TAP_SCHEMA limitations of the metadata approach. But
at the same time Marcus's original proposal is probably the
more friction-less approach so I think that would also be a
good way forward if the alternative is too contentious.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Stelios Voutsinas</div>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
dal <a class="moz-txt-link-rfc2396E" href="mailto:dal-bounces@ivoa.net"><dal-bounces@ivoa.net></a> on behalf of Grégory
Mantelet via dal <a class="moz-txt-link-rfc2396E" href="mailto:dal@ivoa.net"><dal@ivoa.net></a><br>
<b>Sent:</b> Thursday, July 17, 2025 2:02 AM<br>
<b>To:</b> Paul Harrison<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:dal@ivoa.net">dal@ivoa.net</a><br>
<b>Subject:</b> Re: TAPRegExt: sync/async preference</font>
<div> </div>
</div>
<div>Thank you Paul for this suggestion. I just commented on
the original idea.<br>
<br>
Anyway, with the suggestion you've proposed, it means that a
service able to<br>
provide such advanced/smart sync response would have to:<br>
<br>
- either always run the query asynchronously<br>
(even though the sync layer said it timed out, it would
keep running in the<br>
potential hope someone may click on the "wait for the
response" link and<br>
switch on the async endpoint)<br>
<br>
- or will have to restart the query if the user click on the
"wait for the response"<br>
link<br>
<br>
Maybe there is another technical server solution. If not,
any of these solutions<br>
convince me that this server complexity worth the effort.
Let's keep things<br>
simple.<br>
<br>
I still think that such behavior should be applied on client
side. And that would<br>
be even more possible thanks to what Stelios has just
proposed with a<br>
standardized error type in DALI answers. If a sync query
fails with a time out<br>
error, the client can propose to try again in async mode.
Then, as I said, a client<br>
may also decide to run everything on async mode and show
immediately<br>
the result if it succeeded quickly (as TAPHandle does).<br>
<br>
Cheers,<br>
Grégory<br>
<br>
<br>
<div class="moz-cite-prefix">On 15/07/2025 18:22, Paul
Harrison wrote:<br>
</div>
<blockquote type="cite"><br
id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On 15 Jul 2025, at 14:41, Gregory MANTELET via
dal <a class="moz-txt-link-rfc2396E"
href="mailto:dal@ivoa.net" moz-do-not-send="true">
<dal@ivoa.net></a> wrote:</div>
<br class="Apple-interchange-newline">
<div><br
style="font-family:Helvetica; font-size:15px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span
style="font-family:Helvetica; font-size:15px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">To
be honest, I don't like either this false-</span><i
style="font-family:Helvetica; font-size:15px; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">sync</i><span
style="font-family:Helvetica; font-size:15px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"><span
class="Apple-converted-space"> </span>endpoint.
Changing that way</span><br
style="font-family:Helvetica; font-size:15px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span
style="font-family:Helvetica; font-size:15px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">the
behavior of this endpoint is clearly a breaking
change</span></div>
</blockquote>
</div>
<br>
<div>What I suggested is a backwards compatible extension
for “dumb" clients assuming that they always treat a
non-2xx http response is an error. It is still a sync
endpoint that can time-out (which could happen in the
current UWS/TAP standard definitions) - it is just that
the timeout *could* contain information that would allow
a slightly less dumb client to see that it might be
possible still to obtain the result if they are prepared
to wait longer and do the full async protocol. Simple
clients do not have to do any more work than before.</div>
<div><br>
</div>
<div>Paul.</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<title></title>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta http-equiv="content-style-type" content="text/css">
<style type="text/css">
#signature {
color:#1f497d;
font-family: Calibri, sans-serif;
margin-top: 15px;
}
#signature strong {
font-size: 12pt;
}
#signature .cargo {
font-size: 10pt;
margin: 15px;
}
#info {
margin-top: 15px;
font-size: 12pt;
font-family: Tahoma, sans-serif;
}
#info ul {
list-style-type: none;
font-size: 8pt;
}
#info .label {
color:#1f497d;
}
#legal {
font-family: Arial, sans-serif;
font-size: 7pt;
}
</style>
<div id="signature">
Tamara Civera Lorenzo<br>
<span class="cargo">Scientific Database Engineer</span>
</div>
<div id="info">
<table>
<tbody>
<tr>
<td> <img src="cid:part1.pSlhO4qw.tIiDfOmj@cefca.es"
alt="Logotipo CEFCA"> </td>
<td> <strong>Centro de Estudios de Física del Cosmos de
Aragón</strong>
<ul>
<li>Plaza San Juan nº 1, piso 2º</li>
<li>44001 Teruel</li>
<li><span class="label">Tfno:</span> +34978221266 - <span
class="label">Ext: 1116</span> </li>
<li><span class="label">Fax:</span> +34978602334 </li>
<li><span class="label">Email:</span> <a
href="mailto:tcivera@cefca.es"
class="moz-txt-link-freetext">tcivera@cefca.es</a></li>
<li><span class="label">Url:</span> <a
href="http://www.cefca.es"
class="moz-txt-link-freetext">http://www.cefca.es</a></li>
</ul>
</td>
</tr>
</tbody>
</table>
</div>
<div id="legal">
Este correo electrónico y, en su caso, cualquier archivo adjunto
al mismo, contiene información de carácter confidencial
exclusivamente dirigida a su destinatario. Queda prohibida
su divulgación, copia o distribución a terceros sin la previa
autorización escrita de nuestra entidad. En el caso de
haber recibido este correo electrónico por error, por favor,
elimínelo de inmediato y notifique esta circunstancia
al remitente.
En conformidad con lo establecido en las normativas vigentes de
Protección de Datos a nivel nacional y europeo, le informamos
que en la Fundación CEFCA tenemos un interés legítimo para
tratar sus datos de contacto con la finalidad de informarle
de nuestras actividades y gestionar sus peticiones. Sus datos
no serán cedidos a terceros, y los conservaremos sólo mientras
sea necesario para el mantenimiento de nuestra relación o
la gestión de su solicitud. Puede ejercer sus derechos de
acceso, rectificación y supresión de sus datos, así como los
derechos de limitación y oposición a su tratamiento con
fines promocionales, escribiendo a <a class="moz-txt-link-abbreviated" href="mailto:rgpd@cefca.es">rgpd@cefca.es</a>. También puede
dirigirse a nuestro Delegado de Protección de Datos para
cualquier consulta relacionada con nuestra política de
privacidad, escribiendo a <a class="moz-txt-link-abbreviated" href="mailto:dpd@cefca.es">dpd@cefca.es</a> .
<hr>
This email and, when appropriate, any file attached to it,
contains confidential information exclusively addressed to its
recipient. It is forbidden the disclosure, copying or
distribution of this email to third parties without the prior
written authorization of our entity. Should you have received
this email by mistake, please delete it immediately and notify
the sender of this circumstance.
According to the provisions of current Data Protection
regulations at national and European level, we inform you that
at CEFCA we have a legitimate interest to process your contact
data in order to inform you of our activities and manage your
requests. Your data will not be transferred to third parties,
and we will keep it only as long as it is necessary for the
maintenance of our relationship or the management of your
request. You can exercise your rights of access, rectification
and deletion of your data, as well as the rights of limitation
and opposition to its treatment for promotional purposes, by
writing to <a class="moz-txt-link-abbreviated" href="mailto:rgpd@cefca.es">rgpd@cefca.es</a>. You can also contact our Data
Protection Officer for any query related to our privacy policy,
writing to <a class="moz-txt-link-abbreviated" href="mailto:dpd@cefca.es">dpd@cefca.es</a> .
</div>
</div>
</body>
</html>