Running TAP/async jobs immediately: Proposal Text

[Mark Taylor]

For reference: we have been here before
(http://mail.ivoa.net/pipermail/dal/2016-May/007497.html)

Basically: Walter is right, UWS is explicitly agnostic about this,
so it's not treading on UWS's toes, and it's not unreasonable,
for TAP to clarify the way that it is using UWS and say that an
initial PHASE=RUN can be attached to the job creation.
Adjusting UWS to get off the fence about this would also be
a possibility (though given that UWS is not up for change right
now, not such a grat idea).
But, it's a change to the current standard, which is always slightly
painful.  And to argue for it on the grounds of N vs N+1 (N>=3)
connections seems a bit disingenuous, since if you want a quick in,
quick out job you will probably be using the sync endpoint.
But it's probably harmless enough, I don't have strong opinions.

But note: although such a standards change would allow clients to
submit async requests in this way, it wouldn't force them to do that.
And given that generic clients would need to switch this behaviour
on discovered TAP version, it may not be worth their effort
(read: even if initial PHASE=RUN becomes legal in TAP 1.1,
I'm not guaranteeing to do it like that in topcat/stilts,
though I seem to have suggested the opposite in the mail I
quoted above).  So anybody running a TAP service which
*requires* PHASE=RUN on initial job creation in order for
the job to be accepted may find that such a change to the
TAP or UWS text does not solve all their problems.
[And in general: I think in most cases it's much better practice
for non-compliant services to get compliant than to push for
changes in future versions of standards so they don't have to]

Mark


On Thu, 29 Sep 2016, Walter Landry wrote:

> Hi Marco,
> 
> Looking at Section 2.2.3.1 of UWS, setting PHASE=RUN at job submission
> is explicitly listed as a possibility.  It is just not required.  So
> we could change that section to read
> 
>   2.2.3.1. Creating a Job
> 
>   POSTing a request to the Job List creates a new job (unless the
>   service rejects the request). The response when a job is accepted
>   must have code 303 "See other" and the Location header of the
>   response must point to the created job. The response when the job is
>   rejected should be a HTTP 403 Forbidden status style response.
> 
>   This initial POST will in most cases carry parameters for the
>   protocol that is using the UWS pattern, as detailed in 4 .In
>   addition for the initial post may contain job control parameters if
>   allowed by the implementing protocol (i.e. if UWS job control
>   parameter names do not clash with the implementing protocol
>   parameters).
> 
>   If the the parameter PHASE=RUN is included as part of the initial
>   job submission, the UWS must immediately attempt to start the job as
>   in Section 2.2.3.5.
> 
> Section 2.2.3.5 is then changed to
> 
>   2.2.3.5. Starting a Job
> 
>   When a job is initially submitted, including the parameter PHASE=RUN
>   instructs the UWS to immediately attempt to start the job.
> 
>   Alternately, POSTing the single parameter PHASE=RUN to the
>   /{jobs}/{job-id}/phase URI will also instruct the UWS to start the
>   job.  The response to this request must have code 303 "See other"
>   and the Location header of the response must point to the
>   /{jobs}/{job-id} URI so that the client receives the phase that the
>   job has been set to. Typically a UWS will put a job into a QUEUED
>   state on receipt of this command, but depending on how busy the
>   server is, the job might be put almost immediately (and without
>   client intervention) into an EXECUTING state.
> 
> Cheers,
> Walter Landry
> 
> p.s. For queries that are usually very fast (e.g. cone searches that
> are usually very small), there would typically be a single fetch.
> 
> Marco Molinaro <molinaro at oats.inaf.it> wrote:
> > Hi Walter, all,
> > section 5 in TAP is example usage of UWS within TAP.
> > My feeling is that your proposal would better fit the UWS
> > specification, on which TAP relies upon (that's why I CC'ed grid).
> > 
> > Putting this into TAP looks like overriding a behaviour defined
> > by another spec and may create confusion to REC adopters.
> > 
> > That said I'm not against a "create&run" posting, even if this can
> > be mimicked at client level (sure, not saving on the number of
> > connections).
> > 
> > Cheers,
> >      Marco
> > 
> > PS - I don't think it's 3 vs. 4 connections...isn't it N vs N+1?
> > there (usually) can be more fetch calls.
> > 
> > 2016-09-28 22:57 GMT+02:00 Walter Landry <wlandry at caltech.edu>:
> >> Hello Everyone,
> >>
> >> Since there is a new draft TAP spec, can we insert language allowing
> >> users to immediately run async jobs?  Being able to immediately run
> >> jobs is a nice convenience for the users.  Also, if a query is usually
> >> very fast, it can reduce network traffic: 3 connections (submit,
> >> check, fetch) instead of 4 (submit, run, check, fetch).
> >>
> >> As far as I can tell, this would require changes to section 5.1.1 and
> >> 5.1.3.  I have attached proposed text below.
> >>
> >> Cheers,
> >> Walter Landry
> >>
> >>
> >> ============================
> >>
> >> 5.1.1  Creating and Executing a Simple Query
> >>
> >> Asynchronous queries are created in the same way as synchronous, using
> >> one of the async endpoints, for example:
> >>
> >>   HTTP POST http://example.com/tap/async
> >>   LANG=ADQL
> >>   QUERY=SELECT * FROM magnitudes AS m WHERE m.r>=10 AND m.r<=16
> >>
> >> The service's response to this request is an HTTP redirect with a URL
> >> fo the query job:
> >>
> >>   HTTP status 303 'See other'
> >>   Location: http://example.com/tap/async/42
> >>
> >> The query result or an error document can then be retrieved from a URL
> >> associated with the job. This is an application of the UWS
> >> pattern.  The query is then executed with a separate request to run the job URL:
> >>
> >>   HTTP POST http://example.com/tap/async/42/phase
> >>   PHASE=RUN
> >>
> >> Alternatively, the job can be executed immediately by setting
> >> "PHASE=RUN" in the initial submission.  The state of the job can be
> >> retrieved from the phase resource:
> >>
> >>   HTTP GET http://example.com/tap/async/42/phase
> >>
> >> The client may have to check the phase multiple times until the job
> >> finishes. Once the returned value is COMPLETED, the results can be
> >> obtained from the results resource:
> >>
> >>   HTTP GET http://example.com/tap/async/42/results/result
> >>
> >>
> >> ============================
> >>
> >> 5.1.3  Running a Query
> >>
> >> The phase URL shows the progress of the job. If not otherwise
> >> specified, when the job is created by the service the phase will
> >> normally be set to PENDING.  It might be set to ERROR if the service
> >> has rejected the job. If the phase is ERROR, then the error URL should
> >> lead to a an error document explaining the problem. If the phase is
> >> PENDING, then the client needs to commit the job for execution.
> >>
> >> The client runs the job by posting to the phase URL:
> >>
> >>   HTTP POST http://example.com/tap/async/42/phase
> >>   PHASE=RUN
> >>
> >> The phase may also be set during the initial submission.  The service
> >> replies with a redirection to the job URL
> >>
> >>   HTTP status 303 'see other'
> >>   Location: http://example.com/tap/async/42
> >>
> >> ... (rest of section is unchanged)
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/