UWS 1.1

[Patrick Dowler]

Yes, that helps. I ended up implementing server-side polling (of the 
phase in the job db) with escalating interval: 1, 2, 4, 8... then 
staying at 8 (sec). I don't think I can feasibly make another choice 
since I have no idea how long the job will take.

The 8 sec is quite large compared to the ~15ms to pull the current job 
state from the database so not much point in making it bigger. And it is 
progressively small compared to how long the job has been running so 
far. This seems pretty decent for most situations....

Except: Under heavy load, our apache-ajp-tomcat setup hits a limit and 
new connections are held by apache until some ajp resources are 
available. That means we generally do not want connections to hang 
around for a long time consuming that limited resource, so my current 
implementation has a maximum WAIT time. The normal result is to return 
the current state of the job after that time, which would force the 
client to check and possibly redo the get-and-wait. The max is 60 sec. 
Would you expect a normal return without a phase change (eventually) or 
should we just close the connection and let client error handling retry
(like it would for a timeout)? I think just returning the job document 
in the current (unchanged) state seems right and it seems to me the 
server must be able to place a limit since this does consume resources.

* side issue with long waits: as far as I know, there is no good way 
with http to determine if the client is still there waiting without 
writing some output. In theory, I could probably set the response 
headers and write a single white space every so often (without making 
the document invalid) to probe that output stream, but I'm not sure that 
will fail until I fill up at last one unflushable buffer somewhere... 
further argument that the server should be able to place a wait limit.


Pat



On 21/05/15 02:10 PM, Mark Taylor wrote:
> Not sure if this helps, but for my usage purposes I'd consider anything
> within a second to be pretty good service.  I typically poll every 5
> seconds for TAP jobs (should really back off after a while but I don't).
>
> On Thu, 21 May 2015, Patrick Dowler wrote:
>
>>
>> I have (crudely) implemented this and it should go live in a day or so... will
>> announce it here when it is available.
>>
>> One thing that came to mind while implementing this is that it is really quite
>> complex if you have load-balanced web servers. As such, I ended up using
>> internal polling in case events don't get propagated(*) from the server
>> running the job to the server someone is blocking on... but I don't know how
>> responsive the client wants. It seems it would be nice if the client could say
>> "if I had to do it, I would poll every 300ms but I'm being nice and WAIT=30
>> (sec)"... The best solution is to have a distributed event mechanism on the
>> server side but I can see that not having a great cost:benefit ratio. Maybe if
>> the client could give a hint about responsiveness then server-side polling
>> could be tuned sufficiently... WAIT=30&THREATEN-TO-POLL=100ms ? :-)
>>
>> * meaning that right now I don't do this at all :-)
>>
>> Pat
>>
>>
>> On 21/05/15 05:50 AM, Mark Taylor wrote:
>>> Hi grid,
>>>
>>> I'm interested in implementing a UWS-1.1-aware client in order to
>>> improve the way that I do blocking to wait for an asynchronous
>>> TAP response (sec 2.2.1.1).
>>>
>>> Are there some deployed TAP services out there that implement
>>> UWS-1.1 blocking behaviour?  Preferably using the UWS 1.1 namespace?
>>> I'd like to test client behaviour against them.
>>>
>>> Thanks
>>>
>>> Mark
>>>
>>> On Tue, 19 May 2015, Paul Harrison wrote:
>>>
>>>> Hi,
>>>>
>>>> We are hoping to advance UWS 1.1 to PR status at (or perhaps before) the
>>>> Sesto Interop Meeting. I believe that there are now several
>>>> implementations that are being updated to adhere to the new version. It
>>>> has been brought to my attention that the UWS 1.1 schema was not available
>>>> in the text of the last WD that was published to the IVOA document store.
>>>> For those that need a copy, it is available in volute
>>>>
>>>> https://code.google.com/p/volute/source/browse/trunk/projects/grid/uws/doc/UWS.xsd
>>>>
>>>> In addition I have made some small edits to the standard document - in
>>>> preparation for PR - again this version is as yet only available in volute
>>>>
>>>> http://volute.googlecode.com/svn/trunk/projects/grid/uws/doc/UWS.html
>>>>
>>>> I believe that there is only one outstanding issue in this document in
>>>> section 2.2.1.1
>>>> (http://volute.googlecode.com/svn/trunk/projects/grid/uws/doc/UWS.html#blocking)
>>>> highlighted in yellow. It concerns how other standards might signal which
>>>> version of UWS that they are using. My current feeling is to remove this
>>>> text entirely and leave it up to the other standard documents to signal
>>>> the mechanism that they want to use. I think that part of the cause for
>>>> this versioning issue to come up was the lack of clarity of the namespace
>>>> for the UWS schema - which is now addressed by the release of a schema
>>>> with a new 1.1 namespace - therefore it is safe to remove this issue from
>>>> the UWS specification. Any objections?
>>>>
>>>> As has recently been mentioned on the lists, we hope to have a session in
>>>> Sesto on the topic of versioning of schema to form some guidelines for
>>>> good practice.	
>>>>
>>>>
>>>> Regards,
>>>> 	Paul.
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>>> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
>>>
>>
>> --
>>
>> Patrick Dowler
>> Canadian Astronomy Data Centre
>> National Research Council Canada
>> 5071 West Saanich Road
>> Victoria, BC V9E 2E7
>>
>> 250-363-0044 (office) 250-363-0045 (fax)
>>
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-9288776  http://www.star.bris.ac.uk/~mbt/
> .
>

-- 

Patrick Dowler
Canadian Astronomy Data Centre
National Research Council Canada
5071 West Saanich Road
Victoria, BC V9E 2E7

250-363-0044 (office) 250-363-0045 (fax)