UWS 1.1

Grégory Mantelet gmantele at ari.uni-heidelberg.de
Tue May 26 10:58:54 CEST 2015 

Hello Pat, Mark and Grid,

     UWS 1.1 will be also implemented in the CDS/ARI's UWS Library. I 
have still few things to test or finish on it, but basically job list 
filtering and the blocking behaviour are working. Developments are still 
in progress but available on GitHub in the branch "uws1.1": 
https://github.com/gmantele/taplib/tree/uws1.1

     @Mark: if you want, I can provide you a .war archive with a very 
simple uws that you could deploy locally in order to test Topcat on it.


     About the blocking behaviour, I have two comments:

     First, the UWS Lib. is managing jobs completely in memory. This 
lets be notified immediately about a phase change thanks to a listener 
(already in place in the lib. for a while now). Thus, no polling is 
performed.

     Second, and it is the most important point for me: the client 
abortion. As Pat said, there is no way to detect client abortion (i.e. 
the client is not waiting any more for a server response)....at least 
using the Servlet API in Java. A such detection would be possible using 
directly sockets, but this part is hidden by the web application server 
(e.g. Tomcat).
     It is now several days I am spending to find a solution, but the 
only work-around I found was to send after a while a redirection to the 
same request (with the ?WAIT parameter ; with a reduced waiting time if 
some was set). In this way, if the client gave up, the redirection is 
not done, and otherwise the server receives a new waiting request and 
executes it. In both cases, the original waiting request is finished on 
server side.
     Well, this is quite fine if the client is a browser - it is totally 
transparent - but another client (like Topcat) should know about that 
trick. Honestly, I am not keen with this idea although it limits the 
resources consumption on server side by avoiding real unlimited wait but 
by doing like it does. The other solution, as Pat said, is to set a 
timeout (= max server waiting time). I would certainly do the same, 
except if the "request redirection" solution is fine for you.

     By the way, I also want to briefly point out that the same problem 
(client abortion detection) exists also for TAP synchronous requests.


     To conclude this email, I would like to know whether I have 
understood well about the ARCHIVED phase. A job is going in this phase 
ONLY IF the destruction time is reached. Is it right or is there a 
manual way for a user to put a job in this phase?
     Then, I suppose that a normal ?action=delete request is enough to 
anyway really delete this job, isn't it ?

Regards,
Grégory



On 22.05.2015 23:14, Patrick Dowler wrote:
>
> Yes, that helps. I ended up implementing server-side polling (of the 
> phase in the job db) with escalating interval: 1, 2, 4, 8... then 
> staying at 8 (sec). I don't think I can feasibly make another choice 
> since I have no idea how long the job will take.
>
> The 8 sec is quite large compared to the ~15ms to pull the current job 
> state from the database so not much point in making it bigger. And it 
> is progressively small compared to how long the job has been running 
> so far. This seems pretty decent for most situations....
>
> Except: Under heavy load, our apache-ajp-tomcat setup hits a limit and 
> new connections are held by apache until some ajp resources are 
> available. That means we generally do not want connections to hang 
> around for a long time consuming that limited resource, so my current 
> implementation has a maximum WAIT time. The normal result is to return 
> the current state of the job after that time, which would force the 
> client to check and possibly redo the get-and-wait. The max is 60 sec. 
> Would you expect a normal return without a phase change (eventually) 
> or should we just close the connection and let client error handling 
> retry
> (like it would for a timeout)? I think just returning the job document 
> in the current (unchanged) state seems right and it seems to me the 
> server must be able to place a limit since this does consume resources.
>
> * side issue with long waits: as far as I know, there is no good way 
> with http to determine if the client is still there waiting without 
> writing some output. In theory, I could probably set the response 
> headers and write a single white space every so often (without making 
> the document invalid) to probe that output stream, but I'm not sure 
> that will fail until I fill up at last one unflushable buffer 
> somewhere... further argument that the server should be able to place 
> a wait limit.
>
>
> Pat
>
>
>
> On 21/05/15 02:10 PM, Mark Taylor wrote:
>> Not sure if this helps, but for my usage purposes I'd consider anything
>> within a second to be pretty good service.  I typically poll every 5
>> seconds for TAP jobs (should really back off after a while but I don't).
>>
>> On Thu, 21 May 2015, Patrick Dowler wrote:
>>
>>>
>>> I have (crudely) implemented this and it should go live in a day or 
>>> so... will
>>> announce it here when it is available.
>>>
>>> One thing that came to mind while implementing this is that it is 
>>> really quite
>>> complex if you have load-balanced web servers. As such, I ended up 
>>> using
>>> internal polling in case events don't get propagated(*) from the server
>>> running the job to the server someone is blocking on... but I don't 
>>> know how
>>> responsive the client wants. It seems it would be nice if the client 
>>> could say
>>> "if I had to do it, I would poll every 300ms but I'm being nice and 
>>> WAIT=30
>>> (sec)"... The best solution is to have a distributed event mechanism 
>>> on the
>>> server side but I can see that not having a great cost:benefit 
>>> ratio. Maybe if
>>> the client could give a hint about responsiveness then server-side 
>>> polling
>>> could be tuned sufficiently... WAIT=30&THREATEN-TO-POLL=100ms ? :-)
>>>
>>> * meaning that right now I don't do this at all :-)
>>>
>>> Pat
>>>
>>>
>>> On 21/05/15 05:50 AM, Mark Taylor wrote:
>>>> Hi grid,
>>>>
>>>> I'm interested in implementing a UWS-1.1-aware client in order to
>>>> improve the way that I do blocking to wait for an asynchronous
>>>> TAP response (sec 2.2.1.1).
>>>>
>>>> Are there some deployed TAP services out there that implement
>>>> UWS-1.1 blocking behaviour?  Preferably using the UWS 1.1 namespace?
>>>> I'd like to test client behaviour against them.
>>>>
>>>> Thanks
>>>>
>>>> Mark
>>>>
>>>> On Tue, 19 May 2015, Paul Harrison wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are hoping to advance UWS 1.1 to PR status at (or perhaps 
>>>>> before) the
>>>>> Sesto Interop Meeting. I believe that there are now several
>>>>> implementations that are being updated to adhere to the new 
>>>>> version. It
>>>>> has been brought to my attention that the UWS 1.1 schema was not 
>>>>> available
>>>>> in the text of the last WD that was published to the IVOA document 
>>>>> store.
>>>>> For those that need a copy, it is available in volute
>>>>>
>>>>> https://code.google.com/p/volute/source/browse/trunk/projects/grid/uws/doc/UWS.xsd 
>>>>>
>>>>>
>>>>> In addition I have made some small edits to the standard document 
>>>>> - in
>>>>> preparation for PR - again this version is as yet only available 
>>>>> in volute
>>>>>
>>>>> http://volute.googlecode.com/svn/trunk/projects/grid/uws/doc/UWS.html
>>>>>
>>>>> I believe that there is only one outstanding issue in this 
>>>>> document in
>>>>> section 2.2.1.1
>>>>> (http://volute.googlecode.com/svn/trunk/projects/grid/uws/doc/UWS.html#blocking) 
>>>>>
>>>>> highlighted in yellow. It concerns how other standards might 
>>>>> signal which
>>>>> version of UWS that they are using. My current feeling is to 
>>>>> remove this
>>>>> text entirely and leave it up to the other standard documents to 
>>>>> signal
>>>>> the mechanism that they want to use. I think that part of the 
>>>>> cause for
>>>>> this versioning issue to come up was the lack of clarity of the 
>>>>> namespace
>>>>> for the UWS schema - which is now addressed by the release of a 
>>>>> schema
>>>>> with a new 1.1 namespace - therefore it is safe to remove this 
>>>>> issue from
>>>>> the UWS specification. Any objections?
>>>>>
>>>>> As has recently been mentioned on the lists, we hope to have a 
>>>>> session in
>>>>> Sesto on the topic of versioning of schema to form some guidelines 
>>>>> for
>>>>> good practice.
>>>>>
>>>>>
>>>>> Regards,
>>>>>     Paul.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> Mark Taylor   Astronomical Programmer   Physics, Bristol 
>>>> University, UK
>>>> m.b.taylor at bris.ac.uk +44-117-9288776 http://www.star.bris.ac.uk/~mbt/
>>>>
>>>
>>> -- 
>>>
>>> Patrick Dowler
>>> Canadian Astronomy Data Centre
>>> National Research Council Canada
>>> 5071 West Saanich Road
>>> Victoria, BC V9E 2E7
>>>
>>> 250-363-0044 (office) 250-363-0045 (fax)
>>>
>>
>> -- 
>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
>> m.b.taylor at bris.ac.uk +44-117-9288776 http://www.star.bris.ac.uk/~mbt/
>> .
>>
>

More information about the grid mailing list