new UWS 1.1 WD -- mime and job list

[Paul Harrison]

On 2014-09 -30, at 21:13, Patrick Dowler <patrick.dowler at nrc-cnrc.gc.ca> wrote:

> 
> 
> On 30/09/14 12:11 PM, Kristin Riebe wrote:
>> We recently came across another problem of our application: we have
>> users with hundred thousands of jobs, so that our web application ran
>> out of memory when the job list should have been returned.
>> We consider some kind of pagination now or restricting the returned list
>> to the latest 1000 jobs or so.
>> Maybe such a pagination could be considered for UWS 2.
> 
> This could be done with optional parameters in the same offset/limit style as VOSpace:
> 
> id=<starting jobID>
> limit=<number of records to return>
> 
> The service has to return a consistent ordering. This also allows a service to place an upper bound and default value on "limit". The downside (also in vospace) is that to be robust you always make a last call with the final ID you saw and get just that one back because it is the only sure way to know you are done... at least, in vospace that was deemed better than a "truncation marker" in the output to tell you more records exist.

We had previously decided that pagination was a UWS 2 sort of thing, as it would represent too great a change for implementors, and as you point out it is tricky to  return the total number of jobs that are in the list (though it could probably be done with custom HTTP header values).

An alternative to Kristin’s problem that the current 1.1 draft gives is to make use of the “Archived” phase of a job - I am not sure if the reason that there are so many jobs in the list because they are old, but if so then many could be put into the “Archived” phase so that they should not be displayed by default.

Another alternative is to introduce more standard filters on the job list - the current UWS 1.1 draft introduces ?PHASE which filters on the job phase, but we could introduce more standard filters. e.g.

?NEWER=2014-09-05  only show jobs modified since date
?NEWEST=100 only show the 100 most recent jobs.

In general I think that filters on the job list are nicer than pagination, as we can define much better semantics for the filters. As Pat says it is important that the job that you are most interested in appears in the list!

I think that there would be nothing stopping a server implementation from choosing to include the filter parameter in the redirection URL that is used in many places in UWS - as long as we have standard filters then a client could infer that there might be more jobs available by reading the URL.

Are there more ideas for standard filters on the Job list that people would like to include?

 
> 
>> Also, in that context, I would prefer if the location header in the
>> response after deleting a job does not necessarily have to point to the
>> job list, since then again we get into trouble each time one of the many
>> jobs is deleted. A simple reponse confirming a successful deletion could
>> be enough.
> 
> It is worse if you decide that job listing is not permitted and your job list returns an authorisation failure :-). I agree with this as well.
> 
> 

I am not sure that you are allowed to have an authorisation failure on the job list - you are supposed to show the list of jobs that the user is allowed to see - if that happens to be a list with no entries, then that is what you should show. However, I have some sympathy with the suggestion - the original design of UWS was driven by the desire to make it easy to operate directly from a browser - however, it really should be seen as an API and perhaps always redirecting to the Job list is not the best - it is something to discuss at the Interop (won’t be there in person unfortunately, but will try to Skype in if possible).