Modification Of UWS 1 to 1.1

[Petr Skoda]

Hi Pierre,

I am sorry I could not answer earlier but still just in time before 
Thursday's UWS session...

> I try to reply point by point
> UWS is the language to manage job in asynchronous way at distance, we are 
> clear that we define how to interface UWS service.

I think the UWS is not a language (like JDL - which is already a part of 
implementation and as said in 1.3. the "change of JDL changes the 
service contract) - I feel the suggested Prarameter Description language 
as presented by Carlo Zwolf, Franck le Petit and Paul Harrison
http://www.ivoa.net/internal/IVOA/InterOpOct2011Theory/PDL_2011.10.17.pdf
(I am following the Interop despite not being there;-)

is such a JDL (in fact the parameter description)


> We want it to be simple just to simplify his implementation.

It is your wish to make your life simple - but in situation that something 
has to be a global recommendation to solve the requirements of most of 
community, the things are never simple (I know the situation with "Simple"
Spectra Access ;-)) and Generic data sets ...


>> creation of new job and run together.

> From my point of view. The management of : witch job has to be sent in 
> parallel, what is the available RAM, what is the processors speed is not the 
> user problem.

I am afraid you are watching the situation as an informatician but not as 
a scientist. The wise scientist knows quite well his needs and makes 
decisions to optimize the time when he gets PROPER results. It was even 
more critical when the mainframe has limited amount of quick (fast) 
queues and long-run ones with different memory and CPU time limits -
I am sure that this has to be eventually put in UWS somehow as well.


> He has to send job. He can send many of them, he don't have to know if other 
> users send job at the same time. This is an infrastructure problem manage by 
> the provider that can have many kind of cluster, scheduler, batch queue.

Yes - the direct management of computation (how many nodes, which type, 
what size of memory...) has to be enforced by the manager of the 
infrastructure after investigation of typical requirements of users, 
avarage job size and run time etc .....

But as I tried to explain (probably not well ;-) our (i.e. my view as your 
oponent and my colleagues behind VO-KOREL) understanding of UWS idea is :

UWS defines the way how the asynchronous service (and stateful, 
job-oriented) can be implemented without any intention to restrict its 
usage to particular service contract or without any limits of possible 
implementation. UWS is a PATTERN or RECIPE HOW to arrange things

Your idea about its usage its already very concrete:
The user has a ANONYMOUS grid where he submits UNRESTRICTED number of 
ARBITRARILY large jobs and it is a task of a scheduler to process all of 
them accordingly to policy and rules implemented by data provider.

> Usually the scheduler send this job on the available CPU using the knowledge 
> of the ram, the number of CPU and the time reserve for execution.
> This is how it work on every cluster.

Well its a further part of system - the real execution (i.e  allocating 
RAM and executing the binary on given node) - see below:


> Don't reinvent the scheduler, job are on queue and with you mobile you can 
> see the status of all your job if you have a web interface to see so.

Its not about the scheduler - its about my knowledge an experience

>> There is yet no dédicated space to the user out of VOSpace.
In current VO services - we are providing this space. But I do not see the 
reason why to use VOSPACE if things are stored on given URL. VOSPACE is 
for exchange of data between services.


> So Input parameters are anyway destroyed after the job.
In your case yes, in our we have user space and files where it is stored.

I think we are talking about different parts of "job run" - see below

> But you tell me that you want intermediate result from a job you decided to 
> stop.
> we have to think about that and reintroduce abort if it's necessary.

It is allowed by UWS and we need it (imagine the log file with progress of 
convergence of some model)
If it is aborted manually (I am impatient)  or by exceeding the allowed 
CPU time (in given queue) or the timeout by Exec Duration, the results may 
still be very usefull telling that the job is almost done but the user was 
just optimistic to give small tollerance in convergence - but it may as 
well tell you this is nonsense - it does not converge with such a set of 
params ..... or it is oscillating ...


> User will not talk UWS, he will have an interface and will play with it
> we only describe exchange message between this interface and the server.

Exactly

> we have to define all the possible user requirement in this exchange message.

YES - the requirement of user interface is :
I want to upload large data (set of spectra) and initial parameters to my 
REMOTE USER SPACE for several experiements and after getting results I can 
decide to modify parameters and rerun the job with the same data or run 
another experiment already PENDING.

> I don't see where the lowering comfort is. But I am open to any extension if 
> our proposition does not fulfil the requirement.

if you want remove PENDING and excute immediately after the job creation, 
then you break the requirements given above. If not, then I am sorry for 
misunderstanding....

>>> We have remove pending phase as describe before. Job can be on
>>> | suspended phase, but it's only server action.

> We are only try to make the standard more clear by defining message sent 
> back from every actions and limiting the number of method to do the same 
> action to limit ambiguity not comfort. Then implementation will be 
> easier and more efficient.

Nothing against easier and more readable and clear formulations. But you 
have to conserve the functionality .....

>
>> 
>>> | We have removed some useless messages on our point of view like abort | 
>>> from the user. Because it make the same thing as delete as you can
>>> | not retrieve the result as explain in 1.0 version.
>> 
>> I do not understand why you could not retrieve results after abort -
>> and it does not do the same:

>> As we understand it - the DELETING means removing all remnants of the job - 
>> including the results.
> Yes
>>




> As I say before
> We have to face the problem of intermediate result

The people do not like the blind running of a code - they like some 
intermediate results to be produced to check the progress of a job (of 
course sometimes it may not be easy as the system cannot transfer from 
given node to  output node of a cluster - howver some distributed file 
systems try to do this anyway ...


> But not to substitute ourselves to the job manager.

The user has to be his jobs manager ;-)
But he has to have the interface allowing him to control the jobs ...

> Otherwise when there is multiple user and multiple CPU, it become 
> unmanageable. It's not the purpose of UWS, it's provider internal business.

Well in every OS the user has a right to kill HIS jobs if something is 
going wrong. It would be very dumb to allow the user to execute many jobs 
(and let the machine go to knees) just to play with different parameter 
sets as he wants to get some results without more advanced thinking about 
the nature of a problem

(its a experience with VO-KOREL  there is a lot of 
parameters and as you know the global optimization task requires some 
problem knowledge to restrict yourself to the critical parameters to 
converge and to given range of probable solutions --- this is well 
understood by users, nevertheless they try to put the burden of decision 
on "dumb" computer if having enough freedom - then they run hundreds of 
jobs in parallel day and night not to let the machine to idle ;-)

So user in restricted environment (e.g. limited CPU, memory etc) will have 
to decide what jobs to run and while computing, he may discover another 
idea to get a solution faster. Then he naturalely decides to abort the job 
and run another.

The same concerns the PENDING state - he might prepare it but still waits 
for results or error on currently running.

>> Especially the isolation of individual users to see only their own jobs,
> This can be done only if we identify user.

Well in non anonymous environment you have clear identification of user 
(he has somehow to authenticate)

In anonymous you should probably to arrange somehow some temporary ID.
But as UWS is Stateful -you have to remember the state (and user) to be 
able to identify him again after re-connection.

in 2.1.8 (added in UWS 1.0) is defined the Owner but only in case of 
authenticated service. I think it is not forbiddent to use this object 
even for anonymous service using some other means of users' identification

Here I am not sure about how to impose the rule of STATEFULNESS for 
anonymous user - it was not our case ...
in principle knowing the job id anybody could get someone's results after 
completion - probably not wanted by most of community.

> If we want to define the mechanism of identification, then we propose to use 
> HTTP authentication mechanism that is an RFC
This is what we are doing as well ...

>> As i said we are probably doing something revolutionary with UWS and so our 
>> interpretation may be wrong (I hope Paul and Dave will comment on their 
>> primary ideas of UWS - but I liked the original design as very flexible)

I am explaining below what I think is different in my vision and yours


> But you know better the code than the user. So you are able to make an 
> initial estimation of time duration.

NO - in certain cases it depends on parameters - and its almost 
unpredictable - in case of multiparametric optimization.

Of course if you  just run image processing you can scale easily the job 
time using the size of image as parameter.
I am afraid it is a special case, however.

> The queue priority depend on the resource you ask. You can not have the same 
> priority for a quick job mono processor and a multi processor job that use 
> all the memory of the machine and run on 200h.

Exactly - that's why I think the user has to be provided with different 
queues and there should be some UWS parameter hinting the UWS service to 
which one should be the job submitted. Then this "crystal ball number" may 
be replaced with the default time limit of given queue - but it is still 
difficult as usually the CPU time not the wall time is used in queue 
policies....

> So if you say my job is a 5mn job, then I change my mind it will use 200h.
YES -- I can ABORT  all my jobs to  get into the limits allowing me to use 
the long-term queue - depends on job management policy  an my status (be 
allowed to use more memory, CPU time etc ...)

> It's not acceptable for the other users and it brake all the mechanism of the 
> job queue that decide the job scheduling process.

User should have the freedom to decide what he wants if being allowed to 
do it. If you remove the ABORT there is no way how to stop wrongly 
prepared job , If you replace it with DELETE then all the effort is lost, 
while in ABORT case the user could check the results produced so far (to 
recognize the problem diverges and so he has to think more about the 
physical nature and reformulate the problem)

> In UWS you ask a service the way to query them and retrieve result. From 
> this information you are able to make a client that send job and 
> retrieve results. But we didn't give any recommendation on web 
> interface.

It was the original objection of Mathew as well - UWS is for running 
automatic submission - mainly for TAP. It was not originally intended for 
running browser interaction.

As I said we tried to find another (more fancy for scientists - I have 
asked many of them) application of UWS (as it is a general pattern without 
restriction). But so far I do not see any reason why only the blind 
automatic execution of jobs should be the official way, if different idea 
can fit well with current standard.

> Our purpose is not to reduce functionality, but just to remove ambiguity.
This is philosophical retorics ;-)
You are reducing the functionality as you do not allow me to prepare job 
without run and you removed my right to deside about its fate (STOPING it)
The ambiguity is often only apparent - tiny differences may have crucial 
consequences (see the arbitrary EULA ;-)

> If you can send a job from 3 different manner, it mean that people 
> implementing server side  will have to face choice usually not clear
> it mean that client must implement the 3 manner
> it meant that the description of the service have to face a description of 3 
> maners.

can you exactly describe which 3 means of starting you have in mind ?

The job must be CREATED by uploading of parameters. And as it said in 
2.2.3.1 you may add the job control parameter  telling PHASE=RUN, so the 
jobs is run immediately - so all your services may define in parameter 
sets this implicitly (but I would still let to user to check this option 
in input form or add it to the client string etc ...)
If you remove the PENDING phase you restrict the user's freedom.

Its a key issue - you are forcing anybody to accept your solution as you 
feel currently powerful enough to dictate others your particular solution.
And what with the poor older services having PENDING, ABORT etc - it looks 
like the MS or Oracle policy -  drop all what is workig as we decided to 
make you happy with our new version which is the only one we support.

(sorry nothing personally against your group - I am sure you wanted to 
make more developers happy without seeing the consequences)
But OTOH I feel this is the crucial point of misunderstanding between VO 
developers and potential VO users and probably one of the reasons why the 
VO interest ist diminishing in wide astronomical  community  (e.g. seen 
in state of VO affairs in ESO ).

The developers want to enforce the astronomers to accept their 
simply-to-implement and conceptionally  clean solution of VO application 
or service while the scientists want to have something different which is 
however harder to implement or not obvious to developer.

All the IVOA rules tried to maintain the back compatibility and most of 
standards were extended to employ new features, never cut in funcionality!

> I just say, give me a good reason to create a job and not sending it. Why if 
> you create a job you don't want to send it.

Probably the concepts we are talking about is different -

Our solution as I said is a CLOUD -like - so the user has some private 
space allocated where he can upload the data+parameters. He can use this 
space as he needs within certain limits.
If he uploads a lot of data, the disk quota system stops him to create new 
jobs (so he has to decide what to remove - i.e. data, input parameters and 
results) - this is what I understand the DELETE operation.

By CREATE of job I mean - allocation of this private space where the data 
and parameters are uploaded - this is what I call PENDING - after the 
PHASE=RUN is this volume sent by some scheduler to particular machine for 
execution or submitted to some queue - here it comes the private policy of 
the scheduler to power - the state is QUEUED.

Then the particular node may be given the processing code and data and he 
runs it (maybe I am wrong but in cluster or grid the code is not on a node 
but it is uploaded there by the scheduling node from user's space .
So here the UWS should decide what numerical code to run (it knows he 
service URL) and this should be sent with user's data wrapped somehow for 
deployment).

In other words - there is a UWS server that has to handle user's space 
where the data and params are uploaded.  There must be some server 
containing the binaries of code of all supported services and there is a 
entry point (schedulling node) of grid or cluster which does not care 
about UWS but gets a binary of given service, parameters and data of user
and according to his scheduler decides on which node to run it.

You may have a networked (clustered filesystem) and homogenized cluser and 
then all the binaries are same and only the name of binary code is sent in 
some job descritpion language to given node referring to shared user's 
space where the data and params were stored.

He can run limited number of parallel jobs (we are thinking about some 
policy of different memory requirement, time limits etc even if this a 
task of scheduler - but the user has to have some options like you can run 
5 short low memory jobs - other will be queued or one large for longer 
time memory)

Even if we do not let him decide we can let the scheduler decide but 
then some queue classes must be defined and he must decide how to run the 
job.

Even in case of such a TAP application the user may want to query whole 
universe ;-) (large amount of data expected in large cone, larger range of 
magnitudes etc ....) but it may take long time - or he decides to restrict 
the query and wants to have results quickly ....

> And why you are not able to wait 
> until the time you are ready to send directly the job without going to this 
> pending status.

As I tried to explain above - the user in our case have the allocated 
storage on our servers which can use for long time directly - there is a 
storage of this experiments, comments, results, graphs etc ...
Its a CLOUD concept which is extremely comfortable for users like the 
eshop - all my purchases are there so I can re-order again the same goods
- well its a different ;-) but here I can see the improvements in 
parameters on the same data and re-execute the jobs with the same data 
(already uploaded - may be large - in my space) and changed parameters 
(one number modified from my mobile ;-)

Of course in case I re-submit the same job (i.e. I take the same 
data-sets, same parameters, allow to edit the parameters) I do not force 
PENDING phase - as I suppose I want to see results of my modified job (or 
even the same - as the server version of the binary may be upgraded) 
immediately.

So once again - creating jobs means uploading large data + params to 
server user's space.

But then I can use this space to run the same job with small change in 
parameters without need of uploading all the data again.
I agree this is a particular case but in general it may be extended to 
concept of workflows - the data (e.g. processing of image mosaic will take 
time to upload) but then I may play with workflow description from mobile 
and check the quality of solution (mostly not the image itself but some 
quality plot, statistics etc ...)


Probably you should ask users of your services wheteher they would like to 
have history of their jobs without need  uf re-uploading it always.
There are still slow networks in hotels, or using GPRS etc ....
What we are allowing the users is a "supercomputing" on my phone

And I am pretty sure this will be required soon by all astronomers
to have in VO - as the mobility freedom is increasing  - tablets etc ..
There will have to be the proper model for VO to provide such cloud 
services - instead of running Aladin of large computer (lot of memory for 
Java etc ..) the simple browser will connect Aladin in cloud  to see the 
DSS and 2MASS thumbnail of region where the supernova has just exploded.

The same in theory case.

To summarize for further consideration in GWS session - I do not support 
your idea of removing PENDING, ABORT etc ...
I think the critical issue is the solution  of jobs isolation - 
authentication and some users's identification for open (public) services 
(e.g. cookie's ).  The new version of UWS should include somehow the 
parameter description language and will need some concept of queue 
priorities.


I would even appreciate the another phase ;-)
the STOPED as logical extension of ABORT -
stop executing, free processor, allow the user to see the results for 
short time and decide if ABORT or CONTINUE - the state is same as in 
SUSPENDED but the system does not run it again if troubles (e.g. memory 
limit)  are gone but the user has to put it in EXECUTING state 
intentionally. Of course it will be harder to implement in GRID 
environment but I am sure modern systems will alow this (something like 
checkpoint storage, unswapped job etc ...)

The user can already run long job and he is not allowed to run another but 
he may want to stop the execution to get resources, run the quicker one 
and then continue (for another month ;-) to compute again ...

Of course the crucial question is what happens with computing node - is it 
a temporary stop for several minutes where the process is just swapped out 
(and will be killed later) or kind of checkpoint where all the space of 
running jobs is stored in some user space from what it can be recalled 
after weeks etc ....

I am not experienced in such a technologies on clusters, but it may be 
even some special feature of application binary that allows to store the 
intermediate results of long computation and the newly executed job may be 
given the parameter to continue after re-reading such a state (something 
like the SAVE in IDL or Save session in most unix applications ...)


To be honest I dislike the idea of automatic Destruction Time removing my 
results shortly after being computed. I would first decide the user to 
decide what he wants to maintain withing limited space (not to remove jobs 
older e.g. than 2 month and keep current as the older onec may be 
important well represnenting some solution of my problem that I still want 
to work on, but the current are results of multiple demos of some silly 
experiment.

But if the destrution Time is long enough (e.g. month or more) then the 
idea in last paragraph of 2.2.3.3 is nice (if quota exceeded, further jobs 
are stored only shorter times - if the user is warned about this, he will 
be carefull not to loose fancy new results ;-)

And I like the 4.3 - the idea of CEA v2 very much.

Best regards,

Petr


*************************************************************************
*  Petr Skoda                         Phone : +420-323-649201, ext. 361 *
*  Stellar Department                         +420-323-620361           *
*  Astronomical Institute AS CR       Fax   : +420-323-620250           *
*  251 65 Ondrejov                    e-mail: skoda at sunstel.asu.cas.cz  *
*  Czech Republic                                                       *
*************************************************************************