(Resend) New TAP services at ESAVO
Carlos Rios
crios at sciops.esa.int
Fri Nov 12 07:37:43 PST 2010
Dear all,
Carlos Ríos (a new ESAVO member) sent this mail on Nov 12th to the dal at ivoa.net, but it looks like it didn't go through.
I am re-sending it here in his name, and will check whether the mail appears this time in the archived mailing list.
Cheers,
P.
----------------------------------------------------------------------------------------------------
Dear all,
we would like to inform you about TAP progress within ESAVO.
We started the development taking the reference of the code the CADC (Canadian Astronomy Data Centre) made publicly available at the Google SVN (http://code.google.com/p/opencadc/) and, modifying and adapting it to our needs, finally we have published our own TAP service.
Our source code is available as well at Google SVN at this URL: http://code.google.com/p/esavogpl/ under VOTAP label. Also accessible at http://code.google.com/p/esavogpl/source/browse/#svn/trunk/apps/votap .
The whole code includes the main published functionality of CADC and some new functions I will explain below in this mail.
This new TAP service is oriented to access to our catalogues, and for the being time, this service has been used to make the 2XMMi and AKARI catalogues available, having further plans to make some other catalogues available in a future(Planck, GAIA, etc).
More details on the functionality of the code and the implemented services follow:
1.- Main changes included in VOTAP from the CADC code
Some ADQL functions have been added to the original source code: Area, Distance, mathematical functions, a daemon to delete obsolete jobs, secure http access to the service (https) with Tomcat Basic Authorization, the the ability to modify some parameters for a job, delete a job...
Some modified functions over the CADC original code: Coordsys, point and circle management (and its handling of column values)
2.- Available services within VOTAP
These are the links the VOTAP service provides to any user who wants to work with it:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync synchronous implementation of TAP
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async asynchronous implementation of TAP
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/availability indicates if the service is alive (accepting queries)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/capabilities indicates the available resources provided in the TAP service
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/tables returns the information of the available catalogue tables and its columns
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/schemas returns the accessible schemas to the service
To perform our tests we have used the 2XMMi catalogue, with a PostgreSQL database (catalog table name: tap_schema.source_cat_epic)
3.- ADQL and mathematical functions supported
Set of mathematical and trigonometrical functions (ADQL compliant) supported:
abs(x) ceiling(x) degrees(x) exp(x) floor(x) log(x) log10(x) mod(x, y) pi() power(x, y) radians(x)
sqrt(x) rand(x) (*) round(x, n) truncate(x, n) acos(x) asin(x) atan(x) atan2(y,x) cos(x) sin(x) tan(x)
(*): rand(x) function is not fully supported ( in our implementation no seed is used , and this function is provided with a simple (without seed -> random() ) randomize use.
Set of specific geometrical functions ADQL compliant supported:
AREA BOX CENTROID CIRCLE CONTAINS COORD1 COORD2 COORDSYS DISTANCE INTERSECTS POINT POLYGON REGION
4.- Synchronous access
This access gives you a way to send directly queries to your preferred catalogue. For this, you have to attach to your HTTP request (typing in your web browser) the following parameters (in this order):
1.- REQUEST (MANdatory): It must be always the string "doQuery"
2.- LANG (MAN): It can be "ADQL" (mandatory support in TAP), "SQL" (OPTional) and "PSQL" (OPT and not supported by VOTAP)
3.- QUERY (MAN): It must be the query to execute to the catalogue ( schema_name.table_name pattern)
4.- MAXREC (OPT): It is a number, to be able to truncate the number of returned records in the query. There is an special case for it: MAXREC = 0 returns no data but metadata belonging to the table. This feature is implemented over the LIMIT clause for the PostgreSQL database . And also it could be used for databases handling the TOP -instead of LIMIT- clause to return a limited set of rows (this feature will be available simply uncommenting a portion of code to handle this alternative clause).
5.- FORMAT (OPT): VOTABLE is mandatory for TAP protocol and is supported, CSV and TSV are optional and not properly supported.
6.- VERSION (OPT): To specify the version of TAP protocol version. Only 1.0 is supported
An example to query the 2XMMi catalogue would be:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select c.ra, c.dec from tap_schema.source_cat_epic as c&MAXREC=10
Another one, in this case to AKARI catalogue:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select * from tap_schema.II_298_fis as o JOIN tap_schema.II_297_irc as p on o.objname=p.objname
5.- Asynchronous access
The other way is ,obviously, an asynchronous access to VOTAP. It consists of several endpoints to perform tasks like creating a job, keeping track of the existing jobs, the ability to modify some parameters of them... etc. There are multiple possibilities to use this service in a asynchronous way. Let's have a look to a typical use case within the 2XMMi catalogue:
First, we can call the asynchronous service without any parameters. We will obtain the list of the extant jobs hosted on the server, typing this URL in your web browser:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async
Now, we would be interested in creating a new job with a sample query to the catalog. We have to proceed this way (REQUEST, LANG and QUERY are mandatory, but MAXREC is optional) :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async?REQUEST=doQuery&LANG=ADQL&QUERY=select s.ra, s.dec from public.psource_cat_epic as s&MAXREC=30000
After this calling, we will obtain the response, a votable containing the main information about the created job and its state (PENDING) and its job ID (the identifier for the created job). Let's suppose (for the next examples) that the job id (generated by the server) is 515.
At this moment, the job has been simply stored. To launch the query , the user has to send a new request. Before doing this, you can anytime watch the job information, calling the server through these endpoints:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase The server returns the execution phase where the job is. In this case, PENDING.
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/quote The server returns a time prediction of when the job is likely to be completed.
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/executionduration The server returns a time (in seconds) of the duration for which a job shall run. You can modify this duration with the following instruction:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/executionduration?EXECUTIONDURATION=111 (you change the duration of the execution for the job to 111 seconds)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/destruction The server returns the date and time when the job shall be destroyed. You can also modify this date, doing this:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/destruction?DESTRUCTION=2010-12-31T11:00:00 (you change the destruction date to the 31 of December of 2010 at 11h AM)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/parameters The server returns the parameters you have included for the query in the creation of the job.
As an option, it is possible to modify some parameter of the job (MAXREC and/or LANG and/or QUERY and/or FORMAT) , simply requesting in this way: http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/parameters?MAXREC=10000&LANG=ADQL
Now, it's time for running the query. For this, we have the "phase" endpoint. This endpoint offers you either the possibility of running a job (if its state is PENDING or QUEUED) or aborting the job (if its state is EXECUTING or QUEUED as well) . E.g. to abort the recently created job, we have to call the server in this way:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase?PHASE=abort The server will return the job info, where you can check that its new state is ABORTED.
On the other hand, if you want to run the job , you have to type :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase?PHASE=run The server will answer quickly, with the job info, and you will see its state as EXECUTING. To keep track of the execution of the job, you can choose several options:
* Search the job into the job list http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async
* Watch the job info http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async?JOBID=515
* Watch the phase for the job (waiting for the COMPLETED state) http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase
When the job state reaches the status of COMPLETED, you can already see the result :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/results/result
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/result
or see the error document (if there was an error when processing the job):
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/error
And finally, you can choose the last main option: deleting a job. You will delete a job typing this URL:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/delete
In addition, a daemon would be running to check the destruction dates for every job, to delete the ones exceeding that date.
I think that's all. If you have any question or suggestion about this matter, do not hesitate to let us know.
Kind regards,
Carlos Ríos
(ESAVO)
================================================================================================
This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure,
use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in
error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by
the sender.
Please consider the environment before printing this email.
=================================================================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ivoa.net/pipermail/dal/attachments/20101112/4fbc2785/attachment-0001.html>
More information about the dal
mailing list