New TAP service at ESAVO
Carlos Rios
crios at sciops.esa.int
Fri Nov 12 07:37:43 PST 2010
Dear all,
we would like to inform you about TAP progress within ESAVO.
We started the development taking the reference of the code the CADC
(Canadian Astronomy Data Centre) made publicly available at the Google
SVN (http://code.google.com/p/opencadc/) and, modifying and adapting it
to our needs, finally we have published our own TAP service.
Our source code is available as well at Google SVN at this
URL: http://code.google.com/p/esavogpl/ under VOTAP label. Also
accessible
at http://code.google.com/p/esavogpl/source/browse/#svn/trunk/apps/votap .
The whole code includes the main published functionality of CADC and
some new functions I will explain below in this mail.
This new TAP service is oriented to access to our catalogues, and for
the being time, this service has been used to make the 2XMMi and AKARI
catalogues available, having further plans to make some other catalogues
available in a future(Planck, GAIA, etc).
More details on the functionality of the code and the implemented
services follow:
/*1.- Main changes included in VOTAP from the CADC code
*/Some ADQL functions have been added to the original source code: Area,
Distance, mathematical functions, a daemon to delete obsolete
jobs, secure http access to the service (https) with Tomcat Basic
Authorization, the the ability to modify some parameters for a job,
delete a job...
Some modified functions over the CADC original code: Coordsys, point and
circle management (and its handling of column values)/*
*//*2.- Available services within VOTAP*/
These are the links the VOTAP service provides to any user who wants to
work with it:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync /*synchronous*/ implementation
of TAP
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async */asynchronous/* implementation
of TAP
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/availability
indicates if the service is alive (accepting queries)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/capabilities
indicates the */available resources/* provided in the TAP service
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/tables
returns the information of the */available catalogue tables and its
columns/*
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/schemas
returns the /*accessible schemas*/ to the service
To perform our tests we have used the 2XMMi catalogue, with a PostgreSQL
database (catalog table name: tap_schema.source_cat_epic)
/*3.- ADQL and mathematical functions supported*/
Set of mathematical and trigonometrical functions (ADQL compliant)
supported:
abs(x) ceiling(x) degrees(x) exp(x)
floor(x) log(x) log10(x) mod(x,
y) pi() power(x, y) radians(x)
sqrt(x) rand(x) */(*)/* round(x, n)
truncate(x, n) acos(x) asin(x) atan(x)
atan2(y,x) cos(x) sin(x)
tan(x)
(*): rand(x) function is not fully supported ( in our implementation no
seed is used , and this function is provided with a simple (without seed
-> random() ) randomize use.
Set of specific geometrical functions ADQL compliant supported:
AREA BOX CENTROID CIRCLE CONTAINS
COORD1 COORD2 COORDSYS DISTANCE
INTERSECTS POINT POLYGON REGION
/*4.- Synchronous access
*/
This access gives you a way to send directly queries to your preferred
catalogue. For this, you have to attach to your HTTP request (typing in
your web browser) the following parameters (in this order):
1.- REQUEST (MANdatory): It must be always the string "doQuery"
2.- LANG (MAN): It can be "ADQL" (mandatory support in TAP), "SQL"
(OPTional) and "PSQL" (OPT and not supported by VOTAP)
3.- QUERY (MAN): It must be the query to execute to the catalogue (
schema_name.table_name pattern)
4.- MAXREC (OPT): It is a number, to be able to truncate the number of
returned records in the query. There is an special case for it: MAXREC =
0 returns no data but metadata belonging to the table. This feature is
implemented over the LIMIT clause for the PostgreSQL database . And also
it could be used for databases handling the TOP -instead of LIMIT-
clause to return a limited set of rows (this feature will be available
simply uncommenting a portion of code to handle this alternative clause).
5.- FORMAT (OPT): VOTABLE is mandatory for TAP protocol and is
supported, CSV and TSV are optional and not properly supported.
6.- VERSION (OPT): To specify the version of TAP protocol version. Only
1.0 is supported
An example to query the 2XMMi catalogue would be:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select
c.ra, c.dec from tap_schema.source_cat_epic as c&MAXREC=10
<http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select%20c.ra,%20c.dec%20from%20tap_schema.source_cat_epic%20as%20c&MAXREC=10>
Another one, in this case to AKARI catalogue:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select
* from tap_schema.II_298_fis as o JOIN tap_schema.II_297_irc as p on
o.objname=p.objname
<http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select%20*%20from%20tap_schema.II_298_fis%20as%20o%20JOIN%20tap_schema.II_297_irc%20as%20p%20on%20o.objname=p.objname>/*
*//*5.- Asynchronous access*/
The other way is ,obviously, an asynchronous access to VOTAP. It
consists of several endpoints to perform tasks like creating a job,
keeping track of the existing jobs, the ability to modify some
parameters of them... etc. There are multiple possibilities to use this
service in a asynchronous way. Let's have a look to a typical use case
within the 2XMMi catalogue:
First, we can call the asynchronous service without any parameters. We
will obtain the list of the extant jobs hosted on the server, typing
this URL in your web browser:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async
Now, we would be interested in creating a new job with a sample query to
the catalog. We have to proceed this way (REQUEST, LANG and QUERY are
mandatory, but MAXREC is optional) :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async?REQUEST=doQuery&LANG=ADQL&QUERY=select
s.ra, s.dec from public.psource_cat_epic as s&MAXREC=30000
<http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async?REQUEST=doQuery&LANG=ADQL&QUERY=select%20s.ra,%20s.dec%20from%20public.psource_cat_epic%20as%20s&MAXREC=30000>
After this calling, we will obtain the response, a votable containing
the main information about the created job and its state (PENDING) and
its job ID (the identifier for the created job). Let's suppose (for the
next examples) that the job id (generated by the server) is 515.
At this moment, the job has been simply stored. To launch the query ,
the user has to send a new request. Before doing this, you can anytime
watch the job information, calling the server through these endpoints:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase
The server returns the execution phase where the job is. In this case,
PENDING.
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/quote
The server returns a time prediction of when the job is likely to be
completed.
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/executionduration
The server returns a time (in seconds) of the duration for which a job
shall run. You can modify this duration with the following instruction:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/executionduration?EXECUTIONDURATION=111 (you
change the duration of the execution for the job to 111 seconds)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/destruction
<http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/destruction>
The server returns the date and time when the job shall be destroyed.
You can also modify this date, doing this:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/destruction?DESTRUCTION=2010-12-31T11:00:00 (you
change the destruction date to the 31 of December of 2010 at 11h AM)
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/parameters
The server returns the parameters you have included for the query in the
creation of the job.
As an option, it is possible to modify some parameter of the job (MAXREC
and/or LANG and/or QUERY and/or FORMAT) , simply requesting in this
way: http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/parameters?MAXREC=10000&LANG=ADQL
<http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/parameters?MAXREC=10000&LANG=ADQL>
Now, it's time for running the query. For this, we have the "phase"
endpoint. This endpoint offers you either the possibility of running a
job (if its state is /PENDING/ or /QUEUED/) or aborting the job (if its
state is /EXECUTING/ or /QUEUED/ as well) . E.g. to abort the recently
created job, we have to call the server in this way:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase?PHASE=abort
The server will return the job info, where you can check that its new
state is /ABORTED/.
On the other hand, if you want to run the job , you have to type :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase?PHASE=run
The server will answer quickly, with the job info, and you will
see its state as /EXECUTING/. To keep track of the execution of the job,
you can choose several options:
* Search the job into the job list
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async
* Watch the job info
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async?JOBID=515
* Watch the phase for the job (waiting for
the /COMPLETED/ state) http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/phase
When the job state reaches the status of /COMPLETED/, you can already
see the result :
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/results/result
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/result
or see the error document (if there was an error when processing the job):
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/error
And finally, you can choose the last main option: */deleting a job/*.
You will delete a job typing this URL:
http://esavo02.esac.esa.int:8080/VOTAP/services/TapService/async/515/delete
In addition, a daemon would be running to check the destruction dates
for every job, to delete the ones exceeding that date.
I think that's all. If you have any question or suggestion about this
matter, do not hesitate to let us know.
Kind regards,
Carlos Ríos
(ESAVO)
================================================================================================
This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure,
use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in
error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by
the sender.
Please consider the environment before printing this email.
=================================================================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.ivoa.net/pipermail/dal/attachments/20101112/8208e2e5/attachment-0001.html>
More information about the dal
mailing list