Asynchronous querying and tabular data

[Alex Szalay]

I think that when we get into the async responses, this will
have to be part of the VOStore/VOSpace story, i.e. the output
goes into a friendly VOSpace location and sends the handle
to the result.

--Alex 

-----Original Message-----
From: owner-voql-teg at eso.org [mailto:owner-voql-teg at eso.org] On Behalf Of
Patrick Dowler
Sent: Tuesday, May 01, 2007 2:08 PM
To: voql-teg at ivoa.net
Subject: Re: Asynchronous querying and tabular data


This looks like a pretty good analysis of the situation. The claim is that
in typical DAL the queryData is always fast and thus fine as a sync method
while in TAP the primary/necessary operation is the async query for a query
result because the query is expensive/time consuming. I agree with this in
principle and think we should design for async. Since async has not seen
much support so far, I feel we have a rather clean slate to start from.

However, there are still going to be many cases were a TAP service has
modest content, a sync query would work fine, and a service supporting sync
query only is much easier to implement robustly. Here I am mainly thinking
that when you develop anything async, you bring into the implementation nice
things like server-side resource contention, cleanup after some amount of
time, controlled access to staged results (via obscurity or security), and
all that kind of stuff. For the sync response you just dump it to the output
stream and you are done.

So, priority has to be to make the async operation very functional and
natural, but I think allow a sync-only TAP service to be compliant. 
Practically, one can write some wrapper code to make an async call
synchronous, but I think this is better handled as a client side option.
Thus it looks to me like syncQuery and asyncQuery both need to be optional,
but you obviously have to support one of them :)

Pat

PS - Examples of small TAP  services I can envision:

* Peter Stetson's standard star catalog
* proper motion catalog
* solar system catalog (think of the fun with region/STC in this one :)
* supernova catalog

These would all be relatively small and highly useful TAP services. A simple

sync implementation is quite feasible. OTOH, we are also building a source 
catalog to capture results from the entire CADC data collection; I estimated

roughly 5-6 billion sources (~5TB)...

On Tuesday 01 May 2007 08:18, Kona Andrews wrote:
> Dear all,
>
> Copied below is a useful discussion from a colleague of why access
> protocols like SIAP and SSAP don't extend so gracefully to large
> tabular data queries, and why therefore we shouldn't try to make
> TAP exactly conform to the model assumed by these protocols.


-- 

Patrick Dowler
Tel/Tél: (250) 363-6914                  | fax/télécopieur: (250) 363-0045
Canadian Astronomy Data Centre   | Centre canadien de donnees astronomiques
National Research Council Canada | Conseil national de recherches Canada
Government of Canada                  | Gouvernement du Canada
5071 West Saanich Road               | 5071, chemin West Saanich
Victoria, BC                                  | Victoria (C.-B.)