Asynchronous querying and tabular data

[Patrick Dowler]

This looks like a pretty good analysis of the situation. The claim is that in 
typical DAL the queryData is always fast and thus fine as a sync method while 
in TAP the primary/necessary operation is the async query for a query result 
because the query is expensive/time consuming. I agree with this in principle 
and think we should design for async. Since async has not seen much support 
so far, I feel we have a rather clean slate to start from.

However, there are still going to be many cases were a TAP service has modest 
content, a sync query would work fine, and a service supporting sync query 
only is much easier to implement robustly. Here I am mainly thinking that 
when you develop anything async, you bring into the implementation nice 
things like server-side resource contention, cleanup after some amount of 
time, controlled access to staged results (via obscurity or security), and 
all that kind of stuff. For the sync response you just dump it to the output 
stream and you are done.

So, priority has to be to make the async operation very functional and 
natural, but I think allow a sync-only TAP service to be compliant. 
Practically, one can write some wrapper code to make an async call 
synchronous, but I think this is better handled as a client side option. Thus 
it looks to me like syncQuery and asyncQuery both need to be optional, but 
you obviously have to support one of them :)

Pat

PS - Examples of small TAP  services I can envision:

* Peter Stetson's standard star catalog
* proper motion catalog
* solar system catalog (think of the fun with region/STC in this one :)
* supernova catalog

These would all be relatively small and highly useful TAP services. A simple 
sync implementation is quite feasible. OTOH, we are also building a source 
catalog to capture results from the entire CADC data collection; I estimated 
roughly 5-6 billion sources (~5TB)...

On Tuesday 01 May 2007 08:18, Kona Andrews wrote:
> Dear all,
>
> Copied below is a useful discussion from a colleague of why access
> protocols like SIAP and SSAP don't extend so gracefully to large
> tabular data queries, and why therefore we shouldn't try to make
> TAP exactly conform to the model assumed by these protocols.


-- 

Patrick Dowler
Tel/Tél: (250) 363-6914                  | fax/télécopieur: (250) 363-0045
Canadian Astronomy Data Centre   | Centre canadien de donnees astronomiques
National Research Council Canada | Conseil national de recherches Canada
Government of Canada                  | Gouvernement du Canada
5071 West Saanich Road               | 5071, chemin West Saanich
Victoria, BC                                  | Victoria (C.-B.)