Definitive version of the VOTable schema for web services

Roy Williams roy at cacr.caltech.edu
Wed Jul 30 11:46:59 PDT 2008


Dave Morris wrote:
>  It is not beyond imagining that a valid science query to a large data 
> set could return 21G bytes of data.
>
> Whatever we replace/update VOTable with it should be easy process the 
> service response as a stream of rows,
Dave

VOTable is well suited to handling bulk data. The answer, in my opinion, 
is NOT to encode everything in XML and SOAP, but rather use either the 
binary or FITS data streams in the VOTable specification. The metadata 
stays in XML, the big data streams in fixed-length records. It is easy 
to split into parallel streams, allow parallel and asynchronous 
evaluation, etc. This approach scales well beyond your gigabyte dataset 
-- by factors of a 1000 or more.

Roy



More information about the grid mailing list