Definitive version of the VOTable schema for web services
Roy Williams
roy at cacr.caltech.edu
Wed Jul 30 11:46:59 PDT 2008
Dave Morris wrote:
> It is not beyond imagining that a valid science query to a large data
> set could return 21G bytes of data.
>
> Whatever we replace/update VOTable with it should be easy process the
> service response as a stream of rows,
Dave
VOTable is well suited to handling bulk data. The answer, in my opinion,
is NOT to encode everything in XML and SOAP, but rather use either the
binary or FITS data streams in the VOTable specification. The metadata
stays in XML, the big data streams in fixed-length records. It is easy
to split into parallel streams, allow parallel and asynchronous
evaluation, etc. This approach scales well beyond your gigabyte dataset
-- by factors of a 1000 or more.
Roy
More information about the grid
mailing list