Transfer of large data

[Roy Williams]

The key to efficient transfer of big data is to separate (XML) metadata 
from (binary) data. The metadata can contain pointers to the data 
(http:// or srb:// or gridftp://).

When I buy a single brick at my local hardware store, I take it with me 
to the cashier and deal with metadata (payment) and data (the brick) 
together. But when I buy 1,000 bricks, it is different. I pay the 
cashier and receive a piece of paper (the pointer), then I take the 
paper somewhere else to load the bricks into my truck.

As Andreas points out, VOTable was built in this way, to represent 
table metadata, with pointers to binary or FITS data elsewhere. It is 
not advised to use the TR,TD mechanism of VOTable to represent large 
datasets. In the same way, the VOStore specification is being built 
with the possibility of splitting metadata from data.

Splitting metadata from data is more efficient.
But it requires more effort to make sure the two are properly 
synchronized.

Roy

California Institute of Technology
626 395 3670
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/enriched
Size: 1110 bytes
Desc: not available
URL: <http://www.ivoa.net/pipermail/dal/attachments/20050721/bedac482/attachment-0001.bin>