Transfer of large data

[Anita Richards]

Hi,

http://www.hep.man.ac.uk/u/rich
see Activities and talks

http://www.evlbi.org/evlbi/evlbi.html (e.g. Steve Parsely recent eVLBI 
results, links to Dante/GEANT)

Some info mainly related to transfering radio interferometry data (raw and 
FITS not VOTable) - at much higher data rates (0.5 - 1 or more Gb/s) - but 
there are some issues which may be relevant such as tools for testing 
local networks, since sometimes the slowness is a configuration issue. 
But my impression is that VOTable is not ideal for very large files 
(Gb?)....

The main thing for VOs, though, should be avoiding having to make very 
large data transfers by locating tools with data, optimising workflow 
order etc.?

cheers
a



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Dr. Anita M. S. Richards, AstroGrid Astronomer
MERLIN/VLBI National Facility, University of Manchester, 
Jodrell Bank Observatory, Macclesfield, Cheshire SK11 9DL, U.K. 
tel +44 (0)1477 572683 (direct); 571321 (switchboard); 571618 (fax).


On Thu, 21 Jul 2005, Andreas Wicenec wrote:

> Hi Masahiro,
>
> during our development for the ALMA dataflow to the ALMA archive we 
> encountered the same problems and we found the
> following solution, which gives us a performance of up to 96 MB/s on a 
> Gigabit Ethernet connection.
>
> We are using both HTTP and Corba based AV streaming where the payload is in 
> both cases a message with mime-type
> multipart/related. One of the parts (the first in fact) is a VOTable, which 
> points to binary attachments within
> the same message. You can find a more detailed description along with an 
> example application and message file under
> http://almasw.hq.eso.org/almasw/bin/view/Archive/VOTableDataTransfer. I 
> actually made a proposal for some very small
> modifications in VOTable, which would make the integration of such message 
> files even simpler, but unfortunately
> that was rejected at the time, maybe we should try again.
>
> The other point is that Apache Axis is not implemented to support streaming 
> as this would probably solve the main
> issues you found.
>
> Andreas
>
>> Dear all,
>> 
>> Have you experienced large data transfer (e.g. >100MB) between distributed 
>> VO machine? Would you tell your practical solution?
>> 
>> Our JVO team has built a SkyNode server which receives a query and returns 
>> a result VOTable through Web Services, using Apache Axis. Then we encounter 
>> a performance problem in large data transfer using SOAP messages; Large 
>> amount of memory is allocated to hold the entire data as Java objects. It 
>> slows processing speed, too. Sending large data through SOAP messages seems 
>> unrealistic.
>> 
>> So we are considering SkyNodes pass result VOTable using:
>> 
>>    1. Staging (Result data are separately transferred through HTTP or FTP, 
>> like SIAP)
>>    2. Attachment Both will need some extension to SkyNode protocol.
>> 
>> Masahiro Tanaka
>
>