Transfer of large data

[Andreas Wicenec]

Hi Masahiro,

during our development for the ALMA dataflow to the ALMA archive we 
encountered the same problems and we found the
following solution, which gives us a performance of up to 96 MB/s on a 
Gigabit Ethernet connection.

We are using both HTTP and Corba based AV streaming where the payload 
is in both cases a message with mime-type
multipart/related. One of the parts (the first in fact) is a VOTable, 
which points to binary attachments within
the same message. You can find a more detailed description along with 
an example application and message file under
http://almasw.hq.eso.org/almasw/bin/view/Archive/VOTableDataTransfer. I 
actually made a proposal for some very small
modifications in VOTable, which would make the integration of such 
message files even simpler, but unfortunately
that was rejected at the time, maybe we should try again.

The other point is that Apache Axis is not implemented to support 
streaming as this would probably solve the main
issues you found.

Andreas

> Dear all,
>
> Have you experienced large data transfer (e.g. >100MB) between 
> distributed VO machine? Would you tell your practical solution?
>
> Our JVO team has built a SkyNode server which receives a query and 
> returns a result VOTable through Web Services, using Apache Axis. Then 
> we encounter a performance problem in large data transfer using SOAP 
> messages; Large amount of memory is allocated to hold the entire data 
> as Java objects. It slows processing speed, too. Sending large data 
> through SOAP messages seems unrealistic.
>
> So we are considering SkyNodes pass result VOTable using:
>
>    1. Staging (Result data are separately transferred through HTTP or 
> FTP, like SIAP)
>    2. Attachment Both will need some extension to SkyNode protocol.
>
> Masahiro Tanaka