Transfer of large data

[Maria A. Nieto-Santisteban]

At JHU we are planning on including and evaluating the meaga-streamimg 
protocols UDT and SOAP* developed by Grossman et al. at the National 
center for Data Mining (NCDM). 

UDT (UDP-based Data Transfer) for files and SOAP* for Web services.

here are some references:


[1] A. Chien, T. Faber, A. Falk, J. Bannister, R. Grossman, and J. Leigh.
Transport protocols for high performance: Whither TCP? Communications of 
the ACM, 46(11):42. 49, 2003.

[2] R. L. Grossman, M. Mazzucco, H. Sivakumar, Y. Pan, and Q. Zhang. 
SABUL, Simple Available Bandwidth Utilization Library for high-speed wide 
area networks. Journal of Supercomputing, to appear.

Cheers,

Maria


On Thu, 21 Jul 2005, Masahiro TANAKA wrote:

> Dear all,
> 
> Have you experienced large data transfer (e.g. >100MB) between
> distributed VO machine?  Would you tell your practical solution?
> 
> Our JVO team has built a SkyNode server which receives a query and
> returns a result VOTable through Web Services, using Apache Axis.
> Then we encounter a performance problem in large data transfer
> using SOAP messages; Large amount of memory is allocated to hold
> the entire data as Java objects. It slows processing speed, too.
> Sending large data through SOAP messages seems unrealistic.
> 
> So we are considering SkyNodes pass result VOTable using:
>  1. Staging (Result data are separately transferred through HTTP or
> 	     FTP, like SIAP)
>  2. Attachment
> Both will need some extension to SkyNode protocol.
> 
> Masahiro Tanaka
> 

-- 
------------------------------------------------
Maria A. Nieto-Santisteban (nieto at pha.jhu.edu)
Johns Hopkins University
3400 N. Charles St.
Physics & Astronomy Department
Baltimore, MD 21218 (USA)

Tel: 	1 410 516-7679  Fax: 	1 410 516-5096