[MITgcm-devel] speeding up scp for many files over "long distances"

[Ed Hill]

Hi folks (esp. Patrick and Baylor),

I wrote a bash script that uses multiple simultaneous scp processes to
speed up the transfer of multiple files from "remote" locations such as
NASA Ames, GFDL, etc.  Its called "multi_scp" and you can get it at:

  http://mitgcm.org/cgi-bin/viewcvs.cgi/development/edhill/misc_scripts/

Its really easy to use--just run "multi_scp -h" for help.

The technique is useless for systems that are "close" to each other (eg.
two machines here on campus).  But it can give *dramatic* speedups under
some (usually, when the machines are far apart) circumstances:

  http://mitgcm.org/eh3/lou.nas.nasa.gov/lou_data_xfer.png

Also, you're going to need to have SSH public/private keys setup to make
this script practical.  Theres a good SSH pub/priv key tutorial at:

  http://www-106.ibm.com/developerworks/linux/library/l-keyc.html

Ed

-- 
Edward H. Hill III, PhD
office:  MIT Dept. of EAPS;  Rm 54-1424;  77 Massachusetts Ave.
             Cambridge, MA 02139-4307
emails:  eh3 at mit.edu                ed at eh3.com
URLs:    http://web.mit.edu/eh3/    http://eh3.com/
phone:   617-253-0098
fax:     617-253-4464