[MITgcm-devel] cs510 on IBM p690

[Dimitris Menemenlis]

> can you do a 27 CPU test on columbia

27-cpu will not work on columbia because of limited memory

54-cpu works fine

Could Martin's problem be an MPI buffer overflow, which is not caught by MPI 
implementation?  Or maybe a bug in the particular MPI library that he is using? 
  In running on Columbia we had to play around with a bunch of MPI variables 
For a 216-CPU configuration, we are using following environment variables:

limit descriptors unlimited
limit stacksize 2000m
limit coredumpsize 1
module load modules scsl.1.6.1.0 intel-comp.9.1.039 mpt.1.12.0.nas
setenv MPI_DSM_DISTRIBUTE
setenv MPI_BUFS_PER_PROC  512
setenv MPI_BUFS_PER_HOST  512
setenv MPI_MSGS_PER_HOST 2048
setenv MPI_MSGS_PER_PROC 1024
setenv MPI_MSG_RETRIES 5000
setenv KMP_STACKSIZE 1000m
setenv KMP_LIBRARY turnaround

Not sure in detail what all of the above do but they are needed both to speed up 
the code "and" to avoid MPI buffer overflows.

D.