See NamdAtNICS for information on Kraken, a Cray XT5 with compute node Linux (officially called Cray Linux Environment or CLE).

Environment variables for large runs

NAMD sends many unexpected messages. Cray MPI reduces the cutoff for eager protocol messages for larger jobs. NAMD benefits greatly from fixing this value so that most messages are sent via eager protocol via:

setenv MPICH_MAX_SHORT_MSG_SIZE 50000

MPICH errors on Cray XT

It may be necessary to set MPICH_PTL_OTHER_EVENTS to a very high number (perhaps 20000 but please experiment yourself) to avoid these errors:

[0] MPICH PtlEQPoll error (PTL_EQ_DROPPED): An event was dropped on the
OTHER EQ handle.  Try increasing the value of env var
MPICH_PTL_OTHER_EVENTS (cur size is 2048).

Running SMP binaries on Cray XT

You must use the -d option to aprun to specify the number of threads NAMD will launch (ppn + comm thread) or all will be mapped to a single core. For example, the command used to run one process per node on Kraken is:

aprun -n $NUMNODES -N 1 -d 12 $NAMDBIN +ppn 11 +setcpuaffinity +pemap 1-11 +commap 0 $CONFFILE >& $LOGFILE

Two processes per node:

@ PROCESSES = $NUMNODES * 2
aprun -n $PROCESSES -N 2 -d 6 $NAMDBIN +ppn 5 +setcpuaffinity +pemap 1-8,10,11 +commap 0,9 $CONFFILE >& $LOGFILE

Three Processes per node:

@ PROCESSES = $NUMNODES * 3
aprun -n $PROCESSES -N 3 -d 4 $NAMDBIN +ppn 3 +setcpuaffinity +pemap 1-5,7,8,10,11 +commap 0,6,9 $CONFFILE >& $LOGFILE

The -d option is definitely needed if the +pemap and +commap options are not used.

Building on Cray XT5 (new way)

First switch to the GNU compilers with "module swap PrgEnv-pgi PrgEnv-gnu"

To build Charm++ (from the charm source directory):

./build charm++ mpi-crayxt --no-build-shared --with-production

To build NAMD use CRAY-XT-g++.

Building on Cray XT5 (old way, not recommended)

The XT5 uses Opteron processors, runs Linux. and requires and MPI-based NAMD. The only strange thing is that there is no mpicc/mpiCC, you just use cc and CC. Usually pathcc, pgcc, and gcc are available, and you use modules to select which one is mapped to cc/CC. If it works gcc will probably give the best performance. The Intel compiler would probably be better but is usually not available.

To build Charm++ (from the charm source directory):

env MPICXX=CC MPICC=cc ./build charm++ mpi-linux-x86_64 --no-build-shared -O -DCMK_OPTIMIZE=1

To build NAMD use arch CRAY-XT-pgcc for the PGI compilers or CRAY-XT-g++ for the GNU compilers.

For NAMD 2.7b1 you will need to change the CXX definition in arch/CRAY-XT-pgcc.arch to:

CXX = CC -D'nearbyint(X)'='floor((X)+0.5)'

Noise on Cray XT5

Its been noticed that when using all 12 cores on each node on the Cray XT5 machine, NAMD performance suffers. Hence, it is advised that NAMD users try running with 11, 10 or even 9 cores per node to see which configuration gives them the best performance for a given number of nodes.

Here are a few different aprun commands to try (when running on 100 nodes, 1200 cores):

aprun -n 1200 ./namd2 apoa1.namd (use all 12 cores per node)

aprun -n 1100 -N 11 -cc 1,2,3,4,5,6,7,8,9,10,11 ./namd2 apoa1.namd (exclude core 0)

aprun -n 1000 -N 10 -cc 1,2,3,4,5,6,7,8,10,11 ./namd2 apoa1.namd (exclude cores 0 and 9)

aprun -n 900 -N 9 -cc 1,2,3,4,5,7,8,10,11 ./namd2 apoa1.namd (exclude cores 0, 6 and 9)