optimising namd ibverb runs

From: Thanassis Silis (djnass_18_at_hotmail.com)
Date: Mon May 18 2015 - 06:48:03 CDT

I am running some relatively small andm simulations in a system of 6 blade processing servers. Each has the following specs


each of the 4 processors has 8 cores and due to hyper-threading 16 threads are available.
Indeed, cat /proc/cpuinfo returns 64 cpus on each system.

I have created a nodelist file using the infiband interface ip address - I am also using the ibverbs namd executable. I have run several test simulations to figure out which setting minimizes processing time. Overall it seems that for 64 cpus/system * 6 systems = 384 cpus , I get to minimize the processing time by using "+p128 +setcpuaffinity"

This seems odd as it is 1/3 of the available cpus. It's not half - which would seem sensible (if one of each core's threads works, it utilizes the full resources of the other thread of the core and this maximizes performance).

One of the things I tried was to let the system decide which cpu's to use, with
charmrn namd2 ++nodelist nodelist +setcpuaffinity `numactl --show | awk '/^physcpubind/ {printf "+p%d +pemap %d",(NF-1),$2; for(i=3;i<=NF;++i){printf ",%d",$i}}'` sim.conf > sim.log

and also to manually assign worker threads and comminucation threads. I may (or may not!) have managed that with the command
charmrun namd2 ++nodelist nodelist +setcpuaffinity +p64 +pemap 0-63:16.15 +commap 15-63:16
In this above command, I am not sure how should I "see" the 64 * 6 cpus. as 6 same systems ? (so add +p64), or aggregate them to 384 cpus (so add +p384 above). I did try +p384 but it seems to be even slower - way too many threads have been spawned.

So I am fuzzy. Why do I get minimized process time when 1/3 of the 384 cpus are used and no manual settings are in place? Are charmrun and namd2 clever enough at this version (2.10) that they assign worker and comm threads automagically?

Is there some other parameter that you suggest I should append, because at the very least using 1/3 of the cpus seems Very odd.

Thank you for your time and input.

This archive was generated by hypermail 2.1.6 : Tue Dec 27 2016 - 23:21:08 CST