Page History

570,020

Xeon Phi 7250 68C 1.4Ghz

KNL: 1 node 68 cores (1 socket) * 8305 nodes (96Gb+16Gb)/node

SKL: 1 node 40 cores (2 sockets * 20 cores/socket) * 132 nodes (192Gb/node)

Building EMOD3D was somewhat tricky. I ended up having my own version of CMake 3.9 (existing module has no ccmake, and later versions of CMake are buggy), and fftw3 (existing module didn't have fftw3f, and CMake failed to pick up.

Originally build with Intel tool chain, but EMOD3D had rounding error issues, and it generates incompatible random numbers (different from Maui). For best (and consistent) result, using GNU tool chain is highly recommended.

The following modules are used.

craype-network-opa

gcc

craype-mic-knl

mvapich2

mvapich2 is required as mpi4py doesn't seem to work properly with openmpi

Don't bother with fftw3 module. We need to build fftw3 from scratch: only fftw3f (single) version is needed.

FFTW3

export MPICC='mpicc -fPIC -march=knl'

export CC='gcc -fPIC -march=knl'

./configure --enable-float --enable-sse --enable-threads --host=x86_64-pc-linux --enable-shared --prefix=/home01/hpc11a02/gmsim/Environments/nurion/ROOT/local/gnu

make all install

EMOD3D

mkdir build

cd build

export FFTW_DIR=/home01/hpc11a02/gmsim/Environments/nurion/ROOT/local/gnu

cmake ..

GMT

Prerequisite

• curl
• sqlite-snapshot-202004061816,
• zlib-1.2.11,
• libpng-1.6.37,
• tiff-4.1.0,
• GraphicsMagick-1.3.35,
• proj-7.0
• gdal-3.0.1
  
  

Except for GDAL, this works:

$HOME=/home01/x2319a02

$ PKG_CONFIG_PATH=$HOME/gmsim/Environments/nurion/ROOT/local/gnu/lib/pkgconfig
$ ./configure --prefix=$HOME/gmsim/Environments/nurion/ROOT/local/gnu & make all install

For GDAL,

module add netcdf

CPPFLAGS=-I$HOME/gmsim/Environments/nurion/ROOT/local/gnu/include PKG_CONFIG_PATH=$HOME/gmsim/Environments/nurion/ROOT/local/gnu/lib/pkgconfig
./configure --prefix=/$HOME/gmsim/Environments/nurion/ROOT/local/gnu --with-proj=$HOME/gmsim/Environments/nurion/ROOT/local/gnu & make all install

(Edit: I had to manually add CPPFLAGS into config.status (2022/11/25)

For GMT,

go to build

cmake -DDCW_PATH:PATH=$HOME/gmsim/Environments/nurion/ROOT/share/dcw-gmt-1.1.4 -DGSHHG_PATH:PATH=$HOME/gmsim/Environments/nurion/ROOT/share/gshhg-gmt-2.3.7 ../

make all install

!WARNING!

"qsub" MUST be executed in $SCRATCH directory.

isam

$ lfs quota -h /home01

$ lfs quota -h /scratch

1 gujwa = KNL 6,400 node time (100 SRU time) = 435,000 core hours

XXX sec * 4350/3600 = core hours