367,024

Xeon Phi 7250 68C 1.4GHz

KNL: 1 node 68 cores (1 socket) = 272 hyper threads BUT 64-68MPI tasks advisable * 4200 KNL nodes (96Gb+16Gb)/node

SKX: 1 nodes 48 cores (= 2 sockets* 24 cores/socket) = 96 hyper threads * 1,736 nodes

SKX is slightly more expensive than KNL

$HOME: 10Gb (200,000 files)
$WORK: 1Tb (3mil files) : not for high IO, large files. nobackup, no purge

$SCRATCH: unlimited. nobackup, deleted if not accessed for 10 day.

 /nesi/project/nesi00213 == $HOME/project

/nesi/nobackup/nesi00213 == $HOME/nobackup or $SCRATCH/nobackp

Building

Intel

module add fftw3/3.3.8 intel/18.0.2 impi/18.0.2 cmake/3.10.2

MPI_C_LIB_NAMES = mpifort;mpi;mpigi;dl;rt;pthread

MPI_dl_LIBRARY = /usr/lib64/libdl.so

MPI_pthread_LIBRARY = /usr/lib64/libthread.so

MPI_rt_LIBRARY =  /usr/lib64/librt.so

By default gcc-6.5 creeps in and it attempts to build with gcc-6.5 instead of icc. Enforce it with CC=icc.

I found "make VERBOSE=1" extremely useful to debug building issues

GCC

1. Load the correct modules:
   module add git/2.24.1 cmake/3.16.1 TACC impi/17.0.3 libfabric/1.7.0
autotools/1.1 xalt/2.8 gcc/7.1.0 python3/3.6.1 hdf5/1.10.4

2. Build FFTW (3.3.8)

   ./configure --enable-float --enable-sse --enable-threads
--host=x86_64-pc-linux --enable-shared --prefix=$SOMEWHERE
make all install

   1. Now that initial set up has been completed the following commands can be used for the GCC workflow:
      1. activate_env /work/06833/sungbae/stampede2/Environments/stampede_gcc
      2. module restore gcc_modules
      3. export LIBRARY_PATH=$LIBRARY_PATH:~

3. Build EMOD3D
mkdir build
cd build
FFTW_DIR=$SOMEWHERE cmake ../
make

Issue

emod3d has a rounding error issue with icc and returns wrong "ny" failing post-emod3d test. Rob Graves fixed this by converting float to double in the function get_n1n2() in misc.c. The fix is included in 3.0.6 (On Nurion, however, this fix was found to be not enough)

Running

Project name must be CamelCase: DesignSafe-Graves

Slurm script needs -N for number of nodes

#SBATCH -N 4
#SBATCH --ntasks=160

Instead of "srun" it uses "ibrun"

Workflow

A number of hardcoded bits assuming NeSI machine need to be updated. Check workflow and qcore "stampede" branches.

https://github.com/ucgmsim/slurm_gm_workflow/tree/stampede

https://github.com/ucgmsim/qcore/tree/stampede

(python3_stampede) sungbae@stampede21(1):~$ /usr/local/etc/taccinfo
---------------------- Project balances for user sungbae ----------------------
| Name Avail SUs Expires | |
| DesignSafe-Graves 19974 2020-09-30 | |
------------------------ Disk quotas for user sungbae -------------------------
| Disk Usage (GB) Limit %Used File Usage Limit %Used |
| /home1 0.8 10.0 7.82 1853 200000 0.93 |
| /work 10.0 1024.0 0.97 55539 3000000 1.85 |
| /scratch 11.0 0.0 0.00 4032 0 0.00 |
-------------------------------------------------------------------------------

570,020

Xeon Phi 7250 68C 1.4Ghz

KNL: 1 node 68 cores (1 socket) * 8305 nodes (96Gb+16Gb)/node

SKL: 1 node 40 cores (2 sockets * 20 cores/socket) * 132 nodes (192Gb/node)

Building EMOD3D was somewhat tricky. I ended up having my own version of CMake 3.9 (existing module has no ccmake, and later versions of CMake are buggy), and fftw3 (existing module didn't have fftw3f, and CMake failed to pick up.

Originally build with Intel tool chain, but EMOD3D had rounding error issues, and it generates incompatible random numbers (different from Maui). For best (and consistent) result, using GNU tool chain is highly recommended.

The following modules are used.

craype-network-opa

gcc

craype-mic-knl

mvapich2

mvapich2 is required as mpi4py doesn't seem to work properly with openmpi

Don't bother with fftw3 module. We need to build fftw3 from scratch: only fftw3f (single) version is needed.

FFTW3

export MPICC='mpicc -fPIC -march=knl'

export CC='gcc -fPIC -march=knl'

./configure --enable-float --enable-sse --enable-threads --host=x86_64-pc-linux --enable-shared --prefix=/home01/hpc11a02/gmsim/Environments/nurion/ROOT/local/gnu

make all install

EMOD3D

mkdir build

cd build

export FFTW_DIR=/home01/hpc11a02/gmsim/Environments/nurion/ROOT/local/gnu

cmake ..

GMT

Prerequisite

• curl
• sqlite-snapshot-202004061816,
• zlib-1.2.11,
• libpng-1.6.37,
• tiff-4.1.0,
• GraphicsMagick-1.3.35,
• proj-7.0
• gdal-3.0.1
  
  

Except for GDAL, this works:

$HOME=/home01/x2319a02

$ PKG_CONFIG_PATH=$HOME/gmsim/Environments/nurion/ROOT/local/gnu/lib/pkgconfig
$ ./configure --prefix=$HOME/gmsim/Environments/nurion/ROOT/local/gnu & make all install

For GDAL,

module add netcdf

CPPFLAGS=-I$HOME/gmsim/Environments/nurion/ROOT/local/gnu/include PKG_CONFIG_PATH=$HOME/gmsim/Environments/nurion/ROOT/local/gnu/lib/pkgconfig
./configure --prefix=$HOME/gmsim/Environments/nurion/ROOT/local/gnu --with-proj=$HOME/gmsim/Environments/nurion/ROOT/local/gnu & make all install

(Edit: I had to manually add CPPFLAGS into config.status (2022/11/25)

For GMT,

go to build

cmake -DDCW_PATH:PATH=$HOME/gmsim/Environments/nurion/ROOT/share/dcw-gmt-1.1.4 -DGSHHG_PATH:PATH=$HOME/gmsim/Environments/nurion/ROOT/share/gshhg-gmt-2.3.7 ../

make all install

!WARNING!

"qsub" MUST be executed in $SCRATCH directory.

isam

$ lfs quota -h /home01

$ lfs quota -h /scratch

1 gujwa = KNL 6,400 node time (100 SRU time) = 435,000 core hours

XXX sec * 4350/3600 = core hours