Done

  • Latest version of GMT was ported to Kupe
  • nhm2srf, nhm2vm, gcmt2srf , srfinfo2vm codes are modified to suit HPC environment 
  • Fully automated 2 paths 
    1. GCMT -> SRF, SRF info->VM 
    2. NHM-> SRF, SRF info->VM 
  • Only 1 slurm script to submit. 
  • Automated VM integrity test.

 

Issues and solutions

  • GMT doesn't run in MPI unless all processes run GMT in unison. Abandoned MPI and chose multiprocess. 
  • GMT 6.xx "spatial" output was broken. Downgraded to GMT 5.4.3. The bug reported. (http://gmt.soest.hawaii.edu/issues/1245)
  • HDF5 on Cray : File locking issue: Needs `$ export HDF5_USE_FILE_LOCKING=FALSE` (https://github.com/ALPSCore/ALPSCore/issues/348)
  • gs for psconvert is not available on Kupe compute node. Copying and placing gs binary doesn work as files in `/usr/share/ghostscript/9.15`/ are missing. ImageMagick can be tried instead. For now, we don't produce png.

 

NHM2SRF_VM.sl
#!/bin/bash
# script version: slurm
# Please modify this file as needed, this is just a sample
#SBATCH --job-name=nhm2srf_vm
#SBATCH --account=nesi00213
#SBATCH --partition=NeSI
#SBATCH --ntasks=1
#SBATCH --time=03:00:00
#SBATCH --output nhm2srf_vm-%j.out
#SBATCH --error nhm2srf_vm-%j.err
###SBATCH --mail-type=all
###SBATCH --mail-user=test@test.com
###SBATCH --mem-per-cpu=16G
###SBATCH -C avx
#SBATCH --hint=nomultithread
## END HEADER
source machine_env.sh
# user specific parameters
export NUMPROCS=40
export SELECTION_FILE=`pwd`/nhm_selection_geo_si.txt #nhm_selection_sample2.txt #
# important paths
export SRFGENPATH=/nesi/nobackup/nesi00213/tmp/Pre-processing/SrfGen
export QCOREPATH=/nesi/nobackup/nesi00213/tmp/qcore
export GMPEMODELSPATH=/nesi/nobackup/nesi00213/tmp/post-processing/im_processing/computations/GMPE_models
# leave these alone
export PYTHONPATH=$QCOREPATH:$PYTHONPATH:$SRFGENPATH:$GMPEMODELSPATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/nesi/nobackup/nesi00213/gmt/gmt-stable/lib64
# to fix HDF5 file locking issue https://github.com/ALPSCore/ALPSCore/issues/348
export HDF5_USE_FILE_LOCKING=FALSE
echo "SRF generation starts"
date
python2 $SRFGENPATH/NHM/nhm2srf.py $SELECTION_FILE -n $NUMPROCS
echo "SRF generation completed"
date
echo "VM generation starts"
srun python2 $SRFGENPATH/srfinfo2vm.py "autosrf/*/Srf/*.info" -n $NUMPROCS
date
echo "VM generation completed"

When submitted and executed, it creates two directories `autosrf` and `autovm` 

baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autosrf$ ls
AhuririR     Caswell4          FiordMar1&2    Hundalee       MilB5GeoR2    NorthCant13   Springbank
Akatore      Caswell5          FiordSZ03      Hunter         MilfordB1     NorthCant2    Springfield
Albury       CaswellH10        FiordSZ09      HuttPeelNorth  MoonlightNth  NorthCant4    Spylaw
AlpineF2K    CaswellH67        FiveFingers    HuttPeelSouth  MoonlightSth  NorthCant8    Stonewall
AlpineK2T    CaswellH8         Fowlers        Hyde           MS01          OhariuS       Swedge1
AlpineR      CaswellH9         FoxPk          Inangahua      MS02          OldMan        Swedge2
Ashley       CBalleny          GeorgeR1       IrishmanC      MS04          Omihi         Swedge3
AwatereNE    Chalky1to3        Gimmerburn     JorKekCha      MS05          Opawa         Swedge5
AwatereSW    Chalky4to8        Grandview      JorKekNeed     MS09          Ostler        Swedge6to10
AwatNEVer    Cheeseman         Hanmer         KaiwaraN       MtWhite       Otematata     TaieriR
AwatNEVerCl  ClarenceCentr     Harper         KaiwaraS       Nevis         PaparoaRF     Takitimu1
Barefell     ClarenceNE        Hauroko        Kakapo         NMFZ1819      Pegasus1nw    Takitimu2
Billys       ClarenceSW        HikWgtnmax     KekerenguBF    NMFZ4647      Pisa          TeAnau
BlueLk       Cust              HikWgtnmin     KekNeed        NMFZB0        Port2GreyL    Torlesse
BlueMtn      Cw4Swedge411      Hokonui        Kelly          NMFZB1        Port2GreyS    UpperSlope
BooBooALL    Cwedge123         Hollyford      Kirkliston     NMFZB2        Poulter       WaimeaN
Brothers     Dalgety           Hope1888       LeesV          NMFZE1        QuartzCk      WaimeaS
Browning     DoubleHill        HopeConway     LindisP        NMFZE2        Raggedy       Waipiata
BrunAnt      Dryburgh          HopeConwayOS   LkHeron        NMFZF1        Rakaia        WairarapNich
CardronaN    Dunstan           HopeCW         logfile.txt    NMFZF2        Ranfurly      Wairau
CardronaS    Esk               HopeTARA       LongVlly       NorthBranch   Settlement    Waitangi
Caswell1     fault_traces.gmt  HopeTeRapa1n2  Lowry          NorthCant1    SFiordMg13    Waitohi
Caswell211   FernGully         Hororata       Lyell          NorthCant10   SFiordMg1to9  WharaToCampB
Caswell3     Fidget            HumpR          Maimai         NorthCant11   Solander      WhiteCk


baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autosrf/Akatore$ ls
Srf  Stoch
baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autosrf/Akatore$ cd Srf
baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autosrf/Akatore/Srf$ ls
Akatore_HYP01-04_S1244.info  Akatore_HYP02-04_S1284.info  Akatore_HYP03-04_S1324.info
Akatore_HYP01-04_S1244.srf   Akatore_HYP02-04_S1284.srf   Akatore_HYP03-04_S1324.srf
Akatore_HYP01-04_S1254.info  Akatore_HYP02-04_S1294.info  Akatore_HYP04-04_S1334.info
Akatore_HYP01-04_S1254.srf   Akatore_HYP02-04_S1294.srf   Akatore_HYP04-04_S1334.srf
Akatore_HYP01-04_S1264.info  Akatore_HYP03-04_S1304.info  Akatore_HYP04-04_S1344.info
Akatore_HYP01-04_S1264.srf   Akatore_HYP03-04_S1304.srf   Akatore_HYP04-04_S1344.srf
Akatore_HYP02-04_S1274.info  Akatore_HYP03-04_S1314.info  Akatore_HYP04-04_S1354.info
Akatore_HYP02-04_S1274.srf   Akatore_HYP03-04_S1314.srf   Akatore_HYP04-04_S1354.srf
baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autosrf/Akatore/Srf$

 

Similarly, `autovm` directory contains velocity model files for each location.

baes@kupe01: /scale_akl_nobackup/filesets/nobackup/nesi00213/tmp/auto_preproc/nhm_test/autovm/Akatore$ ls
gridfile_rt01-h0.400      model_coords_rt01-h0.400  params_vel.py   VeloModCorners.txt
gridout_rt01-h0.400       model_params_rt01-h0.400  params_vel.pyc  vp3dfile.p
model_bounds_rt01-h0.400  nzvm.cfg                  rho3dfile.d     vs3dfile.s

 

To Do

  • Technical debts: /nesi/nobackup/qcore /nesi/nobackup/post-processing need to be updated, and other codes that depend on old versions need to be revised
  • No labels