Running hazard search for entire Cybershake on Mahuika

...

pip install geopy psycopg2 pyarrow shapely --user

Consider upgrading pip by

...

then update PATH to use the pip in $HOME/.local/bin/pip

Setup

Your Cybershake root directory must contain Runs and Data.

• Runs: Make sure all the IM Agg have been computed and stored under each fault

...

• 's Runs directory. ie.  /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/

...

• Runs/XXXX/IM_

...

• agg
• Data: Containing VMs and Sources 

Go to the Cybershake root directory (eg. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6) and execute the setup.py

============ BELOW is going to be replaced by a single python script setup.py ===============

Generate a fault list.  The optional  --run_dir will check if all the simulation result in run_dir have a matching fault data in srf_dir. 

baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/gen_fault_listtool/setup.py --help 

usage: gen_fault_listsetup.py [-h] [--runn NAME] cybershake_dir RUN_DIR] srfempirical_dir outll_csvfile
positional arguments:
  srfcybershake_dir        Cybershake Root directory eg.
         Source directory
  out_csv               /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6
  empirical_dir         Empirical data root directory eg.
                        /nesi/nobackup/nesi00213/empiricals/v18p5
  ll_file               Path to Outputthe CSV.ll filenamefile
optional arguments:
  -h, --help            show this help message and exit
  --run_dir RUN_DIR  Run directory for extra validation


n NAME, --name NAME  Optional name for Cybershake.

module add Python/2.7.14-gimkl-2017a 

The following example was taken from v18p6_3 directory.

baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$v18p6_3$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/gen_fault_list.py Data/Sources  /nesitool/setup.py . /nesi/nobackup/nesi00213/empiricals/v18p6 non_uniform_whole_nz_with_real_stations-hh400_v18p6.ll


/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/imdb.h5
{empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p6, fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/v18p6_fault_list.csv,
  hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard,
  ll: --run_dir /nesi/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll,
  name: v18p6_3, root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3,
  runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs,
  srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources}

Config file has been successfully written: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/config.yaml
Warning: More faults found than simulated results : Ignoring these
['OtaraWest02', 'OtaraWest01', 'Wairaka05', 'OtaraEast03', 'FiordSZ09', 'FiordSZ03', 'TaurTrE02', 'Tuakana11', 'FiordSZ09Ohena02', 'MaungatiE01', 'Ohena01']
Output CSV successfully written: /nesi/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/v18p6_fault_list.csv

...

/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/

...

baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/data_import/create_cybershake_csvs.py -v v18p6 

This creates CSV files under v18p6/CSV

baes@mahuika02: /nesiv18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll

Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/CSV$ ls -ltr
total 4609
-rw-rw-r-- 1 baes nesi00213 4103996 Oct 16 02:50 SimRun.csv
-rw-rw-r-- 1 baes nesi00213  341250 Oct 16 02:50 SimRunMap.csv
-rw-rw-r-- 1 baes nesi00213     236 Oct 19 02:05 SimRunSet.csv

Currently, SimRunSet.csv needs to be manually edited. [!!! Should be in the seisfinder root directory as a config file. This needs an overhaul ]

_3/imdb_calc.sl
sbatch imdb_calc.sl
Submitted batch job 1874061

This creates a config.yaml that contains the all the necessary config info, creates fault lists, converts .ll into .csv etc, and builds IMDB by crafting a .sl script and automatically submitting it. (This takes about an hour)

In earlier versions, setup.py used to produce hazard_calc_X.sl scripts. We have a better solution, and no longer support this. This step will be deprecated. 

Written /scale_wlg_nobackup/filesets

id,name,type,simRunSetPath,empiricalPath,faultListPath
1,v18p6,1,/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/,/nesi_3/Hazard/hazard_calc_SA_0p5.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/empiricals/v18p5/,/nesiRunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_PGA.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6_fault_list.csv

...

python /nesi/project/nesi00213/deploy/seisfinder2/utils/data_import/insert_csvdata_to_db.py /nesi_3/Hazard/hazard_calc_SA_0p2.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p1.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6.db /nesi_3/Hazard/hazard_calc_SA_5p0.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/CSV

Edit  `common/constants.py` [!!! Should be in the seisfinder root directory as a config file. This needs an overhaul ]. You need to specify DB_PATH and TEMP_DIR 

import os

DB_PATH = '/nesi_3/Hazard/hazard_calc_SA_0p2.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6.db'
VERSION_OPTIONS = ["18p9p3"]
SEARCH_OPTIONS = ["event", "site"]
EVENT_SEARCH = 1
SITE_SEARCH = 2
CLI_PROGRAMS_OPTIONS = ["guided", "argument"]
TIME_OPTIONS = ["historical", "future"]
LOCATION_OPTIONS = ["single", "multiple"]
SINGLE_LOCATION = 1
MULTIPLE_LOCATION = 2
TEMP_DIR = "/nesi_3/Hazard/hazard_calc_SA_3p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_2p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/hazard/" # location where any user can read, write and execute
TMPLOCATION_TO_CREATE_HAZARD_FILES = os.path.join(TEMP_DIR,"Hazard")
SEISFINDER_RUN_CMD = 'python seisfinder.py'

Notice VERSION_OPTIONS above. Edit /common/versions/18p9p3.ini. Specifically, we will be enabling site search for v18p6.   [!!!! Needs an overhaul !!!!]

...
[site]
quakes = v18p6
...

Generate a list of locations .csv from a .ll file. 

_3/Hazard/hazard_calc_SA_5p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_1p0.sl

A sample config.yaml looks like this

empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p6
fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv
hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard
ll: /scale_wlg_nobackup/filesets

bash /nesi/project/nesi00213/deploy/seisfinder2/utils/ll2csv.sh /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll

...

name: v18p6_3
root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6

...

_3
runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs
srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources

Run

Go to Cybershake root directory. (Make sure machine_env.sh has been imported and)

baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --location -42.7012 172.8003 --hazard

This calculates hazard for the given location for SA_5p0.

You can use a CSV file that contains latitude and longitude of multiple locations.

baes@mahuika02: ~$

================= RUN ================

Edit  execute_hazard_search.sl, especially the line starting with "srun" at the bottom.

#!/bin/bash
# script version: slurm
# Please modify this file as needed, this is just a sample
#SBATCH --job-name=hazard_search_multi
#SBATCH --account=nesi00213
#SBATCH --partition=prepost
#SBATCH --ntasks=1
####SBATCH --cpus-per-task=36
#SBATCH --time=00:59:00
#SBATCH --output hazard_search_multi-%j.out
#SBATCH --error hazard_search_multi-%j.err
###SBATCH --mail-type=all
###SBATCH --mail-user=test@test.com
###SBATCH --mem-per-cpu=16G
###SBATCH -C avx
#OpenMP+Hyperthreading works well for VM
###SBATCH --hint=nomultithread
## END HEADER
source machine_env.sh
date
srun python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-mversion v18p6_3 --im SA_5p0 --csv-location /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.csv SA_5p0
date

Submit this job. 

sbatch execute_hazard_search.sl

When the job has completed, check the last -10 lines of .out log file.

baes@mahuika02: ~/tmp$ tail -10 hazard_search_multi-85422.out
Instructions in the multicase are not written in extense as it is not useful.                    Please use the scripts below
Please go to hazard/
Then run the scripts created as:
bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/hazard_calcs.sh
bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_im_value.sh
bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_exceedance_rate.sh
Note: The IM_VALUE and EXCEEDANCE should be adjusted on the deagg_* scripts
Mon Sep  3 03:30:39 UTC 2018

This means it has created a temp directory /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 and  placed empirical files (symbolic links) and bash scripts that contain Python commands. 

Let's call this temp location  a working directory.

There is hazard_calcs.sh in the working directory.  For entire cybershake, this can be very very long. We will be using SLURM's parallel for loop to run this in parallel, but it needs to be split to avoid SLURM complaining about too many lines. This involves quite a bit of steps, but it can be largely automated by executing prepare_hpc_hazard_calc.sh. Go to the working directory, and execute it. You just need to supply the path to the working directory.

_2/test.csv --hazard

Note that once you set parameters, cs-version, im and locations etc., the system remembers them. So you can simply execute the following command and get the same result

baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --hazard

This can be useful if you want to compute deagg for the same location and same setting. You just need to put additional parameters such as exceedance as below.

baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --deagg --exceedance 0.0051

You can make the system forget the setting by --clear switch

baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --clear

If you put --script option, it won't do the calculation. Instead it will output a file hazard_calcs.sh in the current directory. This is useful if you wish to use NeSI to run hazard calculation for all locations and produce hazard map.

baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --csv-location /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_2/test.csv --hazard --script

Suppose there is ~/hazard_calcs.sh. You can run prepare_hpc_hazard.calcs.sh for SLURM submission.

baes@mahuika02: ~$

baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 $ bash /nesi/project/nesi00213/deploy/seisfinder2/site/mahuika/prepare_hpc_hazard_calc.sh .

The above does EVERYTHING for you. It copies the .sl template to the working directory, splits the hazard_calc.sh, and generates all the .sl files ready for submission.

 

Found working directory /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0
Found 80 unique locations
Making a slurm script for processing locations 0 - 80
....Done!!
Go to the following directory and run submit_all.sh
/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm

Everything you need is placed in the directory. Go to the directory and execute submit_all.sh (SLURM doesn't like to process more than 10000 lines. If there are more locations than the limit, it will create multiple .sl scripts and submit_all.sh will submit them all for you)

Code Block
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm$

 

Code Block

baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0$ bash ./submit_all.sh Submitted batch job 69390 Submitted batch job 69391 Submitted batch job 69392 Submitted batch job 69393 Submitted batch job 69394 Submitted batch job 69395 Submitted batch job 69396 Submitted batch job 69397 Submitted batch job 69398 Submitted batch job 69399 Submitted batch job 69400 Submitted batch job 69401 Submitted batch job 69402 Submitted batch job 69403 Submitted batch job 69404 Submitted batch job 69405 Submitted batch job 69406 Submitted batch job 69407 Submitted batch job 69408...

 

Hazard map: When everything is completed, you can submit the following .sl script that has been automatically generated for you. Just edit the exceedance value if required.  For details, see https://github.com/ucgmsim/seisfinder2/tree/master/CLI/hazard

Code Block
baes@mahuika02: ~/cybershake/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_search_v18p6_SA_5p0$5p0/slurm$ sbatch execute_hazard_map.sl

 

 

  

Sample Outputs

Hazard Curve

site_search.py --cs-version 18.6 --location -43.89 170.435 --hazard --im PGA

Image Added

Deaggregation

./site_search.py --cs-version 18.6 --location -43.89 170.435 --deagg --im PGA --exceedance 0.01

GM Selection

site_search.py --cs-version 18.6 --location -43.89 170.435 --gmsel --im PGA --imvalue 0.1

Image AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added