Running hazard search for entire Cybershake on Mahuika
You may wish to install python packages geopy, psycopg2, numpy, pyarrow, etc. Use pip install --user to install missing packages.
pip install geopy psycopg2 shapely --user
Consider upgrading pip by
pip upgrading pip --user
then update PATH to use the pip in $HOME/.local/bin/pip
Setup
Setup part may need updating. imdb_create needs to be added too
Your Cybershake root directory must contain Runs and Data.
- Runs: Make sure all the IM Agg have been computed and stored under each fault's Runs directory. ie. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Runs/XXXX/IM_agg
- Data: Containing VMs and Sources
Go to the Cybershake root directory (eg. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6) and execute the setup.py
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$ python /nesi/project/nesi00213/deploy/seisfinder2/tool/setup.py --help
usage: setup.py [-h] [-n NAME] cybershake_dir empirical_dir ll_file
positional arguments:
cybershake_dir Cybershake Root directory eg.
/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6
empirical_dir Empirical data root directory eg.
/nesi/nobackup/nesi00213/empiricals/v18p5
ll_file Path to the .ll file
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME Optional name for Cybershake.
module add Python/2.7.14-gimkl-2017a
The following example was taken from v18p6_3 directory.
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3$ python /nesi/project/nesi00213/deploy/seisfinder2/tool/setup.py . /nesi/nobackup/nesi00213/empiricals/v18p6 non_uniform_whole_nz_with_real_stations-hh400_v18p6.ll
/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/imdb.h5
{empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p6, fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv,
hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard,
ll: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll,
name: v18p6_3, root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3,
runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs,
srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources}
Config file has been successfully written: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/config.yaml
Warning: More faults found than simulated results : Ignoring these
['OtaraWest02', 'OtaraWest01', 'Wairaka05', 'OtaraEast03', 'FiordSZ09', 'FiordSZ03', 'TaurTrE02', 'Tuakana11', 'Ohena02', 'MaungatiE01', 'Ohena01']
Output CSV successfully written: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv
/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/imdb_calc.sl
sbatch imdb_calc.sl
Submitted batch job 1874061
This creates a config.yaml that contains the all the necessary config info, creates fault lists, converts .ll into .csv etc, and builds IMDB by crafting a .sl script and automatically submitting it. (This takes about an hour)
In earlier versions, setup.py used to produce hazard_calc_X.sl scripts. We have a better solution, and no longer support this. This step will be deprecated.
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p5.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_PGA.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p2.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p1.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_5p0.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p2.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_3p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_2p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_5p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_1p0.sl
A sample config.yaml looks like this
empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p6 fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard ll: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll name: v18p6_3 root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3 runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources
Run
Go to any directory (preferably somewhere your home)
Make sure machine_env.sh has been imported and
baes@mahuika02: ~$ python ~/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --location -42.7012 172.8003 --hazard
This calculates hazard for the given location for SA_5p0.
You can use a CSV file that contains latitude and longitude of multiple locations.
baes@mahuika02: ~$ python ~/seisfinder2/site/site_search.py--cs-version v18p6_3 --im SA_5p0 --csv /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_2/test.csv --hazard
If you put --script option, it won't do the calculation. Instead it will output a file hazard_calcs.sh in the current directory. This is useful if you wish to use NeSI to run hazard calculation for all locations.
baes@mahuika02: ~$ python ~/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --csv /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_2/test.csv --hazard --script
Suppose there is ~/hazard_calcs.sh. You can run prepare_hpc_hazard.calcs.sh for SLURM submission.
baes@mahuika02: ~$ bash /nesi/project/nesi00213/deploy/seisfinder2/site/mahuika/prepare_hpc_hazard_calc.sh Found working directory /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0 Found 80 unique locations Making a slurm script for processing locations 0 - 80 ....Done!! Go to the following directory and run submit_all.sh /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm
Everything you need is placed in the directory. Go to the directory and execute submit_all.sh (SLURM doesn't like to process more than 10000 lines. If there are more locations than the limit, it will create multiple .sl scripts and submit_all.sh will submit them all for you)
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm$ ./submit_all.sh Submitted batch job 69390 Submitted batch job 69391 Submitted batch job 69392 Submitted batch job 69393 ...
Hazard map: When everything is completed, you can submit the following .sl script that has been automatically generated for you. Just edit the exceedance value if required. For details, see https://github.com/ucgmsim/seisfinder2/tree/master/CLI/hazard
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm$ sbatch execute_hazard_map.sl