Running hazard search for entire Cybershake on Mahuika
You may wish to install python packages geopy, psycopg2, numpy, pyarrow, etc. Use pip install --user to install missing packages.
pip install geopy, psycopg2, numpy, pyarrow --user
Make sure all the IM Agg have been computed and stored under each fault of Cybershake IMs directory. ie. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/IMs/XXXX/IM_agg
Edit `common/constants.py` [!!! Should be in the seisfinder root directory as a config file. This needs an overhaul ]. You need to specify DB_PATH and TEMP_DIR
import os DB_PATH = '/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6.db' VERSION_OPTIONS = ["18p9p3"] SEARCH_OPTIONS = ["event", "site"] EVENT_SEARCH = 1 SITE_SEARCH = 2 CLI_PROGRAMS_OPTIONS = ["guided", "argument"] TIME_OPTIONS = ["historical", "future"] LOCATION_OPTIONS = ["single", "multiple"] SINGLE_LOCATION = 1 MULTIPLE_LOCATION = 2 TEMP_DIR = "/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/hazard/" # location where any user can read, write and execute TMPLOCATION_TO_CREATE_HAZARD_FILES = os.path.join(TEMP_DIR,"Hazard") SEISFINDER_RUN_CMD = 'python seisfinder.py'
Notice VERSION_OPTIONS above. Edit /common/versions/18p9p3.ini. Specifically, we will be enabling site search for v18p6.
Generate a fault list. The optional --run_dir will check if all the simulation result in run_dir have a matching fault data in srf_dir.
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/gen_fault_list.py --help usage: gen_fault_list.py [-h] [--run_dir RUN_DIR] srf_dir out_csv positional arguments: srf_dir Source directory out_csv Output CSV filename optional arguments: -h, --help show this help message and exit --run_dir RUN_DIR Run directory for extra validation baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/gen_fault_list.py Data/Sources /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6_fault_list.csv --run_dir /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Runs Warning: More faults found than simulated results : Ignoring these ['FiordSZ03', 'FiordSZ09'] Output CSV successfully written: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6_fault_list.csv
Build the DB out of Cybershake data. Identify where Cybershake Runs directory is located (eg. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Runs) and go to the root of Cybershake. (eg. /nesi/nobackup/nesi00213/RunFolder/Cybershake/ ) [!!! Should be in the seisfinder root directory as a config file. This needs an overhaul ]
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake$ python /nesi/project/nesi00213/deploy/seisfinder2/utils/data_import/create_cybershake_csvs.py -v v18p6
This creates CSV files under v18p6/CSV
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/CSV$ ls -ltr total 4609 -rw-rw-r-- 1 baes nesi00213 4103996 Oct 16 02:50 SimRun.csv -rw-rw-r-- 1 baes nesi00213 341250 Oct 16 02:50 SimRunMap.csv -rw-rw-r-- 1 baes nesi00213 236 Oct 19 02:05 SimRunSet.csv
Currently, SimRunSet.csv needs to be manually edited. [!!! Should be in the seisfinder root directory as a config file. This needs an overhaul ]
id,name,type,simRunSetPath,empiricalPath,faultListPath 1,v18p6,1,/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/,/nesi/nobackup/nesi00213/empiricals/v18p5/,/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6_fault_list.csv
Note that the name "v18p6" should be the same as the name specified in the version .ini file. Finally, Add CSV data into the DB. You can specify the location of DB file (if it doesn't exist, it will automatically create one)
python /nesi/project/nesi00213/deploy/seisfinder2/utils/data_import/insert_csvdata_to_db.py /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6.db /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/CSV
Generate a list of locations .csv from a .ll file.
base ll2csv.sh /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll
This creates /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.csv
Edit execute_hazard_search.sl, especially the line starting with "srun" at the bottom.
#!/bin/bash # script version: slurm # Please modify this file as needed, this is just a sample #SBATCH --job-name=hazard_search_multi #SBATCH --account=nesi00213 #SBATCH --partition=prepost #SBATCH --ntasks=1 ####SBATCH --cpus-per-task=36 #SBATCH --time=00:59:00 #SBATCH --output hazard_search_multi-%j.out #SBATCH --error hazard_search_multi-%j.err ###SBATCH --mail-type=all ###SBATCH --mail-user=test@test.com ###SBATCH --mem-per-cpu=16G ###SBATCH -C avx #OpenMP+Hyperthreading works well for VM ###SBATCH --hint=nomultithread ## END HEADER source machine_env.sh date srun python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py -m v18p6 /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.csv SA_5p0 date
Submit this job.
sbatch execute_hazard_search.sl
When the job has completed, check the last -10 lines of .out log file.
baes@mahuika02: ~/tmp$ tail -10 hazard_search_multi-85422.out Instructions in the multicase are not written in extense as it is not useful. Please use the scripts below Please go to hazard/ Then run the scripts created as: bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/hazard_calcs.sh bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_im_value.sh bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_exceedance_rate.sh Note: The IM_VALUE and EXCEEDANCE should be adjusted on the deagg_* scripts Mon Sep 3 03:30:39 UTC 2018
This means it has created a temp directory /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 and placed empirical files (symbolic links) and bash scripts that contain Python commands.
Let's call this temp location a working directory.
There is hazard_calcs.sh in the working directory. For entire cybershake, this can be very very long. We will be using SLURM's parallel for loop to run this in parallel, but it needs to be split to avoid SLURM complaining about too many lines. This involves quite a bit of steps, but it can be largely automated by executing prepare_hpc_hazard_calc.sh. Go to the working directory, and execute it. You just need to supply the path to the working directory.
baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 $ bash /nesi/project/nesi00213/deploy/seisfinder2/site/mahuika/prepare_hpc_hazard_calc.sh .
The above does EVERYTHING for you. It copies the .sl template to the working directory, splits the hazard_calc.sh, and generates all the .sl files ready for submission.
baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0$ bash ./submit_all.sh Submitted batch job 69390 Submitted batch job 69391 Submitted batch job 69392 Submitted batch job 69393 Submitted batch job 69394 Submitted batch job 69395 Submitted batch job 69396 Submitted batch job 69397 Submitted batch job 69398 Submitted batch job 69399 Submitted batch job 69400 Submitted batch job 69401 Submitted batch job 69402 Submitted batch job 69403 Submitted batch job 69404 Submitted batch job 69405 Submitted batch job 69406 Submitted batch job 69407 Submitted batch job 69408
Hazard map: When everything is completed, you can submit the following .sl script that has been automatically generated for you. Just edit the exceedance value if required. For details, see https://github.com/ucgmsim/seisfinder2/tree/master/CLI/hazard
baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_5F73LY_baes$ sbatch execute_hazard_map.sl