Versions Compared

Old Version 35

changes.mady.by.user Sung Bae

Saved on

compared with

New Version Current

changes.mady.by.user Sung Bae

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
This page is outdated

Running hazard search for entire Cybershake on Mahuika

...

Code Block
pip install geopy psycopg2 pyarrow shapely --user

Consider upgrading pip by

...

then update PATH to use the pip in $HOME/.local/bin/pip

 


Setup

Your Cybershake root directory must contain Runs and Data.

  • Runs: Make sure all the IM Agg have been computed and stored under each fault

...

  • 's Runs directory. ie.  /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/

...

  • Runs/XXXX/IM_

...

  • agg
  • Data: Containing VMs and Sources 

Go to the Cybershake root directory (eg. /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6) and execute the setup.py


Code Block
languagebash

Setup

Code Block
module add Python/2.7.14-gimkl-2017a
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$ python /nesi/project/nesi00213/deploy/seisfinder2/utilstool/setup.py --help 

usage: setup.py [-h] [-n NAME] cybershake_dir empirical_dir ll_file
positional arguments:
  cybershake_dir        Cybershake Root directory eg.
                        /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6
  empirical_dir         Empirical data root directory eg.
                        /nesi/nobackup/nesi00213/empiricals/v18p5
  ll_file               Path to the .ll file
optional arguments:
  -h, --help            show this help message and exit
  -n NAME, --name NAME  Optional name for Cybershake.




module add Python/2.7.14-gimkl-2017a 



The following example was taken from v18p6_3 directory.

Code Block
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6$v18p6_3$ python /nesi/project/nesi00213/deploy/seisfinder2/utilstool/setup.py . /nesi/nobackup/nesi00213/empiricals/v18p5v18p6 non_uniform_whole_nz_with_real_stations-hh400_v18p6.ll -n v18p6

This creates a config.yaml that contains the all the necessary config info, then creates CSVs out of IM Agg, and add the data into the DB, creates fault lists, converts .ll into .csv etc. 

 

A sample config.yaml looks like this

Code Block
csv_dir: 


/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/CSV
db: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/v18p6.db
_3/imdb.h5
{empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p5
v18p6, fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv,
  hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard,
ims_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/IMs
  ll: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll,
  name: v18p6
_3, root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3,
  runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs,
  srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources

 

Notice VERSION_OPTIONS above. Edit /common/versions/18p9p3.ini. Specifically, we will be enabling site search for v18p6.   [!!!! Needs an overhaul !!!!]

Code Block
...
[site]
quakes = v18p6
...

 

Run

...

}

Config file has been successfully written: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/config.yaml
Warning: More faults found than simulated results : Ignoring these
['OtaraWest02', 'OtaraWest01', 'Wairaka05', 'OtaraEast03', 'FiordSZ09', 'FiordSZ03', 'TaurTrE02', 'Tuakana11', 'Ohena02', 'MaungatiE01', 'Ohena01']
Output CSV successfully written: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv
/scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll

Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6

...

_3/imdb_calc.sl
sbatch imdb_calc.sl
Submitted batch job 1874061


This creates a config.yaml that contains the all the necessary config info, creates fault lists, converts .ll into .csv etc, and builds IMDB by crafting a .sl script and automatically submitting it. (This takes about an hour)


Info
In earlier versions, setup.py used to produce hazard_calc_X.sl scripts. We have a better solution, and no longer support this. This step will be deprecated. 

Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_0p5.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_PGA.sl
Written

Navigate to the directory and find .sl scripts that have been already created for you.

 

A sample hazard_calc_SA_5p0.sl

Code Block
#!/bin/bash
# script version: slurm
# Please modify this file as needed, this is just a sample
#SBATCH --job-name=hazard_search_multi
#SBATCH --account=nesi00213
#SBATCH --partition=prepost
#SBATCH --ntasks=1
####SBATCH --cpus-per-task=36
#SBATCH --time=00:59:00
#SBATCH --output hazard_search_multi-%j.out
#SBATCH --error hazard_search_multi-%j.err
###SBATCH --mail-type=all
###SBATCH --mail-user=test@test.com
###SBATCH --mem-per-cpu=16G
###SBATCH -C avx
#OpenMP+Hyperthreading works well for VM
###SBATCH --hint=nomultithread
## END HEADER
SITECODE=/nesi/project/nesi00213/deploy/seisfinder2/site
source $SITECODE/mahuika/machine_env.sh
date
srun python $SITECODE/site_search.py -d /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6 -m v18p6 _3/Hazard/hazard_calc_SA_0p2.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/non_uniform_whole_nz_with_real_stations-hh400_v18p6.csv SA_5p0 
date

 

Submit this job. 

 

Code Block
sbatch _3/Hazard/hazard_calc_SA_0p1.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_5p0.tar.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_5p0.sl

When the job has completed, check the last -10 lines of .out log file.

 

Code Block
baes@mahuika02: /nesi0p2.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_calc_SA_3p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Hazard$$ tail -10 hazard_search_multi-85422.out
Instructions in the multicase are not written in extense as it is not useful. Please use the scripts below
Please go to hazard/
Then run the scripts created as:
bash /home/baes/cybershake/v18p6_3/Hazard/hazard_calc_SA_2p0.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_searchcalc_v18p6_SA_5p0/hazard_calcs.sh
bash /home/baes/cybershake/v18p6.sl
Written /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_im_value.sh
bash /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0/deagg_calcs_exceedance_rate.sh
Note: The IM_VALUE and EXCEEDANCE should be adjusted on the deagg_* scripts
Mon Sep  3 03:30:39 UTC 2018

This means it has created a temp directory /home/baes/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 and  placed empirical files (symbolic links) and bash scripts that contain Python commands. 

 

Let's call this temp location  a working directory.

...

_calc_SA_1p0.sl



A sample config.yaml looks like this

Code Block
empirical_dir: /nesi/nobackup/nesi00213/empiricals/v18p6
fault_list: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/fault_list.csv
hazard_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard
ll: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/non_uniform_whole_nz_with_real_stations-hh400_v18p6_land.ll
name: v18p6_3
root_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3
runs_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Runs
srf_dir: /scale_wlg_nobackup/filesets/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Data/Sources

 

Run

Go to Cybershake root directory. (Make sure machine_env.sh has been imported and)

Code Block
baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --location -42.7012 172.8003 --hazard

This calculates hazard for the given location for SA_5p0.

You can use a CSV file that contains latitude and longitude of multiple locations.

Code Block
baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --csv-location /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_2/test.csv --hazard

Note that once you set parameters, cs-version, im and locations etc., the system remembers them. So you can simply execute the following command and get the same result

Code Block
baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --hazard

This can be useful if you want to compute deagg for the same location and same setting. You just need to put additional parameters such as exceedance as below.

Code Block
baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --deagg --exceedance 0.0051

You can make the system forget the setting by --clear switch

Code Block
baes@mahuika02: ~$ python
Code Block
baes@mahuika02: ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0 $ bash /nesi/project/nesi00213/deploy/seisfinder2/site/mahuika/prepare_hpc_hazard_calc.sh .

The above does EVERYTHING for you. It copies the .sl template to the working directory, splits the hazard_calc.sh, and generates all the .sl files ready for submission.

Submit jobs: sbatch each .sl or use submit_all.sh, which submits all the jobs at one.
site_search.py --clear


If you put --script option, it won't do the calculation. Instead it will output a file hazard_calcs.sh in the current directory. This is useful if you wish to use NeSI to run hazard calculation for all locations and produce hazard map.

...

Code Block
baes@mahuika02: ~$ python /nesi/project/nesi00213/deploy/seisfinder2/site/site_search.py --cs-version v18p6_3 --im SA_5p0 --csv-location /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_2/test.csv --hazard --script


Suppose there is ~/hazard_calcs.sh. You can run prepare_hpc_hazard.calcs.sh for SLURM submission.

Code Block
baes@mahuika02: ~$ bash /nesi/project/nesi00213/deploy/seisfinder2/site/mahuika/prepare_hpc_hazard_calc.sh 

Found working directory /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0
Found 80 unique locations
Making a slurm script for processing locations 0 - 80
....Done!!
Go to the following directory and run submit_all.sh
/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm

 

Everything you need is placed in the directory. Go to the directory and execute submit_all.sh (SLURM doesn't like to process more than 10000 lines. If there are more locations than the limit, it will create multiple .sl scripts and submit_all.sh will submit them all for you)

Code Block
baes@mahuika02: /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/SA_5p0/slurm$ ./submit_all.sh ~/cybershake/v18p6/Hazard/hazard_search_v18p6_SA_5p0$ bash ./submit_all.sh
Submitted batch job 69390
Submitted batch job 69391
Submitted batch job 69392
Submitted batch job 69393
Submitted batch job 69394
Submitted batch job 69395
Submitted batch job 69396
Submitted batch job 69397
Submitted batch job 69398
Submitted batch job 69399
Submitted batch job 69400
Submitted batch job 69401
Submitted batch job 69402
Submitted batch job 69403
Submitted batch job 6940469390
Submitted batch job 6940569391
Submitted batch job 6940669392
Submitted batch job 69407
Submitted batch job 6940869393
...

 

Hazard map: When everything is completed, you can submit the following .sl script that has been automatically generated for you. Just edit the exceedance value if required.  For details, see https://github.com/ucgmsim/seisfinder2/tree/master/CLI/hazard

Code Block
baes@mahuika02: ~/cybershake/nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_3/Hazard/hazard_search_v18p6_SA_5p0$5p0/slurm$ sbatch execute_hazard_map.sl

 

 
  
Sample Outputs
Hazard Curve
site_search.py --cs-version 18.6 --location -43.89 170.435 --hazard --im PGA
site_search.py --cs-version 18.6 --location -43.89 170.435 --hazard --im PGAImage Added
Deaggregation
./site_search.py --cs-version 18.6 --location -43.89 170.435 --deagg --im PGA --exceedance 0.01
 View file
nameDeaggregation_Cyber_Station0101d2c_PGA_IMVal0.157364.txt
height250
 View file
nameDeaggregation_Empirical_Station0101d2c_PGA_IMVal0.157364.txt
height250
GM Selection
site_search.py --cs-version 18.6 --location -43.89 170.435 --gmsel --im PGA --imvalue 0.1
Image AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added
 View file
nameautomated_selection.txt
height250