...
Make sure you are currently under the test_calculate_ims folder, run: pytest
| Code Block |
|---|
$ pytest -v -s test_calculate_ims.py |
CHECKPOINTING & SPLITTING A BIG SLURM
...
The output IM_calc folder is under:
Sample command to run checkpointing:(use generate_sl2.py)
| Code Block |
|---|
$ python generate_sl2.py -s /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_batched/v18p6_1k_under2p0G_ab/Runs -srf /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Runs/test_srfs/ -ll /scale_akl_nobackup/filesets/transit/nesi00213/StationInfo/non_uniform_whole_nz_with_real_stations-hh400_v18p6.ll -np 80 -o ~/test_obs/IMCalcExample /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/test_check_point/ > test.sl |
To submit the slurm script:
| Code Block |
|---|
$cp test.sl /nesi/nobackup/nesi00213/tmp/auto_preproc
$sbatch test.sl |
The reason that we have to run test.sl under '/nesi/nobackup/nesi00213/tmp/auto_preproc' is otherwise slurm cannot find machine.env specified by the test.sl script:
Splitting a big slurm
Splitting a big slurm script into several smaller slurms is need due to maximum number of lines allowed in a slurm script on Kupe.
...
We have 61 rrup files to run, by specifying -ml 100 (100 python calls to calcualte_rrups.py per slurm script), we expect 1 sim slurm scripts to be outputted.
Command to run checkpointing and splitting:
| Code Block |
|---|
python generate_split_sl3sl.py -s /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6_batched/v18p6_1k_under2p0G_ab/Runs -srf /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/Runs/test_srfs/ -ll /scale_akl_nobackup/filesets/transit/nesi00213/StationInfo/non_uniform_whole_nz_with_real_stations-hh400_v18p6.ll -np 80 -o ~/test_obs/IMCalcExample/ /nesi/nobackup/nesi00213/RunFolder/Cybershake/v18p6/test_check_point/ -ml 100 |
Output:
To submit the slurm script:
| Code Block |
|---|
$cp test.sl /nesi/nobackup/nesi00213/tmp/auto_preproc
$sbatch test.sl |
The reason that we have to run 'test.sl' under '/nesi/nobackup/nesi00213/tmp/auto_preproc' is otherwise slurm cannot find machine.env specified by the test.sl script:
Todo: tidy up check_point.py, generate_sl[*|2|3].py
TODO
- Creation of semi-automatic slurm generation that will have all the calls to produce the results as needed.
- Progress printing statements
- Rrup calculation on a smaller station list - currently when generating the slurm script it does the full grid even for stations outside the domain
...