IM Calculation Improvement

Submission integrated into automatic workflow

Large simulations chunked during loading and then processed. Memory usage is higher than expected but do not envision any cases where the memory usage exceeds limitations.

Management DB

Improved speed to return jobs needing to be submitted

Added queue, start and end time parameters to be stored in the DB.

Added retries count to automatically retry simulations

Daemon

Adjustable wall-clock parameter

18p9 run

Selected the runs to be done

Hossack 501r
PaeroaC 167r
Mangatete 334r
AlpineF2K 1r

Testing Plan

Day 1:

• Generate list of faults

• Autogenerate SRF and VM

Day 1 (night):

• First run of 18p9

Day 2

• Fix automation steps

Day 2 (night)

• Second test run

Day 3

• Repeat as necessary

Day n (waiting for storage issues to be fixed)

• Port workflow to Maui

Day n + 2

• Run 18p9 test on Maui

• Fix automation as necessary

Day n + X

• Resubmit 18p6 magnitude faults on Maui

Testing Results

Day 1

• Remove dependancy from SRF_generation on post-processing (change to empirical engine)
• Permission issue on VMs generated

Day 2

• VM code missing plotting data and incorrect QCoRE repository location – ensure QCoRE is configured correctly
• HF issue - issue with random seed / -i parameter

Day 3

• BB - DT/NT relation between HF and LF inconsistent resulting in failing BB
  • Missing parameter in the vm-generation step. Going to make generation scripts / parameter checks before running.
• Retry parameter not functioning correctly - now logging into the management error log on each retry and each reset of the retries
• IM calc tasks:
  
  • Completion checking
  • Automation
  • Inconsistent naming

Day 4

• Task time missing log - functionality existed but did not call the functions.

Test - Missing Batch 1 - Cybershake 18p6

This batch was missed because of an issue with large binary files - these submissions were moved from the automatic submission and weren't run.

• IM Calc not updating to in-queue in DB creating lots of repeated submissions of IMCalc job. (fixed)

Was plotted manually for validation.

Test - Wrong Magnitude - Cybershake 18p6

SRF / VM Generated

• Determined faults that didn't have VMs generated and found the issue

Simulations Run

• 1 LF simulations had errors - potentially IO error (re-running fixed this)
• IM Calc marked as failed but completed. (fixed)
• Need to fix error logs
• Only some tasks automatically retried

Viewing output now

Downloading IMs and plotting on hypocentre

Maui / Mahuika Run

SRF and VM generation running on Mahuika after a sucessful GMT deploy

Still have some issues with plotting SRFs

After some initial simulation tests a fully automated v18p9 is running.

Will analyse metadata from run.

Day N+1

• Issue with IM_calc deployment.(fixed):
  
  • Cython is not included in default python module on Maui.
  • File/Script permission issue that preventing others to run calculate_ims.py

Outstanding Tasks

Permission issue on VMs generated

Automatic plotting of single realisation for each fault

Cleanup of intermediary steps in workflow

Slurm script generation for SRF/VM plotting on Mahuika (including SRF and VM plots)