Task

Expected time reqd.

People

Notes

1. Source          (HOBY)

GeoNet Mw email to SRF

(a)  from Mw tensor get Rup parameters

(b)  From Rup parameters use createSRF.py to make rupture SRF

(c)  Execution of createSRF.py

1minute

5mins

10sec

6min total

Hoby

a) Manually extract location, dip,strike, rake from Mw tensor (Ristau) into setSrfParams.py

b) Change TYPE=2 (point source to finite fault)

c) Run "createSRF.py" (Make sure to use python2)

Plotting of SRF

(a)  SRF plane(s) standard plotting

(b)  SRF on map to verify location (to come after domain set in step 2)

3mins

5mins

2mins

10min total

Hoby

(a) change manually the SRF and output name in plotSRF.py (Viktor is currently working in improving this part)

Execute ‘plotSRF.py’ (Make sure to use python2)

(b) plot_srf.py file.srf

2. Velocity Model Domain   (ETHAN/ROBIN)

From rupture location to VM dev and domain

(a)  From location and Mw determine VM origin, 3D extent, and simulation duration

(b)  Plot VM domain extent

(c)  Generate VM with H=0.1km for this domain, then extract slices

1mins

20 mins (on multiple cores)

15 mins for slices

4 mins for H=0.2km VM creation (30 mins for H=0.1km)

20min total

Ethan, Robin,…

1. Tidy up params_vel.py (some params are redundant) and include corners for plotting before mesh generation
   
2. Run gen_corrds during VM and place all model params alongside VM files (Done)
   
3. Integrate the gen_cords into the first step to enable the domain to be plotted on a map and checked / changed if necessary (Done)
   
4. Split the VM generation from VM interrogation / slice generation to allow the VM to be copied while slices are being generated
5. Will need a wall clock time estimation done before VM calculation (Done)
   
6. Merge  EXTENT_Z_SPACING, EXTENT_LATLON_SPACING and HH (Done)
   
7. Change names of to be consistent  ORIGIN_LAT, ORIGIN_LON,ORIGIN_ROT retain these names , (MODEL_LAT,_LON,_ROT ) (Done)
   
8. Write xy2ll into the gen_params.py script to write the domain corners into params_vel.py (Done)
   
9. Plot the vm domain box within gen_params.py
   
10. Include gen_cords.py within genDomain to keep coords and .p .s and .d binary files in together (Done Duplicate of 2)
    

 There was an error in the auto-VM generation on hypocentre which needed to be fixed, and then the code had to be obtained from github again. (Done)

(Sung) == items below have beem completed after Feb. drill

1. nx, ny, nz are automatically updated if user manually edits other values like (extent_x,_y,_zmax)

2. if extent_x, extent_y and extent_zmax are not divisible by hh, it automatically increases extent_* to the first divisible value

 3. it will automatically work out flo  (based on min_vs/(5*hh).  The initial value of flo is automated, but user can manually override it by editing params_vel.py  (More in item 4)

4.In params_vel,py,  extent_x, extent_y, and extent_zmax, sim_duration and flo have automatically generated initial value.  User can manually edit params_vel.py to override them, but user MUST run *validate_parameters.py*, which will check the integrity of  the parameter file, and adjust automated values such as nx, ny, nz and sufx.  This will overwrite params_vel.py  (also keeps a backup of old params_vel.py) and show before and after changes side-by-side.

(Ethan)

Remove topography from preliminary plot to speed up investigateDomain.py

Change default CPTs to extend their range

From non-uniform station grid, get statgrid points

(a)  Locations of statgrid points

(b)  Locations of GeoNet strong motion stations

(c)  From NZ Vs30 model get Vs30 values at all points

(ignore for now)

1-2 hours!!

10mins

~2hours

Ahsan, Chris

[need .ll file for all of NZ (Ahsan)] (Done)

run statcords (to get station locations corresponding to actual VM domain]

(a) Plot VM domain extent, stations (statgrid and GeoNet),

(b) color based on Vs30 values

2mins

(a) Viktor

(b) Not done

Only done as part of plotting PGA, PGV, PSA, PSA RATIOS

Where to get vs30 values?

3. Observed strong motions  (SUNG)

Obtain observations from GeoNet

(a)  Get recorded motion from GeoNet ftp

(b)  Process and reformat using defaults

(c)  Produce plots of acc/vel time series, response spectra

(d) Produce IM data files for stations on map plots in GMT

(e) Produce Vs30 input files for stations

 (a) to (d) took 7.5 mins.

(e) took 40 mins

Ahsan

Tasks (a) to (d) are automated but (e) is still manual and took the longest time. In (e) the computational time itself only took about 2 mins, it was the formatting that took the most time.

2017-02-23_133554 Mw4.7 simulated as a point source. Since  the earthquake occurred within the FVM 

domain, no new velocity model was generated. The three key steps (in absence of observed data) were
(1) generating the point source (srf file)
(2) Running the simulation
(3) Producing time slice maps/movie, PGV and MMI maps.

The main takeaway lessons were that
(1) Generating a velocity model may not be necessary for each simulation drill
(2) A need for quickly deciding whether a new velocity model should be created. Show existing domain on Map!?
(3) May need to generate file names and directory structure at the time of source generation for efficiency.

Plot IM values at GeoNet stations on map (for various IMs)

3 mins

Viktor

Viktor was away, and I (Ahsan) performed this task. It was some what unclear how the GMT plotting scripts (behind the scenes) got its input, e.g the domain boundary. The code ran without problems. There was a need to display the color bar on a log scale, which I was not able to do through the GMT scripts but had to rewrite the natural log of the data instead of the plain data. It was unclear what the log=True option achieves in the python wrapper to the GMT scripts.

Plot observed vel time series on map

5mins

Viktor

skipped, need to finish script that does this

Plot IM values as a function of Rrup and compare with empirical ground motion model (various IMs)

Ahsan

4. Perform GM Simulation   (SUNG)

Submit GM Simulation

(a)  prepare input params.py, pointing to SRF, VM, and stations of interest etc. (gen_cords.sh and statlist2gp.py need to be run)   File transfer and install

(b)  LF sim (queue time + run time)

(c)  Once LF sim complete, run HF and BB sim

(a)5mins

(b)1:40~3:30 hours (queue+run)

(c)13 min

Sung

(1) We need a standardized event name stored somewhere.

(2) File transfer to Fitzroy could be more simplified. Occasionally I had no right permission to access the files generated by other team members.

==> Solution: Each module (VM, Source, Station) can have an "install" script that is executed by the person-in-charge that transfers the necessary files to the appropriate location (derived from the standardised event name)

(3) Many codes from various repositories are increasingly becoming inter-dependent. We need to make a shared library soon.

(4) Wall clock limit estimation is reasonably accurate, with Fitzroy being prone to fluctuate occasionally. (IO suspected). DB needs some tweak to mark outliers.

5. Post-processing   (VIKTOR)

LF Time slice animation

(a) tar, download data from Fitzroy to local

(b)  From TS produce PGV and MMI plot

(c)  Produce TS animation

9 min

Viktor

1) The animation is working fine for single event. However, if there are 2 SRF files in the run, you need to modify the parameter file such that it will generate the animation of the specific event that you want

2) we might want to generate the animation for the multiple events as well??

3) It would be great if we could include "*modelparams*" in the tar file from Fitzroy, which is needed for (5f) plotting

Script preparation and test

At SMS locations compare sim vs. obs

(a) compare waveforms

(b) Compare IMs as a function of Rrup (multiple IMs; sim, obs, empirial)

(c) Compare sim, obs, empirical response spectra at individual locations

(d) Plot overall bias as a function of vibration period (both sim and empirical)

(e) Preparing input for (f)

(f) Plot bias(residual) at specific locations on a map

10mn

(a)–(d) 20 mins

(e) 4 mins

(f) 5 mins

Hoby

(f) requires modelparams file/Srf; need to make sure we have them copied in hypocentrer

1) Include ground motion duration in the PP plotting

2) Add the empirical ground motion duration

2) get plotSRF improved

2days

2days

2-5 days

 (1) automate process to produce Vs30 files and their formatting.

(2) Produce plots that need Rrup within the existing geoNet real-time processing scripts as opposed to separately running

groundMotionStationsAnalysis code separately.

(3) Include header information for IM data files to be plotted on map using GMT. Currently this is done manually.

(4) Incorporate change to write custom header to .000 .090 .ver GP format files. Issue brought up by Chris McGann

(1) 2 hours

 (2) 2 hours

(3) 2 hours

(4) 2hours

1. Create shared parameter files that can be imported into other steps in the workflow (Saves need for configuring params.py and get_cords.sh)
2. Automated file transfer to Fitzroy
3. Automated installation of a job
   
   

1. 2 days
2. 2 days
3. 2 days

1. Finish script to plot observed timeseries on map
2. Run plots automatically when data has been retrieved
3. Determine format and implement for plotting stations on map with PGV, PGA, PSA data etc...
   and figure out easiest way to plot PGV stations on overlay

2.5 days
1 - 2 days
2 days 

1. Non-uniform grid should probably run after the VM
2. Include non-uniform grid generation to the automatized workflow
3. Parameters for the grid should be read from the VM file
4. Make sure the format of the output is usable

Task

Expected time reqd.

People

Notes

1. Source          (HOBY)

GeoNet Mw email to SRF

(a)  from Mw tensor get Rup parameters

(b)  From Rup parameters use createSRF.py to make rupture SRF

(c)  Execution of createSRF.py

1minute

5mins

10sec

6min total

Hoby

a) Manually extract location, dip,strike, rake from Mw tensor (Ristau) into setSrfParams.py

b) Change TYPE=2 (point source to finite fault)

c) Run "createSRF.py"

Plotting of SRF

(a)  SRF plane(s) standard plotting

(b)  SRF on map to verify location (to come after domain set in step 2)

3mins

5mins

2mins

10min total

Hoby

(a) change manually the SRF and output name in plotSRF.py (need to be improved)

Execute ‘plotSRF.py’

(b) plot_srf.py file.srf

2. Velocity Model Domain   (ETHAN/ROBIN)

From rupture location to VM dev and domain

(a)  From location and Mw determine VM origin, 3D extent, and simulation duration

(b)  Plot VM domain extent

(c)  Generate VM with H=0.1km for this domain, then extract slices

1mins

20 mins (on multiple cores)

15 mins for slices

4 mins for H=0.2km VM creation (30 mins for H=0.1km)

20min total

Ethan, Robin,…

1. Tidy up params_vel.py (some params are redundant) and include corners for plotting before mesh generation
2. Run gen_corrds during VM and place all model params alongside VM files
3. Integrate the gen_cords into the first step to enable the domain to be plotted on a map and checked / changed if necessary
4. Split the VM generation from VM interrogation / slice generation to allow the VM to be copied while slices are being generated
5. Will need a wall clock time estimation done before VM calculation
6. Merge  EXTENT_Z_SPACING, EXTENT_LATLON_SPACING and HH
7. Change names of to be consistent  ORIGIN_LAT, ORIGIN_LON,ORIGIN_ROT retain these names , (MODEL_LAT,_LON,_ROT )
8. Write xy2ll into the gen_params.py script to write the domain corners into params_vel.py
9. Plot the vm domain box within gen_params.py
10. Include gen_cords.py within genDomain to keep coords and .p .s and .d binary files in together

1. There was an error in the auto-VM generation on hypocentre which needed to be fixed, and then the code had to be obtained from github again.

From non-uniform station grid, get statgrid points

(a)  Locations of statgrid points

(b)  Locations of GeoNet strong motion stations

(c)  From NZ Vs30 model get Vs30 values at all points

(ignore for now)

1-2 hours!!

10mins

~2hours

Ahsan, Chris

[need .ll file for all of NZ (Ahsan)] (Done)

run statcords (to get station locations corresponding to actual VM domain]

(a) Plot VM domain extent, stations (statgrid and GeoNet),

(b) color based on Vs30 values

2mins

(a) Viktor

(b) Not done

Only done as part of plotting PGA, PGV, PSA, PSA RATIOS

Where to get vs30 values?

3. Observed strong motions  (SUNG)

Obtain observations from GeoNet

(a)  Get recorded motion from GeoNet ftp

(b)  Process and reformat using defaults

(c)  Produce plots of acc/vel time series, response spectra

 45 mins total for (a)

 and (b)

60 mins for generating Vs30 values. (one-off task)

5 mins for (c)

Ahsan

Majority of time spent trouble shooting. See comment below

GeoNet people make errors which can cause errors and breakdown our code. The data location has files that should have been removed. Take a look at ftp://ftp.geonet.org.nz/strong/processed/Proc/2013/08_Aug/2013-08-16_023105/Vol1/data . Files ending with _ should not be there and nor should V2A and V3A data be there. You will also see three stations named like Q13924J03.V1A that don't follow the naming convention used for all the other data files. This broke down my code. For the next drill, I will change the code to catch these as exceptions and continue smoothly.

Plot IM values at GeoNet stations on map (for various IMs)

3 mins

Ahsan

Calculations of PGVs, PGAs and pSA for set of chosen periods took about 3 mins, these were then passed on to Viktor who plotted them.

Plot observed vel time series on map

5mins

Viktor

skipped, need to finish script that does this

Plot IM values as a function of Rrup and compare with empirical ground motion model (various IMs)

Ahsan

4. Perform GM Simulation   (SUNG)

Submit GM Simulation

(a)  prepare input params.py, pointing to SRF, VM, and stations of interest etc. (gen_cords.sh and statlist2gp.py need to be run)   File transfer and install

(b)  LF sim (queue time + run time)

(c)  Once LF sim complete, run HF and BB sim

(a)5mins

(b)5 hours (queue+run)

(c)13 min

Sung

gen_ts was broken (fixed)

Will be nice to have auto file transfer (in 3 separate streams esp. for VM model)

Will need to separate HF from BB and run it alongside LF.

5. Post-processing   (VIKTOR)

LF Time slice animation

(a) tar, download data from Fitzroy to local

(b)  From TS produce PGV and MMI plot

(c)  Produce TS animation

10mins

25mins

25min

Viktor

needed to fix up scripts to work with new parameter style

mainly needed to fix script to use new parameter style, should run automatically after step A
run time only ~3 minutes 

need to automate a little more, run time ~5 mins

Script preparation and test

At SMS locations compare sim vs. obs

(a) compare waveforms

(b) Compare IMs as a function of Rrup (multiple IMs; sim, obs, empirial)

(c) Compare sim, obs, empirical response spectra at individual locations

(d) Plot overall bias as a function of vibration period (both sim and empirical)

(e) Preparing input for (f)

(f) Plot bias(residual) at specific locations on a map

10mn

(a)–(d) 30 mins

(e) 4 mins

(f) 5 mins

Hoby

(f)Viktor

1. Testing the Script using only the observation and remove irrelevant stations
2. Make sure to use fd_*.ll station lists instead of geonet*.ll
3. Need to add Ground motion duration
   
   

default parameters used, runs in 2 mins but needed to fix a few things up

1) Include ground motion duration in the PP plotting

2) Add the empirical ground motion duration

2) get plotSRF improved

2days

2days

2-5 days

(1) 2 hours

 (2) 5 hours

(3) 2 hours

1. Create shared parameter files that can be imported into other steps in the workflow (Saves need for configuring params.py and get_cords.sh)
2. Automated file transfer to Fitzroy
3. Automated installation of a job
   
   

1. 2 days
2. 2 days
3. 2 days

1. Finish script to plot observed timeseries on map
2. Run plots automatically when data has been retrieved
3. Determine format and implement for plotting stations on map with PGV, PGA, PSA data etc...
   and figure out easiest way to plot PGV stations on overlay

2.5 days
1 - 2 days
2 days 

1. Non-uniform grid should probably run after the VM
2. Include non-uniform grid generation to the automatized workflow
3. Parameters for the grid should be read from the VM file
4. Make sure the format of the output is usable

Task

Expected time reqd.

People

Notes

1. Source          (HOBY)

GeoNet Mw email to SRF

(a)  from Mw tensor get Rup parameters

(b)  From Rup parameters use createSRF.py to make rupture SRF

(c)  Execution of createSRF.py

1minute

5mins

10sec

6min total

Hoby

a) Manually extract location, dip,strike, rake from Mw tensor (Ristau) into setSrfParams.py

b) Change TYPE=2 (point source to finite fault)

c) Run "createSRF.py"

Plotting of SRF

(a)  SRF plane(s) standard plotting

(b)  SRF on map to verify location (to come after domain set in step 2)

3mins

5mins

2mins

10min total

(a) Hoby

(b) Viktor

Execute ‘plotSRF.py’ (select title for top of script)

(a) Hoby : Just needed to change SRF name 

(b) Viktor: make 2 plots on 1 image, 1 zoomed in with slip visible, 2 whole of NZ showing outline/hypocentre

Finished automating. only SRF needed, no reliance on params.py etc...

2. Velocity Model Domain   (ETHAN/ROBIN)

From rupture location to VM dev and domain

(a)  From location and Mw determine VM origin, 3D extent, and simulation duration

(b)  Plot VM domain extent

(c)  Generate VM with H=0.1km for this domain

2mins

1mins

15 mins (on multiple cores)

20min total

Ethan, Robin,…

1) Origin lat/lon/depth written into 'generate_parameters.py' python script

2) Run 'generate_parameters.py', output is a text file which is read by the next step

3) Run 'genDomain.py' this generates the velocity model, extracts slices. Stores the results in Rapid_Model

Need to implement duration calculations

Need to streamline Gen_Coords to sync with model parameters

Run gen_cords.csh and plot

Create .p/s/d files

From non-uniform station grid, get statgrid points

(a)  Locations of statgrid points

(b)  Locations of GeoNet strong motion stations

(c)  From NZ Vs30 model get Vs30 values at all points

(ignore for now)

1-2 hours!!

10mins

~2hours

Ahsan, Chris

[need .ll file for all of NZ (Ahsan)] (Done)

run statcords (to get station locations corresponding to actual VM domain]

(a) Plot VM domain extent, stations (statgrid and GeoNet),

(b) color based on Vs30 values

2mins

(a) Viktor

(b) Not done

Where to get vs30 values?

3. Observed strong motions  (SUNG)

Obtain observations from GeoNet

(a)  Get recorded motion from GeoNet ftp

(b)  Process and reformat using defaults

(c)  Produce plots of acc/vel time series, response spectra

Ahsan

Plot IM values at GeoNet stations on map (for various IMs)

1 hour (GMT)

Ahsan

updating e3d.par domain info

Plot observed vel time series on map

5mins

Viktor

need to automate location of stations involved, zoom levels etc... code exists, needs to be brought together / cleaned up

Plot IM values as a function of Rrup and compare with empirical ground motion model (various IMs)

20min

1hour 30mins

Ahsan

Richard's code runs for 3 mins. Had to go through a number of iterations to get nice looking plots.

Has a good idea to reduce the time significantly. Knows what parameters to change.

Not easy to automate.

4. Perform GM Simulation   (SUNG)

Submit GM Simulation

(a)  prepare input params.py, pointing to SRF, VM, and stations of interest etc. (gen_cords.sh and statlist2gp.py need to be run)

(b)  LF sim (queue time + run time)

(c)  Once LF sim complete, run HF and BB sim

20mins

512 cores (8node) [wait+run 2hours]

30mins

4hours

Sung

1. Hoby uploaded SRF to Fitzroy 
2. Sung uploaded VelModel and model_params (output of gen_cords.sh) to Fitzroy
3. Sung created a recipe for the drill with params.py having all the parameters set, and other files copied from 2010Sep sim.
4. Modification to install.sh was not necessary, but we will eventually need to.
5. Almost forgot to produce .startcords from .ll. For HF, it is advised to use the FD version of .ll (ie. output of statlist2gp.py) in params.py

Good opportunity for automation. (a) can be done in a sec.

5. Post-processing   (VIKTOR)

LF Time slice animation

(a) tar, download data from Fitzroy to local

(a)  Produce TS animation

(b)  From TS produce PGV plot

(c)  Produce MMI plot from PGV

30mins

30mins

5min

5min

Viktor

(can be streamlined, at later date)

vm domain used by default, determine if regional sites should be added (crowding), padding around vm based on rot, domain if rotation small

mostly automated TS animation, some parameters needed such as colour range

single script does both PGV, MMI - need to finish automating colour range for PGV

Script preparation

At SMS locations compare sim vs. obs

(a) compare waveforms

(b) Compare IMs as a function of Rrup (multiple IMs; sim, obs, empirial)

(c) Compare sim, obs, empirical response spectra at individual locations

(d) Plot overall bias as a function of vibration period (both sim and empirical)

(e) Preparing input for (f)

(f) Plot bias(residual) at specific locations on a map

10mn

(a)–(c) 30 mins

(d) 20mn

(e) 10 mins

1. Make sure to use fd_*.ll station lists instead of geonet*.ll
2. There were some errors during the bias plotting, so I need to remove that part of the code to plot the IMs
   
   

make a script that plots just this and loops over period values - done self contained input file used and model_params

1) Update all_geoNet_stations.ll to include SMS that closed in 2010. This will add more stations to the list and we won't have a problem when dealing
with historical events. Kevin would need to update the Vs30 stations list accordingly.

2) Some python scripts (code that downloads data, code that processes data) that run one after another can and will be automated.

3)Coordinates with Viktor for plotting IMs on map. Currently I and Viktor separately gather velocity model domain data etc,for GMT plotting,
which is a manual task. Supply Viktor IMs data file for him to plot.

1. Create shared parameter files that can be imported into other steps in the workflow (Saves need for configuring params.py and get_cords.sh)
2. Automated file transfer to Fitzroy
3. Automated installation of a job
   
   

1. 2 days
2. 2 days
3. 2 days

1. Fix plotting script for SRF to show fault location clearly, Create plotting script that takes PGV and MMI, create plotting script for station resuduals.
2. Automatically calculate default parameters where possible, allow scripts to be run where paths / needed inputs can be passed to them.
3. connect scripts where input from one should automatically run another where possible eg: create PGV, MMI data and instantly plot it

1 - 2 days
1 - 2 days
1 - 2 days