Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Observed ground motions
    1. Ahsan

      NOTE: Installation is required only once, unless there is a need to update. geoNet
      package is self contained and should work on your pcs and laptops too.

      0) INSTALL geoNet.
      From 
      https://github.com/ucgmsim/Pre-processing/tree/master/geoNet
      clone geoNet and follow the installation instructions.

    2. Copy event_info.txt  found at geoNet/examples and modify. Then do the following:
      from geoNet import realtime
      realtime.run("event_info.txt")

      Alternatively performs the steps (1) to (5) outlined below.

      1) GET THE REQUIRED SCRIPTS.
      From geoNet/examples copy

      getData.py
      event_statsll.py
      processData.py
      plot_accvel.py
      plot_psa.py

      to your working directory

      2) GET THE DATA FOR THE EARTHQUAKE.

      In getData.py set BASE_URL to the location to the geoNet URL e.g
      BASE_URL="ftp://ftp.geonet.org.nz/strong/processed/Proc/2017/02_Feb/2017-02-01_102129/Vol1/data/"

      save and run.

      3) GET THE LIST OF STATIONS FOR THE EARTHQUAKE
      Run event_statsll.py which creates event_stats.ll and places it
      in your working directory.

      4)PROCESS THE GEONET VOL1 DATA TO GET NORTH, EAST AND VERTICAL COMPONENTS
      Run processData.py which filters and rotates the SMS data placing them
      in Vol1/data/accBB etc.
      Note: It is better to redirect the std output to a file like so
      python processData.py > processData_output

      5) CREATE PLOTS FOR OBSEREVED DATA
      Run plot_accvel.py and plot_psa.py

  2. Ground motion simulation computation (Sung)
    1. Upload Rupture model to /nesi/projects/nesi00213/RupModel/Realtime/MonYEAR where MonYear is like Dec2016, Jan2017. Under this directory, you need Srf and Stoch.
    2. Upload Velocity model to /nesi/projects/nesi00213/VelocityModels/. Currently we have Canterbury or South_Island. Unless it is a Canterbury specific model, place it under South_Island. We will be doing some clean up later.
      Your velocity model should include rho3dfile.d, vp3dfile.p, vs3dfile.s, params_vel.py and outputs from gen_cords (gridfile, gridout, model_bounds, model_coords, model_params).
    3. Upload station files to /nesi/projects/nesi00213/StationInfo. This should include geonet_stations_xxx.ll, .vs30 and .vs30ref.
    4. Go to /nesi/projects/nesi00213/RunFolder and execute ./install.sh. and follow the instruction.
    5. During install.sh, it will advise you expected wallclock time. Enter a sensible wall clock time.
    6. Follow the remaining instruction.

  3. simulation post-processing
    1. Viktor/Hoby
    2. General Plotting Instructions:
      run on hypocentre, not fitzroy
      make sure the scripts are in PATH (hypocentre):
      export PATH=$PATH:/nesi/projects/nesi00213/gm_sim_workflow/devel
    3. Plotting Station based data such as IMs, Vs30, Observed PGA, pSA:
      plot_stations.py datafile.ll
      Run on hypocentre, datafile has 6 line header as described at the top of plot_stations.py
    4. Plotting Timeslice based data (includes PGV, MMI, timeslice animation) on hypocentre:
      plot_transfer.py auto /nesi/projects/nesi00213/RunFolder/baes/RT2017Jan17_VMSI_20170117_drill_200m-h0p200_EMODv3p0p4_170116 .
      If you have a different username on fitzroy then you will need to add a 4th parameter, your username on fitzroy 
      will copy the remote fitzroy folder to the current directory (.) and start plotting using defaults
      run plot_ts_sum.py to re-run PGV and MMI plots
      run plot_ts.py to re-run timeslice image creation 





Key lessons from past drills:



  1. March 2017 drill (2014 Mw6.3 Eketahuna Earthquake)

 

Task

Expected time reqd.

People

Notes

1. Source          (HOBY)

GeoNet Mw email to SRF

(a)  from Mw tensor get Rup parameters

(b)  From Rup parameters use createSRF.py to make rupture SRF and to plot

(c)  Execution of createSRF.py

 

1minute

 

 

25sec

2min total

 

Hoby/Viktor

a) Manually extract location, dip,strike, rake from Mw tensor (Ristau) into setSrfParams.py

b) Change TYPE=2 (point source to finite fault)

Use either version "3.3" or "5.3.2a"

c) Run "createSRF.py"

Please note that the execution time of createSRF.py may change based on the event size

    

 

 

 

 

 

 

 

2. Velocity Model Domain   (ETHAN/ROBIN)

From rupture location to VM dev and domain

(a)  From location and Mw determine VM origin, 3D extent, and simulation duration

(b)  Plot VM domain extent

(c)  Generate VM with H=0.1km for this domain, then extract slices

1mins

20 mins (on multiple cores)

15 mins for slices

4 mins for H=0.2km VM creation (30 mins for H=0.1km)

20min total



March Drill

Setup code - 4 mins

generateDomain initial (inc. getting CMT data) - 2+2=4 min

creating VM (inc. manual adjustment) - 10+1 = 11

genDomain - 6+6+6 = 18

 

Total ~37-40 minutes

 

Ethan, Robin,…

  1. Tidy up params_vel.py (some params are redundant) and include corners for plotting before mesh generation
  2. Run gen_corrds during VM and place all model params alongside VM files (Done)
  3. Integrate the gen_cords into the first step to enable the domain to be plotted on a map and checked / changed if necessary (Done)
  4. Split the VM generation from VM interrogation / slice generation to allow the VM to be copied while slices are being generated
  5. Will need a wall clock time estimation done before VM calculation (Done)
  6. Merge  EXTENT_Z_SPACING, EXTENT_LATLON_SPACING and HH (Done)
  7. Change names of to be consistent  ORIGIN_LAT, ORIGIN_LON,ORIGIN_ROT retain these names , (MODEL_LAT,_LON,_ROT ) (Done)
  8. Write xy2ll into the gen_params.py script to write the domain corners into params_vel.py (Done)
  9. Plot the vm domain box within gen_params.py
  10. Include gen_cords.py within genDomain to keep coords and .p .s and .d binary files in together (Done Duplicate of 2)

 There was an error in the auto-VM generation on hypocentre which needed to be fixed, and then the code had to be obtained from github again. (Done)

(Sung) == items below have beem completed after Feb. drill

1.nx,ny,nzare automatically updated if user manually edits other values like (extent_x,_y,_zmax)

2. if extent_x, extent_y and extent_zmax are not divisible byhh, it automatically increases extent_* to the first divisible value

 3. it will automatically work outflo  (based on min_vs/(5*hh).  The initial value offlo is automated, but user can manually override it by editing params_vel.py  (More in item 4)

4.In params_vel,py,  extent_x, extent_y, and extent_zmax, sim_duration andflohave automatically generated initial value.  User can manually edit params_vel.py to override them, but user MUST run *validate_parameters.py*, which will check the integrity of  the parameter file, and adjust automated values such asnx,ny,nzandsufx.  This will overwrite params_vel.py  (also keeps a backup of old params_vel.py) and show before and after changes side-by-side.

(Ethan)

Remove topography from preliminary plot to speed up investigateDomain.py

Change default CPTs to extend their range

 

March Drill

(Robin)

Issues encountered/comments

  1. nx and ny were incorrectly doubled (already fixed).
  2. Development oversights leaving out the writing of params, and importing of shared.

Overall if everything went smoothly and no manual adjustment was required, this would total something around 15 minutes. (note this is also dependent on the VM domain size/source magnitude.

 

From non-uniform station grid, get statgrid points

(a)  Locations of statgrid points

(b)  Locations of GeoNet strong motion stations

(c)  From NZ Vs30 model get Vs30 values at all points

 

(ignore for now)

1-2 hours!!

10mins

~2hours

 

 

Ahsan, Chris

 

 

[need .ll file for all of NZ (Ahsan)] (Done)

run statcords (to get station locations corresponding to actual VM domain]

(a) Plot VM domain extent, stations (statgrid and GeoNet),

(b) color based on Vs30 values

2mins

(a) Viktor

(b) Not done

Only done as part of plotting PGA, PGV, PSA, PSA RATIOS

Where to get vs30 values?

3. Observed strong motions  (SUNG)

Obtain observations from GeoNet

(a)  Get recorded motion from GeoNet ftp

(b)  Process and reformat using defaults

(c)  Produce plots of acc/vel time series, response spectra

(d) Produce IM data files for stations on map plots in GMT


(e) Produce Vs30 input files for stations

 (a) to (d) took 7.5 mins.


(e) took 40 mins

Ahsan

  

Tasks (a) to (d) are automated but (e) is still manual and took the longest time. In (e) the computational time itself only took about 2 mins, it was the formatting that took the most time.

2017-02-23_133554 Mw4.7 simulated as a point source. Since  the earthquake occurred within the FVM 

domain, no new velocity model was generated. The three key steps (in absence of observed data) were
(1) generating the point source (srf file)
(2) Running the simulation
(3) Producing time slice maps/movie, PGV and MMI maps.

The main takeaway lessons were that
(1) Generating a velocity model may not be necessary for each simulation drill
(2) A need for quickly deciding whether a new velocity model should be created. Show existing domain on Map!?
(3) May need to generate file names and directory structure at the time of source generation for efficiency.

Plot IM values at GeoNet stations on map (for various IMs)

3 mins

Viktor

Viktor was away, and I (Ahsan) performed this task. It was some what unclear how the GMT plotting scripts (behind the scenes) got its input, e.g the domain boundary. The code ran without problems. There was a need to display the color bar on a log scale, which I was not able to do through the GMT scripts but had to rewrite the natural log of the data instead of the plain data. It was unclear what the log=True option achieves in the python wrapper to the GMT scripts.

Plot observed vel time series on map

5mins

Viktor

skipped, need to finish script that does this

Plot IM values as a function of Rrup and compare with empirical ground motion model (various IMs)

10 mins

Ahsan

This was still performed with groundMotionSationsAnalysis code by passing Obs data twice (as Obs and Sim data). This task will be performed along side task (c).

4. Perform GM Simulation   (SUNG)

Submit GM Simulation

(a)  prepare input params.py, pointing to SRF, VM, and stations of interest etc. (gen_cords.sh and statlist2gp.py need to be run)   File transfer and install

(b)  LF sim (queue time + run time)

(c)  Once LF sim complete, run HF and BB sim

 

(a)5mins

(b)1:40~3:30 hours (queue+run)

(c)13 min

 

Sung

(1) We need a standardized event name stored somewhere.

(2) File transfer to Fitzroy could be more simplified. Occasionally I had no right permission to access the files generated by other team members.

==> Solution: Each module (VM, Source, Station) can have an "install" script that is executed by the person-in-charge that transfers the necessary files to the appropriate location (derived from the standardised event name)

(3) Many codes from various repositories are increasingly becominginter-dependent. We need to make a shared library soon.

(4) Wall clock limit estimation is reasonably accurate, with Fitzroy being prone to fluctuate occasionally. (IO suspected). DB needs some tweak to mark outliers.

 

5. Post-processing   (VIKTOR)

LF Time slice animation

(a) tar, download data from Fitzroy to local

(b)  From TS produce PGV and MMI plot

(c)  Produce TS animation

9 min

 

 

Viktor

1) The animation is working fine for single event. However, if there are 2 SRF files in the run, you need to modify the parameter file such that it will generate the animation of the specific event that you want

2) we might want to generate the animation for the multiple events as well??

3) It would be great if we could include "*modelparams*" in the tar file from Fitzroy, which is needed for (5f) plotting

 

Script preparation and test

At SMS locations compare sim vs. obs

(a) compare waveforms

(b) Compare IMs as a function of Rrup (multiple IMs; sim, obs, empirial)

(c) Compare sim, obs, empirical response spectra at individual locations

(d) Plot overall bias as a function of vibration period (both sim and empirical)

(e) Preparing input for (f)

(f) Plot bias(residual) at specific locations on a map

10mn

(a)–(d) 20 mins

 

 

 

 

 

 

 

 

(e) 4 mins

(f) 5 mins

Hoby

 

 

 

 

 

 

 

 

 

 

 

 

I have still used Richard code because Daniel/Jason mentioned that they sill have some issue in the std of bias between Obs and empiricalGMM. Once the PP is done there will be significant save in time in terms of file preparation because I will use the same config. as Ahsan

 

 

 

 

 




 

 

(f) requires modelparamsfile/Srf; need to make sure we have them copied in hypocentrer

 

 

 

  1. What to do next

 TasksTime estimation
Hoby

1) Srf plot location need to be modified

2)

2)

2days

2days

2-5 days

Ethan
1) Streamline the VM generation with gen_coords.csh
2) Write scrip to stitch together multiple PDFs in PDFtk
3) VM domain baseplot to be passed to plot SRF by others
2days (with other minor tasks)
Ahsan

 (1) automate process to produce Vs30 files and their formatting.

(2) Produce plots that needRrupwithin the existing geoNet real-time processing scripts as opposed to separately running

groundMotionStationsAnalysis code separately.

(3) Include header information for IM data files to be plotted on map using GMT. Currently this is done manually.

(4) Incorporate change to write custom header to .000 .090 .ver GP format files. Issue brought up by Chris McGann

(1) 2 hours

 (2) 2 hours

(3) 2 hours

(4) 2hours

SungThe following 3 items are easy to automated, and where we can easily make mistakes.
  1. Create shared parameter files that can be imported into other steps in the workflow (Saves need for configuring params.py and get_cords.sh)
  2. Automated file transfer to Fitzroy
  3. Automated installation of a job

  1. 2 days
  2. 2 days
  3. 2 days
Viktor
  1. Finish script to plot observed timeseries on map
  2. Run plots automatically when data has been retrieved
  3. Determine format and implement for plotting stations on map with PGV, PGA, PSA data etc...
    and figure out easiest way to plot PGV stations on overlay

2.5 days
1 - 2 days
2 days 

Daniel
  1. Non-uniform grid should probably run after the VM
  2. Include non-uniform grid generation to the automatized workflow
  3. Parameters for the grid should be read from the VM file
  4. Make sure the format of the output is usable
 

 

  1. Feb 2017 drill (2007 Gisborne Earthquake)

...