Significant portions of QuakeCoRE research require large-scale computationally-intensive numerical ground motion (GM) simulations.
The amount of data and complexity of computation make the large-scale simulation practically impossible to run on a researcher’s workstation. QuakeCoRE started collaboration with New Zealand eScience Infrastructure(NeSI), the national high performance computing (HPC) provider to gain the necessary computational capacity and execution speed.

Computational resources

QuakeCoRE ground motion simulation activities at the present time (as at Sept 2016) are largely run on the NeSI 'Fitzroy' Power6 cluster (located at NIWA Greta Point, Wellington), following the decommissioning of the previously utilised NeSI 'Foster' BlueGeneP (located at UC, Christchurch). QuakeCoRE’s GM simulation workflow was selected as one of the benchmark tests for NeSI’s new HPC system procurement process in 2017, and we intend to be early users of these new HPC platforms in mid-2017 to extend our computational capabilities.  The figure below illustrates our current and forecast core hour consumption (based on PowerPC core hours)

Ground motion simulation codes utilised

We are agnostic to specific ground motion simulation methodologies - our workflow is intended to be modular so that any methodology can be inserted into this 'plug and play' environment.  That said, for various current and historical reasons, we have principally utilised the following codebases from others:

• Rob Graves code-base:
  
  • The 'emod3d' wave propagation code for anelastic wave propagation (staggered grid finite difference code on a regular cartesian grid)
    
  • Graves and Pitarka rupture generator (various versions)
  • Pre- and post-processing csh scripts (many have been converted to python scripts, but several still remain)
  • This is the principal
    
• SPECFEM3D Cartesian (https://geodynamics.org/cig/software/specfem3d/)
  • Spectral element (high-order finite element) code for anelastic wave propagation.  Can handle irregular grids and surface topography.
    
  • This is the primary code used for full waveform tomography, because of its built-in adjoint tomography routines
    

Version-control

The codebase for ground motion simulation is maintained under Git version control and can be accessed from:

• emod3d current implementation: https://github.com/ucgmsim/EMOD3D
  
• pre-processing: https://github.com/ucgmsim/Pre-processing
  
• New Zealand velocity model versions: https://github.com/ucgmsim/Velocity-Model
  
• Post-processing: https://github.com/ucgmsim/groundMotionStationAnalysis

Additional repos exist, and the above list (last updated Oct 2016) is not intended to be exhaustive

Verification of code base

Adequate verification is obviously central to ensure a robust and extensible codebase.  Are individual functions are verified following best practice software protocols.  For this wiki page, we focus on high-level exercises for the verification of wave propagation codes.  Of particular attention are:

• Verification of workflow porting from Foster to Fitzroy : Porting of QuakeCoRE simulation workflow from IBM BlueGeneP ('Foster') to IBM Power6 ('Fitzroy') (emod3d-oriented wave propagation and relevant pre- and post-processing).
• Verification of SPECFEM3D computations on Fitzroy against existing emod3d computations