Current Usage (as of 18 Mar 2021)
Total | Used | Avail | |
---|---|---|---|
Isilon | 22T | 19T | 3.7T |
Dropbox | 3,675 TB | 85.96 TB (by all UC staff) | 3,589.04 TB |
Isilon for data that needs more frequent access.
Cybershake
Cybershake 20+ Inputs Table
We will keep the latest CS data on Isilon
What is this data? | Source | VM (.cfgs) / vm_params.yaml | BB | IM | Note | |
---|---|---|---|---|---|---|
22p12 | Dropbox | Same as left | Same as left | Same as left | All good | |
22p11 | 100m Wellington region | Dropbox | Same as left | Same as left | Same as left /isilon/CS_IMs/v22p11 | Source median missing |
400m CS rerun of 20p4/5 (Abandoned) | ||||||
22p4 | 100m last batch South Island | Dropbox | Same as left | Same as left | Same as left /isilon/CS_IMs/v22p4/NeSI (faults not covered in KISTI runs) | All good |
22p2 | 100m South Island batch | Dropbox | Same as left | Same as left | Same as left /isilon/CS_IMs/v22p2 | All good except TeAnau IM (incomplete) |
21p6 | 100m South Island Continuation (contains a few different batches in the one version) | Dropbox | Same as left | Same as left | Same as left /isilon/CS_IMs/v21p6 | Many missing items. See Stocktake.csv. |
21p1 | First batch of 100m South Island | Dropbox | Same as left | Same as left | Same as left /isilon/CS_IMs/v21p1_im | All good |
20p9 | Dropbox | Same as left | Same as left | Same as left /isilon/CS_ims/ | Missing all vm_params.yaml Stocktake.csv stating RELs csv/info are missing, but they are good. (TODO) | |
20p6(=20p4_200) | Same as 20p4 /isilon/cybershake/v20p4/Sources | Isilon: /isilon/cybershake/v20p4/200m_vmcf
| Dropbox (8.0Tb) /isilon/cybershake/v20p6/ | /isilon/CS_ims/ | ||
20p5 | RELXX.csv/.info are missing. | Dropbox | /isilon/CS_ims/ | |||
20p4 | Isilon: /isilon/cybershake/v20p4/Sources (full) | /isilon/cybershake/v20p4 | Dropbox (11Tb) | /isilon/CS_ims/ | ||
19p5 | Full Srf/Stoch v19p5_sources – Dropbox (766Gb) | Dropbox (vm_params) | Dropbox (11Tb) | /isilon/CS_ims/ | ||
18p6 | Full Srf/Stoch v18p6_sources – Dropbox (846Gb) | Nearline /nesi/nearline/nesi00213/RunFolder/to_nearline_18p6/ v18p6_waveform_backup.tar (Can't retrieve from it) (43.67Tb) | /isilon/CS_ims/ | |||
18p5 | Isilon: /isilon/cybershake/v18p5 | |||||
18p4 | Isilon: /isilon/cybershake/v18p4 | |||||
17p9 | Isilon: /isilon/cybershake/v17p9 |
Note:
For a Cybershake version (eg. v20p9), we keep 3 items
- Sources: .csv and .info per fault and all realisations. All files packed into one tar file. v20p9_Sources.tar . (pre-v20p4, we kept all Srf and Stoch files)
- VMs: Only nzvm.cfg per fault. All files packed into one tar file. v20p9_VM_cfgs.tar
- BB: BB.bin files for all realisations packed into a tar file for each fault. (eg. AlpineF2K.tar)
An automated archive script can package all CS files and generate a Dropbox upload command that can be executed from NeSI login node (in "screen" session)
Dropbox has a maximum file size of 350Gb. The auto script will split the TAR files into multiple parts of 340Gb size. To join them (before extracting from TAR),
cat ${TARNAME}.parta* > ${TARNAME}.joined
tar xvf ${TARNAME}.joined
Empirical DB
Mostly on Isilon. Upload to Dropbox in progress
location | notes | |
---|---|---|
v20p4 | /isilon/seistech/empiricals/v20p4 (Dropbox) Nearline : /nesi/nearline/nesi00213/seistech/empiricals/v20p4 (to be deleted) | ds, flt in db format Files below are > 340Gb and split parts are uploaded to Dropbox Br_10_ACTIVE_SHALLOW_ds.db CY_14_ACTIVE_SHALLOW_ds.db CB_14_ACTIVE_SHALLOW_ds.db BSSA_14_ACTIVE_SHALLOW_ds.db ASK_14_ACTIVE_SHALLOW_ds.db |
v19p11 | /isilon/seistech/empiricals/19p11_13stations (Dropbox) /isilon/seistech/empiricals/19p11 (Dropbox) | |
v19p9 | /isilon/seistech/empiricals/19p9_complete (Dropbox) /isilon/seistech/empiricals/19p9_1655 (Dropbox) /isilon/seistech/empiricals/19p9 (Dropbox) | |
v18p6 | /isilon/seistech/empiricals/18p6_new (Dropbox) /isilon/seistech/empiricals/18p6 (Dropbox) Nearline /nesi/nearline/nesi00213/empiricals/v18p6 | Nearline versions are TAR per each IM type (eg. SA_0p5.tar, PGA.tar) |
v18p5 | Nearline /nesi/nearline/nesi00213/empiricals/v18p5 | Nearline versions are TAR per each IM type (eg. SA_0p5.tar, PGA.tar) |
Old Staff Data
Nearline : /nesi/nearline/nesi00213/RunFolder/to_nearline/RunFolder.tar. ← Hoby, Asahn
This page summarizes the archiving of some of the current CyberShake results.
v17p9
Aka the first batch of faults for the South Island. This version was archived on the RCC legacy storage in a very naive way. Every fault was saved in a single tar file. Thus every tar file has all the realizations, and for each realization we have a LF, HF and BB. This archived version needs to be cleaned to reduce the file size and have easier access to the waveforms.
v18p4
This is the batch that completes all the inland faults on the South Island. Note that both results need to be aggregated to have everything. To archive this version, we decided that we would zip the three components of the waveform for a given station into a single file. The way this was archived is very simple as it can be shown in the following block:
Code Block |
---|
wdl16@dev01-quakecore:/rcc/home/projects/quakecore/cybershake/v18p4$ tree -L 4
.
├── IMs
│ └── database.db
├── metadata.json
└── Runs
├── AhuririR
│ ├── AhuririR_HYP01-03_S1244
│ │ ├── Acc
│ │ ├── Vel
│ │ └── xyts
.....
├── Akatore
│ ├── Akatore_HYP01-04_S1244
│ │ ├── Acc
│ │ ├── Vel
│ │ └── xyts
..... |
This means that every fault has the realizations. For each realization we have an Acc, Vel and xyts directories, where we have the accelerations, velocities and e3d files respectively. Please note that only the Acc and Vel files are zipped. There is also a simple metadata.json file and the IM database for that particular version is also archived. The metadata file has the following contents:
Code Block |
---|
{
CyberShakeVersion : 'v18p4',
Path : '/rcc/home/projects/quakecore/cybershake/v18p4',
AccPath : '/rcc/home/projects/quakecore/cybershake/v18p4/Runs/*/*/Acc',
VelPath : '/rcc/home/projects/quakecore/cybershake/v18p4/Runs/*/*/Vel',
HasXYTS : 'yes',
XYTSPath ; '/rcc/home/projects/quakecore/cybershake/v18p4/Runs/*/*/xyts',
WaveFormSize : '423GB',
SizePlusXYTS : '475GB',
WaveFormCompression : 'zip'
IMDBSize : '8.2GB',
FaultNumber : 107,
Description : 'Rest of South Island faults. v17p9 + v18p4 are all the faults on the SI'
} |
This file was generated manually, but we could have some way to automate the generation after the archiving has finished.
v18p5
This version completes the NZ faults by adding the North Island faults. This is the next archiving to be done, and it will follow the same structure as v18p4.
TODO
- Add symlinks to v18p4 to give the impression that it is actually v17p9 + more faults. The same will be needed for v18p5.
Future
...
...