Validation:

Issue:  Estimation was somewhere between 500-700 (can't remember the exact number), based on nn_core_usage it used 1700 core hours. (Sprint 18)

Resaon: sreport does not show physical core hours, but instead always shows hyper threaded on maui (but physical on mahuika if billing flag is set...) (Sprint 19)

Done: 

  • Emailed NeSI regarding core hours logging (Sprint 19)
    • Confirmed that hyper threaded core hours are shown by sreport instead of whats actually billed (physical core hours) (Sprint 19)
  • Changed all estimation to physical core hours (Sprint 19)
  • After cleaning training data and re-training, estimation for validation v19p2p4 (627 sims)
    • LF estimation: Pretty good, est - 496, act 499
    • HF estimation: Room for improvement, 
      • Validation data and Cybershake data don't seem to match up very well (different core hours for similar parameters)
        Only for very short HF runs though (all HF validation sims are very small)
      • Est: 41, act: 18.12 
    • BB estimation: Same as HF
      • Est: 70, act: 31
    • Both HF and BB struggle with estimation for short simulations (as mentioned above I think this is because Cybershake short sims
      and validation don't seem to match up very well)


  • Had a brief look at the metadata of the run and this showed number of core hours used was 700-800 (Sprint 18)
  • What's going on here? – DONE
    • Determine if metadata is correct, need a reliable to get core hour used to confirm metadata values... – DONE
  • Robin is doing another run (when?), get estimation dataframe and compare to metadata after. – DONE


General:

Issue: Estimation is not as accurate as I'd expect it to be, especially EMOD3D

Todo:

  • Have a look at the training data, pretty sure there is some bad data in there – DONE
  • Look at specific samples were estimation has large errors – DONE
  • Look at effect of out of bound estimation -- TODO!
    • Potentially add model that is capable of extrapolation TODO!


  • No labels