The purpose of this document is to describe the standards and procedures to follow during the software testing phases of the QuakeCoRE ucgmsim git repositories.
1. Scope
2. Test priorities
(1) Prioritize GitHub repositories for testing
Identify in-use/abandoned repositories
Identify type of individual application (script/library)
Different type of application has different testing strategy
Count usage frequency of individual applications
Summary: https://docs.google.com/document/d/1ntFrNsMBo1G5bqTCnJIYVsTiuLApUbrf4vNyYMdI298/edit
3.Test plan
(1) Test method:
Black-box: testing without knowledge of the code, compare if the test output is the same as the sample output; often used in testing scripts
White-box: testing with knowledge of the code, often used in unit test for library.
(2) Test framework:
- Pytest
(3) Test design:
Identify type of the application: Script: tested as a whole; Library: unit testing
Identify input and output: what is used to test; what is produced from the test; contact the developer if confused
Formulate tailored testing plan for individual application if needed: eg. design output folder structures, writing config/readme files
Select benchmark/correct sample output
Write testing script for the application
Execute the test
Compare test output with sample output
(4) Test execution
(5) Test report
Examples
(1) How to identify the type of application and corresponding test method
(2) How to design the structure of a testing folder
(3)How to write a test for Script file
(4) How to write a test for Library file
step 1. Identify a function to test: srf_dt
step 2. Identify input file paths for testing
step 3. Set up functionalities to create/remove test output folder which is handled by set_up and tear_down module (as shown in the test example for script in (3))
step 4. Write unit test called 'test_dt' for function 'srf_dt'. Use the builtin pytest.mark.parametrize decorator to enable parametrization of arguments for a test function. The @parametrize
decorator defines two different (test_dt,expected_dt)
tuples so that the function 'test_dt'
will run twice using them in turn. The expected value eg '2.50000e-02' should be manually picked but not by open(file, 'r') from the sample output file.
step 5. Run Pytest
4. Test Responsibility
(1) Scripts tests are conducted by a person that is not involved in the development of the script.
(2) Library tests/Unit tests are created and executed by the developer of the unit.
All Scripts and Library files must pass the tests before being accepted to the usgmsim Git repository.