Testing Standards for ucgmsim Git repositories

The purpose of this document is to describe the standards and procedures to follow during the software testing phases of the QuakeCoRE ucgmsim git repositories.  

1.  Scope

3.Test plan

 (1) Test method:  

• Black-box: testing without knowledge of the code, compare if the test output is the same as the sample output; often used in testing scripts 

• White-box: testing with knowledge of the code, often used in unit test for library.

(2) Test framework:

• Pytest

(3) Test design:

• Identify type of the application: Script: tested as a whole;  Library: unit testing 

• Identify input and output: what is used to test; what is produced from the test; contact the developer if confused 

• Formulate tailored testing plan for individual application if needed: eg. design output folder structures, writing config/readme files

• Select benchmark/correct sample output 

• Write testing script for the application 

• Execute the test 

• Compare test output with sample output

(4) Test execution

(5) Test report

Examples

(1) How to identify the type of application and corresponding test method

(2) How to design the structure of a testing folder

(3)How to write a test for Script file

(4) How to write a test for Library file

Step 1. Identify a function to test: srf_dt

Step 2. Identify input file paths for testing

Step 3. Set up functionalities to create/remove test output folder which is handled by set_up and tear_down module (as shown in the test example for script in (3))

Step 4. Write unit test called 'test_dt' for function 'srf_dt'.  Each test unit should have prefix 'test_' as part of the test function name. Use the builtin pytest.mark.parametrize decorator to enable parametrization of arguments for a test function. The @parametrize decorator defines two different (test_dt,expected_dt) tuples so that the function 'test_dt' will run twice using them in turn. The expected value eg '2.50000e-02' should be manually picked but not by using python reading functions from the sample output file.

Step 5. Run Pytest

4. Test Responsibility