Testing Your Programs
Bart Massey 2013-10-22
- Misc notes
- Triangle Testing
- Testing Basics
- Advanced Testing
Yesterday's CSOD was http://thedailywtf.com/Comments/Deterministic-Programming.aspx.
Today's CSOD is http://thedailywtf.com/Articles/Unit-Tested.aspx.
Homework 3 will be up shortly.
There are three basic methods of verifying and validating code.
- Formal methods
Inspection and formal methods can also be used to verify and validate other things.
We will mostly focus on testing and inspection in this class.
A Test Is An Input-Output Pair
This includes "should fail" negative tests
How to get correct outputs? (the "effective oracle" problem)
- Slow but correct code (but few tests)
- Working backward from output to input
- "Easy" inputs
From Glenford Myers "The Art of Software Testing"
Let's write a program printing whether a triangle is scalene, isosceles or equilateral
- The program should take three numeric arguments
- It should print the appropriate message after analysis
Write as many tests as you can think of that might be useful here
Let's look at an implementation
Write as many more tests as you can think of
Now we iterate a bit
Testing Doesn't Work
Space of possible tests is ridiculously big, like 10^1000
Tests can only fail
Test failures usually give minimal insight
Best that can be hoped for is to test enough cases a user might encounter that the program is known to "kind of" work
Black Box vs White Box
Black Box: Tests given without knowledge of implementation
White Box: Use implementation knowledge to construct tests
Both are valuable
Idea: Divide input or output space up such that only one representative in each domain need be tested
- How to draw domain boundaries?
- There are still a lot
- Better idea: formal methods plus testing
What should be tested?
- System (acceptance)
Integration testing can be
Random (e.g. "fuzz testing")
How much of the program has been tested?
- All statements?
- All branches (each way)?
- All code paths?
- All data patterns?
Automated tools (e.g.
100% coverage is impossible
Untested code is broken code
Attempt to find out how good the tests are
Use SCMS to reliably remove seeded faults (!)
Tests need to be maintained with code
Tests need to be runnable automatically
Test failures need to be logged as tickets until fixed