Testing Your Programs

Bart Massey 2013-10-22


  • Misc notes
  • V&V
  • Triangle Testing
  • Testing Basics
  • Advanced Testing

Misc Notes


  • There are three basic methods of verifying and validating code.

    • Testing
    • Inspection
    • Formal methods
  • Inspection and formal methods can also be used to verify and validate other things.

  • We will mostly focus on testing and inspection in this class.

A Test Is An Input-Output Pair

  • This includes "should fail" negative tests

  • How to get correct outputs? (the "effective oracle" problem)

    • Slow but correct code (but few tests)
    • Working backward from output to input
    • "Easy" inputs

Triangle Testing

  • From Glenford Myers "The Art of Software Testing"

  • Let's write a program printing whether a triangle is scalene, isosceles or equilateral

    • The program should take three numeric arguments
    • It should print the appropriate message after analysis
  • Write as many tests as you can think of that might be useful here

  • Let's look at an implementation

  • Write as many more tests as you can think of

  • Now we iterate a bit

Testing Doesn't Work

  • Space of possible tests is ridiculously big, like 10^1000

  • Tests can only fail

  • Test failures usually give minimal insight

  • Best that can be hoped for is to test enough cases a user might encounter that the program is known to "kind of" work

Black Box vs White Box

  • Black Box: Tests given without knowledge of implementation

  • White Box: Use implementation knowledge to construct tests

  • Both are valuable

Test Domains

  • Idea: Divide input or output space up such that only one representative in each domain need be tested

    • How to draw domain boundaries?
    • There are still a lot
    • Better idea: formal methods plus testing

Testing Kinds

  • What should be tested?

    • System (acceptance)
    • Unit
    • Integration
    • Regression
  • Integration testing can be

    • bottom-up
    • top-down

Testing Methods

  • Random (e.g. "fuzz testing")

  • "User Tests"

  • Code-driven

  • Domain-driven

  • Coverage-driven

Coverage Testing

  • How much of the program has been tested?

    • All statements?
    • All branches (each way)?
    • All code paths?
    • All data patterns?
  • Automated tools (e.g. gcov) exist

  • 100% coverage is impossible

  • Untested code is broken code

Fault Seeding

  • Attempt to find out how good the tests are

  • Use SCMS to reliably remove seeded faults (!)

Testing Infrastructure

  • Tests need to be maintained with code

  • Tests need to be runnable automatically

  • Test failures need to be logged as tickets until fixed

Last modified: Thursday, 24 October 2013, 12:16 AM