Bart Massey 2013-10-22

## Overview

• Misc notes
• V&V
• Triangle Testing
• Testing Basics

## V&V

• There are three basic methods of verifying and validating code.

• Testing
• Inspection
• Formal methods
• Inspection and formal methods can also be used to verify and validate other things.

• We will mostly focus on testing and inspection in this class.

## A Test Is An Input-Output Pair

• This includes "should fail" negative tests

• How to get correct outputs? (the "effective oracle" problem)

• Slow but correct code (but few tests)
• Working backward from output to input
• "Easy" inputs

## Triangle Testing

• From Glenford Myers "The Art of Software Testing"

• Let's write a program printing whether a triangle is scalene, isosceles or equilateral

• The program should take three numeric arguments
• It should print the appropriate message after analysis
• Write as many tests as you can think of that might be useful here

• Let's look at an implementation

• Write as many more tests as you can think of

• Now we iterate a bit

## Testing Doesn't Work

• Space of possible tests is ridiculously big, like 10^1000

• Tests can only fail

• Test failures usually give minimal insight

• Best that can be hoped for is to test enough cases a user might encounter that the program is known to "kind of" work

## Black Box vs White Box

• Black Box: Tests given without knowledge of implementation

• White Box: Use implementation knowledge to construct tests

• Both are valuable

## Test Domains

• Idea: Divide input or output space up such that only one representative in each domain need be tested

• How to draw domain boundaries?
• There are still a lot
• Better idea: formal methods plus testing

## Testing Kinds

• What should be tested?

• System (acceptance)
• Unit
• Integration
• Regression
• Integration testing can be

• bottom-up
• top-down

## Testing Methods

• Random (e.g. "fuzz testing")

• "User Tests"

• Code-driven

• Domain-driven

• Coverage-driven

## Coverage Testing

• How much of the program has been tested?

• All statements?
• All branches (each way)?
• All code paths?
• All data patterns?
• Automated tools (e.g. gcov) exist

• 100% coverage is impossible

• Untested code is broken code

## Fault Seeding

• Attempt to find out how good the tests are

• Use SCMS to reliably remove seeded faults (!)

## Testing Infrastructure

• Tests need to be maintained with code

• Tests need to be runnable automatically

• Test failures need to be logged as tickets until fixed