feedback-directed random test generation.pptVIP

Feedback-directed Random Test Generation (to appear in ICSE 2007) Carlos Pacheco Shuvendu Lahiri Michael Ernst Thomas Ball MIT Microsoft Research January 19, 2007 Random testing Select inputs at random from a program’s input space Check that program behaves correctly on each input An attractive error-detection technique Easy to implement and use Yields lots of test inputs Finds errors Miller et al. 1990: Unix utilities Kropp et al.1998: OS services Forrester et al. 2000: GUI applications Claessen et al. 2000: functional programs Csallner et al. 2005, Pacheco et al. 2005: object-oriented programs Groce et al. 2007: flash memory, file systems Evaluations of random testing Theoretical work suggests that random testing is as effective as more systematic input generation techniques (Duran 1984, Hamlet 1990) Contributions We propose feedback-directed random test generation Randomized creation of new test inputs is guided by feedback about the execution of previous inputs Goal is to avoid redundant and illegal inputs Empirical evaluation Evaluate coverage and error-detection ability on a large number of widely-used, well-tested libraries (780KLOC) Compare against systematic input generation Compare against undirected random input generation Outline Feedback-directed random test generation Evaluation: Randoop: a tool for Java and .NET Coverage Error detection Current and future directions for Randoop Random testing: pitfalls Feedback-directed randomtest generation Build test inputs incrementally New test inputs extend previous ones In our context, a test input is a method sequence As soon as a test input is created, execute it Use execution results to guide generation away from redundant or illegal method sequences towards sequences that create new object states Technique input/output Input: classes under test time limit set of con