metamorphic testing techniques to detect defects in.ppt

22 March 2005 Kaiser: COMS W4156 Spring 2005 Metamorphic Testing Techniques to Detect Defects inApplications without Test Oracles Christian Murphy Thesis Defense April 12, 2010 Overview Software testing is important! Certain types of applications are particularly hard to test because there is no “test oracle” Machine Learning, Discrete Event Simulation, Optimization, Scientific Computing, etc. Even when there is no oracle, it is possible to detect defects if properties of the software are violated My research introduces and evaluates new techniques for testing such “non-testable programs” [Weyuker, Computer Journal’82] Motivating Example: Machine Learning Motivating Example: Simulation Problem Statement Partial oracles may exist for a limited subset of the input domain in applications such as Machine Learning, Discrete Event Simulation, Scientific Computing, Optimization, etc. Obvious errors (e.g., crashes) can be detected with certain inputs or testing techniques However, it is difficult to detect subtle computational defects in applications without test oracles in the general case What do I mean by “defect”? Deviation of the implementation from the specification Violation of a sound property of the software “Discrete localized” calculation errors Off-by-one Incorrect sentinel values for loops Wrong comparison or mathematical operator Misinterpretation of specification Parts of input domain not handled Incorrect assumptions made about input Observation Many programs without oracles have properties such that certain changes to the input yield predictable changes to the output We can detect defects in these programs by looking for any violations of these “metamorphic properties” This is known as “metamorphic testing” [T.Y. Chen et al., Info. Soft. Tech vol.4, 2002] Research Goals Facilitate the way that metamorphic testing is used in practice Develop new testing techniques based on metamorphic testing Demonstrate the effectiveness of metamorphic testi