Analysis is necessary – but far from sufficient.PPTVIP

Jon Pincus (Microsoft Research) Analysis is necessary – but far from sufficient Jon Pincus Reliability Group (PPRC) Microsoft Research Why are so few successful real-world development and testing tools influenced by program analysis research? Outline Provocation Successful tools Analysis – in context Implications for analysis Conclusion Success: a simple view A tool is successful if people use it Not if people think it’s interesting – but don’t try it Not if people try it but don’t use it Not if people buy it but don’t use it (“Shelfware”) Some examples of success Purify BoundsChecker PREfix (2.X and later) Especially interesting because 1.0 was unsuccessful Why do people use a tool? If it helps them get their work done … … more efficiently than they would otherwise … without making them look (or feel) bad. Aside: look at organizational and personal goals. See Alan Cooper’s books, e.g. About Face Value vs. Cost Value: the quantified benefit from the tool Cost: primarily time investment Licensing cost is typically much smaller (Value – Cost) must be Positive Positive fairly quickly More positive than any alternatives Value and cost are difficult to estimate … … and others’ estimates are often questionable An example Purify 1.0: Virtually zero initial cost on most code bases “trial” license easy to integrate Immediate value Companies then invested to increase the value E.g., changing memory allocators to better match Purify’s (and buying lots of licenses) Characteristics of successful tools Successful tools almost always address significant problems, on real code bases, give something for (almost) nothing, and are easy to use. Significant problems Nobody fixes all the bugs. What are the key ones? Often based on most recent scars Often based on development or business goals Examples: Purify: memory leaks BoundsChecker: bounds violations Lint (back in KR days): portability issues Real code bases Large code bases in nasty languages (e.g