Edited by Waldemar Karwowski, Boca Raton, FL CRC Press Determining Usability Test Sample Si.pdfVIP

International Encyclopedia of Ergonomics and Human Factors, 2006, Second Edition, Volume 3 Edited by Waldemar Karwowski, Boca Raton, FL: CRC Press 3084 Determining Usability Test Sample Size Carl W. Turner*, James R. Lewis?, and Jakob Nielsen? *State Farm Insurance Cos., Bloomington, IL 61791, USA ?IBM Corp., Boca Raton, FL 33487 USA ?Nielsen Norman Group, Fremont, CA 94539, USA 1 INTRODUCTION Virzi (1992), Nielsen and Landauer (1993), and Lewis (1994) have published influential articles on the topic of sample size in usability testing. In these articles, the authors presented a mathematical model of problem discovery rates in usability testing. Using the problem discovery rate model, they showed that it was possible to determine the sample size needed to uncover a given proportion of problems in an interface during one test. The authors presented empirical evidence for the models and made several important claims: ? Most usability problems are detected with the first three to five subjects . ? Running additional subjects during the same test is unlikely to reveal new information. ? Return on investment (ROI) in usability testing is maximized when testing with small groups using an iterative test-and-design methodology. Nielsen and Landauer (1993) extended Virzi’s (1992) original findings and reported case studies that supported their claims for needing only small samples for usability tests. They and Lewis (1994) identified important assumptions about the use of the formula for estimating problem discovery rates. The problem discovery rate model was recently re-examined by Lewis (2001). 2 THE ORIGINAL FORMULAE Virzi (1992) published empirical data supporting the use of the cumulative binomial probability formula to estimate problem discovery rates. He reported three experiments in which he measured the rate at which usability experts and trained student assistants identified problems as a function of the nu