keppel,g.wickens,t.d.designandanalysis.docVIP

Keppel, G. Wickens, T. D. Design and Analysis Chapter 6: Simultaneous Comparisons and the Control of Type I Errors ? You should design your research with specific questions in mind, which you then test with specific analyses. However, your design will often lend itself to additional analyses, which may well allow you to learn more about the operation of the variables in question. These additional analyses come with the “burden” that you’ll have a greater chance of making a Type I error among the additional analyses. This chapter covers means of controlling Type I error among these simultaneous comparisons. 6.1 Research Questions and Type I Error ? The family of tests is a set of tests you intend to compute to address a set of research questions. The familywise Type I error rate (?FW) is the probability of making at least one Type I error in the family of tests when all H0 are true. ? When you consider the huge set of possible post hoc tests one might compute, then you are considering the experimentwise error rate (?EW). Needless to say, it will typically be the case that ?FW ?EW. ? With a per-comparison error rate (?), you can compute the familywise Type I error rate for a number of comparisons (c) as: ?FW = 1 – (1 – ?)c (6.1) ? Thus, using KW51 as an example, if you intended to compute three comparisons using ? = .05 for each comparison, your ?FW would be .14. Though not as accurate as the above formula, for a quick and dirty estimate of ?FW you could simply use c? (which in this case would give you an estimate of .15). Of course, as the number of comparisons grows, so does ?FW. To convince yourself of this relationship between c and ?FW, compute ?FW for the number of comparisons indicated below: Number of Comparisons (c) ?FW 4 5 6 ? The formula for computing ?FW only works for orthogonal comparisons (i.e., assumed independence), but ?FW also increases with an increasing number of nonorthogonal comparisons. Thus, because for KW51