Inflated Type I Error Rate
Suppose a researcher measures each of several people on seven variables. Also suppose that the true correlation between each pair of these variables is exactly 0.00 in the population associated with the researcher's sample. Finally, suppose our researcher computes a value of r for each pair of variables, tests each r to see if it is significantly different from 0.00, and then puts the results into a correlation matrix. If the.05 level of significance is used in conjunction with the evaluation of each r, the chances are about 66 percent that at least one of the rs will turn out to be significant. In other words, even though the alpha level is set equal to .05 for each separate test conducted, the collective Type I error risk has ballooned to about .66 because 21 separate tests are conducted.
My caution here is simple. Be wary of any researcher's conclusion if a big deal is made out of an unreplicated single finding of significance when the hypothesis testing procedure is used simultaneously to evaluate many null hypotheses. In contrast, give researchers extra credit when they apply the Bonferroni or Dunn-Sidak technique to hold down their study-wide Type I error risk.
(From Chapter 8 in the 6th edition, pp. 181-182)
Copyright © 2012
Schuyler W. Huck