Sample Size and Power (A)

As was pointed out earlier in this chapter, the sample size used by a researcher is one of the factors that influences whether or not a false null hypothesis will be rejected. With large samples, it is possible that a false Ho will be rejected--even if there is little or no practical significance associated with the findings. As proof that this can happen, I briefly summarized, near the end of Chapter 7, a study in which a tiny correlation coefficient (-.03) ended up being statistically significant. In that study, the sample size was 21,646.

The sample size, if too large, will make the statistical test too powerful in the sense that null hypotheses that are false by a trivial amount will be declared statistically significant. Such a finding has statistical significance but no practical significance. As we have seen, a small strength-of-association index or an observed effect size provides a red flag that serves to alert the researcher and you that an unimportant finding has been declared statistically significant because of a large sample size.

The sample size, on the other hand, can be too small and, as a consequence, cause the findings to be misleading. Due to the fact that there is a direct relationship between the sample size and the probability of rejecting a false Ho, a statistical test based upon an insufficient amount of data will likely lead to a fail-to-reject decision--even if the discrepancy between the arbitrary null hypothesis, on the one hand, and the reality of the population(s), on the other hand, is so large as to deserve the label important or noteworthy. If a researcher doesn't reach a reject decision when Ho is "off target" by a wide margin, then a major Type II error is committed.

(From Chapter 8, p. 184)

Copyright © 2003 Schuyler W. Huck
All rights reserved.