Statistical Significance vs. Practical Significance (B)

Earlier in this chapter, you saw how researchers can do certain things in an effort to see whether a statistically significant finding is also meaningful in a practical sense. Unfortunately, many researchers do not rely on computed effect size indices, strength-of-association measures, or power analyses to help them avoid the mistake of "making a mountain out of a molehill." They simply use the six-step version of hypothesis testing and then get excited if the results are statistically significant.

Having results turn out to be statistically significant can cause researchers to go into a trance in which they willing allow "the tail to wag the dog." Consider, for example, Excerpt 11.34 [not shown here]. In this instance, the "tail" was the presence of a statistically significant difference between the two means, and the "dog" was the researchers' assessment as to whether this difference was meaningful in a practical sense.

When the researchers stated that "despite the small difference in means, there was a significant difference," they imply that their statistical analysis has come along and magically transformed a "molehill" of a mean difference into a "mountain" that deserves others' attention. Had they not been blinded by the allure of statistical significance, the researchers would have focused on the "small difference" and not the "significant difference," and perhaps they would have said "although there was a statistically significant difference between the means, the mean difference was small."

(From Chapter 11, p. 318)

Copyright © 2003 Schuyler W. Huck
All rights reserved.