Examples of the Distinction Between Statistical and Practical significance
Example #1: Statistical vs. Practical Significance
Earlier in this chapter, you saw how researchers can do certain things in an effort to see whether a statistically significant finding is also meaningful in a practical sense. Unfortunately, many researchers do not rely on computed effect size indices, strength-of-association measures, or power analyses to help them avoid the mistake of "making a mountain out of a molehill." They simply use the six-step version of hypothesis testing and then get excited if the results are statistically significant. Having results turn out to be statistically significant can cause researchers to go into a trance in which they willing allow "the tail to wag the dog." Consider, for example, Excerpt 11.34 [not shown here]. In this instance, the "tail" was the presence of a statistically significant difference between the two means, and the "dog" was the researchers' assessment as to whether this difference was meaningful in a practical sense.
When the researchers stated that "despite the small difference in means, there was a significant difference," they imply that their statistical analysis has come along and magically transformed a "molehill" of a mean difference into a "mountain" that deserves others' attention. Had they not been blinded by the allure of statistical significance, the researchers would have focused on the "small difference" and not the "significant difference," and perhaps they would have said "although there was a statistically significant difference between the means, the mean difference was small."
(From Chapter 11, p. 318)
Example #2: Practical and Statistical Significance in Two-Way ANOVAs
In Excerpt 14.36 [not shown here], notice how three F-ratios turned out to be significant, yet two of these were labeled by the authors as being of only minor practical significance. Clearly, these researchers (as well as those who conducted the studies appearing in Excerpts 14.34 and 14.35) were aware of the fact that a single inferential result can turn out to be, at the same time, both significant from a statistical perspective and nonsignificant from a practical perspective. They deserve credit for conducting their data analyses with this important distinction in mind, and even further credit for incorporating this distinction into the written summaries of their investigations. Unfortunately, most researchers formally address only the concept of statistical significance, with the notion of practical significance automatically (and incorrectly) superimposed on each and every result that turns out to be statistically significant. In your reading of research reports, remain vigilant for instances of this unjustified interpretation of results.
(From Chapter 14, pp. 421-422)
Copyright © 2012
Schuyler W. Huck