Overlapping Distributions

Our next-to-last warning [in Chapter 21] is really a reiteration of an important point made earlier in this book regarding "overlapping distributions." If two groups of scores are compared and found to differ significantly from each other (even at "impressive" p-levels), it is exceedingly likely that the highest scores in the "low" group are higher than the lowest scores in the "high" group. When this is the case, as it almost always is, a researcher should not claim--or even suggest--that the individuals in the "high" group outperformed the individuals in the other group. What can be said is that people in the one group, on the average, did better. Those three little words on the average are essential to keep in mind when reading research reports.

Excerpt 21.34 [not shown here] nicely illustrates the likely solution of overlapping distributions. As you can see, a Mann-Whitney U test showed that the two groups being compared were significantly different (p<.05). Look closely, however, at the range of scores in each group. Scores in the experimental group ranged from 24 to 53; scores in the comparison group extended from 30 to 62. Anyone from either group who had a score between 30 and 53 was in a "zone" where the two distributions overlapped. Most likely, the majority of the individuals in each group had scores that positioned them in this zone. If you can picture the distributions of scores corresponding to these two groups (with part of that picture involving a place where the two distributions overlap each other), then you should be able to see why it is wrong to say that "students in the experimental group scored lower" (with the implication being that they all scored lower).

(From Chapter 21, pp. 674-675)

Copyright © 2003 Schuyler W. Huck
All rights reserved.