Warning: ANCOVA Used With Intact Groups
Because ANCOVA produces adjusted means, many applied researchers evidently think that this statistical procedure was designed to permit nonrandomly formed groups to be compared. Over the years, I have repeatedly come across research reports in which researchers talk as if ANCOVA has the magical power to equate such groups and thereby allow valid inferences to be drawn from comparisons of adjusted means.
Besides ANCOVA's statistical inability to generate unbiased adjusted means when intact groups are compared, there is a second, logical reason why you should be on guard whenever you come across a research report in which ANCOVA was used in an effort to equate intact groups created without random assignment. Simply stated, the covariate variable(s) used by the researcher may not address one or more important differences between the intact groups. Here, the problem is that a given covariate variable (or even a set of covariate variables) is limited in scope. For example, the covariate variable(s) used by a researcher might address knowledge but not motivation (or vice versa)
In summary, be extremely cautious when confronted with research claims based upon the use of ANCOVA with intact groups. If an important covariate variable was overlooked by those who conducted the study, pay no attention whatsoever to the conclusions based upon the data analysis. Even in the case where data on all important covariate variables were collected and used, you still should be tentative in your inclination to buy into the claims made by the researchers.
(From Chapter 15 in the 6th edition, pp. 359-360)
Copyright © 2012
Schuyler W. Huck