Consider the Null Hypothesis Before Looking at the p-Level
My first point [here at the end of Chapter 17] is simply a reiteration that he data-based p-value is always computed on the basis of a tentative belief that the null hypothesis is true. Accordingly, the statistical results of a study are always tied to the null hypothesis. If the researcher's null hypothesis is silly or articulates something that no one would defend or expect to be true, then the rejection of the null hypothesis, regardless of how "impressive" the p-value, does not signify an important finding.
If you think that this first point is simply a "straw man" that has no connection to the real world of actual research, consider this real study that was conducted not too long ago. In this investigation, chi square compared three groups of teachers in terms of the types of instructional units they used. Two kinds of data were collected from the teachers: (1) their theoretical orientation regarding optimal teaching-learning practices and (2) what they actually did when teaching. The results indicated that skill-based instructional units tended to be used more by teachers who had a skill-based theoretical orientation, that rule-based instructional units were used moreso by teachers who had a rule-based theoretical orientation, and that function-based instructional units were utilized to a greater extent by teachers who possessed a function-based theoretical orientation. Are you surprised that this study's data brought forth a rejection of the chi-square null hypothesis of no relationship between teachers' theoretical orientation and type of instructional unit used? Was a study need to reach this "finding"?
(From Chapter 17, p. 470)
Copyright © 2012
Schuyler W. Huck