The 7-Step and 9-Step Versions of Hypothesis Testing

Dear Students,

In an attempt to help you with the 7-step and 9-step versions of hypothesis testing, I'd like to do two things right now in this email message. First, I'm going to type out a paragraph of material that appears in the 4-page journal article that I gave you at the end of class yesterday. Then, I'll try to draw a connection between the quoted paragraph and the two versions of hypothesis testing discussed in Ch. 8.

Here's the paragraph from the Olds/Abernethy article: "Post-Exercise Oxygen Consumption Following Heavy and Light Resistance Exercise":

Statistical Analyses

Standard descriptive statistics were used in all analyses. One-way analyses of variance (ANOVA) was used to compare ROC following the three conditions. Fisher's PLSD test was used in post hoc analyses. Effect sizes were calculated between the resting and light, and [between] the resting and heavy conditions. The effect size is the difference between the mean values of two samples divided by the standard deviation. Effect sizes are often used to give an idea of the magnitude of the difference between groups, and to circumvent difficulties arising when the sample size is small (failure to reach statistical significance) or large (where small differences may be statistically, but not practically, significant). Effect sizes of about 0.2., 0.5, and 0.8 are often categorized as small, moderate, and large, respectively.

Ok. That's the paragraph from the Olds/Abernethy article. And here come a few comments from Sky:

  • Don't worry AT ALL about the 2nd or 3rd sentences; they are NOT germane to the material in Chapter 8.

  • The most important part of this passage is contained in the next-to-last sentence. The exceedingly important point made by the authors in this sentence is as follows:
    1. In hypothesis testing, a sample size that's two small is likely to produce a fail-to-reject decision . . .even if there's a large and true difference between the means of the populations associated with the two samples being compared. In other words, a sample size that's two small will tend to bring forth a Type II error, thus causing the researcher to "miss" detecting something important that occurred.
    2. In hypothesis testing, a sample size that's too large is likely to produce a reject decision . . . even if there's a small and trivial difference between the means of the populations associated with the two samples being compared. In other words, a sample size that's too big can create a situation wherein a "significant difference" is shown to exist between two sample means . . . when, in fact, the difference between the sample means is tiny and "significant" only in a statistical sense and NOT AT ALL in a practical sense!!!!!!!!!

  • The concept of "effect size" is at the heart of the 9-step version of hypothesis testing . . . and the ES is simply a judgment by the researcher as to the line of demarcation between "small" (i.e., trivial) differences, on the one hand, and "large" (i.e., important) differences, on the other. The 9-step version of hypothesis testing requires a researcher to specify his/her opinion as to ES . . . and then the appropriate sample size is determined (via a formula or a chart) so the researcher will have a decent chance of reaching a "reject" decision if there is a bigger-than-ES difference between the study's population means. What's a "decent chance" is usually considered to be about 80%, and this number, when converted from a percentage to a decimal, is called the test's "power." In summary, the 9-step version of hypothesis testing allows a researcher to design his/her study so the sample size(s) won't be too small or too large; this is done by (a) asking the researcher to specify ES, (b) asking the researcher to specify power, and (3) asking the researcher to determine (via formula or chart) the "proper" number of subjects to have in the study's sample(s). These 3 steps are positioned as Steps #4, #5, and #6 of the 9-step version of hypothesis testing.

  • In the 7-step version of hypothesis testing, the researcher simply adds a step after rejecting or failing-to-reject the null hypothesis with the 6-step procedure (wherein there's no pre-planned ES, power specification, and sample size determination). The 7th step involves using the sample data to either (1) show that sample size was not "too small" or "too large" or (2) look at the sample means and evaluate the observed difference as being either small-and-trivial or big-and-important. The first of these options asks the researcher to specify ES (as in the 9-step version of hypothesis testing) and then determine, via formula or chart, how much power there was in light of the sample sizes that were used. The second of these options allows the researcher to compute a strength-of-association index, or to estimate the observed magnitude of effect.

  • In the Olds/Abernethy study, the 7-step version of hypothesis testing was used. After rejecting or failing-to-reject any null hypothesis, the researchers computed the "effect size" based upon the sample data. They deserve high marks for going past the 6 steps that form the most basic version of hypothesis testing. What they did in the 7th step, using my terminology, was to compute an estimate of the observed magnitude of effect. I prefer my longer term to their short phrase "effect size," because they computed their "effect sizes" from the data and are, in a very real sense, telling us "what they got" (rather than using the term "effect size" to tell us their OPINION as to the line of demarcation between small and large differences). Olds and Abernethy do, however, tie their computed "effect sizes" back to some judgmental "benchmarks," for they make reference to Cohen's standardized numerical values of 0.2 (a "small" difference), 0.5 (a "medium" difference), and 0.8 (a "large" difference). Sky Huck

Copyright © 2012

Schuyler W. Huck
All rights reserved.

| Book Info | Author Info |

Site URL: www.readingstats.com

Top | Site Map
Site Design: John W. Taylor V