PBA Home > Institutional Research & Analysis > Assessing the size or importance of a difference > Effect Size

Effect Size*

One criticism of statistical significance testing concerns the number of negligibly small research results that are labeled significant simply because they are statistically significant (Carver, 1993). It is well known that statistical significance is related to sample size; large samples are more likely to yield statistically significant results, even though the results may not be very substantive. Effect sizes express the difference between two means in terms of standard deviation units. Cohen (1988) proposed that effect sizes of .2, .5, and .8 be considered small, medium, and large, respectively. These are just guidelines, however, and may not be appropriate for all situations. In addition, interpreting effect sizes is a somewhat subjective process.

By itself, a test of statistical significance will not afford much information about the substantiveness of the difference between two means. For this reason, it is often helpful to supplement the information provided by such a test with an estimate of an effect size.

An effect size for the difference between two means, assuming equal population variances, is estimated by

where the pooled sample standard deviation is estimated by

In the first equation, is the larger of the two sample means, and the subscripts (1 or 2) denote group membership. Sometimes effect sizes are computed by simply dividing the difference in means by the standard deviation of a comparison group (e.g., a group of peer institutions), rather than using a pooled standard deviation. Additional information on effect sizes can be found in Cohen (1988), Glass (1976), Glass, McGaw, and Smith (1981), and Hedges and Olkin (1985).

References

Carver, R. P. (1993). The case against statistical significance testing, revisited. Journal of Experimental
    Education, 61 (4), 287-292.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5 (10), 3-8.
Glass, G. V, McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

*Adapted, with the author's permission from:
Schiel, J. (1998). Interpreting differences between mean ACT Assessment scores. (ACT Research Report
    No. 98-1). Iowa City, IA: ACT. (Available at http://www.act.org/research/reports/pdf/ACT_RR98-01.pdf)

PBA: L:\mgt\IR\eff_size.doc

Last revision 10/26/06