PBA Home > Institutional Research & Analysis > Assessing the size or importance of a difference

Assessing the size or importance of a difference

In all kinds of analysis, we compare two or more results. E.g.:

• Average responses on a survey from two groups of students, or for two sets of colleges.
• Graduation rates (which are percentages) at two times, or for two groups of students.

We usually want to know – Is there a difference? And if so, is it a “big” difference, or important, or worth noting?

The answer depends on three different tests for a single comparison!

Test 1: Statistical significance. Is the difference reliable? Or could the difference we’re seeing likely be due to chance? We use standard statistical tests for this. Reliability is strongly affected by the number of cases – for the same difference in averages, the more cases, the more reliable and the more likely the difference is “statistically significant.”

• Rule of thumb: We in PBA will not draw attention to, or report as a difference, anything that’s not sufficiently reliable. So the reader need not worry about it.

Test 2: Effect size. Is the difference noticeable, given the distributions of responses or scores? Effect size is an alternative to standard statistical tests, and expresses the difference in terms of standard deviation units. Click here for technical details.

• Rule of thumb: We in PBA will draw attention to, or report as a difference, reliable differences as follows:
• Effect size of .35 or greater (in absolute value): Noticeable, detectable difference
• In general, we interpret effect sizes in the following manner:
• .20 or less -- small, not noticeable, "undetectable by naked eye"
• .21-.49 -- small to medium
• .50-.79 -- medium to large
• .80 or greater -- large, can't be missed

• More on effect size interpretation

Test 3: Importance. Is the difference important? A difference may be reliable, and noticeable, but of no importance – e.g., a height difference of 0.2 inches between two populations. There is no statistical test for importance – you have to decide, based on the context.

• Rule of thumb: None.

PBA: L:\mgt\IR\StatRulesOfThumb.doc

Last revision 10/26/06