PBA Home > Institutional Research & Analysis > FCQ > Statistics Comparing the Section to the Department and Campus

Statistics Comparing the Section to the Department and Campus

   A    tenured and tenure-track 
   B    other primary instructors, including GPTI, adjunct, visiting, and
        honoraria
   C    teaching assistants. 

In the rest of this text, "group" refers to the evaluated instructor's department/instructor group or campus/instructor group.

AVERAGE(AVG): The department or campus group average is the mean of the section averages of all sections in the group. It is NOT a weighted average. Each section in the group contributes equally to the group average, regardless of section size. Except in very small groups (under 5 sections), group averages and medians are very similar.

   Example: Dept XYZ has 4 sections in instructor group A.  Their averages
   on the course rating are 3.3, 3.8, 3.0, and 2.7.  The department group
   average is the sum of these (12.8) divided by the number of sections
   (4) = 3.2.  

The STD is expressed in terms of points on the FCQ 0-4 rating scale. It is usually between .40 and .70 for groups with 40 or more sections, and between .10 and .90 for smaller groups. If all sections in the group have similar average ratings, the STD is small (zero if all ratings are identical). If some sections have high ratings and some low, the STD is higher.

   Examples   Section ratings                        Avg       STD
              3.0  3.0  3.2  2.7  3.1  2.8  2.9     2.96      0.16
              3.0  2.0  3.8  2.4  3.1  3.6  2.8     2.96      0.59

   Example: for Boulder, the spring 1992 campus average instructor rating
   for instructor group A was 3.29, with a standard deviation of 0.58. 
   Thus we'd expect about 2/3 (67%) of all sections to have ratings
   between 3.29 - 0.58 = 2.71 and 3.29 + 0.58 = 3.87.  In fact, 70% do. 
   Half of the remaining 30% have average ratings above 3.87, and half
   have averages below 2.71. 

DIF: labelled on the reports as "strength/reliability of difference between your rating and group average." Beginning in September, 1997, with section reports for summer 1997, DIF takes on the values ---, --, -, blank, +, ++, +++. This change makes DIF codes on the individual section reports consistent with those on the Instructor Summary.

At most, about 13% of all sections in a group will have a +++ or --- on a given question; an additional 18% or so will have a ++ or --, and approximately another 30% will have a + or a -. The remaining sections will have a blank in the DIF column, indicating that the difference between section and group means is negligible. NOTE: If the average rating for a group is very high, it's possible that no sections will differ positively from the average enough to have a +++, ++, or + in the DIF column for that question.

In technical terms the DIF is a coded Z-score. We print + or - when the section average is between .5 and 1 STDs from the group average, ++ or -- when the difference is 1 to 1.5 STDs, and +++ or --- when the difference is 1.5 STDs or more. A blank means that the difference is less than .5 STD.

When the group's STD is larger, it takes a greater difference between the section average and the group average to produce each level of DIF. So if there's a wide range of average ratings among sections in the group, it can be impossible for any one section to get a +++ or --- (1.5 or more STDs away from the group average) because it would have to rate above 4.0 or below 0.0 to do so. Conversely, if most sections in a group have almost the same average ratings, a section with an average only .20 away from the group average may get a +++ or ---.

PRIOR TO SUMMER 1997 there were no +++ or --- values and ++, --, +, and - were harder to get--that is, plusses and minuses were assigned to a smaller proportion of sections. ++ or -- indicated that the difference between the section average tand the group average was significant at the .05 probability level, and + or - indicated a difference significant at the .15 probability level. We determined what to print as follows:

   Z = (section average - department or campus group average)
       ------------------------------------------------------
                 STD for department or campus group

   If absolute value of Z is > 1.96 print ++ or -- (-- if Z is negative)

   Otherwise if absolute value of Z is > 1.44 print + or -

   Otherwise print nothing (blank). 

PERCENTILE = the proportion of sections in the group with average ratings lower than that of the section in question. Thus 95 means 95% of sections had lower ratings, while 20 means that only 20% had lower ratings. Percentiles are calculated by putting sections in order of their ratings, then, starting at the top, assigning the highest 1% of the sections to percentile 99, the next highest 1% to percentile 98, etc. When there are ties--sections with exactly the same average rating--all sections with that rating get the highest percentile assigned to any section with that rating.

Percentiles on the section reports are to the nearest 5, and about 5% of all sections in a group will have each possible value. We show percentiles to the nearest 5 to reduce computation time and to reduce the temptation to over-interpret small differences in percentiles. Rounding is done as follows:

                1st- 6th rounded to  5th
                7th-11th rounded to 10th
               12th-16th rounded to 15th
                    . . . 
               82nd-86th rounded to 85th
               87nd-91st rounded to 90th
               92nd-99th rounded to 95th (about 8% of all sections)

AVERAGES and DIFs vs. PERCENTILES

Sections with one, two, or three pluses or minuses in the DIF column will almost always have percentiles between 70-99 or 1-30. However, the converse is NOT true. Some sections MUST have very low and very high percentiles, and their averages will not necessarily differ significantly from the group average. (Prior to spring 97, sections with a single or double plus or minus in the DIF column almost always had percentiles of 85-95 or 5-15.)

Last revision 12/19/06