1. The TS completion rates were higher than WS response rates in four colleges, lower in two; overall, the TS completion rate was 50% vs 31% for the WS.
   
   
2. Of the 242 faculty called, 69% were successfully contacted; 28% of those refused to participate in the TS.
   
   
3. The TS group reported less familiarity with their dean's activities than did the WS respondents.
   
   
4. In general, average TS ratings are one or two grade increments higher (e.g., C to C+ or C to B-) than their WS counterparts.
   
   
5. The ratings for each of the deans increased by about the same amount; therefore, the rank ordering of the deans remained nearly the same.
   
   
6. Almost all of the TS respondents answered at least one open-ended question, whereas less than one-quarter of the WS respondents provided at least one comment.
   
   
7. Overall, the TS comments are shorter, and seem no more useful to the persons rated, than the WS comments.
   
   
8. The direct cost per respondent per dean rated was about $12 for the TS and $0.76 for the WS.

1. The TS produced generally higher response rates, somewhat more positive ratings of deans, and a higher incidence of faculty comments.
   
   
2. The TS ratings produce an equivalent rank ordering of deans from respondents who were generally less familiar with their dean's activities; TS comments are short and no more useful than WS comments.
   
   
3. The TS mode of administrator appraisal is not as cost-effective as the WS mode.

• The phone interviews yielded higher response rates in some but not all colleges. In two colleges the proportion of telephone-sample raters completing interviews was 23-25 percentage points higher than the proportion of eligible raters returning written surveys of college administrators. For the remaining four colleges phone-interview proportions were higher in two (by 6 and 12 percentage points) and lower in two (by 4 and 13 percentage points).
  
  
• About 30% of faculty and instructors reached by telephone refused to give ratings over the phone (range 16% to 41% over colleges). Those contacted were not asked why they wished not to be interviewed; we speculate that the reasons include
  • Lack of familiarity with the dean's performance
  • Inopportune timing (e.g., during office hours)
  • Fears about the confidentiality of ratings and comments
  • A general dislike of telephone surveys, or of surveys in general
  • The fact that some had already sent written ratings.
  
  
• The phone interviews did succeed in getting ratings from some faculty and instructors who had not completed and returned their written surveys. Almost all phone respondents remembered receiving the written survey, but between 30% and 60%, over colleges, said they had not returned any written ratings.
  
  
• In both phone interviews and on the written survey, raters were asked to rate their own "familiarity with [your dean's] performance in his position." Interviewed faculty and instructors said they were less familiar with their dean's performance than the faculty and instructors who returned the written survey. In the larger colleges, 75% of written-survey respondents, but only 25% of phone respondents, said they had "high" familiarity with the dean's performance. Thus, the phone interview ratings may be based, at least in part, on the opinions of faculty and instructors who are not very knowledgeable about their dean's performance.
  
  
• Whether or not a respondent had returned written forms was in part a function of familiarity with the dean. 40-70% of phone respondents (over colleges) said they had completed and returned written forms. Combining over colleges, 72% of those saying they had "high" familiarity with the dean's performance said they had returned written forms, vs. 35% of those stating "medium" familiarity and 23% of those stating "low" familiarity. Thus it appears that lack of familiarity with administrator performance may play a large role in depressing response rates on the written survey.
  
  
• Across all schools/colleges, phone respondents gave slightly more favorable ratings of their deans than written respondents did. The change was one or two "grade notches" (e.g., C or C+ to B-, or 1/3 to 2/3 of a grade point) for every college. However, the relative ranking of deans is similar in phone and written ratings.
  

This finding parallels the results of an experiment carried out in 1997, in which 10% of all eligible raters were "prompted" several times to increase response rates. Compared to unprompted raters, prompted raters had a higher response rate (60% vs 36%,) and gave slightly more favorable ratings (by .25 of a grade point), but their ratings yielded the same rank ordering of administrators. See the full writeup of the prompting experiment for details.

The more positive ratings obtained on the phone may be due to any combination of several reasons:

• Phone respondents' lower familiarity with the dean's performance
• A fear of compromised confidentiality on the part of phone respondents, leading to a reluctance to give negative ratings
• A bias within written-survey respondents, with more dissatisfied faculty and instructors overrepresented
• A bias within phone-survey respondents, with more satisfied faculty and instructors overrepresented
• The fact of rating only the dean (on the phone) vs. many administrators (in writing)
• Chance
  


• Four items were asked on both written and phone surveys: Overall performance, scholarly credibility, effectiveness of leadership, and respect for faculty views. The pattern of results is the same for all four, with slightly higher average ratings from phone interviewees but the same rank-ordering of administrators.
  
  
• The phone interview posed four additional questions, on the dean's goals and future vision; accessibility and rapport with faculty; handling of matters such as salary, promotion, tenure, and workload; and handling of budget allocations. Average ratings for all are similar to those on the "standard" items, and ratings on the additional items correlate highly (.70 or greater) with ratings on the overall performance item.
  
  
• A much higher percentage of phone respondents than written survey respondents made open-ended comments about their dean. Between 88% and 100% of phone respondents (over colleges) responded to at least one of the three open-ended questions in the phone interview, whereas 4-23% of written questionnaire respondents wrote an open-ended comment.
  
  
• The phone comments are shorter, and seem no more thoughtful, constructive, or useful, than the written comments. In some colleges, almost all the written comments are longer (many several sentences) than any of the oral comments.
  
  
  

• Is a higher response rate obtained with the phone interview than the written survey?
• Are faculty and instructors more willing to provide open-ended comments in a phone interview than on a written survey, and are the phone-obtained comments richer, longer, and more useful than the written comments?
• Are the ratings obtained over the phone the same as, higher than, or lower than those obtained with the written survey, and are the phone-obtained ratings in the same rank order across administrators as those obtained from the written survey?
  

• three general questions which asked a respondent who his/her dean is, if he/she had received the written survey, and if he/she had returned it
• the same five closed-ended questions used to assess administrator performance on the written survey
• an additional four closed-ended questions and three open-ended questions not on the written survey.
  

• phone, $12 ($1,450 ongoing direct costs for 121 raters rating one administrator each, or a proportional increase in cost for more raters and/or more administrators)
• written, $0.76 ($2,550 ongoing direct costs for 511 individuals returning forms and rating an average of 6.6 administrators each).