Previous surveys by the Administrator Appraisal Program (AAP) have consisted of a short questionnaire mailed to all faculty members on an annual basis, seeking input on each member’s dean and all campus-wide administrators. This questionnaire provided an opportunity for written comments from faculty, but the principal evaluation tool was grading of an administrator’s performance on the familiar scale of A-F with reference to a limited number of factors.

The revised AAP conducted this year was different in several respects. First, in order to provide more in-depth feedback, the number of administrators under review was limited. Faculty had the opportunity to provide AAP feedback to the review/ reappointment process when the president, chancellor, vice chancellor or the dean of their school or college was undergoing a third- or fifth-year review. Second, faculty were requested to complete and return a "BFA Satisfaction Survey" addressing campus-wide concerns such as salary and benefit programs. Third, the survey questionnaires dispensed with the A-F grading, substituting scaled opinions of administrator effectiveness in more areas. The questionnaires contained approximately 20 questions addressing the effectiveness of each administrator's performance in several areas. 11 questions--addressing general administrative style, salary process, and diversity--were common for all administrators. The remaining questions were performance-based and tailored to the specific circumstances of the administrator and school or college. Responses were solicited using an effectiveness scale of 1 to 5, in which 1 = very ineffective and 5 = very effective (the ratings in between the committee interpreted as 2 = less than effective, 3 = effective, 4 = more than effective), plus a "Don't Know" option.

The evaluation process for Vice Chancellor DiStefano was designed to take into account the broad scope of faculty and varying response rates from different schools and colleges. While the chairs and directors of A&S have regular opportunities to interact with the Vice Chancellor, most individual faculty members do not, and therefore may be less well informed or uninformed about many aspects of the VCAA’s performance. Reflecting this feature of the comprehensive range of the survey were substantial percentages of "Don't Know" responses on faculty questionnaires, ranging from 23% to 60% in the Random Sample and All Raters pools on the 17 questions.

In an attempt to address this problem, as well as the problem of low response rates in previous years, the AAP Committee sent the questionnaire to all faculty designated as eligible by the BFA, and then analyzed the responses from four overlapping groups among the recipients. The first group of recipients was all UCB faculty (313 responses). The second group was a random sample of 286 recipients (110 responses) from among all UCB faculty members. The third group was the 47 chairs and directors of A&S units (33 responses). The fourth group was made up of those schools and colleges outside of A&S which had a response rate high enough to be representative (76 responses).

The AAP Committee had agreed, on the advice of several faculty who specialize in survey methods, that a 60% return rate would be needed for a representative statistical study. We had hoped that three telephone promptings, combined with the fact that VCAA DiStefano was being reviewed at the time we undertook the survey, would lead to an adequate response level on the part of the large random sample, which we had planned to make the focus of our analysis. Unfortunately, the response rate fell short of the necessary rate for a representative sample, with response rates of 30% for All Raters and 37% for the Random Sample.

The respondents from among the Chairs and Directors of A&S constituted 72% of that population; they therefore formed a statistically sound survey in the sense that we can place confidence in these results as fairly representing the opinions of this group. In order to have results from other schools and colleges included as well, we combined the results of the schools and colleges which had response rates over 60%. These were Law, 63% (33 responses); Libraries, 60% (26 responses); and Education, 86% (25 responses). Although the response rate for the Random Sample and All Raters groups (hereafter referred to as RS and All) is too low to assure that they are representative, we do refer to the results of these surveys for comparative purposes below.

The AAP Committee hopes to provide a developmental picture of strengths and weaknesses for administrators standing for review or reappointment in AY 2000-2001. Broadly viewed, we considered three general categories in evaluating responses regarding VCAA DiStefano:

Strengths to Build On represent areas of the administrator’s performance which were rated as highly effective by a substantial majority of the faculty (60% and higher 4 or 5 ratings).

Issues to be Mindful Of were judged to be effective or better by the majority of the respondents, but less than effective by a significant minority of respondents (25% - 39% ratings of 1 or 2).

Areas that Need Improvement were areas judged by a significant fraction of the faculty as unsatisfactory (over 40% 1 or 2 ratings).

VCAA DiStefano received an extremely high proportion of ratings of effective or above. Since the chairs and directors (C/D) and the combined grouping of Law, Libraries and Education (LLE) had a response rate that gave us confidence in identifying representative opinions among these groups, we review these results first. Over two-thirds (68% - 89%) of C/D respondents and over three-quarters (76% - 93%) of LLE respondents judged the Vice Chancellor to be effective to very effective (rating of 3, 4, or 5) on all questions. Corresponding levels in the other two populations were RS: 61%-83%, All: 57%-81%.

The committee defined these to be areas in which more than 60% of the respondents gave ratings of 4 or 5. The Vice Chancellor received ratings in this category on 4 of the 17 questions in the C/D appraisals and on 11 questions in the LLE grouping.

The highest ratings among the C/D respondents had to do with the treatment of faculty and personal character:

Interestingly, although the LLE group found Strengths in almost three times as many survey areas as the C/D group did, one question appeared in the C/D ratings for this category that did not appear in the LLE, Recognizing importance of non-tenure track faculty. This question may draw especially high ratings among the Chairs and Directors because A&S has been much more affected by the Instructors’ Bill of Rights than any other school or college.

In addition to the first three above, the LLE grouping showed high appraisal scores for open communication and inclusive process:

• Communicating openly with faculty on campus issues
• Involving faculty in decisions appropriately
• Creating an atmosphere of trust
• Managing conflicts among faculty and staff

The remaining areas of strengths in the LLE grouping were focused on support for innovation and on timeliness:

• Supporting innovation in teaching
• Supporting innovation in research
• Working with faculty on new programs
• Making decisions in a timely fashions

All of the above were given high though not equivalent ratings in the other groups as well, with a range of 43% - 68% 4s or 5s in these areas for C/D respondents; RS: 40% - 60%; and All: 45% - 59%.

The performance of VCAA DiStefano was never placed in the "Needs Improvement" category by the C/D, LLE and RS respondents. Only one question, Leading UCB to an equitable, merit-based salary system, fell narrowly into this category in the All pool.

The average ratings of the remaining areas were in the range of effective and above. In some of these areas, however, there was a notable percentage of respondents (25% - 39%) who assigned ratings of 1 or 2. While none of the questions surveyed in the combined LLE grouping received ratings in this category, ten questions in the C/D appraisals did.

Because some of these have appeared above in areas of high ratings, it should be explained that even though these represent areas of concern, they are also bi-modal, with higher percentages of ratings at the positive end of the scale in most cases. The areas of divided opinion which also appeared in areas of strength are:

• Supporting innovation in research
• Working with faculty on new programs
• Involving faculty in campus decisions
• Managing conflicts among faculty and staff
• Involving faculty in decisions appropriately
• Making decisions in a timely fashion

The two questions associated with funding issues are also included in the category of issues to be mindful of:

• Leading UCB to an equitable, merit-based salary system
• Securing funds for faculty compensation and recruitment

The remaining two in this category are:

• Promoting the University’s research mission with outside constituencies
• Recruiting and retaining diverse faculty, staff and students

Results in the RS and All pools showed the same patterns, except that the former had somewhat fewer than the C/D group in this category (6), while the All Raters group and somewhat more (14). While the pattern of highest and lowest ratings are the same throughout all four groups of respondents, the difference between the LLE group, with a return rate of 70%, and the All pool, with a return rate of 30%, is notable, since the former group had no questions with percentages of 1 or 2 ratings low enough to appear in this category, whereas in the latter only two questions did not fall into this category.

Since the Satisfaction Survey focused on conditions of the University as a whole, it may offer a helpful context in which to consider responses to administrators as well. For instance, Vice Chancellor DiStefano’s lowest rating in all population pools was in response to Leading UCB to an equitable, merit-based incremental salary system, with mean (average) scores ranging from 2.7 - 3.2. While these ratings point to the importance of continued examination and revisiting of faculty concerns, the Satisfaction Survey also suggests that discontentment about salary is a systemic issue revolving around inadequate University resources in general. The Vice Chancellor’s ratings are high in regard to the responses to relevant questions in the Satisfaction Survey, such as Salary relative to peers at other research institutions, with mean ratings of 1.9 - 2.2 and Equitable distribution of salary, 2.1 - 2.8. (We use mean scores for ease of comparison between the Satisfaction Survey and the appraisal of VCAA DiStefano.)

Two patterns stand out in Vice Chancellor DiStefano’s appraisals. One is an extremely high percentage of ratings in the categories of effective to very effective, with no areas falling into the category of "Need Improvement." The other is a bi-modal distribution pattern in which the concentrations of his lowest ratings were generally combined with greater percentages of ratings at the positive end of the spectrum.

The areas of potential vulnerability which may be indicated by those questions which received the lower ratings of those with bi-modal distribution are focused primarily on funding issues, such as salary and securing funds for compensation and recruitment. Promoting the research mission outside the University and attracting and retaining a diverse faculty also appeared as areas of concern in several of the respondent pools.

In regard to his strengths, Vice Chancellor DiStefano received not only exceptionally large percentages of ratings of effective or higher on every question surveyed, but also particularly high ratings in areas relating to treatment of faculty and shared governance process, such as inclusiveness, integrity, trust and open communication.

Addendum: Activities and Accomplishments