Report of the
Concerning
Philip DiStefano, Provost,
The Administrator Appraisal Program (AAP) seeks to provide in-depth feedback with a high rate of faculty response, in order to assure a representative survey. Faculty have the opportunity to provide AAP feedback to the review/ reappointment process when the president, chancellor, provost, or the dean of their school or college is undergoing the third- or fifth-year review. Faculty are also requested to complete and return a “BFA Satisfaction Survey” addressing campus-wide concerns such as salary and benefit programs. In regard to the appraisal of administrators, the surveys contain questions addressing the effectiveness of each administrator's performance in key areas. Eleven questions--addressing general administrative style, salary process, and diversity--are common for all administrators. The remaining questions are performance-based and tailored to the specific circumstances of the administrator and school or college. Responses are solicited using an effectiveness scale of 1 to 5, in which 1 = very ineffective and 5 = very effective (the ratings in between the committee has interpreted as 2 = less than effective, 3 = effective, 4 = more than effective), plus a "Don't Know" option.
The evaluation process for Provost DiStefano was designed to take into account the broad scope of faculty and varying response rates across different schools and colleges. While chairs, directors and vice chancellors have regular opportunities to interact with the provost, most individual faculty members do not, and therefore may be less well informed or simply uninformed about many aspects of the provost’s performance. Reflecting this feature of the comprehensive range of the survey were substantial percentages of "Don't Know" responses on faculty questionnaires, ranging from 19% to 58% in the Random Sample and All Raters respondent pools on the 18 questions.
In an attempt to address the problem of different levels of familiarity, the AAP Committee sent the questionnaire to all faculty designated as eligible by the BFA, and then analyzed the responses from three groups among the recipients. The first group of recipients was all UCB faculty.[2] The second group was a random sample of 127 recipients from among all UCB faculty members. The third group was 101 faculty who were in a position to have the most knowledge of the provost’s work: all UCB chairs and directors, the Boulder Faculty Assembly Executive Committee, and all deans, vice chancellors and associate vice chancellors, with the exception of any who are not rostered faculty (80 responses)
The AAP Committee and Boulder Faculty Assembly have agreed, on the advice of faculty who specialize in survey methods, that a 60% return rate is needed for a representative statistical study. The random sample and knowledgeable groups both met this requirement, with a 61% response rate for the random sample (78 out of 127) and a 79% response for the knowledgeable sample (80 out of 101). The response rate for the entire UCB faculty is too low at 32% to assure that it is representative.
The AAP Committee hopes to provide a developmental picture of strengths and weaknesses for administrators standing for review or reappointment in AY 2003-2004. Broadly viewed, we considered four general categories in evaluating responses regarding Provost DiStefano:
Strengths to Build On represent areas of the administrator’s performance which were rated as highly effective by a substantial majority of the faculty (60% or higher 4 or 5 ratings).
Assets to Protect were areas where at least half of the responding faculty found the administrator’s performance to be better than effective (50% - 59% of respondents gave ratings of 4 or 5). Thus, any reallocation of effort to correct weaknesses or changes in priorities should try to protect these strengths.
Issues to be Mindful Of were judged to be effective or better by the majority of the respondents, but less than effective by a significant minority of respondents (25% - 39% ratings of 1 or 2).
Areas that Need Improvement were areas judged by a significant fraction of the faculty as unsatisfactory (40% or more 1 or 2 ratings).
On every question over three-fifths of the random sample (60% - 88%) and over four-fifths (81% - 92%) of the knowledgeable respondents gave the provost ratings of 3, 4, or 5. In none of the samples did the provost receive a rating of Needs Improvement (40% or more 1 or 2 ratings), with the exception of one question in the responses of the random sample.
The committee defined these to be areas in which 60% or more of the respondents gave ratings of 4 or 5. The provost received ratings in this category on 10 of the 18 questions in the random sample and on 15 of the 18 questions in the knowledgeable sample.
Both the random and knowledgeable samples rated 9 issues as strengths. The highest ratings among these have to do with the treatment of faculty and personal character:
·
Listening
to faculty, staff and students
·
Treating
all faculty fairly and inclusively
·
Acting
with integrity
·
Creating
an atmosphere of trust
Support for research and teaching as well as good communication and management skills were also rated at this level in both samples:
·
Supporting innovation in research
·
Supporting innovation in teaching
·
Managing conflicts among faculty, staff and
students
·
Communicating openly and honestly with
faculty
·
Working with faculty on new programs
Making decisions in a timely fashion was rated as a
strength in the random sample but not the knowledgeable group.
An additional six areas were rated as strengths by the knowledgeable group but not the random sample.
·
Appropriately involving faculty in decisions
·
Leading CU-B to an equitable salary system
·
Involving shared governance in major campus
issues
·
Securing funds for faculty compensation and
recruitment
·
Promoting the research mission to outside
constituencies
·
Communicating the standards for promotion and
tenure
Assets to Protect (50% - 59% more
ratings of 4 or 5)
Five of the areas which showed as strengths in the knowledgeable sample fell into the category of assets in the random sample. These were:
·
Appropriately involving faculty in decisions
·
Leading CU-B to an equitable salary system
·
Involving shared governance in major campus
issues
·
Securing funds for faculty compensation and
recruitment
·
Promoting the research mission to outside
constituencies
In the knowledgeable sample, Recognizing the importance of non-tenure-track faculty and Making decisions in a timely fashion were also rated in this category.
Issues to be Mindful Of
(25% - 39% ratings of 1 or 2)
In the knowledgeable group,
none of the questions fell into this category.
In the random sample, 4 questions received 28% - 37% 1 or 2
ratings. A division of opinion
concerning two of these is suggested by the fact that they appear in the
category of Assets above as well, since they received a higher percentage of 4
or 5 ratings. These two areas are:
·
Leading CU-B
to an equitable salary system
·
Securing funds
for faculty compensation and recruitment
The remaining questions
that the random sample rated as issues to be mindful of are:
·
Recruiting and
retaining diverse faculty, staff and students
·
Communicating
standards for promotion and tenure
The knowledgeable sample never placed the performance of Provost DiStefano in the “Needs Improvement” category. In only one case, Recognizing the importance of non-tenure-track faculty, did it fall narrowly into this category in the random sample.
COMPARISON TO BFA SATISFACTION SURVEY RESULTS
Since the Satisfaction Survey focused on conditions of the university as a whole, it may offer a helpful context in which to consider responses to administrators as well. This comparison may be especially telling in regard to a provost, a position which is in many ways associated with the state of the university overall. For instance, two of the questions on the Satisfaction Survey with the lowest mean (average) scores were about compensation: Salary compared with peer institutions and equitable distribution of salary, with mean scores of 2.2 and 2.6 respectively. (We use mean scores for ease of comparison between the Satisfaction Survey and the appraisal of Provost DiStefano.) Two of Provost DiStefano’s lowest ratings in the random sample also came in response to questions about compensation, Leading UCB to an equitable, merit-based incremental salary system, and Securing funds for faculty compensation and retention.
While these ratings point to the importance of continued examination and revisiting of faculty concerns, the Satisfaction Survey also suggests that discontentment about salary might reflect a systemic issue stemming from inadequate university resources in general. It is also noteworthy that even in this area of overall concern, the provost’s means of 3.3 and 3.4 on the two questions cited above are much higher than those given to the university as a whole. It may be useful to note that the knowledgeable group gave him exceptionally high mean ratings on the two questions about salary, 4.0 and 3.8.
Similar results for the Satisfaction Survey and the provost’s survey were observed in another area that had some of the lowest ratings for both surveys, revolving around university efforts to recruit and retain diverse faculty (Satisfaction Survey mean=2.9) and students (Satisfaction Survey mean=2.5). The means for the provost on the single item concerning this issue were comparable to those of the Satisfaction Survey for both the random sample (2.9) and the knowledgeable group (3.3).
COMPARISON TO PREVIOUS AAP SURVEY
When an administrator has undergone the AAP in a previous review year,
the committee tries to provide a comparison of significant high and low points
as well as a general overview. In the
case of Provost DiStefano, there are several challenges in doing so. One is that the response rate of the random
sample was only 38% in the earlier appraisal, not high enough to be reliably
representative in the view of the committee and the BFA. While the knowledgeable sample had a return
rate of over 70%, it was constituted quite differently in the previous
AAP. Since we had only targeted the
chairs and directors of A&S last time, due to the evaluation of the dean of
A&S occurring simultaneously, we had tried to give a cross-campus view by
including those schools which had responded with over 60% in that category as
well (as we explained at the time).
Another factor is that the previous AAP, in AY 2000-01, evaluated the performance
of Philip DiStefano as Vice Chancellor of Academic Affairs rather than as
Provost. The positions have many areas
of overlap, and the questions are for the most part the same, but the
difference should be noted.
With all those factors taken into account, a comparison nevertheless
suggests that the satisfaction level registered in regard to DiStefano’s
earlier appraisal has risen. In the
2000-01 evaluation by the random sample, for instance, only one area was rated
as a Strength (60% or more ratings of 4 or 5), and five as Assets (50% - 59%
ratings of 4 or 5). In the 2003-04
evaluation by the random sample, ten areas are rated as Strengths and five more
as Assets, with only 3 of 18 questions receiving less than 50% ratings of 4 or
5.
Despite the higher percentages of positive ratings this
year, four of the five questions which were viewed by the random sample as
Issues to be Mindful Of in the 2000-01 appraisal remained in the same category
again for 2003-04, with one area now moving into the range of Needs
Improvement. (In the 2000-01 evaluation,
no areas were placed in this category). These were Leading CU-B to an equitable salary system, Securing
funds for faculty compensation and recruitment and Recruiting and retaining diverse faculty, staff
and students, with the fourth—Recognizing
the importance of non-tenure-track faculty—falling narrowly into the category of Needs Improvement.
CONCLUSIONS
Many of Provost
DiStefano’s ratings fell into the category of Strengths (60% or more 4 or 5 ratings). This finding occurred for both the random
sample (10 of 18 questions categorized as Strengths) and the knowledgeable
group (15 of 18 questions). This latter group, selected for their familiarity
with the provost’s work, had a response rate of 79%, and the random sample had
a rate of 61%. The AAP Committee considers the response rates of both groups
sufficient to produce reliable survey results.
The remaining areas had
high percentages of positive ratings.
However, the fact that four of them also fell into the category of
Issues to Be Mindful Of (25% - 39% ratings of 1 or 2) and one of them into that
of Needs Improvement (40% or more 1 or 2 ratings) suggests a division of
opinion in regard to these areas among the campus at large as compared with the
knowledgeable population (in the latter no areas fell into these
categories). Four of these areas were
among the five Issues to Be Mindful Of in DiStefano’s earlier AAP evaluation in
2000-01. These questions focused on funding issues such as equitable salary
distribution and securing funds for compensation and recruitment; recruiting
and retaining diverse faculty, students and staff; and recognizing the
importance of non-tenure-track faculty, the last of which narrowly fell this
time into Needs Improvement. Most of
these are areas which tend to receive the lowest ratings of the Satisfaction
Survey, in which faculty evaluate the university as a whole. The additional question in the Issues to Be
Mindful Of category in the 2003-04 appraisal was new to the survey this year,
and concerned faculty confidence in the fairness of the promotion and tenure
process.
There was widespread
faculty agreement concerning the high level of Provost DiStefano’s performance
in all other areas addressed in the survey.
These included communication and management skills; collaborative work
with faculty such as shared governance and general decision-making; support for
research and teaching missions; and considerations of personal character, such
as integrity and the ability to create an atmosphere of trust.
[1] The members of the committee are Chair Cathy Comstock, Co-Director, Farrand Academic Program; Professor Chick Judd, Psychology; Professor Wayne Gazur, Law; Jeff Schiel, Assistant Director, Office of Planning, Budget, & Analysis; Professor James Palmer, Film Studies; Dean Anne Heinz, Continuing Education; Professor Betty Jackson, Leeds School of Business; Professor Richard Laver, Mathematics; Professor Judith Glyde, Music; Professor Bruce Henderson, Journalism.
[2] Faculty are defined as all members of the Boulder Faculty Senate, including all rostered contract instructors and adjoint faculty.