Report of the Boulder Faculty Assembly

Ad-Hoc Committee to Review Faculty Course Questionnaires

Submitted by

Martha Hanna, Chair of the Ad-Hoc Committee

September 24, 2003

Summary

The following report summarizes the key findings and recommendations of the Boulder Faculty Assembly (BFA) Ad-Hoc Committee to review the faculty course questionnaire and its use on the Boulder campus.  Based on a relatively high response rate to a BFA Question of the Month it appears that respondents believe that 1) the current FCQ form lacks credibility with the faculty, 2) the current form is susceptible to abuse by students, and 3) the FCQ process can lead to grade inflation.  A representative of the BFA attended a workshop on faculty evaluation and assessment systems and returned with these observations.  1) Our current form does not ask enough questions for reliable conclusions. 2) Our current form does not have a clearly defined response scale.  3) Our questions are too broadly framed to make consistent conclusions.  4) Our FCQ responses can put some faculty who consistently teach lower level courses in math and science at a disadvantage when compared to their colleagues in the social sciences and humanities.

            The Ad-Hoc committee has designed an alternative FCQ survey that attempts to conform with the findings of the scholarly literature and better meets the Regents’ objectives as stated in Policy 4-B (See Appendix C.)  It is the committee’s recommendation that this form be reviewed by the BFA for consideration of formal adoption.


Table of Contents

                                                                                                            page

Overview                                                                                              1

Background and Findings of the CEDA Workshop               2

Committee Deliberations                                                                    4

Question-of-the-Month:  Summary and Findings                               7

Conclusion                                                                                          17

Appendix A:                                                                                        20

Alternative Faculty Course Questionnaire

Appendix B:                                                                                       21                  

Boulder Faculty Assembly Question of the Month

Summary of Responses: FCQ Topic

Appendix C:                                                                                       25

Regents’ Policy 4-B


Overview

In October 2002, the Executive Committee of the Boulder Faculty Assembly authorized the creation of an ad-hoc committee to review the Faculty Course Questionnaire (FCQ) that is currently used on the Boulder campus.  The ad-hoc committee, chaired by Martha Hanna, Vice-Chair of the Boulder Faculty Assembly (2002-2003), included faculty, graduate student, and undergraduate student representation.  It met weekly through the Spring 2003 semester, and those members of the committee who were available to meet during the summer term continued to do so.  The following report summarizes the key findings and recommendations of the committee, based on its review of relevant scholarly analyses of course evaluations; the deliberations of the committee; and a careful analysis of responses to a BFA Question-of-the-Month survey.  Our most important findings are:

1.      The current form lacks credibility among the faculty.  Very few of the faculty who responded to the Question-of-the-Month survey believe that the FCQ in its current form adequately meets the objectives defined by the Board of Regents in their policy 4-B, adopted April 1986.  Teacher-course evaluations were first introduced on the Boulder campus in 1975.  

2.      The current form is susceptible to abuse.  Although faculty recognize that only a minority of students use the FCQ forms to express abusive comments, more than half of all faculty respondents had received abusive comments on the FCQ forms. 

3.      Faculty who teach courses in subject matters that students might categorize as “non-traditional” or “out of the main stream” are especially susceptible to abusive comments.  Faculty who teach about ethnic diversity, homosexual identity, or left-wing politics (to cite the most frequently mentioned cases) indicate that some students who are uncomfortable with any political, sexual, social, or ethnic identity different than their own express their hostility in personalized written attacks on the instructor.  More women faculty than men commented on their students’ intolerance in this regard.

4.      Faculty believe that the FCQ contributes to grade inflation.  Many faculty respondents believe that students retaliate for low grades received in a course by giving the instructor a very low rating.  Although individual respondents indicate that they have remained strict graders, they fear that other faculty members give high grades in order to receive high evaluations in return. 

Background and Findings of the CEDA Workshop

Two circumstances contributed to the creation of the ad-hoc committee.  (1) In March, 2002, Martha Hanna participated as a representative of the Boulder Faculty Assembly in a two-day workshop organized and run by the Center for Educational Development and Assessment (CEDA), a private consulting company that specializes in the study of systems of faculty evaluation and assessment; (2) members of the BFA Committee on Women learned that female faculty in the Business School had received sexually-explicit, derogatory remarks on FCQs submitted in Spring 2002.  This discovery, combined with key findings from the workshop, suggested that the FCQ format used on the Boulder campus might be significantly flawed. Key points emphasized in the workshops, as they relate to and touch upon the system of faculty evaluation in place at CU-Boulder, include: 

1.      At least 90% of course evaluation forms currently in use at American universities are not scientifically designed and are, therefore, less than reliable guides to teaching competence.  A well-designed form should ask only those questions that students are in a position to answer; it should not ask questions that the students are ill-qualified to answer, such as “Is the teacher familiar with the latest developments in his/her field?”  Nor should it ask questions that are multi-faceted and thus might not elicit clear responses (“Does the faculty member speak clearly and start lectures on time?”).  Furthermore, a well-designed form should ask enough questions about any given topic (clarity of presentation, fairness of grading system) to elicit a statistically reliable response.  At least 4 questions per category need to be included for a statistically reliable response, and they need to be written in such a way that a “D” answer might in some cases be a strong endorsement of the teaching and in others a strong criticism of it.  This guards against the possibility that a student will categorically mark nothing but “D” or nothing but “A”.  The current form in use on the Boulder campus does not ask enough questions in each designated category to provide statistically reliable responses.

2.      Course evaluations should have a clearly defined scale.  Each point on the scale should be defined, not just the two end points.  CU’s FCQ does not have a clearly defined scale.  Only the end-points are defined, where A is “very good” and F is “very poor.”  There are at least two problems with this:  it uses letter grades (rather than “Very Effective, Effective, Moderately Effective, Somewhat Ineffective, Ineffective, for example), and it leaves the middle points undefined.

3.      Course questionnaires should ask specific, narrow questions from which global assessments can be deduced.  Broadly framed questions, such as questions #11 (“This course, compared to all your other university courses”) and #12 (This instructor, compared to all your other university instructors”) on the CU form, usually do not produce a response consistent with the responses given in the more narrowly defined questions.  This means that students can, and often do, give inconsistent responses.

4.      There are two documented ways in which course evaluation results are skewed: lower-level classes, regardless of class size, consistently produce lower ratings; and science and mathematics courses produce lower ratings than humanities and social sciences courses.  Thus faculty who consistently teach nothing but lower-level courses might be at a disadvantage if their teaching is judged only on FCQ scores.  So, too, for faculty in science, engineering, and mathematics if their scores are compared with those of their colleagues in the social sciences and humanities.

Committee Deliberations

Several important ideas and considerations emerged from the committee’s discussions:  

1.      Confidentiality of responses is not always observed/honored, and students have concerns about being chastised for legitimate concerns mentioned on the form.   At the same time that we have to worry about students using the veil of confidentiality to make abusive comments, we also have to be concerned that students are not penalized or intimidated when the confidentiality of their responses is not protected.  This is more likely to be a serious problem in small classes where it is easy to identify a student's handwriting, and in graduate courses where a student might take several courses consecutively from the same professor.

2.      Timing of the FCQs might contribute to hostile comments.  Students feel especially stressed in the last week of classes, and if they already have a good sense of what their final grade might be for the course, they can take their frustration out on the faculty member.

3.      Students are not fully aware of how FCQs are used in assessing teaching and allocating merit points.  It might be appropriate to impress upon students the importance of the process, and thus call upon them to participate in a more serious and reflective manner.  Some faculty were concerned, however, that hostile students would be even more inclined in such circumstances to use the FCQ as an instrument to punish a faculty member whom they do not like or, conversely, to reward a faculty member whom they do like, regardless of the quality of instruction provided in the course.

4.      On-line evaluations might prompt students to be more thoughtful in their comments.  They could respond to the questionnaire at times that are more convenient for their schedules, and hence might be more likely to respond in a considered and thoughtful manner.   However, because logging on to the student's account requires giving an identification number, this procedure might compound the confidentiality problem identified above.  Although students might be less likely to write abusive comments if they think their comments can be traced, they also might be less likely to say anything.  (Unfortunately, anecdotal evidence has not substantiated the belief that on-line responses might be more civil than responses given on paper.  While the response rate can be higher for large courses, say those over 100 students, several instructors who teach large lectures claim that students’on-line comments are still consistent in tone with the in-class FCQ comments.  Furthermore, some instructors object to making FCQs available on-line to students who rarely attend class, which is the case with many large lower level courses.)   Given that the LETTS committee, representing all four campuses and chaired by Professor Richard Bakemeier, will be analyzing the advantages and disadvantages of using on-line surveys during the course of its deliberations this year, the BFA ad-hoc committee believes that it would premature and inappropriate to make a recommendation at this time on this particular issue.

5.      There was a general consensus that we should move away from the A - F scale currently used on the forms.   The sense of the committee on this point coincides with the scholarly literature cited at the CEDA workshop and was reiterated by several faculty respondents to the BFA Question of the Month who believe that the use of letter grades indirectly contributes to grade inflation.   Students who believe that they are “grading the instructor” rather than evaluating the course often give the instructor the same grade on the evaluation as they expect to receive for the course. 

6.      The ad-hoc committee believed that it would be appropriate to propose an alternative form the structure of which would be consistent with the findings of the scholarly literature.  This form would have more questions than the current FCQ, would identify several categories of questions that fall within students’ legitimate competence, and would ask several questions in each category with a balance of negative and positive questions.  The committee spent several meetings reviewing questions (drawn from the scholarly literature on designing a credible course questionnaire) that would meet these criteria for acceptability.   Recognizing that students have legitimate concerns and interests in the evaluation process and that they can speak with authority about several aspects of the classroom experience, the ad-hoc committee explicitly drew upon the opinions and experience of the undergraduate and graduate students on the committee in creating an alternative survey. Our proposed survey is found in Appendix A.

7.      The committee would use the BFA Question-of-the-Month procedure to poll instructional faculty on their sentiments, judgments, and experiences with the current FCQ.   The Executive Committee of the BFA approved the  ad-hoc committee’s request to submit a Question-of-the-Month to the faculty and this survey went on-line in April 2003. 

Question-of-the-Month:  Summary and Findings

The ad-hoc committee believed that it would be valuable to survey the instructional faculty (including tenure and tenure-track faculty, non-tenure track faculty, and graduate student instructors) to determine whether the faculty believe that the FCQ as currently designed and implemented meets the objectives defined by the Board of Regents.   To this end, the committee submitted a draft “Question-of-the-Month” to the BFA Executive Committee and revised the questions to be included on the survey in response to recommendations from members of the Executive Committee.  Although some faculty respondents to the Question-of-the-Month would subsequently object that the survey was not well designed, the overwhelming majority of the Executive Committee approved the questionnaire.  The Question-of-the-Month asked the following questions:

Question #1:  The first objective defined by the Regents is that FCQs “should help improve instruction and student learning.” What is your assessment of the degree to which the FCQs have helped you improve your classroom instruction?      

A)    Not Met at All

B)     Largely Unmet

C)    Partially Met

D)    Largely Met

E)     Fully Met

Question #2: The third objective defined by the Regents is that FCQs should “support the faculty evaluation process and faculty rewards system.” Based on your experience, how well do the FCQs meet this objective?

A)    Not Met at All

B)     Largely Unmet

C)    Partially Met

D)    Largely Met

E)     Fully Met

Question #3:  Faculty have expressed concerns that in some instances the FCQs lend themselves to the expression of disparaging and abusive comments. Have you experienced disparaging remarks on the FCQs?

Yes or No

Two-hundred-ninety-one instructional faculty responded to the Question-of-the-Month.  Of these, 43 were full professors (6 female, 34 male, 3 undefined),  28 were associate professors (13 female, 15 male),  39 were assistant professors (22 female, 15 male, 1 undefined), 13 were senior instructors (9 female, 4 male), 26 were instructors (19 female, 7 male), 105 were graduate student instructors (either GPTIs or Teaching Assistants) (58 female, 46 male, 1 undefined), and 37 indicated their rank as “other” or “none” (15, 10, 3) (2, 2, 5).

Responses to Question #1:  90% of all respondents felt that the FCQ did not meet, or only partially met, the Regents’ objective that the FCQ should “help improve instruction and student learning.”  This percentage appears consistent for each faculty rank and for male and female instructors across ranks.  Skepticism about the usefulness of the FCQ in improving teaching seems especially marked among tenured and tenure-track faculty and Senior Instructors.  Thus 57% of Full Professors, 50% of Associate Professors, 58% of Assistant Professors, and 70% of Senior Instructors responded to this question by choosing either answer A (Not Met at All) or Answer B (Largely Unmet). When broken down by gender, 48% of female faculty respondents, 50% of male faculty respondents, and 58% of those who chose not to identify themselves by gender selected either (A) or (B).

Responses to Question #2:   86% of all respondents felt that the FCQs did not meet, or only partially met, the Regents’ objective of supporting the evaluation process of faculty and contributing usefully to the faculty rewards system.  A substantial number of  faculty respondents, regardless of rank or gender, selected either answer (A) Not Met at All or  (B) Largely Unmet to this question:  63% of Full Professors, 50% of Associate Professors, 49% of Assistant Professors, 77% of Senior Instructors, and 69% of Instructors.   When broken down by gender, 58% of female faculty respondents, 60% of male faculty respondents, and 58% of those who chose not to identify themselves by gender selected either (A) or (B).

Taking responses to Questions 1 and 2 together, it becomes evident that for those faculty members who chose to respond to the survey, there is a substantial credibility problem with the current FCQ:  a majority of faculty do not believe that it either helps to improve teaching and student learning or supports the evaluation process of faculty.  

Responses to Question #359% of all respondents have experienced abusive comments on the FCQs.  When analyzed by rank, 68% of full professor respondents, 71% of associate professor respondents, 61% of assistant professor respondents, 85% of Senior Instructor respondents and 65% of Instructor respondents had received abusive comments on the FCQs; in only the Graduate Instructor category had fewer than 50% of all respondents (46%) received abusive comments.   Female faculty members are more likely to experience disparaging remarks on their FCQs than their male peers:  65% of female faculty respondents indicated that they had received abusive comments versus  51% of male respondents.  Some interesting differences emerge when the data are analyzed by gender and rank combined:  for example, 100% of the female respondents who are full professors have received offensive remarks compared to 62% for male full professors; similarly, 100% of female Senior  Instructor respondents have received offensive comments on the FCQ, whereas only 50% of  male Senior Instructor respondents have done so.  The majority experiencing insulting comments on the FCQs and the differences between male and female faculty members are shown in the charts in Appendix B.  Respondents were also given an opportunity to describe the nature of the abusive comments and the impact such comments have had on their attitude towards teaching.   What follows is a summary of dominant issues identified in these comments. 

1.  Faculty members believe that students who are disgruntled, for whatever reason but usually because they have received low grades, are prone to use the FCQ to “get back” at the instructor.  One respondent noted:  “In order to get even with faculty for undesirable grades, students include disparaging remarks and give very low ratings.”  Another observed:  “I have only received disparaging remarks on maybe one out of 40 FCQs, but they almost always correspond to grading.  Consequently, I believe that the current FCQ offers unhappy students an opportunity to “vent” rather than to offer helpful comments.”   This female Instructor concluded that “positive comments usually state what aspects of a course were most interesting or helpful, negative comments rarely contain specific or useful suggestions.  Perhaps their only value is in learning to acquire a thick skin.”  A male Full Professor expressed the opinion of many respondents when he noted:  “There is a tendency among some students to trivialize the process and to retaliate for poor exam scores by nasty remarks.” Graduate student instructors expressed similar sentiments.  Many implied that they had received negative comments as a form of revenge for disagreements over grading or because they were not an “easy A.”

2.  Because many faculty believe that students use the FCQ to retaliate for low grades, many fear that the FCQ process, with its explicit use of “grades” on the questionnaire, contributes to grade inflation and a general erosion of academic standards.  One respondent (Full Professor, Male) wrote:  “My biggest concern is that too many professors have told me that they hand out high grades in the hope of receiving high grades from students in return.  This movement away from honesty and the maintenance of standards is a slide down to mediocrity.”  He concluded that we should abandon the process by which students are, in effect, “grading professors.” An assistant professor made a similar point:  “I think the answers should NOT be structured in the “A,” “B,” “C,” format, as this can lead to abuse or to paying the instructor back.  I feel strongly about this.” Yet another faculty member  noted that students who had received poor grades from him in a previous class gave him straight “Fs” in a subsequent course as a “way of payback.  This was alluded to in the comments and verified by word of mouth.  It is difficult to receive high FCQs if you are known to be a tough instructor.”  Another senior faculty member made a similar point: “The students resent my rigor in grading and resort to gutter language to express their unhappiness.  The FCQs have led to horrendous grade inflation.  They have outlived their usefulness, and the time has come for them to be discontinued or radically changed.”  An Associate Professor commented that he had “been called everything in the book on FCQs” (a gutter language was part of the lexicon).  He believed that “FCQs provide the poor students a chance to vent after having blown off class for half a semester.  I have also been threatened in FCQs.”  Such offensive remarks “made [him] even more of a disciplinarian in class . . .  On day one I tell students that this is the hardest class they will ever have . . .   Usually, 2-3 get up and leave immediately, but there are always 2-3 who think I am joking, then end up writing mean-spirited FCQs.”  Graduate student instructors admit that the existence of the FCQ puts pressure on them to accommodate students who expect high grades, extensions on assignments, or other concessions.  One student-instructor wrote:   “Whenever I have to give a low grade on a quiz or not allow a student to turn in an extremely late assignment, I wonder if that is going to mean that I will have to read some disparaging comments on the FCQ.”  Another confessed:  “They have…made me feel that I need to appease my students more than I need to convey information on the course.  I have not done this, but they make me question whether I should.”  Whether or not the FCQ does contribute to grade inflation, the belief that it does exists, and thus undermines the credibility of the process in the eyes of instructional faculty. 

 3.   Instructional faculty of all ranks note with frequency that they have received negative comments about their physical appearance.    In many instances, these are remarks about clothing or hair style; in the most extreme cases, they are lewd remarks about the instructor’s body.  Negative remarks about clothing are, it seems, commonplace and both men and women note that they have received comments criticizing the way they dress or do their hair.  Explicitly sexualized comments seem to be directed only at female professors.  One Senior Instructor indicated that she had received “sexist remarks about the kind of clothes” she should wear; comments of this type made her hesitant to read the FCQs “because I’m concerned they will be of an inappropriate personal nature.”  Another Senior Instructor wrote:  “I have been called a bitch by at least one student every semester.  I have had students tell me to fuck off more than one semester.  I have been told to get breast implants.  I was called a dyke and a lesbian.”    An Assistant Professor noted that students made comments about her legs; an Associate Professor received at least one remark “about my sexual availability, i.e. the lack thereof.”  Some students make racist comments or comments criticizing a faculty member for his/her accent.  One tenure-track faculty member, who has lived in the United States for more than twenty-five years and speaks English fluently, noted that some students used the fact that she spoke with a slight accent “as the basis of some very nasty attacks on my teaching ability.” 

4.  Faculty who teach courses on subjects that some students judge to be out of the main stream receive comments that suggest these students are very uncomfortable with any political, sexual, social, or ethnic identity different than their own.  This is true of faculty who teach about ethnic diversity, homosexual identity, or left-wing politics.  The junior faculty member who received “sexist ‘banter’ along the lines of ‘nice legs’ (and less printable comments)” also had students criticize her political viewpoint (“I object to being taught by a socialist”); she concluded that such remarks made her “feel hostile to my students.”   An assistant professor who teaches Native American studies noted that she received “disparaging remarks [to the effect] that I am anti-white, anti-christian, or that I have no business being a teacher.”  She concluded that “the remarks lead me to conclude that the students are partially judging me on my ethnicity, gender, and/or appearance and not on my abilities. . . .  Such remarks so offended me  . . . that I no longer read them.” A tenured faculty member, criticized for including queer studies material in her syllabus,  received such “virulent responses” as “Professor imposes her disgusting sexuality upon us.”  Remarks of this type prompted her not to teach this particular class as a large lecture course again and made her grateful for the fact that she had tenure.   Yet another tenured, female faculty member, who had received negative comments on her syllabus (which included a consideration of ethnicity and race) ended up “lowering [her] expectations on what issues of diversity, particularly in relation to race, could be safely discussed in the university classroom.” More women faculty than men commented on their students’ intolerance in this regard.

5.  Among those faculty who had experienced negative/abusive comments, senior male faculty and some graduate student instructors seemed to be able to ignore or minimize the impact of such comments.   Female faculty of all ranks and junior faculty, male and female, were more likely to find such comments seriously demoralizing.  One male full professor noted that he had received comments about “accent, dress, age, opinions” but he felt able to “ignore them and laugh them off, though I know lot of colleagues (mostly female) who find them very offensive.”  Even though abusive comments are the exception rather than the rule in any given class, many faculty respondents found the remarks so depressing that they were less likely to want to read the FCQs in the future (thus eliminating any potential value the forms might have), were more likely to think poorly of their students, and in the most extreme cases were less likely to want to continue with a teaching career.  One assistant professor received “insulting and ill-informed comments” about his teaching style; and “a drawing of an alien with the words ‘die earthling.’”  He observed that such remarks “discouraged me from continuing teaching, and from continuing at CU, because I put so much work and time into teaching and receive disparaging, ill-informed, and sometimes insulting comments back on the FCQs.  I am currently looking for research jobs that do not include teaching .  .   . in part because a subset of the FCQs drive home the fact that teaching at CU is unrewarding and commonly discouraging.”  One female respondent who did not indicate her rank responded:  “They have made me take the FCQs less seriously.”  A female full professor indicated:  “they have made me hurt, angry, and resentful.  I would not make such comments to any of them.”  An associate professor recognized that most of her students “seem to be generally appreciative of my teaching style and base of knowledge” and only “one or two” wrote negative comments, covering topics as various as clothing style and professional competence.  Nonetheless these comments were very demoralizing:   “I find the comments very difficult to deal with.  They are distinctly damaging to my morale and my opinion of my students.”  Another faculty member, who admitted that “the abusive comments really sting,” was increasingly skeptical of the value of FCQs and had come “to question the reliance of the university on FCQs in evaluating teaching.”  An assistant professor noted that she had “received comments to the effect that I know nothing at all, and also comments on my appearance.”  Remarks about her intelligence “did not help at all and made me not want to teach!  The comments on my appearance were inappropriate.” Among graduate students, 62 of the 134 respondents indicated that they had received insulting, disparaging, or offensive comments.  More than one-third (22 of 62) indicated that the comments were of an inappropriate, personal nature:  these comments including everything from explicit references to the physical appearance, race, sexual orientation of the instructor to sexually graphic suggestions and, in one instance, violent remarks.  Some were able to ignore the remarks or not take them seriously.  Others, however, found such remarks discouraging, upsetting, or anxiety producing.  One respondent noted that “…barbed remarks from a few disgruntled students seem to obliterate those from the overwhelming majority of satisfied or grateful students.”

Several faculty expressed the concern that students are not held responsible for their remarks, whereas instructors must be responsible for the comments they provide on student papers, tests, etc. 

Conclusion

Any review of course evaluations must recognize that three distinct questions are embedded in the review process:  (1) What should the questionnaire ask? (2) How should it be administered (in class vs. on-line, for example)? and (3)  How should the information gathered from the questionnaire be used – and by whom – in the evaluation of faculty?  THE COMMITTEE MAKES THE FOLLOWING RECOMMENDATIONS ON THESE QUESTIONS.

Recommendation 1:  What should the questionnaire ask?   It is the committee’s recommendation that the set of questions included in Appendix A be considered for review by the BFA with the intention that the Boulder campus adopt a new form. THE COMMITTEE UNDERSTANDS THAT, IF APPROVED BY THE BFA, THE QUESTIONS WILL NEED TO BE REVIEWED AND APPROVED AT THE CAMPUS LEVEL.   THE COMMITTEE THEREFORE OFFERS THE FOLLOWING MOTION.

THE AD HOC COMMITTEE TO REVIEW FACULTY COURSE QUESTIONNAIRES MOVES THAT THE BOULDER FACULTY ASSEMBLY APPROVE IN PRINCIPLE THE SET OF QUESTIONS INCLUDED IN APPENDIX A OF THIS REPORT FOR USE IN PLACE OF THE CURRENT QUESTIONS ON THE FACULTY COURSE QUESTIONNAIRE.

RECOMMENDATION 2:  Given that the FACULTY COUNCIL’S LETTS committee is currently addressing the second question, our committee RECOMMENDS THAT THE BFA TAKE NO FURTHER ACTION ON WHETHER FCQS SHOULD BE ADMINISTERED IN CLASS OR ONLINE UNTIL THE LETTS COMMITTEE COMPLETES ITS DELIBERATIONS. 

Recommendation 3:  Faculty responses to the BFA Question of the Month suggest that there is room for improvement and clarity in how the FCQ RESULTS ARE used “to support the faculty evaluation process and the faculty rewards system.”  Faculty respondents do not believe that the current system effectively addresses this Regental objective.  To a certain extent this skepticism derives from the belief that the current FCQ is fundamentally flawed and thus cannot give an accurate assessment of classroom performance.   OUR FIRST RECOMMENDATION, IF ADOPTED, SHOULD ADDRESS THIS CONCERN.

Other procedural concerns need to be addressed, however.  Committee deliberations revealed that there is no uniform policy on campus relating to how the evaluations are to be used.  Who, for example, should have access to the written comments?  And how, if at all, should the written comments – as distinct from the quantitative data derived from the multiple-choice questions -- be used in evaluating faculty for merit-based raises, contract renewals, and promotion? 

The committee RECOMMENDS THAT a clear, readily available policy statement BE DEVELOPED, outlining precisely who has access to the narrative and quantitative data provided on the FCQs and how this information is to be used in the evaluation of faculty.   IN PARTICULAR, IT IS THE COMMITTEE’S UNDERSTANDING THAT THE WRITTEN COMMENTS ON FCQS ARE CONFIDENTIAL AND SHOULD BE ACCESSIBLE TO THE INSTRUCTOR OF RECORD ONLY, BUT ARE ROUTINELY REVIEWED BY MANY DEPARTMENT CHAIRS AND/OR DEANS NEVERTHELESS.   IF, INDEED, THE WRITTEN COMMENTS ARE CONFIDENTIAL, THEY SHOULD NOT BE AVAILABLE TO DEPARTMENT CHAIRS OR DEANS, NOR SHOULD THEY BE A FACTOR IN EVALUATING FACULTY FOR MERIT-BASED RAISES, CONTRACT RENEWALS, OR PROMOTIONS.  THE COMMITTEE RECOMMENDS THAT THE CONFIDENTIALITY OF WRITTEN COMMENTS ON FCQS BE VERIFIED BY THE CAMPUS ADMINISTRATION.  THE COMMITTEE THEREFORE OFFERS THE FOLLOWING MOTION.

THE AD HOC COMMITTEE TO REVIEW FACULTY COURSE QUESTIONNAIRES MOVES THAT THE BOULDER FACULTY ASSEMBLY WORK WITH THE CAMPUS ADMINISTRATION TO DEVELOP A CLEAR, READILY AVAILABLE POLICY STATEMENT OUTLINING PRECISELY WHO HAS ACCESS TO THE NARRATIVE AND QUANTITATIVE DATA PROVIDED ON THE FCQS AND DESCRIBING HOW THIS INFORMATION IS TO BE USED IN THE EVALUATION OF FACULTY. 

APPENDIX A: 

ALTERNATIVE FACULTY COURSE QUESTIONNAIRE

Interaction with and Attitudes towards Students. 

The instructor was too involved with lecture to be aware of the class.

The instructor motivates me to do my best work.

The instructor encourages students to ask questions.

The instructor is not available outside of class.

The instructor treats students with respect.

The instructor used student questions as a source of discovering points of confusion.

The instructor was concerned with whether or not students learned the material.

The instructor shows enthusiasm for teaching.

The instructor seemed genuinely interested in what he/she was teaching.

Course Presentation and Clarity.

The instructor presented material in a clear manner.

The course material did not appear to be presented in logical content units.

The instructor was well prepared for each class.

At times it was difficult to hear what the instructor was saying. [Upon further reflection would it be better to say "understand" rather than "hear"?  If you can't hear it you certainly can't understand it, but you might be able to hear and still not understand -- this touches upon the accent issue.]

The course emphasized comprehension over memorization.

The instructor created an interactive learning environment.

Grading, work-load, and assignments

The instructor's methods of evaluation were fair.

The instructor is consistent when dealing with students.

The assignments were returned in a timely manner.

Assignments helped me to understand the course material.

The instructor's expectations of the students were NOT clearly defined.

The workload required for this course was heavy.

Student Outcomes

I would not recommend this instructor to other students.

The course challenged me intellectually.

I learned a lot from this course.

I feel I performed up to my potential in this subject.

Open-ended questions:

What specific advice would you give that would help you improve your learning in this course?  Please address such issues as classroom environment, assigned readings, presentation style, etc.

What in your judgment works especially well in this course?

Appendix B

Boulder Faculty Assembly Question of the Month

Summary of Responses: FCQ Topic

The purpose of the online survey was to evaluate the FCQs from the faculty member’s perspective.  The survey assesses if the Regents’ objectives are being fulfilled through the FCQ process.  148 full time and 142 part time instructors responded.  There is an 11% response rate for full time instructors and 17% response rate for part time instructors.

Question #1:  The first objective defined by the Regents is that FCQs “should help improve instruction and student learning.” What is your assessment of the degree to which the FCQs have helped you improve your classroom instruction?

A)    Not Met at All

B)     Largely Unmet

C)    Partially Met

D)    Largely Met

E)    Fully Met


Total Response:

90 % of the total responses felt that the FCQ did not meet or only partially met the Regents’ objectives.  Although there were slight differences in response distributions by rank, this percentage appears consistent for each faculty rank group and for both males and females. 

Question #2: The third objective defined by the Regents is that FCQs should “support the faculty evaluation process and faculty rewards system.” Based on your experience, how well do the FCQs meet this objective?

A)    Not Met at All

B)     Largely Unmet

C)    Partially Met

D)    Largely Met

E)     Fully Met

 


Total Response:

86% of the total respondents felt that the FCQs did not meet or only partially met the objective of supporting the evaluation process of faculty and the faculty rewards system.  Throughout the different rankings of faculty members and gender differences the majority of the respondents replied that the FCQs did not meet or largely unmet this objective.  Distributions are heavily skewed right which also depicts respondents not supporting the evaluation process.

Question #3: Faculty has expressed concerns that in some instances the FCQs lend themselves to the expression of disparaging and abusive comments. Have you experienced disparaging remarks on the FCQs?

            Yes or No

Total Response:

 


Total Response:

59% of total respondents have experienced abusive comments on the FCQs. This percentage remains consistent throughout different faculty ranking and gender with the exception of graduate part-time instructors.  Only 46% of graduate instructors said they had received abusive or disparaging comments.  Also, female faculty members usually experience more disparaging remarks on their FCQs. For example, 100% of the female respondents who are full professors and senior instructors have received offensive remarks compared to 62% for male full professors and 50% for male senior instructors. 


Appendix C

Regents’ Policy 4-B

4-B. FACULTY-COURSE EVALUATION

Faculty-Course Evaluation shall be implemented at the University of Colorado for all courses and their sections offered by any of the University of Colorado campuses.  The faculty-course assessment and evaluation process shall be the responsibility of each campus.  The overall purpose of all assessments and evaluations should be to enhance learning.  Multiple measures will be used to assess and evaluate a faculty member's teaching.  Each campus shall design an evaluation form that meets that individual campus's specific needs so long as such forms are uniform for that campus, include evaluation of individual faculty, and are adaptable to the individual campus's research and testing services.  The evaluation system shall be designed to provide published information to students, faculty, departmental administration, and the University's administration to accomplish the following objectives:

1. Improve instruction and student learning (formative evaluation).  Use of formative evaluation is optional.  When used, this goal is to be achieved through the use of a formative (diagnostic) assessment early enough in the semester to afford useful written narrative feedback from students.  Numerical feedback shall not be required as part of the formative evaluation methods.  Students should be educated in appropriate assessment feedback procedures prior to conducting this and other assessments.  The results of formative assessments are confidential and available only to the faculty members.  Other possible methods to achieve this goal include:

·      independent observations

·      video

·      educational specialists

·      faculty/peer visitations

·      student/faculty focus groups

2.       Provide students with an evaluation of the course and the faculty member based on previous students' assessments (summative evaluation).  The numerical results of the student evaluation are public and available to students and others.

3.       Support the faculty evaluation process and faculty rewards system (summative evaluation).  This goal also is achieved through the use of a summative evaluation in conjunction with other means of teaching assessment and evaluation, including but not limited to the following techniques:

·         students' evaluations that are public

·         independent experts' observations

·         faculty/peer visitations

·         supervisor's observations

·         longitudinal assessment to measure year-to-year improvement

·         availability of remedial assistance to support improvements in   teaching effectiveness

Individual campus committees shall a4.se be established to oversee the design, implementation, and information distribution process of the Faculty-Course Evaluation for each campus.  The campus committees shall consist of three students appointed by the campus student Government, three faculty members appointed by the campus Faculty Assembly, one member from the campus testing and processing services or campus equivalent, and one member from the campus Chancellor's Office.  Student and faculty appointments will give special consideration to draw on the broadest representation among the schools and colleges on the campus.

The Chancellors shall be responsible for funding and providing a yearly operations budget.

(Adopted April 17, 1986, pp. 445-453., amended August 3, 2000, p. 11, attachment 1)