Instrument Methodology, Spring 2017

The faculty involved in the redesign project stated the desire to see the questions changed in three vital ways:

1. Questions should be focused on observable behaviors that students could evaluate
2. That questions provide specific, actionable feedback that would lead professors to practices they can change in their classrooms
3. To the extent possible, questions minimize the potential for bias based on race, gender, disability, and other variables

Studies of student evaluations vary in their findings (c.f. Braga, Paccagnella & Pellazari, 2014; Cashin, 1995; Weinberg, Hashimoto & Fleisher, 2009). But one trend is consistent for student evaluations of teaching effectiveness: that students have a difficult time referencing their learning against that of their peers. For instance, with a question such as, “Did you learn a lot compared to other classes?” there is a problem both of referencing “a lot” (which could have different meanings to different students) and that students will inevitably have had diverse classes and experiences (c.f. Braga, Hashimoto and Fleisher, 2009).

To avoid these problems, the FCQ committee sought to use or manipulate questions designed from previous sources that focused more on observable classroom teaching behaviors. For instance, instead of asking a student if they learned a lot, the new FCQ asks:

• I was encouraged to reflect on what I was learning
• I was encouraged to evaluate arguments, evidence, assumptions and conclusions about key issues

These questions are more focused than a general learning question and are intended to provide professors with meaningful information about where they can improve their classroom practices. 

In general, the new FCQ questions were created from questions asked by other institutions, from the National Survey of Student Engagement, and from questions asked in previous research studies. A list of references can be found on the redesign website. Further, the most recent version of the FCQ was reviewed by the Faculty Assemblies on each campus and by the CU Boulder Discipline Based Educational Research (DBER) group — an interdisciplinary group of professors who seek to improve STEM-based educational practices through research.

After the questions were designed, faculty on each campus helped the redesign team to test the new instrument and process over the course of two semesters. The results of each test, were used to revise and modify the questions. A list of the spring 2017 pilot courses and faculty (Excel) can be found on the redesign website. Over 4,000 students (unduplicated) received the new FCQ during this testing process. Each student was asked to provide their impressions of the new tool. Further, pilot faculty were either surveyed or interviewed about the experience with the new FCQ.

Based on analysis (conducted by CU Boulder’s Institutional Research department located in the Office of Data Analytics as well as faculty from all three campuses that included feedback from pilot course professors and students), the questions on the FCQ have been changed and refined multiple times.

The project is committed to continuous improvement and therefore will continue to analyze the questions and examine feedback provided over the next year. It is possible that further changes will be implemented over time and that the FCQ instrument will continue to improve as it evolves.

Further information about the process of the new FCQ, the technology of the new FCQ and suggestions for how the new FCQ should be used can be found on the FCQ redesign website. Further, each campus has provided its own resources that are geared towards helping professors improve their classroom practices and incorporate formative assessment practices that utilize campus technology, such as surveys, Learning management Systems (LMS) and observations. Professors are encouraged to seek other means of improving their classroom practices that will bolster the feedback provided by the FCQ.