Every term, students at the University of Colorado Boulder, University of Colorado Colorado Springs (UCCS) and University of Colorado Denver evaluate their courses and instructors using a questionnaire called the Faculty Course Questionnaire (FCQ). In summer 2016 the three campuses came together to update the current FCQ question sets and also move away from the traditional paper and pencil method and toward a more sustainable method of administration.
The redesign team completed the first pilot in fall of 2016. As part of this effort, the team tested two new instruments to potentially replace the present FCQ and compared results against the traditional paper and pencil method. They also gathered information from pilot faculty and students, learned more about the vendor and its tools and made adjustments for the upcoming spring 2017 pilot.
Pilot administration, response rates and instruments
Across all three campuses more than 40 classes and 2,500 students participated in the FCQ fall pilot. Average online pilot response rates for these courses varied between 41 and 67 percent, while response rates for the traditional FCQ varied between 65 and 74 percent. The project team’s analysis of response rates has shown considerable spread in class-specific response rates with some classes hitting 80 percent. More detailed data about the pilot is available on the FCQ Redesign Project website.
After testing two different instruments as part of this pilot, the project team has decided to eliminate one of the instruments from consideration because of user experience and other issues. In the second pilot, which is underway this semester, the project team will focus exclusively on the other instrument, which received a more positive response from participants.
Steve Vassallo, assistant vice chancellor for integrated planning and decision support in the Office of Data Analytics, says this iterative approach will continue until an instrument and process are refined to a point that they can be used by all faculty across the CU system.
“The project team is using these pilots to test out different instruments and processes and then talking to faculty and students about their experiences,” Vassallo said. “We will continue iterating on the previous round until we have an approach that the team is confident in.”
Vasallo explained additional actions are underway to ensure the online response rate achieves levels consistent with the more successful class-specific response rates. In particular, instructors whose classes showed response rates higher than 60 percent were more likely to have given in-class time for response and/or utilized micro-incentives.
“We need to learn from those faculty who implemented the tool successfully last fall and replicate what they did in other classes.”
Additionally, during the first pilot, some pilot courses received both the traditional paper FCQ and the online tool. In contrast, other classes received only the online pilot, which greatly improved response rates. During spring 2017, only the online questionnaire will be administered to pilot courses.
What’s next for the FCQ Redesign Project?
Follow-up surveys were sent to pilot faculty and students asking for feedback regarding different aspects of the pilot administration. This feedback has been considered by the redesign committee alongside the pilot results and will inform the spring 2017 pilot both in terms of process and instrumentation. For example, the project team learned of the need to provide more explicit instruction on how to administer the pilot FCQs so as to ensure best practices are followed.
The FCQ will continue to be administered in its current form for the coming semester but will likely change to an online format within the coming year.
“There is a real desire to move to a more sustainable method of administering the FCQ,” Vassallo Said. “But the project team is not willing to do so until we know we have a tool in place that will work, which means getting acceptable response rates.”
If you have questions or feedback about this initiative, please email fcqredesign@colorado.edu.
