Collaborators: Sricharan Reddy Varra, Benjamin Vernon, Irina Wagner, Emilie Young, Chris Zimmer
The feedback card in Buff Portal monitors the pulse of the student experience. Since Buff Portal's beta release in March 2019, the User Experience Research & Design team (UXRD) has received over 14,000 feedback card responses from students. In the feedback card, students are asked to rate their satisfaction with Buff Portal on an emoji scale and leave a comment. The emoji scale helps gauge the overall satisfaction of students, while the text comments provide the Buff Portal team with more specific information about the experiences of our users - what’s working and what needs to be improved. While the text comments are the most useful data on Buff Portal’s usability, they are also the most time-consuming to analyze.
To alleviate the pressures of tackling comment analysis during times of high-volume feedback, OIT’s UXRD team has been seeking out ways to automate comment analysis using natural language processing approaches. In the fall of 2020, OIT’s UXRD team partnered with CU Boulder’s Laboratory for Interdisciplinary Statistical Analysis (LISA) for support with this task. The goal of our collaboration was to create a script that would automatically analyze the sentiments and key terms of the text comments at a given window of time. This would allow the Buff Portal team to quickly identify issues with the portal (or other CU tools) before a large number of students is affected.
We partnered with Sricharan Reddy Varra and Chris Zimmer, two statistical collaborators with LISA, to leverage AWS’s Comprehend tool and custom developed scripts in Python for natural language processing and sentiment analysis around key phrases. Using these approaches in the fall, the team was able to get to 70% precision in evaluating the feedback card comment sentiment on a positive to negative scale. We also identified a very promising path for categorizing and automatically assigning topics to the comments (e.g., “registration” vs. “grades”).
The Buff Portal team is thankful for this partnership with LISA, specifically, all of the hard work that Sricharan Reddy Varra and Chris Zimmer put into this project. We continue to iterate on the original Python script created by Sricharan Reddy Varra and we are excited to leverage this analytical approach to continue to improve how we support students and their technology experience at CU Boulder.