| Home |
| About the Program |
| Directory of Teaching Scholars |
| Communiqué |
| The Story of the President's Teaching Scholars Program since 2001: Retreat Reports |
| Research in Learning and Teaching |
| Outreach |
| Scholars' Research, Initiatives, and Studies |
| Teaching Scholars' Projects |
| PTSP Biannual Retreat Reports |
| PTSP Selection: Call for Nominations |
University
of Colorado |
Classroom Learning Analysis StudiesPresident’s Teaching Scholars Program Professor and President’s Teaching Scholar Clayton Lewis, Computer Science SummaryWe request funding for a research program of in-depth interviews of students and faculty, patterned on the Harvard Assessment Project (Light, 2001). Based on the experience at Harvard, interviews with students will reveal student behaviors associated with better and worse educational experiences and provide specific data and guidance for improved student learning environments, both inside the classroom and outside, including their bonding with CU-Boulder. This research interview program will provide feedback on the effectiveness of our programs for students, allowing schools and colleges to improve them. IntroductionThe Harvard Assessment Project is a program of in-depth interviews of a sample of Harvard undergraduates each year. While the interviews are open-ended, allowing students to reveal concerns and ideas not anticipated by project staff, they are structured so as to focus on matters whose importance has been indicated by the results from prior years. Over time, this approach has enabled Harvard to detect potentially interesting patterns in the responses from a few students, to explore these patterns in the whole sample in a following year, and then develop a sharp enough picture from a large enough sample of students to form the basis for action. CU System can profit from the same approach. While Harvard has published some of its own findings, it is obvious that there are big questions about how well these findings fit CU, with its different student body and environment. Here are some examples of issues our project could address: *The Princeton Review rated CU Boulder, for example, the Number One Party School in the US in 2003 and Number 9 in 2004. Is this an accurate reflection of our students' experiences? What do our students think about the relationship between effort and accomplishment in their lives? How much work do faculty believe students must do to do well in their courses? How do students make friends and how important is that to them? How many students meet faculty in their offices to enhance their learning and thereby make acquaintance and become mentored by the faculty member? *"Clicker" response units support a radical and very promising change in the way large lecture courses are taught. What do students think about them? Anecdotally, some students fail to understand the motivation for clicker use, and resent it. *Alumni fundraising is becoming increasingly important in public higher education. What are student attitudes about their future role in this? *Harassment of faculty by students continues to be a problem. What student and faculty attitudes contribute to this problem? This list of issues could be extended indefinitely. Whenever a problem surfaces, and whenever a new initiative is mounted, the need arises for in-depth information about what students and faculty think and do. Our well-organized student survey programs (including the National Survey of Student Engagement-- NSSE) provide some indices of student attitudes, but they do not reveal the logic behind them. And we have no comparable data on faculty attitudes and experience. In 1986 Harvard found itself in the same situation, which is why they instituted the Harvard Assessment Project. For 17 years the project has been producing a steady stream of insights into the student behaviors associated with academic success. Based on these findings, Harvard has made substantial changes in its programs, its advising and its policy. For example:
A similar program at CU will complement our existing survey programs, including NSSE. The interview format common among ethnographic researchers will allow us to go beyond student reports to understand the larger context of student experience and decision-making. For example, we can probe why students meet with faculty, when they do, and why they do not, when they do not. IllustrationA key principle of the Harvard Assessment Project is that their data gathering is always oriented toward influencing policy and practice, rather than simply describing the status quo. Here is how this orientation might play out, using the Princeton Review party school rating as a trigger. The Project Committee would formulate a set of hypotheses. One hypothesis could be that (H1) heavy partiers are students from wealthy families who feel little need to prepare for careers. Another could be that (H2) heavy partiers lack other recreational interests, like participatory sports or outdoor activities. Another could be that (H3) heavy partiers feel disconnected from and not challenged by their studies. Each of these hypotheses has implications for policy or practice, which could be implemented if the hypothesis was supported. H1: Mentoring and advice to families could target students with weak career motivation. H2: The campus could more actively promote club sports and outdoor recreation. H3: Programs identified in the NSSE as less challenging and engaging could be shaped so as to strengthen them. The latter case illustrates how the interviews could help make more use of NSSE data, by linking these data to specific issues of concern for the campus. Possible companion project on faculty experienceCU could extend the Harvard Assessment process to include early career faculty as well as students. Over the years, PTSP Director Mary Ann Shea has conducted interviews with new faculty at CU. These interviews have produced potentially important research findings: *Incidents of incivility occur in the classroom that seriously disturb new faculty. New faculty lack the means to deal effectively with these incidents. *New faculty sometimes devote too much time to their teaching responsibilities, starving other aspects of their work. *Despite increased efforts in orientating new faculty, many new faculty still do not know what is expected of them at comprehensive review and continuously flounder while searching for that information. Together, the findings show the need for further work on new faculty and early career faculty, including mentoring for both, and suggest specific targets for these efforts. This effort could help CU deal more effectively with issues of faculty retention, including problems of diversity. Expanding the scope of this research, allowing it to be carried out annually, so that each year's faculty interviews can be targeted based on earlier findings, and making it part of a targeted effort to assess the impact of CU's efforts to improve faculty effectiveness, would yield substantial dividends in both policy and practice. A similar companion project on the experience of our graduate students would also be appropriate. Method(We thank Elaine Seymour and Liane Pedersen-Gallegos from Ethnography and Evaluation Research, CARTSS, for advice on method and budgeting.) Like the Harvard Assessment Project, the proposed project differs from a traditional, one-shot research project in that it calls for an ongoing program of research, with new research questions emerging as the program goes forward, rather than being specified up front. We proposers can only suggest examples of research questions that could be addressed; the real questions must be hammered out by the stakeholders in the project, in a process based on that used at Harvard, and outlined below. Since the details of sampling and analysis depend on what the research questions are, they cannot be specified in this proposal. Instead, we describe the outline of the process to be followed, with budget projections based on our estimates of an appropriate level of effort for the project. Student Sample. Harvard’s main sample is 385 students (about 6% of all undergrads) drawn at random from all enrolled students. A sample of the same size at CU would represent about 1.5% of CU undergrads. We suggest starting with this sample size, so that we can be guided in our design by the logistics of the Harvard experience. Targeted Samples. When major new initiatives are undertaken, for example the proposed Center for Research in Undergraduate Science Learning (CRUSL), special samples of affected students and faculty can be added to the project. This allows us to determine (for example) how student attitudes are affected by the initiative, and whether programs intended to inform faculty about research relevant to teaching are working or not. Sampling by College and Campus. This initiative can be funded as a campus program, or could be funded from Quality for Colorado funds allocated to the colleges. In the latter case, samples of students and faculty would be formed for each participating college. Participating deans would receive reports each year presenting the results and policy implications for their college. As noted below, dean’s representatives would also participate in designing the research questions used in the interviews each year. Governance.The success of the Harvard Assessment Project is due in part to wide participation in the creation of each year's research questions, so that it can be shaped to meet the needs of a variety of stakeholders. We propose that our questionnaire be composed by the Colorado Assessment Committee, consisting of the FTEP director, representatives of Faculty Affairs, Student Affairs, Planning, Budget and Analysis, and the Boulder Faculty Assembly, including representatives of campus entities with assessment expertise, and designees of the deans. Process.The project will interview each student twice, in spring and fall (following Harvard practice). The annual cycle for the project as a whole is this:
|