Map A to Z Index Search CU Home University of Colorado
 
  Strategic Planning Institutional Research and Analysis Campus Budget and Finances About PBA

PBA Home > Institutional Research & Analysis > Outcomes Assessment > Unit Summaries > Department of Political Science

Department of Political Science
Last updated 5/2/2002

Knowledge and skill goals for this undergraduate degree program are recorded in the most recent CU-Boulder catalog.

In some summaries of assessment activity, goals are referred to by number (e.g., K-2 is knowledge goal 2).

Assessment Activities in 2002-03
Assessment Activities in 2001-02
Assessment Activities in 2000-01
Assessment Activities in 1998-99
Assessment Activities through 1996-97

Assessment Activities in 2002-03

(Undergraduate Curriculum Committee: Prof. Mapel-Chair, Prof. Mewes, Prof. Bair, Andrew Thangasamy-graduate student)

Introduction

This report summarizes the Departmentís ongoing process of evaluating undergraduate degree program performance. After taking a close look last year at faculty teaching portfolios, the Department returned this year to its regular practice of focusing primarily on student portfolio assessment.

The Process of Portfolio Assessment

In 2003, the UCC performed two tasks: reviewing a sample of portfolios of graduating seniors and collecting portfolio materials for continuing majors, both by gathering tests and essays from a new cohort of freshman majors begun last year.

These portfolios contain a history of student performance as indicated by final exams and papers finished between Fall of 1999 and Fall of 2003. Identifying information was deleted from this material, leaving course number, assignment, and semester in which the work was done. The professorís name and grade assigned, if originally indicated, were also removed. Three class tests and papers constituted a minimum portfolio; most portfolios had four or five assignments. Seven portfolios were evaluated, totaling 23 exams and essays altogether. This was a randomly drawn sample, with GPAs varying from 2.25-3.89.

Each studentís written work was evaluated on four characteristics plus an estimated GPA. Those characteristics were:

  1. Basic knowledge of facts and understanding of issues
  2. Ability to evaluate conflicting knowledge
  3. Ability to use appropriate evidence and reach reasoned conclusions
  4. Overall writing ability

Students were rated on achievement during their fourth year of examinations and papers as well as on improvement over the entire period. Achievement was rated on a scale running from "Excellent" to "Good", "Fair", "Poor", and "Not Applicable" when there was insufficient evidence, e.g. when an assignment failed to require a particular intellectual task such as making comparative judgments. "Improvement" reflected a global impression of a studentís performance across all four characteristics. Raters were asked simply to indicate whether they perceived any improvement in performance between studentsí freshman/sophomore and junior/senior level course work by answering "yes", "some", and "none." The complete set of scores for the set of portfolios is shown in the table below.

Results

As one can see from the attachment, there are two cases where the estimated GPAs of students were considerably higher than their actual GPAs. We have no ready explanation for these two discrepant scores, although it may simply be that students have a tendency to perform somewhat better on assignments for classes in their own major than they perform in classes generally. Otherwise, the faculty (and graduate student) scoring of students was fairly consistent, and after discussion of the results, the committee unanimously agreed on the following conclusions. The committee concurred that, with one exception, all of the students showed clear improvement. The committee also agreed that this improvement was particularly evident in the writing skills of the students. The students who showed less improvement clearly had better skills at the beginning, a result that we have consistently found over several years of doing portfolio assessments. In contrast, students who were below par at the beginning of their careers showed the most dramatic improvement, once again in their writing skills in particular. It is hard to say, however, whether this means that the Department does especially well at teaching students who have poor skills at the outset of their education, or only that any improvement in the work of such students is bound to show up with particular clarity. Whatever the explanation, the overall results of the assessment are encouraging.

Another thing that the department seems to do fairly well is in striking a balance between creative and critical thinking. In addition to the usual exams and essay questions, for example, there were assignments that asked students to demonstrate their understanding of a particular set of issues by writing short stories or plays. When assignments were more conventional in character, professors did well in forcing students to engage in critical thinking by making assignments particularly explicit in this regard. Almost all of the assignments asked students "to make a counter-argument and refute it" or "consider alternative explanations." There were a few assignments, however, that did not provide sufficiently explicit directions to engage in critical thinking. This failure may have made a couple of students score lower than otherwise, through no fault of their own. This is certainly one place where we can continue to improve as a department, by being more consistent across courses in forcing students to engage in analysis that requires the consideration of alternative explanations or interpretations.

Raw scores of portfolios

Student Faculty Rater Understands Issues/ Factual Knowledge1 Evaluates Conflicting Arguments1 Uses Appropriate Evidence/ Reaches Reasoned Conclusions1 Overall Writing Ability1 Shows Improvement Over Time2 Est. GPA Actual GPA
(A) Baird 4 4 4 4 3 3.8  
Mewes 3 3 3 3 3 3.6  
Thangasmy 4 4 4 3 3 3.7  
Mapel 3 4 3 3 3 3.8  
Average 3.5 3.75 3.5 3.25 3 3.725 3.892
(B) Baird 2 3 3 3 3 3.5  
Mewes 3 2 3 2 3 3.0  
Thangasmy 3 3 3 2 3 3.25  
Mapel 3 3 3 3 3 3.4  
Average 2.75 2.75 3 2.25 3 3.2875 2.359
(C) Baird 2.5 2.5 3 3 3 2.7  
Mewes 3 2 2 3 3 2.6  
Thangasmy 3 2 2 3 3 3.0  
Mapel 3 2 3 3 3 3.1  
Average 2.875 2.125 2.5 3 3 2.825 2.972
(D) Baird 3 4 4 4 3 3.5  
Mewes 4 3 4 4 3 3.5  
Thangasmy 4 3 3 3 3 3.5  
Mapel 3 3 4 3 3 3.4  
Average 3.5 3.25 3.75 3.5 3 3.475 3.328
(E) Baird 3 3 4 3   3.2  
Mewes 2.5 2 2 2 3 2.9  
Thangasmy 4 3 3 3 3 3.5  
Mapel 3 2 3 3 3 3.5  
Average 3.125 2.5 3 2.75 3 3.275 2.7
(F) Baird 4 3 4 4 2 3.4  
Mewes 3 3 3 3.5 3 3.5  
Thangasmy 3 3 3 2 3 3.4  
Mapel 4 3 4 3 3 3.5  
Average 3.5 3 3.5 3.125 2.75 3.45 2.25
(G) Baird 3 2 2 2.5 3 3.2  
Mewes 2 2 2 1.5 3 2.5  
Thangasmy 3 2 2 2 2 2.7  
Mapel 2 2 2 2 2 2.9  
Average 2.5 2 2 2 2.5 2.825 2.941
1Rating scale: 4=Excellent, 3=Good, 2=Fair, 1=Poor
2Rating scale: 3=Yes, 2=Some, 1=None

[Top]


Assessment Activities in 2001-02

The political science department has assessed success and failure in undergraduate teaching in a number of ways in the past. We have looked at the historical development of our honors program, for example, and have also critically examined the results of several different kinds of surveys. Our primary method of assessment for many years, however, has been to review the portfolios of a random sample of graduating seniors. These portfolios contain a history of student performance as indicated by final examination and written papers collected from students as freshmen and again when these students are seniors. Portfolios are stripped of information that would identify a specific student, and student performance is then evaluated by members of the Undergraduate Curriculum Committee in terms of an estimated GPA plus progress made in four areas: basic knowledge of different areas of political science, ability to make comparative judgments and to think critically, ability to draw reasoned conclusions from evidence, and writing ability. The department continues to collect these data from freshmen, and plans to do a portfolio evaluation in 2002-03.

This year, however, we have elected to use a different approach to evaluating and trying to improve our undergraduate teaching. We have adopted this approach because we believe that it is also crucial to begin assessing the teaching process itself as a way of determining what works and what doesn't in educating our undergraduates. For this reason, the Undergraduate Curriculum Committee collected and critically evaluated the teaching portfolios of sixteen members of the department. These portfolios contain a very extensive set of materials, including syllabi, exams, paper assignments, web site pages, reading and studying guides, bibliographies and other kinds of teaching aids. In addition, the committee met with instructors teaching the large lecture courses to discuss how they are being taught and how they might be improved in order to do a better job helping students learn critical thinking skills.

General Findings and Recommendations

The committee was generally encouraged by the obviously great effort that has gone into faculty teaching materials, as well as by their wide variety. Virtually without exception, the political science faculty devote a great deal of thought and energy to course design. Given the very wide variety of materials contained in the teaching portfolios, it was also clear to the committee that it would be inappropriate to suggest a single model for constructing courses. Nevertheless, the committee does believe that faculty members can learn a great deal from one another about teaching, and that the more discussion and cooperation we can generate, the more students will benefit. We begin by noting a few of the specific innovations that our faculty have introduced in their courses. We suggest that it might be worthwhile for other members to experiment with similar projects (the committee can put faculty members together who wish to collaborate). Our report then moves to some more general recommendations.

Much of the specific experimentation we noted in the teaching portfolios involved the use of different media and technology in and outside of the classroom. One faculty member who teaches Comparative Politics, for example, has a quite ambitious, free-standing "film festival" of relevant foreign movies running side by side with her more traditional lectures, exams and paper assignments. A second faculty member has set up distance-learning, allowing students in his course to collaborate electronically with students at another university where a parallel course is being taught. Many other faculty members have built sophisticated web sites which give students much greater ability to interact and to make use of a wide range of resources on the net. In cases where course web sites are long and complex, at least some of these sites have internal links to make them more student-friendly. The general lesson that the committee draws from such experiments is that our faculty need to encourage each other to make greater use of technology in their teaching. One of the best ways in which this can be done is through the summer Faculty Teaching Excellence Program (FTEP) workshop on integrating technology in the classroom. The committee notes that several of our colleagues have now taken advantage of this course. The FTEP workshop discusses the pedagogy of using technology to its full advantage, teaches faculty how to employ various tools in the classroom, such as Powerpoint, and helps faculty build and evaluate their own course web sites. The FTEP workshop appears to be the single best resource available on campus for learning make use of new technology in the classroom. The committee strongly recommends that more political science faculty make use of it.

Turning to other areas, our first concern is with the large, introductory lecture courses. As part of our effort to make these courses more effective, the department passed a motion to include in these classes an assignment that requires TAs to work individually with students on critical thinking. One example might be a requirement that each student works with his or her TA on several drafts of a critical thinking paper. Another example might be a set of problems that must be tackled one-on-one with a TA. Thus far, writing assignments of the sort just described have been assigned in the large introductory lecture courses in American politics, international relations, comparative politics, and political theory.

The committee met with faculty teaching these kinds of courses to find out how these assignments have worked and to see what might be done to improve them. We were particularly interested in having faculty explain what was unique in their approach to these classes and how they used TAs. Are TAs encouraged to develop a separate "story-line" in their sections, for example, or are they expected to stick closely to the material in lecture? This discussion was particularly valuable in establishing the need to continue experimenting with the use of TAs and in establishing norms about working closely with TAs on how these requirements are implemented. One of the committee's recommendations is that we make it a regular practice for faculty teaching these large courses to meet and discuss how these courses should be taught. While it is obvious that each faculty member will use TAs somewhat differently, faculty should be expected to meet extensively with TAs every week to discuss both the course material and the way that material is being taught in sections.

On the basis of our examination of the teaching portfolios, the committee has two other major recommendations to offer. First, we suggest that the department develop a general web-page or set of pages that can be linked to all of the courses taught in the department. In particular, we believe it would be useful for students to be able to link to a departmental web-page on how to read critically, another page on critical thinking, and yet a third on stages in writing a research paper. A good example of how the reading page might be done can be found on one faculty web site that uses a special acronym to help students remember different aspects of critical reading. A web page on critical thinking might also be devised drawing on several different ideas contained in these teaching portfolios. For example, one faculty member first teaches a range of informal fallacies and mistakes in reasoning, and then has his students write a paper deliberately committing as many of these errors as possible. We would also like to note that where an assignment is both detailed and difficult, some syllabi include words of encouragement. We suggest that faculty think of their syllabi as something that can be humorous, encouraging, perhaps inspiring and otherwise communicative, rather than as just a list of readings and requirements.

The committee's second general recommendation, or set of recommendations, concerns the design of syllabi and course material. Again, we caution that one size does not fit all. Nevertheless, the committee had a chance to look at over thirty syllabi in political science, and we believe that faculty members might improve their teaching by including some of the following features in their syllabi (assuming they don't already):

  • Especially in large, lower-division courses, faculty should try harder to avoid assigning wide-open paper topics. Paper topics that are more closely tailored by the instructor give students needed structure and direction, are easier to compare and grade fairly, and provide students much less opportunity and incentive to cheat by making use of course paper web sites.
  • In a couple of cases, the committee found that there were too few readings and too few assignments for students. We recommend more viewpoints represented in the readings and as more grade-points, either by breaking large projects into stages, or through a series of shorter papers or problem-sets. For example, one syllabus we examined broke the course down into five distinct parts, each with an associated, short critical thinking paper. This kind of organization gives students more of a chance to evaluate their own progress and does not put them in a just a couple of "make or break" situations.
  • One syllabus we examined framed every reading assignment between a short paragraph explaining the purpose of the readings and another short paragraph posing critical reading questions. The result was that the syllabus became far more than a list of readings. Instead, the syllabus revealed that entire shape of the course and developed an overall argument in the process of making assignments.
  • A few of the syllabi we examined mentioned UROP opportunities. Where appropriate, we encourage faculty to follow suit. The UROP program needs to be better advertised and more fully used by students and faculty, and the syllabus is one of the easiest places to do this.
  • Although virtually every syllabi included a statement on disabilities policy, there should also be statements of policy on a variety of other issues, including allowance for religious observance, proper classroom behavior, observance of the honor code and standards for what will count as plagiarism. The Office of Undergraduate Education can provide URLs with information about CU policy on all of these issues.
  • Finally, several syllabi included quite extensive materials for each course, including not only a syllabus, but some or all of the following:
    • critical thinking reading response questions
    • review sheets and questions
    • extensive grading criteria, matched with concrete examples of how a paper can fail to meet those standards
    • research paper evaluation forms, with specific stages for writing the paper
    • mid-semester course evaluations
    • matching "for and against" reading lists
    Such extra material enables students to take greater responsibility for learning by giving them a structure and a set of tools to use. We strongly encourage the political science faculty to look over this list and think about what else they can add to their course materials. If faculty members want to look at specific examples of these materials, they should contact the committee.

Use of assessment in instructional planning/design

We have been asked how our student assessment process has influenced our curricular development and other educational programs related to unit instruction. In particular, we have been asked: 1) whether we have spent more time discussing the teaching of critical thinking in all classes; 2) what has been done more specifically in the large lecture courses, including information on the effects of our assessment process.

With respect to the first question, there is on-going discussion of the problem of teaching critical thinking, both in the Undergraduate Curriculum Committee and in the department as a whole. In the summer of 2002, for example, the committee assessed the teaching portfolios of sixteen members of the department and drafted a report with several recommendations about how to improve course materials to enhance critical thinking. At the departmental level, we have also been engaged in discussions of teaching critical thinking, in particular during our program review process (PRP) last year, when we made use of the results of outcomes assessment in our report on the undergraduate program. PRP included several recommendations about improving critical thinking in all classes, including a systematic assessment of the teaching process itself (which we have started, as noted above), supplementing FCQs with evaluation of more extensive teaching materials, etc. In addition, the department made more specific recommendations about teaching the large lecture classes. All of the large, lecture courses below include a mid-term exam (sometimes two), a final exam, and various recitation assignments. Here this report focuses only on the extra assignment that was included in order to strengthen the critical thinking component. Here are some examples of what was done:

  • Prof. Chan, Political Science 2223, Introduction to International Relations: Paper requirement (limited to 10 pages) on topic jointly assigned by instructor and TA; each student required to schedule a meeting with recitation leader to go over research plan for paper and to seek approval of and feedback on plan.
  • Prof. Greenberg, Political Science 1001, The American Political System: Introduced a critical writing assignment on a detailed topic about the extent of democracy in American politics; each student required to meet with his or her TA in a private session at least one time to work on this paper.
  • Prof. Niles, Political Science 2012, Introduction to Comparative Politics: Research paper (10 pages), on any topic related to course, to be done in five stages with approval of TA.
  • Prof. Steinmo, Political Science 2012, Introduction to Comparative Politics: Students choose a topic with TA's approval; give to TA for approval a 3 page prospectus paper with topic, question/hypothesis, and evidence; meet again with TA to discuss progress; turn in 10-15 page research paper.
  • Prof. Gries, Political Science 2012, Introduction to Comparative Politics: Five page paper, based on following coverage in The New York Times of a specific region of the world over ten weeks; paper outline must be submitted to TA in advance and student must also meet with the TA to discuss the writing of the paper.
  • Prof. Denning, Political Science 2004, Survey of Western Political Thought: Students given an essay question in advance for a paper of five pages due before the mid-term (essay questions also appear on the mid-term). Students then meet after the mid-term in two one-on-one meetings to work with TAs. The first meeting discusses topics and requirements and students are assigned the task of drafting a detailed outline of paper; the second meeting with the TA assesses the outline and comments directly on content and style.

The Undergraduate Curriculum Committee met in mid-September with all of the faculty responsible for these courses in order to discuss how better to teach the critical thinking component of the respective courses. Topics included: getting everyone to use their TAs in the required way; uniform expectations about supervising TAs; whether the critical thinking assignments focused primarily on factual questions or on questions dealing with critical thinking; whether paper topics should be open-ended or tailored; increasing the number of grade points. The Department is committed to pursuing this matter and plans to have a regular meetings in the future to monitor and discuss these classes.

A Note on Graduate-Level Assessment

At the graduate level, we emphasize the training of the next generation of political scientists in the knowledge and best practices of the discipline. Together with our students, we seek to explore and expand what we can know about politics and government and to reason about the classic questions of

[Top]


Return to the index of unit summaries

L:\ir\Outcomes\0A0102\polisci.doc and OA0203\polisci.doc

Last revision 05/05/03


PBA Home  |   Strategic Planning  |   Institutional Research & Analysis |  Budget & Finances  | 
Questions? Comments? |  Legal & Trademarks |  Privacy
15 UCB, University of Colorado Boulder, Boulder, CO 80309-0015, (303)492-8631
  © Regents of the University of Colorado