ASSESSMENT OF UPPER-DIVISION WRITING CLASSES
JUNE 2001

During the 2000-2001 academic year, the University Writing Program (UWRP) assessed its upper-division program through two methods: by analyzing a sample of 232 student evaluations and 167 writing lab surveys completed at the end of the Fall 2000 semester and by forming an outcomes faculty committee who scored and discussed sample writing folders from 48 students randomly chosen from Fall 2000 UWRP 3020 courses.

Overall, student evaluation results were quite positive, with more than three-quarters of the respondents--an average of 79%--rating each of 29 categories "1" or "2" on a 4-point scale. Writing lab surveys reveal that few students--only about 12% of the sample analyzed--use the lab, but of those students who use it, 95% reported their lab visit to be valuable. Among students who didn't receive help from a lab tutor, many reported that they would seek help if different hours were offered or that they would consider visiting the lab for help with future academic projects.

For the first time in UWRP history, faculty members were appointed to serve on an outcomes assessment committee. The six instructors appointed--Don Eron, Karen Gasser, Christine Macdonald, John Piirto, Petger Schaberg, and Tobin von der Nuell--scored student folders according to an analytic scoring guide. They also met twice during the Spring 2001 semester to discuss their experiences as scorers and the results of their scoring sessions. Their meetings raised important pedagogical questions about UWRP 3020 courses and resulted in several recommendations for future discussion and assessment designs. Foremost among these recommendations are a revised scoring guide that uses a holistic rather than an analytical rubric, continued assessment committee meetings to allow program assessment to go beyond quantitative data, and greater faculty involvement in discussion of assessment issues, including regularly scheduled faculty meetings during which faculty can discuss student papers in terms of pedagogy and individual assessment.

Details of these various assessment methods follow.

STUDENT EVALUATIONS

In addition to university-mandated FCQs, students in all UWRP courses completed an internal student evaluation form. For the assessment, a sample of evaluations from 17 sections of the course was analyzed. Every third section of the UWRP 3020 course was chosen from a filebox of all UWRP evaluations completed at the end of the Fall 2000 term. Evaluations had been filed alphabetically by instructor name, so the sample method ensured that no more than one section per teacher would be analyzed. The form covered five topics: the student, outcomes, the instructor, the workshop, and the courses. A copy of this form with results from the survey's 232 respondents is attached in Appendix A.

The Student

This section of the evaluation seeks demographic data on students--their year of study, their major, their previous experience in writing courses, the frequency of writing required in their other courses, and their expected grade in the UWRP course. Many of these results are self-explanatory. A few, however, deserve special mention.

The first question on the form confirms a demographic that has been troublesome to UWRP faculty for several years. While UWRP 3020 is designated as a third-year course, 75% of students in the sample reported being 4th- or 5th-year students. The UWRP 3020 curriculum is designed to coincide with students' early forays into the subject matter of their major disciplines, teaching them to focus on a single topic in depth. Ideally, students then transfer this kind of writing focus to their upper-division coursework. However, many students wait to take the course until their senior year. Those who try to enroll during their third year are often blocked out by those who've waited and who therefore have higher priority on the SIS system. This problem with enrollment costs students the benefits the course is designed to offer. Students' comments about the course reveal that they are aware of this loss:

• "If I had taken this earlier on in my college [career] it might have been more useful."
• "Why do we take this class as graduating seniors when we have already completed most of our writing for our college careers?"
• "What bothers me about UWRP is that I was not able to get into it until my senior year...It just seems that now it's not as useful as it could've been before."
• "This course was excellent. I only wish I took it as a freshman, not a graduating senior. It really got me to think creatively and analytically."

With this report, UWRP repeats its long-standing request that the registration system be programmed to alert students who have acquired 60 credit hours that they need to enroll in an upper-division writing course. At 90 credit hours, students would be blocked from enrollment unless an advisor saw fit to override the block. With such a flag on the registration system, students would be better able to use UWRP skills in their other coursework.

It is encouraging to note that an overwhelming majority of students--98%--reported that they write "occasionally" or "frequently" in their other courses. However, students often tell UWRP instructors that few of their other courses require that they write, and that their writing in other courses is likely to be summary rather than analysis. To get a better picture of the writing students do in their other courses, this question could be revised to ask about the kinds of writing students do as well as how often they are asked to write. By understanding more specifically what kinds of writing students are expected to do in other courses, UWRP faculty might be better able to demonstrate how writing skills transfer from one context to the next, a need indicated by other data from the evaluation.

Over the years, UWRP courses have often received poor ratings in response to an FCQ question about the "fairness" of grading in the class. While it's impossible to know how students interpret the word "fair," the students represented in the sample evaluations show expectations for strong grades, with 84% expecting a grade of A or B, and only 12% expecting a C or lower. The actual grades of the 285 students in the sample sections reveal an average grade of B- (2.8), with 65% of students earning an A or B, 32% earning a C, and 2% each receiving D or F. Such grades may indicate that students have an unrealistic view of what grades mean, equating "fairness" not with quality of work but with perceptions of their own effort or class participation.

Outcomes

Specific responses to the "Outcomes" section of the evaluation can be found in Appendix A. The questions in this portion of the evaluation asked students to rate their learning in the course on a scale of "1" (definitely) to "4" (not at all). A four-point scale was chosen to force students to one side or other of the scale. Scores of "1" or "2," then, can be read as positive. Student responses to questions about course outcomes demonstrate that over three-quarters of the students--78%--gave a "1" or "2" rating to each of the 18 categories listed. In only three of the 18 categories did the percentage of "4" ratings rise above 10%.

An especially high percentage of students--85% or more-- gave high scores to their ability to clarify their writing, establish a clear introduction, thesis, and proofs for their thesis, and to provide useful feedback about their peers' writing. In only two categories did the percentage of students rating a skill "1" or "2" drop to 65 or lower. Whereas students gave quite positive feedback to the course's success in teaching them to write clear introductions, to develop focused theses, to support their theses with proof, and to write substantive body paragraphs, many fewer students reported learning to write effective conclusions (65% "1" or "2" ratings). The other question receiving lower numbers of positive feedback (61%) asked if students learned "types of writing useful in college courses." It seems appropriate, then, for faculty to provide more instruction on effective conclusions and especially to explain how the analysis and argument focus of UWRP 3020 can be used in other university writing tasks.

The Instructor

Students rated the instructor's teaching ability in five separate areas. Ratings of "1" were consistently most frequent on this part of the evaluation, with instructors getting especially positive feedback for clear classroom guidance and for making students think.

The Workshop

Students' responses to six questions on the "1" to "4" scale in this section of the evaluation provided feedback about the workshop format of the class. In writing workshops, students critique each other's work and make suggestions for substantive revision. Again, the responses were positive overall. In fact, 91% of students reported putting "a lot of thought and effort" into their drafts by responding with "1" or "2" ratings. On the other hand, only 71% gave "1" or "2" ratings in response to a question about the amount of effort they put into commenting on their peers' work. The discrepancy of these responses underscores observations classroom teachers often make about students' failure to understand that peer review is an important part of both the class expectations and the development of strong writing skills.

The final question in the section of the evaluation asked students: "Did you consult with your instructor when you needed clarification about workshop comments?" Almost a quarter of the students replied "No." This failure on students' part to clarify suggestions for revision is troublesome, and is echoed in the low number of students taking advantage of UWRP writing labs established for their benefit. Certainly, UWRP teachers are more than willing to meet with students, as is evidenced by the crowded hallways outside their offices. While there is no way to know why these students didn't request clarification, possible reasons include scheduling conflicts with faculty office hours, reluctance to meet with the teacher one-on-one, or apathy. Better publicity about the writing lab schedule may be one way to provide options for students needing extra help, especially if scheduling is the main reason that students don't meet with their teachers when they have questions.

The Course

Students were asked to give short-answer responses to three questions about the particular topic of the UWRP 3020 course they took, about the readings chosen for the course, and about the course's impact on their ability to analyze and argue about issues.

The first of these questions seemed to confuse several students. The phrase "this topic" seemed unclear, and should be revised before the end of the next semester.

Some students noted that texts and topics were difficult, but often wrote that the difficulty provided a welcome challenge and varied opportunities for writing. Indeed, students' responses to the questions were generally quite positive. For instance:

I felt that this is the ONLY class that showed me how to critically analyze/think about my writing.

Papers made me think and write in a way that I am not used to. I was challenged to go further and push the details. ...I feel that I am now better able to write/develop a strong argument rather than simply retelling the story.

I write a lot of argument papers for my major and it helped with some of my assignments.

This course has improved my ability to write an organized, fluent, and solid essay. I definitely learned a lot about working in stages.

This course made me always ask WHY?

[I] learned a lot during this semester; probably the best ...of all CORE classes!

My ability to express or reason have always been weak & it helped me organize my thoughts better by seeing that readers are lost if I make assumptions about what they know.

It [the class] has taught me that all sorts of interesting connections can be made by thinking through something.

Certainly such student comments are gratifying to UWRP faculty, whose commitment to a focus on critical reading/thinking and thesis-driven, conceptually rich writing is at the core of their curriculum. But many students questioned how UWRP 3020 fits into their own university focus. For instance:

As a 4th year Poly Sci student who has written over 30 papers in the past 2 1/2 years and done well, I find it discouraging to be required to take a class that is structured completely different and not perform well...

I felt like I learned a very particular structure/style of writing. I do not think it is particularly applicable to other papers I write or writing for the future.

This class did not help me because it was too different from my other classes.

This topic leads to critical reading of stories and papers but seems very focused to only one type of writing. It doesn't seem that it will apply to as many other classes or future assignments that I would have liked it to.

I learned to analyze a paper. But the style of writing for UWRP is specific only to UWRP classes. I want to learn to write papers for other classes.

I think this course is too subject-specific, and a lot of it cannot be used elsewhere.

I will never have to analyze a book. I needed to learn how to analyze real world situations and problems, not something from this book.

You need to offer more choices for science majors.

I learned a lot, but I felt the writing style will not apply to my other course work or professional writing.

Clearly, a significant number of students believe that their upper-division writing course should more clearly mimic the kinds of writing expected in their disciplines. Each of the comments above came from a student whose UWRP course had a literary focus--poetry, short story, classic literature. But the students whose comments are recorded here came from a wide variety of majors. It is true that a majority of the UWRP's course offerings are literary. It is also true that these courses focus on the critical thinking and skills development necessary to write well in any discipline. Additionally, teachers' choice of particular topics reflects their interests and allow them to bring both knowledge and enthusiasm to their classes. Nevertheless, these comments reflect a valid concern about the topics upper-division students should be writing on. Certainly, many students enjoy a break from their disciplinary reading, and are comfortable that they can use the skills developed writing about poetry or the works of Jane Austen as they write for their own disciplines. As one student commented about a class that focused on poetry, "[I] enjoyed the reading--just for enjoyment; [I] can win an argument anytime now." Other students echoed the universality of the concepts covered in UWRP courses. One student comments, "I feel I can better analyze issues now and argue my point clearly. ... I know from the way I read things in my other courses now that I did learn how to do this well." Another says that the UWRP 3020 course has helped him/her "create other thesis-based papers in other classes." Some students are, in fact, able to transfer skills from one context to another.

But others, it seems, have not yet made this cognitive leap, and may need a course that explores writing within a topic more familiar to them. Indeed, several researchers have revealed that students tend not to transfer writing skills learned in one class to another class. UWRP students' comments that they couldn't use what they learned in UWRP 3020 in other courses may mean that, because the UWRP topic was unfamiliar to them, they needed to put as much effort into learning the subject as into mastering the skills necessary to write focused analyses and arguments. Thus, students are sometimes unable to see the similarities among the kinds of writing demanded in their various courses.

The UWRP can take steps to accommodate these students by offering more UWRP 3020 course topics in disciplines outside of literature, art, and music so that students in other disciplines, particularly the natural sciences, can learn writing skills in a subject with which they are more familiar. Of course, such accommodation would necessitate using faculty whose knowledge of such topics complemented their teaching of writing.

Another and perhaps more feasible approach is for UWRP 3020 faculty to show students explicitly how skills learned in UWRP courses can transfer to writing demanded in other courses. This approach, called the "high road" by researchers D. N. Perkins and Gavriel Salomon ("Are Cognitive Skills Context-Bound?" Educational Researcher, 18.1, 1989) and by Julie Foertsch ("Where Cognitive Psychology Applies: How Theories About Memory and Transfer Can Influence Composition Pedagogy," Written Communication, 12.3, 1995) requires teaching students "how to analyze the underlying discourse conventions that operate in whatever discipline, classroom, or writing context they encounter" (p. 379). By adopting such an approach, UWRP faculty would not only provide invaluable skills to students, but would show them to actively and appropriately transfer them as well.

WRITING LAB SURVEYS

Students completed writing lab surveys at the same time they completed student evaluations. Not all students completed lab surveys; only 167 surveys were collected from the 17 sample sections. Of those 167 students, only 20, or 12%, had used the writing lab. All but one of these students were native English speakers. Specific results for the survey can be found in Appendix B.

Students who did use the lab

Of those students who did use the lab, most used it only once or twice. Data from those students reporting five or more uses indicates that they attended not the one-on-one tutoring lab, but the special grammar sessions conducted by a UWRP instructor as part of the UWRP lab. These students also were in this instructor's class, and so had extra encouragement to be there to focus on basic writing skills with which they struggled.

Certainly the low number of lab users is disappointing, especially in light of the almost unanimous feedback that the lab was useful to students who used it. Students received help with a wide range of questions, from grammar to conceptualization of an argument to logic and organization to style. But the surveys also indicate that only students whose instructors recommended the lab made use of it. In lower-division UWRP courses, lab tutors routinely visit classes at the beginning of the semester and explain what the lab is and what kinds of help it can provide. But lab tutors do not visit upper-division classes, and so students often don't hear about it (or forget that they've heard about it from their teachers in the first week of class) unless their teacher specifically recommends a visit. It seems advisable to have tutors visit upper-division courses as well, to briefly introduce the lab's services. While there are many more upper-division courses than lower-division courses offered, these visits can be scheduled if each lab tutor arranges to visit several classes over the course of the first few weeks of the semester.

Students who did not use the lab

One hundred forty-seven students (88%) reported not using the lab. Four of these students reported speaking English as a second language. Of these 147, only a very few (five, or 3%) were turned away because of tutors' full schedules. Each of these incidents occurred at mid-term, suggesting that UWRP faculty need to encourage students to schedule appointments early, and that tutors may want to work some flexibility into their hours at particular times during the semester.

A majority of students said that they were unlikely to go to the lab regardless of its hours, but 31% also commented that they would have been more likely to use the lab if it had drop-in hours. It seems that students want the flexibility of getting help when it's convenient for them and when they're struggling with a particular problem rather than having to put the problem on hold until they can schedule an appointment.

Such an arrangement may be offered to students under the new Program in Writing and Rhetoric (PWR), which plans to establish a Writing Center within the next few years. This center will have the advantage of being centralized, whereas current lab tutors have offices scattered throughout campus. This centralization will allow the Writing Center to establish a presence that the current scattered lab offices have been unable to establish.

The new Writing Center will likely employ peer tutors as well as instructor-tutors. Survey responses indicate that 31% of the respondents would be reluctant to work with a peer tutor, while 34% would feel comfortable working with either a trained undergraduate peer tutor or an instructor. This data presents hurdles for those involved in establishing the new Writing Center, namely, to change students' attitudes about working with peer tutors and to ensure that tutors get the proper training to build a positive reputation for the Center.

A quarter of the respondents indicated that they would have attended small-group sessions. They earmarked grammar instruction and group work on their own drafts as especially promising small-group activities. The UWRP already offers small group grammar labs, although many students may not be aware of them or may not be able to work them into their schedules. Specific lab time devoted to small group work on papers could provide students from different sections of the course with an opportunity for rich exchange of ideas and workshop techniques and may be worth piloting on a small scale.

Responses from students who had not used the writing lab revealed an overall attitude that the lab would not be useful to them. Many consistently indicated that they did not try to use the lab (97%), that they wouldn't use the lab even if its hours were more accommodating (40%), that they wouldn't have attended small-group sessions if they had been offered (75%). However, 60% of the respondents said that they would use the lab in the future for non-UWRP projects--especially for help writing application essays, honors theses, or course papers. It is curious that these students see a use for the lab as they tackle other writing tasks, but thought it unnecessary for their own writing course. This response may indicate that students, overall, felt comfortable with the feedback they received from their own writing teachers and that their teachers' office hours accommodated any one-on-one help they needed.

OUTCOMES ASSESSMENT COMMITTEE REPORT

For the 2000-2001 academic year, six instructors were assigned places on the outcomes committee as part of their service requirement. These six instructors were Don Eron, Karen Gasser, Christine Macdonald, John Piirto, Petger Schaberg, and Tobin von der Nuell, each of whom had the opportunity to attend the Conference on College Composition and Communication for professional development. Four committee members did attend, each for the first time. Because the conference was held in Denver, attendees did not incur exorbitant travel expenses; thus it was financially feasible to fund their registration fees through the assessment budget. Those who attended the conference found it valuable on several levels, most notably as a chance to hear first-hand about assessment research and current questions, and as an opportunity for professional development in their own areas of expertise and interest. It is recommended that conference attendance continue to be funded for assessment committee members, even if this means earmarking assessment funds for travel.

This committee undertook assessment of the UWRP 3020 course. Assessment committee members scored assessment papers collected from Fall 2000 UWRP 3020 courses and held two full-day assessment meetings during which they discussed curriculum and assessment in light of the papers they had scored.

Quantitative Assessment

All teachers of UWRP 3020 during the Fall 2000 semester were asked to keep folders of papers submitted by two students from each of their classes. To achieve a random sampling of papers, the Outcomes Assessment Coordinator selected these students randomly from course rosters. Faculty were to submit clean copies of all final, major assignments from these two students and submit them at semester's end. They were to include a copy of the assignment for each paper. Not all faculty submitted assessment papers, and some instructors' submissions were unusable because they were not clean copies or were incomplete. Usable folders were sorted so that each assessment committee member received 17 student folders. No committee member scored papers from his/her own classes. Each committee member looked at a set of folders identical to those also reviewed by another committee member so that scores could be compared between pairs of scorers.

Papers were scored analytically according to the score sheet found in Appendix C. This score sheet asks scorers to rate each paper on a four-point scale (1 = excellent, 2 = good, 3 = fair, 4 = poor) in each of 13 categories. This score sheet was similar to the one used for earlier assessments; the number of categories was reduced from 15 to 13 by folding two categories into categories seen as inseparable from them. No norming session was conducted before the scoring. In light of questions being raised in the field of composition and rhetoric about the need for reliability and about the influence of what Bob Broad has labeled "Teacher's Special Knowledge" ("Reciprocal Authorities in Communal Writing Assessment: Constructing Textual Value within a 'New Politics of Inquiry,'" Assessing Writing, 4.2, 1997), the assessment goal was not necessarily to achieve numerical reliability in scorer pairs, but to solicit scores that would provide information about the papers and the way they are seen by different instructors. Such information would then guide discussion of curriculum and pedagogy within the UWRP.

Indeed, the scores revealed wide discrepancies between numerical scores given by each pair of scorers, but discussion revealed similar strengths and weaknesses for many of the papers. These discussions will be explained in the Qualitative Assessment portion of this report.

Numerically, scores from the internal scorers are worse than the "average" score of 2.5 one might expect from a representative sample, with the overall paper scores averaging around 2.8. While this score might seem alarming, especially in light of the 2.3 overall average during the last UWRP assessment, it reflects a difference in scorers' interpretation of the scale rather than a decline in student performance. Only two of the six instructors on the committee had scored outcomes assessment papers in previous years, and their scores were, in fact, more closely aligned than those of other scorers. Interestingly, the instructors new to assessment interpreted the scale on the score sheet to correspond to letter grades, a parallel not intended during the scale's development.

Seen in this way, average scores of "2.8" correspond to a letter grade of B-, which is consistent with the UWRP 3020 grades in the student evaluation sample discussed above. Scorers were quick to point out that, in fact, most of the papers were competent and demonstrated many strengths. The scores suggest that the papers' biggest strengths were in clarity, organization to support a thesis, style/tone/diction, and freedom from surface-level errors.

As has been the case throughout the UWRP's assessment history, students' weaknesses seem to lie in making/defending inferences, fully developing ideas, and dealing with sophisticated rather than predictable ideas. Cognitive development experts often report that, as college juniors and seniors, many students have just begun to synthesize ideas from many different areas of study and experience and have found that these ideas often conflict. That conflict prompts more sophisticated thinking and new integration strategies. This developmental theory is supported by instructors' frequent reports that older students are more likely to enter the class exhibiting these kinds of skills. But whatever the reason for these recurring weaknesses, UWRP faculty need to discuss creative ways to challenge students to better develop these skills.

In previous years, assessment papers' scores also showed weaknesses in representation of counterarguments. Partly, this weakness was caused by unclear directions about scoring this category, and so scorers often gave papers a low rating either because they didn't appropriately represent a counterargument, or because it was unnecessary, given the paper's rhetorical situation. This year, scorers were given clear instruction to mark this category "NA" where appropriate and to give scores in this category only to those papers whose thesis demanded it. Nevertheless, the scores vary wildly. Making decisions about the appropriate use of countervoice depends so much on considerations of rhetorical situation that more emphasis on such considerations probably needs to be incorporated into both the scoring rubric and the course itself. To this end, the committee will meet during the 2001-2002 academic year to design a holistic rubric that incorporates these skills and that allows discussion based on a single numerical score for a paper rather than many subscores. Such a rubric seems more in line with the grade mindset to which committee members are accustomed.

Qualitative Assessment

The two assessment meetings, held January 10 and May 12, focused not on discussions of papers' numerical scores, but on questions raised by those scores. The earlier meeting's purpose was to outline students' general strengths and weaknesses, as evidenced by the papers, and to talk about the pedagogical concerns raised by the papers. The later meeting used discrepancies between members of scoring pairs as a springboard for discussing issues of grading, of the influence of "Teacher's Special Knowledge" or TSK--the effect that a teacher's knowledge about a given student and about classroom particulars has on teacher evaluation of that student's work, and of basic but often unexplored questions about course curriculum.

The January meeting was called soon after committee members had scored the sets of folders. Each pair of scorers met to review their scores and to talk over any scores that seemed unusually discrepant. The pairs recognized that their scores were often quite different, but found that their overall impression of the papers and even of particular papers was consistent. One scorer remarked that his and his partner's scores were "pretty much the same, but a number apart." Agreement with this comment reveals that scorers interpreted the scale differently.

Overall, though, scorers agreed on general strengths and weaknesses in the papers. A primary concern was that students often introduced their papers poorly, leading to weak theses. Committee members agreed that some assignments led to this problem because they didn't provide adequate demand for attention to a paper's rhetorical context. Indeed, the difference in the kinds of assignments given to students may warrant some assessment in its own right. As Edward White notes, "...the assignment variable really complicates the validity of portfolio assessment: some assignments are so creative and supportive that most students write very well on them, while other assignments are so vague and boring that very few students will produce good writing" ("Portfolios as an Assessment Concept," New Directions in Portfolio Assessment, Black, Daiker, Sommers, and Stygall, eds., 1994). Although the UWRP does not currently use portfolios in its assessment, the student folders collected have many portfolio qualities; certainly this caution about assignments applies here.

And, while scorers saw weaknesses in theses and in the conceptual development of many papers, they agreed that the papers revealed writing toward a thesis and a discernible structure, even if the thesis was not particularly strong. Students also tended to make assertions rather than develop ideas, though there was evidence that they were struggling to analyze. In short, the papers suggest that students understand what a strong paper should do, but haven't had the practice to incorporate those elements into their own work.

Committee members' further discussion focused on three areas: the nature of class assignments, the reading/writing process, and student development.

One consideration about assignments is whether or not section topic makes a different in students' performance. UWRP 3020 courses are topic-specific, with a majority of the topics being literary. Visual and musical arts are also represented. While some courses broaden the range of topics, from history to criminal justice to environmental issues, non-literary topics are in the minority. Unfortunately, topics outside of the arts were poorly represented in the assessment sample. Papers from several of these sections were not submitted for assessment, and in other cases the submissions were unusable. It is unfortunate that so many non-literary topics were unrepresented in the sample, because they would have provided data through which to analyze the effect of topic on student performance.

Regardless of topic, the committee agreed that some assignments led to better papers than others. Specifically, assignment sheets that gave students clear parameters (e.g., to analyze according to some established set of criteria) led to "tighter" papers than did assignments that were much more open-ended (e.g., "Articulate and develop a thesis provoked by an issue in the text."). However, scorers also suggested that tight parameters don't always allow for students to think outside of the parameters, and so may hinder students' ability to think through a text on their own. Scores didn't show a distinction between those papers with very clear parameters and those with broad paper guidelines. Perhaps, then, as the committee suggested, important aspects of an assignment are that it meets the teacher's style, that it provides guidance for the student, even if that guidance is not written down on an assignment sheet, and that assignments throughout the semester evolve to teach and to link different kinds of skills.

The committee also spent time discussing issues of reading/writing process, both on the students' part and on their own. In reading student papers, the scorers noticed that student writers often seemed unaware of rhetorical situation, particularly of an audience for a given thesis or of the possibility that an audience might be skeptical of their claims. In the same vein, they also noted that student papers were often "writer-based" rather than "reader-based," focusing on getting down a set of information rather than conveying it in the context a reader needs. Notably, audiences were not often identified on assignment sheets. Nor does the score sheet mention rhetorical context/audience particularly. Yet these concerns were a major focus of the committee's discussion, revealing that more emphasis on rhetorical situation is needed both in the course and in the assessment design.

Committee members' comments corroborated the quantitative analysis above and led them to discuss the consistent weakness in the areas of idea development, defending inferences, and working with sophisticated ideas as a matter of student development. What can realistically be expected of an undergraduate student in a writing course? And what is the relationship between realistic expectations and the expectations that faculty actually have? Development comes, of course, with time and with opportunity to experiment with ideas and with writing strategies. UWRP's focus on workshop and revision accommodates this kind of experimentation to some degree, but often neglects the invention stage of writing, assuming that students will do this on their own. To better help students mature in these more demanding stages of writing, it might be appropriate to spend focused class time on invention, rather than expect students to leap from the discussion of a text to an introductory sketch without being guided through some invention techniques.

In short, the committee was satisfied with student achievement, but also intrigued with issues whose further discussion might lead to more mature student writers.

The May meeting focused on three particular papers, using a norming session format to spur discussion of issues raised by discrepant scoring. Papers were discussed not in terms of the rating scale, which most of these scorers had used only once, but in terms of grades, a currency they have been using for years in the UWRP. In many cases, scores from a pair of scorers had varied widely. In a few particularly remarkable cases, scores in every category between the two scorers were 1.5 points or more apart on the 4-point scale. Subsequently, one paper discussed at the meeting was representative of this marked disagreement. A second paper had received consistently high scores from both its scorers. A third had received discrepant scores, and came from a class whose instructor also sat on the committee. These three papers provided fodder for an often heated discussion of course goals.

Focused discussion of particular papers did not necessarily better align committee members' evaluation of the papers. A closer look at the paper that, numerically, seemed most problematic left one of the paper's scorers wondering why she had given it strong scores. Committee members awarded grades to this paper ranging from D to C+. Members who thought it deserved the higher grade argued on the basis of the thought the paper exhibited, while members who gave it low grades argued that, evidence of thinking aside, the paper simply didn't work because it was too difficult to follow.

Perhaps more interestingly, the paper that received relatively high scores from both of its original scorers got committee grades ranging from C+ to B+. Most committee members agreed that the paper was in the B range--that, unlike many student papers, it recognized a sophisticated counterargument, used strong vocabulary, made a clear distinction between description and analysis, demonstrated a critical reading of the texts at hand, and was fluent. They praised the instructor's assignment and the choice of texts, which, they said, led to a complexity behind the paper that encouraged critical thinking. One instructor, however, argued that the thesis was not clear, that it did not go far enough to overcome a descriptive approach to the assignment, and that the student missed many opportunities to make the paper truly outstanding. This instructor's understanding of the paper's topic--Japanese internment camps during WWII--was informed by texts given in the assignment but also by his own knowledge of the subject. After this discussion, committee members were left asking, "What do we really respect in a student's paper?"

Clearly, the discussion of these two papers raised important, fundamental questions that cannot be answered easily, but which the UWRP should address through its faculty meetings, its brown-bag discussions, and perhaps in professional development seminars. One thing is clear, though: even numerically "reliable" scores do not give a good picture of assessors' evaluation of student writing. Indeed, Pamela Moss, in a landmark article "Can There Be Validity Without Reliability?" (Educational Researcher, March 1994) argues that rater disagreement offers "an impetus for dialogue, debate, and enriched understanding informed by multiple perspectives..." (p. 9).

The third paper elicited the widest grade discrepancy of all--from C- to B+. And this discrepancy rested on a single issue: the distinction between true analysis and summary/description. This paper had been written by a student of one of the committee members, so the committee could ask this instructor directly if the paper merely summarized a text on which the assignment was based, or if the student had arrived at her conclusions analytically. The discussion of this paper, then, turned to two, intertwined concepts: the importance of subject knowledge in paper scoring and TSK--teacher's special knowledge.

As assessment expert Edward White comments, when assessment is conducted by people outside the classroom and so asks scorers to rate papers according to criteria that don't consider their subject knowledge, scores can go far afield. By way of example, White writes (New Directions in Portfolio Assessment):

..I had strong reactions to two particular papers, one on the War of the Spanish Succession, which I thought was witty and wise, and the other on Faulkner's novel The Sound and the Fury, which, though written fluently, reflected serious misunderstanding of the book and included foolish statements about the author. As it turned out, I stood alone on the assessment team in my scoring of both papers... My ignorance of European history had misled me on the Spanish Succession paper, which placed the war in the wrong century and had distorted most of its facts and interpretations. And I was the only one in the room who knew the Faulkner novel, which I occasionally teach, so that paper's superficial fluency impressed the other readers, who had no way of knowing how uninformed the paper was (p. 33).

In the UWRP case, instructors who were unfamiliar with the short story about which the student wrote read the paper either as mere summary of one of the text's metaphors or, assuming the classroom teacher would not have accepted such an approach, as strong analysis of the same metaphor. Only the instructor himself, informed by his own knowledge of the student and class workshops (TSK), was able to weigh the student's work. This discussion left the committee in hearty agreement that TSK is, and should be, an important part of assessment and that more faculty involvement in such discussions would lead to more valuable program assessment.

In addition to a new holistic rubric, the committee will design and administer a faculty survey that will allow them to identify faculty assessment needs and begin to design a more inclusive assessment process. While these steps are being taken, the committee recommends:

• regularly scheduled assessment sessions during which faculty can share student papers to discuss strengths, weaknesses, and grades;
• faculty seminars focusing on assignment development;
• contemplation/discussion of questions of pedagogical issues-including grading, student development, skills transfer, and program goals-in light of current publications in composition and rhetoric.

University Writing Program summary - index page

Return to the index of unit summaries

Return to the Outcomes Assessment Table of Contents

[Questions? Comments?]   [PBA Home]   [CU-Boulder Home]

L:\www\outcomes\units\uwrp.htm
Last updated October 31, 2000