PBA Home > Institutional Research & Analysis > Outcomes Assessment > Unit Summaries > Department of Communication

Department of Communication
Last updated 3/27/2002

Knowledge and skill goals for this undergraduate degree program are recorded in the most recent CU-Boulder catalog. In some summaries of assessment activity, goals are referred to by number (e.g., K-2 is knowledge goal 2).

Outcomes Assessment 2000-2001

Outcomes assessment and the various departmental responses has to be considered within the context of enrollment and curricular trends within the department. Of most significance was the attempt to continue to meet departmental instructional goals while accommodating a doubling of the number of major within the previous three-year period, without an additional allocation of instructional resources.

Curriculum Reforms

During 2000-2001 we significantly changed the curriculum based on outcomes findings and enrollment pressures. Our analyses, based in part on prior outcomes findings, identified several problems. While many of these resulted from rapid growth they could be partly addressed by curricular reform. For example, we discovered that students were often taking courses out of sequence (often because of closed classes), thus they entered more advanced courses without the general background necessary to aid effective learning. Instructors felt the need to repeat material to assure understanding by all in class. Further developments of and stronger enforcement of specific prerequisites aggravated the enrollment problems and could not address the sequencing problem sufficiently.

Further, we identified a problem with what we called the excluded middle. We had good solid introductory courses in each content area and very good advanced seminars but lacked sufficient number and availability of 3000-level courses to provide specialized advanced content to the ever-increasing number of students. And finally, we found that critical thinking skills were often learned in isolation and not carefully integrated into the research and subject matter of the discipline. Each of these problems had consequences for student learning and what they carried out of the major. In response we took a new look at our overall undergraduate curriculum.

The approved revised curriculum is available online. To address the sequencing and adequate background problems, we changed COMM 2210 from a second-year to a first-year course. COMM 2210 (now 1210) is the first general theory course that students take in the major. Further we increased the size of the second level theory course (COMM 3210), thus getting students into it earlier, and we increased the number of 2000-level requirements. For the "excluded middle" problem, we have gradually greatly expanded the size and number of our 3000-level offerings. Since this helps the mid-level content problem at the potential expense of writing opportunities, we are developing a new 1000-level course with more direct writing instruction. And finally we have integrated critical thinking skills into our 4000-level seminars.

Owing to these changes and to work anticipating a new A&S policy allowing us to restrict the enrollment in our major, our outcomes assessment was somewhat abbreviated from prior years. As students progress through these new requirements we anticipate much more focused assessment to determine their consequences.

Outcomes Assessment Procedures

This past year the department used three measures to assess its undergraduate program: (1) A 50-item multiple choice test that measures the department knowledge goals; (2) a fifteen-item survey assessing satisfaction with the major, administered to graduating seniors; and (3) an internship survey administered to field supervisors that assesses displayed communication understanding and skills.

Assessment Procedures History

In 1989-90 and 1990-91 the department assessed knowledge goals by examining students' answers to items from classroom exams in several upper-level courses. In addition, three outside experts evaluated the skills-related goals by rating students' performance on written and oral assignments in COMM 4800 (Current Issues in Communication.) These assessments led to changes in the structure of some courses, and course coordinators were appointed and encouraged to supervise, evaluate, and assist graduate-student teachers. The department initiated a three-day orientation for incoming graduate-student teachers and a corresponding semester-long workshop on teaching. A system of peer teaching evaluation was also established.

In 1991-92 the department suspended the use of outside reviewers for assessing skills. It was proving too costly and too complex- outside reviewers were increasingly unwilling to participate. That year's focus was the development of a 50-item multiple-choice exam to test all of the program's knowledge goals. The first version of the test was administered to 50 graduating seniors in Spring semester, 1992. For 1992-93, a faculty review committee rewrote items that statistical analysis of the pilot version showed to be poor discriminators. In 1993-94 there was some further revision of a few items and the addition of 12 new items, and the test was administered in a required senior level communication course. In 1994-95, the practice of comparing scores of majors at entry and exit, as well as to non-majors, was started. A survey for supervisors of interns was piloted in the 94-95 year and implemented as a formal part of the program in 95-96.

In 1997-98, because of changes in course offerings the knowledge assessment test was revised. In addition, six items were added to the test, to be filled out only by seniors. The knowledge items are now reviewed annually. The senior survey was to measure satisfaction with the major, both overall and in regard to the major's focal skills. Finally, we decided to track the number of majors graduating with professional or university honors. Number of honors students seemed a good indicator of how the program was doing with regard to its strongest students.

In 2000-01, we developed a new self-report instrument for graduating seniors and decided to not independently track our honors students.

Latest Outcome Results and Interpretations

Measure 1: 50-Item Knowledge Test

The current knowledge test assesses general knowledge of communication principles and studies and represents a reasonable distribution of the four main content areas of the department: (1) communication theory and method, (2) organizational communication, (3) argument and rhetorical theory, and (4) interpersonal interaction. Each area had 12-13 questions for a total of 50 items. In the spring semesters of 1998 and 1999 this test was administered in all sections of the introductory public speaking class (Comm 1300) that enrolls a significant number of non-majors as well as students beginning the major. Each spring including 2001 the test was also administered to communication students in senior seminars (4000-level classes). Mean student scores are noted in Table 1 for four groups: lower division and upper division majors, and lower and upper division non-majors.

Inspection of the table shows senior majors to perform better than either lower-class majors or non-majors of either rank. T-tests comparing seniors with the other groups were significant at the p=.01 level or higher. Given there is considerable choice in the classes a communication major may take, and the exam measures all possible course content, the pattern shows exiting seniors to know more that either of the three groups. Since most students specialize in two or three of the four possible areas, the differences are artifactually small, but still statistically significant.

Although this index provides a general sense that students are learning discipline-relevant information, the department sees the index as a relatively general measure. We would like to see a bigger difference in the performance of seniors relative to the other three groups. This profile led the department to add a course requirement for all communication majors. As of summer 2000, all communication majors were required to take Comm 2210 (Perspectives on Communication) rather than selecting from a menu of introductory course possibilities. This curriculum change increased the particular theoretical knowledge that could be expected of all majors yet still retain students' ability to tailor course selection to their intellectual interests. Our new requirements now go well beyond that change. We think that our new requirements should have a greater impact. But we may have to further tailor the test to adequately measure the actual impact.

Spring semester 2001 was our first opportunity to accomplish a longitudinal study comparing the scores of the graduating seniors to the scores of first-year students three years earlier (These are not, or at least not entirely, the same individuals, but are samples from the same cohort.) See Table 2. We expect students graduating in Spring 2002 to even further exceed entry level scores. In the future, the new requirements will help insure that students have a better sequence of courses and larger amount of content knowledge. Higher difference scores should result since we are increasing the number of core ideas with which majors can be assumed to be familiar, and enabling instructors teaching senior seminars to assume a larger common knowledge base for students in the class. Instructors can be more demanding of students in upper-level classes.

Measure 2: Senior Survey

For spring 2001 we developed a more complete senior assessment tool. This instrument focused more on self-assessment of knowledge and skills and less on satisfaction. Graduating seniors were asked to rate themselves on 14 different specific skills and knowledge sets that are core to the department's mission and directly related to students' personal, professional and civic life, as well as to assess the quality of the major. The instrument is here. [Heather - make "here" a link to l ir outcomes oa0001 commsurv.doc.] Each of the items is marked with a 5-point response scale, from "strongly agree" to "strongly disagree," and each makes a statement that "My studies in communication have [improved my ability or knowledge]…." in a given area, as listed in Table 3 below. In addition, open-ended items asked the student to describe what he or she liked best and least about his or her studies. This instrument was emailed to all graduating students; about 30% (n=43) returned the instrument. In the spring of 2002 we have included the survey with a form related to graduation and asked students to return them together; we hope this will improve the response rate.

Overall satisfaction with the major has remained fairly constant over the past few years. Negative comments in the open-ended section related mostly to the difficulty of getting into classes and redundancy across courses. We hope to soon have the enrollment under control. Better sequencing of courses should lead instructors to feel less need to repeat material. The survey indicates that we need to continue to focus on writing. Surprisingly, we also need to do more with teamwork and collaboration. Hopefully the new placement of critical thinking will aid in some of the other categories.

Measure 3: Assessed Competence of Communication Interns

A significant number of majors do an internship in a professional setting related to one in which they may have a job interest following graduation. In an internship, students spend a certain number of hours in the field, keep a journal, and write up a paper linking a theory of communication with their observation and experience in the internship site. Toward the end of each semester, interns' field supervisors are sent a 7-item questionnaire and are asked to evaluate their intern. Five of the items ask the supervisor to rate the student on 7-point scales where 1 = poor and 7 = excellent. Items included, for instance, "I would describe the intern's ability to express himself/herself orally as" followed by a seven-point scale. Besides assessing oral expression, other items ask about the intern's quality of writing, ability to adapt messages to particular audiences, listening skills, and critical thinking and problem-solving ability. In addition, supervisors were asked for open-ended comments and what internship grade they would assign. A summary of the quantitative part of the field supervisory assessment is below. For 97-98 there was a 68% return rate, for 98-99 a 76% return rate, 00-01 a 73% return. Individual comments were largely positive, often exceptionally so. Not infrequently, supervisors commented that they would hire the student if a position came open. In the 97-98 there were occasional concerns expressed about students' writing skills; this occurred less frequently in 98-99 and 00-01.

In sum, feedback from field supervisors suggests the department is doing a good job preparing its students for the wide range of professional positions that they might take upon graduation. In fact they think we are doing a better job than the seniors themselves think. The department regards the internship program as important to the quality of its program and intends to continue encouraging students to get involved in this distinctive type of learning.

L:\ir\Outcomes\OA0001\comm.doc

Last revision 01/28/03