PBA Home > Institutional Research & Analysis > Outcomes Assessment > Unit Summaries > Department of Physics

Department of Physics
Last updated 2/18/2003

Knowledge and skill goals for this undergraduate degree program are recorded in the most recent CU-Boulder catalog. In some summaries of assessment activity, goals are referred to by number (e.g., K-2 is knowledge goal 2).

Assessment activity in 2001-2002
Activity in 2000-2001
Activity in 1998-1999
Activity prior to 1998

Actvities in 2001-2002

This period's assessment activities centered on learning in some of the large undergraduate physics classes. In particular, a systematic study of the online homework system CAPA (Computer- Assisted Personalized Assignments) was carried out in the calculus-based introductory physics course during the fall 2001 semester by Andrea Pascarella as part of her dissertation research in the physics department. This study, which was supervised by Prof. Valerie Otero of the School of Education and Prof. John Taylor of the physics department, looked at the effects CAPA had on student learning and attitudes.

The (about 500) students in this class were split into two groups. One group was initially assigned to CAPA; the other group was assigned to traditional homework. At mid-semester the groups switched identities (the students who began the course using CAPA had to complete traditional homework).

Exam scores and Force and Motion Concept Evaluation gains showed no statistically significant differences between the groups. Written quizzes and exams were collected from a smaller sample of students and analyzed using a problem-solving rubric. No statistically significant differences in the problem solving abilities of the groups were seen. Student opinions about the effect each homework type had on their learning were elicited. Students with non-expert-like epistemologies felt that CAPA was a better learning tool while students with expert-like epistemologies believed that traditional homework was a better learning tool.

Problem-solving interviews were conducted weekly with 9 students. From the analysis of this data a problem solving characterization of students using CAPA and traditional homework was inferred. Four types of problems solvers emerged - the CAPA Thinker, Traditional Thinker, CAPA Guesser, and Traditional Guesser. Thinkers tend to have expert-like epistemological beliefs. Guessers generally have non-expert-like epistemologies. On quantitative problems traditional homework promoted metacognitive processes in the Traditional Thinker and CAPA hindered self-evaluation among CAPA Thinkers. On qualitative problems, the opposite was observed to occur. When the students switched homework types at mid-semester it was expected that CAPA Thinkers would become Traditional Thinkers and vice versa. Similar results were expected among Guessers. However, there were students who "switched" from Traditional Thinkers to CAPA Guessers as well as students who "switched" from CAPA Guesser to Traditional Thinker. Since students and instructors prefer the use of CAPA over traditional homework, and since the use of CAPA does not seem to have any significant deleterious affects on learning outcomes (as measured by exams) the Physics Department plans to continue to use CAPA in most of the large introductory courses that it offers.

Activities in 2000-01

As part of the physics department's ongoing assessment activity we focused our current outcomes assessment on issues of interest to our graduate student population. These issues include comprehensive exams, coursework, quality of life and life after obtaining an M.S. or Ph.D.

Three surveys were used as the primary source of data collection. Students who graduated between 1993 and 1999 were surveyed via the mail about their current efforts and their opinions of various aspects of the Department. The Office of Planning, Budget and Analysis aided with the administration of this survey. In addition, current graduate students were asked to complete two different surveys. Both of these surveys were administered by the physics department. The first survey focused specifically on opinions about the comprehensive exams. In conjunction with the program assessment that took place this year within the physics department a second survey was given to the graduate students. The response rates to these surveys were 38%, 63%, and 49%.

For the past few years the format of our comprehensive exams has been of some concern to the department. An overwhelming majority of graduate students (92%) believe that the written portion of the exams is useful for learning even though most students (70%) think that the exam either is or might be an unnecessary source of pain and suffering. When asked about the written exam's usefulness for preparing for a career in physics, 49% of respondents rated the exam as either somewhat or very useful. The majority of students (85%) feel that the written exam might be better in a different format. Only 72% of students feel that the oral portion of the comprehensive exams is useful for learning. Compared to the written portion of the exam, fewer respondents (45%) feel that the oral exam either is or might be an unnecessary source of pain and suffering. When asked about the oral exam's usefulness for preparing for a career in physics 64% of respondents rated the exam as either somewhat or very useful. The students who failed part or all of the comprehensive exams were asked about the feedback, advice, and support that they received. A majority of these students (82%) felt that they did not receive adequate feedback. The overwhelming complaint expressed by these students was the inability to view their exams to see what errors were made.

In response to faculty concerns about the comprehensive exams the department voted to change the structure of the exams. The new format consists of two elements. First, students must receive passing grades in the core graduate classes offered in the department. Second, each student will be required to write a formal paper summarizing a broad research topic and present a talk on this topic to a committee of examiners. These examiners will ask the student questions related to the research topic as well as general physics. The student will be evaluated on his/her research paper, presentation, and ability to answer the questions posed by the examining committee. It is important to note that one of the committee members will be the student's advisor. If the student should fail it is the duty of his/her advisor to discuss whatever problems exist and help the student to improve.

Our current graduate students (~90%) find their courses useful and reasonably challenging without being unnecessarily difficult. The instructors for these courses were rated as either effective or very effective by the majority of graduate students (90%). However, it is important to note that only 53% of our recent graduates found the classroom instruction to be either satisfactory or very satisfactory. These graduates also indicated that they did not receive enough preparation in optics, experimental physics, and computing. The current graduate students were asked about what changes they would like to see regarding graduate coursework in the department. Of the students who responded to this question, the majority stated that they would like to see more electives/specialized courses.

We were particularly interested in exploring graduate student opinions of the two-semester Mathematical Methods course. While most students enrolled in the first semester of the sequence only a little more than half went on to enroll in the second half. Students were asked to rate the usefulness of the course. The results are displayed in the table below.

The effectiveness of this course varies with the instructor. The more useful courses included specific instruction on mathematical topics useful for the core graduate courses. These results indicate that it might be useful to standardize the instruction in this course to include a fundamental foundation of knowledge in the same way that the other graduate core classes are standardized.

The current graduate students were asked about their interactions with their advisors and about the typical length of their workweek. Most students work between 40 and 60 hours a week. Extremely few students spend more than 60 hours per week working. It is interesting to note that 70% of the respondents feel that they are putting in the right amount of time necessary to complete their educational goals while 20% feel that they should work more. With respect to one's personal life, 20% of the respondents feel that they are working too much while 58% feel that working roughly the right amount each week.

Overall, most graduate students find their advisor either helpful or very necessary. The majority of students feel that their advisor demands an appropriate amount of time from them, provides enough feedback, and is available a sufficient amount of the time. Most recent graduates (63%) were either satisfied or very satisfied with their thesis advising. However, it is noteworthy to mention that only 43% of the graduates who received an M.S. were satisfied or very satisfied with their thesis advising.

Our recent graduates are employed in the following sectors: industry (34%), academia (21%), government lab (16%), and other (28%). This is different from just seven years ago when most graduates (82%) could be found in academia or government labs. In retrospect our recent graduates (72%) were either moderately satisfied or dissatisfied with the career advising they received. The M.S.-only graduates were less satisfied. Current graduate students (72%) do not feel that they get enough information on career/job options and they (92%) would like to see new methods for bringing job related information to them. The job market our graduates are entering has changed within the last decade. The majority of students are no longer going into academia or government labs. With the changing times we need to change our approach to career advising.

Activities in 1998-99

As an outcomes assessment activity relating to the CU Physics non-major undergraduate offerings, a first analysis of Medical College Admissions Test (MCAT) scores earned by CU-Boulder students was undertaken.

The table below shows CU-Boulder students' performance on the physical science portion of the MCAT and the corresponding national averages for the years 1994, 1996, 1997 and 1998.

Table. Mean Physical Science scores on the MCAT by year for CU-Boulder medical school matriculants and applicants, and corresponding national means (maximum score is 15).

It is clear that both the CU-Boulder applicants and matriculants exceeded the national averages for each of the years sampled (where data is available). This speaks well to the level of instruction received by CU-Boulder students in both physics and chemistry. No time-trends are clear. This activity should be revisited in the future, however, as pedagogy changes, for example, with the introduction of computer-assisted physics homework assignments (CAPA) in the general physics courses taken by pre-med students.

*to medical schools nationwide

Activity prior to 1998

Since 1989-90 the department has evaluated these goals by analyzing all term papers and project reports written by students in senior-level capstone courses. From 1989-90 through 1992-93 the capstone and critical thinking course was PHYS 4420 (Atomic and Nuclear Physics), taken by seniors in their final semester. In the Spring of 1996 and 1997 this practice of assigning term papers which then needed to be presented orally to a group was continued. Lab projects in PHYS 3330 (Junior Laboratory) were also analyzed during this period. In 1993-94 the department began two additional critical thinking courses, PHYS 3340 (Introduction to Research in Optical Physics) and PHYS 4430 (Introduction to Research in Modern Physics). The latter course is also listed as PHYS 5430, taken by a small number of graduate students.

All Arts & Sciences-Physics and most Engineering-Physics majors take at least one of the three critical thinking courses. PHYS 3340 and 4430/5430 involve experimental laboratory projects. Since 1994-95 term papers and oral presentations from these courses and the second semester junior laboratory PHYS 3330 were evaluated together. This practice of blending three courses in optics and modern physics taught by three faculty members was continued in 1996-97. The juniors were thus exposed to the more mature work habits and reporting skills of the graduate students. The impact of the very recent preparation of the Bose-Einstein condensate was seen in 1995-96 by the efforts of several student groups in ambitious atomic trapping experiments.

The overall impression of the written reports for PHYS 4420 over the years was very positive; the major weakness was the clarity and style of the writing although the 1991-92 committee felt that even this had improved. Subsequent years' assessments found some cases of fuzzy wording, but more of clearly stated thinking. Most reports included very good surveys of relevant research literature and indicated that the students had generally assimilated the physics even though the technical details were sometimes beyond their abilities. In 1997, the instructor in Physics 4420 requested an initial draft that he went over with the student. This led to far better structured term papers but the quality of the work remained mixed.

In 1991-92 about 30% of the students in an upper-level laboratory course taken by nearly all physics majors completed a survey related to the department's knowledge and skill goals. The respondents were generally satisfied with their training, said that they learned a great deal, and felt that the writing aspects of the lab were useful, although some said that the course should teach more structured scientific writing.

In 1992-93 a more comprehensive survey of physics alumni was conducted as part of a systematic program review. About 75% of the alumni responding were employed in jobs related to their physics training. On the whole they rated their undergraduate preparation well, with most ratings in the good-to-excellent range. There was some sense that the program should incorporate more practical training and better academic and career advising, but it was clear that these alumni thought their program's overall quality was high.

These surveys mentioned time-management skills as an unmet need; this has also been recognized by the lab instructors during 1995-96. They suggest that lab teams working on projects be required to give weekly progress reports to develop this skill.

In 1990-91 the department began monitoring the GRE scores of students applying to graduate school as a way to further evaluate the performance of the best seniors. Results have been mixed. CU-Boulder physics majors who take the GRE are decidedly superior to comparable students nationwide in general analytic and verbal skills, averaging at or above the 75th percentile. Their scores on the specialized part of the exam are distributed similarly to those of the national sample of physics majors, indicating that CU-Boulder students applying to graduate school perform at least adequately compared to their peers. The outcomes committee notes that relatively few of our students take the advanced physics part of the GRE. This may be related to the current widely-advertised weak employment situation for academic positions in physics. This situation has continued to the present date.

Course-taking patterns support this interpretation. For example, in spring of 1995 two of three senior-level electives were cancelled due to low student interest. The one that did generate sufficient enrollment was a course in solid state physics, a career field particularly attractive to students planning to go into industry or other applied work instead of to graduate school. The theory course in nuclear and particle physics was taught in the spring of 1996 to a fairly large class, and used, as designed, for this assessment. In 1997 the situation in all these courses remained similar.

The department has made curricular changes almost continuously. PHYS 2170 (Foundations of Modern Physics) for sophomore majors was introduced in 1991-92 along with PHYS 2140 (Methods of Theoretical Physics), a math-methods course for majors that has a significant computer physics component. There were plans to further emphasize writing and logic in freshman and sophomore laboratory courses. The 1992-93 outcomes report described a major attempt to modernize the curriculum and make it more flexible so that undergraduates could have more research experiences. Our computer lab, which was started in 1991 has been heavily used by the students. Homework in courses assign problems that make use of computers.

Physics 2000 is a major new initiative in the Physics department. It will attempt to use the latest computer simulation developments, and the Web, to develop a visual approach to the physics being taught in the classroom. It will attempt to extend these "simulation courses" to the high schools in the state. Various workshops, including regional high school physics teachers, have taken place to inform everybody of this activity.

Two areas have seen marked increases in involvement by the students. Physics honors theses have increased markedly since a few years ago. A survey of the faculty indicate that in the present period there are an average of eight students doing honor thesis at any one time. The students' work in these activities has been described by the faculty on the average as excellent. Secondly, student involvement in research under the University Research Opportunities Program (UROP) has increased markedly. This is an excellent avenue for undergraduate students to work closely with the faculty; participating in research activities allows the student to interact with graduate students, with research associates, and with the faculty member, and allows him or her to observe how research is done and to apply what has been learned in the classroom.

These new areas of activity have added a new dimension to the students instruction. They are now graduating with a reasonable amount of research experience and many of them graduate with an honors thesis. These new areas of involvement need to be evaluated.

Some graduating students have been interviewed to hear their assessment. Some of them are now participating in a mentoring program for undergraduates. They find this effort gratifying and educational. The involvement in research (UROP) is also a good aspect of our present program.

Index of unit summaries

l:\ir\Outcomes\OA0001\physics.doc, OA0102\physics.doc

Last revision 02/18/03