2010-2011 Annual Assessment Report - Computer Science Program
The curriculum goals of the unit, as currently stated in the catalog or other departmental documents, are as follows:
“The mission of the Department of Computer Science of the University of Colorado at Boulder is to provide an excellent undergraduate education that provides mastery of computer science principles and practices and prepares students for careers as computer specialists as well as for graduate study in computer science.
The educational objectives of the computer science BS degree are to produce students who three to five years after graduation:
- are prepared to be valued individual contributors in a software-oriented organization, to be programmers and designers in an entrepreneurial pursuit, to lead small projects and generally begin preparation for a management career, or to succeed in rigorous postgraduate programs.
- are able to focus their careers on pure computer science technology or to bring computer science expertise to a companion discipline.
- are prepared, where appropriate, to specialize in a broad spectrum of computer science sub-disciplines, ranging across formal computer science (e.g., computational science, bioinformatics, and theory), cognitive science (e.g., human/machine learning, human-computer interaction, collaborative work, and human language technologies), and core computing (e.g., systems, networks and software engineering).
A list of program outcomes for computer science graduates has been developed that satisfy the requirements of ABET (the Accreditation Board for Engineering and Technology) in the Criteria for Accrediting Computing Programs for general computing programs (ABET Criterion 3) and for computer science programs in particular (ABET Criterion 9). As defined by ABET, outcomes are “statements that describe what students are expected to know and are able to do by the time of graduation.”
The program outcomes of the CS B.S. degree are, that at the time of graduation, CS graduates will possess:
- an ability to apply knowledge of computing and mathematics appropriate to the discipline
- an ability to analyze a problem, and identify and define the computing requirements appropriate to its solution
- an ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs
- an ability to function effectively on teams to accomplish shared computing design, evaluation or implementation goals
- an understanding of professional, ethical, legal, security and social issues and responsibilities for the computing profession
- an ability to communicate effectively about computing topics with a range of audiences
- an ability to analyze impacts of computing on individuals, organizations, and society
- a recognition of the need for and ability to engage in continuing professional development
- an ability to use current techniques, skills, and tools necessary for computing practice
- an ability to apply mathematical foundations, algorithm principles, and computer science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices
- an ability to apply design and development principles in the construction of software systems of varying complexity
Are these goals still accurate/current or do they need to be revised? What is the unit’s schedule for reviewing the curriculum goals?
These goals are still accurate and current to our program. As ABET updates their criteria on what amounts to a yearly basis, the undergraduate committee reviews these curriculum goals once a year during the Spring semester.
During the last review period, how has the department/program assessed how well it has accomplished its curriculum goals?
Available assessment data collected during this reporting cycle are reviewed in the following sections.
May/August & December 2010 Post-Graduation Survey (32 respondents):
The College of Engineering and Applied Science administers a survey that targets alumni approximately six to eight months after they graduate to determine if they are employed, pursuing graduate school, etc. The CS department had 32 respondents across the PostGrad survey time periods listed above. Twenty-eight of the 32 CS respondents were employed, with an average salary of $64.3K. One respondent has already completed an advanced degree, two respondents said they are currently attending graduate school, and 15 are considering graduate school in the future. Only two respondents indicated that they were unemployed and looking for a position. Twenty-five of the respondents were employed in full-time engineering positions. All of the respondents (100%) were satisfied with their jobs, with 50% saying they were extremely satisfied, 28.6% saying they were very satisfied, and 21.4% "satisfied for now". The majority (82%) stated that their jobs were completely related to their CS major, with another 14.3% indicating that their current position had much relation to their CS major.
Summer 2011 Alumni Survey (9 respondents, although 2 did not complete the survey):
The College of Engineering and Applied Science administers a survey each summer that targets alumni approximately four years after graduation. Nine CS alumni (nine men) began the survey and seven completed it.
Satisfaction. The respondents’ overall satisfaction with their CS major was high—4.29 on a scale of 1 (not at all satisfied) to 5 (extremely satisfied). Indeed of the seven respondents who answered this question, the majority (6/7) were either extremely satisfied (3/7), or “very satisfied” (3/7) and the remaining respondent was “satisfied” (1/7). All nine respondents felt that their CS major helped them to meet their educational goals with five saying “yes, definitely” and four saying “yes, somewhat.” All seven respondents indicated they would recommend CU and the College of Engineering and Applied Science to a friend.
Employment. Of the seven respondents who completed the entire survey, all seven reported being employed full-time; one was attending graduate school. The majority was employed in Colorado, one was a graduate student at the University of Pennsylvania and one works as a software developer in Wisconsin. The cross-disciplinary nature of computer science is on display when you look at the nature of their employers: only two were directly working in the computer/IT industry; others worked for a consulting firm, health care, and other industries, and one was employed in academia. In addition, two of seven respondents indicated that they have led small teams creating software systems (note: the number of respondents change on some of these metrics because two respondents to the entire survey did not complete all of the questions, answering some and ignoring others), and three of seven respondents indicate that they specialize in one or more sub-areas of computer science; the most common specializations are operating systems and “other” (two out of seven respondents) followed by human/machine learning, computer architecture, networks (one respondent each). While this is an admittedly small sample size, it provides strong evidence that our program is meeting its program objectives: With respect to objective 1, all graduates were employed, three went to graduate school (with another one considering applying to graduate school), and two have assumed leadership roles in software development settings. With respect to objective 2, many of the respondents are working in companion disciplines, while others are working squarely within a computer science-related field. With respect to objective 3, three of seven respondents indicated that they are specializing in one or more sub-areas of computer science with four sub-areas called out explicitly. Finally it should be noted that the work of our graduates is highly valued as reflected in the jump in their reported salaries. The majority of respondents graduated with a B.S. degree in 2006 and reported an average initial salary of $64K; four years later, their average current salary is $91K.
ABET Outcomes. With respect to ABET concerns, we asked our alumni to reflect on how well our program prepared them with the skills listed in our ABET objectives. All of the outcomes are rated above average (range: 3.43 to 4.00 on a 5 point scale) by the seven people who responded to these questions with the exception of skills related to objective g (analyze impacts of computing on individuals, organizations and society). This outcome received an average score of 2.86 that we will seek to improve over time with an expanded range of courses in our Human-Centered Computing track and our professional development offerings at the undergraduate level. With respect to professional development, over half of the respondents are considering (16.67%) or have gone on (50%) to seek additional professional or graduate education, with the degrees being sought in CS (MS or PhD degree) and business (MBA). Overall, these results indicate that our degree program is effective in meeting the program outcomes listed above.
December 2010 Senior Survey (7 respondents):
The College of Engineering and Applied Science administers a survey to all graduating seniors; seven CS majors responded in December 2010 (for a response rate of 64%), although one of the seven did not complete the entire survey.
Success Factors. Students were asked what factors contributed to their success in earning an engineering degree. The top three factors cited were the support of friends (86% of respondents), real world opportunities (86%), and the support of family (71%). Other success factors cited by a majority of respondents were working with others (57%), natural ability (57%), future rewards (57%), and self-motivation (57%).
Satisfaction. Of the six respondents who answered a question asking how satisfied they were with their choice of major, two were "extremely satisfied" and one was "very satisfied" with the remaining three respondents indicating that they were “satisfied” for an overall rating of 3.83 which falls between “satisfied” and “very satisfied.” Two of seven respondents said that their CU experience "definitely" met their educational goals while the remaining five said that it had "somewhat" done so. When students are asked if they would recommend their major to a friend, they responded with an average score of 4.67 out of 5, which falls between “yes with some reservations” and “yes with no reservations,” which compares with an average score of 4.67 for CS students rating CU Boulder as a whole. In addition, 67% of respondents felt their time in our major made them “highly prepared” for a career in engineering while the other two respondents indicated that they were “prepared” or “well prepared” for such a career. These responses indicate that 100% of our graduating seniors in December 2010 felt ready for a career in engineering. Six of our seven respondents gave the department’s computing facilities an average score of 4.67 out of 5, which falls between ratings of “good” and “excellent”. Six students responded to a question asking how satisfied they were with the advising they received and gave our staff advisor an average score of 4.33 out of 5 that falls between “very satisfied” and “extremely satisfied.”
Employment. Four of the seven graduating seniors had secured full-time employment after graduation, with three of those jobs located in Colorado. Two of those four students indicated their starting salary for these positions with the average between the two being $73.5K. Three of the four students with jobs reported that those jobs were “completely related” to their CS degree while the remaining student indicated that their job was closed related to their CS degree. Two students indicated they would be looking for jobs but had not yet started their job search. Two students indicated that they would eventually pursue graduate school, one indicated that they earned their MS degree here at CU alongside their BS degree, and the remaining students indicated that they were unsure if they would pursue graduate school in the future.
ABET Questions. The seven respondents gave high ratings (3.5 or above) to 14 of our 18 ABET-related questions. This result provides strong evidence that our students believe our program is meeting its program outcomes. The weakest rating was a 3.2 (“moderately well prepared”) with respect to written communication skills. Our program continues to require the highest amount of humanities credits of all departments to address this aspect of student preparation.
Highlights/Worst Aspects. Along with favorite classes and teachers, students also mentioned internship opportunities and the CU environment among highlights; among worst aspects, students mentioned a few bad professors (across multiple departments), high workloads in classes, and staying motivated in classes that they found uninteresting. Several students specifically mentioned the need to improve the CS database class and we will look at what we can do to improve this class in response.
May 2011 Senior Survey (23 respondents):
The College of Engineering and Applied Science administers a survey to all graduating seniors; 23 CS majors responded in May 2011 (for a response rate of 45%; three of the respondents did not complete the survey).
Success Factors. Students were asked what factors contributed to their success in earning an engineering degree. The top three factors cited were the support of friends (65% of respondents), working with others (65%), and the support of family (70%). The other success factor cited by a majority of respondents was natural ability (57%).
Satisfaction. Of the 20 respondents who answered a question asking how satisfied they were with their choice of major, three were "extremely satisfied" and 12 were "very satisfied" with the remaining five respondents indicating that they were “satisfied” for an overall rating of 3.9 that falls just shy of the “very satisfied” rating. Five of 22 respondents said that their CU experience "definitely" met their educational goals while the remaining 17 said that it had "somewhat" done so. When students are asked if they would recommend their major to a friend, they responded with an average score of 4.1 out of 5, which falls between “yes with some reservations” and “yes with no reservations,” which compares with an average score of 4.21 for CS students rating CU Boulder as a whole. In addition, 11% of respondents felt their time in our major made them “highly prepared” for a career in engineering while 74% indicated that they were either “prepared” or “well prepared” for such a career. These responses indicate that 84% of our graduating seniors in May 2010 felt ready for a career in engineering with the remaining 16% indicating that they were “slightly prepared.” Nineteen of our 23 respondents gave the department’s computing facilities an average score of 3.47 out of 5, which falls between ratings of “fair” and “good”. Twenty students responded to a question asking how satisfied they were with the advising they received and gave our staff advisor an average score of 4.2 out of 5 that falls between “very satisfied” and “extremely satisfied.”
Employment. Eight of the 23 graduating seniors had secured full-time employment after graduation, with six of those jobs located in Colorado. One additional student reported receiving a job offer for full-time employment but had not yet accepted it at the time of the survey. Four of those eight students indicated their starting salary for these positions with the average salary being $64.6K. Five of the eight students with jobs reported that those jobs were “completely related” to their CS degree while two students indicated that their jobs were closed related to their CS degree. Seven students indicated that they would eventually pursue graduate school.
ABET Questions. The 19 students who responded to questions related to ABET (which are questions that help our department when it seeks accreditation) gave high ratings (3.3 or above) to all 18 of our ABET-related questions. This result provides strong evidence that our students believe that our program is meeting its program outcomes.
Highlights/Worst Aspects. Along with favorite classes and teacher, students also mentioned research and internship opportunities, working in groups on projects, and the CU environment among highlights; among worst aspects, students mentioned disengaged professors (across multiple departments), high workloads in classes, and staying motivated in classes that they found uninteresting.
Fall 2010 Internship Survey (76 respondents):
Of the 76 CS students who responded to the internship survey, 35 of them reported participating in an engineering-related internship over the summer. The positions were held anytime between freshman and senior year, and most students had positions that were one to six months in duration. The vast majority of these students found their position useful/extremely useful in reaching their educational goals (93% rated 4 or 5 on a 5 point scale) and would highly recommend their internship to a friend (83% rated 4 or 5 on a 5 point scale). The average hourly salary reported by 25 of the students was $15.43 which corresponds to an average monthly salary of approximately $2700.
Fall 2010 FCQ Data:
Of the undergraduate class sections taught that semester, the average for the course on a scale of 1 (low) to 6 (high) was 4.4. With respect to instructors, the average on a scale of 1 (low) to 6 (high) was 4.8.
Spring 2011 Freshman Survey (6 respondents):
The College of Engineering and Applied Science administers a survey in the spring to students finishing their freshman year. Computer Science had six students respond to this survey, which corresponds to roughly 14% of the incoming students for our program during the 10-11 academic year. This percentage represents a steep decline over the proceeding year’s 47% response rate. We will explore ways to greatly increase the percentage of students responding to this survey in future years.
Understanding of Engineering: Our respondents indicated that their first year of classes contributed to their understanding of engineering as a field of work and study. All six respondents rated their understanding of engineering as a three out of five before they enrolled in classes; after one and a half semesters of classes these same respondents rated their understanding of engineering as a 3.8 out of five, nearly a one-point gain in that metric.
Reasons for Studying Computer Science at CU: More than any others, four reasons were cited for choosing computer science: interest or talent in the field (4/6), good salary or earnings potential (5/6), challenging and creative field (4/6), versatile and marketable career options (5/6). Other reasons cited by students were family influence (3/6) and the ability to contribute to society (2/6).
Most respondents indicated (4/6) they received the right amount of information (or close to that) about the college before enrollment; only one claimed to receive no information, which is surprising and likely an outlier given that no student has selected this option before in the time we’ve been tracking the results of these surveys.
Living Situation and Academic Support: Three of our respondents live on campus (50%), a significant drop from two years ago where the response indicated that 92% of our freshman lived on campus.[1] On average, they rate the living situation as supportive for an engineering student—3.67 rating on a scale of 1 (not at all supportive) to 5 (very supportive). Two of our respondents indicated engagement with the BOLD Center, with one of those respondents finding its services very helpful.
First-year classes: Thirty-three percent of the respondents (2/6) took a one-credit introductory course, split between the department’s freshman seminar class and the general college introductory course. For one of these students, the intro course confirmed their choice of major, while the other student discovered that their interests lie elsewhere. (Again the sample size of this survey is too low to be considered representative of our student body as a whole.)
There was significant variation in the amount of studying outside of class that respondents reported, ranging from 0-5 hours/week to more than 15 hours/week during the first semester. There was a slight shift towards more studying in the second semester with the lower bound now being at least 6 hours of studying per week. Two of the students reported taking advantage of regular tutoring.
Most students reported studying in groups with only one student reporting that they always study alone. Four of the six respondents reported working with other students in a team, and their satisfaction (4.0) and perception of benefits (3.83) were above par (on a scale of 1 to 5).
Quality of Experience: Respondents were asked to rate their quality of experience with six items on a scale of 1 (low) to 5 (high). There was significant satisfaction with the college’s facilities (3.60), staff (3.75) and faculty (3.8). Out of all responses in these three areas, there were just two ratings below a 3. Satisfaction was also strong in their satisfaction with other students (3.80).
The respondents unanimously said that they would recommend CU’s College of Engineering and Applied Science to a friend, though one had strong reservations and four had some reservations.
As reported above, the overall experience with faculty was very positive, and one student explicitly mentioned that taking CSCI 1300 (CS 1) was the best part of his college experience. Furthermore, all of our respondents report seeking the assistance from a TA or faculty member outside of class, with one student reporting that they sought such assistance at least once a week on average. We think this type of interaction is important and we continue to work on ways to better encourage and reward faculty/student interaction outside the classroom in the first year.
Finances: Two of the six respondents had an outside job beyond being a full-time student; the reasons given for working were exclusively financial. In one instance, the job was equivalent to part-time work at 21+ hours per week in the fall semester but decreased to 16-20 hours a week in the spring semester, and the other student worked only 1-5 hours per week in the spring. Two of those positions were on-campus jobs, while the third was off campus. Five of the six respondents received some financial aid, but they were not fully satisfied with the level of support, averaging 2.40 (between slightly satisfied and satisfied) on a scale of 1 to 5.
Spring 2011 FCQ Data:
Of the undergraduate class sections taught that semester, the average for the course on a scale of 1 (low) to 6 (high) was 4.9. With respect to instructors, the average on a scale of 1 (low) to 6 (high) was 5.2.
Additional Information
We had one new tenure-track faculty join our department during the 2010-2011 academic year. Aaron Clauset joined the department in Fall 2010; his primary research explores the physics of complex systems, focusing on methods for understanding the structure and function of networks on the origins of morphological diversity in biology and on patterns in terrorism and warfare.
The department’s efforts on obtaining accreditation during the during the 2008-2009 academic year was rewarded when the department received full accreditation in summer 2010, backdated to Fall 2008.
What has the department/program concluded with respect to the outcomes of its undergraduate curriculum?
The department is pleased with the results contained in the assessment data. While there are certain negative aspects identified, overall the majority of our students are satisfied with their degree and the advising they receive while here. The overwhelming majority of our students moves on to get jobs related to their CS degree and enjoy what they do. In addition, the data from our surveys indicate that we are on track for meeting our program objectives and program outcomes. We attribute this to the hard work we invested preparing for ABET accreditation and the overall quality of our degree program.
What changes in the curriculum or in major requirements have occurred as a result of your assessment of your undergraduate program?
During the 2010-2011 academic year, we made small adjustments to the undergraduate degree, revising the requirements of the senior thesis to make the reporting aspects of that capstone experience more streamlined. In addition, we officially shut down the “health informatics” portion of the computational biology track. This decision occurred due to a lack of resources to teach the courses required by students in that track. All students that were in that track at the time were allowed to complete their work and graduate with a degree in that track, but no new students were allowed to join that aspect of the track.
Overall, we are quite pleased with the state of our undergraduate program, especially with the success of our accreditation efforts.