2010-2011 Annual Assessment Report - Architectural Engineering Program

The curriculum goals of the unit, as currently stated in the catalog or other departmental documents, are as follows (per the catalog): 

“The educational objective of the architectural engineering bachelor's degree program is to have students acquire the broad knowledge and skills necessary to successfully pursue careers and become leaders in the building engineering profession, and be able to advance the state of the industry in any of the four core disciplines:

• building electrical and lighting systems
• building heating, ventilating, and air conditioning systems
• building structural systems
• construction and construction management”

Are these goals still accurate/current or do they need to be revised?

These overall goals are accurate and published in the 2010-2011 catalog. 

What is the unit’s schedule for reviewing the curriculum goals?

The Joint Evaluation Committees (JECs) in the Department of Civil, Environmental, & Architectural Engineering (CEAE) review these goals when they meet.  There are multiple JECs in CEAE representing each sub-discipline and each meets on a 3-year rotating basis.  The JEC is composed of at least three practicing engineers from disciplines covered by the particular JEC, preferably with experience employing program graduates, at least one faculty member from each discipline covered by the particular JEC, and at least one current undergraduate student from the discipline covered by the particular JEC. 

The building systems, structural, construction, and geotechnical JECs are relevant to the AREN degree.  The building systems JEC met in April 2009 and will meet again in spring 2012.  The Structural JEC met most recently in spring 2010, while the construction and geotechnical JECs met in spring 2011.  The findings and recommendations of the JEC’s are communicated to the Curriculum Committee who has the responsibility for recommending curriculum changes for approval by the faculty as a whole. The Department Executive Committee and Chair are responsible for allocating teaching, facilities and material resources to implement changes.

During the last review period, how has the department/program assessed how well it has accomplished its curriculum goals?

A number of assessment instruments are administered by the College of Engineering & Applied Science that gather data relevant to AREN.  These assessment tools and dates applied since the previous annual assessment report are:

• Freshman Survey (spring 2011)
• Senior Survey (data from December 2010 and May 2011 graduates) – assessment of program outcomes
• Post-Graduation Surveys (given to alumni approximately 6 months after they graduate; data from May/Aug 2010 and December 2010 available in this cycle)
• Alumni Surveys (data from summer 2011 surveys of 2006 graduates) – assessment of program objectives
• Internship Surveys (fall 2011)
• Employer Survey (spring 2011) – assessment of alumni toward achievement of outcomes / objectives

In addition, all graduating students are required to take the national Fundamentals of Engineering (FE) exam.  The results of this exam are released approximately 6 months later, so the fall 2010 and spring 2011 results are available.  These results help us assess both our program outcomes and objectives.

The Faculty Course Questionnaires (FCQs) administered by the University are also used to assess individual courses.  In spring 2008 and onward, questions related to the program outcomes were added to the FCQs for selected required courses.  These data help us assess which specific courses are helping us meet our program outcomes.

Joint Evaluation Committees (JECs), described above, assess and evaluate the program objectives, courses in subspecialty curriculum, and outcomes.  In the 2010/2011 academic year the construction engineering and management (CEM) and geotechnical engineering JECs met as previously stated.

In addition, the curriculum committee in CEAE meets at least monthly and assesses the program. The curriculum committee reviews the feedback from the JECs, and independently reviews FE exam information and other assessment data.

What has the department/program concluded with respect to the outcomes of its undergraduate and (and if applicable, graduate) curriculum?

We have concluded that the AREN program is strong, with appropriate objectives and outcome goals that are being met and serve our students and the profession.  The senior design course is very successful. The large student enrollment in upper division CVEN courses that include both AREN and CVEN majors, continues to be a concern.  However, the number of incoming first year students into AREN has declined over the past few years and this enrollment decrease will reach the upper division courses in a couple of years.  The department will deal with future growth as freshman enrollment rebounds.

The specific data that has been collected and reviewed by our program is summarized below.

Freshman Survey

In spring 2011:only 9 AREN students responded, so caution should be used when generalizing the results.On average, the AREN students were fairly certain of their major (average 4.0 on 5 scale, 5 = completely certain).The students were also fairly certain that they would stay in engineering until graduation, averaging 3.8 of 5 (5=completely certain); 11% were not at all certain of this.The AREN students indicated that their understanding of engineering as a career increased from 2.78 before they enrolled to 4.00 ‘now’ (after about 5-7 months of college; where 1 = not at all clear and 5=very clear).The AREN students rated the quality of their experiences with 7 different aspects of the College of Engineering (staff, faculty, Dean staff, facilities, students, tutoring, and BOLD staff) at 3.25 to 3.60 (5=high).These data indicate good experiences among the students.

The top items that students reported would make their experience in engineering more satisfying were: more help / review sessions (67%), smaller classes (56%), and offer more free electives (56%).

For faculty advising, student rated six different aspects ranging from 2.00 (slightly satisfied) to 3.00 (satisfied).The weakest items were faculty interest and willingness to help and provided career advice; and the strongest element was faculty availability.First year students participate in a group advising forum with faculty rather than being assigned individual faculty advisors, so these rather low ratings are not surprising.For staff advising, the average ratings were higher, ranging from 3.00 (satisfied) to 3.89 (4=very satisfied).The weakest ratings were the extent to which the staff demonstrated interest and a willingness to help and provided career advice.The strongest area was the extent to which a staff advisor made them aware of academic options, enrichment opportunities, and extracurricular activities.

Senior Survey

The feedback to the senior survey includes 12 student respondents in Dec. 2010 (an 80% response rate) and 33 in May 2011 (a 77% response rate); these students may not respond to all of the questions on the survey.The data below will be presented as a weighted average of the results from the academic year.

There was high satisfaction with major choice, averaging 3.98 on a 5.0 scale, based on 42 total responses.Satisfaction with staff advising averaged 4.11 and faculty advising averaged 3.88 (on a 5-point scale; 4 = very satisfied).

The satisfaction with 4 aspects of curriculum ranged from 3.5 to 4.0 on 5.0 scale, based on the average of 42 responses (3 = satisfied, 5 = extremely satisfied).The students’ satisfaction with the availability of electives was rated at only 2.8.It is unclear if this lower rating is due to the few electives allowed or that electives are not offered.The accreditation requirements have limited the flexibility in the program and therefore the number of electives.The AREN curriculum includes no free electives.Most undergraduate technical elective courses in AREN or CVEN that students normally take are offered 1 semester per year.

For the capstone design course, the average ratings were 4.2 that the course reinforced concepts learned in other courses and 4.0 that the capstone prepared them for an engineering career (scale 1 to 5).

On average, the 45 students rated the importance of all 18 of the ABET criteria as 3.7 to 4.7 (3=moderate, 4=very, 5=extremely), indicating they were adequately informed of the importance of these aspects.

Students rated the achievement of 17 of the ABET criteria as 3.4 to 4.3 (3=moderate, 4=very, 5=extremely), indicating they felt adequately prepared.The only weakness with an average rating below 3.0 was “knowledge of contemporary issues” with an average rating of 2.8.This outcome had a similar rating by previous graduating seniors.We will continue to monitor “knowledge of contemporary issues” to see if the students continue to give low ratings to this topic, and if so incorporate more of this content into existing required courses.However, it is very close to moderately equipped and doesn’t appear to be a significant weakness.

The majority of the AREN students held internships, with 58% of the Dec. 2010 and 51% of the May 2011 graduates reporting they had at least 1 internship.Internship satisfaction averaged 4.3 (4 = very satisfied, 5 = extremely satisfied).Internship usefulness averaged 4.5 (4 = very useful; 5 = extremely useful).

Post-Graduation Survey

Twenty-one former students who graduated in 2010 responded to the survey about six months after graduation.Of those, 76% were employed, 14% were in graduate school, 5% were in the military, and 5% listed other.The average annual salary of all respondents was ~$51,000.The average overall satisfaction with their job was 3.7 (3=satisfied for now, 4=very satisfied) among 17 respondents.The relation of their job position to their major averaged 3.5 among 17 respondents (3=some relation to major; 4=much relation to major).

For graduate school, 14% were currently working on an advanced degree, 52% were considering pursuing an advanced degree, and 33% had no interest in pursuing an advanced degree.

Eighteen of the graduates (86%) indicated they would likely pursue professional engineering licensure (PE) in Architectural, Civil, and/or Mechanical Engineering.Eleven (52%) indicated that they would pursue LEED certification.

The curriculum is geared to graduate school, employment and certification, which seems appropriate given the career paths indicated in this survey.Very high success in employment of post graduates is indicated.

Alumni Survey

The alumni survey was administered in Summer 2011 to alumni who graduated in 2006.Eight of the 26 alumni contacted responded to all or part of the survey; a 31% response rate, although 1 of the respondents did not complete the entire survey.In terms of employment, 86% indicated that they were employed and 14% listed other; none were unemployed.The nature of work reported was: 57% design, 14% management, 14% service, and 14% others.The type of employers or businesses reported was: 43% consulting engineering firm, 29% other, 14% construction industry, and 14% civil engineering firm. Of those employed in Architectural Engineering, 33% were employed in construction engineering/management, and 17% each in illumination/lighting, structural, building energy systems, and acoustics; none reported HVAC (heating, ventilation, air conditioning).The extent to which their engineering education contributed to their success was rated on average as 4.00 (on a scale of 1 to 5; 4=high).

With regards to graduate studies, 43% were considering graduate studies and 57% had no plans for graduate studies.Of the degrees being pursued, 67% were interested in an MBA and 33% in an MS.With respect to continuing education, distance education or short courses, 50% of the students were considering these and 50% had no plans for these.All respondents had attended two or more professional conferences.Two alumni (33%) had already obtained professional licensing or certification, 1 planned to, 2 were not sure, and 1 said no.This included 2 PE licenses in Architectural Engineering, 2 EIT/FE, 1 LEEDS certification, and 1 other.

For all of the 11 questions pertaining to the program objectives, the responses averaged 3.5 to 4.6 that the objectives were both appropriate and achieved (on 5 point scale where 3 = agree, 4=strongly agree, and 5=very strongly agree).The average satisfaction with choice of major was very high among the 7 respondents at 4.6; all were satisfied or higher.

The alumni survey in 2011 did not indicate any significant weaknesses in the program.

Internship Survey

The fall 2011 internship survey had 31 respondents majoring in AREN.The survey had students describe what they did in summer 2011, and 52% worked at an engineering related internship, 29% had a job not considered an internship, and 19% attended summer school at CU.

Employer Survey

There were 49 employers who hire AREN majors and responded to the spring 2011 survey.These same employers also hire CVEN majors (43%), mechanical engineering majors (39%), electrical engineering majors (35%), and a variety of other disciplines.The respondents primarily represented consulting engineering firms (41%), construction firms (22%), or other (17%).Of the respondents, 83% had hired engineering graduates from CU.When CU graduates were compared to engineering graduates from other colleges, the average ratings ranged from 3.53 to 4.00 (on a 5-point Likert scale where 3 = about the same as other colleges, 4 = somewhat better than other colleges).The primary strengths of the CU graduates were: ability to apply knowledge of math, science, and engineering (4.0), ability to function on multi-disciplinary teams (4.0), and understanding of professional and ethical responsibility (4.0).Knowledge of contemporary issues was the lowest rated item, which agrees with some of the student feedback.The employer survey data indicates that CU students are strong relative to their peers.

FE Exam

All AREN students are required to take the Fundamentals of Engineering (FE) exam before they graduate.In fall 2010 a total of 21 CU AREN students took the FE exam; an additional 35 took the exam in spring 2011.The overall pass rate in Oct. 2010 was at a ten year historic low of 38%, which was lower than the national average of 66%.Spring 2011 performance was significantly improved, with an overall pass rate of 77% (the highest since Oct. 2006) compared to a national average for AREN majors of 70%.

Topics with weak performance on the general exam in the morning (based on CU students more than 6% below nationwide peers on percentage of correct answers to questions) were: ethics and business practices (fall 2010, -6.7%), engineering economics (fall 2010, -7.1%), statics (~-10%), dynamics (fall 2010, -8.3%), strength of materials (~-6.4%), material properties (fall 2010, -9.8%), and thermodynamics (fall 2010, -11%).

There is no Architectural Engineering subject exam in the afternoon, unlike most other engineering majors which have an afternoon exam in their field.The AREN majors from CU who took the general exam in the afternoon did better than students who took the civil engineering exam in the afternoon.Specifically, 10 of 21 students in fall 2010 took the general afternoon with a 50% overall exam pass rate.In spring 2011, 27 of 34 students took the general afternoon exam with an 85% overall exam pass rate.For the general exam in the afternoon, there was weak performance compared to national AREN peers on biology (-8% in spring 2011) and electricity and magnetism (-11% in fall 2010). Biology is not required nor accepted as an elective for CU AREN majors, so this result is not surprising nor of concern.Performance on electricity and magnetism matched national AREN peers in spring 2011; this area will continue to be monitored.

For the civil engineering exam in the afternoon, there was weak performance on hydraulics/hydrology (-14% in fall 2010; not relevant for AREN), soil mechanics and foundations (~-12%), environmental engineering (-7%; not relevant for AREN), structural analysis/design (-9% to -18%), and materials (-6%).

To improve student performance, faculty made a stronger push to get students to attend optional FE review sessions that were organized well in advance and generally held in the early evening.One concern is that some students do not believe that future engineering licensure is important to their career and therefore do not make an effort to do well on the exam.For example, the average rating by AREN students in the 2010/2011 Senior Survey on the question related to preparation for the FE was only 2.71/5. Among December 2010 graduates, 25% indicated that they had not prepared for the FE exam at all and 25% had only prepared a little.The results were better among May 2011 graduates, with only 9% preparing not at all and 33% a little.FE review sessions in spring semester are now integrated into the AREN senior design course.The better preparation appeared to have resulted in vastly improved pass rates.Therefore, we believe that our students have adequate knowledge but many simply to not take the FE exam seriously despite the fact that the average rating of the importance of becoming a PE for long term career aspirations was rated as 3.76 (3 = moderately important, 4= very important).

Student Ratings of Courses on FCQs

Questions were added to the FCQs that students complete at the end of each course asking them to rate 20 different statements regarding the ABET A-K outcomes and architectural engineering program specific criteria.These questions were added to selected courses.Students rated on a scale of 1 to 6 or not applicable (NA) statements such as “the course improved my ability to design and conduct experiments.”In fall 2010 and spring 2011, these questions were on the FCQs for 6 AREN courses (thermodynamics, fluids/heat transfer, illumination 1, mechanical systems for buildings, electrical systems, senior design) and 7 CVEN courses that AREN students were required to take on the “old” block plan that was in force when the majority of the AREN students started the program (geomatics, analytical mechanics 1, mechanics of materials 1, structural analysis, introduction to construction, steel design and reinforced concrete design).Using the academic year data for the 13 courses in combination, the maximum scores for each of the 20 outcomes ranged from 4.6 to 5.8.This indicates that all of the required outcomes are being satisfactorily covered by the curriculum.The strongest outcomes based on the highest average student rating of 4.9 to 5.0 were knowledge of math, science, and engineering; ability to apply knowledge in a specialized area related to architectural engineering; and the ability to solve engineering problems.

JEC Reviews 

The construction engineering and management (CEM) JEC met on February 16, 2011.Prior to the day-long meeting at CU, the faculty prepared an extensive self-study document.The 12 individuals who served on the JEC committee included 5 CU faculty/senior instructors, 4 individuals working in the CEM industry for consultants and contractors, and 3 CU students (2 senior civil engineering students and 1 former AREN student currently in the master’s program). The committee first reviewed the program objectives and outcomes, but did not have any specific comments.There were 8 recommendations from the JEC, which related to the CVEN and/or AREN curriculum: reduce class size, increase CEM faculty, provide structured internships, provide in-class lab time, develop CEM classroom space, maintain or expand surveying experience, teach management skills, integrate safety into the curriculum. The CEM faculty discussed these issues, and determined that two items could be accommodated within existing courses (management and safety).Some changes recommended by the CEM JEC might impact non-structures emphasis areas, such as expanding surveying/geomatics.This recommendation was in direct opposition to the structures JEC the previous year that recommended removing that course requirement.After a discussion in the curriculum committee, it was decided to retain the current course.Other comments cannot be readily accommodated due to resource limitations.

The geotechnical engineering JEC met on February 4, 2011.The seven committee members included 3 industry members (CDOT and 2 consultants), 2 CU faculty members, 1 CU BS student, and 1 CU BS/MS student.The JEC carefully reviewed all of the undergraduate geotechnical courses and had 10 comments.But AREN students only take one of these courses in the CEM or structural concentrations, CVEN 3708 Geotechnical Engineering 1. The only relevant comment from the JEC is to include expansive soils information, which the faculty determined will be enhanced in CVEN 3708.

Faculty

Assistant Professor Brent Protzman left the building systems faculty to pursue industrial opportunities in summer 2011.Senior Instructor William Yearsley left the construction engineering and management group in summer 2011 to pursue industrial opportunities.Faculty searches are being conducted to fill these positions and a new instructor was hired and started in Fall 2012.

Curriculum issues

Curriculum issues are discussed by the faculty at large.All changes in the courses that comprise the curriculum must be approved by the faculty.The AREN curriculum went through a major curriculum change that took effect starting Fall 2009.Students who started on this new curriculum have not yet graduated.No new changes in the AREN curriculum were made in 2010/2011.

More extensive data analysis was completed in support of writing the ABET self-study completed in June 2011. The CEAE faculty concluded that most assessment measures indicate a robust program that is meeting its curricular goals. 

What changes in the curriculum or in major requirements have occurred as a result of your assessment of your undergraduate program?

The increased assessment data that is available from the College-wide surveys and the outcomes assessments by the students at the end of our courses helps the JECs to accurately assess our program.  The AREN curriculum just underwent significant changes, as summarized above.  It is too early to tell if these changes will result in improved outcomes, but the impact of these changes should be evident in future assessment cycles.