Map A to Z Index Search CU Home University of Colorado
 
  Strategic Planning Institutional Research and Analysis Campus Budget and Finances About PBA

PBA Home > Institutional Research & Analysis > Outcomes Assessment > AOC meeting notes 12/11/00

Assessment Oversight Committee (AOC) meeting notes 12/11/00

Attendees:

Michael Grant, chair
Gordon Brown
Sam Fitch
Stephen Jones
Padraic Kenney

Merrill Lessley
Michael Main
Ronald Melicher
Lou McClelland, staff
Perry Sailor, staff

Absent:

Shelley Copley
Elease Robbins

James Sherman
Kumiko Takahara

Handouts:

  1. UCB Graduation-year assessment plan, Quality Indicator 16, approved by CCHE, 11/00.
  2. NCA suggestions and requirements for assessment at UCB, from 4/00 site visit report.
  3. PBA unit tracking, a one-page listing of academic units with date of last assessment report, date of last substantive assessment update, indicator of strong assessment procedure, date next report due, unit assessment coordinator, assessment budget 98-99, degrees 98-99, assessment $/degree 98-99, assessment $/degree 9596-9899.
  4. Photocopy of main undergraduate outcomes assessment web page
  5. Assessment methods used by academic departments and programs at UCB.
  6. Comments from departments on ETS Major Field Tests. (not discussed)
  7. Assessment budget breakdown for 2000-1, by unit, from Mike Grant, with info on what the money was spent on.

Grant, introduction

  • Welcome, thanks
  • Three reasons why AOC forming now
    • NCA (North Central Association) recommendation, in their 10-year accreditation site visit report, for more action on assessment, and required 2003 update report. The chancellor has delegated responsibility for the action and report to VC DiStefano, who in turn delegated to Mike Grant, who has called the AOC together. See handout 2.
      • NCA mentioned especially the lack of an oversight committee, a need for more assessment of graduate-level education, and a need to demonstrate faculty involvement in and use of assessment. Overall, NCA says that we can't demonstrate what's been done, like the plant collector whose specimens don't make it to the herbarium so are of little utility.
    • CCHE quality indicator system requirement that each institution submit a "graduation year [senior] assessment plan." The UCB plan was approved in November 2000; the committee will oversee implementation. See handout 1.
    • Conviction that assessment is important and can lead to better units, better education for students, and a better institution. Consistent with national trends and emphasis in assessment.
  • UCB and UCB units have been doing assessment over the last 10 years, but there's lots of variance. The intent of the renewed action is not uniformity, but benefit to units and meeting requirements.
  • The AOC will focus on undergraduate learning in the major discipline first because this is where most activity has occurred already and because this is the focus of the CCHE requirement. Later (but well before the 2003 NCA deadline) the committee will address assessment at the graduate level, and of undergraduate "general education" areas.
  • The AOC will have a formal charge. It is not ready yet; will be similar to what's listed under AOC's role in the CCHE plan (handout 1).

McClelland: Reviewed the plan submitted to and approved by CCHE (handout 1).
Major elements:

  • Background, history of UCB actions
  • Why change is needed
  • Plan to reinvigorate senior-year assessment. Roles/responsibilities of Mike Grant, AOC, PBA, academic units, students.

Sailor: Reviewed the individual unit summary for political science, as an example.

Units are supposed to do assessment every year, and report in alternate years, but some have not reported in several years. Most individual unit reports are longitudinal.

The remainder of the minutes summarizes issues raised.

Diversity of assessment methods

  • Over the last 10 years UCB units have used many different assessment methods (see handout 5), including embedded testing, review of student papers and projects, capstone courses and projects, portfolio evaluations, nationally-normed tests, unit-specific tests, pre/post testing, student surveys and interviews
  • Some units, such as political science, have themselves tried many methods
  • Different issues in assessment arise for
    • Large and small units
    • Units with and without a single course taken by most senior-year majors
    • Units in different discipline clusters
    • Different types of learning goals (even within units). Skill goals such as interpretation and documentation may require very different assessments than knowledge goals.
    • Situations where comparisons among students, and/or between institutions, are to be made. For example, portfolios can give a deep, rich picture of student work and achievement, but the lack of a set of standard stimuli makes comparison of portfolios difficult.
  • Diversity of methods is not a problem, but an asset.

Standardized tests for senior-year assessments

  • CCHE does not require that all, or any, departments or fields use standardized tests for senior-year assessments
  • However, CCHE does have a special interest in standardized tests.
  • See Appendix B of the plan submitted to CCHE (handout 1) for the UCB position on standardized tests. Briefly, we will use the ETS Major Field Achievement Tests on a three-year cycle, with a sample of seniors, in disciplines where the content is appropriate and the department does not veto use.
  • See comments from departments on ETS Major Field Tests (handout 6). These were not discussed at the meeting

Unit motivation ("unit" means department, college, or program)

  • Some units have found assessment useful in revising curricula (e.g., theatre and dance)
  • Many seem to take a pro forma approach to assessment, simply to meet requirements
  • Even units such as Political Science, with thoughtful, continuously-revised processes, don't involve most faculty in collection or use of assessments
  • Most committee members who spoke did not support the possibility of linking assessment to the "unit merit" process, feeling that this might do more harm than good.
  • General support for
    • Demonstrating to units the ways in which assessment has and can help units improve. Possible methods: once/year gathering, outside consultants, written materials.
    • Clearer guidelines and more readily available help to units doing assessments
    • Increased use of "persuasion" or out-and-out requests from the VC for unit compliance
    • Rewards, recognition for units doing especially good jobs
  • The AOC role would be to check what units are doing, offer constructive feedback, and "tell departments what each other do/have gained."

Student motivation

  • Motivation is not an issue with work done in and for courses (however, evaluation simply by the instructor alone is usually not sufficient)
  • Motivation is often an issue for
    • Tests not used in grading students, or to meet a requirement.
    • Surveys, where low response rates can make valid inference impossible
  • No good answers

Priorities in allocating money, effort

  • The VCAA's office manages a budget for undergraduate assessment. Funds are requested by units (A&S departments, colleges). In addition, PBA spends time and money on central surveys (e.g., of seniors) and assisting units with surveys, testing, and reporting and using results.
  • Mike Grant reviewed handout 7, showing A&S units' assessment budget requests and what the money has been spent on: faculty time (e.g., in summer), staff and TA/RA time, test materials, survey materials, external reviewers, videotaping, etc.
  • We reviewed the handout 3, a one-page listing of academic units, showing assessment dollars per bachelor's degree granted. The variance is enormous, from zero to over $200 per graduate.
  • Should allocations of time and money be equal to each unit? Proportional to number of graduates? Based simply on who requests what?

Next steps

  • AOC charge (Mike with VCAA)
  • Future meetings - Mike's office will schedule for 90-120 minutes each, monthly.
  • Handbook for departments - Mike will draft from materials on PBA website
    • Goals/reasons for assessment, what departments have done, check points, resources to support department activities, examples of use/benefits
  • Assignments, unit reports
    • Perry Sailor sent email on 12/19/00 to committee members, assigning each four unit summaries to review. Five large units were assigned to two members, the remainder to one only.
    • The list of assignments is reproduced at the end of the minutes
    • Members should be prepared to report on their reviews at the next meeting
  • Minutes - PBA will put AOC member lists, meeting dates, minutes, etc. on the assessment website

Sailor email to committee, 12/19/00, with assignments

Assessment Oversight Committee members,

As we discussed at the meeting of the Assessment Oversight Committee on 12/11, here are assignments for each member to review several of the individual units' reports that are currently posted on the web site. Five of the biggest programs (Business, Engineering, Psych., EPOB, and English) were assigned to two reviewers).

Elease Robbins: Classics, History, Sociology, Business

Gordon Brown: Ethnic Studies, University Writing Program (click on the General Education link to find this one), Linguistics, Women's Studies

James Sherman: Communication, Humanities, Spanish/Portuguese, Psychology

Kumiko Takahara: Economics, International Affairs, Speech/Language/Hearing Sciences, Engineering

Merrill Lessley: EPOB, Geography, Psychology, Computer Science

Michael Grant: East Asian Languages/Literature, Kinesiology, Theater/Dance, English

Padraic Kenney: Anthropology, French/Italian, Political Science, Applied Math

Ronald Melicher: Chem/Biochem, Germanic Studies, Russian, Music

Sam Fitch: Film Studies, Critical Thinking (click on General Education), Math, EPOB

Shelley Copley: English, Univ. Mathematics Program (click on General Education), Philosophy, Business

Stephen Jones: American Studies, Fine Arts, Physics, Engineering

Also, we'll be getting minutes of the meeting to each of you soon. And I've sent a packet of the handouts from the meeting to those of you who missed it.

thanks,
Perry

Perry Sailor, PhD
Senior Researcher
Planning, Budget, and Analysis
University of Colorado at Boulder
Campus Box 15
Boulder, CO 80309-0015
Voice (303) 492-2514 Fax: (303) 492-0996
Email: perry.sailor@colorado.edu


PBA: PS-- L:\IR\OUTCOMES\aoc\notes0012.doc - 12/19/00.

Last revision 03/01/02


PBA Home  |   Strategic Planning  |   Institutional Research & Analysis |  Budget & Finances  | 
Questions? Comments? |  Legal & Trademarks |  Privacy
15 UCB, University of Colorado Boulder, Boulder, CO 80309-0015, (303)492-8631
  © Regents of the University of Colorado