0:00:00.531,0:00:06.960
[Stephanie Foster:] Hello! This is
introduction to rubrics. I'm Stephanie Foster,
0:00:06.960,0:00:11.280
Assessment Lead in the Center for Teaching &
Learning at the University of Colorado Boulder.
0:00:11.840,0:00:16.400
In this session, you will learn about the
appropriate uses for rubrics and become
0:00:16.400,0:00:20.800
familiar with the essential components of
designing a rubric for class assignments
0:00:22.880,0:00:30.720
The use of rubrics as an assessment tool dates
back to the mid-1990s. So, not so long ago.
0:00:30.720,0:00:36.480
Rubrics have since been growing in popularity
and use in higher education since at least 2005
0:00:36.480,0:00:40.000
and are currently being used
in many different contexts.
0:00:41.120,0:00:47.040
A rubric is a guide that articulates the
expectations for an assignment and communicates
0:00:47.040,0:00:54.160
the level of quality for performance or learning.
Rubrics are typically used to score student
0:00:54.160,0:01:00.160
performance on defined criteria and levels of
quality, or intellectual or skill development
0:01:00.160,0:01:08.160
over time. Rubrics can be simple and short or
complex; they can be qualitative, providing
0:01:09.920,0:01:16.800
feed– narrative feedback, or quantitative,
providing numerical scores. Or all of the above.
0:01:18.400,0:01:22.400
There are two main types of
rubrics used in higher education:
0:01:22.400,0:01:28.000
the analytic rubric is what we'll spend our time
talking about today but I'd like to show you both.
0:01:28.000,0:01:33.200
So the analytic rubric specifies at least two
characteristics to be assessed at each performance
0:01:33.200,0:01:39.520
level and provides a separate score for each
characteristic. You would use analytic rubrics
0:01:39.520,0:01:46.480
for identifying particular strengths and areas of
improvement across a research paper, for instance,
0:01:47.360,0:01:51.760
and you can use it to provide detailed
formative feedback on student work.
0:01:53.280,0:01:58.720
A holistic rubric, on the other hand, provides
a single score based on an overall impression
0:01:58.720,0:02:04.560
of a student's performance on a task. You should
use a holistic rubric when a single dimension is
0:02:04.560,0:02:11.520
adequate, so perhaps you have a short assignment
or you want to give quick or summative feedback.
0:02:12.480,0:02:19.440
I'd like to show you an example of both an
analytic and a holistic rubric used for the
0:02:19.440,0:02:25.920
same task. When I was at George Mason
University, I created a product rubric
0:02:25.920,0:02:30.960
for the– looking at the results of
undergraduate research products.
0:02:33.600,0:02:42.720
So for this purpose, we defined an analytic
rubric. We wanted to understand the dimensions
0:02:42.720,0:02:46.640
of what went into that student product.
Now, a student product might be
0:02:47.600,0:02:55.920
a research paper or report; it might be a poster
at an event; but basically this is the the end of
0:02:55.920,0:03:02.480
their work so this is summative evaluation. So you
see the dimensions listed on the left, starting
0:03:02.480,0:03:09.200
with articulation of problem purpose or focus and
continuing down the line to quality of delivery.
0:03:10.320,0:03:16.160
So in this case, in this rubric,
you will see that the dimensions
0:03:17.280,0:03:23.840
are then aligned with four levels of
performance, from expert to novice, and in each–
0:03:24.800,0:03:32.400
each lines up with a description of student
performance. So in this case we're looking at the
0:03:32.400,0:03:39.920
fourth dimension down, analysis or interpretation.
You'll see that at the emerging level, the product
0:03:39.920,0:03:45.840
would show that the student used evidence
to support a limited analysis of the problem
0:03:46.400,0:03:52.480
and that their interpretation is partially linked
to the theoretical framework or scholarly model.
0:03:53.680,0:03:58.800
If you just move up to the proficient level
of performance, you'll see that there there's
0:03:58.800,0:04:04.080
an improvement the evidence now supports an
adequately complex analysis of the problem
0:04:04.640,0:04:09.040
and the interpretation is adequately
linked to the theoretical framework or
0:04:09.040,0:04:17.200
scholarly model. And as you see improvement,
you can move up into the expert category. Now,
0:04:18.240,0:04:22.880
one of the really great uses of an
analytic rubric is that you can define
0:04:22.880,0:04:28.560
where specifically where students are performing
really well and other areas where they're
0:04:28.560,0:04:34.720
performing not so well. So they may be stronger
in articulating their problem, for instance,
0:04:34.720,0:04:41.840
and weaker in, say, their quality of
delivery or their implications or impact.
0:04:43.200,0:04:49.520
Now for that same product that we're looking at,
we could use a holistic rubric and here's the
0:04:49.520,0:04:55.120
example. So you'll notice we use the same four
levels of performance from expert to novice,
0:04:55.120,0:05:00.880
but in this case there's a very
brief summary under each of those–
0:05:02.800,0:05:09.840
under each of those scales. So, the
reason we designed the rubric in this way
0:05:10.560,0:05:15.840
is that the– we had two different purposes in
mind. The first was that the analytic rubric
0:05:15.840,0:05:25.920
would be used by the instructor or the mentor to
look in-depth at a student product. We felt like
0:05:25.920,0:05:30.480
that instructor or mentor was the only person who
could really look at the product in that detail.
0:05:31.680,0:05:37.120
But we were also holding celebrations of
scholarship and presentations where students would
0:05:37.120,0:05:44.080
present their work or show their posters in
poster sessions, and we wanted to be able to
0:05:44.880,0:05:52.400
do an assessment of those products quickly by
reviewers who were not the mentors, so we would
0:05:52.400,0:05:58.640
need something where they could very quickly
assess the entire product and give it one score
0:05:59.280,0:06:07.120
so both the analytic and the holistic rubrics
are– have important uses at certain times.
0:06:09.440,0:06:15.840
Now why should you use rubrics? Essentially,
a rubric is a communication tool between the
0:06:15.840,0:06:24.000
instructor and the student. So a rubric creates
a common framework and clear expectations for
0:06:24.000,0:06:32.800
how students will perform. It creates consistency
and fairness using the same criteria and standards
0:06:32.800,0:06:39.760
across all of the students in your class,
and across reviewers or multiple graders. I
0:06:39.760,0:06:45.440
should note here that it's important that if
you are using multiple reviewers or graders
0:06:45.440,0:06:53.360
that you need to spend time training them on
the rubric and norming their scoring across
0:06:54.000,0:07:00.880
samples, so that graders are not interpreting
the language in different ways, and thus this
0:07:00.880,0:07:06.240
consistency and fairness notion would kind of
go out the window. So you want to make sure
0:07:06.240,0:07:10.880
that everybody's using this in the same
way to promote consistency and fairness.
0:07:12.320,0:07:20.720
Rubrics can promote transparency, so that progress
towards a final product and grades are clear.
0:07:20.720,0:07:28.240
It reduces mystery when and helps students to
understand why they earn the grade or the feedback
0:07:28.240,0:07:36.560
that they receive. Some faculty think that rubrics
can be used for faster assessment and it can be
0:07:36.560,0:07:43.840
done more efficiently when you have a rubric with
well-defined categories and criteria for success.
0:07:44.480,0:07:50.320
Now I might add here that it's important when
you're giving student feedback that you always add
0:07:50.320,0:07:57.040
something particular to that student work: that
you don't only use the rubric but that students
0:07:57.040,0:08:02.640
also receive written feedback and comments
from you that are specific to their assignment.
0:08:04.000,0:08:07.280
Rubrics can be used to identify
strengths and weaknesses, so
0:08:08.000,0:08:12.720
using the example that I just showed you,
it could show exactly where students are
0:08:12.720,0:08:18.000
doing well and where they need more support.
So usually when students earn a B on a paper,
0:08:19.440,0:08:25.280
you assess the paper in a holistic way to earn a
B but there are specific components that are not
0:08:25.280,0:08:32.800
all earning B grades. You may have strengths in
some areas and areas maybe the conclusion was not
0:08:32.800,0:08:40.080
a B, maybe the conclusion was a D, but overall the
student performed well, but this could really help
0:08:40.080,0:08:46.880
to identify specific strengths and weaknesses. And
finally, rubrics are used for objective criteria.
0:08:46.880,0:08:54.400
Rubrics are criterion-referenced: this means
that you compare all student work against the
0:08:54.400,0:09:04.880
criteria for success, and not norm-referenced, so
you're not comparing students to each other. So
0:09:04.880,0:09:13.360
when should you use rubrics? A rubric can be used
for assignments for which there's more more than
0:09:13.360,0:09:18.320
one answer, so really, there's no point in using
rubrics for a multiple choice exam, for instance;
0:09:20.000,0:09:25.040
you can use rubrics for formative or summative
assessment and what's the difference here? I
0:09:25.040,0:09:30.640
really like what evaluator Robert Stake
wrote, that when the cook tastes the soup,
0:09:30.640,0:09:35.600
that's formative. You still have time to
improve it. But when the guests eat the soup,
0:09:35.600,0:09:42.480
that's summative, you're done. So– so you can use
rubrics in both a formative and a summative way.
0:09:44.800,0:09:54.240
You can use rubrics for both process or product or
performance. A process element might be something
0:09:54.240,0:09:58.800
like how well a student communicated
with team members for a group project.
0:10:00.160,0:10:06.640
A product or performance might be a final research
paper or the performance of a choreography.
0:10:08.800,0:10:16.640
Rubrics are very useful when done appropriately
for peer review so students can use rubrics to
0:10:16.640,0:10:22.560
look at each other's work and get feedback
for improvement. And finally, rubrics
0:10:22.560,0:10:28.480
are useful for self assessment and improvement.
This can promote higher-order thinking such as
0:10:28.480,0:10:34.080
critical thinking and self-reflection, and can
communicate that back with their instructor.
0:10:38.560,0:10:46.000
Now let's talk about how do you create a rubric?
In its simplest form, a rubric includes five
0:10:46.000,0:10:53.360
things. The essentials are a task description,
the outcomes or dimensions to be rated
0:10:53.360,0:10:59.840
(these are the rows of the rubric), the levels of
performance or the scale (those are the columns),
0:11:00.640,0:11:06.400
a description of each characteristic at each level
of performance or scale (this– this is the content
0:11:06.400,0:11:13.840
of each of the cells), and a scoring strategy.
We are going to focus on numbers one through four
0:11:14.480,0:11:18.080
the scoring strategy will be
covered in a separate presentation.
0:11:19.760,0:11:24.880
So I'm going to continue to use the example that
I showed you earlier from George Mason University;
0:11:25.520,0:11:30.800
you should have a link to that in the content on
the website where you're viewing this video right
0:11:30.800,0:11:37.280
now. A task description: so on the first
page there's actually quite a bit of text
0:11:37.280,0:11:42.640
but if you sort through that text, ultimately
you'll find the task description. This rubric
0:11:42.640,0:11:46.800
is designed to evaluate the product of an
undergraduate research or creative project.
0:11:47.520,0:11:51.760
Products may include written
documents, poster presentations,
0:11:51.760,0:11:56.240
oral presentations or performances,
artistic expressions, and interviews.
0:11:58.720,0:12:03.040
So when you write your rubric, you should be
very clear about the activity, the assignment,
0:12:03.040,0:12:08.720
the performance or presentation
that's being assessed. Okay, the next
0:12:09.520,0:12:16.640
uh– when you take a look at an analytic rubric,
you'll see these uh different pieces. So you want
0:12:16.640,0:12:23.440
to define your dimensions. Your dimensions
are either about the product or performance,
0:12:23.440,0:12:28.000
which we've discussed, so, for instance,
the use of evidence to make an argument
0:12:28.000,0:12:33.520
could be a dimension, the use of examples could
be a dimension, or the organization of ideas,
0:12:34.240,0:12:42.240
or it can be about process. So we use the example
of communication for teamwork, for example,
0:12:42.240,0:12:48.080
or did the student follow proper protocols. This
is whether they– how they perform the process
0:12:49.040,0:12:55.600
of producing the work. And across the top you'll
see the scales or level of– levels of performance.
0:12:55.600,0:13:00.560
This describes how well the task is
performed. So you might have different levels:
0:13:00.560,0:13:06.160
you could have language describing those levels
such as exemplary, proficient, or needs work;
0:13:06.880,0:13:11.440
complete, partial, or none; or you
could have letter grades across the top.
0:13:14.880,0:13:20.800
So let's look at dimensions or outcomes and
in this case I'm continuing to use the example
0:13:20.800,0:13:26.880
of this– the product rubric. The outcomes or
dimensions to be rated, these are the rows;
0:13:26.880,0:13:34.400
these are the skills knowledge and or behavior to
be demonstrated. You should specify which skills,
0:13:34.400,0:13:39.920
knowledge, or behaviors you are looking for here,
and limit the characteristics to those that are
0:13:39.920,0:13:47.120
most important to the assignment. So one thing I
want to caution faculty on here is something I see
0:13:47.120,0:13:54.960
quite often and that is when I interview faculty
about an assignment they say, "I want students to
0:13:55.520,0:14:02.800
develop critical thinking skills. I want them
to understand how to use multiple sources to
0:14:02.800,0:14:09.360
make an argument." They almost never say things
like, "I want students to use proper grammar,"
0:14:09.360,0:14:15.520
"I want them to use proper syntax and have no
spelling errors," and yet when I take a look at
0:14:15.520,0:14:23.680
their rubric, their rubric significantly weights
things like grammar, syntax, and spelling errors.
0:14:24.320,0:14:33.040
So if those are things that you want to grade
students on, then those things need to be included
0:14:33.040,0:14:39.280
in your rubric and be clearly spelled
out. Now of course you're going to have
0:14:40.640,0:14:45.680
things that happen in student assignments that
are unexpected. Maybe somebody does something
0:14:45.680,0:14:52.640
that is wonderful and you want to reward them
for that. Maybe they do something unexpected in
0:14:53.760,0:14:58.960
various ways and you want to be able to comment
on that. You should always leave room for
0:14:58.960,0:15:04.640
the the unexpected or the wonderful things that
you see in student work. But by defining these,
0:15:05.360,0:15:12.560
and clarifying these expectations, you do that
not only for the student but for yourself.
0:15:14.080,0:15:18.560
Now when you're thinking about scales or levels
of performance, these are the labels that you
0:15:18.560,0:15:24.800
use to describe the levels of performance,
and they should be clear and meaningful. One
0:15:24.800,0:15:34.240
of the things that I like to do that I prefer
is to always have positive language. Positive
0:15:34.240,0:15:40.320
and developmental language. So what I mean by
developmental is that we know that students
0:15:40.320,0:15:47.360
are learning and growing and we want to represent
that. So this is not representing a failure,
0:15:47.360,0:15:53.520
but that you are performing at a certain level
and we know that you'll be able to improve. Now of
0:15:53.520,0:15:59.520
course you may use letter grades, or you may use a
word, uh, instead of "novice" here you might have
0:15:59.520,0:16:04.720
"unacceptable," or maybe that's off the scale:
maybe you also have an unacceptable category or
0:16:05.280,0:16:10.480
maybe you just don't see an element in the
student work at all, so they're required
0:16:10.480,0:16:15.840
to articulate their problem and maybe they
don't do that at all. So that's not a novice,
0:16:15.840,0:16:23.200
that's– that's a zero because they didn't do that
and so you just need to be clear what you expect.
0:16:27.440,0:16:35.200
So when you're completing in an analytic rubric,
the description of each of these dimensions
0:16:35.200,0:16:44.720
at– and according to each level of performance,
it's good practice to start with the top category.
0:16:44.720,0:16:50.720
So this describes the best work you expect using
all of the characteristics. And then define the
0:16:50.720,0:16:56.560
lowest category. What's an unacceptable product,
or the bare minimum that you expect to see? And
0:16:56.560,0:17:04.080
then develop the descriptions of the intermediate
level products in the categories. You should make
0:17:04.080,0:17:10.320
sure that the language from column to column is
similar, and that your syntax and wording are
0:17:10.320,0:17:18.560
aligned. So you should use specific descriptions,
avoiding words like "good" or "excellent" but use
0:17:18.560,0:17:24.800
words that are clearer to students and
provide areas that they can use to improve.
0:17:26.880,0:17:34.240
I like to start my list of outcomes with
the content, ideas, arguments, and then move
0:17:34.240,0:17:40.560
to things like organization, grammar, and
citation if those things are being evaluated.
0:17:45.040,0:17:52.240
Now what I've given to you today has been a
very brief introduction to rubrics with not
0:17:52.240,0:17:57.360
a lot of examples. If rubrics is something
that you really want to learn more about,
0:17:57.360,0:18:02.240
there are a couple really great books
on the subject, pretty recent books;
0:18:02.240,0:18:07.600
the second one, "Introduction to Rubrics," should
probably be on every faculty member's shelf.
0:18:09.920,0:18:17.760
And one more slide; I want to talk about
expanding our use of rubrics. So, a 2009
0:18:17.760,0:18:24.960
review of research on the use of rubrics in higher
education found that students tended to think of
0:18:24.960,0:18:32.560
rubrics as helping them learn and achieve, so,
that formative use of rubrics, while instructors
0:18:32.560,0:18:39.200
focused almost exclusively on rubric use for quick
grading. So there's a bit of a mismatch there,
0:18:40.000,0:18:44.720
and I think that the way that students are
looking at it is the way that I look at it,
0:18:44.720,0:18:51.600
although I do enjoy quick grading myself, but I'd
like to encourage you to think about using rubrics
0:18:51.600,0:18:57.280
as an instructional guide instead of or
in addition to the use for grading. So
0:18:57.280,0:19:03.920
how can you use rubrics to help provide
a richer, more complex communication
0:19:04.880,0:19:11.840
to students about expectations and about
their ability to to grow and to learn
0:19:11.840,0:19:18.240
through this assignment? But here's the key: you
can't just hand out the rubric to students. They
0:19:18.240,0:19:25.280
must be taught how to use it for self-assessment
and improvement. They must get practice and
0:19:25.280,0:19:31.600
understand the language. So this is important for
self-assessment and for peer assessment as well.
0:19:33.360,0:19:40.800
Rubrics can be developed to assess learning
performance over time, such as in a portfolio from
0:19:41.520,0:19:48.480
one semester, from multiple semesters, or over the
course of a student's academic career. Portfolios
0:19:48.480,0:19:58.000
are very often used in certain academic fields
such as teaching or education programs; they
0:19:58.000,0:20:07.680
are often used in performance or visual arts, for
example; and in a variety of other areas as well.
0:20:09.760,0:20:13.680
Rubrics can be used to assess learning
across sections of the same course.
0:20:14.240,0:20:21.440
So if you want to understand, for instance, the
impact of different teaching strategies across the
0:20:22.160,0:20:28.560
different sections or other elements, you might
use a rubric to do that kind of assessment,
0:20:28.560,0:20:34.640
or across courses. So you can use the same
rubric for similar kinds of assignments
0:20:34.640,0:20:41.280
across courses. And finally, rubrics can be
used in a very interesting way as a program
0:20:42.080,0:20:48.640
guide to make decisions about [a] program's
curriculum and program assessment tools. So
0:20:48.640,0:20:56.240
rubrics can be used for faculty to have shared
understandings about what students are learning,
0:20:57.280,0:21:03.040
where you want to see students go, and,
ultimately, what the values of the department
0:21:03.600,0:21:09.440
are in that program. And it can also be used
to then create program assessment tools such
0:21:09.440,0:21:19.760
as surveys uh and– and other kinds of um and to
guide other kinds of assessment and evaluation.
