Colorado
Learning Assessment Studies
Studies
of Student Engagement
Written by
Professor and President’s Teaching Scholar Clayton Lewis, September
2002
Introduction
In 1999 the President’s Teaching Scholars identified student
engagement as the key issue they wished to illuminate in a program
of research. The scholars discussed the striking differences between
students who are attentive in class and those who read the newspaper,
between students who participate actively in discussions and those
who do not, between students who pursue course topics beyond the
requirements and those who do not, and other behavioral contrasts.
They speculated that these differences could be related to differences
in how much students learn, and they shared the opinion that these
differences have a large effect on the professional satisfaction
and enjoyment of teaching. They outlined a program of study, starting
with a review of the literature, that would clarify the nature of
student engagement and how it could be assessed, that would provide
insight into the factors that promote or inhibit it, and that would
determine what its effects on learning, if any, are. While some
participants in the discussion felt that the indicators of engagement
mentioned above must be strongly related to learning, others were
skeptical, feeling that learning could occur without these symptoms,
and, for that matter, that students could be strongly engaged without
that fact being readily observable.
A group of scholars, representing several different disciplines,
undertook to explore the educational research literature, looking
for pertinent background information. We had trouble finding material
that looked useful, especially given the range of methodological
commitments represented in the group. The best leads seemed to be
in the work on intrinsic motivation (for example, the work of Edward
Deci and Richard Ryan, http://www.psych.rochester.edu/SDT), but
the group did not see a clear way forward in developing these ideas
in our own work. In response to this situation, some of the scholars
began their own empirical work, seeking information from students
about possible indicators of engagement, and about the relationship
between engagement and learning.
We had the opportunity to consult Professor Uri Treisman of the
University of Texas, a leader in the promotion of student academic
success. He encouraged our exploration, telling us that other groups
of teachers had also identified student engagement as a crucial
matter, and suggested we investigate the work of Carol Dweck of
Columbia University. The study group, and the larger group of scholars,
found Dweck’s work interesting and potentially very useful,
and some scholars began to collect data relating to Dweck’s
framework. In brief, Dweck finds that students’ beliefs about
the relationships among effort, aptitude and accomplishment, what
she calls self-theories, have a profound effect on engagement and
learning.
At this writing, the scholars have carried out seven studies of
various aspects of student engagement. The findings bear on the
following issues:
Is self-reported engagement associated with learning? Grades?
Is self-reported effort related to engagement? To learning?
What self reported behaviors are associated with students’ self-reports of engagement?
Do these associations differ for different kinds of courses?
Are Dweck’s self-theory categories related to engagement?
Is self-reported engagement associated with learning? Grades?
Fred Coolidge asked students in his introductory statistics course
to rate their own learning and their engagement in the class. He
found that while 70% of the students reported learning “a lot”
or “a huge amount”, only 15% reported being engaged at
those levels, indicating that students can learn a lot without feeling
engaged. At the same time, there is a positive association between
engagement and learning in these data, with a correlation of .275,
statistically significant.
Gene Abrams collected similar data in six undergraduate math classes,
but using course grades rather than self-ratings as the measure
of learning. He found correlations ranging from .20 to .78, averaging
.51. All but two of these correlations are highly significant; four
were higher than .5, quantitatively quite strong relationships.
A correlation of .5 means that a quarter of the variation in course
grades can be accounted for by reported engagement.
In the following year Abrams gathered further data in seven undergraduate
math classes, using self-ratings of learning rather than grades.
He found correlations ranging from .202 to .725, again averaging
.51. All but two of these correlations are significant; four were
higher than .5, again quantitatively quite strong relationships.
Bill Briggs, Mitch Handelsman, and their colleagues Nora Sullivan
and Annette Towler, studied students in a liberal arts mathematics
class. They did not find significant correlations between self-rated
engagement and exam or course grades. However, using another comparison
technique they did find that students reporting higher engagement
did earn higher exam and course grades, but only when the engagement
rating was for relative engagement, that is, how engaged the students
felt in this class as compared with other classes. The differences
were quantitatively meaningful, with the more engaged students outscoring
the less engaged by about 14 points out of 100 on the final, and
5 points in overall course grades.
Briggs et al. also asked 266 students in six classes in math, political
science and psychology about a variety of behaviors and attitudes
associated with engagement. They used these responses to identify
four engagement factors that were associated with different aspects
of engagement. These factors were highly predictive of grades in
the 40-student math class, accounting for 26% of the variance in
homework grades, 28% of the variance in midterm grades, and 30%
of the variance in final exam grades.
Is self-reported effort related to engagement? To learning?
In his survey of seven math classes, Gene Abrams asked students
to report how many hours per week they devoted to work in the class,
so that he could determine whether self-reported engagement was
related to the effort students invested. Here the seven correlations
ranged from .202 to .419, averaging .28, with only the two largest
being significant. The same data also allowed Abrams to assess the
relationship between effort and learning. Here the correlations
were lower, ranging from 0 to .512 and averaging .20. Only the largest
correlation was significant.
What self reported behaviors or other indicators are associated
with students’ self-reports of engagement?
Do these associations differ for different kinds of courses?
Briggs, Handelsman and Sullivan have developed a Student Engagement
Questionnaire, which asks students to report how characteristic
of them are a range of behaviors that faculty and students see as
indicators of engagement. They also asked the students to rate their
level of engagement in the class, both in the absolute and in comparison
with other courses. The same behaviors are associated with absolute
and relative engagement: the four behaviors most strongly associated
with absolute engagement, having fun in class, really desiring to
learn the material, thinking about the course between class meetings,
and using office hours, are the same as those most strongly associated
with relative engagement. Further, the ten behaviors least strongly
associated with the two measures of engagement are also the same.
Indicators of engagement did differ somewhat between classes. In
math courses the strongest indicators were desiring to learn and
coming to class every day, while in psychology classes having fun
and listening carefully were most important. There were also some
differences between lower division and upper division courses. For
example, going to the professor’s office hours was one of seven
behaviors more strongly associated with engagement in upper division
than in lower division courses.
Dennis VanGerven asked students in undergraduate anthropology classes,
one large class (334 students) and two small sections (about 20
students each) to say what indicators or behaviors they associate
with engagement. He grouped the responses into thematic categories:
things the student does, the way the student feels, things the professor
does, and aspects of the class format. The numbers of responses
from students in the large class differed across these categories
in the order mentioned, with 42% falling in the “things the
student does” category, 30% in “the way the student feels”,
25% in “things the professor does”, and 3% in class format.
He grouped the responses within these categories into sub themes;
of these, the most frequently reported were “concentration
in class and attendance”, 20% of responses; “professor’s
attitude”, 18%; “discuss/apply material outside class”,
15%; and “student feels connection to the class and material”,
14%.
In the small class, the order of importance of the indicators was
different. While “things the student does” was still most
important, the other factors were much less often cited, totaling
only 24% of the reports, and “things the professor does”
dropped to last place, at 3% of reports. The most often cited subcategories
for students in the small class were “concentration in class
and attendance”, at 49% of reports, and “discuss and apply
material outside of class”, at 24%. “Professor’s
attitude”, which was the second most important subcategory
for the large class, contributed less than 3% of the reports in
the small class.
VanGerven also asked the students to rate their overall level of
engagement in the class. Rated engagement in all three classes was
high, 4.3 for the large class, and 4.3 and 4.1 in the two small
sections, on a five-point scale. Thus students in both large and
small classes are engaged, but they report different indicators
of engagement to be important.
While VanGerven’s categories do not correspond fully with those
used by Briggs and colleagues, we can see that “thinking about
the course between class meetings” a factor that emerges as
important in the Briggs et al. study, also is frequently cited by
students in both the large and small classes in the VanGerven study.
On the other hand, the most strongly represented subcategory in
the VanGerven data, “concentration in class and attendance”,
corresponds to indicators that were not so important in the Briggs
et al. data, such as “listening carefully in class”, and
“coming to class every day”, both items in the middle
of the Briggs et al. importance ranking.
Are Dweck’s self-theory categories related to engagement?
Dweck has found that students who believe accomplishment is based
mainly on fixed aptitudes, what she calls an “entity”
self-theory, are likely to be less engaged than students who feel
accomplishment can be increased by effort, what she calls an “incremental”
self-theory. In their study Briggs et al. found that self-theory
was related to one form of engagement, what they term “emotional”
engagement. “Emotional engagement” included “Finding
ways to make the course material relevant to my life," "Applying
course material to my life," "Finding ways to make the
course interesting to me," "Thinking about the course
between class meetings," and, "Really desiring to learn
the material." Overall, the association between engagement
factors and self-theory was highly significant but not strong, accounting
for 5% of the variance in the strength of endorsement of incremental
self-theory.
Burkhart surveyed 58 students in a lower-division reasoning course,
collecting information about self-theories, confidence, and learning
goals at the beginning and end of the course, and responses about
various engagement-related behaviors at the end of the semester.
The questions examined both an endorsement of entity theory in general,
and endorsement of it specifically for mathematics. As Dweck would
predict, general entity theory in the pre-survey was significantly
negatively associated with engagement. At the same time, entity
theory specifically for mathematics in the pre-survey was negatively
associated with engagement. Neither relationship was significant
in the post-survey.
Following Briggs et al., we can separate the engagement items in
Burkhart’s study into categories corresponding approximately
to the Briggs et al. factors. Considering the Burkhart items related
to the Briggs et al. “participation engagement” factor,
there was a negative association with general entity theory on the
pre-survey, as Dweck would predict, with no other significant relationships.
For the single Burkhart item that seems to fall in the Briggs et
al. “emotional engagement” category, discussing course
material with friends and family, a puzzling pattern appears. General
entity theory, assessed both in the pre- and post-survey, is negatively
associated with this behavior, as Dweck would predict. But entity
theory specifically for math, assessed in both surveys, is positively
associated with this form of engagement.
Two of the Burkhart items tap forms of active disengagement: reading
unrelated material in class and discussing unrelated material in
class. Interestingly, entity theory in general and for math, assessed
in the post-survey, are negatively associated with these behaviors.
The remaining Burkhart engagement items might be placed in a “basic” engagement category, as they include such base-line expectations
as attending class, taking notes, and paying attention. There were
no significant associations between this class of behavior and entity
theory.
VanGerven surveyed students in a large anthropology class (about
300 students) on their level of engagement in his course, and in
their courses generally. He also asked for their judgments on whether
ability is fixed, Dweck’s entity theory, and whether entity
theory is true for Anthropology courses specifically. He also asked
whether students sometimes valued “doing well” in a class
over learning. He gave the survey twice, once in mid-semester and
once at the end.
Dweck would predict that students who endorsed entity theory would
report lower levels of engagement in the class, and lower engagement
in classes generally. The data do not show these relationships to
be significant on either survey. Dweck would also predict that students
who endorse entity theory for Anthropology specifically would report
lower engagement, but this relationship also was not significant
in either survey.
However, as Dweck would also predict, students who endorse entity
theory are more likely to favor “doing well” over learning.
This relationship was highly significant in both surveys. At the
same time, students who favor “doing well” over learning
rate themselves less engaged in courses generally in the earlier
survey (relationship not significant in the later survey.) In the
Burkhart data, general entity theory was also associated with favoring
“doing well” over learning, in the pre-survey, but the
relationship with entity theory for math was not significant, and
neither relationship was significant in the post-survey.
What do these findings mean?
The data indicate that there is a positive relationship between
engagement and learning, sometimes quite a strong relationship.
The Briggs et al. work suggests that the relationship becomes clearer
when different forms of engagement are distinguished, and one can
speculate that the linkages between forms of engagement and learning
will be different for different students and for different educational
situations. For example, the behaviors that are helpful for a weak
student may be different from those helpful for a strong student,
and the behaviors helpful for learning a skill may be different
from those helpful for understanding theoretical ideas. The Briggs
et al. findings on relative vs. absolute engagement suggest that
engagement may act on student’s allocation of effort. For students
with a more or less fixed time budget, only relative engagement
would produce increased effort in a given class, while for other
students absolute engagement could have an effect.
The Abrams data on engagement, effort, and learning also suggest
the complexity here. There engagement was not strongly predictive
of time invested, but also time invested was not strongly predictive
of learning (contrary to the broad conclusion in educational research
that time on task is perhaps the most robust predictor of learning.)
Plainly some forms of engagement, such as paying careful attention,
do not result in people spending more time, and people can spend
time in more or less effective ways. A more refined analysis will
be needed to expose the machinery at work.
Turning to the connection between Dweck self-theories and engagement,
the relationships are spotty (present here, absent there) and sometimes
puzzling, as in the case of the positive association between entity
theory for math and discussing course material with family and friends.
The relationship between entity theory and preferring “doing
well” over learning is perhaps a bit more consistent, but not
strongly so.
These results also suggest that a more refined look at engagement
behaviors, as represented by the Briggs et al. work, would be useful.
Going forward, our work might include an effort to determine how
students respond to particular kinds of situations, for example,
receiving a poor grade on an assignment. Self-theory for a student
in this situation might be more predictive of their response, and
thence of their learning, than for other students in other situations,
and only certain engagement behaviors might be pertinent here, as
well.
Pushing our studies in this direction will require some refinements
of method. We should adopt the Briggs et al. engagement questionnaire,
gaining more comparability across studies. We might also seek to
distinguish stronger and weaker students, not possible in most of
the studies done so far, as a way to get at least some of the differences
among students’ situations in the courses we study. These moves
are possible while staying with survey methodology.
Further progress calls for more detailed data gathering, by direct
observations of student behaviors, and/or diary studies. We should
develop some hypothetical models of student responses to different
learning situations, make predictions about the factors, including
(potentially) student learning goals, student self-theories, and
forms of engagement, and attempt to test these.
|
