Megan Bresnahan, Alison Graber, Alison Hicks and Caroline Sinkinson

University Libraries

Rationale for co-authoring:

We chose to co-author our report, because of our shared goals and the enhanced learning that may occur through a broad look at our instructional programming. We believe that through multiple assessment cycles and the exploration of several assessment mechanism, we may gather indicators for change in our teaching and programming. A central mission of the University Libraries is information literacy education across the curriculum and throughout disciplines. We constantly seek strategies to meet the maximum number of students at relevant moments in their academic careers, but to fully reach this goal, we need to embrace a collaborative assessment culture. Working collaboratively, we will increase our organizational knowledge and strengthen our information literacy efforts through shared responsibility and reflection.

 

Spring 2012 Methods

 

Sample Group: PWR First-Year Writing Classes

Course context description:

The University Librarians at CU work closely with the Program for Writing and Rhetoric first-year writing class. Every section of the class attends an in-person library seminar in addition to completing an online tutorial. Librarians facilitate over one hundred seminars per semester, each of which may be at very different stages in the research process depending on the PWR faculty member’s scheduling preferences. Students may be in the initial stages of research, for example generating a topic of interest, or they may be further along, for example searching for specific research studies that respond to components of their argument or inquiry. As we coordinate the scheduling of the seminars, we request details about the assignment, students’ research stage,and students’ topics from the faculty member.

 

In the past, we have relied solely on the needs expressed by the faculty member and not those expressed by students. Because both perspectives are enormously useful to librarians preparing for a teaching session, this assessment pilot sought to capture student needs and affective responses experienced when embarking on research. Additionally, the pilot sought to uncover indicators for change to our teaching methods, seminar focus, and additional learning opportunities. 

 

Methods: 

We sent a pre-seminar questionnaire and a post-seminar questionnaire to ten sections of the first-year writing class during the spring semester. We designed the questions to inform both teaching and learning.

 

Pre-seminar questionnaire:

    • Assessment for teaching: Student responses will help guide our seminar plans, focus, and engagement strategies. The questions serve both as a needs analysis and as information that will help us engage with learners.
    • Assessment for learning: Student responses will help students articulate what they know and understand about the assignment and research.
    • Questions:
      • What is your topic of research? [Answers may guide seminar focus, serve as sample, or act as an engagement mechanism]
      • What do you already know about your topic? [Answers will determine stage of research process: background research, question formulation, in-depth analysis, tracing conversation, synthesis and integration, ethical use]
      • What do you need to know about your topic? [Answers will guide types of tools (databases etc.) and skills  covered in the seminar]
      • What intimidates you the most about doing research? [Content useful for presenting relevance, customization, and personalization of seminar; long term analysis of results may reveal bottlenecks and  roadblocks as perceived by students]

 

 

 

 

 

Post-seminar questionnaire:

    • Assessment for teaching: Student responses (i/ii) will help us understand meaningful and significant seminar content (what worked). Student responses (iii/iv) will help us adjust our teaching or extend our teaching with additional learning materials.
    • Assessment for learning: The questions are also designed to help students articulate process and strategies about information seeking. Questions i/ii draw students attention to change/ adaptation and failure of their own expectations or preconceptions.
    • Questions:
      • Please describe one or more takeaways from today's seminar that you will use for this project. [Investigates students’ comprehension and application of content provided in seminar]
      • Please describe changes you will make to your former approach to doing research? What habits will you change? [Expectation failure]
      • What questions do you still have about research for this research project? [Identifies what more students need assistance with or confidence in; long-term analysis will serve to identify content areas missing, which we may address through scaffolding, online/ extended learning opportunities or drop-in workshops]
      • If you were to attend another library workshop, what would you like to learn? [Provides information about other curiosities students have about research; provide information about what more we need to address in our teaching and instructional programming]
      • Comments

 

 

 

 

Delivery:

We sent students an email invitation to respond by answering via a Google Form two-three days prior to the seminar (pre) and immediately following the seminar (post).

 

Summary of Results

 

Pre-Seminar Questionnaire

              Sample of Topics:

      • Don’t have a topic yet?
      • Social networking
      • Global Warming
      • Caffeine consumption and impact on quality of life
      • Decriminalization of marijuana in Boulder
      • 1960’s pop art
      • Journalism: the state of the field
      • Elementary school lunches leading to childhood obesity
      • Tourism impact on indigenous cultures in Kenya
      • Ice Hockey: evolution of regulations and the game
      • Contrasting the comparing the differences of communication in the new era. Facebook, email, twitter vs newspapers and books and how this is evolving, changing, affecting society.

 

 

 

             

              Sample student responses questions 1-5

Sample 1: (1 Topic) Vaccinations for children (2 Know) They are required for children to enter school, there is debate over whether this should be the case or not. (3 Need to know) The arguments on either side. (4 Concern) Finding sources to use.

 

Sample 2: (1 Topic) Right to bear arms (2 Know) There are many death because of this issue. (3 Need to know) I want to read about the counter argument about why there should be gun use. (4 Concern) Knowing where to find the evidence I need.

 

Sample 3: (1 Topic) I think that my topic will be over pop art in the 1960's.(2 Know)

I know about Andy Warhol, Richard Hamilton, Roy Litchenstein, and other artists. I know that the idea behind pop art is the incorporation of popular culture into artistic expression and is a statement about the attitudes that lead to the state of the popular culture at the time. (3 Need to know) I need to know more about the time period leading up to the 60s that lead to the pop art movement and what many people thought about the art that was created. (4 Concern) The idea of citing my search. I always get the citations wrong or I feel like it is really time consuming for me to add the citations.

 

Sample 4:  (1 Topic) The effect of social media on society today. (2 Know)  That kids and adults have definitely been affected by social media and are more likely to say lies and be mean over social media because they aren't face to face. Also that kids lack a lot of communication skill because they do not partake in as much face to face communication due to social media. (3 Need to know) I need to know about it more in-depth, I only know the basic arguments but i don't know the studies or facts that go along with the topic. (4 Concern)  That my topic might be too broad and i will have to narrow it down which will cause me not to find a significant amount of research.

             

                            Question 5 themes: concerns and intimidations (listed highest to lowest)

      • Adjusting (narrowing my topic)
      • Finding enough information
      • Judging validity/ credibility/ relevance
      • Writing and synthesis
      • Time commitment
      • Citing
      • Reading/ comprehension
      • Too much information
      • Knowing enough
      • Not intimidated
      • Background research
      • Opposing views

 

 

 

 

                            Sample textual responses to Question 5:

      • “Being overwhelmed by the huge amount of information, especially information that may seem useful at first glance, but will end up making me waste a huge amount of time. Databases. Reading huge,dense, boring articles. Finding specific, useful information. Having too broad a topic. Narrowing a topic. Pretty much everything.”
      • “Nothing, I'm compassionate about my topic and enjoy researching it on my own time.”
      • “If my subject turns out to be a bust and I can't find a lot of research I'll have to start all over again and be behind.”
      • “I'm worried I won't be able to easily find at least seven journal articles, or how to identify one from a regular web site.”
      • “I feel like what intimidates me the most is putting all the pieces together.”
      • “nothing. Im excited to learn more about this topic.”
      • “the library and the vast amount of sources there are.”

 

 

 

Observations:

      • Students topics were extremely  broad, yet, finding enough information was one of the highest concerns.
      • Students seem to think there are ‘right’ / ‘wrong’ sources and do not recognize that they play a part in the research as well, in interpretation, synthesis, meaning making.
      • Students may benefit from time management tips, reading, synthesis, and note taking tips, as well as focussed sessions on citation/ plagiarism.

 

 

 

 

Post-Seminar Questionnaire

Question 1 themes: seminar take-aways  (Listed highest to lowest)

      • Exposure to new search tools or databases
      • Search strategies
      • Research management
      • Topic formation/ narrow/ broad

 

 

 

Question 2 themes: changes to students’ research habits  (Listed highest to lowest)

      • Use databases in addition to Google
      • Develop search strategies and keywords
      • Investigate more interesting topics to me
      • Develop research questions around my topic
      • Research management
      • Distinguish scholar/ popular
      • None

 

 

 

Question 3 & 4 Summary: On-going questions or additional workshop needs:

                            Advanced

      • How to search for books in more detail.
      • It would be great to learn more research tips, like *. I am sure there are a million out there
      • I would like to learn more about the different databases that I could use for other papers in the future. Also, more about Norlin library, because I am not sure where to find some of the books I look up.
      • I would like to see how people in my major use these tools learned today.
      • I would like to learn more about how to find the proper subject for my topic.
      • I would like to investigate other databases a little closer to get even more options for research, as well as more details on what to scan for in the articles found in the databases.
      • I would like to learn more about visual reference that the library has to offer.

 

 

 

                            Citation & Writing

      • Citation styles for survey, interview, or own experiment.
      • More on analyzing sources and using information from those sources in text.

 

 

 

                                          Working

      • I still need to work on finding a topic that will potentially have enough sources and [hopefully] interest me. That's for me to figure out but it would be nice if you could [hypothetically] just give me the answer.
      • I still need to work on finding articles that are more specific to the topic.
      • I don't necessarily have any as of now, but if I need help I would definitely go ask the research center for help.
      • I am still trying to narrow down my project which can be a little difficult when looking at so many articles, but I think it is good to start off broad anyways!
      • I don't know what I would like to learn. Anything that would help but what that would be I don't know (which probably doesn't help you very much or at all.)
      • Nothing in particular, seeing as how the workshop answered all of my previous questions.
      • None (several)

 

 

 

 

              Comments:

      • Overall the seminar was really helpful! I will definitely use the techniques we discussed.
      • Super helpful, even as a junior learned some things. thanks!

 

 

 

 

Observations:

The questions and responses improved our seminars in several ways. Gathering details about students’ topics, strategies, and intimidations  through the pre-seminar questionnaire helped us to authentically address and engage students in the seminar. For example, by identifying the areas of most concern (intimidation), we could adjust the seminar to highlight and enhance strategies relevant to those concerns. For example, using  the examples from this pilot, we might focus seminar content on topic formation and adjusting the narrowness or broadness of a topic.  Similarly, if student topics are clearly well developed as opposed to overwhelmingly broad, we could adjust the seminar to focus on discipline and format specific resources as opposed to general background gathering resources. These are only a few examples of how we can use student needs assessment to alter our teaching strategies, but each possibility presents an opportunity to encourage adaptive and tailored instruction. [Workshopping sample of student topics, problem-based driven by student topic or fears etc.] Students’ responses revealed knowledge, needs, and interests that we took into account during preparation, revision, and enhancement of  individual seminars as well as our broader  instructional programming. Furthermore, if analysed over time, the results may serve to identify consistent trends and bottlenecks commonly encountered by students.

 

Results gathered through the post-seminar questionnaire provided indicators for change to teaching strategies when student take-aways did not match student learning objectives. Additionally correlating student requests for additional learning with student status, assignment, and affiliation may demonstrate best practices for scaffolding and mapping of our instructional efforts. Finally, we may also develop alternative strategies, such as online learning objects or extended learning options, to meet student learning needs.

 

This initial pilot of the questionnaires revealed a few areas of improvement.  Firstly, student completion of questionnaires varied widely; however, in the instances when teaching faculty endorsed and encouraged students, there was visible impact on participation. We will work more actively to seek teaching faculty assistance. Given that the post-seminar questionnaire had far fewer completions, we may consider encouraging completion during the last few minutes of the seminar. Finally, Questions 3 (What questions do you still have for this research project) and Question 4 (If you were to attend another seminar what would you like to learn) may be rephrased to elicit more useful responses.

 

Sample Group: Spanish Classes

Course context description:

An information literacy class is integrated into the Spanish and Portuguese department curriculum through the inclusion of one 50 minute research seminar in every 3000 Advanced Spanish Language Skills class. A class at the 3000 level ensures that every departmental major receives specialized research instruction that aims to form a purposeful basis for their literary, language, cultural, business or professional work in the department. In this class, students have to write a short research paper on a topic of their choice, that uses some Spanish resources. Students have typically never researched in Spanish before and are just becoming aware of the disciplinary context and discourse community of their work. The librarian coordinates with each instructor (typically a graduate student) to tailor the session to what the instructor perceives to be the major needs of the class.

 

Methods

As above.

 

Summary of Results

Pre-Seminar Questionnaire

Sample of Topics:

    • Coffee Production in South America
    • Human rights violations in Argentina during the Dirty War
    • Comparing religions in ancient South and Central America
    • Sea turtles in Costa Rica
    • Jorge Luis Borges
    • Spain's law of forgetting, its purpose and how it's affecting spanish society today

Sample student responses questions 1-4

    • (1 Topic) Immigrants from Mexico or Latinamerica. (2 Know) The difficulties of migrant workers, the undocumented immigrants issue in politics and how that affects immigrants, and the cultural gaps that need to be bridged in order for a migrant family's kids to succeed. (3 Need to Know) More sources, as well as a good novel for a literature review. (4 Concern) How long it will take, and the inaccuracies of most internet search results
    • (1 Topic) The Quechua of Peru.  (2 Know) I know that they are an indigenous tribe in Peru, and that they are prominent in the Andes region. (3 Need to Know) I need to know their family life, their traditions, and their way of life. (4 Concern) Trying to find sources in Spanish, not really sure where to go about doing this. How to convert some of the info I find in English into Spanish and how to use this info in my paper.
    • (1 Topic) Mexican Drug Cartel. (2 Know) I know it has been a huge problem for the last several years, and that it has severely hindered tourism in Mexico. (3 Need to Know) How/when it started. What is being done to alleviate the situation? Who (really) is involved? (4 Concern) I need to know where the bulk of my resources will be located. It'd be nice to have resources other than online news articles.

Question 5 themes: concerns and intimidations (listed highest to lowest)

    • Finding Spanish sources
    • Reliability
    • Too much information
    • Citing
    • Reading Spanish articles
    • Not finding information on a topic
    • Narrowing a topic
    • Writing
    • Choosing a research question
    • Finding books
    • Time

Sample textual responses to Question 5:

    • I am never necessarily sure how to know if a source is reliable. Also, I am intimidated by this research paper as it will be in Spanish, and I'm not sure how available information on my subject in the Spanish language will be.
    • In the past, choosing good research questions has troubled me. I feel pretty confident about the questions I have now though. And I know that the wonderful library staff will help me finding the resources I need.
    • Sorting through all the material and extracting what is relevant. Also citing the sources correctly is always tedious.
    • Nothing. I'm awesome. But for real, I am worried slightly worried about citing resources in Spanish.

Observations:

    • Students were most nervous about finding and using Spanish sources, perhaps indicating that they didn’t recognise transferability from current/English research process
    • Students recognised for the most part that their topics were broad, and that they would need to narrow in order to not feel inundated.
    • Fewer people also mentioned problems with citing and study skills, perhaps due to previous classes/experience. 

 

Post-Seminar Questionnaire

Question 1 themes: seminar take-aways  (Listed highest to lowest)

    • Finding Spanish articles
    • Choosing better keywords to search
    • Library Resources
    • Inter-Library Loan

Question 2 themes: changes to students’ research habits  (Listed highest to lowest)

    • Using scholarly articles/databases
      • “I didn't know we had this before the class.”
    • Choosing better keywords/search strategies
      • “I will definitely put a greater emphasis on the keywords that I use.  When putting in synonyms of the same word, I sometimes got completely different results.”
    • Using more reliable sources
    • Doing better Google searching
      • “I had never heard of those Spanish-specific search tools.  I'd never even heard of the Spanish Googles.  Learning about those alone has helped me a lot.”

 

 

 

 

Question 3 & 4 Summary: On-going questions or additional workshop needs:

    • Citing
    • More on Find it at CU
    • More on Spanish Search Engines
    • How to find books
    • Norlin navigation
    • More searching mechanics

 

Comments:

    • Engaging and helpful!
    • Not really. I was very surprised by how helpful the seminar was. I learned a lot and now have a good idea on how to move forward in researching my topic in a different language.
    • I loved the instructors accent!

 

Observations:

    • The data/information gathered gives me a much clearer idea about student topics, which, in turn, will influence how I structure the classes next semester. More students were studying non Latin-American/Hispanic topics than I expected, which means I need to be more up to date with finding material in other databases in Spanish. It will also give me ideas about how to update the online webpage with more helpful sample topics, and further resources.
    • I was pleased with how students seem to have absorbed and appreciated the learning objectives of the class. It was great to get feedback about the need for information on how to find information in Spanish.
    • I also thought that it was interesting how students seemed to have built on their prior knowledge- ie many students recognised common pitfalls (eg too much information) rather than worrying that they wouldn’t have enough. It would be interesting to see if that could be related to their PWR class (see above) It was good to gain a sense of where students are in thinking about their research process- respondents seemed more mature and thoughtful than I expected!
    • Next steps include tying this assessment in with other Spanish classes I teach, as well as potentially trying to tease out the effects of information literacy on intercultural competence.

 

 

 

Sample Group: Engineering & Physics Classes

Course context description:

The Gemmill Library supports several science departments and the College of Engineering and Applied Sciences.  The instruction librarian provides information literacy instruction sessions for undergraduate and graduate students enrolled in these disciplines throughout the academic year.  This assessment focused on 4 class sessions, 2 in 3000 and 4000 undergraduate level environmental engineering courses and 2 graduate physics writing courses.  In each of these 4 class sessions, students are working on in-depth semester long research projects and need to be able to effectively find, use, and synthesize scholarly literature in the sciences.

 

Methods

This assessment was modified slightly from the prior samples presented above in order to capture additional evidence about the database sources used by science students.  This is an example of how the assessment strategies within University Libraries can be modified and tweaked to adjust to the needs of librarians, who support a variety of disciplines and levels of expertise among students, while maintain a cohesive assessment initiative.

 

Pre-seminar questionnaire:

    • Questions:
      • What is your topic of research? [Answers may guide seminar focus, serve as sample, or act as an engagement mechanism]
      • Where have you already looked for information on your research topic? / Where do you plan to look for information about your research topic? [Answers will provide information about where students tend to look for information when they are searching for scholarly literature]
      • What do you already know about your research topic? [Answers will determine stage of research process: background research, question formulation, in-depth analysis, tracing conversation, synthesis and integration, ethical use]
      • I feel confident in my ability to find scholarly sources (journal articles, books) relevant to my research topic. [scale 1-5]
      • I feel comfortable using library databases to find articles related to my topic. [scale 1-5]
      • What do you need to know about your topic? [Answers will guide types of tools (databases etc.) and skills  covered in the seminar]
      • What intimidates you the most about doing research? [Content useful for presenting relevance, customization, and personalization of seminar; long term analysis of results may reveal bottlenecks and  roadblocks as perceived by students]

 

 

 

 

Post-seminar questionnaire:

  • Questions:
  •  

      • I feel confident in my ability to find scholarly sources (journal articles, books) relevant to my research topic. [scale 1-5]
      • I feel comfortable using library databases to find articles related to my topic. [scale 1-5]
      • This session will help me with my research [scale 1-5]
      • How could what you learned in this session apply to your research/assignment? [Expectation failure]
      • What did you like least about this session?
      • What questions do you still have about your research topic? ? [Identifies what more students need assistance with or confidence in; long-term analysis will serve to identify content areas missing, which we may address through scaffolding, online/ extended learning opportunities or drop-in workshops]
      • Comments

 

 

Summary of Results

Pre-Seminar Questionnaire

Sample of Topics:

    • Molecular quantum physics - wave-like interference pattern (between electrons) as a hydrogen molecule is double photo-ionized
    • Super accurate gravitational mapping of Earth
    • Highly scattering materials and photo-acoustic effect for imaging purposes and biomedical applications.
    • Chemistry of effluent organic matter and its impacts on advanced oxidation for wastewater treatment and photochemical processes in the environment
    • Engineered interventions to minimize microbial activity that causes Acid Mine Drainage

Sample student responses questions 1-4

    • (1 Topic) Ozone exposure of older low-income adults living in urban neighborhoods (2 Already looked) Google search (3 Know) Ozone exposure seems to be more severe among this population.  (Need to know) How to find statistics on my topic. (Concern) Finding the most recent information.
    • (1 Topic) Entropic theory of gravity (2 Already looked) Google (3 Know) it is a conceptual theory that describes gravity as an entropic force.  (Need to know) How to find the most relevant and important articles related to Entropic Theory. (Concern) Narrowing my topic.
    • (1 Topic) Effects of fire on the release of Mercury from forest soils (2 Already looked) Google Scholar, Web of knowledge (3 Know) I know that this is a major problem in the western united states.  (Need to know) How to find environment articles (Concern) Too much information.

Questions 5 and 6 self-ratings (scale of 1-5)

  • Confidence: 3.6
  • Comfort: 3.55

Question 7 themes: concerns and intimidations (listed highest to lowest)

  • Time
  • Currency
  • Search terms
  • References
  • Too much information
  • Selecting the right source

Sample textual responses to Question 7 (Concerns):

  • I feel like my topic is too broad and I am concerned about knowing where to look for relevant research articles.
  • I feel comfortable with the research part, my advisor has given me many articles, but I am concerned about keeping track of what I read and citing.
  • Some of the articles I have already found are too specific for my topic.  I feel like I need more background.
  • I am having trouble finding articles on my topic, and I am not sure what search terms to use.
  • Finding IEEE fulltext.

 Observations:

  • Several students expressed concern related to topic scope, either too broad or too specific
  • A few of the graduate students mentioned that their advisors had already supplied many of the articles they need on their topic, almost as if the search process could be skipped.
  • The graduate students were concerned with keeping track of their research and saving time in this process.
  • Surprising range of high quality scholarly databases listed in question 2, students seemed to have some awareness of where to look beyond google.

Post-Seminar Questionnaire

Questions 1-3 self-ratings (scale from 1-5):

Confidence: 4.62

Comfort: 4.43

Helpful: 4.62

 

Question 4 themes: changes to students’ research habits  (Listed highest to lowest)

 

 

      • Development of effective search strategies
        • “Ways to search (synonyms), different databases and how to limit and narrow the search”
        • “limit and narrow the search”
      • Content-specific identification of sources
        • “I didn't know how to find chemical articles, and I do now for my paper”
        • “Finding non-us information for my topic”
      • Exposure to new databases:
        • “Knowing new databases like Engineering Village and Web of Science”
      • Adopting citation tools
        • “The citation information will be used for my bibliography”

 

 

 

 

Question 5 themes: What questions do you still have?

 

  •  Time for hands-on practice of skills.
    •  “I want to use Zotero, but need to practice
    •  “I need to try the search strategies on my topic”
  • Questions about keeping track of research
    •  “Do you need to share something about literature management? Or it's not your field”
  • Questions about citation manager:
    • “You could mention "sync" in zotero.  That's useful if you use different computers”

 

Comments:

  •  “Very helpful and informative”
  • “This will really help”
  • “Wow”

 

Observations:

  • Students need more time to apply the information presented in the session.  Follow-up assessment related to how the content is applied later is needed.
  • However, seeing some evidence of that application, in relation to my learning goals, is promising.

 

Future:

 

While these questionnaires present an initial phase in our assessment efforts, we also recommend analysis that more accurately investigates student application of learning. Teaching-librarians typically only interact with students for sixty to ninety minute sessions, which is an insufficient time frame to instill a critically reflective information literate posture. Therefore, to assess CU students’ information literacy, we suggest performance assessment mechanisms such as: database research logs, student research reflections, keyword development worksheets, annotated bibliographies, student interviews amongst others.

 

Additionally, we intend to pursue the development of a storehouse of assessment tools and outcomes that will be available to all teaching-librarians. Our long term goal is to also create a reporting system, which will archive assessments across time. With these strategies, we are confident that we can encourage a collaborative assessment culture in the libraries.

 

 

University of Colorado at Boulder CU Home CU Search CU A to Z Campus Map