Research In Depth with the D’Mello Lab: 2020 Spring Newsletter Feature Article

Interview with Sidney, expanded version.

Developing Models of How Team Collaborative Problem-Solving Works using Sensor Technology, and Developing and Testing an Artificial Intelligence-based Team Facilitator to Improve Teamwork

Sidney D’Mello sees a future workplace that is embedded in technology where teamwork – already increasingly critical – will itself be redefined.  He states: “Teams will need to develop better skills in handling complex problems as routine work will be increasingly delegated to artificial intelligence (AI) technologies such as personal digital assistants. Teams will need to rapidly adapt to fluid membership and changing work structures with the growing gig economy, with crises such as COVID-19, and as new workers enter the workforce bringing new cultural practices. Individuals will need to be able to perform effectively in heterogeneous teams as the workforce becomes more diverse and as globalization increases. The future of teamwork will require integration of technological advances to facilitate team performance, yet we are largely relying on tools and techniques from the 20th century for team facilitation. “

To address the imminent needs of the future, Sidney’s group is developing ways to understand collaborative problem solving and other forms of teamwork by considering how social, emotional, and cognitive components work together. For example, one project aims to develop a theoretical “framework to assess and identify collaborative problem-solving (CPS) skills in computer-based educational environments” whereas another aims to develop an intelligent (AI-based) team facilitator to improve team performance.

Through these explorations, Sidney hopes to “contribute to a new understanding on how 21st century teams can manage complexity, how team heterogeneity can lead to team effectiveness, and identify successful strategies for team adaptability.”.

In pursuit of these goals, he is currently involved in three projects: one sponsored by the National Science Foundation (NSF) is focused on discovering how interpersonal interactions arise and influence collaborative problem solving in digital STEM learning environments while designing next-generation STEM learning environments that aim to make CPS more enjoyable, engaging, and effective. A related project sponsored by the Department of Education’s Institute of Education Sciences (IES) focuses on developing and tested a “computational model that can enable automated detection of evidence of collaboration from data captured during collaborations; a third more recent study, sponsored by the, NSF utilizes sensor technologies for tracking team behavior in real-world workplaces, developing models of teamwork, and developing, testing, and refining an AI-based team facilitator tool.

ICS: Sidney, why did you get involved in this work?

SD: : People always thought that relatively complex cognitive jobs would always be protected. But with AI, it’s not just procedural work that is at threat right now, but even cognitive tasks like accounting and banking. However, people have unique human capabilities in two important areas (among others): problem solving that is non-routine; and collaboration, working on complex problems together. But historical research has shown that two heads are often not better than one or not as better as they should be. There is something about collaboration that is very ineffective – teams rarely achieve what they should be able to achieve. So, we need better methods to make collaboration more effective.
 
ICS: What are the main aims of your larger project with middle school students?

SD: The research aims to address three key questions: (1) what is the CPS construct in general especially in the context of computerized learning environments, and what student actions and behaviors constitute evidence of effective CPS; (2) what is the nature of the association between CPS and student learning outcomes, particularly in computerized educational environments; and (3) how can a model be developed to analyze and automate assessment of collaborative problem solving skills.
 
ICS: What makes this study’s approach different from other CPS research?

SD: First we aim to really understand collaboration from a combination of social, emotional and cognitive components. Typically, collaboration is investigated from a singular context of a cognitive perspective or motivationally or emotionally. The reality is that these components work together. So, our work is different in that we are looking at collaboration from this (multi-context) lens. 
 
Second, people interacting with each other and the environment opens up and increases what you can observe such as, are they looking at you, are they smiling, making eye contact, facial expressions, nodding together, all which are complicated patterns of behaviors so you need new ways to measure, new units of analysis to measure. And the research is showing that you can look at collaboration and produces measure that are predictive of group success. So we are able to take a highly complex process, collaboration, and distill it to useful and usable information.
 
Collaboration is messy! But we have shown that you can actually make a lot of sense of it (here Sidney laughs and says, “like banging our heads on the wall”) by taking very small steps, to go step by step and really target (and clarify) something really complicated.
 
ICS: Your focus is CPS within computerized educational environments. Are you looking at collaboration as a holistic interaction between people and the environment, not just people to people interaction?

SD: Yes, any complex cognitive system involves people interacting with each other and any tangible tools. There’s a classic example from the 1980’s about two pilots landing an airplane. Where is the cognition on how to land the plane? Is it just in one pilot’s head? Both pilots’ heads? No. The cognition on landing the plane is in both the pilots heads and in the airplane. The airplane is part of the cognitive system of landing the airplane. We adopt on, and expand on this approach, called distributed cognition, in our thinking.
 
ICS: The NSF study with college students is developing an AI-based facilitator tool that uses multisensory analysis and real-time dynamic input in an effort to promote better communication between people. So in this case, the AI facilitator is like the airplane, part of the cognitive system of two and three people communicating with one another. What is happening with this study?

SD: First, we started off with dyads and triads working on problems using videoconferencing. We would their facial expressions (emotional communication), eye gaze (where they look and what they attend to), speech intonation, language contents, and physiology (heart rate variability, skin conductivity physio arousal). These data are used to understand the unfolding collaboration on a second-by-second-basis using AI-based computational modeling.
 
Next, we are embedding these models in the collaboration interface where it can jump in to provide real-time feedback on collaboration by sending private zoom messages to the users. The advantage of personal cueing is to help each person improve their communication. How well an individual can ‘read’ the visual, verbal, and other cues during a conversation varies greatly, so by being able to give feedback, the tool helps improve collaborative interactions. For example, I may say something with the best intentions and I may have offended somebody and not be able to pick that out, but the tool would pick it out simply by measuring jump in skin conductance or increased hear t rate.
 
With COVID-19, the study has shifted to be a Zoom user study now and this summer we’ll look at the data, conduct small scale tests to make improvements. Then we hope to follow up with an experimental large scale study with about 100 teams, so 300 people, to find out whether having the intelligent collaboration interface compared to a control condition improves how people collaborate and the outcomes of the collaboration.

What next?

• There are also studies looking at human-agent teaming, when you have a human and a computer as a team and trying to understand this pairing.
• Excited for neurosci focused research made possible by the equipment grant for the functional Near Infrared Spectroscopy (fNIRS) scanner- allows us for the first time to study what is literally happening in the brain when working in teams.
• Also study with UC Irvine and Notre Dame on ‘Future of work of teams’, with the study being conducted ‘in the wild’, actual workplaces, studying work-teams for 2 or 3 months. 

Sidney’s work highlights the unique strengths of ICS, with a focus on multi-institution and multi-disciplinary research collaborations.

You can read more about each study featured here at:
Abstract for Intelligent Facilitation for Teams of the Future via Longitudinal Sensing in Context and A Theory and Data Driven Approach for Identifying Evidence of Collaborative Problem Solving Skills