Amelia Kelly
Former Visiting Scholar • VP of Speech Technology, SoapBox Labs • Fulbright TechImpact Scholar 2020/2021
Institute of Cognitive Science

As a Fulbright TechImpact scholar for 2020/2021, I plan to carry out a project called CARNIVAL: Child Automatic speech RecognitioN to measure academically productIVe tAlk in the cLassroom. Automatic speech recognition (ASR) is the process by which a computer can transcribe the linguistic contents of an audio file.

In CARNIVAL I focus on one particular application of child ASR in elementary school science, technology, engineering and mathematics (STEM) education. Teachers and instructors in K-12 STEM classrooms are moving towards an educational system that encourages children to discuss ideas, actively listen to one another, and to construct arguments and explanations based on evidence and reasoning (Michaels et al. 2002). This approach has been shown to particularly help girls and other members of groups underrepresented in STEM to gain confidence in their mathematical abilities (Suresh et al 2019). To facilitate “academically productive” talk, teachers encourage the use of “talk moves”: phrases and questions that turn classrooms into knowledge-building communities where students intellectually engage with each other’s ideas (Michaels et al. 2012).

Increasingly, teachers are using recordings of classroom sessions to monitor the use of talk moves in the classroom. However, transcribing classroom sessions is expensive and time-consuming. The TalkBack platform from the University of Colorado (CU) Boulder avoids this expense by providing teachers with feedback on their instruction by using adult speech recognition to transcribe the teachers’ speech and identify certain talk moves in class. Child speech recognition is a much more difficult task than adult ASR because of the physiological and behavioural differences between adults and children. In order to monitor talk moves it is necessary to have accurate transcriptions of both the teachers’ and the students’ speech.

Due to the lack of available child ASR, the TalkBack system is as yet unable to accurately transcribe the students’ speech and report on their contribution to academically productive talk during classroom sessions. The proposed project will focus on using the latest advances in machine learning and artificial intelligence to improve ASR technology for elementary school child speech in the classroom. This will be achieved by developing a bespoke state-of-the-art child speech recognition system for TalkBack using speech data from children speaking and learning in classroom environments. The result will be an ASR system that can take, as input, an audio file containing child speech, and return a text transcription of the content of that audio file.

By collaborating with the Institute of Cognitive Science (ICS) at CU Boulder, I propose to integrate the child ASR system into the TalkBack platform, which will be used by K12 STEM teachers to monitor their use of academically productive talk in class. The ability to transcribe the students’ speech is integral to the system’s ability to classify and monitor talk moves, and will add a new layer of functionality to this platform.

Research summary

I am VP of Speech Technology at SoapBox Labs, the world's leading company for child speech technology. For the last 7 years, I have worked exclusively in child speech, leading a team of computational linguists, speech scientists and engineers in building accurate and scalable real-world child speech ASR systems and related technologies. I have seen first-hand the transformational power of child speech and language technologies when used in the educational sector, and believe that when used at scale as a tool by teachers and educators, they will play a transformational role in tackling the worldwide literacy crisis.

In my career to date I have worked with IBM Watson in Dublin, during which time I secured a patent in cognitive computing, and Fluential in Sunnyvale, CA, where I helped develop intent recognition technology. In my early days with SoapBox Labs I was affiliated with Trinity College Dublin as a postdoctoral researcher. I hold a bachelor of science degree in Physics and Astronomy from NUI Galway, Ireland, an M.Phil in Linguistics and a PhD in speech Technology, both from Trinity College Dublin.

Bio

I am an artificial intelligence engineer and scientist specialising in automatic speech recognition of children’s voices. Currently VP of Speech Technology at SoapBox Labs, I hold a B.Sc. in Physics and Astronomy from NUI Galway, and an M.Phil and PhD in Linguistics and Speech Technology from Trinity College Dublin. I have 13 years experience in speech signal processing, natural language processing, machine learning and artificial intelligence and hold a patent in the area of cognitive computing. I am a regular speaker at international technical conferences and industry events and I am a Fulbright TechImpact Scholar for 2020/2021.