Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home (2023)

Abstract: Enriching the quality of early childhood education with interactive math learning at home systems, empowered by recent advances in conversational AI technologies, is slowly becoming a reality. With this motivation, we implement a multimodal dialogue system to support play-based learning experiences at home, guiding kids to master basic math concepts. This work explores Spoken Language Understanding (SLU) pipeline within a task-oriented dialogue system developed for Kid Space, with cascading Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components evaluated on our home deployment data with kids going through gamified math learning activities. We validate the advantages of a multi-task architecture for NLU and experiment with a diverse set of pretrained language representations for Intent Recognition and Entity Extraction tasks in the math learning domain. To recognize kids’ speech in realistic home environments, we investigate several ASR systems, including the commercial Google Cloud and the latest open-source Whisper solutions with varying model sizes. We evaluate the SLU pipeline by testing our best-performing NLU models on noisy ASR output to inspect the challenges of understanding children for math learning in authentic homes.

## introduction and background the ongoing progress in artificial intelligence (ai) based advanced technologies can assist humanity in reducing the most critical inequities around the globe. the recent widespread interest in conversational ai applications presents exciting opportunities to showcase the positive societal impact of these technologies. the language-based ai systems have already started to mature to a level where we may soon observe their influences in mitigating the most pressing global challenges. education is among the top priority improvement areas identified by the united nations (un) (i.e., poverty, hunger, healthcare, and education). in particular, increasing the inclusiveness and quality of education is within the un development goals foot_0 with utmost urgency. one of the preeminent ways to diminish societal inequity is promoting stem (i.e., science, technology, engineering, math) education, specifically ensuring that children succeed in mathematics. it is well-known that acquiring basic math skills at younger ages builds students up for success, regardless of their future career choices @xcite . for math education, interactive learning environments through gamification present substantial leverages over more traditional learning settings for studying elementary math subjects, particularly with younger learners @xcite . with that goal, conversational ai technologies can facilitate this interactive learning environment where students can master fundamental math concepts. despite these motivations, studying spoken language technologies for younger kids to learn basic math is a vastly uncharted area of ai. this work discusses a modular goal-oriented spoken dialogue system (sds) specifically targeted for kids to learn and practice basic math concepts at home setup. initially, a multimodal dialogue system @xcite is implemented for kid space @xcite , a gamified math learning application for deployment in authentic classrooms. during this preliminary real-world deployment at an elementary school, the covid-19 pandemic impacted the globe, and school closures forced students to switch to online learning options at home. to support this sudden paradigm shift to at-home learning, previous school use cases are redesigned for new home usages, and our dialogue system is recreated to deal with interactive math games at home. while the play-based learning activities are adjusted for home usages with a much simpler setup, the multimodal aspects of these games are partially preserved along with the fundamental math concepts for early childhood education. these math skills cover using ones and tens to construct numbers and foundational arithmetic concepts and operations such as counting, addition, and subtraction. the multimodal aspects of these learning games include kids' spoken interactions with the system while answering math questions and carrying out game-related conversations, physical interactions with the objects (i.e., placing cubes and sticks as manipulatives) on a visually observed playmat, performing specific pose and gesture-based actions as part of these interactive games (e.g., jumping, standing still, air high-five). our domain-specific sds pipeline (see figure 1 ) consists of multiple cascaded components, namely automatic speech recognition (asr), natural language understanding (nlu), multimodal dialogue manager (dm), natural language generation (nlg), and text-to-speech (tts) synchronizing the agent utterances with virtual character animations on student user interface (ui). here we concentrate on the spoken language understanding (slu) task on kids' speech at home environments while playing basic math games. such application-dependent slu approaches commonly involve two main modules applied sequentially: (i) speech-to-text (stt) or asr module that recognizes speech and transcribes the spoken utterances into text, and (ii) nlu module that interprets the semantics of those utterances by processing the transcribed text. nlu is one of the most integral components of these goal-oriented dialogue systems. it empowers user-agent interactions by understanding the meaning of user utterances via performing domain-specific sub-tasks. intent recognition (ir) and named entity recognition (ner) are essential sub-tasks within the nlu module to resolve the complexities of human language and extract meaningful information for the application at hand. given a user utterance as input, the intent classification aims to identify the user's intention (i.e., what the user desires to achieve with that interaction) and categorize the user's objective at that conversational turn. the entity extraction targets locating and classifying entities (i.e., specific terms representing existing things such as person names, locations, and organizations) mentioned in user utterances into predefined task-specific categories. in this study, we present our efforts to convert the task-oriented sds @xcite designed for school use cases @xcite to home usages after covid-19 and inspect the performance of individual sds modules evaluated on the home deployment data we recently collected from 12 kids individually at their homes. the current work focuses on assessing and improving the slu task performance on kids' utterances at home by utilizing this real-world deployment data. we first investigate the asr and nlu module evaluations independently. then, we inspect the overall slu pipeline (asr+nlu) performance on kids' speech by evaluating our nlu tasks on asr output (i.e., recognized text) at home environments. as the erroneous and noisy speech recognition output would lead to incorrect intent and entity predictions, we aim to understand these error propagation consequences with slu for children in the math learning domain. we experiment with various recent asr solutions and diverse model sizes to gain more insights into their capabilities to recognize kids' speech at home. we then analyze the effects of these asr engines on understanding intents and extracting entities from children's utterances. we discuss our findings and observations for potential enhancements in future deployments of this multimodal dialogue system for math learning at home. ## experimental results to build the nlu module of our slu pipeline, we train intent and entity classification models and cross-validate them over the kid space home poc dataset to decide upon the best-performing nlu architectures moving forward for home. for language representations, we observe that incorporating diet with the bert family of embeddings from autoencoders achieves higher f1 scores relative to the gpt family of embeddings from autoregressive models. we cannot reveal any benefits of employing math-specific representations with diet, as all such models achieve worse than diet+bert results. one reason we identify is the mismatch between our early math domain and advanced math corpora, including college-level math symbols and equations, that these models trained on. another reason could be that such embeddings are pretrained on smaller math corpora (e.g., 100 million tokens) compared to massive-scale generic corpora (e.g., 3.3 billion words) that bert models use for training. diet+convert is the clear winner for intents and achieves second-best but very close results for entities compared to diet+labse. convert and labse are both sentence-level embeddings, but convert performs well on dialogue tasks as it is pretrained on large conversational corpora, including reddit discussions. based on these results, we select diet+convert as the final multitask architecture for our nlu tasks at home. next, we evaluate our nlu module on kid space home deployment data collected at authentic homes over 12 sessions with 12 kids. each child goes through 5 activities within a session, as described in 3.1. in the performance drops we previously observed at school @xcite . we witness distributional and utterance-length differences between poc/training and deployment/test datasets. realworld data would always be noisier than anticipated as these utterances come from younger kids playing math games in dynamic conditions. to further improve the performance of our kid space home nlu models (trained on poc data) by leveraging this recent deployment data, we experiment with merging the two datasets for training and evaluating the performance on individual deployment sessions via leave-one-out (loo) cv. at each of the 12 runs (for 12 sessions/kids), we merge the poc data with 11 sessions of deployment data for model training and use the remaining session as a test set, then take the average performance of these runs. that would simulate how combining poc with real-world deployment data would help us train more robust nlu models that perform better on unseen data in future deployment sessions. the overall f1-scores reach 96.5% for intents (2.3% gain from 94.2%) and 99.4% for entities (0.1% gain) with loocv, which are promising for our future deployments. to inspect the asr module of our slu pipeline, we experiment with rockhopper, google, and whisper-base/small/medium asr models evaluated on the same audio data collected during home deployments. using the manual session transcripts as a reference, we compute the average wer for kids with each asr engine to investigate the most feasible solution. for slu pipeline evaluation, we test our highestperforming nlu models on noisy asr output. ## error analysis for nlu error analysis, ## conclusion to increase the quality of math learning experiences at home for early childhood education, we develop a multimodal dialogue system with playbased learning activities, helping the kids gain basic math skills. this study investigates a modular slu pipeline for kids with cascading asr and nlu modules, evaluated on our first home deployment data with 12 kids at individual homes. for nlu, we examine the advantages of a multi-task architecture and experiment with numerous pre-trained language representations for intent recognition and entity extraction tasks in our application domain. for asr, we inspect the wer with several solutions that are either low-power and local (e.g., rockhopper), commercial (e.g., google cloud), or open-source (e.g., whisper) with varying model sizes and conclude that whispermedium outperforms the rest on kids' speech at authentic home environments. finally, we evaluate the slu pipeline by running our best-performing nlu models, diet+convert, on vad-asr output to observe the significant effects of cascaded errors due to noisy voice detection and speech recognition performance with kids in realistic home deployment settings. in the future, we aim to finetune the whisper asr acoustic models on kids' speech and language models on domain-specific math content. moreover, we consider exploring n-best-asr-transformers @xcite to leverage multiple whisper asr hypotheses and mitigate errors propagated into cascading slu.