End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics (2022)

Abstract: The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education. The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions. We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing MathBERT representations for potential enhancement to the Natural Language Understanding (NLU) module. We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.

## introduction following the advances in artificial intelligence (ai) research, building innovative applications to support education can present exciting opportunities with a positive and broader social impact. the united nations (un) sustainable development goals 1 (sdgs) @xcite represent an urgent call for action to help address critical global problems, where education is among the top five of these development areas (i.e., poverty, hunger, health, education, gender equality). across many 1 https://sdgs.un.org/goals disciplines, improving the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages, which is a fundamental component of comprehensive and successful stem education (i.e., science, technology, engineering, and mathematics). languagebased ai systems are starting to mature to a point where they could play a significant role in helping students learn and practice mathematical concepts. despite its importance, applied natural language processing (nlp) technologies for enhancing mathematics education still remain a highly under-explored area of research. this study presents a task-oriented spoken dialogue system (sds) developed to facilitate playbased learning of basic mathematical concepts for early childhood education. the system has been developed in the lab and evaluated via real-world deployments at school while the students are learning and practicing basic math concepts with multimodal interactions. these fundamental early math concepts and basic operations include constructing numbers using ones and tens, counting, addition, subtraction, measurement of length and size, etc. the multimodal interactions involve speech-based interactions to answer early math-related or gamerelated questions, counting and placing learningspecific tangible objects (i.e., manipulatives) in a visually observed space, touch-based interactions with the 1-to-100 number grid projected on the wall, to name a few. this work discusses our efforts to improve the modular sds pipeline built for game-based math learning and perform an end-to-end evaluation with various sds components using real-world deployment outputs. the main sds module we focus on investigating is natural language understanding (nlu). the nlu arguably is the most critical component of goal-oriented dialogue systems that enables efficient communication between humans and intelligent conversational agents via application-specific comprehensive sub-tasks. intent recognition (ir) is at the heart of these nlu tasks, where the goal is to identify users' objectives from the input text and determine their intentions. the representation of human language is a crucial factor determining the success of conversational agents, especially in real-world applications. in the math learning domain, this language representation gains further significance due to the quite peculiar nature of mathematical language. with that motivation, we explored employing math-bert @xcite representations for potential enhancement to the nlu module of our play-based math-learning system. mathbert is a recently proposed language model created by pretraining the well-known bert-base model @xcite on a large mathematical corpus. we compared the nlu results obtained by simply using bert @xcite and @xcite representations versus the new mathbert representations. we further investigated the two variations of mathbert models, one pre-trained with mathematics-specific vocabulary and the other with bert-base vocabulary, to see their effects on our domain-specific math-learning nlu task. as most of the application-specific and modular sds pipelines do, our task-oriented sds contains particular building blocks or modules for automatic speech recognition (asr), natural language understanding (nlu), multimodal dialogue management (dm), natural language generation (nlg), and text-to-speech (tts). at its current stage of research & development, for practical reasons, we employ template-based responses at the nlg module with off-the-shelf tts. thus, this study emphasizes more on speech recognition (asr), intent understanding (nlu), and response selection (dm) tasks for a conversational agent that supports elementary math learning. for a complete end-to-end evaluation of our task-oriented sds, we evaluated the asr, nlu, and dm components to understand how error propagation affects the overall performance in real-world scenarios. ## language understanding methods for the nlu module within our early mathlearning dialogue system pipeline, we have examined numerous options for intent recognition and built our nlu models on top of the rasa opensource framework @xcite . ## conclusion and future work improving the quality of mathematics education is vital in developing problem-solving and critical thinking skills for younger learners, which are fundamental for comprehensive stem education. this study showcased a task-oriented sds built to promote play-based math learning for early childhood education. the conversational ai system is implemented and evaluated on real-world deployment data collected in classrooms while the kids are practicing basic math concepts. we presented our attempts to enrich the modular sds pipeline for gamified math learning and prosecuted an endto-end evaluation using several sds components tested on real-world deployment data. for nlp applied to math education, language representations can play a significant role due to the exceptional nature of math language. we investigated employing the language representations from the mathbert model created by pre-training the bert-base on mathematical corpora. we compared the intent recognition results obtained using bert and con-vert representations versus the recently proposed mathbert embeddings on top of the diet architecture for nlu. to perform an end-to-end evaluation of our sds pipeline, we evaluated the asr, nlu, and dm modules to investigate how error propagation affects the overall sds performance in real-world math-learning scenarios. in future work, we aim to explore adopting the n-best-asr-transformer architecture @xcite to utilize multiple asr hypotheses. this approach can improve the intent recognition performances and mitigate recognition errors propagated into the asr+nlu+dm pipeline due to using only the top asr hypothesis. another future direction is to enhance math-specific language representation learning by pre-training the convert model on large math corpora, especially tailored towards early math education (e.g., pre-k to 2ndgrade math curriculum), and then fine-tuning them on our nlu tasks for game-based math learning.