Error-preserving Automatic Speech Recognition of Young E nglish Learners’ Language (2024)

Abstract: One of the central skills that language learners need to practice is speaking the language. Currently, students in school do not get enough speaking opportunities and lack conversational practice. The recent advances in speech technology and natural language processing allow the creation of novel tools to practice their speaking skills. In this work, we tackle the first component of such a pipeline, namely, the automated speech recognition module (ASR). State-of-the-art models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners’ speech. Second, most ASR systems contain a powerful language model, which smooths out mistakes made by the speakers. To give corrective feedback, which is a crucial part of language learning, the ASR systems in our setting need to preserve the mistakes made by the language learners. In this work, we build an ASR system that satisfies these requirements: it works on spontaneous speech by young language learners and preserves their mistakes. For this, we collected a corpus containing around 85 hours of English audio spoken by Swiss learners from grades 4 to 6 on different language learning tasks, which we used to train an ASR model. Our experiments show that our model benefits from direct fine-tuning of children’s voices and has a much higher error preservation rate.

## introduction speaking is one of the core competencies to be developed in foreign language classes and the second most widely used skill in everyday-life communication @xcite . for students to successfully acquire speaking competencies, they must be trained from an early stage in the language learning process and in a systematic manner. however, speech production is a highly complex process that is often not addressed adequately in class-rooms. the main issue is that students often do not get enough speaking opportunities @xcite , and lack extended conversational practice @xcite . the recent advancements in both speech processing @xcite , and conversational dialogue systems @xcite provide an opportunity to increase the speaking practice of language learners using automated tools. the work presented in this paper is part of a larger effort to develop an interactive, voice-driven chatbot with which learners can practice their interactive speaking skills. the bot is designed as a conversation partner that adjusts to the proficiency level and interests of the students and provides corrective feedback to support their language development. one key issue is the automated speech recognition (asr) module, which transcribes the utterances of the language learners into text to be processed in downstream tasks (e.g., speaker-error analysis, dialogue systems, inter alia). the focus of this work is to adapt the asr module to handle children's speech in a language learning environment. the core challenge for the asr system in this setting is not only to transcribe the speech but to make sure that the errors made by the language learners are transcribed faithfully. this is needed to provide language learners with corrective feedback, which is a key component of foreign language development. it prompts learners to notice errors and is likely to lead to utterance repair, which, in turn, facilitates language development @xcite . our investigations showed that current state-of-the-art asr models tend to correct the speakers' errors, which renders giving corrective feedback impossible. the second challenge for the asr system is handling spontaneous children's speech since most of these systems are trained on adult read-aloud error-free corpora recorded by native speakers @xcite . children's speech, especially spontaneous speech of language learners, differs significantly from readaloud speech of native adult speakers (shivakumar and georgiou, 2020). children's' speech has a different range of sound frequencies @xcite , a high within-subjects variability @xcite and a high inter-speaker variability in different age groups @xcite . these challenges yield three research questions, which we address in this work: the "verbatimness" of an asr transcript? 2. how well do current pre-trained asr systems perform on learners' spontaneous english productions, with respect to error preservation and in general? 3. does fine-tuning pre-trained systems with data from young learners lead to improved error preservation in the asr transcripts? ## dataset: spontaneous speech of young learners of english we now describe the dataset that we collected for the purpose of this research. it contains 85 hours of audio recordings of spontaneous speech by young swiss learners of english. each recording is paired with a verbatim transcript that contains error annotations. ## conclusion and outlook our work shows that state-of-the-art asr systems have difficulties handling young learners' speech; furthermore, they tend to correct the errors made by the speakers, which makes the downstream identification of speaker errors and provision of corrective feedback impossible. thus, we collected around 85 hours of children's language learner speech data, which we used to fine-tune a custom model. our model outperforms all the others (including whisper-large) in terms of error preservation (word-based error preservation rate, wepr) and surpasses the english models of comparable size (≈ 300m parameters) by a large margin in terms of word error rate. thus, our research shows the necessity of using targeted data (in this case, children who learn a foreign language) to fine-tune an asr module, which is useful in downstream tasks. the focus of this work lies in a) investigating the utility of existing systems and b) creating an adequate asr system that can be used as part of a language learning support tool to increase the students' speaking opportunities. as a next step, we will investigate how to enhance error preservation. for this, training larger models is the most straightforward approach. however, we also plan to train the asr system jointly with error annotations. for this, we started the creation of more detailed error annotations. initial results have shown that verbal errors are the largest error category for young learners of english in switzerland (with about 22% of all errors) , and within these, wrong subject-verb agreement is most frequent. similarly, investigating how to handle frequent code-switching to german words or sentence fragments is an unsolved issue that needs to be addressed to improve downstream tasks. even whisper-large, which can handle multiple languages in principle, did not perform well in detecting code-switching. finally, we aim to evaluate asr models in the context of integrating them with a conversational agent and corrective feedback. a small model can preserve errors better than stateof-the-art pre-trained models, which was the main scope of this work. no performance tuning. since the scope of this work is to understand if the usage of young learners' speech data is beneficial for our purposes, we did not tune the performance of our model. that is, we did not perform any hyper-parameter tuning or any other methods to increase performance (e.g., joint prediction of errors using a language model). thus, there is still a large margin of improvement using our dataset.