C aptain A - A mobile app for practising F innish pronunciation (2023)

Abstract: Learning a new language is often difficult, especially practising it independently. The main issue with self-study is the absence of accurate feedback from a teacher, which would enable students to learn unfamiliar languages. In recent years, with advances in Artificial Intelligence and Automatic Speech Recognition, it has become possible to build applications that can provide valuable feedback on the users’ pronunciation. In this paper, we introduce the CaptainA app explicitly developed to aid students in practising their Finnish pronunciation on handheld devices. Our app is a valuable resource for immigrants who are busy with school or work, and it helps them integrate faster into society. Furthermore, by providing this service for L2 speakers and collecting their data, we can continuously improve our system and provide better aid in the future.

## introduction proper pronunciation is needed to build confidence in second language (l2) learners and is essential for effective communication and language acquisition @xcite . l2 adult learners, who might not have regular exposure to the target language during their everyday life, may lack sufficient opportunities to practise and receive corrective feedback. with recent advances in automatic speech recognition (asr) technologies, computerassisted pronunciation training (capt) apps have become more and more effective in helping l2 learners. these apps can immediately give the users feedback on their pronunciation at their convenience. however, while popular languages such as english have many pronunciation applications @xcite @xcite , there are fewer resources available for finnish l2 learners. to the best of our knowledge, there was no similar app for capt in finnish before this work. the main challenge in developing capt applications for finnish and other low-resource languages is the lack of data from l2 speakers. furthermore, if the l2 corpus is not annotated at the phoneme level, it makes developing an app for mispronunciation detection (md) more complicated. we designed our captaina app to function as well as possible using all available data and add the possibility of collecting users' data after the pilot phase (figure 1 ). such information will help evaluate the app's effectiveness for language training and improve our model's performance to better address students' needs in later versions. recent works from wu et al. (2021) and xu et al. ( 2021 ) have demonstrated the effectiveness of end-to-end systems with transformer-based architectures for english md. while we focus more on practicality, we use a similar approach without a detailed annotation dataset for finnish. ## dataset one of the major challenges that we needed to overcome was the limited data at our disposal. we should note that for the english language, several datasets are available with phoneme level annotation @xcite @xcite . unfortunately, no such public finnish resources exist. thus we opted to use the data collected during the digitala project @xcite as our primary corpus. this dataset includes ratings from language experts on pronunciation, fluency, lexical, grammatical and the holistic overall level for each audio file, but it does not have phoneme level information. the digitala corpus consists of free-form and read-aloud speech, from which we selected 768 short read-aloud samples as those matched our intended scenario most closely. this gave us approximately 60 minutes of audio with the overall pronunciation ratings ranging from 1 to 4, with 4 being the best. the rating is for the whole pronunciation task and not individual phonemes. the lowest pronunciation level (1) contains approximately 2,200 phones, the highest one (4) has only 576 phones, while the remaining 14,000 phones are split almost equally between levels 2 and 3. the corpus was also transcribed by third parties who were not language experts. the small size of the digitala corpus and the lack of phoneme annotation meant it was not suitable for training or finetuning for the md task. however, as there were no better alternatives, we used the digitala read-aloud transcript as a replacement for the evaluation set. consequently, we needed another dataset to train our models. after some preliminary experiments, we selected the finnish parliament corpus @xcite , a publicly available corpus without any statistically significant use of dialects @xcite . by training our models for the asr task with suitably chosen native speakers' samples, we expected the models could learn the features of native finnish speech and have the potential to identify deviations made by l2 speakers. as a first step, we filtered the most suitable portion of the data, by selecting speeches with low or average speaking rates (which is the most similar to how l2 learners speak). as an additional step, we also restricted the data by excluding older (50+) speakers, since our target audience is generally younger immigrants. the last step in data prepara-tion was the splitting of the 281 hours of data into 75% for training, and 25% for tuning hyperparameters and evaluating the speech recognition models. we should note that we also used two publicly available reference models, called finnish-nlp foot_0 and finnish-nlp-s foot_1 . both have been trained with 228 hours of finnish parliament data and approximately 47 hours of data from other sources. ## results to validate our models, we computed their character error rate (cer), recall (percentage of mispronunciations correctly detected) and precision (the ratio of detected mispronunciations actually being mispronunciation, according to a native finnish listener) using the digitala read-aloud corpus. the empirical results can be seen in one common type of mistake made by finnish l2 speakers is related to the duration of phonemes. in our test set, excluding other types of mistakes, the xls-r-10 model achieved approximately 68% recall rate where users pronounce short (single) phonemes too long, and 57% where users pronounce long (double) phonemes too short. although our goal was to develop a capt app that works well for all users, due to the composition of the training data and the randomness in the optimization process, the system still developed some biases towards subgroups of users. for samples in the same pronunciation level, the xls-r-10 model has better md for male speakers, with a higher recall rate (66.0% vs females 59.8%), and a slightly higher precision (31.2% vs 27.6%). the gender distribution in the test set is balanced, with 53% male samples and 47% female. we will release more detailed analyses in the future once we collect more data from users. while it is possible to use the training part of the digitala corpus for finetuning our wav2vec 2.0 models, we could not control the pronunciation quality, as the speakers are l2 learners and there is no phoneme annotation. in our preliminary experiments we found that finetuning with bad pronunciation data led to lower performance in md. when making mispronunciation, the users are also advised to refer to the app multimedia pronunciation instructions (figure 4 ). ## conclusion in this paper, we presented the prototype of cap-taina, an app that helps language learners practise finnish pronunciation. because of the lack of data available for phoneme level pronunciation mistakes, our solution is based on multilingual wav2vec 2.0 models, which are finetuned for native finnish asr. by running the l2 learners' utterances through the asr without a language model, we predict pronunciation errors and probability scores that indicate the success of pronunciation. the resulting models are validated by measuring cer, recall and precision for samples of different levels of pronunciation judged by human experts. in the future, we plan to collect user data (feedback and audio) with our app to update the models and improve the self-study application. captaina was trained on a native speaker corpus and evaluated on 1 hour of transcribed l2 data, without requiring detailed annotation from experts or fine-tuning l2 data. its performance could be a motivation to apply a similar setup to other lowresource languages. based on the initial result of captaina, we plan to develop a similar capt app for finland swedish.