Automated speech recognition of I ndonesian- E nglish language lessons on Y ou T ube using transfer learning (2023)

Abstract: Experiments to fine-tune large multilingual models with limited data from a specific domain or setting has potential to improve automatic speech recognition (ASR) outcomes. This paper reports on the use of the Elpis ASR pipeline to fine-tune two pre-trained base models, Wav2Vec2-XLSR-53 and Wav2Vec2-Large-XLSR-Indonesian, with various mixes of data from 3 YouTube channels teaching Indonesian with English as the language of instruction. We discuss our results inferring new lesson audio (22-46% word error rate) in the context of speeding data collection in diverse and specialised settings. This study is an example of how ASR can be used to accelerate natural language research, expanding ethically sourced data in low-resource settings.

## introduction accent, speaker-class characteristics, and the use of dialects are among many factors impacting automatic speech recognition (asr) performance @xcite . the dominance of 'high-resource' languages in natural language processing (nlp) and impact of market forces have produced strong outcomes for some applications of asr when dialects, accented speech or particular speaker populations are excluded @xcite @xcite . however, many human speech scenarios, especially outside monolingual english contexts, require technologies more robust to language mixing and situated language usage -as well as performance measures of these technologies that prioritise the needs of users @xcite . this study worked with data from a non-standard context, that is, data from three youtube channels teaching indonesian with english as the language of instruction. it records the teaching practice of three teachers who: 1) explore a broad definition of 'indonesian language', 2) demonstrate various linguistic behaviours associated with teaching (e.g., hyper-articulation and repetition), 3) would typically be described as 'accented' in either one or both languages, and 4) recorded their speech amidst background noise, adding music and sound effects. we hypothesised that repetition and simplifications in 'teacher-talk' intended to create comprehensible input for students @xcite , and the use of transfer learning, could balance the many challenging characteristics in the data and allow asr to create useful transcriptions for editing and analysis. in recent years, transfer learning approaches have achieved state-of-the-art asr performance on benchmark tasks with small quantities of data @xcite . these approaches fine-tune a base model previously trained on a large dataset. some pretrained models have been made publicly available, allowing more people to take advantage of their performance, and their advantages to be shared more equitably @xcite . this study assessed her indonesian accent to be jakartan, and her english as mixing aspects of hong kong, singaporean, american, and australian accents. here the participant demonstrated some of the linguistic stancetaking described by @xcite , as she explained various address terms or substitutions for english 'you' appropriate in a market in jakarta when buying an iphone. [ ] -square brackets are additions from the transcriber to clarify meaning. ( ) -are translations and notes on linguistic and audio features. underlined text is in a language other than standard indonesian or english claims of state-of-the-art performance from finetuning a pretrained asr model with as little as 10 minutes of labelled data @xcite often depend on large-vocabulary language models @xcite . for contexts where matching language models are not readily available, more realistic results are to be expected, such as in @xcite where median word error rate (wer) ranged from 18-66% for cook islands maori. even with language models, wers remained high for low-resource languages: 32.91% for read speech in bemba language in sikasote and anastasopoulos (2022) and 48% for kurmanji kurdish in @xcite . the aim of this study was to achieve a useful level of accuracy in machine transcription, creating drafts for human correction to expand the online indonesian learning dataset (oil) @xcite . the study used the elpis asr toolkit to fine-tune models with a small set of human transcribed data. we trialled different base models, parameters, and mixes of fine-tuning data against various evaluation measures to better understand the performance of the tools and achieve this goal. the rest of the paper is organised as follows: we begin with sociolinguistic and language-teaching commentary on the data, and then provide information about the experiment design, fine-tuning parameters, and standardised results. we discuss how different models performed on audio from new lessons and for different speakers. finally, we reflect on technologies guided by direct and indirect user need, especially how evaluation measures inform decisions about usability of machine transcription for downstream tasks such as inference editing. ## methodology the experiments used elpis, a tool to aid linguists to apply sophisticated asr tools and approaches such as kaldi @xcite , wav2vec2 @xcite . elpis enabled us to work with asr base models that are available on the hugging face hub foot_0 , a repository of public and private datasets and models suitable for machine learning. models trained in elpis were uploaded to the hugging face hub, and then used for subsequent analysis. an initial dataset of manually transcribed audio from youtube videos totalled 1 hour and 35 min-utes (19 lessons). this initial dataset was used to fine-tune asr base models. inference texts from an additional seven lessons were used as a draft for human editors to expand the corpus to 2 hours and 13 minutes (26 lessons) of transcribed data. we used a mixed methods approach to analyse our results, supplementing standardised asr evaluation with qualitative commentary on transcription workflow and corpus analysis. ## results the speaker specific models fb_jer_e60 and nlp_all_jer_e60 achieved the lowest wer from elpis-internal train/test splits (30.39% and 32.51% respectively). train/test evaluation is compared in the next best training evaluation scores were from models fine-tuned with all our data: ind_nlp_all and fb_all 5 . experimental models finetuned with a subset of data 6 from teachers who were long-term residents of indonesia (models with suffix _natind) had higher wer. while indonesian nlp reports wer of 14.29% for the base model ind_nlp, it did not score well on inference of our multilingual audio ( our own fine-tuned models were a dramatic improvement on these results. the best wers on inference files ranged from 22.86% to 43.65%. no single model consistently achieved the best wer, cer or correct sequence of six or more on inference of new audio ( qualitative human rating of inference indicated inferences from fb_all and ind_nlp_all were suitable for editing (see r in while machine transcription accelerated the transcription process, the process of fine-tuning base models and preparing data required an upfront investment which was not compensated for by this acceleration over seven inference files. we hope that our upfront investment can be useful to others via our models and data on hugging face. the process of editing machine transcriptions revealed workflow and evaluation needs. it also impacted human transcriber interpretation of the data, provoking discussion of how multilingual, accented, language teaching plays out. meanwhile, so-called 'errors' in inference were less concerning than they would be in other fields where accuracy is of paramount importance (such as in medical applications of asr @xcite . figure 4: insights into accented speech via error correction of inference. correct sequences: length and location. the placement of correct sequences of inference influenced the usability of an inference as an editable draft. specifically, longer correct sequences and those that were left-aligned reduced the time spent editing, an impact not measured by wer. similarly, word final spelling errors were less disruptive to the editing process as they required less keystrokes to correct. as an example, for reference 'satu ribu', the inference 'a satu ribu' is more disruptive than 'satu ribua a'. this is despite having lower wer and cer. accented speech. the reflection of speech behaviours in machine transcription also stimulated reflection on teacher pronunciation. the use of context and language knowledge in understanding and interpreting teacher speech is highlighted in the following examples. in a lesson from gunawan tambunsaribu (gun) the word final 't' in 'foot' was converted to a 'd' in the machine inference (see figure 4 ). human analysis found the production of 'oo' (in 'foot') by the participant matched with the common grapheme to phoneme pair in words like 'too' and 'roo'. however, 'foot', confusingly given it's spelling, is pronounced like 'put'. the inference highlighted the transfer of indonesian vowel production and possibly a speech error resulting from irregularities in orthographic conventions in english. similar to a language model (lm), a human transcriber editing the inference in figure 4 would step through each word, finding 'adde' to be a non-word. presuming correction to 'edge' was substituted, the sentence 'the edge of the food' would be judged improbable and corrected despite the vowel production described in the previous paragraph. further, a human and a lm would preference 'foot' over 'food' as the preceding data indicates body parts are likely, being the topic of the lesson. in another inference, the transcription of 'tv' as 'tipi' matched the participant's production of the word. the inference reflected a common characteristic displayed by indonesian speakers in which fricatives and plosives foot_4 are not always differentiated @xcite . language models. often the addition of a lm will be used to improve asr and other nlp. however, in this application of asr, introducing a lm would be unlikely to assist as code-switching behaviours, non-standard grammar and accents, as well as situated language from the language learning setting has largely been excluded from language technologies @xcite . in other words, lms built from data similar to ours are not yet available. in our study, human transcribers took on the role of lm correction. however, this placed signficant demands on transcribers to be multilingual and knowledgeable in the language learning setting. these demands make transcription and error correction of this data a true bottleneck. optimistically perhaps, we see this work as potentially enriching for teachers and their reflective teaching practice. it can bring attention to interlanguage and movement between native speaker modelling and intermediary productions of sounds and language structures. future work. asr systems fine-tuned with very small quantities of data often rely on lms trained with large amounts of text data @xcite . these systems typically use a multilingual base model that has been fine-tuned to a monolingual language, with a monolingual lm foot_5 . in this setting, further work to develop a complex multilingual lm could improve results with a pre-trained multilingual model fine-tuned with multilingual data. a major challenge in the development of multilingual lms for contexts such as this is the varying inter-utterance and intra-utterance code-switching that occurs in teacher speech @xcite . these switches are likely to disrupt potential identification of language for an n-gram lm. an n-gram sequence identified as english may in fact erroneously negate a correctly identified indonesian word in the sequence. further work to investigate initial diarisation/language identification may be a fruitful approach to handling this language complexity. such an approach was taken in @xcite to assist with mixed data from adult and child speakers. in this setting, multiple mono-lingual lms used on identified languages and then compiled could be helpful @xcite . however, with the degree of hyperarticulation and accent evident in this study's audio, reliable language identification itself is likely to be difficult. the prevalence of merged word errors identified in inference texts (e.g. 'reduplicationand' rather than 'reduplication and'), would be resolved in a monolingual system through the use of a lm. given a lm may not work well on data like this, future work for complex language systems could investigate the benefits of a rudimentary splitting step based on matches with combined bigrams or trigrams from a multilingual vocabulary list. audio content analysis indicates that errors concentrated at the beginning and ends of files were associated with background music. given the consistent poor inference text in these sections, better performance would be likely by excluding these sections of the files. ## conclusion our findings offer a reality check of asr performance with 'difficult' data, including newer techniques of transfer learning. our results clearly indicated that publicly available models for indonesian are not suitable for processing holistic language teaching data. inference from a model fine-tuned on a small dataset of complex language was much more useful. the wer remained high, however, rather than discarding results based on the industrystandard/internal expectations, we persisted and edited inference text to expand our dataset. the resulting insights into user workflows encourage investigation of task-specific evaluation measures. meanwhile, insights into data characteristics that were highlighted by editing the inference texts go some way to counterbalancing the time spent in interactions with asr output by language-teaching professionals. our ethically sourced dataset foot_6 and best models foot_7 are available on hugging face.