An (unhelpful) guide to selecting the best ASR architecture for your under-resourced language (2023)

Abstract: Advances in deep neural models for automatic speech recognition (ASR) have yielded dramatic improvements in ASR quality for resource-rich languages, with English ASR now achieving word error rates comparable to that of human transcribers. The vast majority of the world’s languages, however, lack the quantity of data necessary to approach this level of accuracy. In this paper we use four of the most popular ASR toolkits to train ASR models for eleven languages with limited ASR training resources: eleven widely spoken languages of Africa, Asia, and South America, one endangered language of Central America, and three critically endangered languages of North America. We find that no single architecture consistently outperforms any other. These differences in performance so far do not appear to be related to any particular feature of the datasets or characteristics of the languages. These findings have important implications for future research in ASR for under-resourced languages. ASR systems for languages with abundant existing media and available speakers may derive the most benefit simply by collecting large amounts of additional acoustic and textual training data. Communities using ASR to support endangered language documentation efforts, who cannot easily collect more data, might instead focus on exploring multiple architectures and hyperparameterizations to optimize performance within the constraints of their available data and resources.

## introduction the majority of significant academic and industry research on automatic speech recognition (asr) @xcite @xcite @xcite has been evaluated on a small set of english language datasets @xcite . word error rates (wer) for english asr now approach those of human transcriptionists @xcite , and speakers of english can now reliably use asr for text entry when using mobile devices. this level of accuracy, however, is attainable only for the handful of the world's 7000 languages that, like english, have abundant training resources. most of the world's languages, even ones spoken by tens of millions of speakers, currently lack datasets prepared specifically for training asr models. the datasets that do exist are typically much smaller than english asr datasets that have been available for decades, with no more than a few dozen hours of acoustic training data. as the common voice project @xcite has shown, collecting large amounts of data for widely spoken languages is possible, but using this kind of platform is likely to be impractical for the roughly 40% of the world's languages that are endangered @xcite . a similar percentage of languages -again, even many that are widely spokenlack an established writing system, which presents other obstacles to building large asr corpora. fortunately, existing methods for training accurate asr models for english and other highresource languages can be adapted to low-resource settings. some toolkits include recipes for smaller datasets that require the training of fewer parameters. other approaches rely on fine-tuning acoustic models pre-trained on massive multilingual speech datasets. most recent work using these approaches, however, does not compare the performance of multiple competitive architectures across multiple diverse small asr datasets. thus, while we have access to transformative technology that can be harnessed to build reasonable models for languages with limited resources, we do not know which of the popular architectures is "better" or whether features of a particular dataset or language might make one architecture more suitable than another. in this paper we explore four different popular asr architectures, three of which are currently considered state of the art, that can be used even in language hh:mm # speakers # lm tokens audio quality audio source bemba train test 17:17 02:00 8 2 96k variable read speech wolof train test 16:49 00:55 14 2 600k high read speech swahili train test 10:00 01:45 n/a n/a 3m variable read speech and broadcast news seneca train test 09:57 02:04 11 11 76k variable fieldwork fongbe train test 07:35 01:45 25 4 990k high read speech iban train test 07:00 01:00 17 6 200k high broadcast news hupa train test 06:06 01:31 1 1 41k variable fieldwork oneida train test 03:23 00:51 7 4 18k variable fieldwork quechua train test 03:00 00:45 n/a n/a 8.1k variable conversations bribri train test 00:29 00:11 n/a n/a 4k variable fieldwork guarani train test 00:19 00:07 n/a n/a 1.2k variable read speech ## related work although most of the notable advances in asr have focused on english and a few other languages with abundant data, there has been substantial interest in asr for languages with minimal training resources for quite some time @xcite . much of the work from the 2010s focused on the languages of the iarpa babel project @xcite @xcite . research initiated with the babel datasets on methods of transfer learning and data augmentation in low-resource settings has continued apace @xcite @xcite . with the success of the kaldi toolkit, researchers began to collect and freely distribute their own kaldi-ready datasets for under-resourced and endangered languages, several of which are explored in this paper @xcite @xcite @xcite . more recent work has explored training monolingual endto-end models with substantially larger datasets than those used here @xcite , as well as transfer learning and fine-tuning from pretrained multilingual @xcite or english models @xcite . ## datasets five of the datasets explored here are freely available datasets built by researchers, sometimes in collaboration with speech communities, specifically for training asr models for widely spoken but under-resourced languages of the global south: bemba (sikasote and anastasopoulos, 2022), fongbe (laleye et al., 2016), wolof @xcite , swahili @xcite , and iban @xcite . three datasets (quechua, bribri, guarani) were created from existing recordings for the 2022 americasnlp workshop shared task foot_0 . the remaining datasets for three endangered languages of north america (hupa, oneida, and seneca) were created using existing linguistic and community fieldwork recordings available to the authors through the affiliation of one of the authors with one of these communities and the generosity of the community elders. while nearly any recorded speech can be transcribed and used to train an asr system, a common approach for building a new asr dataset is to ask speakers of the language to read aloud provided texts, which obviates the laborious task of transcription. with this strategy, speakers are often recorded in a studio or similarly controlled environment, resulting in more consistent recording quality. alternatively, datasets can be created from existing audio data such as radio broadcasts or linguistic fieldwork recordings. such recordings are often already transcribed but need to be segmented and time-aligned with the transcripts, which must often be done by hand. information about the linguistic characteristics of the eleven languages is provided in ## asr architectures the goal of this work is to explore whether any one of several popular and state-of-the-art asr architectures is especially well suited for building models with small amounts of training data. we train models on the the eleven datasets described in section 3 using four different architectures: ## results figure 1 shows the word error rates (wer) for four of the five approaches (kaldi dnn, wav2vec xlsr with and without a language model (lm), and whisper) when trained and tested on each of the eleven datasets. note that prior baselines reported in the papers associated with the datasets for wolof, swahili, fongbe, hupa, and iban, using non-s.o.t.a. architectures, and bemba, using a slightly different configuration of wav2vec xlsr, are lower than the best reported architecture here. no prior wer results have been reported for the oneida, quechua, bribri, and guarani datasets. we observe a large variation in wer across languages, which should not be surprising given the great variability in the quantity of training data, the type and audio quality of data collected, and the linguistic features of these languages. datasets of less than 3 hours had consistently high wers, but across the other datasets, there does not appear to be a clear relationship between amount of audio training data and wer. though not shown in figure 1 , espnet yielded the worst performance by far for all languages, with only wolof, the second largest dataset, achieving a wer below 65%. again, this is not surprising given that this espnet recipe @xcite was proposed for a much larger 60-hour indigenous language dataset. more interestingly, we see no consistent ranking of the remaining four approaches across the eleven datasets. using an lm during decoding with wav2vec xlsr always yields some improvement in wer over not using an lm, but the differences are often quite small. notably, swahili, which has the largest lm, sees only a tiny reduction in wer when that lm is used during decoding. the kaldi hybrid dnn, despite being outdated, outperforms more than one of its state of the art rivals for seneca, fongbe, iban, and quechua. whisper is dramatically better than other models for wolof and hupa, figure 1: word error rates (wer) for each dataset under each architecture. a lower wer indicates fewer errors and higher accuracy. there is no clear winner among the other models, with even the kaldi hybrid dnn often outperforming more recent state of the art fine-tuning approaches. where wer has been previously reported, one or all of these models outperform the reported baselines in the published papers for these datasets. but substantially worse for fongbe and quechua. though closely related and typologically similar, seneca and oneida show very different patterns, as do fongbe and wolof, two related languages with datasets recorded under similar conditions. the wer for swahili is relatively stable across architectures, while wer is quite variable for wolof, hupa, fongbe, and oneida. the rankings do not appear to be related to the method of speech collection (read vs. spontaneous) or the consistency of audio quality. in addition, whether or not a language is tonal, like bemba, fongbe, and bribri, does not appear to predict the relative rankings of the four architectures. we do note, however, two potential patterns, which merit further investigation with a larger set of languages. first, fongbe, the only language of the eleven with isolating morphology (i.e., limited affixation) is one of only two languages where whisper yielded the highest wer of the four systems. second, the three languages with the largest phonesets, wolof, swahili, and hupa, yielded the same relative ranking, with whisper performing the best and kaldi the worst. although there is certainly not enough information here to draw conclusions, it is plausible that the design of a particular training architecture or the content of the pretrained models could render a system more appropriate for a language with a particular linguistic property. ## conclusions under-resourced language communities, whether large or small, need to know how to invest their lim-ited resources when developing an asr system for their language. our findings suggest, unfortunately, that there are no obvious or simple guidelines to follow. our future work will expand the set of languages explored here in order to establish connections between expected model performance and linguistic features and dataset characteristics. we also plan to explore the impact of language model size and domain on asr accuracy and the relationship between language model and morphology.