How Important is a Language Model for Low-resource ASR ? (2024)

Abstract: N-gram language models (LMs) are the innovation that first made large-vocabulary continuous automatic speech recognition (ASR) viable. With neural end-to-end ASR architectures, however, LMs have become an afterthought. While the effect on accuracy may be negligible for English and Mandarin, jettisoning the LM might not make sense for the world’s remaining 6000+ languages. In this paper, we investigate the role of the LM in low-resource ASR. First we ask: does using an n-gram LM in decoding in neural architectures help ASR performance? While it may seem obvious that it should, its absence in most implementations suggests otherwise. Second, we ask: when an n-gram LM is used in ASR, is there a relationship between the size of the LM and ASR accuracy? We have discovered that gut feelings on this question vary considerably, but there is little empirical work to support any particular claim. We explore these questions “in the wild” using a deliberately diverse set of 9 very small ASR corpora. The results show that: (1) decoding with an n-gram LM, regardless of its size, leads to lower word error rates; and (2) increasing the size of the LM appears to yield improvements only when the audio corpus itself is already relatively large. This suggests that collecting additional LM training text may benefit widely-spoken languages which typically have larger audio corpora. In contrast, for endangered languages where data of any kind will always be limited, efforts may be better spent collecting additional transcribed audio.

## introduction including an n-gram language model (lm) during decoding within an automatic speech recognition (asr) system is a long-established critical practice @xcite @xcite . this has motivated appreciable research exploring different ways to build or augment lms in order to improve asr accu-racy @xcite @xcite . with recent advances in end-to-end neural asr, the role of the lm has diminished somewhat dramatically @xcite , with the lm becoming largely optional. using an lm in decoding is often challenging within many state-of-the-art deep learning asr toolkits, where it is typically a poorly documented command-line option or a feature that users themselves must implement @xcite . a number of studies have incidentally shown that decoding with an lm within s.o.t.a. neural architectures yields competitive or superior performance @xcite @xcite @xcite , but the impact of the lm in these papers is discussed briefly, if at all. additionally, the languages in these studies -typically english -are not at all constrained by data availability. recent work by @xcite on 40 indian languages with more limited resources shows a consistent improvement in accuracy when using an lm, but the corpora are still enviably large, with over 17k hours of audio and billions of words. two recent studies probe the effect of lm size on asr performance in more genuinely underresourced settings. sikasote and anastasopoulos (2022a) report that a larger lm actually yields mildly worse performance for a 19-hour bemba corpus. @xcite explore the effect of lm size in six small corpora, finding a lack of word error rate (wer) improvement when using larger lms, even in simulated low-resource settings. the contradictory results in this prior work raise two questions. first, is an n-gram lm still a valuable component in asr when resources are limited? second, if using an lm does make a difference, how much additional text data beyond the transcripts of the audio training data should be used to train the lm -or in other words, how large should the lm be? training on additional texts could decrease the out-of-vocabulary (oov) rate and reduce the overall sparsity of the lm, but the impact of the lm and its size on asr accuracy remains strangely unexplored with few studies addressing the issue directly or thoroughly (sikasote and anastasopoulos, 2022a; @xcite . this study aims to provide empirical evidence to answer these research questions. we explore the impact of decoding with lms of different sizes within two asr architectures on a diverse set of small corpora. we very deliberately choose to work with many corpora of varying sizes, recording quality, and linguistic properties rather than simulating a range of sizes using a single corpus. we conduct further regression analyses to explore for each language whether asr performance is impacted by lm size and other factors of interest. addressing these questions has important implications for creating asr datasets for underresourced languages. for many indigenous and endangered languages, additional texts for training larger lms often come from a domain (e.g., the bible) that is drastically different from that of the audio (e.g., linguistic fieldwork). this sort of domain mismatch might yield richer lms that poorly model the target data. an improved understanding of the impact of n-gram lms on asr accuracy in such settings can help language communities choose how to allocate their limited resources for building more robust asr systems. ## related work there has been continuous progress over the past decade in advancing asr technologies for lowresource languages @xcite @xcite . some of these languages, such as wolof @xcite and swahili @xcite , have large speaker populations; with reasonable time and financial support, more data is obtainable. for others, especially indigenous and endangered languages, it is difficult to collect additional data, but asr can facilitate the creation of this data @xcite @xcite . our work goes beyond prior work that examines the impact of n-gram lms on asr performance (sikasote and anastasopoulos, 2022a; @xcite . first, we study nine languages from six language families, covering a diverse set of typological properties. second, we compare two asr architectures (section 3.3) for each language. this experimental design allows us to see whether the impact of lm size is consistent across both languages and models. ## experiments it is not our goal to find the best performing asr architecture or parameterizations. we are also not comparing wer across languages, a meaningless exercise given the range of languages and corpus size. instead, our goal is to examine, for each language, the effect of (1) decoding with an lm in an end-to-end system, and (2) increasing the size of the lm, in order to see whether similar qualitative trends hold across these nine diverse languages. ## results the wer results are presented in ## conclusions and future work our results over nine languages, corroborated with regression analysis, show that: (1) decoding with an lm yields consistently lower wer; (2) a larger lm infrequently improves asr accuracy. we propose that in order for a larger lm to be helpful, it needs to: (1) be much larger relative to the size of the transcripts of the audio; (2) result in lower oov rates for the audio test data; (3) be coupled with sufficient audio training data. we hope these results can be taken as guidance for asr dataset creation strategies for low-resource languages. for widely-spoken languages, where additional text data is often readily available, resources might be best directed toward gathering that data. for indigenous and endangered languages, limited resources might be better spent ethically increasing the amount of audio data. in addition to experimenting with more languages, particularly endangered languages, we see designing informative data selection and data augmentation methods for lm building as a fruitful avenue for future work. this in turn can perhaps mitigate the issues of (audio) data limitations particularly for endangered languages.