Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation (2023)

Abstract: The performance of automatic speech recognition (ASR) systems has advanced substantially in recent years, particularly for languages for which a large amount of transcribed speech is available. Unfortunately, for low-resource languages, such as minority languages, regional languages or dialects, ASR performance generally remains much lower. In this study, we investigate whether data augmentation techniques could help improve low-resource ASR performance, focusing on four typologically diverse minority languages or language variants (West Germanic: Gronings, West-Frisian; Malayo-Polynesian: Besemah, Nasal). For all four languages, we examine the use of self-training, where an ASR system trained with the available human-transcribed data is used to generate transcriptions, which are then combined with the original data to train a new ASR system. For Gronings, for which there was a pre-existing text-to-speech (TTS) system available, we also examined the use of TTS to generate ASR training data from text-only sources. We find that using a self-training approach consistently yields improved performance (a relative WER reduction up to 20.5% compared to using an ASR system trained on 24 minutes of manually transcribed speech). The performance gain from TTS augmentation for Gronings was even stronger (up to 25.5% relative reduction in WER compared to a system based on 24 minutes of manually transcribed speech). In sum, our results show the benefit of using self-training or (if possible) TTS-generated data as an efficient solution to overcome the limitations of data availability for resource-scarce languages in order to improve ASR performance.

## introduction self-supervised learning (ssl) enables speech representation learning without the need for (manually) labeled data. although this approach is very effective, pre-training an ssl model is costly. this cost (e.g., training time, resources, and memory) increases with the number of languages added to the model. furthermore, transferring information across languages, or extending a pre-trained model to new data or to a different domain is computationally expensive, and catastrophic forgetting may occur @xcite . to alleviate this, ssl models are therefore often fine-tuned on the target task with target domain data. for the task of automatic speech recognition (asr), fine-tuning approaches generally require less data, but training asr systems that perform well for languages with very little data remains challenging. this leads to (digitally) underrepresented communities and domains such as minority languages, regional languages and dialects not profiting sufficiently from most recent technological advancements. recent studies explored fine-tuning of pretrained self-supervised models for asr using speech from low-resource languages (e.g., @xcite , and difficulties of modeling resource-scarce languages and dialects were acknowledged in previous work @xcite . it remains an open question to what extent model performance is dependent on the amount of fine-tuning data and the type of language, when the total amount of available data for a language is limited. having a better understanding of how limited training data affects model performance paves the way for creating meaningful speech technology for a wider range of languages. in this paper, we fine-tune pre-trained ssl models for asr using varying amounts of data from four typologically diverse minority languages or language variants: gronings, west-frisian, besemah and nasal, which have a limited amount of data available. we specifically investigate whether data augmentation approaches can be used to generate additional training data to improve the performance of these models, particularly when very little resources are available. by using data from (ongoing) language documentation projects, we evaluate a real-world use of our experimental setup. previous work describes the benefits of data augmentation by adopting a self-training approach, which generates labels (i.e. transcriptions) for unlabeled speech (e.g., @xcite @xcite @xcite @xcite . various self-training methods are proposed, including iterative approaches, decoding with an external (textbased) language model, or filtering approaches that improve the quality of the generated labels. however, limited conclusions can be drawn from these works on the effectiveness of self-training in a very low-resource, real-world setting, as these studies either use datasets with more than 10 hours of data (which may not be available for very small languages), only considered modeling english, or reported average performance over a set of languages that strongly varied in terms of training data size. we therefore complement this work by investigating the benefits of self-training for four typologically different, true low-resource languages. to this end, we use a standard self-training approach to evaluate the potential benefit of a simple system in a real-world setup, which nevertheless yields substantial performance improvements (relative worderror-rate (wer) reductions up to 20.5%). in addition to self-training, several studies (e.g., @xcite @xcite reported on augmenting the training data with synthetic speech generated using a text-to-speech (tts) system. for this reason, we also examine whether this approach is useful in our low-resource setup. we recognize that not all very low-resource languages may have sufficient amounts of data available for tts development, and we therefore only generate synthetic training examples for gronings, one of the four low-resource languages in our dataset that has an existing tts system available. we show the benefit (i.e. up to 25.5% relative reduction in wer) of augmenting the training data by using an existing tts system, and analyze the effect of adding different amounts of synthetic speech on the model performance. our datasets, code, and newly trained models are publicly available. foot_0 ## data as indicated, we use transcribed speech from gronings, west-frisian, besemah, and nasal. for the lat-ter two minority languages, only four hours of manually transcribed speech data are available. for all language varieties, we therefore limit the amount of manually transcribed speech data to four hours. we divide each dataset into 80% for training, 10% for development and 10% for testing. the development and test sets therefore include approximately 24 minutes of speech, and the training set contains approximately 3.2 hours of transcribed speech. in line with @xcite , we allow for speaker overlap between the sets due to the limited number of speakers per language variant, as they found that it had limited effects on the performance of asr models. all data have been anonymized by assigning recordings a random identifier, and no other meta-information that could be used for identifying the speakers were collected or extracted. we obtained consent from the communities to publicly release the datasets for gronings, besemah, and nasal. the west-frisian data can be obtained by emailing the authors (islrn: 340-994-352-616-4). ## methods we fine-tune the pre-trained multilingual xls-r model with 317 million parameters on different amounts of training data from the four languages in our dataset @xcite . note that we chose the smallest publicly available pre-trained xls-r model to minimize the computational requirements needed for (reproducing) this study. xls-r is pretrained on approximately 436,000 hours of speech in 128 different languages. this data was collected from a variety of sources, including parliamentary speech (372,000 hours in 23 european languages), read speech from multilingual librispeech (44,000 hours in eight european languages) and common voice (7,000 hours in 60 languages), speech from youtube from the voxlingua107 corpus (6,600 hours in 107 languages), and conversational telephone speech from the babel corpus (approximately 1,000 hours in 17 african and asian languages). the majority of the training data is from indo-european languages (87%), and the language that is most represented is english (roughly 70,000 hours). while the model does include a small portion of west-frisian data (i.e. 15 hours), this is not the case for gronings, besemah, and nasal. the architecture and pre-training objective of xls-r are similar to those of wav2vec 2.0 @xcite . the model is trained as a single end-to-end system, and consists of a convolutional encoder, a quantizer, and a 24-layer transformer model. speech representations are learned through a contrastive task that is applied to the quantized encoder representations. after pre-training, the model can be fine-tuned for speech recognition using transcribed speech. a linear projection is added on top of the transformer network to predict characters from the transcriptions using connectionist temporal classification (ctc; @xcite . we include a multilingual model in our study, because previous work showed that multilingual pretraining transfers well to low-resource languages (e.g., @xcite . we experimented with fine-tuning other models (for example the dutch wav2vec 2.0 model included by @xcite , but preliminary results showed that xls-r was superior. the hyperparameters of our fine-tuning experiments follow those reported in @xcite for comparable data sizes, except for the learning rate, which we tune on the basis of the development data by evaluating the following range: [5e-4, 1e-4, 5e-5, 1e-5]. in addition, we reduce the batch size and use gradient accumulation to make sure our experiments run on limited compute hardware (i.e. a single nvidia 40 gb a100 gpu). we evaluate the fine-tuned models in terms of word error rate (wer), which is a commonly used evaluation metric based on the number of substitutions, deletions, and additions between two transcripts, and report performance on the test set using the fine-tuned model checkpoint that has the lowest wer on the validation set. additionally, we investigate whether it is beneficial to further pre-train the xls-r model using limited data and computational hardware before fine-tuning the model for asr. as pre-training is computationally expensive, we only evaluate the performance on gronings, for which we perform the broadest range of experiments. specifically, we pre-train on the four hours of gronings training data with the test set samples removed for 100,000 steps and use a learning rate of 1e-5, which was selected after briefly experimenting with a range of learning rates that we evaluated on the validation set. similar to the fine-tuning experiments, we use gradient accumulation and a small batch size. the total computational budget for this study is about 390 hours on a 40 gb a100 gpu (160 fine-tuning runs of roughly 2 hours each, and pretraining runs of roughly 70 hours). we perform all experiments using the huggingface transformers library, version 4.24.0 @xcite . ## experimental setup for each of the languages, we use varying amounts of training data for fine-tuning the multilingual xls-r model. additionally, for gronings, we also fine-tune the xls-r model that is further pretrained on gronings. for all experiments, we start from the full training dataset of 192 minutes (80% of four hours), and divide this set repeatedly into smaller subsets until reaching roughly 20 minutes (50% of each split). consequently, we have training sets of 192, 96, 48 and 24 minutes, respectively. in the self-training approach, we fine-tune the pre-trained xls-r models on one of the subsets of data (i.e. 24, 48, or 96 minutes) as the initial step. we regard this model as the teacher model, which is then used to transcribe the remaining portion of speech data from the full training data (i.e. without the labels). the resulting automatically transcribed data, in conjunction with the original labeled data, is subsequently used to fine-tune a second model, referred to as the student model, which ideally outperforms the teacher model. this approach is shown in figure 1 . for example, we fine-tune a xls-r teacher model on 24 minutes of manually transcribed speech data and use this model to label the remaining 168 minutes of speech data contained in the full training set. the combined data (e.g., 24 minutes of natural speech with correct labels and 168 minutes of automatically transcribed speech obtained through self-training) are subsequently used to fine-tune a new student model. we apply this procedure to each of the three training splits to investigate in which cases self-training may be beneficial in a low-resource setting. our decoding procedure does not use an external language model (lm) due to the limited availability of text-based training materials for all languages, and also to ensure a fair comparison between languages. this is supported by previous work that found no improvement in speech recognition performance when limited amounts of textual data are available for lm training @xcite . note that in addition to the self-training approach, preliminary experiments were conducted with other data augmentation techniques (following @xcite . specifically, we experimented with adding noise to the speech signal, raising or lowering the pitch of the speaker, and simulating far-field speech. these techniques, however, did not improve the speech recognition performance, and we discarded them from our experimental setup to limit the amount of comparisons. ## results we show the word error rates (wers) for gronings, west-frisian, besemah, and nasal in figure 3 . the wers for the development set are presented in appendix a. for each of the languages, we observe a clear performance increase (i.e. lower wers) when the amount of manually transcribed training data becomes larger. the wers decrease between 30.1% and 53.3% when we use the complete set of training data (i.e. 192 minutes of manually transcribed speech data) instead of the 24-minute subset. importantly, figure 3 also shows that selftraining is beneficial for each of the languages. student models improve over their teacher models in almost all cases. the improvement is particularly strong when the teacher model was based on a very small amount of data (i.e. 24 minutes) and ranges between 6.3% and 13.9%. ## discussion and conclusion we investigated whether data augmentation techniques are beneficial to improve the performance of asr systems for four typologically different languages with a limited amount of real-world training data available. we evaluated the performance of xls-r models fine-tuned using varying amounts of training data, showing that the model performance generally improves (i.e. resulting in lower wers) when (more, in the case of self training) augmented training data is used. the greatest performance gains across the four languages were observed when the amount of manually transcribed data used for fine-tuning was increased. nevertheless, we also observed substantial increases in model performance by augmenting very limited moreover, using an existing tts system for generating additional synthetic training data was likewise shown to be beneficial. we observed that the benefit of augmenting the training data via the tts system yielded larger performance gains (even on par with a model fine-tuned on four times the minimum amount of manually transcribed speech data we considered) than using the iterative self-training procedure. however, in contrast to self-training, no beneficial effect was present when increasing the amount of generated data. this pattern held true irrespective of using the general test set for evaluation or an out-of-domain test set instead. while not many minority languages have a suitable tts system available, generating speech data using such a system is very easy as it only requires written text. of course, our results also show that when the material is available to train a tts system (i.e. using audio recordings and associated transcriptions) it is likely better to use these resources directly for training the asr system. while we showed the benefit of iterative selftraining when a very small amount of training data is available, the benefit of supplying more and more self-trained training data was diminishing. our result extends the findings for english by @xcite to a new set of minority languages or language variants. it is possible that the transcriptions generated by a specific teacher model in the selftraining approach contain useful information, but that this is negated to a large extent by the generated errors of the model. as teacher models fine-tuned on larger amounts of manually transcribed training data are expected to yield higher quality transcriptions (as shown in e.g., @xcite , the effect of generating more data might be more beneficial in these cases. however, this should be investigated in future work. when using the tts system for augmenting our training data, we did not see a benefit of increasing the amount of generated synthetic speech. as the additional training data represents data from a single speaker (as the tts system was trained on the basis of data from a single speaker), the model might have been been overfitting to that specific speaker. future work, therefore, needs to investigate alternatives (or additions) to using a tts system for generating additional training data. for example, by investigating whether model performance can be improved using speaker adaptation methods or cross-lingual voice conversion (e.g., @xcite . we found only minor performance gains when we fine-tuned the xls-r model that was further pretrained on gronings (using all training and development data). specifically, self-training appeared to have greater performance gains than continuing pre-training (cpt), and combining cpt and self-training only marginally improved results. given the large computational cost of cpt as opposed to the two data augmentation methods, it is clear that cpt is not cost-effective. it may be that cpt only yields appreciable performance gains once a sufficient amount of unlabeled audio can be obtained (e.g. 200 hours of ainu: @xcite . however, obtaining such a large amount of data for minority languages or language variants such as gronings, besemah, and nasal is unlikely. it is therefore important to further investigate how a limited amount of target language data can be used effectively for self-supervised pre-training. for example, @xcite reported that using an additional 70-hour out-of-domain corpus alongside a 12-hour target corpus was crucial in improving performance. given that similar language regularization approaches have been effective for neural machine translation (e.g. @xcite , it may be possible that this strategy could also be beneficial for further pre-training in speech (e.g., using a 70-hour indonesian speech corpus alongside the target four hour besemah corpus). in conclusion, our results show that dataaugmentation techniques may serve as a costeffective way to improve asr performance for low-resource languages and variants. while the performance of the four systems is not comparable to systems developed for high-resource languages, these systems may serve as a starting point for these language varieties. we hope our experiments help further more inclusive speech technology for low-resource languages.