Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens (2024)

Abstract: While massively multilingual speech models like wav2vec 2.0 XLSR-128 can be directly fine-tuned for automatic speech recognition (ASR), downstream performance can still be relatively poor on languages that are under-represented in the pre-training data. Continued pre-training on 70–200 hours of untranscribed speech in these languages can help — but what about languages without that much recorded data? For such cases, we show that supplementing the target language with data from a similar, higher-resource ‘donor’ language can help. For example, continued pretraining on only 10 hours of low-resource Punjabi supplemented with 60 hours of donor Hindi is almost as good as continued pretraining on 70 hours of Punjabi. By contrast, sourcing supplemental data from less similar donors like Bengali does not improve ASR performance. To inform donor language selection, we propose a novel similarity metric based on the sequence distribution of induced acoustic units: the Acoustic Token Distribution Similarity (ATDS). Across a set of typologically different target languages (Punjabi, Galician, Iban, Setswana), we show that the ATDS between the target language and its candidate donors precisely predicts target language ASR performance.

## introduction for developing automatic speech recognition (asr), 'low resource' languages are typically classified as such based on the availability of transcribed speech. untranscribed speech, texts, or reliable metadata about the language are often assumed to be easily obtainable. this assumption may not hold true for under-described languages with little digital representation. for such languages, we are interested in two questions: 1) does leveraging untranscribed speech from a similar, higher-resource 'donor' language for pre-trained model adaptation help improve speech recognition in the target language, and 2) how do we select the best donor? these questions are of interest as asr system development with little transcribed speech has become viable with multilingual pre-trained transformer models for speech (e.g. wav2vec 2.0 xlsr-128: @xcite . yet, as most languages are under-represented in the pre-training data, directly fine-tuning these models for asr in the target language can yield lower performance compared to their well-represented counterparts @xcite . while recent studies have shown the effectiveness of continued pre-training to adapt these models to the target language @xcite , they involved using 70-200 hours of target language data. for some languages, it may be quite difficult to source this much speech data -even untranscribed. thus, in our first set of experiments, we investigated whether supplementing target language data with data from another language could be a viable approach for model adaptation via continued pre-training (cpt). we selected punjabi as our target language to establish top-line performance when sufficient data is available (70 hours, approximating the setup in @xcite , along with a limited data baseline (when only 10 hours of punjabi is available). we compared this baseline to supplementing the 10 hours of punjabi with 60 hours of data from 8 other indic languages (indo-aryan: hindi, urdu, gujarati, marathi, bengali, odia; dravidian: malayalam, tamil). we finetuned each cpt-adapted model using the same 1 hour of transcribed punjabi speech. results indicated that adding data from unrelated dravidian languages (malayalam, tamil) or dissimilar indo-aryan languages @xcite yielded no better than baseline performance, 25% word error rate (wer). by contrast, we observed improved wers from adding more similar languages (marathi, urdu, gujarati, hindi), with adding hindi coming close to the 70-hour punjabi top-line: 23.2% vs. 22.2%, respectively. in our second set of experiments, we investigated how well measures of similarity between the target and donor languages predicted target language asr performance. we found that commonly used measures based on external typological databases such as lang2vec @xcite were not sufficiently fine-grained for our use case and, crucially, also varied with the quality/completeness of the available metadata for a given target language. to sidestep these issues, we propose the acoustic token distribution similarity (atds), which measures the degree of similarity for two untranscribed speech corpora based on frequencies of occurrence of recurring acoustic-phonetic sequences. this measure extends token distribution similarity @xcite , shown to correlate with positive transfer in continued pre-training for text-based language models. to account for the text-/token-less nature of untranscribed speech corpora, we induce them in a bottom-up manner using wav2seq @xcite , a method for inducing pseudo-tokens using pre-trained speech embeddings. we compared the asr performance from the indic language experiments to various similarity measures and found that atds offered the most accurate ranking. furthermore, atds correctly predicted the best donor language between two options for three non-indic low-resource languages (galician, iban, and setswana). 1 in sum, the main contributions of this paper are: 1) a systematic study of pairwise transfer between languages in continued pre-training and its effects on target language asr performance, and 2) the development, analysis, and first validation of atds -a fine-grained, bottom-up measure of acoustic-phonetic similarity to predict this asr performance. to facilitate reproducibility and further research, we make available all our code, model checkpoints, and experimental artefacts. foot_1 2 background: wav2vec 2.0 in this section, we provide a high-level overview of the wav2vec 2.0 model and highlight specific details about its architecture and training objectives that will be relevant to our later discussions. developed by @xcite , wav2vec 2.0 is a type of self-supervised pre-trained transformer model. in machine learning, supervised learning in-volves the use of human-generated labels (e.g. transcriptions), which can be time-and cost-intensive to create. in self-supervised pre-training, the goal is to first train the model on a proxy task for which it can derive its own labels, before the resulting model is adapted or 'fine-tuned' to the target task (e.g. asr). leveraging self-supervised pre-training has shown remarkable success across a variety of tasks when combined with a transformer-based model @xcite , which excels at learning how various units in a sequence co-occur (e.g. words in a sentence). naturally, this has spurred on much experimentation for leveraging such models for low-resource asr (e.g. @xcite @xcite . as illustrated below in figure 1 , the wav2vec 2.0 architecture consists of three parts: 1) a convolutional feature extractor that extracts learnable features from strided frames in the audio input, 2) a quantiser which clusters the audio features into into a set of discrete code vectors (q 1 , ..., q 5 ), and 3) a transformer attention network that learns contextenriched representations (h 1 , ..., h 5 ). for selfsupervised pre-training, the model is optimised using a joint contrastive loss and a diversity loss. the contrastive loss requires the model to use the neighbouring context to distinguish each masked frame amongst a set of negative distractors. 3 the diversity loss requires the model to make equal use of all code vectors, preventing the model from relying on a small subset which can lead to trivial/sub-optimal solutions. transformer contrastive loss + diversity loss ℒ masked [ k ae t ] h1 h2 h3 h4 h5 we highlight here two important details relevant for our later discussions on acoustic tokens. the first is that the representations learned by the trans-former network are particularly useful for speech applications requiring fine-grained comparisons of acoustic-phonetic content, e.g. speech information retrieval @xcite , second language pronunciation scoring @xcite , spoken dialect classification @xcite -particularly those learned by the middle layers of the transformer network (e.g. layers 12-16 of a 24-layer network). the second detail is that these representations are encoded as vectors in a high-dimensional latent space and emitted at a rate of 49 hz. combined with the diversity loss that requires exploration of this space, there is a many-to-many relationship between phonetic categories and these vectorsboth in the latent space and in time. for example, as illustrated in figure 1 , a single speech sound such as [ae] may last for three time steps (yielding h 2 , h 3 , h 4 ). the first two vectors (h 2 , h 3 ) may be very close in the latent space, as they map to the start of [ae] sounds while the third h 4 may map to [ae] sounds preceding @xcite and is thus located in a different part of the latent space. thus to derive phone-like tokens from these representations, we must group them in the latent space (e.g. using kmeans clustering) and then again in time according to how the grouped units themselves routinely cooccur (e.g. using subword modelling). in this way, the goal of tokenisation for both text and speech is driven by the need to derive units of a practical granularity based on the nature of the input: from coarser-grained words to finer-grained sub-words in text and, inversely, finer-grained subphones to coarser-grained phones in speech. we return to these two details in our development of the acoustic token distribution similarity measure for comparing the acoustic-phonetic similarity of two untranscribed speech corpora. ## a systematic study of pairwise transfer 3.1 motivation since the release of the original english wav2vec 2.0 model pre-trained on the 960 hour librispeech corpus @xcite , additional massively multi-lingual variants have also been developed: xlsr-53, pre-trained on 56k hours from 53 languages @xcite ; xlsr-128, pretrained on 436k hours from 128 languages @xcite ; and mms, pre-trained on 491k hours from 1,406 languages @xcite . in each case, the vast majority of the pre-training data is drawn from european language sources. given the under-represented nature of most languages in these wav2vec 2.0 models, several studies have investigated continued pre-training (cpt) to adapt them for target languages (e.g. @xcite @xcite @xcite . @xcite adapted the xlsr-53 model using 200 hours of ainu. using the same 40 minutes of transcribed ainu for asr fine-tuning, they found that the adapted model resulted in a 8.8% absolute word error rate (wer) decrease over the off-the-shelf xlsr-53 model. for many low resource languages, however, obtaining 200 hours of speech data may not be feasible. in their study of unsupervised domain adaptation for greek, @xcite found adapting the xlsr-53 model via cpt using a small 12-hour dataset of greek read speech to be ineffective. however, they found that successful cpt-based adaptation could be achieved with the use of multi-domain data, e.g. 12 hours of read speech mixed with 70 hours of newscasts. given these findings, we were motivated to investigate whether comparable results could be achieved by supplementing target language data with data from another language. ## a bottom-up approach to similarity 4.1 motivation a common method for calculating similarities between languages is to use language vectors from the @xcite with and without adaptation via continued pre-training (cpt). cpt-adapted models were trained for 10k updates using 70 hours of punjabi for the topline (t), 10 hours of punjabi for the baseline (b), and 10 hours of punjabi combined with 60 hours of data from another language for the experiment conditions (e1, e2, e3). all models were fine-tuned with the same 1 hour of punjabi data. asr performance reported in word error rate (wer) and relative word error rates (werr), relative to the 10 hour cpt baseline (b). for each experiment condition, median and range were obtained from 3 cpt runs per language with different donor data in each run. ## limitations and future directions we limited the scope of this paper to exploring the transfer between languages and as such used the standard wav2vec 2.0 pre-training recipe for model adaptation. we acknowledge that this requires a large compute budget beyond what is affordable in many low resource scenarios. in future, we hope to investigate whether or to what extent transfer learning can be combined with more compute-efficient adaptation methods. to conduct a systematic study of pairwise transfer, we used domain-matched, high-quality asr datasets containing mostly read speech and only examined sourcing data from a single donor language. questions relating to multi-donor and multidomain transfer and how such interactions affect downstream performance in the target language will need to be addressed in future research. ## conclusion for developing automatic speech recognition (asr) systems for languages with very few resources, we demonstrated that massively multilingual pre-trained models for speech can be successfully adapted via continued pre-training using a mix of data from the target language and supplemental data from a similar, higher-resource 'donor' language. additionally, we motivate and propose the acoustic token distribution similarity (atds) -a novel measure of similarity to predict positive transfer between a donor and target language. across a set of typologically different target languages (punjabi, galician, iban, setswana), we show that the atds between the target language and its candidate donors precisely predicts target language asr performance. we attribute this predictive capability of atds to leveraging the knowledge of the pre-trained model to be adapted, its inductive biases and training objectives, and the distributions within the candidate datasets that will be used to adapt it. it is indeed expected that this measure then is able to predict downstream task improvements better than measures based on other models or external information -the latter of which is not always available or reliable for under-described languages. given the high cost associated with continued pre-training, however, we argue that using a well-calibrated, task-specific measure minimises the chance of costly, unexpected outcomes. we make a final observation here that various target-donor language pairs where we observed successful transfer exist in linguistic situations with significant, sustained contact. for example, galician is a minoritised language in spain and virtually all galician speakers are bilingual in spanish (de la fuente iglesias and pérez castillejo, 2020). similarly, @xcite report that finetuning a model pre-trained on french was particularly successful for gwadloupéyen and morisien, two french-based creole languages. these findings suggest that to develop truly inclusive speech technologies in a resource efficient manner, what will be required is a nuanced understanding of what factors linguistic and non-linguistic yield sufficiently high levels of cross-lingual similarity which in turn permit positive transfer.