Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR (2024)

Abstract: Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have been used to address such low-resource settings. In this work, we investigate how these techniques can be effectively combined using a multilingual multimodal model like SeamlessM4T. Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning, thus boosting ASR performance. We also show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over baseline in an extremely low-resource setting without any labeled speech.

## introduction across the languages of the world, the automation of various speech and text tasks has led to the creation of massive multilingual datasets such as multilingual librispeech @xcite , that contain speech, text, and other metadata for a number of different languages. this large-scale collection has catalyzed the emergence of large multilingual automatic speech recognition (asr) models @xcite , which utilize the structural similarities between different languages to learn language-invariant features and boost accuracy. subsequently, multimodal multilingual models, such as m3p @xcite , that bridge the gap between speech and text using joint representation spaces, have also emerged. these models are trained using large amounts of multilingual speech and text data. however, less-spoken languages, especially those from developing countries, do not have such large data corpora available @xcite , thus hurting model performance for extremely low-resource languages (chang et al., * these authors contributed equally to this work. 2023). thus, creating targeted models for severely low-resource languages has become crucial. one efficient way to do this is by adapting existing models to the target language using limited amounts of labeled data. such adaptation has to be done carefully so as to not overfit to the target language characteristics. parameter-efficient fine-tuning (peft) @xcite techniques have gained wide acceptance where only relevant parts of a model are identified and fine-tuned for a specific downstream task. text-only adaptation is another sub-area that is gaining popularity for low-resource asr @xcite . multimodal models have training pathways for both speech and text data, offering a good framework to combine both approaches. multilingual models, on the other hand, allow for cross-lingual transfer @xcite , i.e., using a higher resource language to improve performance on a lower resource language. in this work, we have leveraged the multimodal nature of meta's seamlessm4t @xcite to explore the benefits of speech-based adapter fine-tuning and text-only adaptation. these techniques have been used both in isolation and in combination to identify the best strategy to improve low-resource asr for a number of indic languages. we have also exploited the multilingual nature of the model to use higher-resource languages to improve low-resource asr. thus, our main contributions include: (a) identifying how to combine speech-based parameter-efficient fine-tuning and text-only adaptation to boost low-resource asr, (b) identifying a cross-lingual transfer technique that can give more than 17% relative reduction in wer for a low-resource language without using any speech of that language, (c) the use of small amounts of available data to boost the performance of seamlessm4t @xcite on six indic languages, bengali, gujarati, kannada, maithili, malayalam and odia. text decoder length adapter translated transcript speech speech-based training text encoder multimodal representation space seamlessm4t-nllb w2v-bert 2.0 trainable adapters transcript speech encoder text-based training ## related work one of the key challenges in current asr research is enabling systems to handle multilingual inputs @xcite while minimizing resource requirements in terms of training, inference, and storage costs. currently, the most popular paradigm using multilingual models are to initially pre-train the models in a selfsupervised manner on a large multilingual dataset @xcite before being fine-tuned on a set of target languages @xcite . a general way of performing such model fine-tuning is by updating all the weights or some specific model components while training. these kinds of methods are parameter inefficient and often cause catastrophic forgetting @xcite , for all non-target languages. also, training and storage costs for such methods increase linearly with both the model size and the number of languages. to mitigate these limitations, recent literature on nlp has introduced several parameter-efficient fine-tuning methods @xcite @xcite , often involving trainable modules called adapters @xcite , whose weights are updated while freezing the original backbone. significant efforts are being made to develop better adapter architectures and efficient training methods @xcite to utilize contrastive learning @xcite and metalearning @xcite . these modules can also be used to adapt multilingual asr models for a low-resource setting, with simadapter @xcite being one of the first models to utilize adapters to leverage cross-lingual features. in the context of speech recognition, a lowresource setting could refer to any scenario with insufficient training data. this includes challenges such as recognizing atypical speech @xcite or processing less commonly spoken languages. a recent work @xcite demonstrated the benefits of using adapters for very low-resource languages with less than five hours of training data. for the low-resource situation, task-or language-specific adapter modules showcase superior performance @xcite compared to fine-tuning the model components, but even such approaches are constrained by inherent limitations of the base model. over the past few years, considerable effort has gone into developing multilingual asr foundational models with more generalizable features. these models offer a stronger starting point for low-resource adaptations and enable the use of cross-lingual transfer learning. the exponential growth in computing power has led to the creation of increasingly large language models, which are now used for a wide range of tasks, including as backbones for multimodal asr models @xcite @xcite . for such models, the foundational backbone is expanded using audio tokens generated using techniques like wav2vec @xcite and hubert @xcite in order to learn a joint representation in a multimodal space; the token vocabulary is expanded to encompass both text and audio. note that models with joint multimodal representations are not only useful for asr but can also be integrated with a vocoder for tts or conversational chatbots @xcite . multimodal models can be trained with joint text-audio tasks through self-supervision with masked language modeling and denoising objectives; further fine-tuning is often done with asr and speechto-text or speech-to-speech translation tasks. one of the most recent examples of such a multilingual multimodal model has been seamlessm4t @xcite by meta ai, which is built upon the nllb @xcite backbone and can process speech and text inputs from nearly 100 languages. an implicit advantage of using such multimodal models for low-resource asr is the ability to benefit from text-only learning for shared parameters. in most cases, there is significantly more text data available than speech data. thus, the capability to leverage text-only adaptation for asr models can be highly advantageous in these scenarios. while there is a lot of prior work in the domain of text-only adaptation for asr @xcite @xcite , and there has been some work on a comparative analysis of various fine-tuning strategies for low-resource asr @xcite , to the best of our knowledge, our work is the first to explore them for multilingual multimodal models. ## methodology in this work, we leverage a combination of parameter-efficient adaptation, unlabeled textual data, and minimal amounts of transcribed speech to improve asr performance in low-resource languages using multilingual multimodal models. figure 1 demonstrates the overall workflow of our proposed pipeline. ## discussion we observe that for decoder adapters, it is more beneficial to use text-only adaptation compared to asr-based training; the latter's benefit is mainly derived via the encoder layers. this emphasizes the role played by text data in improving the decoder's ability to enhance the internal language model of the asr system. we also observed that 5-hour text data adaptation, having on average 92% fewer tokens than the full text, performed comparably to full-text data adaptation. this indicates that even limited amounts of text data can significantly boost asr. for a given target language with labeled speech, we found that fine-tuning the encoder adapters was the most accurate and parameter-efficient strategy. however, for cross-lingual zero-shot settings with no labeled data in a target language, we found it beneficial to fine-tune the length adapter with data in a related language rather than fine-tuning encoder adapters; the latter led to overfitting to the related language rather than enabling transfer to the target language. text-based adaptation led to further improvements in the cross-lingual setting, indicating that even without speech data, asr for low-resource languages can be improved by finetuning the length adapter. lastly, a curious observation was that higher cross-lingual transfer was seen for genetically closer language pairs, with odia-bengali outperforming maithili-bengali in terms of relative wer reduction. ## conclusion in this work, we explored the combination of parameter-efficient asr fine-tuning and text-only adaptation techniques to enhance asr for lowresource indic languages using a multi-lingual multi-modal base model (seamlessm4t). we find that a limited amount of text data was sufficient for adaptation, text-based adaptation was superior to asr fine-tuning of decoder adapters, and encoder adapters were most effective in limited speech settings. in cross-lingual settings, however, the length adapter (and not the encoder adapter) was most successful, and text adaptation was additionally beneficial. future work will focus on developing a better understanding of the interplay between different adapters within multimodal models.