Investigating Adapters for Parameter-efficient Low-resource Automatic Speech Recognition (2025)

Abstract: Recent years have witnessed the adoption of parameter-efficient adapters in pre-trained language models for natural language processing. Yet, their application in speech processing remains less studied. In this work, we explore the adapters for low-resource speech recognition, introducing a novel technique - ConvAdapt into pre-trained speech models. We investigate various aspects such as data requirements, transfer learning within adapters, and scaling of feed-forward layers in adapters. Our findings reveal that bottleneck adapters offer competitiveness with full fine-tuning with at least 10 hours of data, but they are not as effective in few-shot learning scenarios. Notably, ConvAdapt demonstrates improved performance in such cases. In addition, transfer learning in adapters shows promise, necessitating research in related languages. Furthermore, employing larger speech models for adapter-tuning surpasses fine-tuning with ample data, potentially due to reduced overfitting than fine-tuning.

## introduction automatic speech recognition (asr) advancements have favored high-resource languages due to abundant data and computing power. however, over 7000 languages are low-resource or zeroresourced, raising concerns of extinction @xcite . large pre-trained self-supervised speech models like wav2vec 2.0 show promise in enhancing asr for low-resource languages through fine-tuning with smaller datasets @xcite . fine-tuning such large and even multi-lingual models with a low-resource language data usually works well in practice but has its own limitations. it involves updating most of the model parameters which is inefficient, resource-intensive and storage-demanding. moreover, it poses challenges in dealing with multiple tasks/languages, causing catastrophic forgetting and complex decision-making for choosing the task sequence @xcite . bottleneck adapters, initially introduced in computer vision @xcite , consist of twolayer feed-forward networks inserted into large pretrained models @xcite . this technique selectively updates adapter parameters while keeping the rest of the model frozen, effectively reducing trainable parameters. it facilitates taskspecific adapter integration into pre-trained models, avoiding the need for full re-training and mitigating catastrophic forgetting @xcite . while adapters have been well studied in natural language processing (nlp) literature @xcite , investigating adapters in the speech signal processing domain is relatively new. bottleneck adapters are implemented for asr with the wav2vec 2.0 english base model @xcite , @xcite , and google universal speech model @xcite . these studies indicate that adapters perform on par with fine-tuning while being parameter efficient. few studies explore bottleneck adapters for specialized tasks like multi-domain asr modeling with transformers @xcite , personalized speech recognition in a multi-turn dialog setting with transducers (rnn-t) @xcite , atypical and accented speech recognition with rnn-t and transformer transducers @xcite . the latter one utilized residual connections within adapters. different adapterbased approaches are compared for several speech processing tasks with three of the state-of-the-art pre-trained models @xcite . the selection of different neural layers to insert the adapters is performed with a two-stage algorithm @xcite . while the prior works rely on bottleneck adapters, chapter technique based on convolutional neural network (cnn) adapters are employed in hubert feature extractor on emotion and speaker tasks @xcite . to the best of our knowledge, no work leveraged convolutional nets as adapters by incorporating them into the contextual transformer layers in the speech processing domain. also, given the limited work on asr modeling using adapters, there exist substantial research gaps that necessitate a comprehensive study of this method. it still remains an open question what the training data size must be for adapter-tuning to perform on par with complete fine-tuning for the low-resource asr task since the prior work investigates the adapter-tuning for speech processing with only high-resource languages such as english, omitting the suitability of the approach for lowresource languages. furthermore, the possibility of scaling the adapter modules or pre-training the adapters with a source language are not explored in the literature. this study aims to address the aforementioned research gaps by conducting a comprehensive investigation of adapters for asr, with a particular focus on the low-resource aspect. through this research, we aim to reduce computational complexity while simultaneously maintaining asr performance, ensuring the representation and preservation of low-resource languages in the field of speech technology. the contribution of this work is four-fold: to this end, we propose a simple yet effective technique convadapt for extreme lowresource parameter-efficient asr. notably, no prior research has explored the data requirements for adapter-based low-resource asr, to the best of our knowledge. let the transformer model representation be d m and the representation from the second fc layer is f m . both d m and the f m have the same dimension of m. the output of the add & norm layer is computed as, we propose the convadapt technique by replacing the bottleneck adapters with cnn-based adapters while keeping them at the same position in the transformer layers (see figure 1 ). the latent representation from the mhsa/ff of the transformer is fed as input to the convadapters after rearranging the tensors to avoid dimension mismatch. we employ two 1-dimensional convolutional layers, each followed by a rectified linear unit (relu). there are 1280 input channels and 1280 output channels, and both kernel size and stride are set to 1. the same padding is used. finally, we rearrange the resultant tensors again to obtain the original dimension and add this to the adapter input through a residual connection. experimental setup: we choose two large pretrained cross-lingual models, namely xls-r and mms @xcite . xls-r has three variants with 0.3 billion (b), 1b, and 2b trainable parameters. we utilize mms containing lang language datasets train set dev set test set group length clean other clean other bn indo-aryan subak.ko 200.3 20.5 -20.3 -(kibria et al., 2022) mt semitic common voice 52.5 2.3 -1.5 -(ardila et al., 2020) masri-headset (mena et al., 2020) mep tube merlin parlament ## conclusion and future scope this study presents a comprehensive analysis of parameter-efficient adapters for large pre-trained speech models. we find that bottleneck adapters are not suitable for few-shot learning, however, they perform competitive to full fine-tuning when at least 10 hours of data are available. our proposed convadapt technique in transformer layers is simple yet effective to deal with extremely low-resource cases. in mid-to-high resource scenarios, bottleneck adapter-tuning surpasses the widely used full fine-tuning technique, signifying its considerable impact in the field. leveraging highercapacity models like xls-r 2b significantly improves adapter-based tuning, countering the overfitting challenge posed by large pre-trained models during full fine-tuning. impressively, adapters we believe that our intriguing findings on adapter-tuning showing remarkable potential for both low-resource and mid/high-resource asr would encourage more research into this direction. future work includes exploring transfer learning in adapters with closely-related languages and performing multiple tasks using a single encoder.