Lightweight Adapter Tuning for Multilingual Speech Translation (2021)

Abstract: Adapter modules were recently introduced as an efficient alternative to fine-tuning in NLP. Adapter tuning consists in freezing pre-trained parameters of a model and injecting lightweight modules between layers, resulting in the addition of only a small number of task-specific trainable parameters. While adapter tuning was investigated for multilingual neural machine translation, this paper proposes a comprehensive analysis of adapters for multilingual speech translation (ST). Starting from different pre-trained models (a multilingual ST trained on parallel data or a multilingual BART (mBART) trained on non parallel multilingual data), we show that adapters can be used to: (a) efficiently specialize ST to specific language pairs with a low extra cost in terms of parameters, and (b) transfer from an automatic speech recognition (ASR) task and an mBART pre-trained model to a multilingual ST task. Experiments show that adapter tuning offer competitive results to full fine-tuning, while being much more parameter-efficient.

## introduction the question of versatility versus specialization is often raised in the design of any multilingual translation system: is it possible to have a single model that can translate from any source language to any target one, or does it have to be multiple models each of which is in charge of one language pair? the former is referred to as a multilingual model, while the latter are bilingual ones. these two paradigms have their own strengths and limitations. from a practical point of view, a multilingual model seems to be highly desirable due to its simplicity in training and deployment, in terms of both time and space complexities. however, in terms of accuracy, a multilingual model could be outperformed by its bilingual counterparts, especially on high-resource language pairs. n × m × speech embeddings attention ffn adapter outputs (shifted) attention attention ffn adapter (a) transformer with adapters. layernorm linear relu linear (b) an adapter cell. in practice, a certain trade-off between the aforementioned factors (and thus more generally between versatility and specialization) has often to be made, and depending on the application, one can be favored more than the other. one way to move along the spectrum between multilingual and bilingual models is to use adapter tuning which consists in freezing pre-trained parameters of a multilingual model and injecting lightweight modules between layers resulting in the addition of a small number of language-specific trainable parameters. while adapter tuning was investigated for multilingual neural machine translation (nmt) @xcite , to our knowledge, this paper proposes the first comprehensive analysis of adapters for multilingual speech translation. our contributions are the following: (1) we show that both versatility and specialization can be achieved by tuning language-specific adapter modules on top of a multilingual system. bilingual models with higher accuracy than the original multilingual model are obtained, yet keeping a low maintenance complexity; (2) starting from a different initialization point, we show that adapters can also be used as a glue to connect off-the-shelf systems (an automatic speech recognition (asr) model and a multilingual denoising auto-encoder mbart @xcite ) to perform the multilingual st task. extensive experiments on the must-c dataset @xcite show that adapter-based fine-tuning can achieve very competitive results to full fine-tuning-while being much more parameter-efficient-in both standard and low-resource settings. our code based on fairseq s2t @xcite ) is publicly available. foot_0 2 related work adapter layers (or adapters for short) were first proposed in computer vision @xcite , then explored for text classification tasks in nlp @xcite . adapters are generally inserted between the layers of a pre-trained network and finetuned on the adaptation corpus. @xcite studied adapters in the context of nmt and evaluated them on two tasks: domain adaptation and massively multilingual nmt. @xcite later introduced monolingual adapters for zero-shot nmt. other research groups made contributions on the use of adapters in nlp @xcite and a framework built on top of huggingface transformers library @xcite was also released to facilitate the downloading, sharing, and adapting state-of-the-art pretrained models with adapter modules @xcite . also very relevant to our paper is the work of @xcite where adapters are used to adapt pre-trained bart @xcite and mbart25 (multilingual bart pre-trained on 25 languages) @xcite to machine translation. as far as speech processing is concerned, adapters were mostly used in asr @xcite @xcite . recently, they have also been explored for st as well but in a limited scope. escolano et al. (2020) addressed a very specific setting (zero-shot st), while @xcite used only a single adapter after a transformer encoder. ## adapters for speech translation in this section, we describe the integration of adapters into a given backbone model for speech translation. as the transformer @xcite has become increasingly common in speech processing, foot_1 it will be used as our backbone. our method, however, can be easily applied to any other architectures, e.g., dual-decoder transformer @xcite . adapter modules can be introduced into a transformer in a serial or parallel fashion. consider a layer represented by a function f that produces an output y from an input x, i.e., @xmath0. this can be an entire encoder or decoder layer, or just one of their sub-layers (e.g., the self-attention or the final feed-forward network (ffn) component). suppose that our adapter layer is represented by a function g. the new "adapted" output is then given by: intuitively, a serial adapter modifies the output directly, while a parallel one performs the operations in parallel before merging its output to the layer. in figure 1a , we show an example of serial adapters being integrated to the transformer, or more precisely to its ffn sub-layers. a common adapter module @xcite is presented in figure 1b . here g is a small ffn with a residual connection. the first linear layer is typically a down projection to a bottleneck dimension, and the second one projects the output back to the initial dimension. bottleneck allows us to limit the number of parameters. other adapter architectures also exist, e.g., stickland and murray (2019) explored parallel adapters consisting of a multi-head attention (mha) layer in a multi-task setup. st dataset from english to eight target languages including dutch (nl), french (fr), german (de), italian (it), portuguese (pt), romanian (ro), russian (ru), and spanish (es). each direction includes a triplet of speech, transcription, and translation. sizes range from 385 hours (pt) to 504 hours (es). must-c-imbalanced we built a low-resource version of must-c, called must-c-imbalanced, in which we randomly keep only x% of the original training data, where @xmath2, fr; @xmath3, it; @xmath4, ro; and @xmath5, pt (same order of the languages in the original must-c if we sort them in decreasing amount of data). the amount of speech data ranges from 41 hours (de) to 504 hours (es) in this version, better reflecting real-world data imbalance scenarios. ## refinement in this section, a fully trained multilingual st backbone is further refined on each language pair to boost the performance and close potential gaps with bilingual models. we compare adapter tuning with other fine-tuning approaches as well as the bilingual and multilingual baselines (the latter being the starting point for all fine-tuning approaches) @xcite bilingual vs. multilingual for the small architecture (@xmath6, the bilingual models slightly outperform their multilingual counterpart (rows 1, 2). looking further into the performance of each language pair, the multilingual model is able to improve the results for 4 out of 8 pairs (de, nl, pt, ru), mainly those in the lower-resource direction, but the joint multilingual training slightly hurts the performance of higher-resource pairs such as es, fr, it, and ro. finally, we observe that the medium model (@xmath7, 10). adapter tuning vs. fine-tuning both recipes yield improvements over the multilingual baseline and recover the lost performance of higher-resource directions compared to the bilingual baseline for the small model (@xmath8. for the medium one (@xmath9, one adapter tuning (row 14) can slightly improve the scores in all directions and even approach the results of the best fine-tuning experiment (row 17) while maintaining much lower model sizes (95.5m vs. 8× 36.3m parameters). ## transfer learning in this section, we show that adapters can be used to combine available pre-trained models to perform a multilingual st task. in particular, we initialize the encoder using a pre-trained asr encoder (on must-c) 3 provided by @xcite and the decoder using mbart50, a multilingual denoising auto-encoder pre-trained on 50 languages @xcite . we tune language independent crossattention and language-specific adapters on top of these backbone models (using must-c as well). the results presented in ## conclusion we have presented a study of adapters for multilingual st and shown that language-specific adapters can enable a fully trained multilingual st model to be further specialized in each language pair. with these adapter modules, one can efficiently obtain a single multilingual st system that outperforms the original multilingual model as well as multiple bilingual systems while maintaining a low storage cost and simplicity in deployment. in addition, adapter modules can also be used to connect available pre-trained models such as an asr model and a multilingual denoising auto-encoder to perform the multilingual speech-to-text translation task.