Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021)

Abstract: We present a simple yet effective approach to build multilingual speech-to-text (ST) translation through efficient transfer learning from a pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning 10 50% of the pretrained parameters. This effectively leverages large pretrained models at low training cost such as wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation. This sets a new state-of-the-art for 36 translation directions (and surpassing cascaded ST for 26 of them) on the large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on average for En-X directions and +6.7 BLEU for X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.6 BLEU on average across 28 non-English directions), making it an appealing approach for attaining high-quality speech translation with improved parameter and data efficiency.

## introduction recent advances in pretraining over unlabeled data and then finetuning on labeled data leads to significant performance improvement in text understanding and generation tasks @xcite @xcite . lately, such text pretraining and finetuning paradigms have been extended to other modalities: audio @xcite , images @xcite , and video @xcite . at the same time, pretraining and finetuning techniques have improved multitasking applications significantly, such as multilingual translation, cross-lingual representations, question-answering and so on @xcite @xcite . in this paper, we advance the one-model-for-all paradigm further by adapting audio and multilingual text pretraining and finetuning to improve multilingual speech-totext translation. our contributions are as follows: we describe our approach in section 2, namely pretrained models, length adaptor, lna finetuning and joint speech-text finetuning as is illustrated in figure 1 . experiments setup and results are elaborated in section 3 and section 4. section 5 provides ablation studies of the proposed finetuning strategy. ## ablation studies ablation on lna finetuning. in for adapting to a single language pair downstream st task (english-german), we find finetuning self attention (+sa) parameters in the decoder did not bring further improvement while significantly increasing the amount of parameters to train. ## related work speech translation. sequence-to-sequence based speech translation has shown very good potential over the traditional cascaded system @xcite @xcite with end-to-end approaches surpassing cascaded system for the first time at @xcite in a shared task setting. however, previous work also indicates that its success heavily relies on large amounts of labelled training data, which is difficult to acquire. in order to mitigate the data scarcity issue, recent research work focuses on multi-task learning @xcite @xcite @xcite , pretraining different components of the model @xcite , transfer learning @xcite and generating synthetic data @xcite . pretraining and finetuning. our work is motivated by the recent success of self-supervised learning for nlp and speech processing applications @xcite @xcite @xcite @xcite @xcite @xcite , which has achieved state-of-the-art results when finetuning on downstream tasks in nlp @xcite @xcite . our work attempts to leverage pretrained components from different modalities (text and speech) to perform the st task. how to efficiently adapt large pretrained models has gained growing interest. @xcite and @xcite represent the stream of work which adds additional "adaptor modules" to achieve fast adaptation to downstream tasks. another category of solutions focus selective finetuning (only subset of parameters) suitable for downstream tasks. our work belongs to the second category of efficient finetuning without adding extra parameters (e.g. adaptor modules). empirical studies shows that finetuning the final layers of bert account for most of the quality gains on downstream tasks @xcite . finetuning layernorm parameters was also found effective for adapting pretrained bart or mbart for machine translation @xcite . a general approach is to automatically learn which layers/parameters from a large-pretrained model to finetune and freeze @xcite , which we found is an exciting direction for future work. ## conclusion we proposed a simple and effective approach to leverage pretrained single-modality models (such as wav2vec 2.0, mbart) to perform speech-totext translation. on two large-scale multilingual speech translation benchmarks, our approach advances the state-of-the-art (+6.6 bleu on average for 36 translation directions in covost 2, and +5.6 bleu for 28 translation directions in europarl). we provide an efficient finetuning strategy which is not only data-and parameter-efficient, but also demonstrates crosslingual transfer ability by only finetuning 10 ∼ 50% of the parameters of large pretrained models.