End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021 (2021)

Abstract: This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation.

## introduction typically, a speech translation (st) system is composed of an automatic speech recognition (asr) and a machine translation (mt) model, which is known as cascade system. however, in recent years, end-to-end models have gained popularity within the research community. these systems are encoder-decoder architectures capable of directly translating speech without intermediate symbolic representations. this approach solves classical shortcomings of cascade st systems, e.g. the error propagation or the slow inference time @xcite . nevertheless, while there are plenty of data available to train asr and mt systems, there are not as many datasets for st, despite some recent efforts @xcite . moreover, this approach is inherently more difficult because the encoder has to perform both acoustic modeling and semantic encoding. for these reasons, end-to-end st systems still struggle to achieve the performance of cascade st models. still, last year's iwslt was the first time an end-to-end system had the best performance in the evaluation campaign @xcite . hence, given the increasing interest in end-to-end st systems, and the potential gains from advancing research on them, we decided to focus on developing such a system for this year's offline task. when there are not enough data for a task, a common practice is to use pre-trained components, like bert @xcite for various nlp tasks. in the st field, the idea of pre-training the encoder for asr was introduced by @xcite and has become a standard technique for developing modern end-to-end systems @xcite . by contrast, pre-training the decoder for mt does not lead to better performance @xcite . recently, @xcite proposed a multilingual st system that combines a pre-trained wav2vec 2.0 @xcite as the encoder and a pre-trained mbart decoder @xcite . furthermore, they proposed a minimalist fine-tuning strategy that trains only the 20% of the model parameters, while achieving similar performance to fine-tuning the whole model. from our perspective, this approach might become a turning point in the field, including bilingual scenarios like the iwslt offline task. hence, we decided to adopt this architecture 1 and fine-tuning strategy in our system ( §2.1). in addi-tion, we introduce an adapter module to extract better representations from the encoder ( §2.2), and we propose a two-step training strategy ( §4.1) that brings improvements to the translation quality. during training, we used data augmentation techniques to boost our system's performance. specifically, we applied randomized on-the-fly augmentations by adding an echo effect and modifying tempo and pitch ( §3.3). since our system works directly on the audio waveform, we could not use specaugment @xcite . instead, we applied masking to the output of the wav2vec 2.0 feature extraction module, thereby obtaining a similar effect. the test data are provided with an automatic segmentation that does not ensure sentence-like segments. considering the trend observed in 2019 and 2020 iwslt offline task, where submission with own segmentation algorithms are strictly better than those with the given segmentation, we also decided to work with a custom segmentation algorithm. we base it on the approach of @xcite , but we replace the silence detection tool with an asr system ( §3.4). our experiments on the iwslt 2019 test set, show that our system works better when the data are segmented with our own segmentation algorithm ( §4.3). ## system description we built an end-to-end st system, mainly composed of pre-trained modules. we couple a wav2vec 2.0 encoder @xcite and an mbart decoder @xcite , following the strategy proposed by @xcite . when combining these two models, there is a length discrepancy between the target sentence length and the encoder output. for this reason, it is necessary to use a module to shorten the encoder output, which we refer to as the length adaptor. additionally, we introduce an adapter module to reduce the gap between the different modalities of the pre-trained models @xcite . a method that @xcite proved to be beneficial for st models. ## data here we introduce the datasets used for our experiments and describe the filtering and data augmentation methods that were employed during training. ## experiments here we describe our experiments, along with their implementation details and the results on must-c and the iwslt 2019 test set. ## conclusions we described the upc machine translation group participation in the iwslt 2021 offline st task. we built our system by combining pre-trained components, using wav2vec 2.0 as an encoder and an mbart decoder. in order to fine-tune such a large model with approximately 770 million parameters, we followed the strategy proposed by @xcite , in which just a 20% of the parameters are trained. originally, this method was proposed for multilingual st, and it had not been applied to initialize a bilingual system yet. with this approach, we got a score of 26.23 bleu in the must-c test set. then, we introduced an adapter module to reduce the gap between the different modalities of the pre-trained components, which brought an improvement of 0.76 bleu. we also explored a two-step training where we initialized the coupling modules before fine-tuning the rest of the model, which resulted in an increase of 1.02 bleu with respect to the original model. furthermore, we applied other techniques like fine-tuning with in-domain data, checkpoint averaging and ensembling our two best models. our final score in the must-c test set was 28.22 bleu. apart from using wav2vec 2.0 as the encoder of our st system, we additionally leveraged it in our asr-based data filtering and as part of our segmentation algorithm. applying this custom segmentation we gained an increase of 2.5 to 3 bleu score in the iwslt 2019 test set, as compared to the result of with given segmentation. as was shown in @xcite , and confirmed in this work for a bilingual scenario, large pretrained models can be very effective in st. we believe that future work should focus on exploring better methods to adapt these pre-trained models to new languages and tasks, with adapter modules being promising candidates.