FST : the FAIR Speech Translation System for the IWSLT 21 Multilingual Shared Task (2021)

Abstract: In this paper, we describe our end-to-end multilingual speech translation system submitted to the IWSLT 2021 evaluation campaign on the Multilingual Speech Translation shared task. Our system is built by leveraging transfer learning across modalities, tasks and languages. First, we leverage general-purpose multilingual modules pretrained with large amounts of unlabelled and labelled data. We further enable knowledge transfer from the text task to the speech task by training two tasks jointly. Finally, our multilingual model is finetuned on speech translation task-specific data to achieve the best translation results. Experimental results show our system outperforms the reported systems, including both end-to-end and cascaded based approaches, by a large margin. In some translation directions, our speech translation results evaluated on the public Multilingual TEDx test set are even comparable with the ones from a strong text-to-text translation system, which uses the oracle speech transcripts as input.

## introduction multilingual speech translation @xcite enables translation from audio to text in multiple language directions with a single model. similar to multilingual text translation, it is sample efficient as the model supports more languages. furthermore, multilingual speech models can facilitate positive transfer across languages by learning a common representation space from speech inputs, typically either raw audio or filterbank features. in this paper, we provide a description of our submission to the first multilingual speech translation task at iwslt 2021. the task evaluates *yun tang and hongyu gong have equal contribution to this work. speech translations from spanish (es), french (fr), portuguese (pt) and italian (it) into english (en) and spanish (es). among them, three translation directions (it-en, it-es and pt-es) are considered zero-shot with respect to the constrained track. in addition, participants are encouraged to submit transcriptions for the relevant languages. our team, fair speech translation (fst), participated in the unconstrained track, where we submitted one primary system and four contrastive systems. we are interested in exploring the effectiveness of building a general-purpose multilingual multi-modality model. we leverage large amounts of data, including unlabelled and labelled data from different modalities, to alleviate the data scarcity issue. we build the multilingual model to perform speech translation and speech recognition tasks for all evaluation directions. our model leverages selfsupervised pretraining on both the encoder and the decoder. the model is further improved by knowledge transferring from the text-to-text translation task to the speech-to-text translation task under the multitask learning framework. finally, we finetune the model on parallel speech translation corpora as well as weakly aligned speech translation data through data mining to achieve the best result. in section 2, we described data sources and our method for speech translation data mining. models and training methods are then described in section 3. finally, we present the results for the primary and contrastive systems in section 4. ## data provided by the iwslt 2021 shared task, the multilingual tedx dataset collected from ted talks provides speech translations in 13 directions @xcite . we focus on the seven competition directions in the shared task: es-en, fr-en, pt-en, it-en, fr-es, pt-es and it-es. ## conclusion in this work, we described our multilingual end-toend speech translation system submitted to iwslt 2021. we leverage the large amount of training data from different domains and modalities to improve the speech translation performance. we adopt a progressive approach to build the model with three stages. compared with our strong baseline, the proposed system achieves 8.6 bleu score improvement, which also outperforms other reported systems, including both end-to-end and cascaded based, by a large margin.