The Y i T rans Speech Translation System for IWSLT 2022 Offline Shared Task (2022)

Abstract: This paper describes the submission of our end-to-end YiTrans speech translation system for the IWSLT 2022 offline task, which translates from English audio to German, Chinese, and Japanese. The YiTrans system is built on large-scale pre-trained encoder-decoder models. More specifically, we first design a multi-stage pre-training strategy to build a multi-modality model with a large amount of labeled and unlabeled data. We then fine-tune the corresponding components of the model for the downstream speech translation tasks. Moreover, we make various efforts to improve performance, such as data filtering, data augmentation, speech segmentation, model ensemble, and so on. Experimental results show that our YiTrans system obtains a significant improvement than the strong baseline on three translation directions, and it achieves +5.2 BLEU improvements over last year’s optimal end-to-end system on tst2021 English-German.

## introduction in this paper, we describe our end-to-end speech translation system yitrans which participates in the offline tracks of the iwslt 2022 evaluation campaign. we evaluate our systems from english to german, chinese and japanese. we aim at exploring the pre-training methods for end-to-end systems, and bridging the quality gap with the cascaded approaches. as self-supervised learning has been shown effective in speech-to-text tasks @xcite @xcite , our teams are interested in building a multimodality pre-trained model with self-supervised approaches by leveraging large amounts of speech and text data. inspired by speecht5 @xcite , we design a multi-stage unified-modal training strategy for pre-training both the encoder and this paper also tries to improve the system performance by exploring various techniques for the related tasks. (1) to boost the performance with advanced speech segmentation @xcite , we apply the pyannote toolkit @xcite and the merge algorithm from @xcite to segment the audio. particularly, to overcome the long sentence problem in the dataset, we design a new segment algorithm. (2) dataset is the key point for a st system to perform well. hence, we conduct refined data filtering and large-scale data augmentation @xcite . (3) we also employ progressive learning, back translation and multi-stage fine-tuning @xcite @xcite when fine-tuning our models. (4) motivated by @xcite , we utilize joint st and mt fine-tuning for our endto-end st models. (5) as comparison, we also build the cascaded systems for all three language pairs by fine-tuning asr and mt models from pre-trained models. the rest of this paper is organized as follows. in section 2, we describe the data preparation, including the data pre-processing, data augmentation, and speech segmentation. section 3 illustrates the unified-modal pre-training methods, and our systems for all three tasks. we share the experimental setting, results, and analyses in section 4. section 5 concludes the submission. we also present the official test results @xcite ## end-to-end yitrans st system recent studies, such as speecht5 @xcite and slam @xcite , have shown that joint pre-training of speech and text can boost the performance of spoken language processing tasks, such as speech translation. this section will mainly introduce the model architecture of our end-to-end yitrans system, and the proposed methods to pretrain and fine-tune the models. ## conclusion in this paper we describe our end-to-end yitrans speech translation system for iwslt 2022 offline task. we explore building st systems from largescale pre-trained models. our proposed multistage pre-training strategy allows the model to learn multi-modality information from both labeled and unlabeled data, which further improves the performance of downstream end-to-end st tasks. our systems are also built on several popular methods such as data augmentation, joint fine-tuning, model ensemble, and so on. massive experiments demonstrate the effectiveness of our system, and show that the end-to-end yitrans achieves comparable performance with the strong cascade systems and outperforms the last year's best end-to-end system by 5.2 bleu in term of english-german tst2021 set.