The Xiaomi AI Lab’s Speech Translation Systems for IWSLT 2023 Offline Task, Simultaneous Task and Speech-to-Speech Task (2023)

Abstract: This system description paper introduces the systems submitted by Xiaomi AI Lab to the three tracks of the IWSLT 2023 Evaluation Campaign, namely the offline speech translation (Offline-ST) track, the offline speech-to-speech translation (Offline-S2ST) track, and the simultaneous speech translation (Simul-ST) track. All our submissions for these three tracks only involve the English-Chinese language direction. Our English-Chinese speech translation systems are constructed using large-scale pre-trained models as the foundation. Specifically, we fine-tune these models’ corresponding components for various downstream speech translation tasks. Moreover, we implement several popular techniques, such as data filtering, data augmentation, speech segmentation, and model ensemble, to improve the system’s overall performance. Extensive experiments show that our systems achieve a significant improvement over the strong baseline systems in terms of the automatic evaluation metric.

## introduction we submit an end-to-end offline speech translation system, a cascaded offline speech-to-speech translation system, and an end-to-end simultaneous interpretation system to the offline-st track, offline-s2st track, and simul-st track, respectively. this paper provides a detailed description of the three systems we submit. there are two commonly used solutions for speech translation models: the end-to-end approach and the cascaded approach. the cascaded system uses a pipeline where an automatic speech recognition (asr) system is followed by a machine translation (mt) system. the asr system first transcribes the speech utterances in the source language into text in the same language, and then the mt model translates the asr output into text in the target language. in contrast, the end-to-end st system directly translates speech utterances in the source language into text in the target language. the scarcity of training data makes end-to-end systems still slightly inferior in translation quality to cascaded systems, which suffer from error propagation and information loss @xcite . cascaded systems continue to dominate the systems submitted at iwslt in previous years @xcite @xcite . however, with the rapid development of pre-training technology, a large number of large-scale pre-training models suitable for various modalities, such as speech @xcite @xcite ) and text @xcite , have emerged. therefore, end-to-end st systems have gradually attracted attention from both the academic and industrial communities in recent years. in our submission, we have opted for an end-to-end approach to establish the st system. we briefly introduce the submitted systems: offline speech translation system. our submitted end-to-end offline speech-to-text translation system is based on two pre-trained models: hu-bert @xcite and mbart @xcite . it has been proven that these two models have strong capabilities on st and mt tasks, respectively. our offline st model consists of a speech encoder, a text encoder, and a text decoder, with all parameters initialized using the pre-trained hubert and mbart models. ## methods we build our offline-st system in an end-to-end manner (end-to-end offline-st) based on the hu-bert and mbart pre-trained models. our simultaneous speech translation system (end-to-end simul-st) utilizes the same model architecture as the offline-st system and adopts wait-k and itst strategies. the cascaded s2st system involves an end-to-end speech-to-text translation model followed by a tts model. ## conclusion in this paper, we present our submissions for the iwslt 2023 shared tasks. we participate in three tracks, namely the offline speech translation track, the offline speech-to-speech translation track, and the simultaneous speech translation track. all of our submissions use large-scale pre-trained models, and we further improve these models using various effective techniques, such as data augmentation, contrastive learning, and model ensembles. extensive experiments validate the effectiveness of our proposed method and demonstrate that our submitted systems are comparable to state-of-the-art baseline systems in terms of performance.