The N iu T rans End-to-End Speech Translation System for IWSLT 23 E nglish-to- C hinese Offline Task (2023)

Abstract: This paper describes the NiuTrans end-to-end speech translation system submitted for the IWSLT 2023 English-to-Chinese offline task. Our speech translation models are composed of pre-trained ASR and MT models under the SATE framework. Several pre-trained models with diverse architectures and input representations (e.g., log Mel-filterbank and waveform) were utilized. We proposed an IDA method to iteratively improve the performance of the MT models and generate the pseudo ST data through MT systems. We then trained ST models with different structures and data settings to enhance ensemble performance. Experimental results demonstrate that our NiuTrans system achieved a BLEU score of 29.22 on the MuST-C En-Zh tst-COMMON set, outperforming the previous year’s submission by 0.12 BLEU despite using less MT training data.

## introduction end-to-end speech translation (e2e st) directly translate speech in the source language into text in the target language without generating an intermediate representation, which has gained significant attention in recent years due to several advantages over cascade methods, including low latency and the ability to avoid error propagation @xcite . in this paper, we describe our niutrans e2e st system that participated in the iwslt23 english-to-chinese offline track, the overview of our system is shown in fig 1 . to improve the performance of our system, we aim to maximize the diversity of our ensemble of e2e st models. our e2e st models are based on the stacked acoustic and textual encoding (sate) method @xcite , which is a framework to make the best of pre-trained automatic speech recognition (asr) and machine translation (mt) * authors contributed equally. components. using this framework, we explore multiple architectures of pre-trained asr and mt models with varying numbers of parameters and input representations such as fbank features or waveform data. pseudo data is a crucial component of e2e st, often generated by ensemble mt systems @xcite . this year, we focused more on the performance of mt models and developed an iterative data augmentation method to leverage text data from all corpora, improving the mt models and enabling the generation of multiple pseudo data. we then used these multiple pseudo data to train diverse e2e st models for optimal performance. our best st ensemble system includes models with different input representations, architectures, and training corpora, achieving a bleu score of 29.22 on the must-c en-zh tst-common set. the remainder of the paper is organized as follows: section 2 describes the data processing, data augmentation and speech segmentation. section 3 outlines the construction of the vocabulary and structures of our asr, mt and st models. the experimental settings and final results are presented in section 4. finally, section 5 concludes the submission. ## model architecture we explored the performances of different asr, mt, and st architectures and found that using larger models is more conducive to performance improvement in all three tasks. ## conclusion this paper presented our submission to the iwslt23 english-to-chinese offline speech translation task. our system aimed to find the optimal ensemble system under the "constrained" training condition. to achieve this goal, we explored different input representations, model architectures, and proposed an ida method to utilize all available texts to improve the mt systems and generate multiple pseudo st data. our final system achieved a bleu score of 29.22 on the must-c en-zh tst-common set, and the results on the iwslt 23 test sets are shown in