The M ine T rans Systems for IWSLT 2023 Offline Speech Translation and Speech-to-Speech Translation Tasks (2023)

Abstract: This paper presents the extscMineTrans English-to-Chinese speech translation systems developed for two challenge tracks of IWSLT 2023, i.e., Offline Speech Translation (S2T) and Speech-to-Speech Translation (S2ST). For the S2T track, extscMineTrans employs a practical cascaded system to explore the limits of translation performance in both constrained and unconstrained settings, where the whole system consists of automatic speech recognition (ASR), punctuation recognition (PC), and machine translation (MT) modules. We also investigate the effectiveness of multiple ASR architectures and explore two MT strategies: supervised in-domain fine-tuning and prompt-guided translation using a large language model. For the S2ST track, we explore a speech-to-unit (S2U) framework to build an end-to-end S2ST system. This system encodes the target speech as discrete units via our trained HuBERT. Then it leverages the standard sequence-to-sequence model to directly learn the mapping between source speech and discrete units without any auxiliary recognition tasks (i.e., ASR and MT tasks). Various efforts are made to improve the extscMineTrans’s performance, such as acoustic model pre-training on large-scale data, data filtering, data augmentation, speech segmentation, knowledge distillation, consistency training, model ensembles, etc.

## introduction in this paper, we describe the minetrans english-to-chinese speech translation systems which participate in two challenge tracks of the iwslt 2023 @xcite evaluation campaign: offline speech translation (s2t) and speech-to-speech translation (s2st). the annual iwslt evaluation campaign compares the models produced by different institutions on the task of automatically translating speech from one language to another. traditional s2t/s2st systems typically use a cascade approach @xcite @xcite @xcite , which combines automatic speech recognition (asr), machine translation (mt), and text-to-speech (tts, for s2st) components. recent advances in end-to-end models @xcite @xcite @xcite that directly translate one language speech to another without intermediate symbolic representations, have shown great potential in overcoming the problems inherent in cascaded systems, such as error propagation and slow inference. despite this, there is still a gap between the two approaches, as end-to-end models have much less supervised training data than sub-tasks, i.e., asr, mt, and tts. last year's iwslt offline s2t track @xcite confirmed this, with the best end-to-end model submission scoring 1.7 bleu points lower than the top-ranked cascade system. this year's competition aims to answer the question of whether cascade solutions remain dominant, particularly in the s2st track, where there has large-scale data for training. in the offline s2t track, minetrans employs a practical cascaded system to explore the limits of translation performance in both constrained and unconstrained settings, in which the entire system consists of automatic speech recognition (asr), and machine translation (mt) modules. we also investigate the effectiveness of multiple asr architectures and explore two mt strategies: supervised in-domain fine-tuning @xcite and prompt-driven translation using chatgpt foot_0 @xcite . in the s2st track, minetrans utilizes a speech-to-unit translation (s2ut) framework to construct an end-to-end system, which is similar to @xcite but removes all auxiliary recognition tasks (i.e., asr and mt tasks). this framework converts target speech into discrete units via our pre-trained hubert and then leverages the standard sequence-to-sequence model to learn the mapping between source speech and discrete units directly. we found that with a largescale dataset, such as 10,000 hours of training data, the previous multi-task learning technique @xcite @xcite is not necessary for model convergence, and this approach can successfully handle the mapping between source speech and discrete units. we also explore various initialization strategies and several techniques to improve model performance, including (1) different selfsupervised pre-trained speech encoders and pretrained text-to-unit models, (2) data filtering and augmentation, consistency training, and model ensembles. to the best of our knowledge, we are the first and only one to successfully train and submit the end-to-end s2st model on this challenging track. our code is open-sourced at: https: //github.com/duyichao/minetrans-iwslt23 . the remainder of this paper is organized as follows: section 2 describes data preparation, including data statistics, data preprocessing, and data filtering. section 3 describes our solution for the offline speech translation track. section 4 describes our solution to the speech-to-speech track. in section 5, we conclude this paper. ## conclusion this paper presents the minetrans system for two challenge tracks of the iwslt 2023: offline speech translation (s2t) and speech-to-speech translation (s2st). for the s2t track, mine-trans employs a cascaded system to investigate the limits of translation performance in both constrained and unconstrained settings. we explore two machine translation strategies: supervised indomain fine-tuning and prompt-guided translation using a large language model. for the s2st track, minetrans builds an end-to-end model based on the speech-to-unit (s2u) framework. to the best of our knowledge, we are the first and only team to successfully train and submit the end-to-end s2st 9 https://github.com/wenet-e2e/wenet/blob/main/ docs/pretrained_models.en.md on this track. this model uses our trained hu-bert to encode the target speech as discrete units and leverages the standard sequence-to-sequence model to directly learn the mapping between source speech and discrete units without the need for auxiliary recognition tasks such as asr and mt. we use several techniques to improve minetrans's performance, including speech encoder pre-training on large-scale data, data filtering, data augmentation, speech segmentation, consistency training, and model ensemble.