The BIGAI Offline Speech Translation Systems for IWSLT 2023 Evaluation (2023)

Abstract: This paper describes the BIGAI’s submission to IWSLT 2023 Offline Speech Translation task on three language tracks from English to Chinese, German and Japanese. The end-to-end systems are built upon a Wav2Vec2 model for speech recognition and mBART50 models for machine translation. An adapter module is applied to bridge the speech module and the translation module. The CTC loss between speech features and source token sequence is incorporated during training. Experiments show that the systems can generate reasonable translations on three languages. The proposed models achieve BLEU scores of 22.3 for en→de, 10.7 for en→ja and 33.0 for en→zh on tst2023 TED datasets. However, the performance is decreased by a significant margin on complex scenarios like persentations and interview.

## introduction speech translation aims to solve the problem of translating speech waveform in source language into written text in target language. cascade systems decompose the problem into automatic speech recognition (asr) to transcribe source speech into source text and machine translation (mt) to translate source text into target text @xcite . it is clear that such architecture has the advantage of ensembling results from state-of-the-art (sota) asr models and mt models and the disadvantages of accumulating subsystem errors and discarding paralinguistic features. recent end-to-end speech translation (e2e st) systems have shown the potential to outperform cascade systems @xcite . however, due to the lack of highquality parallel training data, it is difficult to quantify the gap between the two categories. inspired by @xcite work, this submission explores various techniques to address problems in speech translation. 1) perform finegrained data filtering by calculating wers for speech data and alignment scores for translation data. 2) apply a straightforward split-and-merge method to split long audio clips into short segments. 3) employ a three-stage training strategy to concatenate the finetuned speech module and the translation module. 4) incorporate connectionist temporal classification (ctc) loss to leverage the divergence between speech features and source token sequences @xcite . experiments are carried out to perform speech translation at sentence level and corpus level. the performance of the three pt36 models is finally evaluated on the tst2023 datasets with automatic metrics. the rest of this paper is organized as follows. section 2 describes how speech data and translation data are processed in the experiments. section 3 explains how finetuned models are assembled to perform speech translation on all three languages. section 4 illustrates experiment setups, results and analysis. section 5 concludes the submission. ## conclusion in conclusion, this paper describes the end-to-end speech translation systems for iwslt 2023 offline tasks. built upon pretrained models, the systems are further trained on large amount of parallel data using the three-stage finetuning strategy. the pt36 model consists of an asr12 module with an adapter module for asr and an mt24 module for mt. the training loss sums up the ctc loss for asr and the cross entropy loss for mt. experiments demonstrate that the proposed methods have the potential to achieve a reasonable performance. however, due to limited resources, some modules has not well trained, which has a negative impact on subsequent tasks. therefore, the end-to-end models still underperform sota systems.