Submission of USTC ’s System for the IWSLT 2023 - Offline Speech Translation Track (2023)

Abstract: This paper describes the submissions of the research group USTC-NELSLIP to the 2023 IWSLT Offline Speech Translation competition, which involves translating spoken English into written Chinese. We utilize both cascaded models and end-to-end models for this task. To improve the performance of the cascaded models, we introduce Whisper to reduce errors in the intermediate source language text, achieving a significant improvement in ASR recognition performance. For end-to-end models, we propose Stacked Acoustic-and-Textual En- coding extension (SATE-ex), which feeds the output of the acoustic decoder into the textual decoder for information fusion and to prevent error propagation. Additionally, we improve the performance of the end-to-end system in translating speech by combining the SATE-ex model with the encoder-decoder model through ensembling.

## introduction this paper describes the submission for the iwslt 2023 offline speech translation task @xcite by national engineering laboratory for speech and language information processing (nelslip) at the university of science and technology of china. speech translation (st) solutions include cascaded and end-to-end approaches. the cascaded approach combines automatic speech recognition (asr) and machine translation (mt) systems. the asr system recognizes the source speech as intermediate text in the source language, and the mt system translates the intermediate text into text in the target language. while the end-to-end approach directly translates the source speech into text in target language, without using source language text as an intermediate representation. compared with cascaded approaches, the end-to-end paradigm can overcome higher architectural complexity and error propagation @xcite . the stacked acoustic-and-textual encoding (sate) @xcite ) method combines the acoustic and textual encoders using an adapter module to approach the performance levels of cascaded solutions. furthermore, st can be improved using large-scale and cross-modal pretraining methods @xcite such as whisper @xcite , which leverages large-scale weak supervision, and speechut @xcite , which optimizes the alignment of speech and text modalities by hidden units. in this study, we employ a cascaded approach wherein the asr system is built using the pretrained whisper @xcite to ensure the recognition performance of speech to source language text. furthermore, the mt systems in the cascaded setup are created using diverse techniques like back translation @xcite , selftraining @xcite , domain adaptation and model ensemble. in end-to-end condition, we implement two types of architectures, including encoder-decoder @xcite and stacked acoustic-and-textual encoding extension (sate-ex). for the encoderdecoder, we use the corresponding components of asr models to initialize the encoder, and the corresponding components of mt models to initialize the decoder. for sate-ex, we utilize the textual decoder to receive the output features of the acoustic decoder to assist in generating the target language text, achieving information complementarity of different asr decoding hidden states, and preventing intermediate error propagation. additionally, we employ adaptation training, along with the adaptation module and multi-teacher knowledge distillation of stacked acoustic-and-textual encoding (sate) @xcite to bridge the gap between pre-training and fine-tuning. our approach included the utilization of augmentation strategies commonly used in cascaded systems, like speech synthesis @xcite ## experiments our experimental results are presented in ## conclusion this paper summarizes the results on the iwslt 2023 offline speech translation task. we employ various model architectures and data augmentation techniques to build speech translation systems in cascaded and end-to-end settings. the experimental results demonstrate the effectiveness of strategies such as pre-trained whisper models, adaptation training, and the stacked acoustic-and-textual encoding extension (sate-ex). in future work, we will further investigate sate-ex and explore multimodal representation learning in speech translation.