I 2 R ’s End-to-End Speech Translation System for IWSLT 2023 Offline Shared Task (2023)

Abstract: This paper describes I2R’s submission to the offline speech translation track for IWSLT 2023. We focus on an end-to-end approach for translation from English audio to German text, one of the three available language directions in this year’s edition. The I2R system leverages on pretrained models that have been exposed to large-scale audio and text data for our base model. We introduce several stages of additional pretraining followed by fine-tuning to adapt the system for the downstream speech translation task. The strategy is supplemented by other techniques such as data augmentation, domain tagging, knowledge distillation, and model ensemble, among others. We evaluate the system on several publicly available test sets for comparison.

## introduction historically, speech translation (st) has involved combining automatic speech recognition (asr) and machine translation (mt) systems in a cascade. the asr system would transcribe speech signals into text in the source language, and the mt system would then translate this text into the target language. however, recent developments in deep learning have made it possible to use an end-to-end speech translation model @xcite , which directly translates speech in the source language into text in the target language, without relying on intermediate symbolic representations. this approach offers the advantages of lower latency and avoids error propagation. while cascaded models initially outperformed end-to-end models, recent results from iwslt campaigns @xcite @xcite have shown that the performance of end-to-end models is now approaching that of cascaded solutions. large pretrained models @xcite @xcite have be-come a prevalent basis for speech and language processing work @xcite . through the utilization of pretrained models and subsequent finetuning using a small amount of labeled data, many tasks have exhibited significant improvements in performance @xcite @xcite , some even reaching state-of-the-art results. in this work, we describe our end-to-end system for the offline speech translation task at iwslt 2023 @xcite in the english-german (en-de) language direction. the current year's task not only includes the traditional ted talk evaluation set translated from english to german, but also introduces two additional test sets consisting of acl presentations, press conferences and interviews (empac), which are more complex and challenging. furthermore, this year's constrained data track allows less data than previous years. our team enhances the end-to-end st system within the context of the pretrain-finetune paradigm. we introduce several pretraining stages before finetuning for the downstream st task. furthermore, we implemented dynamic audio augmentation methods to account for differences in audio recording quality. we boost the system's robustness by ensembling multiple individual models and use domain tagging to direct the model towards specific output styles. here, we evaluate our system against various standard public test sets for both speech translation and text machine translation. ## methodology in this section, we introduce the model architecture of our system, and describe some of the methods we incorporated into the design and training process. ## results and analysis we present our experimental results and analyses in this section. ## conclusion in this paper we outline our proposed end-to-end system that incorporates pretrained models trained on large-scale audio and text data to enhance the st performance. the system underwent several stages of additional pretraining followed by finetuning for the downstream speech translation task. we explored several techniques including audio aug-mentation, domain tagging, knowledge distillation and model ensemble to improve the system's performance. we utilize both speech and text inputs, and propose a mixing procedure to unify representations from both modalities to not only increase the amount of available training data but also better adapt the model to downstream speech tasks. we plan to carry out more experiments to further explore the effect of modality mixing and improve the performance of such models for speech-to-text tasks.