Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT 23 (2023)

Abstract: This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.

## introduction in the past decade, the field of speech translation (st) has seen significant advancements, mainly due to end-to-end models that directly translate speech, offering a more efficient method compared to traditional cascade systems @xcite . despite data availability challenges, recent progress has diminished the performance disparity between these approaches @xcite @xcite . critical to the advancements in end-to-end models is the exploitation of asr and mt data through pretraining strategies @xcite @xcite @xcite @xcite . recently, @xcite proposed a method to effectively utilize both asr and mt pretraining to enhance st. this approach involves pretraining an encoder-decoder mt system with available text data, followed by pretraining a speech encoder to generate representations similar to the mt system's encoder (siamese pretraining) using connectionist temporal classification (ctc) supervision @xcite and optimal transport @xcite . the resulting speech encoder and text decoder can be fine-tuned with st data. another way of incorporating asr and mt is to leverage large pretrained speech and text models as a foundation for end-to-end st systems @xcite @xcite @xcite . however, these systems encounter representation discrepancy issues, which can hinder the full exploitation of pretrained foundation models. @xcite ; @xcite aimed to address this by adding coupling modules after the pretrained encoder, while other focus on solving the length discrepancies @xcite @xcite . @xcite tackled the issue by projecting speech and text features to a common semantic space using attention mechanisms and semantic memories. in our work, we tackle the issue of misaligned speech and text encoder representations by adopting the approach proposed by @xcite . our system uses a speech foundation model finetuned on english asr, wav2vec 2.0 @xcite , and an mt foundation model fine-tuned on multilingual mt (en-xx), mbart50 @xcite , as described in section 2.1. building on prior research @xcite , we employ two encoders: an acoustic encoder from wav2vec 2.0 and a semantic encoder from mbart50. coupling modules link these encoders to address length discrepancy. we extend @xcite by applying ctc and ot losses to the outputs of the acoustic and semantic encoders, respectively, add a second auxiliary ot loss for the inputs of the semantic encoder, and keep the text encoder frozen to keep the mt space intact. this method aligns the speech encoder's represenfigure 2: speech translation fine-tuning tations with the mt foundation model, effectively improving the final st system's performance by mitigating representation mismatch. in summary, we participate in the iwslt 2023 offline speech translation task, focusing on translating spoken english to written german, by employing an end-to-end system. we leverage asr and mt foundation models with the siamese pretraining approach, to effectively bring their encoder's representations closer. we furthermore decouple acoustic and semantic modeling in our speech encoder, adjust for the length miss-match between speech and text with several coupling modules, and apply knowledge distillation @xcite from mt @xcite , using mbart50. ## methodology our system, an encoder-decoder transformer, leverages asr and mt foundation models ( §2.1). we initially train the speech encoder with an extended siamese pretraining ( §2.2), and then fine-tune it with the mt decoder for end-to-end st ( §2.3). ## experiments here we describe the experiments we carried out in this work. the implementation details are available in §a.1. iwslt '22 system for the iwslt 2022 offline task, our submission employed a hubert encoder @xcite and an mbart50 (en-xx) decoder, which were efficiently fine-tuned to st with the lna strategy @xcite and parallel adapters @xcite , using datasets such as must-c v2, europarl-st and covost. the architecture included three 1d convolutional layers between the encoder and decoder, resulting in a subsampling of the encoder representation by a factor of 8. the final ensemble also comprised models utilizing knowledge distillation and a wav2vec 2.0 encoder @xcite . baseline our baseline has four main differences compared our last year's best system. we did an initial exploratory analysis of various encoders ( §a.3), including different versions of wav2vec 2.0, and hubert. upon observing no significant differences, we opted to utilize wav2vec 2.0 fine-tuned with pseudo-labels @xcite , a more prevalent choice within the research community. despite the strong performance demonstrated by efficient fine-tuning with lna and parallel adapters, we chose to switch to standard st fine-tuning in order to optimize performance. moreover, we employ a semantic encoder initialized from the mt model. lastly, we also pre-train the foundation models, wav2vec 2.0 with ctc on the asr data of must-c, and mbart50 on the parallel text of must-c. it is important to note that only must-c data was utilized for the baseline. siamese pre-training instead of pre-training the speech encoder with ctc only, we follow the siamese pre-training method ( §2.2), with the encoder architecture described in §2.1, to align the encoder representations with the mt model's representation space. the system, instead of using three layers of 1d convolutions, now incorporates also ctc-based compression, a large adapter, and finally a single layer of 1d convolutions. following the siamese pre-training on must-c's asr data, we jointly fine-tune the model and the mt decoder on the must-c st data. similar to the baseline, the mt model is also fine-tuned on the parallel text of must-c beforehand. more data we extend the previously described process by incorporating additional data. initially, we fine-tune mbart50 using all the mt data ( ## results in in the first part of ## conclusions we described the submission of the upc machine translation group for the iwslt 2023 offline st task. our system leverages asr and mt foundation models and a siamese pretraining step to maximize the transfer learning from mt. we show that siamese pretraining can bring significant improvements to our st models, while fine-tuning with kd can also be helpful. we furthermore show that synthetic data are crucial at improving performance in the iwslt test sets. in future work, we plan to investigate the zero-shot capabilities of optimal transport in the context of foundation models.