Pushing the Limits of Zero-shot End-to-End Speech Translation (2024)

Abstract: Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method’s superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.

## introduction traditionally, the standard approach for speech translation (st) has been the cascade model @xcite , where automatic speech recognition (asr) and machine translation (mt) systems were chained together to produce the desired output. however, in recent years, there has been a paradigm shift towards end-to-end models, which aim to directly map the input speech to the target text without the intermediate transcription step @xcite . such models offer several advantages, including reduced error propagation, more compact architectures, and faster inference times @xcite ). despite their advantages, end-to-end models require parallel st data, which are often limited. to mitigate this data scarcity, several strategies aim to leverage the more abundant asr and mt data @xcite . then, another challenge that arises is the modality gap, which emerges from the inherent differences between the speech and text modalities, manifesting in length mismatch and representation mismatch. convolutional length adapters @xcite or connectionist temporal classification (ctc) @xcite can alleviate the length mismatch to some extent. to ease the representation gap, previous works have proposed a shared encoder @xcite and optimizing distance metrics to bring the speech-text representation closer to the shared semantic space @xcite . due to its ability to handle representations of different lengths, the wasserstein distance @xcite using optimal transport, has emerged as an effective distance metric, specifically during pretraining @xcite . although these methods show promise in reducing the modality gap and increasing translation quality, they still require at least some st data. to this end, we propose zero-shot st with subword compression and optimal transport (zeroswot), a novel approach that aims to bridge the modality gap in a zero-shot fashion, only requiring asr and mt data. using optimal transport, we train a speech encoder based on wav2vec 2.0 @xcite to produce representations akin to the embedding space of a massively multilingual mt model, thus enabling zero-shot st inference across all its supported target languages (fig. 1 ). to map the two embedding spaces, our method learns the tokenization of the mt model through ctc, and compresses the speech representations to the corresponding subwords, thus closing the length mismatch. experiments on @xcite reveal that our method is not only better than existing zero-shot models, by a large margin, but also surpasses supervised ones, achieving state-of-the-art results. on covost @xcite , zeroswot outperforms the original version of the multimodal seamlessm4t (seamless-communication, 2023a), while evaluations on the 88 target languages of fleurs @xcite showcase the massively multilingual capacity of our method. zeroswot is also vastly superior to comparable ctc-based cascade st models, and while it is on par with cascades that utilize strong attention-based encoder-decoder asr models, it ranks better in terms of efficiency. finally, we provide evidence of our method's ability to close the modality gap, both in terms of length and representation. ## experimental setup data. for training we are using common voice @xcite , the speech-transcription pairs of must-c v1.0 @xcite , and lib-rispeech @xcite . we evaluate our models on three multilingual st benchmarks: must-c v1.0 (@xmath0.), covost 2 @xcite (@xmath1.), and fleurs (conneau et al., 2023) (@xmath2.) (appendix a). model architecture. the acoustic encoder follows wav2vec 2.0 large @xcite with @xmath3. the ctc module uses a vocabulary with size |v| = 30, and together with the acoustic encoder are initialized from a ctcfinetuned version of wav2vec 2.0 on librispeech @xcite . the subword encoder has 3 layers with dimensionality of 1024 (35m). the text encoder has either @xmath4, both with dimensionality 1024, and an embedding size of |b| = 256k. we initialize its parameters from two dense versions of nllb @xcite , the 600m (medium) or the 1.3b (large). based on the size of nllb, the resulting zeroswot models used for inference have 0.95b (medium) or 1.65b (large) parameters, while both of them the have the same number of training parameters (350m). we also train a small version based on wav2vec 2.0 base and nllb-medium (appendix c.1). training details. we use adamw @xcite with a learning rate of 3e-4, inverse square root scheduler with warmup, and batch size of 32m tokens. we apply dropout of 0.1 as well as time and channel masking @xcite . speech input is 16khz waveforms, and text is tokenized with sentencepiece @xcite , using the nllb model (appendix c.2). loss. we set α = 0.9 and apply wasserstein losses for @xmath5, 7, 8, 9, 10, 11, 12} (medium) and @xmath6, 14, 16, 18, 20, 22, 24} (large) @xcite . positional and entropy regularization are set to µ = 10 (eq. 14, 15), and λ = 1 (eq. 16). evaluation. we apply checkpoint averaging and use a beam search of size 5. we evaluate with casesensitive detokenized bleu @xcite using sacrebleu @xcite (appendix b). ## conclusions we presented zeroswot, a novel method for zeroshot end-to-end speech translation, that works by adapting a speech encoder to the representation space of a massively multilingual mt model. our method sets new sota in must-c, while it also surpasses larger, supervised models in covost. we additionally showed that zeroswot outperforms cascades, both in performance and efficiency. finally, we provided evidence of closing the modality gap, with respect to both length and representation. future research will expand on low-resource scenarios and speech-to-speech translation.