Towards Zero-shot Learning for End-to-end Cross-modal Translation Models (2023)

Abstract: One of the main problems in speech translation is the mismatches between different modalities. The second problem, scarcity of parallel data covering multiple modalities, means that the end-to-end multi-modal models tend to perform worse than cascade models, although there are exceptions under favorable conditions. To address these problems, we propose an end-to-end zero-shot speech translation model, connecting two pre-trained uni-modality modules via word rotator’s distance. The model retains the ability of zero-shot, which is like cascade models, and also can be trained in an end-to-end style to avoid error propagation. Our comprehensive experiments on the MuST-C benchmarks show that our end-to-end zero-shot approach performs better than or as well as those of the CTC-based cascade models and that our end-to-end model with supervised training also matches the latest baselines.

## introduction speech translation (st) requires knowledge transfer among different modalities, whereas models more often than not perform worse on cross-modal tasks. the st model in real-world applications is usually a cascade approach that first uses an automatic speech recognition (asr) system to transcribe the speech into text and then uses a text machine translation (mt) model. recent end-to-end (e2e) st models remove the need for an explicit asr, with several practical advantages over the cascade models such as reduced latency, reduced error propagation, and shorter pipeline. however, e2e st models are less competitive than cascade models in practice @xcite @xcite because end-to-end data are an order of magnitude less than those for asr or mt, especially for low-resource language pairs. solutions have been proposed to combat this data problem. in @xcite , an adapter with additional parameters is used during fine-tuning to combine the two pretrained models of different modalities. the new module, however, only learns from st data, which is of a greatly reduced quantity. the alignment in building cross-modal representations is also a popular topic. @xcite simply concatenates the representations of different modalities and lets the self-attention learn the cross-modal features. some solutions deal with this problem through mapping features into fixed-size representations @xcite @xcite . the squared error is generally used as the optimization objective @xcite . they may suffer from information loss when representations are compressed or constrained prior. in order to overcome both the data and the length problems, we propose a pre-trainable adapter that connects two pre-trained modules. specifically, we adopt a popular cross-modal st architecture that can be generalized to many existing works. for the alignment adapter, we employ as loss the word rotator's distance (wrd) minimization @xcite @xcite , allowing the adapter to promote the cross-modal representations that match in the space of the semantic encoder. unlike previous works, this strategy allows us to pre-train the adapter. meanwhile, instead of mapping to a fixed length, the ctc enables adjustment of the length of the source modality representation dynamically. this step can guarantee the cross-modal representations become features with a similar but not exactly the same length, and then our proposed wrd objective with optimal transport (ot) solver can align them properly. besides speech translation, our approach can be naturally adapted into image translation. unlike the audio / image encoder layer softmax ctc loss linear + encoder layer wrd loss decoder layer ce loss shrink adapter transcription translation audio cross-modal encoder semantic encoder and decoder (a) model architecture (b) data usage subsample emb. layer linear emb. layer shared shrink layer cross-modal encoder shrink adapter semantic encoder decoder (c) semantic encoder and decoder training source text target text transcription transcription cross-modal encoder shrink adapter semantic encoder decoder wrd audio / image transcription transcription cross-modal encoder shrink adapter semantic encoder decoder wrd translation (d) zero-shot cross-modal training (e) end-to-end finetuning audio transcript domain a audio transcript tgt src src tgt domain b zero-shot setting image assistant translation @xcite , we attempt to translate the text within the image. our goal is also to align the cross-modal representation between images and texts @xcite . the related discussion and experiments can refer to the appendix a.4. the contributions of this paper are as follows: (1) we adopt the wrd loss together with the shrink mechanism to measure two feature sequences in different lengths, enabling the adapter pre-training. ## main method we adopt a popular framework in figure 1(a) , including a cross-modal encoder with a shrink adapter and a semantic encoder/decoder pack. ## conclusion in this paper, we present a zero-shot architecture that takes better advantage of cascade models, bridging the gap between cascade and end-to-end translation models. by leveraging differentiable shrink adapter and wrd loss, our approach is a direct end-to-end st model in the zero-shot setup that matches the cascade system without additional post-processing, e.g., rescoring via an additional language model. our method can also achieve comparable results to recent supervised st models.