Cross-modal Contrastive Learning for Speech Translation (2022)

Abstract: How can we learn unified representations for spoken utterances and their written text? Learning similar representations for semantically similar speech and text is important for speech translation. To this end, we propose ConST, a cross-modal contrastive learning method for end-to-end speech-to-text translation. We evaluate ConST and a variety of previous baselines on a popular benchmark MuST-C. Experiments show that the proposed ConST consistently outperforms the previous methods, and achieves an average BLEU of 29.4. The analysis further verifies that ConST indeed closes the representation gap of different modalities — its learned representation improves the accuracy of cross-modal speech-text retrieval from 4% to 88%. Code and models are available at https://github.com/ReneeYe/ConST .

## introduction end-to-end speech-to-text translation (e2e st) becomes important in many internet products and real applications. an e2e st system accepts audio signals as the input and generates the target translation using a single model. compared with the conventional cascade st models, e2e st models have achieved almost comparable @xcite or even superior @xcite @xcite performance. the performance of an e2e st model is still restricted for languages with relatively small parallel data, compared to text machine translation (mt). existing approaches for st focus on using additional data from mt and automatic speech recognition (asr). this can be realized through pretraining approaches @xcite or multi-task training frameworks @xcite @xcite . we analyze transformer models for speech translation and observe a noticeable modality gap between encoder representations of speech and text from existing st models (as in figure 1a . sec. 6 has more details). an ideal representation should satisfy: if the content of the speech and transcription are similar, their encoded representations should likewise be close to each other (as in figure 1b ). nevertheless, how to learn unified and aligned speech-text representations? inspired by the recent progress of contrastive learning approaches in cross-lingual @xcite and cross-modal vision-and-language domains @xcite @xcite , we designed a simple contrastive learning method for st (const) to learn the representations that meet the aforementioned conditions explicitly. on the one hand, our model inherits the advantages of the previous multi-task learning methods. on the other hand, it reduces the gap between the representations of speech and its corresponding transcription. our contributions are as follows. end-to-end st to alleviate the error propagation in the cascaded st systems and to make the deployment simpler, bérard et al. (2016); weiss et al. (2017) proposed to use an end-to-end architecture to directly translate speech into text in another language, without the intermediate transcription. kano et al. (2017); berard et al. (2018); inaguma et al. (2020); wang et al. (2020a); zhao et al. (2021a) implemented several off-the-shelf encoderdecoder e2e-st models, such as bilstm @xcite and speech-transformer @xcite . however, training an end-to-end speech translation model is difficult because we need to design a cross-modal cross-language model, meanwhile, the speech-transcription-translation supervised data for speech translation is significantly less than that of mt and asr. methods, like data augmentation @xcite @xcite , pre-training @xcite @xcite @xcite @xcite , self-training @xcite , utilizing selfsupervised pre-trained audio representation @xcite @xcite , are proved to be effective. meanwhile, some work has shown that the encoderdecoder model with a single encoder cannot encode speech information well. for example, @xcite first proposed a second encoder to further extract semantic information of the speech sequence. @xcite proposed a stacked acoustic-and-textual encoder and introduced largescale out-of-domain data. also, multi-task frameworks @xcite @xcite are widely applied to further enhance the robustness for st. as a cross-modal task, some work has noted the problem of the modality gap. @xcite designed a fix-size semantic memory module to bridge such a gap, from the neuroscience perspective. however, we find that this approach actually sacrifices the effect of mt. so in this paper, we propose a simple yet effective contrastive learning method to bridge the gap and to improve st performance. cross-modal grounding learning this paper attempts to address the problem in speech translation from the perspective of cross-speech-text representation learning. we are also inspired by crossmodal representation learning in the acoustic word embedding (awe) @xcite @xcite and the visuallanguage pre-training (vlp) @xcite @xcite tasks. these works usually focus on enhancing textual representations with acoustic or visual information, in other words, grounding learning. in this work, we consider the its dual form, i.e., grounding speech representations using text. ## the const approach an end-to-end speech translation model directly translates audio sequence @xmath0, ..., s |s| ) to the text @xmath1, ..., y |y| ) in the target language. speech translation corpus @xmath2, x, y)} provides transcript @xmath3, ..., x |x| ) in the source language, as well. in this section, we present the overall speech translation model and cross-modal contrastive learning. we also provide several feasible strategies to construct more positive and negative pairs to enhance the contrastive learning. ## why does cross-modal contrastive learning work? -analysis on the modality gap as mentioned earlier, the existing multi-task training models cannot address the speech-text modality gap. does const reduce the representation gap between speech and text? to visualize it, we plot the bivariate kernel density estimation (parzen, 1962) (kde) contour of their dim-reduced features, where t-sne (van der @xcite is used to reduce the dimension into two (figure 5 ). ideally, if the representations of speech and its corresponding tran-script are similar, their kdes will be similar, and thus the contour lines will overlap as much as possible. however, figure 5 (a) is the kde contour of the multi-task framework without any explicit modeling to bring two modalities together @xcite . it shows that the representations are so dissimilar that they are organically divided into two clusters, i.e. speech-text modality gap exists. does const reduce the modality gap? as shown in figure 5 (b), compared to the baseline model without contrastive learning, const with cross-modal contrastive learning is able to bring representations of different modalities much closer. this means that the audio representation contains more linguistic information similar to that of the textual transcription, which is more advantageous for the downstream st generation through the shared transformer encoder and decoder. ## case analysis in this section, we use several cases that const generates. we compare our model with the cascaded model and the previous end-to-end model, ## conclusion in this paper, we propose const, a simple yet effective contrastive learning framework bridging the speech-text representation gap and facilitating the st with limited data. we also provide feasible hard example mining methods to learn robust representations. the results on the must-c st dataset prove the effectiveness of the method.