FCGCL : Fine- and Coarse-Granularity Contrastive Learning for Speech Translation (2022)

Abstract: It is notoriously difficult to implement end-to-end speech translation (E2E-ST) model because of the task complexity and data scarcity. Existing techniques often attempt to carry out implicit knowledge transfer from machine translation (MT) to ST model by imposing various constraints. However, in this transfer scenario, a significant problem is that the performance of the MT will drop significantly and the final transfer effect is also restricted. In this article, we recommend Fine and Coarse Granularity Contrastive Learning (FCGCL), which conduct explicit knowledge transfer from MT to ST model. Specially, we ensure through multi granularity contrastive learning that inputs with similar semantic between different modalities are encoded closely in the shared semantic space while inputs with different semantics are kept apart. Experiments on the MuST-C datasets on all 8 languages and further analysis show that our method can effectively improve the E2E-ST performance and achieves an average BLEU of 29.0.

## introduction end-to-end (e2e) speech-to-text translation (st) @xcite @xcite @xcite @xcite @xcite is the task of translating a source-language audio directly to a foreign language text without any intermediate outputs. different from the traditional cascade method @xcite @xcite @xcite which decomposes st into two sub-tasks -automatic speech recognition (asr) for transcription and machine translation (mt) for translation, e2e-st jointly handles them in a single neural network. this endows e2e-st with special advantages, such as lower latency, less error propagation, and fewer parameters. however, e2e-st is a cross-modal translation task, and it is non-trivial to train such a model well. speech is more complicated and finer granularity than text, making it more difficult to extract representations containing rich semantic information. such modal gap between speech and text results in the performance of st model is usually inferior to the corresponding mt model @xcite . to overcome this problem, existing techniques often attempt to carry out implicit knowledge transfer by imposing various constraints @xcite @xcite . in this transfer scenario, a significant problem is that the performance of the mt as a constraint will drop significantly and the final transfer effect is also restricted. in this paper, we propose a cross-modal multi granularity contrastive learning method to make explicit knowledge transfer from mt to st model. the embedding from a st model encoder can be regarded as frame-level representation of fine granularity. correspondingly, the mean vector of embeddings is termed as the coarse granularity sentencelevel representation. the same as st model, there are two types of representations with different granularities obtaining from the mt model encoder. we perform fine and coarse granularity contrastive learning (fcgcl) on both sentencelevel and frame-level to provide comprehensive guidance for the extraction of speech representations to bridge the modal gap. the mt model is pretrained and we frozen its parameters to avoid the performance drop. moreover, another problem needs to be solved is the representation degeneration @xcite @xcite suffered by the pretrained mt model, which is also called anisotropy. in this paper we take a simple whiten-ing approach @xcite to alleviate representation degeneration problem by transforming the representation into a standard normal distribution, which satisfies isotropy. we conduct experiments on the must-c benchmark on all 8 language pairs. the experiment results and detailed analysis verify the effectiveness of our proposed method. ## method in this section we first analysis the basic problem formulation of e2e-st and introduce the overall framework of fcgcl. then the coarse granularity contrastive learning and the whitening operation to alleviate the representation degeneration problem are stated in section 2.3. section 2.4 details the fine granularity contrastive learning and the maximum similarity method which is used to find the corresponding text token for each speech frame in an unsupervised manner. finally, we describe knowledge distillation in section 2.5. ## conclusion in this paper, we recommend fcgcl, which combines contrastive learning and knowledge distillation for explicit knowledge transfer across modalities. in addition, we recommend whitening to solve the representation degeneration problem of text representation in mt model. experiments on the must-c dataset on all 8 languages demonstrate the effectiveness of our method.