CMOT : Cross-modal Mixup via Optimal Transport for Speech Translation (2023)

Abstract: End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport (CMOT) to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text.

## introduction speech translation (st) is the task to translate speech signals in one language into text in another. conventional st systems work in a cascaded mode @xcite , by combining automatic speech recognition (asr) and machine translation (mt), which may suffer from error propagation and high latency. to solve these problems, end-to-end st is proposed @xcite . end-toend st directly generates the target text directly from the source audio without the intermediate transcript, which has drawn more attention in recent years @xcite @xcite @xcite @xcite . end-to-end st has to perform cross-lingual and cross-modal transformation simultaneously, hence bringing more challenges than asr or mt. unfortunately, it is harder to collect parallel speech-totext pairs than text-to-text pairs, which makes st suffer from the under-fitting problem. to solve the data shortage problem, some researchers propose methods to introduce text-to-text parallel data of mt into st, including pretraining @xcite @xcite , multi-task learning @xcite @xcite , knowledge distillation @xcite @xcite , etc. in this way, the knowledge of mt can be transferred to st, helping learn a better crosslingual transformation. although these methods of transfer learning have shown improvements in the translation quality, st is joint learning of cross-lingual and crossmodal transformation, and the effectiveness of transfer learning is also based on the assumption that the cross-modal source will be projected to the common representation space. therefore, the mixup strategy is employed to further improve st by reducing the gap between the speech and text representation spaces. previous methods with mixup strategy usually perform word-level mixup between aligned tokens in different modalities @xcite , hence needing the help of external alignment tools. however, the alignment tools are not always available and require large amounts of additional annotated data for training, so how to obtain such alignments remains a problem for most languages. illuminated by the application of the optimal transport theory in finding the alignment between languages or modalities in recent years @xcite , we propose crossmodal mixup via optimal transport (cmot). we use optimal transport to adaptively find the alignment between speech and text, and use this alignment to achieve token-level mixup. the mixup sequence will serve as a medium between the speech sequence and text sequence to realize cross-modal knowledge transfer. experiments on the must-c dataset show that cmot achieves an average bleu of 30.0. in addition, we prove that optimal transport can help find cross-modal alignments and cmot can help alleviate the modality gap in st. ## method in this section, we will describe our proposed cross-modal mixup via optimal transport (cmot). we find the alignment between speech and text sequences via ot, and mix up the two unimodal sequences to get a cross-modal mixed sequence. we predict the translation with all these sequences and regularize their outputs. figure 1 illustrates the overview of our proposed method. ## experiments we conduct experiments under two settings, the base and the expanded. the former only uses st data, while the latter uses external mt data additionally. we will describe the datasets we use, our experimental setups and our main results in this section. ## conclusion we propose cross-modal mixup via optimal transport (cmot) to adaptively find the alignment between speech and text sequences, and to mix up the sequences of different modalities at the token level. experiments on the must-c benchmark demonstrate the effectiveness of our method, which helps overcome the modality gap and thus improves the performance of end-to-end st systems. we will continue to explore the application of this adaptive cross-modal alignment method in the future.