Understanding and Bridging the Modality Gap for Speech Translation (2023)

Abstract: How to achieve better end-to-end speech translation (ST) by leveraging (text) machine translation (MT) data? Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping. However, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias. We find that the modality gap is relatively small during training except for some difficult cases, but keeps increasing during inference due to the cascading effect. To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (Cress) method. Specifically, we regularize the output predictions of ST and MT, whose target-side contexts are derived by sampling between ground truth words and self-generated words with a varying probability. Furthermore, we introduce token-level adaptive training which assigns different training weights to target tokens to handle difficult cases with large modality gaps. Experiments and analysis show that our approach effectively bridges the modality gap, and achieves significant improvements over a strong baseline in all eight directions of the MuST-C dataset.

## introduction end-to-end speech translation (st) aims to translate speech signals to text in another language directly. compared to traditional cascaded methods, which combine automatic speech recognition (asr) and machine translation (mt) models in a pipeline manner, end-to-end st could avoid error propagation and high latency @xcite . recently, end-to-end st models have achieved comparable or even better results than cascaded st models @xcite @xcite . however, due to the scarcity of st data, it is difficult to directly learn a mapping from source speech to the target text. previous works often leverage mt data to help the training with multi-task learning @xcite . by sharing encoder and decoder between st and mt, the model tends to learn similar representations from different modalities. in this way, the auxiliary mt task can help build the source-to-target mapping. however, there remains a gap between st and mt due to the differences between speech and text. in this paper, we measure the modality gap with representation differences of the last decoder layer between st and mt, because the representation of this layer will be mapped into the embedding space to obtain the final translation. a significant modality gap potentially causes different predictions, which makes st lag behind mt. thanks to multi-task learning, we observe that when training with teacher forcing, where both st and mt use ground truth words as target-side contexts, the modality gap is relatively small except for some difficult cases. however, the exposure bias problem can make things worse. during inference, both st and mt predict the next token conditioned on their previously generated tokens, which may be different due to the modality gap. moreover, different predictions at the current step may lead to even more different predictions at the next step. as a result, the modality gap will increase step by step due to this cascading effect. to solve these problems, we propose the crossmodal regularization with scheduled sampling (cress) method. to reduce the effect of exposure bias, we introduce scheduled sampling during training, where the target-side contexts are sampled be-tween ground truth words and self-generated words with a changing probability. based on this, we propose to regularize st and mt in the output space to bridge the modality gap by minimizing the kullback-leibler (kl) divergence between their predictions. this will encourage greater consistency between st and mt predictions based on partial self-generated words, which is closer to the inference mode. besides, to handle the difficult cases, we introduce token-level adaptive training for cress, where each target token is given a varying weight during training according to the scale of the modality gap. in this way, those cases with significant modality gaps will be emphasized. we conduct experiments on the st benchmark dataset @xcite . results show that our approach significantly outperforms the strong multi-task learning baseline, with 1.8 bleu improvements in the base setting and 1.3 bleu improvements in the expanded setting on average. further analysis shows that our approach effectively bridges the modality gap and improves the translation quality, especially for long sentences. ## preliminary studies on modality gap with multi-task learning, most of the knowledge of mt can be transferred to st. however, the performance gap between st and mt still exists. in this section, we first conduct some preliminary studies with our multi-task learning baseline model to understand where this gap comes from. ## method: cress our preliminary studies in section 3 show that: inspired by these, we propose the cross-modal regularization with scheduled sampling (cress) method to bridge the modality gap, especially in inference mode (section 4.1). furthermore, we propose a token-level adaptive training method for cress to handle difficult cases (section 4.2). ## analysis and discussion results in section 5.3 show the superiority of our method. to better understand cress, we explore several questions in this section. all analysis experiments are conducted on must-c @xmath0. ## related work end-to-end speech translation end-to-end speech translation @xcite has shown great potential for overcoming error propagation and reducing latency compared to traditional cascaded st systems @xcite , 2019; di @xcite . one challenge in training end-to-end st models is the scarcity of st data. to address this problem, researchers employed mt data to help training with techniques like pre-training @xcite @xcite @xcite @xcite @xcite , multi-task learning @xcite @xcite @xcite , knowledge distillation @xcite , and data augmentation @xcite @xcite . however, due to the modality gap between speech and text, it is still difficult to fully exploit mt data with the above techniques. to overcome the modality gap, han et al. (2021) projects features of both speech and text into a shared semantic space. fang et al. (2022); zhou et al. (2023) mixes up features of speech and text to learn similar representations for them. ye et al. (2022) brings sentence-level representations closer with contrastive learning. bapna et al. (2021, 2022); chen et al. (2022); tang et al. (2022); zhang et al. (2022b) jointly train on speech and text and design methods to align two modalities. different from previous work, in this work, we understand the modality gap from the target-side representation differences, and show its connection to exposure bias. based on this, we propose the cross-modal regularization with scheduled sampling (cress) method to bridge the modality gap. exposure bias exposure bias indicates the discrepancy between training and inference. several approaches employ reinforcement learning (rl) @xcite @xcite instead of maximum likelihood estimation (mle) to avoid this problem. however, @xcite shows that rl-based training is unsta-ble due to the high variance of gradient estimation. an alternative and simpler approach is scheduled sampling @xcite , which samples between ground truth words and self-generated words with a changing probability. @xcite extends it with gumbel noise for more robust training. in this paper, we adopt this approach to approximate the inference mode due to its training stability and low training cost. output regularization for mt regularization in the output space has proved useful for mt. @xcite proposes to regularize the output predictions of two sub-models sampled by dropout. @xcite regularizes the output predictions of models before and after input perturbation. in this paper, we regularize the output predictions across modalities, which encourages more consistent predictions for st and mt. ## conclusion in this paper, we propose a simple yet effective method cress to regularize the model predictions of st and mt, whose target-side contexts contain both ground truth words and self-generated words with scheduled sampling. based on this, we further propose a token-level adaptive training method to handle difficult cases. our method achieves promising results on must-c benchmark. further analysis shows that our method can effectively bridge the modality gap and improve the translation quality, especially for long sentences. in the future, we will explore how to apply our method to other tasks.