STEMM : Self-learning with Speech-text Manifold Mixup for Speech Translation (2022)

Abstract: How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech sequences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a self-learning framework. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy, and achieves significant improvements over a strong baseline on eight translation directions.

## introduction speech-to-text translation (st) aims at translating acoustic speech signals into text in a foreign language, which has wide applications including voice assistants, translation for multinational video conferences, and so on. traditional st methods usually combine automatic speech recognition (asr) and machine translation (mt) in a cascaded manner @xcite @xcite @xcite , which might suffer from error propagation and high latency. to break this bottleneck, end-to-end st systems attracted much attention recently @xcite @xcite @xcite , which learn a unified model to generate translations from speech directly. some recent work has shown great potential for end-toend speech translation, even surpassing traditional cascaded systems @xcite . as a cross-modal task, a major challenge in training an end-to-end st model is the representation discrepancy across modalities, which means there is a modality gap between speech representations and text embeddings, as shown in the left sub-figure of figure 1 . existing approaches often adopt a sophisticated mt model to help the training of st, with some techniques like pretraining @xcite @xcite , multitask learning @xcite @xcite and knowledge distillation @xcite @xcite . although these methods have achieved impressive improvements in st task, these methods are not necessarily the best way to leverage the mt knowledge. considering that during training, the input of the translation module only include speech sequences or text sequences, the lack of multimodal contexts makes it difficult for the st model to learn from the mt model. inspired by recent studies on some cross-lingual @xcite @xcite and cross-modal @xcite @xcite ) tasks, we suggest that building a shared semantic space between speech and text, as illustrated in the right sub-figure of figure 1 , has the potential to benefit the most from the mt model. in this paper, we propose the speech-text manifold mixup (stemm) method to bridge the modality gap between text and speech. in order to calibrate the cross-modal representation discrepancy, we mix up the speech and text representation as the input and keep the target sequence unchanged. specifically, stemm is a self-learning framework, which takes both the speech representation and the mixed representation as parallel inputs to the translation model, and regularizes their output predictions. experimental results show that our method achieves promising performance on the benchmark dataset @xcite , and even outperforms a strong cascaded baseline. furthermore, we found that our stemm could effectively alleviate the cross-modal representation discrepancy, and project two modalities into a shared space. ## method in this section, we will begin with the basic problem formulation (section 2.1) and introduce the model architecture (section 2.2). then, we introduce our proposed speech-text manifold mixup (stemm) in section 2.3. finally, we introduce our proposed self-learning framework with stemm in section 2.4 and present two mixup ratio strategies in section 2.5. figure 2 illustrates the overview of our proposed method. ## conclusion in this paper, we propose a speech-text manifold mixup (stemm) method to mix up the speech representation sequences and word embedding sequences. based on stemm, we adopt a selflearning framework, which learns the translation of unimodal speech sequences and multimodal mixed sequences in parallel, and regularizes their output predictions. experiments and analysis demonstrate the effectiveness of our proposed method, which can alleviate the cross-modal representation discrepancy to some extent and improve the performance of st. in the future, we will explore how to further eliminate this discrepancy and fill the cross-modal transfer gap for st.