Improving Speech Translation by Fusing Speech and Text (2023)

Abstract: In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text to improve speech translation. However, speech and text are disparate modalities, we observe three aspects of modality gap that impede their integration in a speech translation model. To tackle these gaps, we propose **Fuse**-**S**peech-**T**ext (**FuseST**), a cross-modal model which supports three distinct input modalities for translation: speech, text and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation and fused speech-text translation. We evaluate FuseST on MuST-C, GigaST and newstest benchmark. Experiments show that the proposed FuseST achieves an average 34.0 BLEU on MuST-C En \rightarrow De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FuseST does not degrade on MT task, as observed in previous works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model. Code is available at https://github.com/WenbiaoYin/FuseST.

## introduction speech translation (st) accepts speech signals as the input and outputs target translation. speech translation can be broadly categorized into cascade system and end-to-end speech translation (e2e st). cascade system @xcite @xcite usually combines automatic speech recognition (asr) and machine translation (mt). the mt subsystem uses asr transcripts as input, which provide clear expression but may contain errors stemming from asr. while e2e st @xcite @xcite ., * corresponding author " i'm [noise] delight [noise] to be here." " i'd like to be here." asr end-to-end st fusest cascade mt subsystem target translation speech: correct but contains noise transcript: clear but contains error (a) (b) fusion transcript semantic error speech noise correct speaker paralinguistics clear fusion: correct and clear 2022) can directly map speech signals to the target translation, thus avoiding the problem of error propagation. we observe a noticeable modality distribution difference between speech and asr transcript as shown in figure 1a . speech signals contain abundant information, such as paralinguistics and speaker characteristics, but they are harder to model and more susceptible to noise. on the other hand, asr transcripts are clear but may contain errors. for example, as shown in figure 1b , the speech signal conveys the message "i'm delight to be here", but the speech signal contains numerous blank segments and background noise. e2e st may encounter challenges in accurately extracting information directly from speech, particularly in the presence of noise, without compromising translation quality. meanwhile, the asr model mistakenly transcribes the speech as "i'd like to be here". the mt subsystem may proceed with translation without awareness of errors in "i'd like". inspired by these findings, we propose a model that fuses speech and asr transcript (fused speechtext) as input to leverage their complementary strengths to improve speech translation. as shown in figure 1b , our model supports three distinct modalities of input for translation: speech, text and fused speech-text. however, speech and text are disparate modalities, we observe three aspects of modality gap that impede their integration in a speech transla-tion model: 1. the speech representation is in continuous space, while the text representation is in discrete space. 2. when using speech and golden transcript as input, the model relies heavily on the golden transcript information and neglects the speech information. this behavior is because the golden transcript is more straightforward and contains precise semantic information, adequate for achieving high translation quality. the model will learn a shortcut to rely solely on the golden transcript. in contrast, speech representations obtained through the pre-trained model are harder to model, they include four types of information @xcite : content, semantics, paralinguistics and speaker characteristics. however, when using speech and asr transcript as input, solely relying on textual information is insufficient to achieve good translation quality. we expect the model to incorporate more speech information. 3. diverse modality inputs lead to distinct hidden states in the encoder and distinct distributions in the decoder. we propose several methods to bridge the modality gap to better integrate speech and text. 1. we explore mapping continuous speech representation to a discrete space using a codebook to align with the text representation. 2. we adopt the prompt tags implicitly guide the model to utilize more speech information when the transcript is inaccurate. we further explore how to explicitly guide the model to fuse speech and text. meanwhile, we apply crossmodal contrastive learning (ccl, @xcite to reduce the gap between the model semantic of speech and its corresponding transcript. 3. we adopt cross-attentive regularization (car, @xcite to align the states of the encoder and cross-modal regularization (cmr) to align the distribution of the decoder. our contributions are summarized as follows: ## related work cascade st cascade st, achieved by concatenating asr and mt components, has been extensively employed in commercial speech translation systems. however, cascade st is vulnerable to challenges such as error propagation and high latency. to overcome the error propagation, @xcite @xcite proposed to feed the mt system with asr data structures; @xcite @xcite proposed to make mt robust to asr errors, for instance by training it on parallel data incorporating factual or emulated asr errors. end-to-end st to overcome the error propagation and high latency in the cascade st systems, @xcite proposed an end-to-end architecture for speech translation, which has attracted extensive attention @xcite @xcite @xcite . however, it is difficult to train an end-to-end speech translation model directly, primarily due to the inherent variability and complexity of speech signals and the scarcity of high-quality speech-translation datasets. some training methods like pretraining @xcite @xcite @xcite , multi-task learning @xcite @xcite , data augmentation @xcite @xcite , meta-learning @xcite , contrastive learning @xcite , knowledge distillation @xcite and curriculum learning @xcite , are proved to be effective. ## case study in this section, we present several cases generated by fusest-mt, fusest-st and fusest-ft. in the first case, the asr model mistakenly transcribes "delight" as "i'd like" due to the highly similar pronunciations of these two words. fusest-mt fails to generate the correct translation as a result of errors present in the asr transcript. meanwhile, fusest-st produces omissions, as modeling direct speech to target translation proves to be more challenging. notably, only fusest-ft translates the sentence correctly, leveraging the complementary strengths of speech and text. in the second case, the speech signal contains numerous background noises; the asr model mistakenly transcribes the "they are" as "their", fusest-mt and fusest-st are mistranslated, and only fusest-ft translates correctly. ## conclusion in this paper, we propose fusest, a cross-modal model which supports three distinct input modalities for translation: speech, text and fused speechtext. we comprehensively analyze the modality gap between speech and text and, utilize multiple techniques to bridge the modality gap. we then fuse speech and text to improve speech translation. experiments and analysis demonstrate the effectiveness of our proposed method.