A Simple Concatenation can Effectively Improve Speech Translation (2023)

Abstract: A triple speech translation data comprises speech, transcription, and translation. In the end-to-end paradigm, text machine translation (MT) usually plays the role of a teacher model for the speech translation (ST) via knowledge distillation. Parameter sharing with the teacher is often adopted to construct the ST model architecture, however, the two modalities are independently fed and trained via different losses. This situation does not match ST’s properties across two modalities and also limits the upper bound of the performance. Inspired by the works of video Transformer, we propose a simple unified cross-modal ST method, which concatenates speech and text as the input, and builds a teacher that can utilize both cross-modal information simultaneously. Experimental results show that in our unified ST framework, models can effectively utilize the auxiliary information from speech and text, and achieve compelling results on MuST-C datasets.

## introduction speech translation (st) is the task that automatically translates a source acoustic speech signal into a text sequence in a target language. with the advance of transformer, recent works on end-to-end speech translation (e2e st) can alleviate many problems usually occurred in the cascade system and achieve comparable performance @xcite @xcite . for the e2e st model, mt is often used as the teacher of st, and methods such as knowledge distillation or contrastive learning are used to bridge the modality gap. the mt teacher only uses the source text (transcription) information. the speech and text modalities are consumed individually by st model. there are two main drawbacks. one is the teacher mt model can not use speech information, which limits the overall model perfor-mance. the other is mt uses text input, st uses the speech input, then close the two individual modalities. there is no unified module can simultaneously use cross-modal information. here, we take a further step towards more effective use of both speech and transcription text in st. inspired by the related works of video transformer @xcite , when processing video, concatenating video information and text embedding information can better model the cross-modal information of the video. we concatenate the preprocessed speech and the transcription text jointly, and encode the two-modal information simultaneously. following the recent popular advance in e2e st with knowledge distillation (kd) @xcite , it provides a practical paradigm for transferring knowledge from rich-resource mt task to limited resource st task. however, we re-define the role of teacher in our framework, because the information of the two modalities can further improve the upper bound of model performance than the single modality. our proposed model, a unified cross-modal concatenate st structure (uccst) introduces the teacher-student learning with kullback-leibler divergence (kl) regularization to transfer knowledge from cross-modal translation model to two subtasks -st and mt. our main contributions can be summarized. (1) compared with the previous st frameworks which can only utilize one single modality text in mt teacher, we design a unified framework that can use both input information of the two modalities simultaneously by concatenating speech and text. ## conclusion in this paper, we designed a unified st framework. compared with the previous st frameworks which can only utilize one single modality text in mt teacher, our method can use both information of the two modalities simultaneously by concatenating speech and text. our st method can better utilize the cross-modal information. experiments show that our method can significantly improve st performance regardless of using the limited ternary data or adding auxiliary external data.