An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation (2024)

Abstract: Consistency regularization methods, such as R-Drop (Liang et al., 2021) and CrossConST (Gao et al., 2023), have achieved impressive supervised and zero-shot performance in the neural machine translation (NMT) field. Can we also boost end-to-end (E2E) speech-to-text translation (ST) by leveraging consistency regularization? In this paper, we conduct empirical studies on intra-modal and cross-modal consistency and propose two training strategies, SimRegCR and SimZeroCR, for E2E ST in regular and zero-shot scenarios. Experiments on the MuST-C benchmark show that our approaches achieve state-of-the-art (SOTA) performance in most translation directions. The analyses prove that regularization brought by the intra-modal consistency, instead of the modality gap, is crucial for the regular E2E ST, and the cross-modal consistency could close the modality gap and boost the zero-shot E2E ST performance.

## introduction speech-to-text translation takes acoustic speech signals as input and outputs text translations in the target language. the conventional cascaded st system consists of an automatic speech recognition (asr) system and a machine translation (mt) module in a pipeline manner @xcite @xcite . recent works on st have focused on the end-to-end system, which learns a unified model that directly generates text translations from speech without any intermediate outputs @xcite . e2e st is a cross-modal task, where the major challenges include parallel st data scarcity and representation discrepancy between speech and text modalities. in order to boost e2e st training, the techniques utilized by existing approaches include pretraining @xcite , multi-task learning @xcite , knowledge distillation @xcite , and cross-modal representation learning @xcite @xcite . however, most methods are far from being widely used due to the sophisticated model architecture, complicated algorithm implementation, and tedious hyperparameter search. consistency regularization has been widely adopted and shown great promise to improve nmt performance @xcite @xcite @xcite . specifically, @xcite introduce an intra-lingual consistency, r-drop, to regularize dropout and improve the supervised nmt performance, and @xcite propose a cross-lingual consistency, crossconst, to learn universal representations and boost the zero-shot nmt performance. given the similar problem formulations between nmt and e2e st, a natural question arises: can we significantly improve e2e st performance by leveraging simple consistency regularization? in this paper, our primary goal is to provide a simple, easy-to-reproduce, but tough-to-beat strategy for learning e2e st models. inspired by @xcite and @xcite , we propose two strategies, simregcr and simzerocr, for training e2e st models in regular and zero-shot scenarios. we show that intra-modal consistency is crucial for the regular setting, and cross-modal consistency is the key to closing the modality gap and boosting the zero-shot performance. the contributions of this paper can be summarized as follows: ## methodology in this section, we formally propose simregcr and simzerocr, the consistency-based strategies for learning e2e st models in regular (section 4.1) and zero-shot (section 4.2) scenarios respectively. we introduce the details of each part below. ## conclusion in this paper, we propose two simple but effective consistency regularization based strategies for learning e2e st models. we analyze the regularization effect of simregcr on the regular e2e st performance and show that simzerocr could effectively close the modality gap. experiments on the must-c benchmark demonstrate the capabilities of our approaches to improve translation performance in both regular and zero-shot settings. given the universality and simplicity of simregcr and simzerocr, we believe they can serve as strong baselines for future e2e st research. for future work, we will explore the effectiveness of consistency regularization on more speech related tasks, such as speech-to-speech translation, speech language modeling, etc.