Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation (2023)

Abstract: Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in end-to-end speech translation (E2E ST). The commonplace ”modality gap” between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the ”capacity gap”: high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the over-fitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for en-fr on the MuST-C dataset.

## introduction end-to-end speech translation (e2e st) employs a direct model to translate source language speech into target language text, which has low latency and can avoid the "error propagation" problem in traditional cascade methods @xcite . however, compared to automatic speech recognition (asr) and machine translation (mt) models used in cascade methods, e2e st models typically have limited training data @xcite , which can result in sub-optimal performance. transferring the knowledge from the related tasks (e.g. asr and mt) is a widely-used approach for e2e st to achieve optimal performance @xcite . however, the difference between tasks and data makes the transfer process more challenging @xcite . the inconsistency of length and representation between speech and text leads to the "modality gap" figure 1 : the modality gap in the pt-ft paradigm. we decoupled the encoder to speech encoder and text encoder @xcite , and compared the accuracy curve of fine-tuning for different inputs of the text encoder. we find that the modality gap only exists at the initial stage of the fine-tuning. @xcite , which exists in scenarios where the inputs of the model change, such as in the pre-training and fine-tuning (pt-ft) paradigm @xcite or in the multi-task learning (mtl) methods @xcite . thus, the connectionist temporal classification (ctc) @xcite based adapters @xcite have been proposed to transform the original speech output into a text-like sequence. recently, consistency training methods have achieved promising results by using a better branch, such as the mix-up branch @xcite or the text branch @xcite , to promote crossmodal learning and support the original speech output. however, we find that the "modality gap" does not exist throughout the training process in figure 1 . while consistency training methods have some regularization function @xcite to help the model overcome over-fitting and be fully trained. a natural question arises: are modality adaption methods always effective when an e2e st model is fully trained? in this work, we aim to investigate how much of the improvement is due to the modality adaption or regularization methods. to achieve this, we adopt the pt-ft and encoder decouple paradigm and establish a framework that incorporates adjustable modality adaption and consistency training methods. through extensive experiments on the must-c en-de and en-fr benchmarks, we observe that: ## experimental setup datasets and pre-processing. we conducted our experiments on the must-c @xcite dataset for two language directions: en-de and en-fr. the dev set was used for validation, and the tst-common set was used for reporting our results. for training the english asr model, we used the librispeech @xcite dataset. the wmt16 en-de and wmt14 en-fr datasets were used to train the mt models. we preprocessed the speech input by extracting 80-dimensional log-mel filterbank features and removing utterances with more than 3,000 frames. the vocabulary, which has a size of 10k, is shared between the source and target languages and was trained using the sentencepiece @xcite model from the must-c dataset. ## conclusion through a case study, we have demonstrated that the "modality gap" in the pt-ft paradigm for e2e st is only present in the early stages of fine-tuning. although a modality adaption method can accelerate the convergence speed, it does not significantly improve the final performance of a fully trained model. however, the over-fitting and "capacity gap" are more critical issues in e2e st, and a good regularization technique can help in fully training the model.