Unveiling the Role of Pretraining in Direct Speech Translation (2024)

Abstract: Direct speech-to-text translation systems encounter an important drawback in data scarcity. A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training process. In this study, we compare the training dynamics of a system using a pretrained encoder, the conventional approach, and one trained from scratch. We observe that, throughout the training, the randomly initialized model struggles to incorporate information from the speech inputs for its predictions. Hence, we hypothesize that this issue stems from the difficulty of effectively training an encoder for direct speech translation. While a model trained from scratch needs to learn acoustic and semantic modeling simultaneously, a pretrained one can just focus on the latter. Based on these findings, we propose a subtle change in the decoder cross-attention to integrate source information from earlier steps in training. We show that with this change, the model trained from scratch can achieve comparable performance to the pretrained one, while reducing the training time.

## introduction in recent years, extensive research has been done in the field of speech-to-text translation (st). these models have transitioned from cascaded systems to direct ones @xcite . while this shift helps mitigate error propagation, it introduces other challenges such as scarcity of training data and the need for the model to tackle translation and speech recognition simultaneously. to bypass these issues, a common approach to train direct st systems involves pretraining the encoder on the automatic speech recognition (asr) task @xcite . this enables the encoder to learn acoustic modeling in the source language by leveraging asr data, and the model can focus on semantic modeling during the st training. various studies have been conducted on pretraining for st. @xcite , introduced a method to enhance st performance for lowresource source languages by utilizing asr pretraining from a high-resource language. @xcite enhanced the performance of an st system by pretraining both the encoder and the decoder on asr and mt respectively. @xcite and @xcite proposed variations of asr pretraining that yielded superior results. however, pretraining has some drawbacks too. it has additional data requirements, which can be a problem particularly in languages that don't have a written form and hence no asr data. furthermore, it complicates the training pipeline and worsens the efficiency of the overall training process. recent studies have already questioned the pretraining approach, @xcite , demonstrating that similar results can be achieved under certain conditions without the need for pretraining. however, the authors show that many strategies need to be simultaneously used to achieve this, such as an exhaustive hyperparameter tunning, ctcbased regularization and their proposed parameterized distance penalty. complementing previous interpretability works in st @xcite , in this study, we conduct the first-ever analysis of the training dynamics of a st system, and based on its results, we propose a subtle modification in the transformer @xcite architecture to bypass the pretraining stage. first, we compare the training dynamics of a conventional system that uses a pretrained encoder with one trained from scratch foot_0 . through this analysis, we observe significant disparities in their behaviors. particularly, we note that when making predictions, the model trained from scratch delays the utilization of information extracted by the encoder until a later stage of training. we hypothesize that this delay occurs due to the complexity of the acoustic modeling task, that in this setting needs to be learned together with the semantic modeling. hence, it takes a significant amount of updates to sufficiently train the encoder so that it can extract meaningful information. consequently, the model ignores the encoder outputs and focuses on training the decoder for language modeling. once the encoder can extract valuable representations, the model has already converged towards language modeling and struggles to rely on the information obtained by the encoder. secondly, we believe that by forcing the model to utilize encoder outputs earlier in the training process, the model would not converge towards language modeling and the encoder would be trained more rapidly, leading to higher-quality representations in its outputs. through a modification in the residual connection in the decoder crossattention mechanism, we force the model trained from scratch to integrate source information from earlier training steps, and we observe a comparable performance than in the pretrained one. overall, the main contributions of our work are: (1) the first study of training dynamics in st, that unveils the role of the pretraining, and (2) a modification in the transformer architecture to bypass the pretraining stage. ## training dynamics in speech translation in this section, we analyse how much two training strategies on a transformer-based system foot_1 rely on the speech source when making predictions along a st training. in particular, we study: (1) a standard st system, consisting on an asr pretraining, followed by a re-initialization of the decoder and a training on st data, and (2) a system that only performs a st training on a randomly initialized model. to measure the amount of input information used by the model to generate a prediction we use the source contribution defined by alti+, covered in section 2.1. to generalize this score to a sentence-wise score, we average the source contribution used for the prediction of each token in a sentence. finally, to obtain a score over a test set, we average again the score obtained by every sentence in the set. given that we use a different source contribution measure than @xcite in their work described in section 2.2, we decide to train an additional mt model, to confirm that the three stages described in their work still happen on our setting. for all our analysis, we store a checkpoint every 1k updates during the full training, and every 100 updates during the first 10k foot_2 . for each of the checkpoints, we evaluate the model computing bleu and source contribution scores on english-german must-c dataset @xcite 4 . ## training st from scratch building on our previous analysis, we hypothesize that forcing a speech translation model trained from scratch to utilize source information from the start could enhance the training process. if the model is required to use the encoder's representations, regardless of their quality, poor representations will negatively impact the model's overall performance. this, in turn, will cause a faster training of the encoder to extract better representations. in particular, considering the results in figure 1 , we observe that both the text translation model and the pretrained speech translation model achieve a stable source contribution of approximately 65%. hence, we consider this proportion to be optimal and aim to enforce it in the speech translation model trained from scratch. to test our hypothesis we propose a subtle architecture modification, that forces the transformer to use source information along the full training. our modification focuses on the cross-attention layer of the decoder, which is the step where source and target information is aggregated, with source information coming from the attention block and target-prefix details coming from the residual flow. ## conclusions in this work, we present the first study on the training dynamics of direct st systems, comparing a standard st model with a pretrained encoder to one trained from scratch. the analysis shows that, without pretraining, the model struggles to incorporate information from the encoder's outputs when making predictions. as an explanation, we suggest the encoder needs more updates than in a text task until it can extract valuable representations of the input tokens. once this is achieved, the model has already converged towards language modeling, hence failing to utilize the information extracted by the encoder effectively even in later steps. to address this issue, we propose a subtle modification to the transformer architecture that forces the model to incorporate source information throughout the whole training. by doing so, we achieve comparable performance in a model trained from scratch to one with pretraining, while reducing training time and data requirements.