Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023)

Abstract: Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. However, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, we propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation. We conduct experiments on the MuST-C dataset. The results demonstrate that our method attains state-of-the-art results. Moreover, when additional data is used, we achieve the new SOTA result on MuST-C English to Spanish task with 20.8% of the training time required by the current SOTA method.

## introduction end-to-end (e2e) models have made significant strides in the artificial intelligence realm, especially in speech translation (st). these models have low latency and less error propagation by providing direct translations from speech inputs @xcite . this approach contrasts with traditional pipeline models that rely on separate automatic speech recognition (asr) and machine translation (mt) systems. however, the e2e model's single-model design poses new challenges due to its need for cross-modal and cross-lingual transfers @xcite . to address this, recent studies have utilized multi-task learning (mtl), leveraging cross-modal or crosslingual training objectives for pre-training or joint training @xcite @xcite . this technique assures good convergence of the models and emphasizes the importance of the auxiliary loss, offering a brand-new perspective for further advancements in e2e st @xcite . however, further exploration is necessary to determine how and to what extent these auxiliary tasks aid the final st model. notably, not all auxiliary tasks in the fine-tuning stage are beneficial. this inconsistency arises because mtl is typically viewed as a multi-objective optimization problem, often resulting in training trade-offs when objectives conflict @xcite . the ideal mtl outcome is to achieve pareto optimality @xcite , indicating solutions are superior to any alternatives. since mtl does not ensure optimal performance for the st task, fine-tuning is crucial to overcome this shortcoming. some studies even underscore a critical conflict between the st task and asr and mt tasks @xcite , necessitating a fine-tuning strategy. so answering these questions is crucial to designing an optimal mtl strategy for the st task. in this paper, we rethink task consistency in mtl and introduce a gradient-based consistency metric, which denotes the consistency of the gradient direction between the st task and other auxiliary tasks. our analysis shows that 1) asr aids the acoustic encoder and mt facilitates the textual encoder in audio-to-text transfer, 2) length inconsistency hinders aligning the representations of the two modalities, 3) disparity between noisy speech features and clean text embeddings as considerable obstacles, and 4) the timing and degree of task influence exhibit significant variation. inspired by the aforementioned observations, we relax the asr task that only uses it to help the acoustic encoder. we propose the looking-back mechanism (lbm) to overcome length consistency. it can significantly shrink the speech length without information loss. to bridge the modality gap, we introduce the local-to-global (l2g) training strategy. by incorporating speech-like noise into the text and utilizing an l2g extractor, we enhance contextual information at each layer. this method effectively guides the interaction between audio and text sequences and aligns with audio processing requirements. we further propose a task-normbased weight decrease method to speed up training, adjusting task weights based on auxiliary task influence and timing, thus avoiding unnecessary training costs. we test our method on must-c 8 language datasets. the results show our method can achieve comparable performance with sota work without fine-tuning on st task. we then set the new sota by utilizing the knowledge distillation method @xcite . with additional data, we achieve comparable performance with that of using only 12.5% ∼ 33.3% training cost compared with the sota work foot_0 . ## task consistency quantification in this section we investigate the consistency issue of multi-task learning on three tasks. we randomly sample n samples as analysis set @xmath0, x, y)}. here s, x, y denote the speech, transcription and translation sequences respectively. we then build the asr set d @xmath1, x)} and mt set d @xmath2, y)}. by inputting the same data (s, x, y) into the model, we observe that different training tasks yield distinct parameter gradients. the losses of different tasks are given by: ## method we propose the improved mtl (called imtl) method from three perspectives: 1) denoising the st sequence, 2) adding noising to mt, and 3) and improving training efficiency. the overall training objective is: the cl denotes contrastive learning and we set the w c to 0.3. based on the above analysis, we use the ctc loss as the asr task. ## conclusion most advanced st methods heavily rely on multitask learning, but few studies focus on the relationship between auxiliary tasks and the st task itself. in this study, we design a gradient consistency metric to analyze the impact of other tasks on the st task during the multi-task learning process. based on our analysis, we propose improved methods that address three key aspects: length, representation, and training efficiency. experimental results on the must-c dataset demonstrate that our approach achieves state-of-the-art performance and significantly improves training efficiency. π(•, •) and apply average pooling to the two sequence representations. the variable τ is a scaler that controls the sharpness of the function output, and in this case, we set τ to 0.1. for each speech input s i , we have its corresponding labeled transcription x i , which forms a positive sample (s i , x i ). additionally, we utilize transcriptions other than x i (denoted as x j for j ̸ = i) to create negative samples.