Mutual-Learning Improves End-to-End Speech Translation (2021)

Abstract: A currently popular research area in end-to-end speech translation is the use of knowledge distillation from a machine translation (MT) task to improve the speech translation (ST) task. However, such scenario obviously only allows one way transfer, which is limited by the performance of the teacher model. Therefore, We hypothesis that the knowledge distillation-based approaches are sub-optimal. In this paper, we propose an alternative–a trainable mutual-learning scenario, where the MT and the ST models are collaboratively trained and are considered as peers, rather than teacher/student. This allows us to improve the performance of end-to-end ST more effectively than with a teacher-student paradigm. As a side benefit, performance of the MT model also improves. Experimental results show that in our mutual-learning scenario, models can effectively utilise the auxiliary information from peer models and achieve compelling results on Must-C dataset.

## introduction speech translation (st) aims to translate speech signals into a foreign language. it is a multi-modality task, closely related to automatic speech recognition (asr) and machine translation (mt). st has a wide range of applications, such as video subtitling @xcite , real-time lecture translation @xcite , and protection of endangered languages @xcite . despite the recent success in end-to-end (e2e) models, currently such systems still face the issue of labelled training data insufficiency @xcite . a recent popular advance in e2e st is the use of knowledge distillation (kd), which can provide an effective paradigm for transferring knowledge from rich-resource to low-resource tasks @xcite . under such paradigm the mt model is considered a teacher that guides the st model, considered a student learning from the teacher. we pose that oneway knowledge transfer in a strict teacher-student relationship maybe sub-optimal for the following reasons: 1. since the mt model is frozen in this one-way knowledge transfer scenario, the success of knowledge transfer and hence the performance of the st task is constrained by the performance of the pre-trained mt model; 2. there is a modality gap between speech and text inputs of the two models, with speech input also containing inherent speaker variability. motivated to address the issues mentioned above, we set out to improve st and mt tasks by training them jointly. instead of freezing teacher model, we introduce a mutual-learning paradigm, which regards st and mt models as peers that learn collaboratively, aiming to iteratively learn and share the knowledge between the two models. originally, mutual-learning was proposed to leverage information from multiple models and allow effective dual knowledge transfer in image processing tasks @xcite . we leverage this idea and adapt it to sequence tasks. our main contributions are: 1. we propose a jointly-trainable mutual-learning paradigm, which improves the distillation method by training together. the search space of mt and st are both enlarged, providing the potential for a more robust local optima. 2. we further improve our mutuallearning method by integrating cyclical annealing schedule, which alleviates the kl vanishing problem suffered by many time-series tasks @xcite @xcite . 3. we implement extensive experiments on must-c en-fr, en-es datasets and illustrate the advantage of our model by empirically comparing with a cascaded model, a knowledge distillation (kd) model and a multi-task learning (mtl) model. the ex-perimental results show our model can effectively leverage the transcript and the auxiliary mt task, and we provide competitive results in all experiments. in addition, as a side benefit, the performance of the mt model also improves. e2e models consist of an encoder that encodes speech input as an intermediate representation, and a decoder that decodes this intermediate representation to a probability distribution over the target text feature space. in the past, the encoder and decoder were based on recurrent neural network architecture, but most recent work utilises transformerbased architecture @xcite @xcite @xcite . ## conclusion we proposed a mutual-learning paradigm for endto-end speech translation to effectively transfer knowledge between st and mt models. experimental results demonstrate that our proposed approach outperforms knowledge distillation, the typical one-way transfer paradigm, as well as, multitask learning, a typical dual knowledge transfer paradigm. we also provide a competitive result compared to a cascaded model, which has thus far been outperforming e2e st models.