CKDST : Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation (2023)

Abstract: Distilling knowledge from a high-resource task, e.g., machine translation, is an effective way to alleviate the data scarcity problem of end-to-end speech translation. However, previous works simply use the classical knowledge distillation that does not allow for adequate transfer of knowledge from machine translation. In this paper, we propose a comprehensive knowledge distillation framework for speech translation, CKDST, which is capable of comprehensively and effectively distilling knowledge from machine translation to speech translation from two perspectives: cross-modal contrastive representation distillation and simultaneous decoupled knowledge distillation. In the former, we leverage a contrastive learning objective to optmize the mutual information between speech and text representations for representation distillation in the encoder. In the later, we decouple the non-target class knowledge from target class knowledge for logits distillation in the decoder. Experiments on the MuST-C benchmark dataset demonstrate that our CKDST substantially improves the baseline by 1.2 BLEU on average in all translation directions, and outperforms previous state-of-the-art end-to-end and cascaded speech translation models.

## introduction end-to-end (e2e) speech-to-text translation (st), directly translating speech in one language into text in another, has recently attracted increasing attention @xcite @xcite . compared with traditional cascaded st, e2e st does not require automatic transcription, which endows itself with less error propagation and lower latency. however, parallel st data that consist of speech inputs and target translations, are proverbially limited, especially in comparison with automatic speech recognition (asr) and machine translation (mt) data. in order to mitigate this issue, previous efforts leverage pre-training approaches @xcite and multi-task learning (mtl) frameworks @xcite @xcite to transfer knowledge from asr and/or mt to st. among them, knowledge distillation (kd) @xcite has proved to be an effective way to improve st performance by transferring knowledge from mt to st @xcite @xcite . however, previous kd approaches to st only explore the classical kd that transfers knowledge from prediction logits, which may not allow for sufficient knowledge distillation. specifically, in classical kd @xcite , two types of knowledge are encoded in prediction logits, target class knowledge from target class logits and nontarget class knowledge from non-target class logits. each type of knowledge contributes to the success of classical logits distillation. however, @xcite have found that the classical kd couples the non-target class knowledge with the target class knowledge. such entanglement may inhibit the transfer of non-target class knowledge and limit the performance of logits knowledge distillation. additionally, due to the modality gap between speech and text, it might be difficult for e2e st to sufficiently capture and translate semantic information embedded in speech inputs to target translations. fortunately, however, in mtl-based e2e st, a speech input is accompanied with its transcription that is used as the input fed into mt. such speech and transcription pairs allow us to distill knowledge from transcription representations to speech representations so as to reduce the modality gap. however, such knowledge distillation has not yet been explored for end-to-end speech translation. in order to address these two issues and efficiently distill mt knowledge to st, we propose a comprehensive knowledge distillation framework for st (ckdst). specifically, we propose cross-modal contrastive representation distillation (ccrd) and simultaneous decoupled knowledge distillation (sdkd) as two essential approaches for ckdst, to transfering knowledge from text representations and to performing more sufficient logits distillation. ccrd applies a contrastive training objective to force e2e st to learn speech representations that are closer to their corresponding textual representations. in doing so, we could increase the mutual information lower bound between speech and text representations @xcite . sdkd is proposed for e2e st to mitigate the issue that the classical kd couples the target class knowledge with non-target class knowledge @xcite . for more effectively transferring logits knowledge from mt to st, we decouple these two types of knowledge in prediction logits and extend the decoupled knowledge distillation to the mtl framework where both st and mt are fine-tuned simultaneously. in a nutshell, our contributions are three-fold. end-to-end speech translation. to alleviate the error propagation in cascaded st and to ease the deployment, @xcite and @xcite propose to use an end-to-end architecture to directly translate speech in one language into text in another, without using the intermediate transcriptions. in recent years, increasing efforts have been done in e2e st @xcite @xcite @xcite @xcite . since the parallel speech translation data is notoriously limited, many approaches have been proposed to solve this problem, such as pre-training @xcite @xcite , multi-task learning @xcite @xcite , and data augmentation @xcite . additionally, knowledge distillation from a well trained mt model to a st model has proved effective in improving st performance. @xcite leverage knowledge distillation to allow the e2e st model to learn the same prediction distribution as the mt model. the mt model is frozen while the st model is being trained. sate @xcite uses both pre-trained asr model and mt model as teacher models to perform knowledge distillation. each pre-trained model serves a different module of the st model, and they are also frozen during training. @xcite propose the online-kd that simultaneously update the st module and the mt module in a multi-task learning framework. however, these efforts only distill the knowledge from prediction logits via classical kd, and the knowledge from encoder representations is ignored. in our work, we comprehensively and efficiently distill knowledge from both encoder representations and prediction logits of mt to st. ## ckdst in this section, we first introduce the model architecture of ckdst and then elaborate the two knowledge distillation approaches in ckdst. ## experiments we compared with state-of-the-art e2e/cascaded st models to examine the effectiveness of the proposed ckdst. ## analysis additionally, we conducted a series of in-depth analyses to further investigate how the proposed methods improve e2e st. ## conclusion in this paper, we have presented ckdst, which comprehensively and effectively distills the knowledge of mt to boost the performance of e2e st through two key approaches: ccrd and sdkd. the former leverages a contrastive objective to maximize the mutual information lower bound between speech and text representations for representation knowledge distillation. the later reformulates the classical kd loss to decouple the target class knowledge and the non-target class knowledge for more effective logits knowledge distillation. our experiments strongly demonstrate that our approaches are able to significantly improve e2e st and achieve new state-of-the-art results on the must-c benchmark dataset.