Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation (2021)

Abstract: A conventional approach to improving the performance of end-to-end speech translation (E2E-ST) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition (ASR) and neural machine translation (NMT) tasks. However, since the input modalities are different, it is difficult to leverage source language text successfully. In this work, we focus on sequence-level knowledge distillation (SeqKD) from external text-based NMT models. To leverage the full potential of the source language information, we propose backward SeqKD, SeqKD from a target-to-source backward NMT model. To this end, we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder. The paraphrases are generated from the translations in bitext via back-translation. We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined. Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance, and the effectiveness is complementary regardless of the model capacity.

## introduction end-to-end speech translation (e2e-st) @xcite , which aims to convert source speech to text in another language directly, is an active research area. because direct st is a more difficult task than automatic speech recognition (asr) and machine translation (mt), various techniques have been proposed to ease the training process by using source transcription. examples include pretraining @xcite @xcite , multi-task learning @xcite @xcite , knowledge distillation @xcite , meta-learning @xcite , twopass decoding @xcite , and interactive decoding @xcite . however, as input modalities between st and mt tasks are different, an auxiliary mt task is not always helpful, especially when additional bitext is not available @xcite . moreover, because monotonic speech-to-transcription alignments encourage the asr task to see surface-level local information, an auxiliary asr task helps the e2e-st model to extract acoustic representations, not semantic ones, from speech. sequence-level knowledge distillation (se-qkd) @xcite is another approach to transferring knowledge from one model to another. recent studies have shown that seqkd has the effect of reducing the complexity of training data and thus eases the training of student models, e.g., non-autoregressive (nar) models @xcite @xcite . paraphrasing, which represents text in a different form but with the same meaning, can also be regarded as seqkd when using neural paraphrasing via back-translation @xcite @xcite . it has been studied to improve the reference diversity for mt system evaluations @xcite and the performance of low-resource neural mt (nmt) models @xcite . in this work, due to its simplicity and effectiveness, we focus on seqkd from text-based nmt models to improve the performance of a bilingual e2e-st model. in order to fully leverage source language information, we propose backward se-qkd, which targets paraphrased source transcriptions generated from a target-to-source backward nmt model as an auxiliary task. then, a single st decoder is trained to predict both source and target language text as in a multilingual setting @xcite . this way, the decoder is biased to capture semantic representations from speech, un-like joint training with an auxiliary asr task. we also propose bidirectional seqkd, which combines seqkd from two nmt models in both language directions. therefore, the e2e-st models can fully exploit the knowledge embedded in both forward and backward nmt models. experimental evaluations demonstrate that se-qkd from each direction consistently improves the translation performance of both autoregressive and non-autoregressive e2e-st models. we also confirm that bidirectional seqkd outperforms unidirectional seqkd and that the effectiveness is maintained in large models. ## method in this section, we propose bidirectional seqkd from both forward and backward nmt models that leverages machine-generated source paraphrases as another target in addition to the distilled translation to enhance the training of a bilingual e2e-st model. let x denote input speech features in a source language and y s and y t denote the corresponding gold transcription and translation, respectively. let d @xmath0, y s i , y t i )} i @xmath1, and ## experimental setting data we used must-c en-de (408 hours) and en-fr (492 hours) datasets @xcite . both language pairs consist of a triplet of (x, y s , y t ). we performed the same data preprocessing as @xcite (see details in appendix a.1). we report case-sensitive detokenized bleu scores @xcite on the tst-common set with the multi-bleu-detok.perl script in moses @xcite . ## conclusion to fully leverage knowledge in both source and target language directions for bilingual e2e-st models, we have proposed bidirectional seqkd, in which both forward seqkd from a source-to-target nmt model and backward seqkd from a target-tosource nmt model are combined. backward se-qkd is performed by targeting source paraphrases generated via back-translation from the original translations in bitext. then, the e2e-st model is enhanced by training to generate both source and target language text with a single decoder. we experimentally confirmed that seqkd from each direction boosted the translation performance of both autoregressive and non-autoregressive e2e-st models, and the effectiveness was additive. multi-referenced training with the original and distilled text gave further gains. we also showed that bidirectional seqkd was effective regardless of model sizes.