A conventional approach to improving the performance of end-to-end speech translation ( E2E-ST ) models is to leverage the source transcription via pre-training and joint training with automatic speech recognition ( ASR ) and neural machine translation ( NMT ) tasks . However , since the input modalities are different , it is difficult to leverage source language text successfully . In this work , we focus on sequencelevel knowledge distillation ( SeqKD ) from external text-based NMT models . To leverage the full potential of the source language information , we propose backward SeqKD , SeqKD from a target-to-source backward NMT model . To this end , we train a bilingual E2E-ST model to predict paraphrased transcriptions as an auxiliary task with a single decoder . The paraphrases are generated from the translations in bitext via back-translation . We further propose bidirectional SeqKD in which SeqKD from both forward and backward NMT models is combined . Experimental evaluations on both autoregressive and non-autoregressive models show that SeqKD in each direction consistently improves the translation performance , and the effectiveness is complementary regardless of the model capacity . End-to-end speech translation ( E2E-ST ) ( Bérard et al . , 2016 ) , which aims to convert source speech to text in another language directly , is an active research area . Because direct ST is a more difficult task than automatic speech recognition ( ASR ) and machine translation ( MT ) , various techniques have been proposed to ease the training process by using source transcription . Examples include pretraining ( Bérard et al . , 2018 ; Wang et al . , 2020c ; Bansal et al . , 2019 ; Wang et al . , 2020d ) , multi-task learning ( Weiss et al . , 2017 ; Bérard et al . , 2018 ; Bahar et al . , 2019 ) , knowledge distillation ( Liu et al . , 2019 ) , meta-learning ( Indurthi et al . , 2020 ) , twopass decoding ( Anastasopoulos and Chiang , 2018 ; Sperber et al . , 2019 ) , and interactive decoding ( Liu et al . , 2020 ; Le et al . , 2020 ) . However , as input modalities between ST and MT tasks are different , an auxiliary MT task is not always helpful , especially when additional bitext is not available ( Bahar et al . , 2019 ) . Moreover , because monotonic speech-to-transcription alignments encourage the ASR task to see surface-level local information , an auxiliary ASR task helps the E2E-ST model to extract acoustic representations , not semantic ones , from speech . Sequence-level knowledge distillation ( Se-qKD ) ( Kim and Rush , 2016 ) is another approach to transferring knowledge from one model to another . Recent studies have shown that SeqKD has the effect of reducing the complexity of training data and thus eases the training of student models , e.g. , non-autoregressive ( NAR ) models ( Gu et al . , 2018 ; Zhou et al . , 2019a ; Ren et al . , 2020 ) . Paraphrasing , which represents text in a different form but with the same meaning , can also be regarded as SeqKD when using neural paraphrasing via back-translation ( Mallinson et al . , 2017 ; Wieting et al . , 2017 ; Federmann et al . , 2019 ) . It has been studied to improve the reference diversity for MT system evaluations ( Thompson and Post , 2020 ; Bawden et al . , 2020a , b ) and the performance of low-resource neural MT ( NMT ) models ( Zhou et al . , 2019b ; Khayrallah et al . , 2020 ) . In this work , due to its simplicity and effectiveness , we focus on SeqKD from text-based NMT models to improve the performance of a bilingual E2E-ST model . In order to fully leverage source language information , we propose backward Se-qKD , which targets paraphrased source transcriptions generated from a target-to-source backward NMT model as an auxiliary task . Then , a single ST decoder is trained to predict both source and target language text as in a multilingual setting ( Inaguma et al . , 2019 ) . This way , the decoder is biased to capture semantic representations from speech , un-like joint training with an auxiliary ASR task . We also propose bidirectional SeqKD , which combines SeqKD from two NMT models in both language directions . Therefore , the E2E-ST models can fully exploit the knowledge embedded in both forward and backward NMT models . Experimental evaluations demonstrate that Se-qKD from each direction consistently improves the translation performance of both autoregressive and non-autoregressive E2E-ST models . We also confirm that bidirectional SeqKD outperforms unidirectional SeqKD and that the effectiveness is maintained in large models . To fully leverage knowledge in both source and target language directions for bilingual E2E-ST models , we have proposed bidirectional SeqKD , in which both forward SeqKD from a source-to-target NMT model and backward SeqKD from a target-tosource NMT model are combined . Backward Se-qKD is performed by targeting source paraphrases generated via back-translation from the original translations in bitext . Then , the E2E-ST model is enhanced by training to generate both source and target language text with a single decoder . We experimentally confirmed that SeqKD from each direction boosted the translation performance of both autoregressive and non-autoregressive E2E-ST models , and the effectiveness was additive . Multi-referenced training with the original and distilled text gave further gains . We also showed that bidirectional SeqKD was effective regardless of model sizes .

Challenge: Existing works on end-to-end speech recognition models use the source transcriptions for performance improvements but it is challenging due to the modality gap.
Approach: They propose a bidirectional sequence knowledge distillation which learns from text-based NMT systems with a single decoder to enhance the model to capture semantic representations.
Outcome: Evaluations on autoregressive and non-autoregressive models show that the proposed method improves in both directions and the results are consistent in different model sizes.