  A conventional approach to improving the performance of end-to-end speech
translation (E2E-ST) models is to leverage the source transcription via
pre-training and joint training with automatic speech recognition (ASR) and
neural machine translation (NMT) tasks. However, since the input modalities are
different, it is difficult to leverage source language text successfully. In
this work, we focus on sequence-level knowledge distillation (SeqKD) from
external text-based NMT models. To leverage the full potential of the source
language information, we propose backward SeqKD, SeqKD from a target-to-source
backward NMT model. To this end, we train a bilingual E2E-ST model to predict
paraphrased transcriptions as an auxiliary task with a single decoder. The
paraphrases are generated from the translations in bitext via back-translation.
We further propose bidirectional SeqKD in which SeqKD from both forward and
backward NMT models is combined. Experimental evaluations on both
autoregressive and non-autoregressive models show that SeqKD in each direction
consistently improves the translation performance, and the effectiveness is
complementary regardless of the model capacity.
