ESP net How2 Speech Translation System for IWSLT 2019: Pre-training, Knowledge Distillation, and Going Deeper (2019)

Abstract: This paper describes the ESPnet submissions to the How2 Speech Translation task at IWSLT2019. In this year, we mainly build our systems based on Transformer architectures in all tasks and focus on the end-to-end speech translation (E2E-ST). We first compare RNN-based models and Transformer, and then confirm Transformer models significantly and consistently outperform RNN models in all tasks and corpora. Next, we investigate pre-training of E2E-ST models with the ASR and MT tasks. On top of the pre-training, we further explore knowledge distillation from the NMT model and the deeper speech encoder, and confirm drastic improvements over the baseline model. All of our codes are publicly available in ESPnet.

## introduction this paper presents the espnet-st group's @xmath0. espnet @xcite has been widely used for many speech applications; automatic speech recognition (asr), textto-speech @xcite , speech translation @xcite , machine translation (mt), and speech separation/enhancement @xcite . the purpose of this submission is not only to show the recent progress on st researches, but * *equal contribution also to encourage future research by building strong systems along with the open-sourced project. this year we focused on (1) sequence-level knowledge distillation (seqkd) @xcite , (2) conformer encoder @xcite , (3) multi-decoder architecture @xcite , (4) model ensembling, and (5) better segmentation with a neural network-based voice activity (vad) system @xcite and a novel algorithm to merge multiple short segments for long context modeling. our primary focus was e2e models, although we also compared them with cascade systems with our best effort. all experiments were conducted with the espnet-st toolkit @xcite , and the recipe is publicly available at https://github.com/espnet/ espnet/tree/master/egs/iwslt21 . ## data preparation in this section, we describe data preparation for each task. the corpus statistics are listed in ## experimental setting in this section, we describe the experimental setting for each task. the detailed configurations for each task are summarized in ## conclusion in this paper, we have presented the espnet-st group's offline systems on the iwslt 2021 submission. we significantly improved the baseline conformer performance with multi-referenced seqkd, multi-decoder architecture, segment merging algorithm, and model ensembling. our future work includes scaling training data and careful analysis of the performance gap in different test sets.