ESP net- ST IWSLT 2021 Offline Speech Translation System (2021)

Abstract: This paper describes the ESPnet-ST group’s IWSLT 2021 submission in the offline speech translation track. This year we made various efforts on training data, architecture, and audio segmentation. On the data side, we investigated sequence-level knowledge distillation (SeqKD) for end-to-end (E2E) speech translation. Specifically, we used multi-referenced SeqKD from multiple teachers trained on different amounts of bitext. On the architecture side, we adopted the Conformer encoder and the Multi-Decoder architecture, which equips dedicated decoders for speech recognition and translation tasks in a unified encoder-decoder model and enables search in both source and target language spaces during inference. We also significantly improved audio segmentation by using the pyannote.audio toolkit and merging multiple short segments for long context modeling. Experimental evaluations showed that each of them contributed to large improvements in translation performance. Our best E2E system combined all the above techniques with model ensembling and achieved 31.4 BLEU on the 2-ref of tst2021 and 21.2 BLEU and 19.3 BLEU on the two single references of tst2021.

## introduction this paper presents the espnet-st group's @xmath0. espnet @xcite has been widely used for many speech applications; automatic speech recognition (asr), textto-speech @xcite , speech translation @xcite , machine translation (mt), and speech separation/enhancement @xcite . the purpose of this submission is not only to show the recent progress on st researches, but * *equal contribution also to encourage future research by building strong systems along with the open-sourced project. this year we focused on (1) sequence-level knowledge distillation (seqkd) @xcite , (2) conformer encoder @xcite , (3) multi-decoder architecture @xcite , (4) model ensembling, and (5) better segmentation with a neural network-based voice activity (vad) system @xcite and a novel algorithm to merge multiple short segments for long context modeling. our primary focus was e2e models, although we also compared them with cascade systems with our best effort. all experiments were conducted with the espnet-st toolkit @xcite , and the recipe is publicly available at https://github.com/espnet/ espnet/tree/master/egs/iwslt21 . ## data preparation in this section, we describe data preparation for each task. the corpus statistics are listed in ## experimental setting in this section, we describe the experimental setting for each task. the detailed configurations for each task are summarized in ## conclusion in this paper, we have presented the espnet-st group's offline systems on the iwslt 2021 submission. we significantly improved the baseline conformer performance with multi-referenced seqkd, multi-decoder architecture, segment merging algorithm, and model ensembling. our future work includes scaling training data and careful analysis of the performance gap in different test sets.