The Volctrans Neural Speech Translation System for IWSLT 2021 (2021)

Abstract: This paper describes the systems submitted to IWSLT 2021 by the Volctrans team. We participate in the offline speech translation and text-to-text simultaneous translation tracks. For offline speech translation, our best end-to-end model achieves 7.9 BLEU improvements over the benchmark on the MuST-C test set and is even approaching the results of a strong cascade solution. For text-to-text simultaneous translation, we explore the best practice to optimize the wait-k model. As a result, our final submitted systems exceed the benchmark at around 7 BLEU on the same latency regime. We release our code and model to facilitate both future research works and industrial applications.

## introduction this paper describes the neural speech translation systems submitted to iwslt 2021 by the volctrans team (also known as the team from bytedance ai lab), including cascade and end-to-end speech translation (st) systems for the offline st track and a simultaneous neural machine translation (nmt) system. we aim at finding the best practice for these two tracks. for offline st, the cascaded system often outperforms the fully end-to-end approach. recent studies on the fully end-to-end approaches obtain promising results and attract a lot of interest. last year's results have shown that an end-to-end model achieves an even better performance @xcite compared with the cascaded competitors. however, they introduce pre-training @xcite @xcite and data augmentation techniques @xcite to end-toend models, while the cascaded is not that strong 1 code and models are available at https: //github.com/bytedance/neurst/tree/ master/examples/iwslt21 enough. hence, in this paper, we would like to optimize the speech translation model in two aspects. first, we are devoted to building a strong cascade competitor and learns the best practice from wmt evaluation campaigns @xcite , such as back translation @xcite and ensemble. second, we explore various self-supervised learning methods and introduce as much semi-supervised data as possible towards finding the best practice of training endto-end st models. in our settings, asr data, mt data, and monolingual text data are all considered in a progressively training framework. the results are very promising, and the final performance on the must-c test set surpasses the end-to-end baseline by 7.9 blue scores, while it is still lagging behind our cascade model by 1.5 blue scores. it is not surprising since some well-optimized methods for mt can not be easily used on st, such as back translation. however, our experience shows that the external data can effectively close the gap between end-to-end models and cascade models. in parallel, we also participate in the simultaneous nmt track, which translates in real-time. our system is based on an efficient wait-k model @xcite . we investigate large-scale knowledge distillation @xcite and back translation methods. specially, we develop a multi-path training strategy, which enables a unified model serving different wait-k paths. our target is to obtain the best translation quality at different latency levels. the remaining part of the paper proceeds as follows. section 2 and section 3 describe our cascade and end-to-end systems respectively. section 4 presents the implementation of simultaneous nmt models. each section starts from the training sources and how we synthesize large-scale data. and then, we give details about the model structure and techniques for training and inference. we con- ## end-to-end speech translation recent studies show that the fully end-to-end solution achieves promising performance when compared with the cascaded models @xcite . this section will introduce how we build our end-to-end models for the offline st task. ## simultaneous translation this section describes our submissions to the text-to-text simultaneous speech translation track for english to german (en2de) and english to japanese (en2ja). for versatility, we adopt identical methods for these two language pairs. ## experimental results we conduct all our experiments using neurst @xcite and report results for the submitted speech translation tasks in this section. it is worth noting that all transcripts and translations in the test sets are removed from the training data. when evaluating the offline st models, tags such as applause and laughing are removed from both hypothesis and reference. we use word error rate (wer) to evaluate the asr model and report case-sensitive detokenized bleu foot_7 for mt. no other data segmentation techniques are applied to the dev/test sets. results on must-c dev and tst-common, as well as dev(v1) and tst-common(v1) from @xcite are listed together, which serve as strong baselines for comparison purpose in the end-to-end speech translation field. when evaluating the simultaneous translation, we use the official simuleval @xcite toolkit and report case-sensitive detokenized bleu @xcite and average lagging @xcite ## final results ## conclusion this paper summarizes the results of the shared tasks in the iwslt 2021 produced by the volctrans team. we investigate the performance of the end-toend solutions with data augmentation and progressively training framework for the offline st task. our end-to-end approach surpasses the last year's best cascaded system by 1 bleu, but it is still lagging behind our cascade model by 1.5 bleu scores on must-c test sets. however, our endto-end solutions achieve promising performance on tst2020 and tst2021. afterwards, we develop the efficient wait-k model with multi-path training, and large-scale knowledge distillation and back translation methods. the final submitted systems exceed the baseline systems at 7 bleu on the same regime. we see the data augmentation technique plays the most important role in these tasks. in the future, we would like to explore a more extensive data condition on both modality and quantity. we hope our practice could facilitate batch research works and industrial applications.