The N iu T rans End-to-End Speech Translation System for IWSLT 2021 Offline Task (2021)

Abstract: This paper describes the submission of the NiuTrans end-to-end speech translation system for the IWSLT 2021 offline task, which translates from the English audio to German text directly without intermediate transcription. We use the Transformer-based model architecture and enhance it by Conformer, relative position encoding, and stacked acoustic and textual encoding. To augment the training data, the English transcriptions are translated to German translations. Finally, we employ ensemble decoding to integrate the predictions from several models trained with the different datasets. Combining these techniques, we achieve 33.84 BLEU points on the MuST-C En-De test set, which shows the enormous potential of the end-to-end model.

## introduction speech translation (st) aims to learn models that can predict, given some speech in the source language, the translation into the target language. endto-end (e2e) approaches have become popular recently for its ability to free designers from cascading different systems and shorten the pipeline of translation @xcite @xcite . this paper describes the submission of the niutrans e2e st system for the iwslt 2021 @xcite offline task, which translates from the english audio to the german text translation directly without intermediate transcription. our baseline model is based on the dlcl transformer @xcite model with connectionist temporal classification (ctc) @xcite loss on the encoders @xcite . we enhance it with the superior model architecture conformer @xcite , relative position encoding (rpe) @xcite , and stacked acoustic and textual encoding (sate) @xcite . to augment the training data, the english transcriptions of the automatic speech recognition (asr) and speech translation corpora are translated to the german translation. finally, we employ the ensemble decoding method to integrate the predictions from multiple models @xcite trained with the different datasets. this paper is structured as follows. the training data is summarized in section 2, then we describe the model architecture in section 3 and data augmentation in section 4. we present the ensemble decoding method in section 5. the experimental settings and final results are shown in section 6. ## training data our system is built under the constraint condition. the training data can be divided into three categories: asr, mt, and st corpora foot_0 . asr corpora. asr corpora are used to generate synthetic speech translation data. we only use the common voice @xcite and lib-rispeech @xcite corpora. furthermore, we filter the noisy training data in the common voice corpus by force decoding and keep 1 million utterances. mt corpora. machine translation (mt) corpora are used to translate the english transcription. we use the allowed english-german translation data from wmt 2020 @xcite and open-subtitles2018 @xcite . we filter the training bilingual data followed @xcite , which includes length ratio, language detection, and so on. st corpora. the st corpora we used include must-c (gangi et al., 2019) english-german 2 , covost @xcite , speech-translation ted corpus 3 , and europarl-st @xcite . the statistics of the final training data are shown in ## model architecture in this section, we describe the baseline model and the architecture improvements. then, the experimental results are shown to demonstrate the effectiveness. ## data augmentation a large amount of the training data is necessary for a strong neural model. however, unlike the asr and mt tasks, annotated speech-to-translation data is scarce, which prevents well-trained st models. this is the main reason why cascaded systems are the dominant approach in the industrial scenarios. in this section, we describe our data augmentation method. we train a deep dlcl transformer @xcite with the 25 encoder layers on all available mt data. to keep the domain consistency with the original st data, we finetune the mt model on the must-c dataset. the model achieves the sacre-bleu of 35.89 of the must-c tst-common test set. for the case-insensitive librispeech dataset, we train a similar mt model except for lowercasing the source text without punctuation during training. then, we generate the german translation from english transcription in the librispeech and common voice asr datasets. furthermore, sequencelevel knowledge distillation @xcite is applied to augment the training data. we generate the translation of the must-c and speech-translation ted st datasets which are more re-lated to the target domain. corrupting the acoustic feature is another data augmentation method, including specaugment, speed perturbation, and so on. specaugment @xcite is a simple data augmentation applied on the input acoustic features. the time masking and the frequency masking are applied in our systems. speed perturbation transforms the audio by a speed rate, which changes the duration of the audio signal. limited by the size of gpu resources, we do not use this method. compared with the perturbed data, we think the synthetic samples improve the robustness more effectively. all available st corpora are shown in ## ensemble decoding ensemble decoding is an effective method to improve performance by integrating the predictions from multiple models. it has been proved in the wmt competitions @xcite . in our systems, we train multiple st models with different training data for diverse ensemble decoding. the models are chosen based on the performance of the development set. this leads to a significant improvement over a single model. ## conclusion this paper describes the submission of the niu-trans e2e st systems for the iwslt 2021 offline task, which translates the english audio to german translation directly without intermediate transcription. we build our final submissions considering two mainstreams: we also find that the distribution of the training data has a great impact on the performance and alleviate it by ensemble decoding. using the given segmentation, we achieve remarkable improvements over the best end-to-end system of last year.