The USTC - NELSLIP Offline Speech Translation Systems for IWSLT 2022 (2022)

Abstract: This paper describes USTC-NELSLIP’s submissions to the IWSLT 2022 Offline Speech Translation task, including speech translation of talks from English to German, English to Chinese and English to Japanese. We describe both cascaded architectures and end-to-end models which can directly translate source speech into target text. In the cascaded condition, we investigate the effectiveness of different model architectures with robust training and achieve 2.72 BLEU improvements over last year’s optimal system on MuST-C English-German test set. In the end-to-end condition, we build models based on Transformer and Conformer architectures, achieving 2.26 BLEU improvements over last year’s optimal end-to-end system. The end-to-end system has obtained promising results, but it is still lagging behind our cascaded models.

## introduction this paper describes the submission to iwslt 2022 offline speech translation task by national engineering laboratory for speech and language information processing (nelslip), university of science and technology of china, china. for years, spoken language translation (slt) has been addressed by cascading an automatic speech recognition (asr) and a machine translation (mt) system. the asr system processes source speech into source text and the mt system translates asr output into text in target language independently. recent trends rely on using a single neural network to directly translate the speech in source language into the text in target language without intermediate symbolic representations. the end-to-end paradigm shows an enormous potential to overcome some of the cascaded systems' problems, such as higher architectural complexity and error propagation @xcite @xcite . last year's results of iwslt 2021 have confirmed that the performance of end-to-end models is approaching the results of cascaded solutions. the best end-to-end submission (under the same segmentation and training data conditions) is 2 bleu points (22.6 vs 24.6) below the top-ranked system @xcite . in this work, we build machine translation systems with techniques like back translation @xcite , domain adaptation and model ensemble, which have been proved to be effective practices in iwslt and wmt @xcite . besides, we further improve cascaded speech translation system performance with methods of self-training @xcite @xcite , speech synthesis @xcite , supervised hybrid audio segmentation (shas) @xcite , etc. in end-to-end condition, we initialize the encoder with the corresponding component of asr models and the decoder with that of mt models respectively @xcite . methods used in cascaded systems and as much semi-supervised data as possible are utilized to improve end-to-end models' performance. furthermore, we try to obtain a better performance with ensemble of cascaded and end-to-end models, which may accelerate the application of end-to-end models in industrial scenarios. the remaining of the paper proceeds as follows. section 2 describes speech recognition, speechto-text translation (s2t for short) and text-to-text translation (t2t for short) data used in our experiments. section 3 and section 4 present our cascaded and end-to-end systems respectively, where the details about model architectures and techniques for training and inference will be described. the experimental settings and final results are shown in section 5. ## end-to-end speech translation as regards model architecture, we investigate 4 variants in end-to-end speech translation. the decoder is 6 layers of transformer with embedding size of 1024, attention head of 16 and ffn size of 8192. the encoder is vgg-transformer, initialized by asr vgg-transformer architecture. the decoder is 6 layers of transformer, initialized by nmt e15d6 variant. ## experiments all our experiments are conducted using fairseq toolkit @xcite . we use word error rate (wer) to evaluate the asr models and report casesensitive sacrebleu scores for machine translation. results of must-c tst-common (tst-com), iwslt tst2018/tst2019/tst2020 are listed together, which can be compared as baselines for other researchers and participants in the future. we also present results of iwslt 2022 testsets in the appendix. ## conclusion this paper summarizes the results of iwslt 2022 offline speech translation task produced by the ustc-nelslip team. we investigate various model architectures and data augmentation approaches to build strong speech translation systems, both in cascaded condition and end-to-end condition. in our experiments, we demonstrate the effectiveness of back translation, knowledge distillation, domain adaptation, ensemble, elegant segmentation. our end-to-end model surpasses the last year's best system by 2.26 bleu, but it is still lagging behind our cascaded model by an average of 1.73 bleu scores on must-c test sets. as a note for future work, we would like to investigate the effectiveness of speech data augmentation and multi-modal representations in end-to-end speech translation.