The AISP - SJTU Simultaneous Translation System for IWSLT 2022 (2022)

Abstract: This paper describes AISP-SJTU’s submissions for the IWSLT 2022 Simultaneous Translation task. We participate in the text-to-text and speech-to-text simultaneous translation from English to Mandarin Chinese. The training of the CAAT is improved by training across multiple values of right context window size, which achieves good online performance without setting a prior right context window size for training. For speech-to-text task, the best model we submitted achieves 25.87, 26.21, 26.45 BLEU in low, medium and high regimes on tst-COMMON, corresponding to 27.94, 28.31, 28.43 BLEU in text-to-text task.

## introduction this paper describes the systems submitted by ai speech co., ltd. (aisp) and shanghai jiaotong university (sjtu) for iwslt 2022 simultaneous translation task. two speech translation systems including cascade and end-to-end (e2e) for the simultaneous speech translation track, and a simultaneous neural machine translation (mt) system for the text-to-text simultaneous translation track. the systems are focused on english to mandarin chinese language pair. for simultaneous speech translation, recent work tends to fall into two categories, cascaded systems and e2e systems. and the cascaded system often outperforms the fully e2e approach. only one work @xcite shows that the e2e model can achieve better results than the cascaded model. in their work they introduce pre-training @xcite @xcite and data augmentation techniques @xcite to e2e models. therefore, in this paper, we hope to optimize the speech translation model from two aspects. first, we aim to build a robust cascade model and learn best practices from wmt evaluation activities @xcite @xcite , such as back translation @xcite @xcite . second, we explore various selfsupervised learning methods and introduce as much semi-supervised data as possible towards finding the best practice of training cascaded speech-totext (s2t) models. in our settings, asr data, mt data, and monolingual text data are all considered in a progressively training framework. we only trained one e2e model, and its bleu is 22.49 with 1272 al. due to the huge difference in the scale of training data from the cascaded model, e2e performance is far lower than that of the latter. the cascaded s2t final performance on the must-c v2 test set is 25.87, 26.21, 26.45 bleu with low, medium and high regimes. in addition, we also participate in the simultaneous text-to-text (t2t) task. our system is based on an efficient wait-k model @xcite and caat model @xcite . we investigate large-scale knowledge distillation @xcite and back translation methods. specially, we develop a multi-path training strategy, which enables a unified model serving different wait-k paths. all mt models are based on transformer @xcite . the organizers use the output of a streaming asr system as input to the text-to-text system, and the results will be shown in the overview paper @xcite . the rest of this paper is organized as follows. section 2 describes the details of the data preprocessing and augmentation. section 3 describes the models used in our system and introduces details of the model structure and techniques used in training and inference. we present experimental results in section 4 and related works in section 5. finally, the conclusion is given in section 6. 2 data preprocessing and augmentation ## experiments in our experiments, pre-norm transformerbase @xcite is used as the offline baseline model to compare with the text-to-text models. the baseline model has 12 encoder layers and 6 decoder layers and it is trained only using bilingual data. we compare the baseline model with three text-to-text models: wait-k @xcite , efficient wait-k and dynamic-caat. for speech-to-text task, we compare the results of asr cascaded dynamic-caat and efficient wait-k respectively. the details of models are summarized in systems are evaluated with respect to quality and latency. quality is evaluated with the standard bleu metric @xcite . latency is evaluated with metric average lagging (al), which is extended to the task of simultaneous speech translation from simultaneous machine translation @xcite . we conduct all our experiments using simuleval toolkit @xcite and report results for the submitted speech translation tasks. latest 6 checkpoints of a single training process are averaged in our experiments. we also adopted fp16 mix-precision training to accelerate the training process with almost no loss in bleu. all models are trained on 8 rtx a10 gpus. all translation systems are followed by a post-processing module for chinese punctuation. ## conclusion this paper summarizes the results of the shared tasks in the iwslt 2022 produced by the aisp-sjtu team. in this paper, dynamic-caat we used outperforms efficient wait-k, and its result is close to offline model in the case of @xmath0. from the experiments we also can see that the pre-trained language model plays a most important role in both asr and e2e translation. because of the huge difference in the amount of data, the performance of the e2e system is much lower than that of cascaded system. in the future, we hope to explore more effective data augmentation experiments applied to e2e translation. we hope that our practice can facilitate research work and industrial applications.