The AISP - SJTU Translation System for WMT 2022 (2022)

Abstract: This paper describes AISP-SJTU’s participation in WMT 2022 shared general MT task. In this shared task, we participated in four translation directions: English-Chinese, Chinese-English, English-Japanese and Japanese-English. Our systems are based on the Transformer architecture with several novel and effective variants, including network depth and internal structure. In our experiments, we employ data filtering, large-scale back-translation, knowledge distillation, forward-translation, iterative in-domain knowledge finetune and model ensemble. The constrained systems achieve 48.8, 29.7, 39.3 and 22.0 case-sensitive BLEU scores on EN-ZH, ZH-EN, EN-JA and JA-EN, respectively.

## introduction we participate in the wmt 2022 shared general mt task, including english↔chinese(en↔zh) and english↔japanese(en↔ja). all of our systems are built with constrained data sets. for model architectures, we exploit several transformer variants including transformer-dlcl @xcite , transformer-ode (bei li, 2021), transformer-rpr @xcite , transformer-coda @xcite . in this year's translation tasks, we mainly employ data filtering @xcite , large-scale back-translation @xcite , knowledge distillation, forward-translation, in-domain knowledge finetune and model ensemble to improve the final model's performance. for the synthetic data generation, we first exploit large-scale back-translation @xcite method to leverage the target-side monolingual data and the knowledge distillation @xcite to leverage the source-side of bilingual data. to use the source-side monolingual data, we explore forward-translation by ensemble models to get general domain synthetic data.furthermore, several data augmentation methods are applied to improve the model robustness, including different token-level noise and different sampling methods. we mainly use three training strategies in the training phase, including the warmup strategy @xcite to adjust the learning rate in training, different sampling methods @xcite and the graduated label smoothing @xcite . in the fine-tuning stage, the test set is clustered into seven categories, and then use the tfidf-ngram algorithm @xcite to search for similar bilingual and monolingual data in all data according to these seven domains. the monolingual data is then generated using forward translation to generate pseudo-data, and finally fine-tuned together with the searched bilingual data. we pay more attention to the differences between different models in this year. we compute self-bleu @xcite from the translations of the models on the valid set to quantify the diversity among different models. to be precise, we use the translation of one model as the hypothesis and the translations of other models as references to calculate an average bleu score. a lower self-bleu means this model is more different from other models. for ensemble method in every category, the self-bleu scores of the models are calculated to represent their differences from other models, and according to the self-bleu scores of the model, the distribution weight when they perform ensemble is calculated through the softmax-temperature @xcite . now seven domain ensemble models are obtained, then use the model for each domain to predict the test set of the corresponding domain separately. this paper is structured as follows: sec. 2 describes the novel model architectures. we introduce our system and training strategy in detail in sec. 3. experimental settings and results are shown in sec. 4. we conduct analytical experiments in sec. 5. finally, we conclude our work in sec. 6. ## analysis to verify the effectiveness of our approach, we conduct analytical experiments on model variants, data augmentation methods, and ensemble strategies in this section. ## conclusion this paper summarizes the results of the shared general mt task in the wmt 2022 produced by the aisp-sjtu team. in this shared task, we participated in four translation directions: @xmath0, @xmath1, @xmath2. we investigate various novel transformer based architectures to build mt systems. our systems are also built on several popular data augmentation methods such as back-translation, knowledge distillation, forward-translation and in-domain finetune. in the future, we hope to explore more efficient model architectures and data augmentation techniques in mt systems. we hope that our practice can facilitate research work and industrial applications.