NT 5 at WMT 2022 General Translation Task (2022)

Abstract: This paper describes the NTT-Tohoku-TokyoTech-RIKEN (NT5) team’s submission system for the WMT’22 general translation task. This year, we focused on the English-to-Japanese and Japanese-to-English translation tracks. Our submission system consists of an ensemble of Transformer models with several extensions. We also applied data augmentation and selection techniques to obtain potentially effective training data for training individual Transformer models in the pre-training and fine-tuning scheme. Additionally, we report our trial of incorporating a reranking module and the reevaluated results of several techniques that have been recently developed and published.

## introduction this paper describes an overview of our submission systems for participating in the wmt 2022 general machine translation tasks. our team, named nt5, is comprised of individuals from four organizations: ntt, tohoku university, tokyo institute of technology, and riken. this year, we focused on bi-directional translation in a single language pair: english-to-japanese (@xmath0. our submission system consists of an ensemble of transformer models @xcite with several recent extensions. we also applied data augmentation and selection techniques to obtain poten-figure 1 : system overview tially effective training data for training individual transformer models in the pre-training/fine-tuning scheme. the models were first trained with a large but possibly noisy parallel corpus for pre-training and then with a small but clean parallel corpus for fine-tuning. additionally, we report our trial of incorporating a reranking module that rescores the n-best lists based on source-to-target, target-tosource, and masked language models. the following section briefly provides an overview of our entire system and each module in more depth. ## system overview figure 1 shows an overview of our system. our submissions are for the constrained track, which only uses parallel and monolingual data that are provided by the wmt shared-task organizers. we selected transformer models @xcite as our base translation model and chose a two-step training strategy: pre-training and finetuning schemes. we first constructed datasets for the pre-training and fine-tuning. the pre-training dataset must be as large as possible, even if the data are noisy @xcite . we first trained the transformer models using only the provided bitext datasets for both translation directions: @xmath1. we refer to these first trained models as initial models. we then generated synthetic datasets for both directions through back-translation @xcite , i.e., translating target-side monolingual data using the initial model in the reverse translation direction. the fine-tuning dataset must be as clean as possible, even if it is relatively small. indeed, in our previous year's submission @xcite , we adapted the models to a news domain in the finetuning phase and drastically improved the translation quality. however, this year's task focused on a general domain, i.e., a test set that consisted of sentences from multiple domains. thus, adaptation by fine-tuning is much more challenging. we tested and combined data selection methods based on sentence embeddings and language models for obtaining fine-tuning data. this process can be viewed as selecting domain adaptation data. by using these datasets, we pre-trained the transformers with pre-training configurations and finetuned the pre-trained models with the fine-tuning configurations described in our system also uses a reranking module. we generated the ten best translation lists as reranking candidates using an ensemble of transformer models. then we selected the best translations based on the weighted sum of the likelihoods obtained from the source-to-target and target-to-source translation models and the masked language models. corpus w/o filtering w/filtering jparacrawl v3.0 25.7 m 25.0 m wikimatrix 3.89 m 3.64 m jesc 2.80 m 2.57 m wiki titles v3 757 k 327 k kftt 440 k 371 k ted talks 242 k 224 k newscommentary v16 1.9 k 1.8 k ## results ## conclusion we described the submission of our joint team @xcite to the wmt'22 general translation task. we participated in the en↔ja translation. our system mainly consists of an ensemble of transformer models with several recent extensions. we also applied data augmentation and selection techniques to train individual transformer models in our pre-training/finetuning training scheme.