SKIM at WMT 2023 General Translation Task (2023)

Abstract: The SKIM team’s submission used a standard procedure to build ensemble Transformer models, including base-model training, back-translation of base models for data augmentation, and retraining of several final models using back-translated training data. Each final model had its own architecture and configuration, including up to 10.5B parameters, and substituted self- and cross-sublayers in the decoder with a cross+self-attention sub-layer. We selected the best candidate from a large candidate pool, namely 70 translations generated from 13 distinct models for each sentence, using an MBR reranking method using COMET and COMET-QE. We also applied data augmentation and selection techniques to the training data of the Transformer models.

## introduction this paper provides a system description of submissions by our team, called skim 1 , at wmt-2023. we took part in english to japanese (@xmath0. we specifically participated in the constrained track, which places restrictions on the available data and pretrained models. the trial of this year's submissions is a reranking part. our submission system consists of multiple translation models, followed by a reranking module @xcite based on comet @xcite and comet-qe @xcite . this reranking approach serves to identify and select high-quality translations from the hypothesis candidate set generated by multiple translation models. among the transformer-based translation models, we also incorporated a large transformer model with 10.5b parameters. we also applied data augmentation techniques based on our previous year's 1 the team name is an anagram of the first letters of the authors' last names. system @xcite . we briefly describe the system overview, including the experimental results that could not be submitted. ## system overview an overview of our submission system is shown in figure 1 . following the development process used for last year's system @xcite , we used transformer @xcite as the model architecture and conducted pre-training and fine-tuning. in the pre-training phase, we used both a synthetic dataset created by back translation @xcite and the provided bitext dataset. here, we refer to the target-to-source translation model to generate this synthetic dataset as the initial translation model. furthermore, we conducted fine-tuning on the translation models derived from pre-training using high-quality bitext ## primary translation module we trained several transformer models for the reranking in the decoding phase. we describe the details of the models in this section. furthermore, alongside the newly trained models, we reused the primary translation models from the previous year's submission system @xcite . ## reranking to enhance translation quality, we applied a reranking process to the candidate set of hypotheses translated by each model described in section 4. we conducted a comparative analysis of the various methods, as presented in the following sections. ## submission system initially, we planned to submit several versions of the system, with the highest-scoring system selected as the final version. however, the reranking process took longer than expected, and we were unlike the post evaluation setting (section 3.3), these models were fine-tuned using all of the wmt'20 test set, the wmt'20 development set, the wmt'21 test set, and the wmt'22 test set. ## conclusion this paper described our submission system for the constrained track of the wmt'23 general translation task. we developed a translation system for en↔ja. we perform reranking on the candidates generated by multiple translation models, which include a large-scale model with 10.5 billion parameters. post evaluation (section 5.2) confirmed the limitations of sentence-level translation quality improvement through model scaling and the effectiveness of our reranking approach.