S amsung R & D Institute P oland submission to WMT 20 News Translation Task (2020)

Abstract: This paper describes the submission to the WMT20 shared news translation task by Samsung R&D Institute Poland. We submitted systems for six language directions: English to Czech, Czech to English, English to Polish, Polish to English, English to Inuktitut and Inuktitut to English. For each, we trained a single-direction model. However, directions including English, Polish and Czech were derived from a common multilingual base, which was later fine-tuned on each particular direction. For all the translation directions, we used a similar training regime, with iterative training corpora improvement through back-translation and model ensembling. For the En → Cs direction, we additionally leveraged document-level information by re-ranking the beam output with a separate model.

## introduction since the transformer architecture became the standard model in neural machine translation, recent advancements in the field have come from two techniques. the first one is deepening the model by adding more layers, mainly in the encoder part, in order to model more complex dependencies @xcite @xcite . this, however, poses problems during the training -too deep models are much harder to train due to the gradient vanishing problem @xcite . the second technique consists in improving the quality of training data by removing spurious translations @xcite and making the data easier to learn through the teacher-student methodology @xcite @xcite . in this submission, we decided to leverage both techniques. we deepened the model with a lexicalshortcuts transformer modification. we also iteratively improved the synthetic corpora by train-ing better and better translation models, backtranslating and distilling the data in each step. the remainder of this paper is structured as follows: section 2 introduces the data used for training, section 3 shows baseline nmt models and our experiments. in section 4 we describe our training regime and results. section 5 is for conclusions. ## nmt system overwiev all of our systems are trained with the marian nmt foot_3 (junczys-dowmunt et al., 2018) framework. ## conclusions in this paper, we have described the submission to the wmt20 shared news translation task by samsung r&d institute poland. all submitted systems were constrained and utilized only the permitted data. with our approach, we were able to leverage two important techniques that improve the translation quality. one method was deepening the model, while still being able to train it effectively. the other one was filtering and improving the quality of the training data and producing high quality synthetic data. our iterative approach of improving the training data and improving the translation model proved to be successful, showing gradual increase in the bleu scores.