S amsung R & D Institute P oland Participation in WMT 2022 (2022)

Abstract: This paper presents the system description of Samsung R&D Institute Poland participation in WMT 2022 for General MT solution for medium and low resource languages: Russian and Croatian. Our approach combines iterative noised/tagged back-translation and iterative distillation. We investigated different monolingual resources and compared their influence on final translations. We used available BERT-likemodels for text classification and for extracting domains of texts. Then we prepared an ensemble of NMT models adapted to multiple domains. Finally we attempted to predict ensemble weight vectors from the BERT-based domain classifications for individual sentences. Our final trained models reached quality comparable to best online translators using only limited constrained resources during training.

## introduction samsung r&d institute poland (srpol) participated in the wmt 2022 general mt task for three translation directions: @xmath0, @xmath1. all our systems were built using only constrained datasets. in contrast to previous years, where the task focused on news translation, this year's task was domain-independent. however, mt models benefit a lot from domain adaptation. therefore, we decided to prepare an ensemble of nmt models adapted to multiple domains to benefit from domain adaptation and improve generalization. we prepared a news profiled model but also a general-purpose one. additionally, we worked on medical and legal domains; however, there was very limited in-domain data in the constraint path for this domains and we had to extract pseudo indomain data from monolingual corpora. our system was implemented using marian framework. the core of the submitted solution is iterative back-translation and iterative distillation * work done while at samsung r&d institute poland. ## english-russian all corpora were preprocessed by removing sentences of inappropriate languages, normalizing punctuation, replacing all russian letters ë (yo) with e (ye), removing duplicate sentences. ## english-croatian we applied similar preprocessing as for russian language. additionally to all available en-hr corpora from opus @xcite we added all available data for serbian language to the training. we used custom validation set based on ted for first iterations and wmt22 dev set for last two iterations. we added directional tokens in front of each sentence that allowed to differentiate between croatian and serbian translation. for back-translation we used news mono corpora and source language from all en-hr parallel corpora as well. additionally to the back-translated corpora we added en-hr parallel data. we performed two iterations of back-translation. after training of first iteration with back-translated data we fine-tuned the model on all parallel en-hr data. after training of the second iteration we fine-tuned the model on ccmatrix corpus @xcite . the back-translation was noised with gumbel noise. after the above we started to apply knowledge distillation and fine-tuning the model on distilled data. we did only 2 iterations of distillation. first distillation was done on ccmatrix corpus and second on tuning corpus (created from dgt, qed, tedtalks, europat, setimes, hrenwac, ted2020 corpora). we experimented with different learning rates in order to find the best performing model after this step. finally, we made an ensemble out of the best-performing models. additionally, we found that a normalization value of 0.5 results in a better score. ## results results of training iterations for english to russian are presented in tuning with domain adaptation corpora has improved slightly (0.1-0.2) some of single models but gave no noticeable improvement on final score of ensembles. ## conclusions we confirmed that iterative knowledge distillation combined with iterative back-translation is sufficient to prepare high-quality translation models. this method gives excellent results on low-resource and mid-resource languages. during the wmt 2022 general mt task, we reached one of the best results among constrained systems. in our work, we compared different methods of back-translation: clean, noised, and tagged. mostly, the tagged back-translation achieved the best results, but for some training iterations, noised backtranslation's results were on-par or better. we compared different sources of monolingual data used for back-translation: commoncrawl and news crawl. the comparison suggests that the choice of the monolingual corpus has a significant influence on final results. our exploration of different filtering methods suggests that while using pre-filtered data (as provided in wmt 2022), it is sufficient to filter only target data, leaving source data unfiltered. we presented a simple and effective method of adding contextual data to the training corpus, which gave a noticeable improvement. we investigated a new method of dynamic ensemble weighting, but the results show no improvement over other methods.