A dam M ickiewicz U niversity at WMT 2022: NER -Assisted and Quality-Aware Neural Machine Translation (2022)

Abstract: This paper presents Adam Mickiewicz University’s (AMU) submissions to the constrained track of the WMT 2022 General MT Task. We participated in the Ukrainian ↔ Czech translation directions. The systems are a weighted ensemble of four models based on the Transformer (big) architecture. The models use source factors to utilize the information about named entities present in the input. Each of the models in the ensemble was trained using only the data provided by the shared task organizers. A noisy back-translation technique was used to augment the training corpora. One of the models in the ensemble is a document-level model, trained on parallel and synthetic longer sequences. During the sentence-level decoding process, the ensemble generated the n-best list. The n-best list was merged with the n-best list generated by a single document-level model which translated multiple sentences at a time. Finally, existing quality estimation models and minimum Bayes risk decoding were used to rerank the n-best list so that the best hypothesis was chosen according to the COMET evaluation metric. According to the automatic evaluation results, our systems rank first in both translation directions.

## introduction we describe adam mickiewicz university's submissions to the constrained track of the wmt 2022 general mt task. we participated in the ukrainian ↔ czech translation directions -a low-resource translation scenario between closely related languages. the data provided by the shared task organizers was thoroughly cleaned and filtered, as described in section 2. the approach described in section 3 is based on combining various mt enhancement methods, including transfer learning from a high-resource language pair @xcite , noisy back-translation @xcite , nerassisted translation @xcite , document-level translation, model ensembling, quality-aware decoding @xcite , and on-the-fly domain adaptation @xcite . the results leading to the final submissions are presented in section 4. additionally, we performed a statistical significance test with paired bootstrap resampling @xcite , comparing the baseline solution with the final submission on the test set reference translations released by the shared task organizers. according to the automatic evaluation results based on comet @xcite scores, our systems rank first in both translation directions. ## data in the initial stage of system preparation, the sentence-level data was cleaned and filtered using the opusfilter @xcite toolkit. with the use of the toolkit, language detection filtering based on fasttext @xcite was performed, duplicates were removed, and heuristics based on sentence length were applied. in particular, we removed sentence pairs with a length ratio over 3 and long sentences (> 200 words). then, using moses @xcite pre-processing scripts, punctuation was normalized and non-printing characters removed. finally, the text was tokenized into subword units using senten-cepiece @xcite with the unigram language model algorithm @xcite . for @xmath0, we used separate vocabularies for the source and the target language. each vocabulary consisted of 32,000 units. we used concatenated data from the flores-101 @xcite vided by the task organizers. ## approach we used the marian @xcite toolkit for all of our experiments. our model architecture follows the transformer (big) @xcite settings. for all model training, we used 4x nvidia a100 80gb gpus. ## results the results of our experiments are presented in in this case, the scores include improvements added by back-translation, ner source factors and finetuning. the document-level evaluation was split into sentence-level decoding and document-level decoding. in the first scenario, the model translates 1 comet scores were computed with the wmt20-comet-da model. we found that the largest gain in the comet value was achieved due to the quality-aware decoding method, at the cost of bleu value. the chrf value remained the same in the @xmath1, while it increased slightly in the @xmath2. as discussed in section 3.3, the inclusion of ner source factors helped the model with the translation of named entities, which is not well reflected in the comet value, as this metric is not sufficiently sensitive to discrepancies in named entities @xcite . ## conclusions we describe adam mickiewicz university's (amu) submissions to the wmt 2022 general mt task in the ukrainian ↔ czech translation directions. our experiments cover a range of mt enhancement methods, including transfer learning, back-translation, ner-assisted translation, document-level translation, weighted ensembling, quality-aware decoding, and on-the-fly domain adaptation. we found that using a combination of these methods on the test set leads to a +0.22 (26.13%) increase in comet scores in the @xmath4.19 (24.18%) increase in the @xmath5, compared with the baseline model. according to the comet automatic evaluation results, our systems rank first in both translation directions.