GTCOM and DLUT ’s Neural Machine Translation Systems for WMT 23 (2023)

Abstract: This paper presents the submission by Global Tone Communication Co., Ltd. and Dalian Univeristy of Technology for the WMT23 shared general Machine Translation (MT) task at the Conference on Empirical Methods in Natural Language Processing (EMNLP). Our participation spans 8 language pairs, including English-Ukrainian, Ukrainian-English, Czech-Ukrainian, English-Hebrew, Hebrew-English, English-Czech, German-English, and Japanese-English. Our systems are designed without any specific constraints or requirements, allowing us to explore a wider range of possibilities in machine translation. We prioritize backtranslation, utilize multilingual translation models, and employ fine-tuning strategies to enhance performance. Additionally, we propose a novel data generation method that leverages human annotation to generate high-quality training data, resulting in improved system performance. Specifically, we use a combination of human-generated and machine-generated data to fine-tune our models, leading to more accurate translations. The automatic evaluation results show that our system ranks first in terms of BLEU score in Ukrainian-English, Hebrew-English, English-Hebrew, and German-English.

## introduction in this study, we utilize fairseq @xcite as our development tool and adopt the transformer @xcite as the primary architecture. the main ranking index for the submitted systems is bleu @xcite , which we also employed as the evaluation metric for our translation system using sacrebleu 1 , consistent with our approach from the previous year. for data preprocessing, we apply punctuation normalization, tokenization, and byte pair encoding (bpe) @xcite across all languages. additionally, we applied a truecase model for english, ukrainian and czech, tailored to the specific characteristics of each language. in terms of tokenization, we utilized polyglot foot_0 for ukrainian and hebrew, and moses tokenizer.perl @xcite for english and czech. moreover, we incorporated knowledge-based rules and a language model to clean parallel data, monolingual data, and synthetic data. for the multilingual translation model, we amalgamated all languages into a single model and supplemented it with an english to russian parallel corpus to enrich the language information. the remainder of this paper is organized as follows: section 2 introduces the translation task and presents statistics of the dataset. section 3 describes our baseline systems and the proposed multilingual translation model. the data selection method is elaborated in section 4. section 5 presents experiments conducted on all translation directions, covering data filtering, model architectures, back-translation, joint training strategies, adaptations of the multilingual model, fine-tuning, data selection, and ensemble decoding. section 6 analyzes the results, providing insights into the efficacy of different techniques. finally, section 7 concludes the paper. ## task description the task at hand focuses on bilingual text translation, with the provided data detailed in ## billingual baseline model and multilingual translation model bilingual baseline model and multilingual translation model: to establish a robust baseline for comparison with our multilingual model, we employed the transformer_wmt_en_de as our bilingual baseline model, which consists of 12 encoding and 12 decoding layers. the multilingual translation model closely mirrors the gt-com2022 (zong and bei, 2022) model, but this year, the focus is on the x to x model. to achieve superior translation quality, we incorporated russian as the primary auxiliary language due to its high similarity with ukrainian. we trained a single multilingual model that encompasses all directions. for all languages in the multilingual model, we applied joint byte pair encoding (bpe) separately. ## data selection we use source test sets to train a text classification model with roberta @xcite . specifically, we use the in-domain test set as positive examples, and another same mount of sentence pairs from the out-of-domain test set as negative examples. we fine-tuned roberta on this labeled dataset to obtain a binary classifier, which can effectively distinguish between in-domain and outof-domain data. we then utilized this classifier to select domain-specific training data from the general training corpus. the selected in-domain training data was used to fine-tune the multilingual neural machine translation model. we also experimented with an alternative data selection approach based on prompt learning. we constructed a prompt template and leveraged the generative power of chatglm-6b @xcite to obtain an domain classifier via p-tuning @xcite . the prompt template is displayed in ## experiment this section outlines the step-by-step experiments we conducted, with the entire workflow depicted in figure 1 . ## result and analysis english-sourced language pairs. as shown in we have noticed a significant improvement, particularly in the low-resource direction of czech to ukrainian, when we added russian (which is a language closely related to ukrainian) to the multilingual corpus. ## conclusion this paper presents gtcom and dlut's neural machine translation systems for the wmt23 shared general mt task. we applied three major techniques to enhance translation quality: backtranslation, a multilingual translation model, and fine-tuning with data selection. by employing these techniques, we achieved significant improvements in automatic evaluation metrics, as demonstrated in