U v A - MT ’s Participation in the WMT 2023 General Translation Shared Task (2023)

Abstract: This paper describes the UvA-MT’s submission to the WMT 2023 shared task on general machine translation. We participate in the constrained track in two directions: English \leftrightarrow Hebrew. In this competition, we show that by using one model to handle bidirectional tasks, as a minimal setting of Multilingual Machine Translation (MMT), it is possible to achieve comparable results with that of traditional bilingual translation for both directions. By including effective strategies, like back-translation, re-parameterized embedding table, and task-oriented fine-tuning, we obtained competitive final results in the automatic evaluation for both English \rightarrow Hebrew and Hebrew \rightarrow English directions.

## introduction multilingual machine translation (mmt) @xcite has attracted a lot of attention in recent years because of 1) its high-level efficiency (multiple translation directions within a single model) and 2) the potential for knowledge transfer, especially for low-resource or even unseen directions. in mmt systems, only one additional tag is introduced to indicate the translation direction, compared to the conventional encoder-decoder architecture. in this competition, we explore mmt with a minimal setting, i.e., using one model for bidirectional translations simultaneously. we leverage all the official parallel data and a substantial portion of the monolingual data generously provided by the wmt23 organizer, as elaborated in section 2. to elevate the quality of our parallel data, we implemented a comprehensive three-step cleaning procedure. additionally, for monolingual data, we further trained an n-gram language model to filter out low-quality sentences, * equal contribution. the backbone of our system is based on a standard transformer @xcite . additionally, we build a re-parameterized embedding table @xcite (see section 3.2) to enhance the representational word similarity between english and hebrew, targeting better knowledge transfer for multilingual translation. the final system involves three stages of training: 1) pretraining with synthetic data (see section-4.1), where we leverage back-translation @xcite to produce synthetic data, and finally add them as additional data within new translation directions in our mmt system to conduct pretraining. 2) training without synthetic data (see section 4.2), where we discard the additional synthetic data and further train our system using real bitext only. 3) fine-tuning with task-related data (see section 4.3), where we copy and fine-tune our system using @xmath0. we observe evident improvements for stage 1 and stage 2, while surprisingly performance drops for stage 3. we report our results, including the offline evaluation and the final online evaluation, in section 5. our constrained system showed comparable performance to unconstrained systems, and outperformed the second-place constrained submission with +10 bleu. ## data in this section, we provide an overview of our data sources and the data cleaning procedures applied to our english-hebrew translation task. we utilize both parallel and monolingual data sets provided by the organizers for training our translation systems. ## experiments we describe the training process of our system in three stages. ## results table-2 shows our offline and online evaluation results according to each training stage described in section 4. we still use flores-200 to conduct offline evaluations. the online results are reported by wmt23 background bleu evaluations. stage-1, -2, and -3 refer to "pretraining with synthetic data", "training without synthetic data", and "finetuning on task-specific data" respectively. the results of online and offline evaluations are quite consistent. both of them achieve best results when training with stage 1 and stage 2. it shows that by step-by-step narrowing training data from mixing with synthetic data to real data distribution, we can further boost our mmt system's performance. however, when we further conduct fine-tuning on direction-specific data, i.e., applying stage 3, there is an evident performance drop. it seems that tuning in a specific direction upon mmt may not be a good practice, at least when the training data are a subset of that for mmt. we leave this point for future exploration. our final system achieves 35.0 and 51.0 in en he and he en direction respectively, which are both in the first place for constrained tracks. ## conclusion in this competition, we show that: 1) it is possible to achieve comparable results with conventional bilingual translation by using mmt training fashion to handle two dual translation directions. 2) previous embedding re-parameterized method @xcite also works for bilingual translation, which is not verified in the original paper. however, when training data scales up to 30+m level, the improvements become marginal. 3) by step-by-step narrowing training data (especially for stage-1 and stage-2) from mixing with synthetic data to real data distribution, we successfully boost the final performance, even in a quite high-resource scenario (30+m).