Achieving State-of-the-Art Multilingual Translation Model with Minimal Data and Parameters (2023)

Abstract: This is LanguageX (ZengHuiMT)’s submission to WMT 2023 General Machine Translation task for 13 language directions. We initially employ an encoder-decoder model to train on all 13 competition translation directions as our baseline system. Subsequently, we adopt a decoder-only architecture and fine-tune a multilingual language model by partially sampling data from diverse multilingual datasets such as CC100 and WuDaoCorpora. This is further refined using carefully curated high-quality parallel corpora across multiple translation directions to enable the model to perform translation tasks. As per automated evaluation metrics, our model ranks first in the translation directions from English to Russian, English to German, and English to Ukrainian. It secures the second position in the directions from English to Czech, English to Hebrew, Hebrew to English, and Ukrainian to English, and ranks third in German to English, Japanese to English, and Russian to English among all participating teams. Our best-performing model, covering 13 translation directions, stands on par with GPT-4. Among all 13 translation directions, our multilingual model surpasses GPT-4 in bleu scores for 7 translation directions.

## introduction since 2023, large language models like chatgpt @xcite have had a profound impact on the field of machine translation, characterized by an ever-increasing scale in terms of parameters and data requirements. many research institutions and language service providers struggle to keep pace with this computational arms race. for smaller teams, the only viable strategy is to maximize model performance under constrained hardware resources. we participated in the wmt 2023 general machine translation task, covering 13 translation directions. given our limited computational power and time constraints, it was infeasible to craft dedicated models for each translation direction, making a large-scale multilingual translation model our optimal choice. we utilized fairseq @xcite to train our baseline multilingual translation model and further employed the hugging face transformers toolkit @xcite to train a multilingual language model. subsequent fine-tuning with task-specific instructions enabled it to perform multilingual translation tasks effectively. ## data filtering and selection we participated in the wmt 2023 general mt task, competing in 13 language pairs including chinese to/from english, german to/from english (at the document level), hebrew to/from english (in a low-resource setting), japanese to/from english, russian to/from english, ukrainian to/from english, and english to czech. given the challenge of concurrently training for 13 translation directions, it was imperative for us to judiciously regulate the size of parallel corpora specific to each direction, as well as the parameter count of the multilingual translation model. this was crucial to ensure training completion within a constrained timeline. for the chinese-english bi-directional translation, our primary sources for parallel corpora were the ccmt corpus (which can be found at: ## system description this section illustrates how the model is trained step by step. ## conclusion this paper describes languagex (zenghuimt)'s translation system for the wmt2023 general mt task. initially, we utilize a comprehensive encoder-decoder structure to establish our baseline system by training across all 13 contest translation directions. in the subsequent stages, we embrace a solely decoder-focused design and harness a multilingual language model, drawing samples from multilingual datasets like cc100 @xcite and wudaocorpora @xcite . this model is then meticulously finetuned using select high-grade parallel corpora from various translation domains, empowering it to execute translation task. our best-performing model, covering 13 translation directions, boasts around 1 billion parameters. this is less than one percent of the parameter count of mammoth models like gpt-4 (openai, 2023), which possess hundreds of billions of parameters. in translation evaluations across all languages, our system stands on par with @xcite . among all 13 translation directions, our multilingual model surpasses @xcite in bleu @xcite scores for 7 translation directions.