Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study (2025)

Abstract: Large language models (LLMs) have shown continuously improving multilingual capabilities, and even small-scale open-source models have demonstrated rapid performance enhancement. In this paper, we systematically explore the abilities of open LLMs with less than ten billion parameters to handle multilingual machine translation (MT) tasks. We conduct comprehensive evaluations on six popular LLMs and find that models like Gemma2-9B exhibit impressive multilingual translation capabilities. We then introduce the Parallel-First Monolingual-Second (PFMS) data mixing strategy in the continual pretraining stage to further enhance the MT performance and present GemmaX2-28, a 9B model achieving top-tier multilingual translation performance across 28 languages. Specifically, GemmaX2-28 consistently outperforms the state-of-the-art (SOTA) models such as TowerInstruct and X-ALMA and achieves competitive performance with Google Translate and GPT-4-turbo.

## introduction large language models (llms) such as gpt models @xcite , palm models @xcite , and others have demonstrated remarkable capabilities in multilingual translation @xcite . recently, significant efforts have been directed towards bolstering the multilingual competencies of open llms across a diverse linguistic spectrum, beyond english and chinese. for example, over 5% of the llama3 @xcite pretraining datasets consist of high-quality non-english data covering over 30 languages; qwen2/2.5 @xcite claim to have multilingual support for over 29 languages; @xcite introduce aya-101, a massively multilingual generative language model supporting 101 languages. @xcite evaluate popular llms on multilingual mt tasks and find that the translation capability of llms is continually evolving. however, the models assessed (xglm @xcite , @xcite , falcon @xcite , bloomz @xcite , llama2 @xcite ) are outdated due to the rapid development of open llms, and the multilingual translation ability of the latest models remains unclear. one natural question arises: 1) how do the latest open llms with practical scale perform multilingual mt tasks? to further boost llms' translation capability, existing approaches usually adopt multilingual corpora during continual pretraining on open llms. specifically, @xcite utilize continual pretraining on monolingual datasets. @xcite apply continual pretraining on monolingual datasets followed by parallel datasets. @xcite perform continual pretraining on a multilingual mixture of monolingual (two-thirds) and parallel (one-third) datasets. however, the optimal approach to leverage monolingual and parallel datasets for multilingual mt remains underexplored. another natural question arises: 2) what are the best practices for leveraging monolingual and parallel corpora to enhance the multilingual mt performance of llms? in this paper, we first evaluate the in-context capabilities of the latest open-source llms including mistral-7b-v0.3 @xcite , qwen2/2.5-7b @xcite , llama3/3.1-8b @xcite , and gemma2-9b @xcite across 28 languages on the multilingual mt tasks. we find that open llms like gemma2-9b exhibit remarkable multilingual translation capabilities but still fall behind strong closed-source models. we then systematically investigate the most effective methods to leverage monolingual and parallel data for further boosting gemma2-9b's capabilities and propose the parallel-first monolingual-second (pfms) data mixing strategy. leveraging pfms strategy when continually pretraining on gemma2-9b followed by instruction finetuning on a small set of high-quality translation pairs, we learn gemmax2-28-9b, a many-to-many multilingual mt model that performs competitively with google translate and gpt-4-turbo across 28 widely spoken languages. the contributions of this paper can be summarized as follows: ## datasets and baseline settings in this section, we describe the evaluation datasets used in sections 4 and 5 as well as the model configurations. ## benchmarking open llms for multilingual machine translation in this section, we report empirical results on tokenizer efficiency and multilingual mt performance across various open-source llms. ## gemmax: boosting multilingual translation with gemma models given its impressive multilingual capabilities discussed in section 4, we select gemma2-9b as our backbone model for learning many-to-many multilingual machine translation across 28 languages. we continue the pretraining of gemma2-9b with multilingual corpora and finetune the continual pretrained model using a small but high-quality parallel dataset with the translation prompt as follows: translate this from [source language] to [target language]:\n[source language]: <source sen-@xmath0:<target sentence>. ## conclusion in this paper, we systematically evaluate the multilingual in-context translation of the latest open llms at a practical scale. while these models demonstrate strong translation capabilities, they still fall short compared to closed-source models. furthermore, we explore the best data recipe for multilingual translation with llms and propose a parallel-first monolingual-second (pfms) data mixing strategy. leveraging the pfms strategy during continual pretraining on gemma2-9b, we achieve significant improvements, bringing its translation performance to a level comparable with gpt-4-turbo and google translate. to the best of our knowledge, gemmax2-28-9b is the open model with the highest translation quality. we aim to develop models that support a broader range of languages and possess enhanced translation capabilities as part of our future work.