Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating massive languages? 2) Which factors affect LLMs’ performance in translation? We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our empirical results show that translation capabilities of LLMs are continually involving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of translation directions but still faces a large gap towards the commercial translation system like Google Translate, especially on low-resource languages. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, LLM can acquire translation ability in a resource-efficient way and generate moderate translation even on zero-resource languages. Second, instruction semantics can surprisingly be ignored when given in-context exemplars. Third, cross-lingual exemplars can provide better task guidance for low-resource translation than exemplars in the same language pairs. Code will be released at: https://github.com/NJUNLP/MMT-LLM.

## introduction with the increasing scale of parameters and training corpus, large language models (llms) have gained a universal ability to handle a variety of tasks via in-context learning @xcite , which allows language models to perform tasks with a few given exemplars and human-written instructions as context. one particular area where llms have shown outstanding potential is machine translation (mt). previous studies have shown the surprising performance of llms on high-resource bilingual translation, such as english-german translation @xcite , even if these 1 code will be released at: https://github.com/ njunlp/mmt-llm . models are not particularly optimized on multilingual data. however, the multilingual translation ability of llms remains under-explored. mmt is a challenging task that involves translating text among different languages and requires semantic alignment between languages @xcite @xcite . it is also unclear that how llm acquires translation ability and which factors affect llm's translation ability. in this paper, we follow icl paradigm and focus on studying llms in multilingual machine translation by answering two questions: 1) how llms perform mmt over massive languages? 2) which factors affect the performance of llms? for the first question, we evaluate several popular llms: english-centric llms, including opt @xcite , llama2 @xcite , falcon @xcite and multilingual llms, including xglm @xcite , @xcite , chatgpt (ope-nai, 2022), @xcite . we consider 102 languages and 606 translation directions (202 english-centric directions, 202 french-centric directions and 202 chinese-centric directions). results show that the multilingual translation capabilities of llms are continually involving and gpt-4 reaches new performance height. compared with the widely-used supervised mmt system nllb (team, 2022), gpt-4 achieves higher performance on 40.91% english-centric translation directions. but compared with the commercial translation system (google translate), llms still have a long way to go, particularly when it comes to low-resource languages. french-centric and chinese-centric translation are also more challenging for gpt-4 than english-centric translation, which further indicates its unbalanced capability across languages. for the second question, we find some new working patterns. first, we discover that llm can acquire translation ability in a resource-efficient way and generate moderate translation even on zeroresource languages. second, llms are able to perform translation even with unreasonable instructions if in-context learning exemplars are given. however, if given mismatched translation pairs as in-context exemplars, llms fail to translate, which is similar to observations from concurrent studies @xcite . this shows the importance of exemplars in icl for machine translation. third, we find that cross-lingual translation pairs can be surprisingly good exemplars for low-resource translation, even better than exemplars in the same language. the main contribution of this paper can be summarized below: ## experiment setup dataset we benchmark multilingual translation on flores-101 @xcite ) dataset foot_0 , which enables an assessment of model quality on a wide range of languages. llms we evaluate translation performance of eight popular llms: xglm-7.5b @xcite , opt-175b @xcite , bloomz-7.1b @xcite , falcon-7b @xcite , llama2-7b @xcite , llama2-7b-chat @xcite , chat-gpt (openai, 2022) and gpt-4 foot_1 (openai, 2023). icl strategy for each model, we report its translation performance with eight randomly-picked translation pairs foot_2 from the corresponding development set as in-context exemplars and "<@xmath0. "<@xmath1. we use line-break as the concatenation symbol. according to our experiment analysis, this icl strategy serves as a simple but strong recipe. all implementation is based on openicl foot_3 @xcite . ## benchmarking llms for massively multilingual machine translation in this section, we report results on multilingual machine translation and introduce our main findings about llms' translation ability. the multilingual translation capabilities of llms are continually involving llms still lag behind the strong supervised baseline, especially on low-resource languages figure 2 shows the translation performance of the supervised systems and gpt-4 on each language. in 40.91% translation directions, gpt-4 has achieved higher bleu scores than nllb, indicating the promising future of this new translation paradigm. but on long-tail low-resource languages, gpt-4 still lags behind nllb, let alone google translate. data leakage issue should be considered before evaluating llms on public datasets. we do not include bloomz's performance on flores-101 in our report because bloomz is instructiontuned with xp3 dataset @xcite , which includes flores-200 dataset. thus bloomz may have been exposed to test cases from flores-101 during training. if so, the evaluation results can not precisely reflect its translation ability @xcite . to illustrate this concern, we take 1000 english sentences from the most recent news spanning august 2023 to october 2023 foot_9 , and ask human experts to translate them into chinese and construct a bilingual no-leakage evaluation set, named news2023. figure 4 shows that bloomz's performance significantly deteriorates on this no leakage set, whereas other models maintain a consistent performance across both datasets. through this example, we wish to draw the community's attention to the potential data leakage issue when evaluating large language models. ## analyzing factors that influence llm's translation performance to better understand how llm acquires translation ability and which factors have influence on its performance, we conduct in-depth analysis. for analysis, we choose xglm-7.5b as an example foot_10 . note that, when studying a certain factor, we keep the remaining factors unchanged. ## conclusion in this paper, we evaluate the multilingual translation ability of popular llms, including chatgpt and gpt-4, on 102 languages and 606 directions, which presents the advantages and challenges of llms for mmt. we find that translation capabilities of llms are continually involving and gpt-4 reaches new performance height. however, even for gpt-4, it still face challenge on low-resource languages. in our analysis, we find that llms exhibit new working patterns when used for mmt. for example, instruction semantics can be ignored during in-context learning and cross-lingual exemplars can provide better task instruction for lowresource translation. more importantly, we find that llm can acquire translation ability in a resourceefficient way, which indicates the promising future of llm in multilingual machine translation.