Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions (2024)

Abstract: Large-scale pretrained language models (LLMs), such as ChatGPT and GPT4, have shown strong abilities in multilingual translation, without being explicitly trained on parallel corpora. It is intriguing how the LLMs obtain their ability to carry out translation instructions for different languages. In this paper, we present a detailed analysis by finetuning a multilingual pretrained language model, XGLM-7.5B, to perform multilingual translation following given instructions. Firstly, we show that multilingual LLMs have stronger translation abilities than previously demonstrated. For a certain language, the translation performance depends on its similarity to English and the amount of data used in the pretraining phase. Secondly, we find that LLMs’ ability to carry out translation instructions relies on the understanding of translation instructions and the alignment among different languages. With multilingual finetuning with translation instructions, LLMs could learn to perform the translation task well even for those language pairs unseen during the instruction tuning phase.

## introduction the emergence of large pretrained language models (llms) @xcite has revolutionized the research of machine translation @xcite . these models have demonstrated remarkable multilingual translation capabilities, without requiring explicit training on parallel corpora. for instance, xglm, a medium-sized multilingual language model, outperforms supervised models using only several examples as demonstrations * equal contribution. † corresponding author. @xcite ; the cutting-edge llm gpt4 has been shown to perform comparably to commercial translation systems on multiple language pairs @xcite . most existing research on llms for machine translation focuses on in-context learning (icl), i.e., taking several parallel sentences as the demonstration to guide llms to perform translation @xcite @xcite . however, these methods rely heavily on the in-context learning ability of llms. for smaller models, e.g., models with only 1b or 7b parameters, the relatively weak icl ability may result in an underestimation of their potential translation ability. instead of relying on the icl abilities, we propose to investigate the ability of llms by directly training them to follow translation instructions. inspired by the recent success of instruction tuning @xcite , we organize multilingual translation tasks as different instances of the translation instruction, with each instance corresponding to a specific language pair. by training the llms to follow these instructions, i.e., with multilingual finetuning with translation instructions (mfti), it is possible to better elicit translation ability inside llms. our results show that by training on a mixed dataset of 1,000 sentences per language pair, mfti outperforms the 8-shot in-context learning by near 3 bleu on average, showing a greater potential of llms' translation ability than previously demonstrated @xcite . in addition, we also discuss how mfti improves the llms and which factors influence the performance. to better understand why llms could follow these instructions, we design a mfti setting where only a subset of the translation instructions, i.e., language pairs, are used for training. thus llms need to generalize their instruction following abilities for those language pairs unseen during mfti. surprisingly, mfti elicits the translation ability not only for trained language pairs but also for those unseen during instruction training. with further experiments and analyses, we find that llms could learn translation behavior in general by being trained to translate even irrelevant language pairs. it is also interesting that with mfti, llms learn to directly align languages through the use of pivot languages, which enhances the instruction-following ability for unseen language pairs. 2 multilingual finetuning with translation instructions ## understanding the potential translation ability of llms in this section, we first assess the overall translation performance of mfti by comparing it to few-shot in-context learning. 1 we then present a detailed analysis of how the corpus for mfti influences the translation quality. ## understanding the ability of carrying out translation instructions in this section, we present a comprehensive analysis on how mfti improves the model's ability to carry out translation instructions. we begin by presenting an overarching experiment where we intentionally withhold certain language pairs during the mfti process, which allows us to study models' ability to carry out translation instructions under different conditions. furthermore, we delve deeper into our analysis by exploring how mfti enhances llms' ability to carry out translation instructions from following perspectives: better understanding of translation instructions (section 4.3 and section 4.4) and better alignment between languages to execute translation instructions (section 4.5). ## conclusion in this paper, we explore multilingual finetuning with translation instructions (mfti), to better unleash the translation ability of multilingual llms. through extensive experiments, we demonstrate that by training on a mixture of 1000 sentences per language pair, mfti achieves better performance than 8-shot icl, indicating the untapped potential of translation ability in llmsv by previous works. moreover, we systematically discuss the working mechanism of mfti by analyzing it from the view of instruction-following. our experiments demonstrate that mfti helps the model better follow the instruction by introducing more language pairs and monolingual sentences, and enhances the direct language alignment by learning from pivot language pairs. our paper also unveils remaining translation issues when adopting llms for zero-shot machine translation, i.e., over/under translation, oscillatory hallucination, and mistranslation caused by incorrect alignments. future work should focus on acquiring more language knowledge from the pretraining phase and designing better regularization terms to solve these problems.