LL a MAX : Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages (2024)

Abstract: Large Language Models (LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we conduct extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages. Through a comprehensive analysis of training strategies, such as vocabulary expansion and data augmentation, we develop LLaMAX. Remarkably, without sacrificing its generalization ability, LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs (by more than 10 spBLEU points) and performs on-par with specialized translation model (M2M-100-12B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model. The code and the models are publicly available.

## introduction large language models @xcite @xcite @xcite exhibit excellence performance in translation tasks involving high-resource languages @xcite ), yet their effectiveness in lowresource translation is suboptimal @xcite @xcite . figure 1 illustrates the number of translation directions with performance exceeding 10 spbleu @xcite score on flores-101 @xcite . it is evident the majority of models are clustered around the origin point for arabic-centric translations, demonstrating a significant disparity when compared to their english-centric performance. figure 1 : we assess translations in both directions, @xmath0, across various models using flores-101 test, with x representing all 101 languages included in flores-101. the results are visualized in a figure where different markers represent various models, a red marker indicates that the language (lg) is arabic, while a blue marker indicates english. we count the number of translation directions that achieve a spbleu score higher than 10. the findings indicate that modest llms demonstrate strong support for english-centric translation, but underperform in arabic-centric translation. this discrepancy is primarily due to the lack of pre-training data for these languages @xcite @xcite . many researchers are actively working to address this issue. @xcite enhance the llms' ability by translating low-resource languages after learning textbooks. @xcite find cross-lingual examples that can provide better task guidance for low-resource translation. in addition to the efforts focus on the fine-tuning stage, some studies have attempted to train a multilingual llm from scratch @xcite , or to train a languagespecific llm @xcite @xcite . however, the languages covered by these works are not extensive @xcite @xcite , and the translation performance is still unsatisfactory @xcite @xcite . to tackle this discrepancy, we conduct a massive multilingual continual pre-training for non-english languages. firstly, we present a comprehensive analysis of critical technical designs, including vocabulary extension (section 2.1) and data augmentation (section 2.2). these analyses establish the groundwork for the training procedure, directly influencing the efficacy and, ultimately, the performance of the llms. subsequently, we apply those strategies in continual pre-training using both parallel and monolingual data to enhance the translation performance of llms across the 102 languages covered by flores-101, particularly for lowresource languages. a primary challenge in expanding language support lies in determining the appropriate vocabulary @xcite . after assessing the impact of adding language-specific tokens from various angles: tokenization granularity, embedding quality, and the model's inner distribution, we find that introducing a small number of new tokens significantly degrades existing llm performance, while a larger new token set increases training complexity and data requirements. surprisingly, adhering to the original vocabulary of llms emerges as the most cost-effective strategy for extending llms to 102 languages. another great challenge in extending language support is the scarcity of data for low-resource languages @xcite . to alleviate the scarcity of training data, we delve into dictionary-based data augmentation @xcite and conduct a comprehensive analysis of various augmentation strategies. this analysis takes into consideration different dictionaries and data sources (monolingual or parallel data). we find that the optimal approach for data augmentation involves using parallel data, with the choice of dictionary correlated to the number of target language entities it covers. finally, we leverage the above discussed techniques to perform large-scale, multilingual continual pre-training on llama series models @xcite , resulting in llamax series models (llamax2 and lla-max3). the llamax2, trained over 60 days using 24 a100 gpus, significantly enhances translation capabilities and achieves comparable performance (evaluated on flores-101) to the specialized translation model m2m-100-12b @xcite . specifically, our method demonstrates an average improvement of more than 10 spbleu compared to baseline models in low-resource-centric translation, as shown in ## training data construction to build powerful llms that support translation across a hundred languages, it is crucial to collect and construct a sufficient amount of data. ## benchmarking results in this section, we present multilingual benchmarking results to comprehensively demonstrate the potential of llamax2. we evaluate translation quality with spbleu @xcite and comet-22 @xcite for both llms and translation models. see appendix b for training details on llamax2 and description of baseline models. we significantly enhances the multilingual translation capabilities of the base llama2 model through massive multilingual continual pretraining. the benefits of our continual pretraining is enhancing the base llm's multilingual translation capabilities. evaluation results on flores-101 benchmark are shown in llamax outperforms other open-source decoder-only llms on multilingual translation by a large margin. next, we compare llamax2-alpaca model with other open-source decoder-only llms built for multilingual purposes ( llamax is closing the performance gap between open-source llm translator and specialized encoder-decoder translation systems. while llamax2 has achieved the state-of-theart translation performance among open-source decoder-only llms, the next critical question is whether we can close the gap between llms and specialized encoder-decoder translation systems. llamax provides a better starting point for specialized instruction-tuning on english task data. in the end, we demonstrate the usage of our continual pre-trained model ( llamax2 we now show that our released checkpoint can be enpowered to handle more multilingual tasks beyond translation. figure 3 presents three example tasks where we use specialized instruction data to unlock llamax2's abilities on specific tasks, such as math reasoning and common sense reasoning. we find that the instruction-tuned llamax2 model outperforms its llama2 model counterpart in non-english performance across all three tasks, demonstrating that provides a better starting point for instruction-tuning with task-specific data. llamax circumvents catastrophic forgetting issue. a common concern with continual pretraining on additional multilingual corpus is that the process might disturb the parametric knowledge and working pattern of the original model, a phenomenon known as catastrophic forgetting @xcite . furthermore, we compare lla-max2 with llama2 on popular english benchmarks that measure a diverse set of core capabilities of llms. experiment results in beyond the english-centric translation is more efficient and effective. we further investigate the necessity and feasibility of multilingual augmentation for an english-centric llm. we can effectively transform a translation task (@xmath1, which allows us to leverage the power of english as a central language, facilitating seamless communication and comprehension across various language pairs. we refer to this experimental setup as a pivot translation experiment. as shown in ## related work multilingual large language models. large language model (llms; openai, 2023; @xcite @xcite trained with englishcentric data can also solve various non-english tasks @xcite @xcite , but the performance between non-english and english is significantly large @xcite . efforts to develop more multilingual llms in two different ways: retraining llms with diverse multilingual data from scratch @xcite ; or continuous training of pre-trained models using language-specific data with the option to expand the vocabulary @xcite @xcite . instead of training from scratch, continual pre-training aims at updating pre-trained models with new data, making the process more efficient and cost-effective (gupta et al., 2023; alves et al., 2024; xie et al., 2023). multilinguality in llms. recent research has shed light on the multilingual capabilities of llms. a comprehensive survey by huang et al. (2024a) discusses various aspects of multilingualism in llms, including training and inference methods, model security, multi-domain with languages culture, and emphasizes the need for language-fai technology. yuan et al. (2024b) analysis multilingualism of llms from the vocabulary sharing aspect. zhao et al. (2024b) delve into the architecture of llms to find how llms handle multilingualism. recently, li et al. (2024) quantify the multilingual performance of llms. these studies provide valuable insights into the multilingual capabilities of llms, and the key technical design of continual pre-training for llamax. ## conclusion in this work, we enhance the series models of llama translation performance for 102 languages through continual pre-training, creating llamax. we compare llamax 's translation capabilities with other decoder-only llms and encoderdecoder models across multiple benchmarks. lla-max is also assessed on general tasks and finetuned with task-specific instructions. our results indicate that llamax improves translation quality while maintaining general capabilities and can serve as a powerful foundation model for downstream multilingual applications.