F uxi T ranyu: A Multilingual Large Language Model Trained with Balanced Data (2024)

Abstract: Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between high- and low-resource languages. To mitigate this challenge, we present FuxiTranyu , an open-source multilingual LLM, which is designed to satisfy the need of the research community for balanced and high-performing multilingual capabilities. The base model, FuxiTranyu-8B, features 8 billion parameters and is trained from scratch on meticulously balanced multilingual data that contains 600 billion tokens covering 43 natural languages and 16 programming languages. We also develop two instruction-tuned models: FuxiTranyu-8B-SFT which is fine-tuned on a diverse multilingual instruction dataset, and FuxiTranyu-8B-DPO which is further refined with DPO on a preference dataset for enhanced alignment ability. Extensive experiments on a wide range of multilingual benchmarks demonstrate the competitive performance of FuxiTranyu against existing multilingual LLMs, e.g., BLOOM-7B, PolyLM-13B, and Mistral-7B-Instruct. Both neuron and representation interpretability analyses reveal that FuxiTranyu achieves consistent multilingual representations across languages. To promote further research into multilingual LLMs, we release both the base and instruction-tuned FuxiTranyu models together with 58 pre-training checkpoints at HuggingFace and Github.

## introduction a well-pretrained base model is crucial for facilitating research and applications of large language models. however, training a base llm from scratch typically demands a substantial amount of data and significant computational resources, posing a barrier to the development of new llms. the majority of llms are usually tailored to specific languages such as english @xcite or chinese @xcite , neglecting the growing demand for multilingual capabilities, especially from low-resource languages. while certain llms like mistral models @xcite demonstrate multilingual capabilities, their coverage is limited, restricting the exploration in massively multilingual settings. recent efforts have been dedicated towards mitigating such language-specific constraints through supervised fine-tuning, as exemplified by okapi @xcite . however, as highlighted by the alignment hypothesis in lima @xcite , the knowledge of llms is predominantly derived from pre-training, while supervised fine-tuning primarily aligns model behavior to instructions, which is a narrow subset of the pre-training data. this makes fine-tuning less effective for boosting multilingual abilities when pre-training is dominated by a few languages. other initiatives have focused on pre-training multilingual llms, such as bloom @xcite and polylm @xcite . nevertheless, these efforts are hindered by their performance, which does not measure up to that of current trending llms. bloom suffers from outdated training data, while polylm is undermined by imbalanced language distribution, with english data accounting for approximately 70% and chinese for ~20%, potentially leading to insufficient learning of under-represented languages. previous studies @xcite disclose three traits of multilingual llms caused by imbalanced language resources: cross-lingual inconsistency, distorted linguistic relationships, and unidirectional transfer between high-and low-resource languages, emphasizing the need for balanced data distribution. recently introduced multilingual llms, e.g., aya 23 models @xcite , have demonstrated remarkable performance on multiple llms pre-training tokens languages base model available pretraining checkpoints available bloom-7b1 (scao et al., 2022a) 300b 46 nls + 13 pls ✓ ✓ aya 23-8b (aryabumi et al., 2024) unknown 23 nls × × polylm-13b (wei et al., 2023) 638b 18 nls ✓ × fuxitranyu-8b 606b 43 nls + 16 pls ✓ ✓ in this work, we present fuxitranyu, a family of multilingual llms supporting 43 natural languages and 16 programming languages. the fuxitranyu initiative aims to mitigate the aforementioned challenges of multilingual llms. the base model comprises 8 billion parameters and has been trained from scratch using approximately 600 billion multilingual tokens. to ensure balanced learning across all supported languages, we have manually controlled the sampling ratio of pretraining data for different languages, striving for as balanced distribution as possible. in line with our commitment to advancing research in multilingual llms, we have also released 58 pre-training checkpoints, resonating with the efforts of llm360 @xcite . in addition to the base model, we develop two instruction-tuned models: fuxitranyu-8b-sft, fine-tuned on a collected high-quality multilingual instruction dataset, and fuxitranyu-8b-dpo, further tuned on preferences with dpo for enhanced alignment ability. our evaluations focus on knowledge, capability and alignment dimensions categorized by @xcite . evaluation results on multilingual discriminative tasks such as multilingual arc, hel-laswag, and mmlu @xcite , xwinograd @xcite , xcopa @xcite , xs-torycloze @xcite , and multilingual generative tasks including wmt and iwslt translation benchmarks @xcite and xl-sum summarization benchmark @xcite , demonstrate fuxitranyu's superior performance compared to bloom-7b1 3 https://cohere.com/command and polylm-13b, as detailed in section 5. the instruction-tuned models, fuxitranyu-8b-sft and fuxitranyu-8b-dpo, also outperform llama-2-chat-7b, mistral-7b-instruct-v0.1, bloomz-7b1, polylm-multialpaca-13b on translation and summarization benchmarks. to further understand the multilingual capabilities of fuxitranyu models, we have conducted neuron-and representation-level analysis, revealing that fuxitranyu-8b learns more languageagnostic representations compared to bloom-7b1 @xcite , which can be attributed to the balanced pre-training data. however, languages with extremely limited resources, such as bengali and tamil, are allocated with fewer neurons. additionally, different layers and components of fuxitranyu-8b handle multilingual text differently, with deep layers being more languagespecific and the importance of attention and mlp components varying across layers. ## related work recent advanced llms @xcite @xcite @xcite @xcite have excelled in nlp and crossmodal tasks, sparking increased research on multilingual llms @xcite @xcite , which aim at broader language support. there are three main approaches to building multilingual llms: pre-training from scratch, continual pre-training, and post-training (e.g., supervised fine-tuning and reinforcement learning from human feedback). pre-training from scratch, like palm 1&2 @xcite , bloom @xcite , and polylm @xcite , leverages extensive training corpora from diverse sources, enabling the incorporation of new knowledge. however, pre-training poses a variety of challenges, such as the need for vast computing resources, which can hinder the development of new multilingual llms. additionally, it also suffers from the curse of multilinguality @xcite @xcite , where the performance of individual languages deteriorates as the number of languages increases. on the other hand, continual pre-training, like aurora-m @xcite , llamax @xcite , is more efficient but risks catastrophic forgetting of previously learned knowledge. supervised fine-tuning (sft) often leverages multilingual instruction data or incorporates translation tasks to address data scarcity @xcite @xcite . however, both continual pre-training and sft rely heavily on high-quality, diverse datasets, which are often limited to many languages. reinforcement learning from human feedback (rlhf) is increasingly used to align models with human preferences @xcite . in multilingual llms, multilingual rlhf data are used to train multilingual reward models @xcite . however, rlhf typically relies on human-annotated data, which can be expensive and time-consuming to collect, especially for under-resourced languages. while these methods can achieve impressive performance, they can also be computationally expensive and may not generalize well to unseen languages. ## pretraining we elaborate on the sources and domains of our pre-training data and the efforts we have made in the pre-processing stage in section 3.1. next, we discuss the details of our fuxitranyu architecture in section 3.2. we present the strategy we used to determine which languages should be supported by the fuxitranyu series of models in appendix a, the details of our tokenizer training in appendix b, and the pre-training settings in appendix c.1. ## experiments we conducted extensive experiments to evaluate the capabilities of fuxitranyu (both the base model and instruction-tuned models) under the multilingual setting. we compared fuxitranyu against strong baselines, including both englishcentric and multilingual models. for englishcentric models, we used llama-2 (llama-2-7b, llama-2-chat-7b) @xcite and mistral (mistral-7b-v0.1, mistral-7b-instruct-v0.1) @xcite as baseline. for multilingual models, we compared fuxitranyu with bloom (bloom-7b1, bloomz-7b1) @xcite , polylm (polylm-13b, polylm-multialpaca-13b) @xcite , and llamax2 (llamax2-7b, llamax2-7b-alpaca) @xcite . 11 we used the lm evaluation harness framework @xcite for all evaluation experiments. discriminative tasks for evaluating discriminative tasks, we used arc @xcite , hellaswag @xcite , mmlu @xcite , xwinograd @xcite , xcopa @xcite , and xsto-rycloze @xcite datasets. specifically for the multilingual evaluation, we utilized the multilingual version of arc, hellaswag and mmlu datasets @xcite and selected 15 languages for the evaluation (ar, bn, de, en, es, fr, hu, id, it, pt, ru, sk, ta, vi, zh). for xwinograd, xcopa, and xstorycloze datasets, we utilized all of the languages provided in the datasets. generative tasks we evaluated the performance towards generative tasks, especially in translation and summarization tasks. for the translation task, we employed wmt14 in en-fr translation direction @xcite , wmt16 in en-de and en-ro translation directions @xcite and @xcite in en-ar translation direction for measuring the translation performance in our models and benchmark models. for the summarization task, we used xl-sum @xcite dataset. we selected 15 languages for the evaluation (ar, en, es, fr, gu, hi, id, mr, pt, ru, sr, ta, uk, vi, zh). ## conclusion in this paper, we have presented fuxitranyu to address the need for open-source multilingual llms. along with the base model, fuxitranyu-8b, we also present instruction-tuned models fine-tuned on multilingual supervised fine-tuning and preference data, fuxitranyu-8b-sft and fuxitranyu-8b-dpo. evaluations on multilingual benchmarks show fuxitranyu outperforms previous multilingual and monolingual llms. furthermore, interpretability analyses underscore the efficacy of the multilingual capabilities embedded in fuxitranyu. computing fertility on the same set of documents across different languages. results are presented in figure 2 , which indicate that the fuxitranyu tokenizer is more efficient than the others in most languages. based on our evaluations and interpretability analysis, we believe that the fertility of the tokenizer positively correlates with the model's performance in specific languages. in the fertility test, we observe that bengali (bn), hindi (hi), and tamil (ta) exhibit high fertility, indicating lower tokenization efficiency in these languages compared to others. consequently, the performance and importance of neurons of these languages in our base model are also suboptimal. further details are discussed in section 6.3.