Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages (2024)

Abstract: While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages (English and French) across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages.

## introduction contemporary large language models (llms; @xcite @xcite @xcite are predominantly trained on multilingual corpora. however, the language distribution in the data is highly imbalanced. for instance, llms like llama-2 @xcite , with english as the primary language, have also been trained on japanese text, yet the quantity figure 1 : comparison between vanilla supervised finetuning (sft), translate-then-sft, and our proposed method. besides using the translated question-answer pairs in the target language (e.g., japanese), sdrrl further leverages the generated answer a ⋆ en by llms in the resource-rich language (e.g., english) and collects self-distillated data (in green box) to help enhance its multilingual capabilities. the imbalanced data distribution above has led to significant limitations in the capabilities of llms across most languages. to enhance the multilingual capabilities, a common approach follows the translating and then supervised fine-tuning (sft; @xcite paradigm, as shown in figure 1 @xcite . specifically, training data is translated into the target language using either the model itself or an external machine translation (mt) system before continuing the training process, thereby offering more data in the target language and improving multilingual capabilities. however, the translate-then-sft method encoun-ters several challenges: first, the multilingual enhancement gained from translated "questionanswer" pairs is limited and may sometimes even degrade the capabilities in the original primary language @xcite . second, constrained by the accuracy of machine translation (especially for the low-resource languages), the translated texts used for training can be highly noisy, containing numerous awkward sentences and incorrect content, adversely affecting the quality of the generated text and the multilingual abilities of the llms. therefore, we explore a new question along this trajectory: besides translating the training pairs, can we enhance the abilities in other languages by leveraging the original relatively strong capabilities of llms in resource-rich language? in this paper, we introduce sdrrl, a method that uses self-distillation from resource-rich languages) to achieve the goal mentioned above. specifically, as illustrated in figure 1 (c), sdrrl comprises two parts: (1) self-distillation: instead of the ground-truth answer, responses from llms in resource-rich languages are collected to construct a transfer set. these are then translated into other languages using machine translation systems and code-switching tools, forming "questionanswer" pairs that are semantically identical but linguistically varied, and conducting sentence-level knowledge self-distillation within the same batch. our experiments, based on llama-2-7b @xcite and seallm-7b @xcite with english as the resource-rich language, demonstrate that even with a smaller set of english instruction data as the transfer set, sdrrl can effectively distill english capabilities into 14 other languages, showing effectiveness in both multilingual comprehension and generation tasks. further analysis indicates that sdrrl helps preserve the original capabilities in high-resource languages and improves the quality of generated responses. ## method in this section, we first revisit the supervised finetuning (sft) and translate-then-sft paradigm, subsequently dividing the discussions into two parts of our proposed sdrrl. in the first part, we construct a transfer set using responses in the resource-rich language from llms through sentence-level self-distillation. in the second part, we employ parallel translation-based instruction data to further improve multilingual generation capabilities. ## conclusion and future work we introduce self-distillation from resource-rich languages (sdrrl) to enhance the multilingual capabilities of llms. sdrrl uses the model itself to generate high-quality responses in resource-rich source languages and their target language counterparts as supervision signals for knowledge distillation, aiming to align additional target languages with resource-rich languages. we conduct comprehensive experiments across 16 languages on llama-2 and seallm. the results demonstrate that, compared to various baselines, our method significantly enhances the performance of target languages while preserving the capabilities of source languages. this highlights the multilingual potential of llms and illuminates paths for further research into multilingual llms.