U ltra L ink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset (2024)

Abstract: Open-source large language models (LLMs) have gained significant strength across diverse fields. Nevertheless, the majority of studies primarily concentrate on English, with only limited exploration into the realm of multilingual abilities.In this work, we therefore construct an open-source multilingual supervised fine-tuning dataset.Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs. Firstly, we introduce a knowledge-grounded data augmentation approach to elicit more language-specific knowledge of LLMs, improving their ability to serve users from different countries. Moreover, we find modern LLMs possess strong cross-lingual transfer capabilities, thus repeatedly learning identical content in various languages is not necessary. Consequently, we can substantially prune the language-agnostic supervised fine-tuning (SFT) data without any performance degradation, making multilingual SFT more efficient.The resulting UltraLink dataset comprises approximately 1 million samples across five languages (i.e., En, Zh, Ru, Fr, Es), and the proposed data construction method can be easily extended to other languages.UltraLink-LM, which is trained on the UltraLink dataset, outperforms several representative baselines across many tasks.

## introduction thanks to the collaborative efforts of the active large language models (llms) community, opensource llms are becoming increasingly powerful @xcite , even outperforming some representative closedsource counterparts @xcite in some specific tasks @xcite . figure 1 : to equip large language models with not only language-specific knowledge but also language-agnostic expertise, we construct the ultralink dataset for multilingual sft. for each language, ultralink consists of four subsets, encompassing chat data with languagespecific content, chat data with language-agnostic content, math data, and code data. these accomplishments are closely related to the contribution of open-source supervised fine-tuning (sft) data @xcite @xcite @xcite , which plays a pivotal role in eliciting the instruction-following ability of llms and aligning the model behaviour with human preferences. nevertheless, the focus of existing works is primarily on the construction of english sft data, resulting in a comparatively limited availability of multilingual sft resources. to mitigate the challenge of data scarcity, some researchers suggest translating english sft data into multiple languages. @xcite utilize chatgpt foot_0 to translate the two essential components, instructions and responses, from alpacastyle @xcite english data to other languages. @xcite propose to translate both the alpaca and the sharegpt foot_1 data. while directly translating english sft data can effectively support multilingual sft, there are still two major drawbacks associated with this approach: we believe that a good multilingual llm should not only possess language-specific knowledge but also be equipped with language-agnostic skills. figure 2 gives an example of the two types of instructions. we thus propose a new approach to better construct multilingual sft data, applicable to any language. compared to conversation translation @xcite , our advantages can be illustrated as follows: figure 2 : examples of instructions with languagespecific and language-agnostic content. we apply the aforementioned approach to four non-english languages, including chinese, russian, french, and spanish. note that our method can also be easily extended to other languages. finally, we train a sft llm on the proposed ultra-link dataset, which outperforms several representative open-source multilingual llms, demonstrating the effectiveness of our dataset. ## data curation automatically generating sft data is now an important research topic for llms @xcite @xcite . for multilingual sft, it is crucial to consider the influence of cultural diversity on language-specific data, while also integrating language-agnostic universal data that is related to the general abilities of llms (i.e., math reasoning). in this work, we propose a data construction framework consisting of two pipelines, as shown in figure 3 . ## conclusion in this work, we propose a knowledge-grounded data augmentation method and a two-stage translation mechanism to construct language-specific and language-agnostic multilingual sft data, respectively. experiments demonstrate that the proposed dataset is effective for multilingual llms.