X -Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024)

Abstract: Large language models respond well in high-resource languages like English but struggle in low-resource languages. It may arise from the lack of high-quality instruction following data in these languages. Directly translating English samples into these languages can be a solution but unreliable, leading to responses with translation errors and lacking language-specific or cultural knowledge. To address this issue, we propose a novel method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. Specifically, the language model first learns to generate appropriate English instructions according to the natural web texts in other languages as responses. The candidate cross-lingual instruction tuning samples are further refined and diversified. We have employed this method to build a large-scale cross-lingual instruction tuning dataset on 10 languages, namely X-Instruction. The instruction data built using our method incorporate more language-specific knowledge compared with the naive translation method. Experimental results have shown that the response quality of the model tuned on X-Instruction greatly exceeds the model distilled from a powerful teacher model, reaching or even surpassing the ones of ChatGPT. In addition, we find that models tuned on cross-lingual instruction following samples can follow the instruction in the output language without further tuning.

## introduction large language models illustrate a series of remarkable abilities, e.g., generating better response and zero-shot task generalization, after tuning on instruction following samples @xcite @xcite . however, @xcite the instruction sample translated from english ignores the language-specific knowledge that "fine" and "mine" sound different in urdu. (b) the cross-lingual instruction tuning sample generated by our method contains language-specific knowledge from the native corpus. the responses in low-resource languages, which are prone to contain unsafe content @xcite and ignore cultural nuances @xcite , are inferior to the ones in high-resource languages. in addition to the limited pre-training corpus in these languages, it may come from the lack of highquality instruction following data. distillation from the teacher model or manual annotation are common methods to obtain highquality instructions. the former cannot be applied to low-resource languages since the teacher model may respond poorly in these languages. considering the high requirements, including creative thinking and professional knowledge, to write instruction-following samples, it is hard to find suitable annotators that excel in low-resource languages. translating english instruction tuning samples into the target language is also unreliable, which will introduce translation errors, especially in the samples with codes, and overlook the cul-tural nuances between languages. as shown in figure 1 (a), the urdu sample translated is wrong due to the pronunciation nuance between english and urdu. therefore, aligning language models in low-resource languages remains a significant challenge. in the preliminary experiment, we found that llama @xcite ) can understand unseen low-resource language instructions quite well through responding in english, when tuned with limited instruction-following samples, but it performs much worse when replying in its own language ( based on the observation, we first adopt the cross-lingual instruction following format like @xcite , in which the languages of instructions and responses are not the same, to exploit the understanding ability rapidly learned in low-resource languages and better generation performance in english. we further propose an automatic pipeline to excavate and refine cross-lingual instruction tuning data from the unsupervised multilingual corpus, which contains language-specific knowledge and native expressions. specifically, the model first learns from seed data to generate appropriate english instructions for the given texts in other languages, which compose candidate cross-lingual instruction following samples. the candidate samples are refined in an iterative manner. in each iteration, an evaluator is trained through a synthetic rating dataset and finds better cross-lingual samples, which are exploited to improve the evaluator in the next iteration. thus, the quality of samples found is improved with the stronger evaluator. the final cross-lingual instruction tuning dataset is sampled from the different clusters of the remaining data after k-th iteration to increase the diversity. we conducted experiments on 10 languages, which include 5 medium-resource languages and 5 low-resource languages foot_0 , and generated 320k highquality cross-lingual instruction tuning samples. on four instruction-following evaluation benchmarks, the response quality of the base model tuning on our data can surpass those generated from translation and distillation baseline models, and even exceed the ones of @xcite in these languages. in addition, we find that crosslingual instruction tuning models can follow the instruction in the language of output without fur-@xmath0.04 5.50 1.35 7.10 (0-10) to sum up, our contributions are as follows: our work is related to multilingual instruction generation and tuning, which will be briefly introduced below. instruction generation large language models obtain instruction following and better zero-shot task generalization ability after tuning on diverse instruction following samples @xcite @xcite . the key challenge is how to build high-quality instruction following samples that have a great influence on the performance of the model after tuning. distilling from more powerful models @xcite @xcite and human labeling @xcite are main methods to generate high-quality instruction following samples. our work is more similar to the self-align methods @xcite @xcite , which show promising results by iteratively self-generating and filtering instructions without strong models and a lot of manual annotation. however, it cannot be applied to the condition when language models are unable to generate high-quality instructions in unseen languages. in contrast, our method avoids the weakness in the low-resource language generation and exploits superior generation ability in english. ## method as shown in figure 2 , we design a three-step pipeline to automatically excavate high-quality cross-lingual instruction following samples from multilingual web corpora: 1) the base language model is fine-tuned to generate candidate english instructions given multilingual corpus (section 3.2). 2) samples with inappropriate cross-lingual instruction and output are discarded to improve the quality of the dataset (section 3.3). 3) we diversify the cross-lingual instruction data by sampling from the multiple clusters of the english instruction embedding (section 3.4). ## x-instruction dataset we apply the automatic pipeline to a large-scale multilingual corpus using llama-2-7b, and construct a cross-lingual instruction tuning dataset with 320k samples for 10 languages, which is named x-instruction dataset. codes and data will be made public after review to advocate future research. ## conclusion in this paper, we propose a pipeline to excavate cross-lingual instruction-tuning samples from multilingual corpora by exploiting the better generation ability in high-resource languages of large language models. we apply it to 10 languages and construct a large-scale cross-lingual instructions dataset named x-instruction. experimental results from gpt-4 and human evaluation demonstrate that models tuned on x-instruction can generate better responses and follow the instructions in the output language without further tuning. we hope x-instruction can bring more attention to the better instruction construction method for low-resource languages, future work could focus on directly generating high-quality instruction following samples in low-resource languages.