C ulture I nstruct: Curating Multi-Cultural Instructions at Scale (2025)

Abstract: Large language models, despite their remarkable success in recent years, still exhibit severe cultural bias. Therefore, in this paper, we introduce CultureInstruct, a large-scale instruction-tuning dataset designed to reduce cultural bias in LLMs. CultureInstruct is constructed with an automatic pipeline, utilizing public web sources and a specialized LLM to generate instruction. Our data comprises 430K instructions, ranging from classic NLP tasks to complex reasoning. CultureInstruct also covers 11 most relevant topics to cultural knowledge, making it highly diverse. Our experiments show that fine-tuning LLMs with CultureInstruct results in consistent improvements across three types of cultural benchmarks, including (i) general cultural knowledge, (ii) human opinions and values, and (iii) linguistic cultural bias. Our best model, Qwen2-Instruct 72B + CultureInstruct, outperforms GPT-4o Mini and GPT-4o with 18.47% and 13.07% average relative improvements on cultural benchmarks.

## introduction natural language processing (nlp) has seen drastic development in recent years, driven largely by advancements in large language models (llms). these models have demonstrated remarkable capabilities in understanding and generating human language across a wide array of tasks, from questionanswering @xcite to complex reasoning @xcite . however, a critical challenge of llms remains unsolved: the integration of cultural knowledge into these models to ensure they operate unbiased across diverse linguistic and cultural contexts. in addressing the problem of cultural biases, some works have attempted to actively mitigate these biases and improve the performance of llms on culturally diverse tasks. researchers have explored various methods, such as prompt engineering with culturally specific context @xcite @xcite to dynamically steer models toward generating culturally appropriate responses in real-time applications. however, prompt engineering is notorious for its unreliability across different downstream tasks. several works tried to continue pre-train llms on non-english languages @xcite but this approach is time-consuming and very costly, despite its effectiveness. thus, it is better to fine-tune llms with instructions related to cultural knowledge. recent works have proposed instruction-tuning datasets that are culture-specific, but they are either limited to a small set of instruction tasks @xcite or only cover one aspect of cultural knowledge, such as cultural norms @xcite and human values @xcite . these datasets, despite their high value in mitigating cultural biases, are only able to solve specific challenges within limited domains of cultural understanding. this narrow focus restricts the generalization potential of llms, making them less capable of handling a wide range of cultural contexts. to bridge this gap, we present culturein-struct, a large-scale instruction dataset of cultural knowledge. cultureinstruct covers various topics related to cultural knowledge, including cultural norms, human values, history, cuisine, art, and more. to construct the dataset, we propose to generate cultural instructions automatically from web documents. specifically, we implement a four-step pipeline to produce cultureinstruct (figure 1 ). at the first step -document filtering, we select culture-relevant documents from dolma @xcite based on keyword filtering and apply a fasttext model @xcite trained on an aggregated set of multiple public cultural datasets to further filter the documents. secondly, at the instruction generation step, we utilize an open-source llm -bonito @xcite and tune it to generate instruction samples from the collected documents. this set of samples then goes through the third step -instruction diversification. in the final step, data decontamination, we remove samples that closely resemble any of the evaluation sets. we validate the effectiveness of cultureinstruct by fine-tuning multiple series of llms, including llama-3.1-instruct (meta, 2024), mistral-0.3-instruct @xcite , qwen2-instruct @xcite , olmo @xcite , and gemma2-instruct @xcite . fine-tuning with culturein-struct outperforms the base llms significantly on multiple types of cultural benchmarks, including: (i) benchmarks for general cultural knowledge -culturalbench @xcite , candle @xcite , and eticor @xcite , (ii) human values -globalopinionqa @xcite ; (iii) linguistic cultural bias -camel @xcite . these benchmarks are particularly relevant because they assess models on diverse aspects of cultural understanding. by evaluating models on such culturally sensitive tasks, we ensure that the fine-tuned models demonstrate not only general competence but also cultural inclusivity and fairness in global contexts. in summary, we make the following contributions: ## data construction in this section, we describe the pipeline for constructing cultureinstruct. as illustrated in figure 1 , the proposed pipeline consists of four processes: (1) selecting culture-relevant documents from dolma @xcite , ( 2 ) generating instruction data from the collected documents, (3) diversifying the instructions, and (4) decontaminate the data from the chosen cultural benchmarks. ## experiments in our experiments, we investigate to what extent cultureinstruct improves the performance of various llms on cultural benchmarks, covering three aspects: (i) general cultural knowledge and cultural norms, (ii) linguistic cultural bias, and (iii) human values. ## conclusions in this paper, we introduce cultureinstruct, an instruction-tuning dataset designed specifically for improving cultural knowledge and reducing cultural bias of llms. cultureinstruct consists of 430k meticulously-crafted instructions, covering 11 most relevant topics to cultural knowledge. our experiments show that by fine-tuning with cultureinstruct, various llms show consistent improvements on three types of cultural benchmarks, namely general cultural knowledge, linguistic cultural bias, and human values. in the future, we will extend the dataset to cover more instruction tasks that are specific to cultural knowledge, as well as extending cultureinstruct to multilingual settings.