CRAFT : Extracting and Tuning Cultural Instructions from the Wild (2024)

Abstract: Large language models (LLMs) have rapidly evolved as the foundation of various natural language processing (NLP) applications. Despite their wide use cases, their understanding of culturally-related concepts and reasoning remains limited. Meantime, there is a significant need to enhance these models’ cultural reasoning capabilities, especially concerning underrepresented regions. This paper introduces a novel pipeline for extracting high-quality, culturally-related instruction tuning datasets from vast unstructured corpora. We utilize a self-instruction generation pipeline to identify cultural concepts and trigger instruction. By integrating with a general-purpose instruction tuning dataset, our model demonstrates enhanced capabilities in recognizing and understanding regional cultural nuances, thereby enhancing its reasoning capabilities. We conduct experiments across three regions: Singapore, the Philippines, and the United States, achieving performance improvement of up to 6%. Our research opens new avenues for extracting cultural instruction tuning sets directly from unstructured data, setting a precedent for future innovations in the field.

## introduction large language models (llms) like chat-gpt @xcite , claude, and gemini @xcite have demonstrated their proficiency in managing diverse tasks related to semantic understanding and text generation. beyond acting as general task-solvers, the ability of llms to understand and reason with cultural nuances could play a crucial role in generating precise and personalized responses to benefit broader communities @xcite @xcite . culture is a comprehensive concept encompassing traditions, customs, beliefs, values, and social norms, all deeply rooted in historical contexts and continuously evolving over time. it is also intrinsically linked to languages and dialects, which can be sparsely represented in available resources. in the domain of llms, which initially train on vast amounts of unlabeled data, knowledge is systematically captured and structured through data-driven techniques. with limited model sizes, knowledge that occurs infrequently is often less effectively captured compared to more frequently occurring information @xcite . additionally, the predominance of english in the pretraining corpus inherently biases these models towards western perspectives, a consequence of the over-representation of english-language sources. this bias means that cultural concepts may be inadequately captured, especially for under-represented regions @xcite . consequently, llms struggle to effectively adapt to and represent diverse cultural concepts due to these inherent limitations in their training data. expanding the cultural reasoning capabilities of llms could potentially be achieved by pre-training them on corpora from diverse languages. however, this approach is still expansive and challenging due to the difficulty in obtaining high-quality multilingual datasets @xcite . meanwhile, instruction fine-tuning could more directly impact end-user applications. however, the development of cultural instruction tuning sets is limited due to the high costs associated with collecting culturally relevant instruction sets, along with challenges in ensuring their quality and diversity. in this study, we pioneer the study of deriving instruction tuning sets from unlabeled corpora. initially, we use keyword filtering to isolate culturally relevant concepts from a vast corpus containing over 600 billion english tokens. subsequently, we then utilized these selected regional text segments to prompt llms for both questions and answers. ## methodology we introduce the craft (cultural reasoning with instruction fine-tuning) method, designed to synthesize cultural instructions from a massive, unlabeled english corpus. the methodology is detailed in the following steps, as illustrated in figure 1 . selective data extraction. we utilize slimpajama @xcite as our primary data source, which comprises an english corpus containing over 600 billion tokens. given the sparse distribution of culturally relevant concepts within this vast dataset, and to manage the processing burden effectively, we propose an efficient filtering process using keywords to identify and extract culture-related concepts. specifically, we curate a collection comprising a minimum of 150 words to represent each region. we then segment the documents into chunks no larger than 512 tokens each. from these segments, we retain only those text chunks that include at least two regional keywords, such as "national day parade" and "merlion" for singapore. through analyzing a subset of over 200 billion english tokens, we successfully extracted 35,000 text segments for singapore, 25,000 for the philippines, and 35,000 for the us. ## experiments settings. for evaluation, we selected four datasets, including three culturally-focused ones: sg-eval, ph-eval, and us-eval @xcite . each of these datasets is a human-crafted collection of multiple-choice questions and answers that probe cultural or regional knowledge. to assess knowledge capabilities in general subjects, we utilize a subset of the mmlu dataset @xcite . to minimize the influence of randomness, we report the average results from five runs, using paraphrased prompt templates. ## conclusion in this paper, we introduce the craft method, designed to synthesize cultural instructions from a vast, unlabeled corpus. we conduct experiments across three regions, with the potential for expansion to additional regions. our pioneering selfinstruction techniques facilitate effective mining from unstructured data sources, enhancing both the diversity and quality of the synthesized instructions compared to previous studies.