Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in I ndonesian and S undanese (2024)

Abstract: Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in 4.5K questions per language (9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally ‘deep’ as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.

## introduction the development of large language models (llms) is significantly impacting nlp, leading to an increasing trend in the automated generation of datasets, particularly for question answering (qa) tasks. however, a major challenge arises with underrepresented languages like indonesian and sundanese due to the need for cultural context. for the generated data to be fully useful, it must not only be linguistically accurate, but it also needs to reflect the cultural nuances, historical references, and social norms. it is not yet clear whether current llms can create qa data that adequately includes the cultural nuances specific to certain languages. another common method for constructing non-english datasets is by using machine translation. although more scalable, this method cannot be straightforwardly applied due to the contextual irrelevancy of the data, primarily due to geographical differences (which could also influence cultural differences). for example, many english common-senseqa questions @xcite include concepts such as snow or any winter sports, which are irrelevant in indonesia due to its tropical climate with only two seasons. this dataset also often includes english-centric names and locations, primarily limited to the us, which are not considered "commonsense" for indonesians. therefore, in this study, we aim to investigate how well current llms can adapt and generate a commonsense qa dataset that is both linguistically accurate and culturally relevant to indonesia. we focus on indonesian, the lingua franca of indonesia, and sundanese, one of the regional languages in indonesia with 34 million speakers, yet still considered low-resource @xcite . to address dataset scarcity, especially in sundanese, we also manually constructed datasets by involving annotators from various regions in indonesia, ensuring a representation of diverse cultural perspectives. to sum up, our contributions are as follows: commonsenseqa dataset using various methods (figure 1 ), including adapting existing english data (llm_adapt) and generating new datasets manually (human_gen) and automatically @xcite . the dataset contains ∼3k llm-generated and ∼1.5k humangenerated question-answer pairs per language, which, to our knowledge, is the largest culturally nuanced commonsense qa dataset in both indonesian and, particularly, sundanese. • we perform a comprehensive analysis to assess the effectiveness of llms in creating a culturally relevant commonsense qa dataset. we find that adaptation from english data is less effective, especially for sundanese. however, gpt-4 turbo can generate questions with some basic local knowledge in indonesian and sundanese when provided with human-created categories and concepts. other datasets like arc @xcite , open-bookqa @xcite , and mcscript @xcite test commonsense reasoning via qa, but some questions require grade-school science knowledge. commonsenseqa @xcite presents the type of "purely" commonsense qa in the form of multiple-choice questions based on knowledge from conceptnet @xcite , built through crowdsourcing. however, the crowdworkers bring their cultural background as part of their common knowledge in building the data, resulting in many questions that are "commonsense" only in the western culture. ## data generation methods to investigate whether llms can generate culturally relevant commonsense qa data in indonesian and sundanese, we build a dataset using various methods with llms as data generators. we also employ humans to generate data for comparison. as illustrated in figure 1 , we apply three dataset generation methods: (1) automatic data adaptation (llm_adapt), where we leverage llms to automatically adapt english commonsenseqa data to our target languages; (2) manual data generation (human_gen), where we ask nativespeaker human annotators to manually construct the dataset; and (3) automatic data generation (llm_gen), where we use llms to generate data based on the list of categories and concepts used in human_gen. while all prompts given to the llm are in english, for the llm_adapt data, we specifically instruct the model to produce responses directly in indonesian. subsequently, we translate the resulting indonesian data into sundanese. in contrast, for the llm_gen data, we instruct the model to give the output directly in indonesian and also sundanese, to closely replicate the data generation process used for human_gen. the generation methods for each dataset are detailed below. ## conclusion in this study, we introduced new indonesian and sundanese commonsenseqa datasets built using various methods: automatic dataset adaptation, direct generation with llms, and manual dataset creation by human annotators from diverse regions. our thorough analysis reveals that adapting existing english data is less effective for sundanese. in contrast, direct generation in the target languages by gpt-4 turbo produces relevant data for both languages, although the cultural depth has not yet matched the human-generated ones. nevertheless, given the limited resources, combining llms and humans for dataset creation can be a practical solution, allowing for more efficient dataset creation by reducing the over-reliance on a single data source. our goal is for llms to serve as beneficial tools for diverse communities, not solely those from higher-resource languages, cultures, or regions. this is especially important since llms are continuously being used to generate data. we hope our work represents a crucial step toward achieving this broader objective.