I ndo C ulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven I ndonesian Provinces (2024)

Abstract: Although commonsense reasoning is greatly shaped by cultural and geographical factors, previous studies have predominantly centered on cultures grounded in the English language, potentially resulting in an Anglocentric bias. In this paper, we introduce IndoCulture, aimed at understanding the influence of geographical factors on language model reasoning ability, with a specific emphasis on the diverse cultures found within eleven Indonesian provinces. In contrast to prior work that has relied on templates (Yin et al., 2022) and online scrapping (Fung et al., 2024), we create IndoCulture by asking local people to manually develop a cultural context and plausible options, across a set of predefined topics. Evaluation of 27 language models reveals several insights: (1) the open-weight Llama–3 is competitive with GPT–4, while other open-weight models struggle, with accuracies below 50%; (2) there is a general pattern of models generally performing better for some provinces, such as Bali and West Java, and less well for others; and (3) the inclusion of location context enhances performance, especially for larger models like GPT–4, emphasizing the significance of geographical context in commonsense reasoning.1

## introduction the reasoning abilities of multilingual language models are frequently evaluated using english texts, potentially amplifying an anglocentric bias toward culture grounded in the english language, and leading to less inclusive models @xcite . cultures, however, vary significantly from one location to another and profoundly shape the way speakers of a language reason @xcite . recent evaluations of models' commonsense reasoning ability @xcite @xcite have been conducted on english datasets such as social iqa @xcite and piqa @xcite , and thus often overlook geographical aspects, thereby risking cultural bias. culture is a multifaceted concept encompassing the way of life @xcite , including our thoughts and actions @xcite . it includes tangible elements like food, art, and clothing, as well as intangible aspects such as ideas, values, attitudes, and norms. culture is shaped by geographical location and ethnicity, influencing the commonsense reasoning of people within a region. for example, in indonesia, it is culturally acceptable to eat rice with your hands but it is considered unusual to use chopsticks. similarly, at traditional indonesian weddings, it is common to sit on the floor while eating, whereas this practice is less common in australia. this work focuses on understanding the influence of geographical contexts in cultural commonsense reasoning, with the main focus on indonesian culture. indonesia is a highly multicultural country @xcite , home to over 1,300 recognized ethnic groups and more than 700 languages @xcite . as the targest archipelagic country in the world, indonesia has a population exceeding 270 million spread across 38 provinces, stretching from aceh province in the west to papua province in the east. few prior studies on commonsense reasoning in indonesian contexts @xcite @xcite have explicitly addressed the geographical nuances and rich diversity of indonesian cultures. this paper introduces indoculture, a novel dataset to evaluate cultural reasoning in eleven indonesian provinces, manually developed by local people in each province based on predefined topics. in prior work, cultural reasoning has primarily relied on datasets constructed through templates @xcite , and online scraping @xcite . while these studies offer valuable insights, they may be susceptible to training data contamination when used to assess large language models (llms). for instance, @xcite reported a zero-shot accuracy of 92% when using chatgpt @xcite to evaluate low-resource data. indoculture contains cultural commonsense knowledge data from eleven provinces in indonesia (blue colored in figure 1 ), namely, aceh, north sumatra, west sumatra, west java, central java, east java, bali, south borneo, east nusa tenggara (ntt), south sulawesi, and papua. these provinces span breadth of indonesia, each representing a major island in the country, with the addition of bali and ntt. figure 1 also shows three examples in indoculture for three provinces: aceh, north sumatra, and papua. 2 the first example focuses on cultural artifact, specifically, the traditional wedding dress from aceh. the second example examines family relationships while the third example focuses on cultural beliefs and norms regarding pregnancy in papua. can large language models effectively reason based on the diverse cultures of indonesia? to capture the rich diversity of indonesian cultures, we predefined 12 fine-grained topics as guidelines for data construction. figure 2 displays the topic distribution in indoculture, with the majority focusing on food, weddings, art, pregnancy and children, and family relationships. additionally, we also pose the question: is there any influence of geographical location on the commonsense reasoning of language models? we address these questions through comprehensive experiments across different language models, incorporating several levels of location granularity as additional context in the prompt. our contributions can be summarized as follows: figure 1 : indoculture covers eleven provinces spanning from eastern to western indonesia. the highlighted regions in the map represent the provinces examined in indoculture. we present examples from aceh, north sumatra, and papua, with three plausible options and correct answers indicated in bold. english translations are provided for illustrative purposes. cultural commonsense reasoning with geographical contexts previous studies have explored commonsense reasoning with geographical context. shwartz (2022) investigated time perception (e.g., morning and night) across different locations, while yin et al. (2022) examined cultural knowledge of language models across five countries using datasets built from templates and translations. other work has focused on automatically extracting cultural knowledge from various sources, including wikipedia @xcite , conversations @xcite , and common crawl @xcite , incorporating location context with the assistance of llms. relatedly, @xcite created a knowledge bank for situational norms, using english-speaking mechanical turk annotators and incorporating a country taxonomy. unlike this work, indoculture specifically concentrates on cultural reasoning across indonesian provinces, developed and validated manually by local people (experts). compared to the automatic method and english-speaking crowd workers for data construction, indoculture arguably contains less noise, and is free from the training data contamination of llms. commonsense reasoning with indonesian contexts as illustrated in figure 1 , indoculture is a sentence completion task in the indonesian language featuring a one-sentence premise, three plausible options, and one correct option to evaluate reasoning ability and cultural knowledge across eleven indonesian provinces. while sentence completion tasks are straightforward for humans, answering indoculture requires machines to engage in cultural reasoning to logically conclude which of the three options is logically consistent with the first sentence @xcite . the dataset includes a total of 2,429 instances. ## discussion a recent study @xcite has demonstrated that evaluating language models using token probabilities in multiple-choice question types does not align well with the generated text. this discrepancy is reported to be more pronounced in models fine-tuned on conversational or safety data. in response to this issue, we conducted a manual evaluation of 200 random samples (in section 4.3) and found that the performance in the generated text, especially for merak-the best open-weight indonesian-centric language model-deteriorates. however, this issue is less apparent in larger models such as llama-3 and gpt-4. conducting manual evaluation on all data and models is expensive, and we plan to address this issue in future works. this work primarily focuses on introducing a novel dataset constructed for evaluating cultural commonsense reasoning within the indonesian context, including preliminary evaluation results based on standard methods used in previous studies @xcite @xcite @xcite . ## conclusion indoculture is a cultural commonsense reasoning dataset encompassing the diversity of indonesian cultures, spanning from aceh province in the west to papua province in the east. through collaboration with local individuals across eleven provinces and rigorous quality control measures, we introduce indoculture for the purpose of evaluating language models. our findings reveal that large language models, whether indonesian-centric or multilingual, demonstrate a limited understanding of indonesian cultures. notably, incorporating location as additional context significantly enhances model performance, particularly for gpt-3.5 and gpt-4.