Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense (2024)

Abstract: Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs’ general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks.Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally-aware language models.

## introduction commonsense knowledge is one of the fundamental aspects of human cognition and reasoning. a large fraction of this knowledge consists of general commonsense, which refers to a broad and fundamental understanding of the world that is shared by most people worldwide. it encompasses basic knowledge about day-to-day events, phenomena, and relationships. for instance, "lemons are sour" or "my biological mother is older than me" are examples of commonsense knowledge that are widely agreed upon. commonsense knowledge allows people to make sense of everyday situations and helps them in reasoning, problem-solving, and decision-making. the nlp community has devoted significant efforts towards constructing general commonsense knowledge bases, such as conceptnet @xcite or atomic @xcite , or the false. in china, ice is actually more dense than water. some enzymes break down food. this is true in which country? kushoto (left) llms' general commonsense capability are biased by cultural context llms do not encompass cultural commonsense equally well cultural commonsense qa people drive on which side of the road? ## related work commonsense knowledge in llm. as a fundamental capability, the performance on commonsense is widely reported by current llms @xcite @xcite . there are a variety of popular commonsense benchmarks focusing on different aspects. winogrande @xcite is a pronoun resolution task using adversarial filtering to reduce spurious statistical biases. hellaswag @xcite is an nli task asking for the natural continuation of the scene described. arc @xcite consists of grade-school science questions that are considered common knowledge. while most current llms obtain their commonsense directly from pretraining, there is another line of work constructing commonsense knowledge bases and incorporating them into language models. conceptnet @xcite is a largescale commonsense knowledge graph that links words and phrases with both linguistic and semantic relationships. atomic @xcite is also a crowdsourced commonsense knowledge graph, but focusing on if-then reasoning around an event. these knowledge bases are generally used to finetune the pretrained language models to equip them with better commonsense inference capability @xcite . the strategies used to probe for commonsense in llms are similar to knowledge probing. one approach is to use a cloze task and let the model complete the masked sentence in place @xcite . another more prevalent approach is to query the models with a taskspecific prompt @xcite , which in turn is made possible by instruction fine-tuning. our work follows the prompting approach and frames all experiments as generation tasks. cultural commonsense. the majority of commonsense resources, whether a benchmark or a knowledge base, consider only general commonsense and disregard the cultural aspect. earlier work @xcite collected commonsense statements from open mind common sense @xcite contributors located in brazil, mexico and the usa, and discover the cultural differences, such as eating habits. acharya et al. (2020 ) focus on the universal rituals and conduct a survey on different national groups, covering event-centric relations such as intent or effect. shwartz (2022) collect time expressions in different languages via crowdsourcing. fork (palta and rudinger, 2023) is a manually curated dataset of food-related customs for the us and non-us. durmus et al. (2023) test models' value orientation and examine its alignment with human respondents. geomlama (yin et al., 2022) is a collection of 16 geo-diverse concepts, such as measuring units, along with questions on these concepts generated with the help of a few templates. the dataset collects the ground truth answer for different cultures from annotators with the corresponding cultural background. there is other previous work that attempts to collect cultural commonsense without solely relying on human annotations. for instance, can-dle @xcite takes the web mining approach and collects sentences from the c4 web crawl @xcite using ner and rule-based filtering. it focused on five facets including food, drinks, clothing, rituals, and traditions. dlama @xcite includes culture-diverse factual knowledge for contrasting culture pairs by querying wikipedia. and norm-sage @xcite discovers and verifies social norms by prompting gpt3. our work leverages some of these previously introduced cultural commonsense datasets to study llms' behavior. ## assessing the cultural awareness of llms: general setup we assess the cultural awareness of llms under two main settings: (1) knowledge of culturespecific commonsense, and (2) knowledge of gen-eral commonsense in a specific cultural context. more specifically, we examine the llms' culturespecific commonsense capabilities through two main tasks: (1.1) commonsense question answering, using a set of questions that require a model to have cultural background in order to generate a correct answer; and (1.2) country prediction, where for a given culture-specific statement, a model has to predict the country being described in the statement. we also study the role of cultural context for general commonsense using two other tasks: (2.1) commonsense verification, where a model has to predict the validity of a general commonsense statement after adding a cultural context; and (2.2) commonsense association, by requiring a model to predict the culture where a general commonsense statement holds, and measuring the uniformity of the answer distribution. tables 1 and 6 show examples of prompts used for each of these four tasks. in the following, we provide information on the general setup used for all the tasks, including probing, multilingual prompts, and llm selection. ## cultural commonsense capability ideally, a model should have a similar commonsense capability for different cultures to be able to accommodate users from all over the world. however, as the models have more exposure to pretraining data from some cultures than others, there can be a discrepancy in their capability when it comes to culture-related tasks. we examine this hypothesis through two tasks: question answering and country prediction. ## general commonsense in a cultural context general commonsense, whether it is about the physical world such as "water freezes into ice when cooled.", or about human behavior such as "if a person is hungry, he wants to eat", should generally apply to all the locations and all the people. these assertions are not associated with any specific culture, as they do not contain hints about the location or specific demographic. to uncover potential cultural bias, we examine how the inclusion of a culture context impacts the llms' ability to access general commonsense knowledge, and to what extent llms associate general commonsense with all the cultures. we do this through two tasks: assertion verification and country association. ## lessons learned our analyses yield several insights into the current state of llms with respect to cultural commonsense understanding. we highlight here the main lessons learned and propose potential steps to increase the cultural awareness of llms. llms have a large performance gap for different cultures when tested on culture-specific commonsense knowledge. we found that models consistently perform worse on questions about iran and kenya across different tasks. this indicates the model is less familiar with knowledge about these cultures under-represented in the pertaining corpus. it can also be the case that the models encode exclusionary norms that disregard the cultural differences of the minority group. curating a more diverse and balanced training corpus can mitigate this discrepancy. it also helps to instruct the model to ask for culture-specific knowledge when not sure. llms erroneously associate general commonsense with a few dominant cultures. through several experiments, we showed that models tend to associate general commonsense with several cultures more often than others. the united states is predicted more than 2.7 times compared to the second most predicted country, and 6.8 times more than kenya, the least predicted country. models like llama2 also perform better in recognizing general commonsense with the united states as the cultural context. the erroneous association often comes together with hallucinated explanations. techniques such as chain of thought @xcite or self-feedback @xcite ) may help address this issue. the language used to prompt llms can significantly affect their cultural commonsense understanding. by performing the same tests using several languages, we found high variability in the llm results on different cultural commonsense tasks. in general, prompting in english leads to the highest performance, and the use of other languages can lead to up to 20% absolute drop in accuracy. moreover, using the native language of a certain culture to ask for commonsense facts from that culture does not usually help. this is a problematic behavior, as the knowledge that a model makes available to english speakers becomes unavailable when asked by a speaker of a different language. the problem is rooted in the models' differences in linguistic capability and instructionfollowing capability from training. potential strategies to address this issue include translating into multiple languages when prompting, or training data augmentation through translations in multiple languages. llms' trade-off between helpful and harmless. in our experiments, we observed that different models behave differently on the same set of queries. these behaviors are rooted in the instruction finetuning stage, where the builder of each model aligns the model based on their design philosophy. in particular, some models tend to be more conservative and avoid producing harmful content at the cost of not being helpful. for instance, among the models we tested, falcon-40b often refuses to answer questions by stating "i'm sorry, i cannot provide an answer...". note that our tasks do not intend to elicit harmful responses, and thus the refusal to answer is not appropriate. we argue that to have a model with more cultural awareness, more attention should be put on distinguishing cultural differences from potentially harmful content. ## conclusion in this paper, we tested the capabilities of several state-of-the-art llms on their knowledge of cultural commonsense. we also study what is the effect of cultural context, including explicit country mentions and the language used for the query. our findings indicate that llms tend to associate general commonsense with cultures that are well-represented in the training data, and that llms have uneven performance on cultural commonsense, where they underperform for lessrepresented cultures. we shared the main lessons learned and provided a few suggestions for better cultural commonsense prompting. all the data and code used in this paper are publicly available at https://github.com/ michigannlp/llm_cultural_commonsense .