S usu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between G hana and the US (2024)

Abstract: Recent work has highlighted the culturally-contingent nature of commonsense knowledge. We introduce AMAMMERε, a test set of 525 multiple-choice questions designed to evaluate the commonsense knowledge of English LLMs, relative to the cultural contexts of Ghana and the United States. To create AMAMMERε, we select a set of multiple-choice questions (MCQs) from existing commonsense datasets and rewrite them in a multi-stage process involving surveys of Ghanaian and U.S. participants. In three rounds of surveys, participants from both pools are solicited to (1) write correct and incorrect answer choices, (2) rate individual answer choices on a 5-point Likert scale, and (3) select the best answer choice from the newly-constructed MCQ items, in a final validation step. By engaging participants at multiple stages, our procedure ensures that participant perspectives are incorporated both in the creation and validation of test items, resulting in high levels of agreement within each pool. We evaluate several off-the-shelf English LLMs on AMAMMERε. Uniformly, models prefer answers choices that align with the preferences of U.S. annotators over Ghanaian annotators. Additionally, when test items specify a cultural context (Ghana or the U.S.), models exhibit some ability to adapt, but performance is consistently better in U.S. contexts than Ghanaian. As large resources are devoted to the advancement of English LLMs, our findings underscore the need for culturally adaptable models and evaluations to meet the needs of diverse English-speaking populations around the world.

## introduction many datasets have been developed to train and test language models' knowledge and reasoning figure 1 : an example from our dataset which asks about drying herbs. the answer choices show two culturally distinct correct responses where the ghanaian consensus aligns more with sun-drying, while the american consensus leans more towards using a food dehydrator. in this paper, we study the implications of cultural differences between the united states and ghana for developing commonsense benchmark evaluations of english llms. our approach is informed by cultural consensus theory, which posits that the culturally "correct" answer lies in agreement with the shared consensus of the group and reflects what is commonly known by the members of that group @xcite @xcite . this can be understood as a culturally-specific refinement of traditional ai accounts of commonsense knowledge as, roughly, "the stuff everyone knows." in contrast to works that primarily use human annotators as post-hoc validators on synthetically generated data @xcite , this work involves human annotators throughout the entire test set creation process: writing free form answer choices ( §2.4), rating the plausibility of answer choices ( §2.5), and validating the final multiple-choice test items ( §2.6). though this process is costly, we believe it is well-motivated for our goal of creating a test set to probe comparative cultural knowledge in llms, while minimizing (to the extent possible) sources of cultural biases or inaccuracies in the test set itself. our process ensures that all answers choices representing a culture are (1) written by a member of that culture, and (2) represent a consensus among other members of the culture. we present a dataset of 525 multiple choice questions (mcq) containing three question versions shown in figure 1 for a downstream question-answering task. for our work, we define a culturally adaptable model as one that can recognize implicit and explicit cultural cues and respond in a culturally ap-propriate manner, e.g., selecting a culturally appropriate answer choice. models with high degrees of cultural adaptability are desirable as they would be useful to a wider set of users and could better serve a variety of multi-or cross-cultural applications. we conduct experiments to assess whether two encoder models and six llms exhibit cultural biases in two ways: (1) by showing a higher preference for answers consistent with one culture over another when presented with culturally unspecified questions, and (2) by showing higher accuracy for some cultural contexts over others when cultural cues are provided. our main contributions are as follows: ## constructing the amammere dataset we construct the amammere dataset through a multi-stage process outlined in figure 2 , and described in detail throughout §2. in the first stage we select and disambiguate mcq questions ( §2.2) from existing commmonsense datasets ( §2.1), covering a range of topics. next, we collect free-form answers (correct and incorrect) to these questions ( §2.4) by surveying separate pools of ghanaian and us annotators (appendix h). in a second survey, annotators from each pool rate mcq answers on a 5-point plausibility likert scale ( §2.5), used to select a "correct" answer and a set of distractors. the newly constructed mcq is then presented in full to annotators from each pool for final validation ( §2.6). our use of likert ratings to select correct and incorrect choices in mcq construction is motivated by our finding (in parallel work) that commonsense mcqs test items exhibit low agreement when the gap in plausibility ratings between the "correct" and distractor answers is small @xcite . ## experiment we experiment with bert-base @xcite , roberta-base @xcite and other generative llms across the setups below. we finetuned these encoder models on a consolidated version of the original datasets and run our evaluations. we also evaluated, in zero-shot setting, six state-ofthe-art instruction-tuned llms: llama2 @xcite with three different model sizes (7b, 13b and 70b), llama3-70b, mistral-7b-instruct-v0.2 @xcite ) and google's gemma-7bit @xcite . we used three different prompt types for these llms as detailed in appendix b. see more experiment details in appendix e. ## results and discussion we evaluate models across our experimental setups to understand their inherent biases and cultural sensitivities. we present the models' preferences for an unspecified cultural setting in the all-answers configuration in ## conclusion we create a new test set to investigate cultural biases in models, focusing on ghana and the us. the focus on the ghanaian population, a harder-tostudy yet important group, underlines the importance of representing diverse cultures in computational research. our study offers multi-stage annotator participation throughout the dataset creation, ensuring we capture nuanced cultural specificities. our experiments show that models favor us culture in unspecified cultural contexts and specifying cultural cues improves accuracy for both groups, albeit, the improvement is more pronounced for the us. these findings highlight the need for ongoing efforts to make models more culturally inclusive, particularly for low-resource languages. mitigation techniques should involve annotators from diverse backgrounds at all stages of dataset creation, and training models tailored to specific cultures.