Culturally Aware Natural Language Inference (2023)

Abstract: Humans produce and consume language in a particular cultural context, which includes knowledge about specific norms and practices. A listener’s awareness of the cultural context is critical for interpreting the speaker’s meaning. A simple expression like *I didn’t leave a tip* implies a strong sense of dissatisfaction when tipping is assumed to be the norm. As NLP systems reach users from different cultures, achieving culturally aware language understanding becomes increasingly important. However, current research has focused on building cultural knowledge bases without studying how such knowledge leads to contextualized interpretations of texts. In this work, we operationalize cultural variations in language understanding through a natural language inference (NLI) task that surfaces cultural variations as label disagreement between annotators from different cultural groups. We introduce the first Culturally Aware Natural Language Inference (CALI) dataset with 2.7K premise-hypothesis pairs annotated by two cultural groups located in the U.S. and India. With CALI, we categorize how cultural norms affect language understanding and present an evaluation framework to assess at which levels large language models are culturally aware. Our dataset is available at https://github.com/SALT-NLP/CulturallyAwareNLI.

## introduction language, as a tool of social interaction, is used in a cultural context that involves specific norms and practices. cultural norms are behavioral rules and conventions shared within specific groups, connecting cultural symbols and values @xcite . they provide contextual knowledge to interpret the meanings behind actions and words. for example, when tipping is customary, a simple expression like "i didn't leave a tip" implies a strong sense of dissatisfaction, while in a culture where tipping is optional, the implicature no longer holds. as nlp systems reach billions of users across cultural boundaries, building culturally aware language models is an emerging requirement @xcite . by the term "culturally aware", we specifically refer to three levels of awareness in language understanding: (1) having knowledge of specific cultural norms; (2) recognizing linguistic context that invokes cultural norms; (3) accommodating cultural variations by making a culture-specific inference. a lack of cultural awareness can result in models with poor accuracy and robustness @xcite , social biases and stereotypes @xcite @xcite , and discrimination against annotators @xcite . to incorporate cultural factors in nlp systems, recent work has built knowledge bases of cultural symbols, practices @xcite @xcite @xcite , and values @xcite @xcite , along with methods to probe or elicit such knowledge from llms. in this work, we take a step towards culturally aware language understanding. unlike previous work focusing on building knowledge bases of cul-tural norms and practices-level (1), we focus on how people make different inferences based on cultural knowledge, i.e., level (2) and (3). we operationalize cultural variations in language understanding through an nli task that measures cultural variations as label disagreement between annotators from different cultural groups. we introduce cali, the first culturally aware natural language inference dataset foot_0 , with 2.7k premisehypotheses pairs. each premise is centered around a type of normative behavior. we ask annotators from two different cultural groups, one based in the united states and the other based in india, to label the entailment relationships in the context of their culture. the label variations between the two groups capture the cultural variations in language understanding. with the new dataset, we categorize how cultural norms affect language inference (section 4) and evaluate at which levels large language models are culturally aware (section 5). we find despite llms having knowledge of many cultural norms, especially ones associated with the u.s. culture, they lack the ability to recognize such norms in the nli task, i.e., level (2), and adjust inference based on cultural norms, i.e., level (3). our work highlights that cultural factors contribute to label variations in nli and cultural variations should be considered in language understanding tasks. ## a culturally aware nli dataset we introduce a new nli dataset that measures cultural variations in language understanding through label variations of the nli task. we choose nli as a starting point, as it is one of the most fundamental and widely used natural language understanding tasks. the dataset generation framework is shown in figure 2 . ## analyzing cultural variations with cali, we empirically investigate (1) how cultural knowledge leads to label variations in the nli task and ( 2 ) what are other factors that contribute to label variations between the two cultural groups. ## evaluating language models we first investigate how well models fine-tuned on existing nli datasets perform on cali. specifically, whether entailment features learned from other datasets are transferable to cali and whether these features are biased towards a particular cultural context. then, we use the three-level cultural awareness defined in section 1 and the categories developed in section 4.1 as an evaluation framework to assess gpt-3.5/4 cultural awareness on nli tasks, where we ask models to make culturespecific inference by prompting models with explicit cultural indicators. ## conclusion we present a framework to study cultural awareness in natural language understanding. we define cultural awareness at three levels: (1) having knowledge of cultural norms, (2) recognizing cultural norms in context, and (3) making culturalspecific inferences. our work focuses on level (2) and (3). we operationalize cultural variations in language understanding through an nli task and contribute the first culturally aware nli dataset, cali. with our dataset, we categorize how cultural norms influence language understanding and present an evaluation framework to assess at which levels llms are culturally aware. we show despite having knowledge of cultural norms, llms still lack awareness on levels (2) and (3). our work highlights that cultural factors contribute to label variations in nli and cultural variations should be considered in language understanding tasks. we advocate for the community to incorporate cultural awareness in nlp systems in future research.