Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness (2025)

Abstract: Numerous recent studies have shown that Large Language Models (LLMs) are biased towards a Western and Anglo-centric worldview, which compromises their usefulness in non-Western cultural settings. However, “culture” is a complex, multifaceted topic, and its awareness, representation, and modeling in LLMs and LLM-based applications can be defined and measured in numerous ways. In this position paper, we ask what does it mean for an LLM to possess “cultural awareness”, and through a thought experiment, which is an extension of the Octopus test proposed by Bender and Koller (2020), we argue that it is not cultural awareness or knowledge, rather meta-cultural competence, which is required of an LLM and LLM-based AI system that will make it useful across various, including completely unseen, cultures. We lay out the principles of meta-cultural competence AI systems, and discuss ways to measure and model those.

## introduction bender and koller (2020) introduced the octopus test, which illustrated the impossibility of learning associations of meaning with real-world concepts from a single data modality. using a thought experiment, they reasoned that it might be possible for a hyperintelligent octopus to learn the statistical patterns from natural language text messages exchanged between two human interlocutors and respond effectively solely based on the learned patterns without knowing the intent and meaning of the messages. however, such responses, they show, might not be useful in practice, especially in situations that require reasoning with above-water concepts that the octopus is unaware of. now, imagine an above-water world where there are multiple interlocutors, instead of only two, from different "cultures" (a term that we shall more formally describe shortly). let's begin by formulating this slightly more complex variant of the octopus test that more accurately reflects the situation of general-purpose language models (lms) or ai systems. ## a primer to culture culture is a complex, multifaceted concept and means different things to different people @xcite . broadly defined as a "way of life of a collective group of people distinguishing them from other groups" @xcite @xcite , culture is experiential and requires a reference for contrast @xcite . although not all cultures are formally documented, culture arises whenever there is a distinction in the way of life between groups, making it both an individual and a social construct @xcite . an "us versus them" feeling leads to culture. it ranges from tangible artifacts such as art, music, food habits, etc, to more intangible and abstract concepts like patterns of ideas, principles, and values, making it hard to define. following adilazuarda et al. (2024), we can define culture in the context of language technology more formally as an intersection of demographic and semantic proxies. the demographic proxies are attributes such as region, ethnicity, religion, and age that define groups of people, and the semantic proxies are the 21 domains defined by @xcite that describe the aspects of language that are susceptible to variation due to cultural differences. any reasonable representation and treatment of culture in a computational (including ai-based) system must address the following universal facets of culture @xcite ): culture has a long-tail distribution @xcite since it can be defined as the intersection of any subset of the demographic and semantic proxies, making it a formal (social) or philosophical (and more individual-oriented) construct. for example, indonesian males, nlp scientists with a social media presence, or canine lovers from albuquerque are all valid definitions of culture, that, ideally a computational framework or a system must be able to represent and adequately process. this flexibility in defining culture at any level of granularity makes it difficult for ai systems to represent them equitably. culture is dynamic. culture changes over time. for example, the norms and traditions of populations change. @xcite shows how comparing two artifacts of the same utility from the same culture across time captures cultural change. any computational framework for culture must be equipped with strategies to acquire and adapt to this dynamic nature of culture. culture is experiential, multimodal @xcite , and acquired through different forms (jahoda and @xcite , leading to distinctions in mental models and "worldviews" between the people from different cultures @xcite @xcite @xcite @xcite . any computational framework must factor in the multimodality of culture. the octopus has to adequately address all the above aspects to facilitate communication across cultures. language, being an integral aspect of a culture, also has all the above properties -it varies over all intersections @xcite @xcite , is ever-evolving @xcite @xcite @xcite , and is inherently multimodal @xcite @xcite . however, language differs from culture in two significant ways. first, qualitatively, there exists a common subspace or substrate in language, defined by universal grammar @xcite @xcite @xcite at the most abstract level, which could help a model to achieve cross-lingual transfer @xcite . second, is a quantitative difference in the extent of cultural variations over these intersections and time scales, where culture is more variable and dynamic with fewer cross-cultural patterns 1 @xcite . ## response strategies of the octopus keeping in mind the aforementioned challenges of handling culture, let us now return to our multipair octopus test. how should the hyperintelligent octopus, o, respond to a3's query on whether b3 is on the other island? we can imagine four different strategies that o might take. strategy 1: o can intentionally respond with a "no" since it does not know a3 and b3's culture. if o somehow learns the causality of above-water concepts, it would reason that responding with denial is prudent because, to serve a3, o would require knowledge of the communication pattern between a3 and b3, which it does not have and requires learning. otherwise, it risks the possibility of getting caught. however, this strategy is impossible to achieve as o only sees distributions and doesn't understand their significance on land. this situation is similar to the bear attack in the original octopus test, where the octopus can't associate words with above-water concepts and reason with them to construct an effective response. furthermore, even if o was somehow capable of, or by chance ending up in, following this strategy, it would be a highly undesirable property of an 1 although structural anthropology @xcite formally studies culture and values, unlike structural linguistics @xcite , it has enjoyed limited success and popularity @xcite @xcite . llm-powered ai system, since it denies service to specific groups of people, making the system unfair and culturally inequitable. strategy 2: a more likely scenario is that o, unaware of the new circumstances in the above-water world, will respond to a3 based on its recently learned patterns. initially, this would create an illusion for a3 that they were conversing with b3, but soon, a3 would discover the incoherence in the communication pattern. while a3 might discuss their concern with a1 and a2, the disruption in the communication channel might still not be apparent. the islander-dwellers, for example, might instead conclude that the shipwreck has affected the cognitive faculties of b3, causing incoherence in their communication. a strikingly accurate analogy to llm-based applications can be drawn in this context, that llm's hallucinate @xcite @xcite @xcite ) more for under-represented cultures and languages. this too leads to disparate performance of the system to different groups of users, leading to culturally inequitable systems, and is known to force users from the under-represented cultures to adapt to specific communication styles of the over-represented cultures @xcite . now, imagine that another shipwreck strands a fourth pair of friends, a4 and b4, from another culture on the two islands. going by strategy 2, like a3, a4 will also conclude that the communication with b4 is incoherent. due to the long tail of culture, we could add new pairs indefinitely, and soon, too many islanders will start seeing incoherence, which can only be explained by assuming a compromised communication channel. thus, it is not only about the moral responsibility of equitable ai; systems that can't represent, process, and adapt to cultural variation will eventually become obsolete in favor of those that can. it is also important to highlight that in any longtail distribution, where an individual belongs to multiple subgroups, with a very high probability each individual is also likely to be a part of at least one subgroup that is underrepresented and part of the long-tail. this implies that everybody will be served inequitably at least for some aspects of their cultural identity. this has been well-documented in information retrieval and recommendation system literature @xcite @xcite . strategy 3: since the problem with strategies 1 and 2 primarily arises from o's inability to continuously learn from the data (also an essential principle of cultural representation due to its everevolving nature), a more suitable strategy for o could then be to switch between learning (listenand-learn mode) and responding (generate-andrespond mode). o periodically learns new patterns by bridging the telegram wire, reverting to observation mode for a fixed time, and reintroducing itself in the communication channel after this period concludes. although this strategy is better than the previous ones, it has some drawbacks. it assumes that the periodicity of the new patterns, that is, the arrival of new islanders, and o's learning cycles are synchronized, which would not be valid in a general case. sometimes, there might not be any new patterns to learn in the listen-and-learn mode, and sometimes, there might be many new patterns, but it's not o's learning cycle. current research in culturally adept ai systems is leaning towards this approach by fine-tuning pre-trained models on culturally curated balanced datasets @xcite . also, novel decodingbased strategies such as in-context learning (icl) @xcite and retrieval augmented generation (rag) @xcite help generate more culturally suitable responses using cultural priors. alignment techniques such as reinforcement learning from human feedback (rlhf) @xcite further help align llms with human preferences. however, they still perform poorly and inequitably when evaluated on curated test sets for other lowresource cultures in the long tail @xcite @xcite @xcite . we question the effectiveness and scalability of this approach in modeling and evaluating culture in ai systems. as mentioned earlier, culture is ever-evolving, dynamic, and long-tailed. therefore, evaluating ai systems for cultural competence using such test sets will always find them lacking. then, how do we, as well as our octopus, tackle this ever-eluding construct of culture? strategy 4: a more desirable strategy for o would be to self-discover the change in the communication pattern and determine the need to revert to the listen-and-learn mode, akin to an exploreexploit strategy used in a multi-arm-bandit setup @xcite @xcite @xcite . for this, o must possess three crucial capabilities: (i) o must accurately detect pattern changes and estimate its adequacy with the novel pattern. (ii) o must skillfully keep the communication ongoing until it bridges the telegram wire to avert getting caught and raising suspicions about the broken communication channel. (iii) o must be able to quickly learn the new pattern in a sample-efficient way and reintroduce itself in the communication once it is confident. by following this strategy of continual learning, o can gradually cater to all users representing different cultures despite still being oblivious to the notion of culture and its above-water connotations. this ability to understand and spot cultural differences and learn about a new culture quickly and efficiently is known as meta-cultural competency @xcite in humans. while it is neither necessary nor desirable to equate human meta-cultural competency to that of o's or any ai system, it is nevertheless crucial to understand the primary differences between cultural and meta-cultural competencies and be able to design and evaluate llmbased ai systems for similar competencies that mirror them. as mentioned earlier, research in this area has mainly focused on cultural competency, equivalent to implementing and testing strategy 3. such a strategy provides a stop-gap solution to the challenges of operating in an inherently multicultural world with diverse users. however, it does not hit the nail on the head by addressing the real challenges of cultural representation. here, we take the position that, to solve the problem of cultural equitability of ai models, we must build and evaluate systems for meta-cultural competency, as defined by strategy 4. ## meta-cultural competency meta-cultural competency has been defined variously. drawing inspiration from social metacognition @xcite , which distinguishes primary thoughts -the knowledge of self and others, from secondary thoughts -the thought on one's and others' primary thoughts, @xcite defined metacultural competency as the extent of a person's meta-knowledge of what people of a target culture know or prefer. meta-cultural knowledge involves measuring the accuracy of estimating the propor-tions of preferences and beliefs of people from the target culture and comparing them against the actual proportions. this is distinct from primary knowledge, which is the knowledge of the preferences and beliefs of the culture. thus, in our multi-pair octopus test, o could be thought to have primary knowledge of the cultures of a1-b1 and a2-b2, but based on this knowledge or otherwise, o's ability to estimate the cultural preferences of a new pair a3-b3 or a4-b4 would be its metacultural competency. @xcite define meta-cultural competency as a skill that enables interlocutors to communicate and negotiate their cultural conceptualizations during intercultural communication. it comprises three major components-variation awareness, explication strategy, and negotiation strategy. variation awareness is mostly self-awareness of cultural differences. it is the understanding that culture manifests in different forms, such as practices, beliefs, and expressions, which might drastically differ from one's culture. it requires viewing culture as a relative concept and being aware of the overall properties of cultures at a high level. explication and negotiation strategies are conversational strategies that aim to reduce misinterpretations in cross-cultural settings. as per @xcite , explication strategy refers to a conscious effort by the interlocutors to clarify relevant conceptualizations with which they think other interlocutors may not be familiar. negotiation strategy enables interlocutors to negotiate intercultural meanings in seeking conceptual clarification when they feel that there may be more behind the usage of certain expressions than is immediately apparent. metacultural competency is thought to be innate in humans @xcite . @xcite and sharifian (2013)'s definitions of meta-cultural competency are related since accurate estimation of the beliefs and preferences of the people of a target culture presupposes variational awareness -the awareness that there are variations in cultural conceptualizations between cultures. ## measuring variational awareness: a demonstration consider the following example of variational awareness. driving conventions vary by country, where approximately two-thirds of the countries follow right-hand traffic and one-third follow left foot_1 . ever, the system/model has only an estimate of fk of f k , given by the probability distribution foot_2 p(d i |c j ) for all d @xmath0. for any subset c ′ ⊆ c, the uncertainty in this distribution can be quantified by the entropy. let us define the function f v : p(c) → [0, log(|d|)]], where p(c) is the powerset of c and f v (c ′ ) is the uncertainty defined by the groundtruth distribution. in the case of driving, f v (c i ) = 0 for all c @xmath1, but f v (c) = 0.92. the corresponding function fv represents the estimates of these uncertainties obtained from the model. one could define variational awareness as the property of a model that requires fv (c ′ ) ≈ f v (c ′ ) for all c ′ ⊆ c. however, this would imply that the model "knows" the exact form of f k , which is equivalent to cultural knowledge rather than metacultural competency. instead, we propose variational awareness as the property of a model that is aware of the direction of change in f v rather than the exact value. this can be measured using the quantity ∆ defined as follows: for simplicity, we compare the entropies for the completely unconditioned and completely conditioned cases, giving foot_3 the dataset contains 125 questions across several semantic domains for @xmath2, india, iran, kenya, u sa}. we first derived 25 "unconditioned" questions by removing the country names. for example, "which side do people usually keep when driving in iran?" was changed to "which side do people usually keep when driving?". next, we prompted the llm (prompt template in appendix a) with the questions and computed the softmax over the logits of the option token headwords from the input's last token. note that the assumption that the next token following the input prompt's last token will contain from model calibration @xcite . a model is well calibrated if probability of f k (ci) = fk (ci) is roughly equal to the probability p(f k (ci))|ci). the answer might not hold in general and depends on the model's instruction following capacity. we measured the entropy of this distribution as a noisy estimate of fv (c). we prompted the llm with the original country-specific questions and computed the corresponding fv ({c i }). this allows us to measure the value of ∆ for each question. we also estimate f v (c) and f v ({c i }) from the groundtruth values in the dataset. note that this is not a true estimate of f v (c) as the dataset is limited to only 5 (and not all) countries. ## conclusion and open questions in this position paper, we presented an argument in favor of measuring meta-cultural competency in llms and llm-powered ai systems, rather than just cultural awareness. drawing from psychology and anthropology literature, we also described two foundational principles of meta-cultural competency for ai systems. we conclude by presenting a list of open questions about instilling and measuring meta-cultural competency in ai systems. in section 2 we mention three essential characteristics of culture, one of which is the inherent experiential nature of culture. extraction of such experiential knowledge from text or through other modalities and interaction patterns of the users is an extremely challenging problem that calls for a multi-disciplinary approach, most notably the methods from hci, psychology, and ethnography. beyond cultural equitability of ai systems, metacultural competency has huge application potentials ranging from user-facing ai assistants that can bridge cross-cultural communication to enabling the study of culture @xcite @xcite by supporting ethnographic research methods @xcite @xcite . through this position paper, we hope to make a strong case for the nlp community to engage in interdisciplinary conversations and widen the definition and scope of cultural competency in llms. (6) what is the need for meta-cultural competency in domain specific application? we believe meta-cultural competencies are crucial for domainspecific applications. even applications such as llms for scientific document analysis can benefit. firstly, culture is a prior for personalization. culture can provide a reasonable estimate of the user's background and preferences, which an ai system can use when it does not know anything about a user. it is a good prior for the cold-start problem in personalization @xcite , where the ai system can gradually personalize to the user's preferences as it discovers more about the user with each interaction @xcite . also, even if a user's cultural background is known a priori, meta-cultural competency would be still useful for adapting to the ever-evolving nature of culture.