Culture is Everywhere: A Call for Intentionally Cultural Evaluation (2025)

Abstract: The prevailing “trivia-centered paradigm” for evaluating the cultural alignment of large language models (LLMs) is increasingly inadequate as these models become more advanced and widely deployed. Existing approaches typically reduce culture to static facts or values, testing models via multiple-choice or short-answer questions that treat culture as isolated trivia. Such methods neglect the pluralistic and interactive realities of culture, and overlook how cultural assumptions permeate even ostensibly “neutral” evaluation settings.In this position paper, we argue for intentionally cultural evaluation: an approach that systematically examines the cultural assumptions embedded in all aspects of evaluation, not just in explicitly cultural tasks. We systematically characterize the what, how, and circumstances by which culturally contingent considerations arise in evaluation, and emphasize the importance of researcher positionality for fostering inclusive, culturally aligned NLP research. Finally, we discuss implications and future directions for moving beyond current benchmarking practices, discovering important applications that we don’t know exist, and involving communities in evaluation design through HCI-inspired participatory methodologies.

## introduction language model-based applications are growing in adoption across the world. to ensure they are adopted responsibly and effectively, an understanding of their cultural impacts and sensitivities is important. cultural misalignments in ai can perpetuate stereotypes, marginalize underrepresented voices, and fail to address the needs of diverse user communities @xcite . in response, the nlp and ml communities have begun to focus on culturally-aligned nlp, a subfield that aims to develop and evaluate systems capable of understanding and appropriately applying cultural knowledge in context @xcite @xcite . the overarching goal is to create nlp systems that can effectively respond to and operate within varied cultural settings @xcite . in this paper, we concentrate specifically on evaluation, as it increasingly shapes the direction of llm development and deployment across diverse cultural contexts. a key challenge, however, is that any decision in the evaluation pipeline-no matter how technical or routine-can carry cultural assumptions or consequences. for example, the tasks selected for evaluation often reflect the developers' cultural context, which may not align with the needs of users from different backgrounds @xcite . metrics assumed to be universal, such as what counts as "well-structured" writing, can vary significantly across cultures. even expectations around interaction style and communication can differ @xcite , affecting how users perceive model outputs. despite this, the community often overlooks these cultural contingencies, focusing attention only on the most obvious or explicit cultural questions (those labeled as "cultural tasks" or "multilingual settings"). as a result, most current evaluation practices reduce culture to static facts, trivia, or proxies like nationality-primarily testing models through isolated factual questions @xcite or their performance on culturally-cued prompts @xcite . while knowledge of cultural facts is important, it fails to recognize the cultural contingencies embedded in seemingly "neutral" evaluation choices. in this position paper, we argue that every evaluative choice should be examined for culturally contingent considerations, not just those in explicitly cultural domains. we argue for a shift toward intentionally cultural evaluation: a systematic approach that foregrounds cultural context throughout the evaluation process. by this, we mean making the cultural context of every evaluative decision explicit and deliberate, rather than leaving cultural influences implicit or accidental. to challenge the current focus on only the most obvious choices like explicitly cultural tasks or multilingual settings, we systematically distinguish and discuss three key aspects of evaluation: (1) what is evaluated (section 2), (2) how it is evaluated (section 3), and ( 3 ) in what circumstances (section 4) the desideratum is defined. we also examine the critical role of researcher positionality in shaping these evaluative choices (section 5). finally, we outline the broader implications of our proposed approach for nlp research and practice (section 6). contributions. our work offers several key contributions. we characterize the cultural contingencies in evaluation and propose building blocks for intentionally cultural evaluation. we find most evaluations reflect a narrow set of cultural assumptions, shaped by those who define the tasks and metrics. the design of "what" gets evaluated is frequently informed by dominant anglocentric perspectives, reifying specific knowledge types and communicative norms while marginalizing others. we show that standard computational practices, such as static reference examples or aggregate metrics, are poorly equipped to assess culturally grounded variation, and argue for reimagining these methods to support more flexible, context-sensitive judgments of model quality. crucially, we argue that culture in evaluation is not merely static content to be measured but is fundamentally tied to the circumstances of evaluation. we show how culture is both embedded in the very language of evaluation and enacted through culturally-contingent interactional patterns. as such, evaluating only static outputs misses key aspects of cultural alignment. finally, we call for greater reflection on the positionality of those evaluating. evaluation of cultural competence in nlp is not neutral-it is shaped by the positionality of researchers and by systemic biases embedded in the broader ai/ml ecosystem. researchers from lower-resource or non-anglophone contexts often face pressure to conform to english-centric benchmarks to gain visibility, placing additional burdens on their work and constraining the development of research agendas grounded in local cultural contexts. this marginalization limits the diversity of perspectives represented in nlp and reinforces existing inequities. further, we suggest implications for moving be-yond decontextualized methodologies toward more situated and culturally responsive methods, surfacing "unknown unknowns," and co-constructing evaluation practices with affected communities. we ground our suggestions using findings from hci studies. in doing so, we support a broader shift in nlp evaluation toward 'thick evaluation' @xcite )-an approach that prioritizes contextsensitive, community-aligned assessments of ai systems. ## what to evaluate to move toward culturally intentional evaluation, we must ask: what tasks contain important, culturally contingent considerations? current evaluations suffer from (a) overly narrow conceptions of 'cultural' tasks and (b) externally imposed definitions of relevance, thus failing to capture true cultural competence in real-world contexts. ## how to evaluate having established what to evaluate, we now address how to evaluate these diverse desiderata. sometimes, what can be feasibly evaluated is constrained by limitations in the how. a major challenge in large-scale cultural evaluation is "values pluralism," the existence of diverse, sometimes fundamentally irreconcilable perspectives @xcite . as datasets grow to encompass more diverse sub-groups, core differences in perspective can render the aggregation across samples less meaningfully representative of a coherent "culture" as a whole @xcite . this pluralism shapes how we can define and measure cultural alignment. ## in what circumstances to evaluate the circumstances of an evaluation are not culturally neutral. yet current practices often fail to account for the deep cultural contingency embedded in two fundamental dimensions: (a) the language in which evaluation is conducted and (b) the interactional context it assumes. ## implications and future directions beyond decontextualized measures. while existing benchmarks serve as useful tools for comparing models' general abilities (section 2), they often fall short in evaluating how models perform in realworld, culturally situated contexts. inspired by behavioral testing approaches like checklist @xcite , which systematically probe linguistic capabilities through targeted test cases, we propose extending existing "universal" benchmarks with explicit dimensions of cultural capability. by incorporating tests for "cultural alignment failures"-such as how models handle culturally specific communication norms, contextually appropriate responses, or regionally relevant content. at the same time, as we discuss in section 3, reference-based evaluation has fundamental limitations for cultural assessment: it cannot capture undesired default behaviors or accommodate the culturally contingent definitions of "good" we identified in subsection 3.1. we need evaluation frameworks that accommodate multiple valid perspectives simultaneously rather than forcing consensus on a single metric. this requires benchmark designs that move beyond static references to holistically assess the acceptability and severity of cultural misalignments while systematically surfacing patterns of bias or insensitivity. while employing llm-as-a-judge systems is one promising method to evaluate such nuances without a single correct answer, as explored for cultural qa @xcite , we caution that this is not a silver bullet. even when trained evaluators or llm-as-a-judge systems are used, the cultural frame embedded in the judge itself risks re-inscribing a dominant perspective, creating epistemic circularity. discovering "unknown unknowns" humancentric evaluation of llms often masks llms' unique, non-human errors; likewise, researchers evaluating cultures as outsiders risk overlooking problems they do not know exist. as discussed in section 2, surfacing these "unknown unknowns"-culturally meaningful tasks, interaction behaviors, and preferences currently invisible to us-is a major evaluation challenge.this includes both task-level gaps and interaction-level misalignments where cultural communication patterns affect model performance across user communities (section 4). to uncover these gaps, we need richer data on real user interactions, especially from underrepresented cultures. while resources like wild-chat @xcite , lmsys @xcite , and anthropic's clio project @xcite provide useful insights, current datasets remain limited in cultural coverage and openness. addressing unknown unknowns also requires methodological support. collaborating with hci researchers can help; for example, interactive systems have been developed to visualize data gaps and guide human-in-the-loop data collection @xcite . the field also needs empirical studies on why researchers overlook these gaps and what interventions can help build more culturally robust evaluation practices section 5. ## conclusion we have argued that evaluation in language technology is never culturally neutral, and that every choice-explicit or implicit-carries cultural consequences. our analysis shows that conventional evaluation practices, from task and metric selection to benchmarking standards, often obscure or marginalize diverse cultural realities. to move beyond these limitations, we advocate for culturally intentional evaluation: an approach that makes cultural context visible, explicit, and central at every stage of the evaluation pipeline. by centering positionality, engaging with affected communities, and embracing context-sensitive, "thick" evaluation practices, the nlp community can develop more equitable, representative, and impactful language technologies. we hope this work catalyzes further reflection and action, inviting researchers to critically reexamine and reimagine the cultural assumptions embedded in their evaluation practices, and to co-create more inclusive and responsive models for the world's linguistic diversity.