Extrinsic Evaluation of Cultural Competence in Large Language Models (2024)

Abstract: Productive interactions between diverse users and language technologies require outputs from the latter to be culturally relevant and sensitive. Prior works have evaluated models’ knowledge of cultural norms, values, and artefacts, without considering how this knowledge manifests in downstream applications. In this work, we focus on extrinsic evaluation of cultural competence in two text generation tasks, open-ended question answering and story generation. We quantitatively and qualitatively evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. Although we find that model outputs do vary when varying nationalities and feature culturally relevant words, we also find weak correlations between text similarity of outputs for different countries and the cultural values of these countries. Finally, we discuss important considerations in designing comprehensive evaluation of cultural competence in user-facing tasks.

## introduction cultural competence is the ability to effectively and appropriately communicate with socioculturally different audiences @xcite . people demonstrate cultural competence by tailoring their utterances to the participants in a conversation @xcite @xcite . these adaptations range from sociolinguistic variations (e.g., using 'soccer' or 'football' depending on the context) to appropriately using facts (e.g., in india, the prime minister is the head of the government, but in the usa, the president is). hence, for effectively serving diverse users, outputs from large language models (llms) need to be culturally relevant @xcite . cultural competence includes a person's knowledge of a culture, which then supplements their skills of effectively communicating with people from that culture @xcite @xcite . so, the cultural competence of llms should also be evaluated along both these aspects. contemporary works have largely targeted the knowledge component of cultural competence by evaluating llms' knowledge of cultural values, norms, and artefacts ( § 2.2). such evaluation is intrinsic because it is decoupled from the manifestation of this knowledge in downstream applications @xcite . in this work, we focus on extrinsic evaluation of cultural competence. extrinsic evaluation setups should closely mimic user interactions with a system @xcite . we select the tasks of story generation and open-ended question answering (qa), both of which have high representation in user interactions with llms @xcite , so . as shown in figure 1 , we obtain model outputs for 35 topics in story generation and 345 topics for qa when an explicit cue of culture, i.e., nationality, is present in the prompt. our data consists of outputs from 6 llms for 193 nationalities, with 5 outputs per prompt and 2 temperature settings, resulting in over 370k story outputs and 3.6m qa outputs. we analyse the extent and the nature of lexical variations in these outputs. further, recent intrinsic evaluations have heavily relied on surveys from cross-cultural psychology, like hofstede's cultural dimensions @xcite and world values survey @xcite as measures of cultural values (e.g., @xcite ; @xcite ; @xcite , inter alia). we evaluate whether the text distributions of outputs correlate with the cultural values of countries, as captured by these surveys. to summarize, our main research questions are: rq1: do models vary outputs when explicit cues of culture are present in the input prompt? rq2: do model outputs contain culturally relevant vocabulary? rq3: are model outputs for countries with similar cultural values, also similar? by measuring the variance in the outputs, we find that models make non-trivial adaptations for different nationalities ( § 5.1). next, inspecting the vocabulary of these outputs, we find that they contain culturally relevant words ( § 5.2). finally, we find only a weak correlation between the text distributions and cultural values of countries, as measured by cross-cultural psychology surveys frequently used in contemporary work ( § 5.3). our findings show that intrinsic and extrinsic measures of cultural competence do not correlate. this necessitates developing holistic evaluations to analyse cultural competence in tasks representative of user interactions with llms. we make our code foot_0 and data foot_1 publicly available for replicating results and use in further analysis. ## extrinsic evaluation of cultural competence jones and @xcite describe extrinsic evaluation criteria as, "those relating to its function, i.e its role in relation to its setup's purpose". so, consider the two broad use cases of llms: (a) classification tasks, and (b) generation tasks. while incorporating cultural knowledge has been shown to benefit classification tasks like hate speech detection and commonsense reasoning @xcite @xcite , to the best of our knowledge there is no prior work focusing on open-ended text generation tasks. specifically, we obtain model outputs when nationalities in prompts are perturbed. we propose quantitative ( § 3.1) and qualitative ( § 3.2) analyses to evaluate these outputs for cultural competence. ## results we now describe our observations on the outputs obtained with the temperature of 0.3. the observations with a temperature of 0.7 are consistent with these findings and are described in appendix c. ## limitations while our work serves as a starting point and a call to focus on the extrinsic evaluation of cultural competence, it is not free of limitations. first, we perform limited qualitative evaluation, and we do not perform any comprehensive human evaluation of the outputs. we describe considera-tions for comprehensive human evaluation in § 6. secondly, our work is anchored on nationalities and relies on imperfect measures of cultural values. however, as we describe in detail in § 6, evaluation of cultural competence demands participatory and intersectional approaches, in addition to accounting for imperfect and static measures of cultures. further, our evaluation of the outputs does not reflect their pragmatic correctness. in other words, have not evaluated whether a model's adaptations for a particular question (eg. 'explain elections...') correctly reflect how the topic is operationalized in the country. such evaluation needs either expert knowledge or a comparison with verified sources. moreover, in measuring the characteristics of the text distributions, we focus only on vocabulary. this provides a starting point for cultural competence. however, culturally sensitive text will need to be evaluated for further characteristics also, for example adhering to the tonality, formality, or other stylistic expectations that might vary culturally. finally, in our evaluation, we prompt the model with the nationality explicitly and in english. however, there might be other implicit cues of culture that trigger adaptations such as the language and dialect of interaction, and topical differences in queries which we do not account for in this work. we hope that future work can address these limitations to holistically evaluate llms for cultural competence in user-facing tasks. ## conclusion in this work, we evaluated cultural competence in two tasks, story generation and open-ended question answering. our data contributions include a hand-curated list of 345 diverse questionanswering topics and 35 story-generation topics. we also obtain model outputs for 6 models and 193 nationalities which we will make available for further analysis. our methodological contributions include conceiving two quantitative and one qualitative analysis for the evaluation of llm outputs for cultural competence. using these methods, we find that models do vary their outputs with varying nationalities ( § 5.1), outputs contain culturally relevant artefacts ( § 5.2), and model outputs weakly correlate with cultural values ( § 5.3). our findings underscore the importance of comprehensive extrinsic evaluation of cultural competence.