CAVA : A Tool for Cultural Alignment Visualization & Analysis (2024)

Abstract: It is well-known that language models are biased; they have patchy knowledge of countries and cultures that are poorly represented in their training data. We introduce CAVA, a visualization tool for identifying and analyzing country-specific biases in language models.Our tool allows users to identify whether a language model successful captures the perspectives of people of different nationalities. The tool supports analysis of both longform and multiple-choice models responses and comparisons between models.Our open-source code easily allows users to upload any country-based language model generations they wish to analyze.To showcase CAVA’s efficacy, we present a case study analyzing how several popular language models answer survey questions from the World Values Survey.

## introduction there is a growing body of work on understanding the biases encoded in large language models (llms). in particular, researchers have striven to measure the culture-and country-specific competencies of llms @xcite , and how they represent subjective country-specific opinions @xcite . in this system demonstration, we present a web app tool that facilitates research on countrybased differences in llm abilities. cava 1 2 3 presents a novel method to visualize and interact with the cultural values expressed by an llm with a map-based interface. there is a range of tools that allows users to evaluate the degree of cultural alignment between an llm and a country with techniques such as performance met-rics, identification and location of keywords, visualization of the distribution of answers, and performing cross-model comparisons. cava's design allows for the easy addition of models and questions, making it adaptable for specific use cases. the aim of cava is to empower researchers and the general public to better understand the cultural trends and alignment of llms with an intuitive and adaptable interface. using cava, we conducted a case study on the religious beliefs of llms and discovered notable patterns of behavior in popular llms. we hope that future users can glean additional insights into similarly impactful topics. ## related work a prevalent approach in current research to assess the cultural alignment of llms involves utilizing established frameworks or surveys such as hofstede's cultural dimensions @xcite or the world values survey (wvs) @xcite . this method typically involves employing prompt engineering to instruct llms to simulate personas from specific countries and then have them respond to the framework or survey. the answers are then compared to the ground truth to quantify the cultural alignment of llms and reveal their cultural biases. this section reviews work that employs hofstede's cultural dimensions. @xcite observed that while all llms struggle to accurately reflect cultural values, gpt-4 demonstrated a stronger understanding of cultural dimensions compared to gpt-3.5 and llama2 when adapted to specific personas. @xcite observed similar struggles, but showed llms are generally capable of grouping countries on each side of a cultural dimension and demonstrated that there is no clear correlation between a language's online presence and the cultural alignment of the country that uses it. in another study, @xcite highlighted how english prompts flatten out cultural differences and bias them towards american culture. tao et al. (2024) demonstrated five openai llms exhibit cultural values aligned with english-speaking protestant european countries. alkhamissi et al. (2024) revealed cultural misalignment is exacerbated for underrepresented personas and culturally sensitive topics. arora et al. (2023) supports these findings albeit with mbert, xlm, and xlm-r. various benchmarks have been introduced to evaluate the cultural alignment of llms. cde-val @xcite is based on hofstede's cultural dimensions. worldvaluesbench @xcite and globalopinionqa @xcite , which comes with a map-based visualization, are based on the wvs and pew global attitudes survey (pew). regional variants of the wvs such as the european values survey (evs) and chinese values survey are also other commonly used surveys for evaluating llms in this regards @xcite . ## description of system cava is a web app centered around an interactive world map displaying an llm's responses to survey questions when it is asked to take on the persona of an individual from each country foot_0 . it consists of two main modes for visualizing the survey results. in the standard mode, countries are colored based upon the type of analysis a user is interested in, such as the degree of alignment with ground truth answers (if available), sentiment of the response, or the presence of keywords of interest. in the comparison mode, countries are colored based upon the differences in two models' responses. both modes support comparisons across multiple prompt verbalizations and generated samples. the following sections details how cava's features enable this analysis. predicted labels. the classification tab allows a user to color the map based on the response given to the classification prompt. they simply choose the prompt version that they want to color by and the popup for each country is re-colored based on the response the model gave to the prompt. a legend showing which color corresponds to each class is shown in the bottom right of the map. moreover, the sidebar also contain a bar chart with the distribution over all the classes for every country. prediction correctness. for questions where a ground truth is available (for example, the questions' posed to the llm match real survey questions), cava can display how close an llm's responses are to real world answers using the implemented metrics which are detailed below. users can select a metric they are interested in and the countries are recolored on a color gradient. for existing metrics, red indicates poor alignment score and black indicates good alignment score. countries without a ground truth distribution remain white. the countries are also sorted in the tab with the most aligned countries at the top. ## analyzing cultural competencies to showcase how cava can be used to assist in analysis of llms' cultural competencies, we posed 25 questions from the world values survey to seven models and visualized the results. ## conclusion and discussion this paper introduces cava, a novel tool for visualizing the cultural competencies of llms across the dimension of geographic locales. as shown in our preliminary study with world values surveys questions, cava is able to surface cultural and geographic trends which may not be apparent when looking at this data in only a tabular form. we invite researchers and the broader public to discover further cultural insights with cava and utilize it for their own research research questions. future work could include adapting cava to be a continual benchmark for closed source models, documenting changes in capabilities over time @xcite . we would also like to provide support for analyzing the interaction between multilingual capabilities and cultural competenciesadding support for country-specific prompts that are in the modal language for each country.