Building Knowledge-Guided Lexica to Model Cultural Variation (2024)

Abstract: Cultural variation exists between nations (e.g., the United States vs. China), but also within regions (e.g., California vs. Texas, Los Angeles vs. San Francisco). Measuring this regional cultural variation can illuminate how and why people think and behave differently. Historically, it has been difficult to computationally model cultural variation due to a lack of training data and scalability constraints. In this work, we introduce a new research problem for the NLP community: How do we measure variation in cultural constructs across regions using language? We then provide a scalable solution: building knowledge-guided lexica to model cultural variation, encouraging future work at the intersection of NLP and cultural understanding. We also highlight modern LLMs’ failure to measure cultural variation or generate culturally varied language.

## introduction people think and behave differently around the world. this is partly due to cultural variation, or the differences among individuals that exist due to some form of social learning @xcite . having a computational method that utilizes language to measure cultural variation could help us better understand the way people communicate @xcite , build more culturally-aware nlp systems @xcite , and advance interdisciplinary research in anthropology, cultural psychology, etc. however, due to a lack of data and scalability constraints, few such methods exist. in this paper, we present measuring regional variation in culture as a problem of interest for the nlp community. we highlight how large language models (llms) struggle with this task, and build a knowledge-guided lexical model as a scalable and reliable solution. specifically, we focus on measuring the cultural dimenision of individualism and collectivism foot_0 across the united states (us) using geolocated tweets. historically, measuring cultural dimensions across regions has been mostly done through questionnaires, such as the world values survey (wvs) @xcite . however, questionnaires are time-consuming and heavily restricted in scope; the most recent wvs wave required four years and averaged only 52 participants per us state. recent work probes llms for cultural values @xcite , but these llms do not reflect all cultures equally @xcite . therefore, relying on llms to measure culture is risky, as they may not generalize well to different populations. the overhead of traditional survey-based approaches and inconsistent cultural awareness of existing llms motivate computational methods that rely on existing language data to measure cultural variation instead. we specifically seek to use geolocated twitter data -instead of selecting a small portion of people to represent a state or county (like traditional survey-based methods), we instead use a massive amount of tweets from that region, thus gaining a larger and more holistic representation of a region's population. we also seek to leverage domain expertise from cultural psychologists. cultural psychologists have spent decades developing non-computational tools to measure cultural constructs like individualism and collectivism @xcite . by encoding this domain knowledge via a set of expertcurated seed words, we can create a method to measure culture that is both scalable and grounded in cultural psychology theory. in this paper, as an example of cultural variation, we will measure individualism and collectivism across us counties using the following resources: ## building knowledge-guided lexica issues with traditional lexica. lexica, or sets of curated words, are a highly scalable and explainable method for analyzing large datasets. however, building a full lexicon linked to cultural theory from the ground up is a time-intensive process, and sometimes takes psychologists years to complete @xcite . additionally, psychologists often make the error of including words in lexica that correlate negatively with the construct they are trying to measure. jaidka et al. (2020) find that removing certain words from lexical categories in liwc 2015 (pennebaker et al., 2015) actually improves overall performance. concept expansion shame disgrace pity shameful shame sacrifice duties honorable obligation loyalty shame sacrifice duties .... list of seed words from cultural psychologist shame + disgrace + pity + shameful ... shame sacrifice duties + loyalty + obligation + honorable ... synonym expansion purification knowledge-guided lexical model pride joy pride joy disgrace loyalty obligation -pity ... these erroneous words can be caused by the following two phenomena: 1. inflated frequency: highly frequent words in a lexical category can correlate negatively with their counterparts (e.g. words like "love" and "lol", contained in the labmt positive sentiment lexicon @xcite correlate negatively with happiness @xcite ) and muddy the results of a lexicon. to mitigate this, we propose a method that still builds on domain knowledge, but produces a lexicon that is internally coherent, bypassing the issues introduced by inflated frequency and polysemy. our approach has two components: expansion and purification. figure 2 details this approach for the collectivism lexicon we generate. using this method, we utilize domain knowledge from an expert psychologist to create a lexical model that has wider coverage of our corpus. we can then use this model to analyze our geolocated twitter corpus and measure regional variation in individualism and collectivism across the us. ## evaluation upon expanding and purifying the lexica, we apply it to our twitter corpus. to get a collectivism score for county c, we sum the weighted frequencies of each word w in our collectivism lexicon l coll , as outlined in equation 3 . we then aggregate these county-level scores to the state-level, and validate our results using past statelevel collectivism research from vandello and cohen (1999); @xcite . we expect our state-level collectivism results to correlate positively with these indicators. as there are no similar resources for individualism, we use the strength of negative correlation to evaluate our state-level individualism results. ## results to further assess the success of our method, we compare our correlations against the following baselines: as the corresponding individualism and collectivism scores for that state. see appendix c for additional details on prompting procedure. ## investigating cultural variation in llm-generated text with a validated method to measure regional variation in individualism and collectivism, we can now answer the question: given geographical information, can llms generate text that mimics real-world cultural variation? ## conclusion & future work we introduce a new problem for the nlp community: measuring regional variation in culture. this is a new class of problem -obtaining labels for culture across states or countries is often infeasible, as there are very few labeled data points to train on. using pre-trained llms to label data is unreliable, as these models lack basic cultural awareness. additionally, scaling to label billions of data points is computationally infeasible. classic deep learning methods fail to solve this problem; therefore, measuring culture necessitates a different approach. we present a method to efficiently measure cultural variation by leveraging domain knowledge from cultural psychology to create knowledgeguided lexica. our lexica filter out erroneous words and ensure internal coherence, bypassing the pitfalls of traditional lexica. by applying these lexica to social media language, we can estimate cultural differences at fine-grained geographic levels, such as states, counties, and communities -a task that modern llms fail to accomplish. future work could build on this method to get deeper insights into communities and cultures. for example, our method could be used to identify tweets that mark cultural differences; we encourage researchers to build more sophisticated models on these identified tweets. additionally, our method is easily extendable to other cultural dimensions, such as power distance, tightness/looseness, etc. this method could also measure cultural variation globally, which requires analyzing different languages. since our method is language agnostic, it can easily extend to non-english settings by leveraging multilingual embeddings.