C ulture B ank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies (2024)

Abstract: To enhance language models’ cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With the pipeline, we construct CultureBank, a knowledge base built upon users’ self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit. Unlike previous cultural knowledge resources, CultureBank contains diverse views on cultural descriptors to allow flexible interpretation of cultural knowledge, and contextualized cultural scenarios to help grounded evaluation. With CultureBank, we evaluate different LLMs’ cultural awareness, and identify areas for improvement. We also fine-tune a language model on CultureBank: experiments show that it achieves better performances on two downstream cultural tasks in a zero-shot setting. Finally, we offer recommendations for future culturally aware language technologies. We release the CultureBank dataset, code and models at https://github.com/SALT-NLP/CultureBank. Our project page is at culturebank.github.io

## introduction large language models (llms) have become instrumental in various applications to interact with diverse user populations, such as in recommender systems @xcite and customer service @xcite . however, these models often mirror western-centric perspectives @xcite , as they are predominantly trained on data that reflect these values and behaviors. such cultural bias can lead to unintended consequences @xcite , e.g., reinforcing stereotypes, alienating non-western users, hindering global deployment and so on. therefore, it becomes increasingly important to develop language technologies that are aware of diverse cultures. we acknowledge that culture is a broad concept, and prior work has attempted to operationalize culture via different proxies @xcite @xcite . in the following paper, we use culture to refer to the knowledge of traditional behavior, norms, values, shared meanings, and patterned ways of behaving which are characteristic of a given society, a group of societies, a certain race or a certain period of time @xcite @xcite @xcite @xcite . to enhance llms' culture awareness, existing studies have developed cultural knowledge databases to represent culture-related knowledge and norms, but they have several limitations. (1) they often rely on formal knowledge sources like wikipedia and online articles @xcite , which miss the rich, evolving and long-tailed cultural nuances experienced by local communities. (2) secondly, these methods tend to present cultural knowledge in an assertive manner @xcite @xcite , failing to capture the fact that cultural practices and values can vary among individuals within the same cultural group. (3) besides, their evaluation methods often rely on classification tasks and question answering @xcite @xcite , which is very different from how llms are deployed in the real world and hence cannot reflect their cultural awareness in practice. to tackle these challenges, we utilize online communities, an important complement to existing work, where people share their cultural ex-figure 1 : overview. our goal is culturally-aware language technologies. to do so, we develop a pipeline and construct culturebank with structured cultural descriptors. each descriptor has a grounded scenario, persona, and question for evaluation. we fine-tune a model on culturebank and improve its performance on two cultural tasks. in short, we make the following contributions. with traditional ethnographic methods, social scientists recorded cultural knowledge through existing historical accounts, ethnographic data, and cultural documents. for instance, behavioral scientists compiled a collection of cultural materials, and released an online database named ehraf (the human relations area files) @xcite . in computer science studies, researchers employ computational methods to automatically construct datasets @xcite from large sources or curate data from crowd source workers @xcite . @xcite built a pipeline to extract assertive cultural commonsense knowledge from c4 @xcite , a large collection of internet data, and @xcite used wikipedia and navigated to related online documents to extract cultural knowledge. data from these sources are much cleaner compared to online communities, and often focus more on normative cultural indicators. since culture is highly heterogeneous, we also need descriptive cultural expressions from sources like online communities. stereokg @xcite used reddit and twitter to extract cultural stereotypes for 5 religious groups and 5 nationalities, but due to the lack of proper filtering, the results are noisy. as an important complement to existing data sources, our work proposes a pipeline to process highly noisy online communities data on a large scale, and show that it can be easily generalized across different platforms, to provide valuable descriptive cultural knowledge. cultural-awareness in language models. previous works have studied cultural dimensions in language models @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . on the evaluation side, prior studies have measured subjective global opinions from llms @xcite , and probed cultural value differences in these models @xcite @xcite . on the model side, culturellm @xcite proposed a cost-effective method to integrate cultural differences into language models with augmented data. this work proposes a grounded way to evaluate cultural awareness to match real-world use cases, and fine-tune a more culturally aware language model with descriptive cultural behaviors constructed from online communities. prior efforts on cultural knowledge base @xcite often represent cultural knowledge in free-text sentences. but free-text contents on online communities are often noisy, and such an unstructured representation hinders further computational operation such as search and filter. therefore, besides providing free-text description, we also develop a taxonomy (shown in ## construction pipeline centering on the proposed taxonomy, we propose a bottom-up pipeline to construct cultural descriptors from online communities (tiktok in the main ## evaluating llms' culture awareness with culturebank-tiktok, we evaluate llms' cultural awareness. prior work asks llms to answer cultural true/false questions @xcite . but llms are used in contextualized settings like a dialogue agent. so we propose a grounded evaluation, that grounds cultural knowledge in a real-world scenario, to test llms' ability to integrate cultural knowledge into their responses. we also perform classification-based direct evaluation in § d.2. grounded data generation. for each descriptor in culturebank, we first use a mixtral-8x7b model fine-tuned on gpt-4-generated examples to generate a relevant consulting scenario, a client persona, and a grounded evaluation question. then we employ a self-refinement method to improve the model generation based on two quality-control metrics at inference time. figure 3 shows a generated example (for the "no tipping in japan" descriptor, the grounded question is "what gesture says 'thanks you' in japan?"). human annotation shows 86% questions are correctly grounded on the original descriptor on 50 random samples. see §d.3 for more details. grounded evaluation. as shown in figure 3 , we present the generated grounded question to the llm for an answer. given the answer, we perform (1) automatic evaluation that uses gpt-4 to judge if the answer entails the original cultural descriptor (entailment score); and (2) human evaluation where two experts compare answers from two llms and select the more culturally-aware one (win rate). ## generalizing to reddit there are different online communities, so it is important to test if our pipeline can be transferred to other platforms. so we apply our pipeline on reddit, another online community. ## recommendations for culturally aware language technologies informed by results on culturebank construction and analysis, cultural awareness evaluation, and fine-tuning, we outline insights towards future culturally-aware language technologies. ## conclusion to conclude, our study introduces a generalizable pipeline for creating cultural knowledge bases from online communities. using the pipeline, we develop culturebank, a cultural knowledge database with 12k cultural descriptors sourced from tiktok, and 11k from reddit. culturebank features agreement levels for nuanced cultural interpretation and contextualized scenarios for grounded evaluation. with culturebank, we assess the cultural awareness of various llms, showing room for improvement. further, fine-tuning an llm with culture-bank leads to better performance on downstream cultural tasks, which showcases the potential of culturebank. finally, drawing from our findings, we close the paper by presenting insights towards future culturally-aware language technologies.