Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning (2024)

Abstract: Exploring the intersection of language and culture in Large Language Models (LLMs), this study critically examines their capability to encapsulate cultural nuances across diverse linguistic landscapes. Central to our investigation are three research questions: the efficacy of language-specific instruction tuning, the impact of pretraining on dominant language data, and the identification of optimal approaches to elicit accurate cultural knowledge from LLMs. Utilizing the GeoMLaMA benchmark for multilingual commonsense knowledge and an adapted CAMeL dataset (English-only) for evaluation of nuanced cultural aspects, our experiments span six different languages and cultural contexts, revealing the extent of LLMs’ cultural awareness. Our findings highlight a nuanced landscape: while language-specific tuning and bilingual pretraining enhance cultural understanding in certain contexts, they also uncover inconsistencies and biases, particularly in non-Western cultures. This work expands our understanding of LLMs’ cultural competence and emphasizes the importance of integrating diverse cultural perspectives in their development, aiming for a more globally representative and equitable approach in language modeling.

## introduction large language models (llms) are capable of performing well across a wide variety of tasks @xcite owing to their ability of generating coherent text that draws from a large corpus of pre-training data. however, some tasks like performing open-ended social reasoning involve questions @xcite which due to being under-specified or requiring a certain level of critical thinking elicit an opinionated answer from the llm that affects different social groups, sometimes in undesirable ways @xcite . the role of culture is undeniable when looking at factors that determine people's beliefs and behavior in social settings. cultural differences exist across countries and they interplay with the dominant language spoken, influencing both individual traits and group behavior. however, it is a well-known problem that multilingual llms are trained on corpora that are not equally representative of all parts of the world, but are rather more "western aligned" @xcite . this leads to potential issues of misrepresentation of culture and a lack of understanding of cultural knowledge in text generated by llms. other work which studies this question brings out the lack of precision in cultural representations (ramezani and xu, 2023), problems of skewing distributions or amplifying biases existing in society @xcite , erasing underrepresented nuances @xcite and also the impact of low resourced languages and cultures they are spoken in @xcite . while these studies have laid the groundwork, there remains a gap in understanding how language-specific instruction tuning might correlate with cultural knowledge. our work aims to first test the hypothesis that instruction tuning on data in a specific language might improve the cultural awareness of the llm for the related culture. we also look at the impact of continued pre-training in a bilingual setting and how that has an influence on multilingual cultural understanding. specifically we design the following research questions for this purpose: rq1 : does instruction tuning on languagespecific data enhance cultural knowledge? rq2a : does pretraining on language-specific data enhance cultural knowledge? rq2b : what is the optimal approach for eliciting cultural information from llms? for this purpose, we translate the instructional data used for training the alpaca model @xcite to five languages @xcite other than english to cover the six distinct cultures. we then use this data to train low rank adapters @xcite ) followed by evaluation on a benchmark for multilingual cultural knowledge in each language to measure the impact of instruction tuning. we also explore whether llms understand tangible cultural nuances like food, beverages, clothing, etc by framing a dataset about different social situations with cultural targets based on a previous study @xcite . we use this to ask: rq3 : do llms understand the nuances of culture and what disparities exist across tangible cultural aspects? overall, our findings show shortcomings in not only how culture is understood by llms, but also in current existing approaches at overcoming them. ## data and methods our study explores the cultural understanding of large language models (llms) through two primary strategies: enhancement of an existing benchmark and the creation of a new, culturally-focused benchmark. our methodology involves translating instructional data from the alpaca dataset (which does not include culturally relevant information) into five additional languages and conducting supervised fine-tuning on various llama 2 model @xcite sizes using these translations (a sample of which is manually verified by speakers and rq2 . lang refers to language-specific variants of alpaca or a language-specific prompt. ## experimental settings we divide our experiments into two distinct categories -first looking at how instruction tuning and pretraining play a role in cultural understanding, and then going deeper into different aspects of cultural nuances. ## results our findings show that llms do not understand the specific details that define culture even when we try different approaches like sft, bilingual pretraining, and prompting in the dominant language. our investigation into whether sft on languagespecific data enhances cultural knowledge involves a series of experiments, detailed in appendix 6, no clear enhancement due to sft, with a lot of variability across cultures the hypothesis that sft with language-specific data substantially improves cultural knowledge is not conclusively supported by our findings. there is notable variability across different cultures. for instance, in china and iran, english-based fine-tuning seems to be more effective, while in india, hindi fine-tuning competes closely. english emerges as the most effective language for eliciting cultural knowledge across various cultures. however, the second-most effective language is not consistently the dominant one. for example, prompting in chinese yields better results than swahili for kenyan cultural questions ( ## related work in the context of understanding cultural biases in large language models (llms), several studies have made significant contributions, each addressing different aspects of this multifaceted issue. tao et al. (2023) use the world values survey to map gpt models on the inglehart-welzel cultural map, highlighting the effectiveness of cultural prompting as a mitigation strategy. durmus et al. (2023) combine datasets from the world values survey and the pew research center's global attitudes surveys to explore models' alignment with western values, using various prompting techniques. the seaeval benchmark @xcite demonstrates the challenges multilingual llms face in multicultural reasoning, affected by factors like positional bias and language nuances. copal-id @xcite finds that llms have a lower understanding of culture-related questions compared to non-culture related ones, especially in multilingual settings. additionally, @xcite pioneered examining cultural alignment for chatbots, revealing chatgpt's american-centric alignment. however, these studies collectively highlight some gaps: a predominant focus on westerncentric perspectives, limited exploration of non-western cultures, and the need for more comprehensive strategies to incorporate a global spectrum of cultural nuances in llms. our work aims to build upon these findings, addressing these shortcomings by examining llms' cultural awareness more holistically and inclusively. ## conclusion this study on the cultural understanding of large language models (llms) reveals significant variations in their ability to encapsulate diverse cultural nuances. our investigations, leveraging the geomlama benchmark and the adapted camel dataset, demonstrate that while language-specific instruction tuning and bilingual pretraining offer some improvements, they fall short of ensuring comprehensive cultural competence, particularly in non-western contexts. the findings underscore the need for incorporating a wider range of cultural perspectives in llm training and development, highlighting the importance of creating models that are not only linguistically adept but also culturally sensitive and globally inclusive.