Are Generative Language Models Multicultural? A Study on H ausa Culture and Emotions using C hat GPT (2024)

Abstract: Large Language Models (LLMs), such as ChatGPT, are widely used to generate content for various purposes and audiences. However, these models may not reflect the cultural and emotional diversity of their users, especially for low-resource languages. In this paper, we investigate how ChatGPT represents Hausa’s culture and emotions. We compare responses generated by ChatGPT with those provided by native Hausa speakers on 37 culturally relevant questions. We conducted experiments using emotion analysis. We also used two similarity metrics to measure the alignment between human and ChatGPT responses. We also collect human participants ratings and feedback on ChatGPT responses. Our results show that ChatGPT has some level of similarity to human responses, but also exhibits some gaps and biases in its knowledge and awareness of Hausa culture and emotions. We discuss the implications and limitations of our methodology and analysis and suggest ways to improve the performance and evaluation of LLMs for low-resource languages.

## introduction large language models (llms), such as chatgpt are rapidly becoming popular and are employed in generating content for various purposes, be it for personal notes, in the workplace, or even for research and education. additionally, these models have a global reach, meaning a diverse set of people with varying cultural backgrounds. as a result, these models need to reflect the cultural differences of people and their emotional sensitivities when generating content. since these models were trained mainly on internet data, there is a high probability that they will be biased toward cultures with languages that are highly resourceful, such as english, japanese, chinese, german, and french @xcite @xcite . previous studies have shown that llms exhibit intersectional biases @xcite , gender stereotypes @xcite , and political biases @xcite . in this paper, we study cultural differences surrounding the representation of hausa culture and emotions in chatgpt. the relationship between language, culture, and emotions has been well established in the literature @xcite @xcite . language serves as a medium through which cultural identities, values, and traditions are expressed and transmitted. cultural narratives, metaphors, and discourses are embedded in language, reflecting the cultural heritage of the community @xcite . the language we use can influence how we experience and express emotions. language can both dampen and intensify emotional experiences. journaling or verbal expression of emotions, for example, can help regulate negative emotions @xcite . our work focuses on the extent to which multilingual llms generate culture-aware responses. we aim to investigate the validity of cultural and emotional responses generated by multilingual llms for low resource languages. in our experiments, validity is assessed by speakers of the hausa language. despite being spoken by approximately 100 to 150 million people globally, hausa remains a low-resource language. hausa is spoken mostly in west african countries such as nigeria, nijer, ghana, cameroon and benin @xcite . the remainder of this paper is organized as follows: section 2 details the experiment design, section 3 provides the results, section 4 discusses the findings and finally, the conclusion is described in section 5. ## experiments as a first step, we prompt chatgpt with 37 questions that are expected to yield culturally-aware responses (more details below). then, we used those as survey questionnaire, and collected the responses from native hausa speakers living in nigeria. we collected two types of responses; first, we asked participants to answer the questions as openended questions. then, we asked them to rate the responses generated by chatgpt using a 5-point likert scale. our experiments involve a comparative analysis of the survey responses and those generated by chatgpt. ## discussion the limited cultural alignment found in this work can be attributed to several factors such as the quality and quantity of the training data, the fine-tuning process, the cultural and linguistic diversity of the users, and the evaluation methodology. these factors are further discussed below. training data: humans grow up learning about cultural values in their lives. this shapes how they understand and respond to cultural questions. however, llms lack this background knowledge or life experience @xcite @xcite . so, when it gathers and organizes information, it doesn't understand the context or importance of cultural diversity. this is due to the fact that chatgpt is trained on a large corpus of conversational data, which may not reflect the cultural and emotional diversity of its users, especially users of low-resource languages like hausa. the training data may also contain noise, errors, or biases that may affect the model's performance. for example, the training data may be skewed towards certain domains, topics, or perspectives that may not be relevant or appropriate for hausa speakers @xcite . therefore, the quality and quantity of the training data are crucial for developing more culture-aware and emotion-sensitive llms for low-resource languages @xcite . fine-tuning process: chatgpt is fine-tuned on a smaller dataset of multilingual conversations, which may not capture the nuances and variations of hausa culture and emotions @xcite . for example, to the prompt foot_0 "how would you feel if your student call you by your first name", chat-gpt's response was "i would feel a bit surprised but might appreciate the informality, recognizing that the student may see our relationship as more casual.". this is an example for an inappropriate response in the hausa culture and traditions and it will be considered very disrespectable to call your teacher by their first name even in an informal setting. therefore, the fine-tuning process needs to be carefully designed and optimized for developing more robust and adaptable llms for low-resource languages. another example, for the question: "how will you feel about prioritizing eating in a restaurant over cooking at home?", more than 68% of the participants considered chatgpt's response as culturally inappropriate. similarly, for the question: "how would you feel about your ward moving out of the house at the age of 18?", less than 2% of the respondents agreed with the response. cultural and linguistic diversity: chatgpt is designed to generate responses for a diverse set of users, who may have different cultural backgrounds, values, and preferences. however, when evaluating on hausa, chatgpt did not seem to capture these cultural and linguistic diversities indicated in lower meteor scores. in particular, chat-gpt may not be able to produce the hausa dialects, idioms, or expressions that may exist within the hausa language and culture. chatgpt may also not be able to adapt to the different contexts, situations, or goals that may influence the model's performance. therefore, the cultural and linguistic diversity of the users poses a challenge and an opportunity for developing more personalized and context-aware llms for low-resource languages. based on these factors, we suggest some ways to improve the performance and evaluation of chat-gpt and llms for low-resource languages, such as hausa. first, we suggest using more diverse and representative data that can cover more topics and scenarios that hausa speakers may encounter in the digital world. for example, we can use data from different sources, such as social media, news, blogs, or forums. we can also use data from different groups of people, such as age, gender, education, or location. these data can enrich the model's knowledge and adaptability and provide a more realistic and authentic evaluation of the model's performance. second, we suggest incorporating human feedback and perspective that can improve the model's performance. for example, we can use methods such as user testing, surveys, interviews, or focus groups. we can also use techniques such as active learning, reinforcement learning, or dialogue management. these methods and techniques can enhance the learning and interaction of the model and provide a more user-centric and user-friendly evaluation of the model's performance. third, we suggest proposing evaluation metrics that can measure various aspects of natural language and human communication considering coherence, relevance, fluency, or sentiment. these metrics can complement the similarity metrics and provide a more comprehensive and holistic assessment of the model's performance. ## conclusion we investigated how chatgpt, a generative large language model (llm), represents the hausa culture and emotions, a low-resource language spoken by over 100 million people in west africa. we compared the responses generated by chatgpt with those produced by native hausa speakers on 37 culturally relevant questions. we employed emotional analysis, semantic and ngram textual analyses. we also collected the ratings and feedback of human participants on the chatgpt responses, and evaluated their cultural authenticity. our results show that chatgpt has a limited degree of alignment with human responses. we found a mismatch in the emotional diversity exhibited in the responses of the participants compared to the responses of chatgpt. we showed that while artificial responses were semantically similar to human participants, they were not aligned with anticipated word choices. finally, participants found that some chatgpt responses were likely, but that others were unlikely to be spoken by a member of their culture. our study highlights the imperative for improving chatgpt and other llms' performance and evaluation in low-resource languages to better represent users' cultural and emotional diversity, crucial for sensitive domains like health and education in order to promote equitable and inclusive participation. we suggest utilizing more diverse data, human feedback, and alternative evaluation metrics. in conclusion, our research underscores the importance of evaluating llms for low-resource languages, exemplified by hausa. future directions involve expanding datasets and establishing crowd-truth @xcite approaches to aid validation strategies of cultural alignment evaluations for researchers and practitioners.