Multilingual Language Models are not Multicultural: A Case Study in Emotion (2023)

Abstract: Emotions are experienced and expressed differently across the world. In order to use Large Language Models (LMs) for multilingual tasks that require emotional sensitivity, LMs must reflect this cultural variation in emotion. In this study, we investigate whether the widely-used multilingual LMs in 2023 reflect differences in emotional expressions across cultures and languages. We find that embeddings obtained from LMs (e.g., XLM-RoBERTa) are Anglocentric, and generative LMs (e.g., ChatGPT) reflect Western norms, even when responding to prompts in other languages. Our results show that multilingual LMs do not successfully learn the culturally appropriate nuances of emotion and we highlight possible research directions towards correcting this.

## introduction the global reach of large language models (lms) today prompts an important question -are multilingual lms also multicultural? we are specifically interested in the multicultural behavior of lms from the lens of emotion. lms are used for many multilingual tasks that require emotional sensitivity and therefore must be able to reflect cultural variation in emotion. for instance, lm-powered therapy bots must delicately adapt the way they speak to patients in different languages @xcite , lms as creative writing assistants must produce content that will elicit the appropriate emotional response in an author's desired audience @xcite , lms used for workplace communication must understand the subtleties of interpersonal interaction @xcite , etc. we define cultural variation in emotion as the nuances in meaning and usage of emotion words across cultures. for example, in english, we have many different words that express anger. one can say "i feel angry," but may also choose to say "frustrated", "irritated", or "furious." the anger invoked "i would feel that facing off against a friend at a friend's house is a fun and exciting experience and a good way to deepen friendships." by a baby crying on an airplane is different from the anger invoked by an unfair grade on an exam; different situations that cause anger will invoke different language to best express it. these nuances in meaning and usage patterns of emotion words exist differently across cultures @xcite . therefore, there is not a perfect one-to-one mapping between languages for emotion words coupled with their meaning and usage patterns. the direct translation for "i feel frustrated" from english to chinese (simplified), for example, is "我感到沮 丧". however, in a situation where a native english speaker would likely say "i feel frustrated," a native chinese speaker may use a different phrase than "我感到沮丧", based on situation, context, and the cultural norms of emotion expression in china. as we rely on multilingual lms today for emotionally sensitive tasks, they must reflect this cultural variation in emotion. however, the widely-used multilingual lms are trained on anglocentric corpora and encourage alignment of other languages with english @xcite , both implicitly and explicitly, during training. the key problem in this approach to building multilingual lms is that any form of alignment destroys a model's ability to encode subtle differences, like the difference between "i feel frustrated" in the united states and "我感到沮丧" in china. in this paper, we investigate whether widelyused multilingual lms reflect cultural variation in emotion. we select four high-resource written languages, two western and two eastern, to focus on in this work -english, spanish, chinese (simplified), and japanese. specifically, we investigate two facets of lms: embeddings and language generation. we make our code public foot_0 and encourage researchers to utilize the analyses outlined in this work as a baseline to measure the cultural awareness of future multilingual models. ## related work a large body of work in nlp focuses on detecting emotion in multilingual text . however, a major oversight in this line of research is that it treats emotion as culturally invariant. work from @xcite gathers a corpus of annotated social media data from 19 languages, but uses machine translation to transfer annotations from one language to another, assuming that translation correctly captures emotional variation. work from @xcite generates lexica to analyze emotion across 91 languages, relying on translations from english lexica and assuming that the affective state of parallel words will be identical. psychologists have characterized emotion as having multiple components -an emotional experience, a physiological response, and a behavioral response tendency @xcite . each of these components vary from culture to culture @xcite , a complexity completely ignored when emotion is treated as a static, transferable label on an utterance of text. using machine translation to transfer emotion labels between languages incorrectly assumes that emotion is experienced identically across cultures. others have also observed that lms can fail to account for cultural context and variation. @xcite find that chatgpt strongly aligns with american values. @xcite use word embeddings to globally measure human values across cultures, and find that these values overlap more when measured via data in english vs. native languages. @xcite probe multilingual lms and discover weak alignment with the cultural values reflected by these lms and established values surveys. in this paper, we focus on emotion, showing a wider variety of anglocentric anchoring by elucidating the underlying mechanisms of this alignment. we investigate emotion embeddings and lm probabilities, as well as affective language generated from multilingual lms. ## investigating emotion embeddings many tasks in multilingual nlp utilize embeddings from pre-trained lms such as xlm-roberta @xcite and mbert @xcite . researchers fine-tune these models for downstream tasks, relying on their learned representations of words and concepts. we scope our investigation to embeddings from the widely used xlm-roberta models. xlm-roberta was trained on text that includes parallel and comparable corpora (e.g., wikipedia) in mul- monolingual roberta happiness sadness anger elation 2.23 9.85 13.05 12.55 multilingual roberta happiness sadness anger elation 4.25 6.68 28.44 28.48 joy joy figure 2: we determine the similarity between the embeddings of monolingual joy and multilingual joy by comparing the distances from joy to other emotions embeddings in both settings. specifically, we calculate the correlation @xmath0.05, 9.85, 12.55.2.23 > @xmath1.44, 6.68, 28.48, 4.25 > to infer similarity. this section investigates the effect of alignment -both implicit and explicit -by analyzing emotion embeddings from monolingual, multilingual, and aligned roberta models (see table a2 ). we further investigate whether this anchoring impacts our ability to visualize known cultural differences (e.g. differences between pride and shame in the us vs. japan @xcite ) when projecting embeddings into the two-dimensional valence-arousal plane @xcite . ## investigating multilingual lm generation we now turn from investigating embeddings to analyzing language generated by language models (gpt-3, gpt-3.5, and gpt-4) to see if multilingual lm completions reflect cultural variation in emotion. in order for lms to be used for tasks that require emotional sensitivity, their responses must align with cultures' socio-cultural norms @xcite ; generated text must reflect users' cultural tendencies and expected affect @xcite . we first analyze token-level completion probabilities from gpt-3, to see if they reflect cultural differences between american and japanese shame and pride. we then prompt gpt-3.5 and gpt-4 in english and non-english languages to respond to scenarios that should elicit different emotional responses across cultures and assess their cultural appropriateness in a small-scale user study. ## conclusion we find that multilingual models fail to fully capture cultural variations associated with emotion, and predominantly reflect the cultural values of the western world. emotion embeddings from multilingual lms are anchored to english, and the text completions generated in response to non-english prompts are not in tune with the emotional tendencies of users' expected culture. for instance, when gpt-4 is prompted in japanese, it responds as an american fluent in japanese but unaware of japanese culture or values. our results caution against blindly relying on emotion representations learned by lms for downstream applications. using machine translation to transfer labels or utilizing multilingual lms in a zero-shot setting for unseen languages has risksthe multilingual representations of emotion learned by these models do not perfectly reflect how their corresponding cultures express emotion. future research directions our paper motivates the need for future work that transcends current anglocentric lms. this could take the form of higher performing, non-english models in a monolingual setting, or of multilingual models trained on more linguistically and culturally balanced corpora. future work should additionally investigate whether state-of-the-art monolingual models in non-english languages succeed in encoding the respective culture's norms. furthermore, we encourage the evaluation of multilingual models on benchmarks that measure cultural awareness in addition to standard metrics.