Multi-lingual and Multi-cultural Figurative Language Understanding (2023)

Abstract: Figurative language permeates human communication, but at the same time is relatively understudied in NLP. Datasets have been created in English to accelerate progress towards measuring and improving figurative language processing in language models (LMs). However, the use of figurative language is an expression of our cultural and societal experiences, making it difficult for these phrases to be universally applicable. In this work, we create a figurative language inference dataset, {pasted macro ‘DATASETNAME’}, for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. We assess multilingual LMs’ abilities to interpret figurative language in zero-shot and few-shot settings. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data, emphasizing the need for LMs to be exposed to a broader range of linguistic and cultural variation during training. Data and code is released at https://anonymous.4open.science/r/Multilingual-Fig-QA-7B03/

## introduction when you are feeling happy, do you think that you are "warm" or "cold"? if you are a monolingual english speaker, you will likely answer "warm", and use expressions like "this really warmed my heart". however, if you are a native hindi speaker, you may answer "cold", and use expressions like दल को ठं डक पढ़ना ("coldness spreads in one's heart" ) @xcite . linguistic communication often involves figurative (i.e., non-literal) language @xcite figurative expression inference yo omah iku kaya istana (the house is like a palace.) omah iku apik banget. (the house is very nice.) omah iku elek banget. (the house is very ugly.) id rambutnya seperti bihun. (her hair is like vermicelli.) rambutnya keriting. (her hair is curly.) rambutnya lurus. (her hair is straight.) hi जीवन मीठा गु लकन्द है । (life is sweet gulkand. ) जीवन अ ा है । (life is good.) जीवन बु रा है । (life is bad.) kn ಅದು ದೋಸೆಯಂತೆ ಗರಿಗರಿಯಾಗಿತ್ತು. (it was crispy like a dosa.) ಅದು ಗರಿಗರಿಯಾಗಿದೆ (it is crisp.) ಅದು ಗರಿಗರಿಯಾಗಿರಲಿಲ್ಲ (it was not crisp.) sw maneno yake ni sumu. (his words are like poison.) maneno yake yanaponya. (his words heal.) maneno yake yanaangamiza. (his words are devastating.) we find english figurative language datasets may not have cultural relevance for other languages ( §2). this is a general challenge in nlp, as assumptions of common knowledge and important topics to talk about vary from culture to culture @xcite . in order to better train multilingual models to interpret figurative language, as well as to understand linguistic variation in figurative expressions, we construct a multilingual dataset, mabl (metaphors across borders and languages), of 6,366 figurative language expressions in seven languages ( §3). examples are shown in we use the dataset to conduct a systematic analysis of figurative language patterns across languages and how well they are captured by current multilingual models ( §4). we find that figurative language is often very culturally-specific, and makes reference to important entities within a culture, such as food, mythology, famous people, or plants and animals native to specific regions. we benchmark multilingual model performance ( §5) and analyze model failures ( §6), finding that zero-shot performance of multilingual models is relatively poor, especially for lower-resource languages. according to @xcite , main factors which poses challenges on the performance in such cases are cross-lingual transfer and concept shift across languages. however, we observe that concept shift seems to play a larger role due to culturally specific examples. adding a few examples in the target language can improve performance of larger models, but this is more beneficial for lower-resource languages. this highlights the importance of including culturally relevant training data, particularly data that highlights not just the existence of a concept, but also how people view that concept within that culture. ## linguistic and cultural biases of existing figurative language datasets to confirm the importance of building a multilingual, multi-cultural figurative language dataset, we first performed a pilot study to examine the feasibility of instead translating an existing figurative language dataset, fig-qa @xcite , from lang. fr hi ja incorrect 13% 40% 21% culturally irrelevant 17% 20% 17% english into other languages. while there are wellknown problems with using translation to create multilingual datasets for tasks such as qa @xcite , it is still worth examining these issues in the context of figurative language in particular. we used the google translate python api to translate the development set into languages that the authors of this paper understood. 3 these were french, japanese, and hindi. each annotator annotated 100 examples for both correctness (whether or not the translation was accurate), and cultural relevance (whether or not the expression was one that would make sense to a native speaker from the culture where the language is predominant). as seen in the shift in topics discussed in cultures associated with different languages can be captured by native speakers familiar with that culture, motivating our collection of natural figurative language examples from native speakers. ## conclusion despite being relatively widespread, figurative language is relatively under-studied in nlp. this is especially true for non-english languages. to enable progress on figurative language processing, we create mabl, a figurative inference dataset across seven languages. we find considerable variation in figurative language use across languages, particularly in the unique objects that people invoke in their comparisons, spanning differences in food, mythology and religion, and famous figures or events. this variation is likely due to differences in cultural common-ground between the countries in which these languages are spoken. we find that multilingual models have considerable room for improvement on this task, and cross-cultural shift may play a significant role in the performance degradation from english. we encourage the nlp community to further examine the role that culture plays in language, and note that figurative language can be used as a testbed to examine crosslinguistic and cross-cultural variations.