Evaluating LLM -Generated Explanations of Metaphors – A Culture-Sensitive Study of Danish (2025)

Abstract: In this study, we examine how well Danish culture-specific metaphors are explained by two of the best performing language models for Danish, namely ChatGPT and Llama. For comparison, the explana- tions are measured against how well cross- lingual (or ’universal’) metaphors are ex- plained by the models; referring here to metaphors that exist in Danish as well as across cultures and languages and in par- ticular in English. To perform our study, we compile a pilot dataset of 150 Danish metaphors and idioms divided tentatively by culture specificity. We prompt the two models and perform a careful qualitative evaluation of the explanations against a four-graded scale. Our studies show that both models are heavily biased towards English since they have much more suc- cess in explaining the metaphors that also exist in English than the culture-specific ones, relying presumably on erroneous transfer from English when dealing with the latter. In particular, the sentiment of the culture-specific metaphors seems to be often ’lost in translation’. We further claim that this strong colouring towards English poses a serious problem in the era of LLMs with regards to developing and maintaining cultural and linguistic diver- sity in other languages.

## introduction metaphorical expressions are an essential part of language and offer considerable cognitive benefits in both oral and written communication by making the content much more personal and engaging @xcite @xcite . metaphorical language is therefore also highly frequent and occur with reference to both universal, culture-specific and personal aspects of life. in other words, metaphors and idiomatic expressions provide an advanced tool for humans to express themselves in abstract and complex situations with reference to highly culture-specific, personal, and opinionoriented values @xcite . with the recent advancements of large language models (llms), however, using metaphors in communication is no longer exclusive to humans. chatbots like chatgpt produce and interpret metaphors when they communicate, and they do so with apparent fluency and equilibrism, in particular for english. a more careful look into the use of metaphors in language models, however, exposes quite a lot of serious problems and cultural biases, even if it is hard to pinpoint exactly from where these problems arise. some may be due to unbalanced training data where some languages are prioritised over others and thereby causing erroneous language transfer and cultural hallucinations @xcite and @xcite . others may derive from a general lack of grounding of the language models with respect to physical objects and spacial conditions, and therefore a lack of ability to 'see' which features from a concrete sense are transferred to the metaphorical meaning; a deficit that may decrease in future with language models becoming increasingly more multi-modal @xcite . under all circumstances, cultural biases in the use of and interpretation of metaphors become particularly evident and problematic when working with the models on medium-resourced languages like the scandinavian ones. standard techniques for evaluating the language models in terms of large-scale benchmarks that are often both rigid and simplistic in nature do not reveal a fully nuanced picture of how this complex figure of speech is dealt with by the models, as mentioned for dan-ish in @xcite . in order to gain better knowledge and understanding of the models' treatment of metaphor in our own language, we therefore aim at i) compiling a pilot metaphor dataset, which is culturally sensitive in that it is developed from danish language resources and from the point of perspective of the danish society, and ii) providing qualitative evaluations by danish native speakers of the explanations given by the models on these metaphors. for our study of model performance, we choose the currently two best performing models in danish according to the scandeval benchmark, namely chatgpt and llama, as reported in @xcite . both chatbots are based on high-performing multilingual transformer models that are well-suited for the kind of conversation on metaphors that we are interested in with our experiments. where llama is a partly open-source model, gpt is a proprietary model. we are however only exploring the models via prompting. the paper is organised as follows: to position our work, we refer in section 2 to related work on metaphors in linguistics, lexicography, and nlp. further, in section 3 we describe the creation of the culture-specific pilot dataset of danish metaphors, looking into the typical source and target domains reflecting cultural aspects of the danish society. section 4 is devoted to our model experiments with chatgpt and llama and explains how we have prompted the models about danish metaphors in both danish and english and with and without a textual context. we describe in section 5 our procedure for evaluating the llmgenerated explanations against a four-graded scale and discuss the annotation agreement results. in section 6 we show and analyse the results and compare how the models deal with culture-specific vs. cross-cultural metaphors, and to which extent the two models differ in performance. all data are made freely available from github foot_0 . finally, in section 7 we conclude and sketch out how our experiments might be scaled up in future work and hopefully used for model improvement. ## related work metaphors have been studied intensively in linguistic theory for decades and are considered an essential figure of speech that is closely related to our conceptual and cognitive system as well as to our culture. the work of @xcite constitutes a landmark in this line of research in stating that metaphors are fundamentally a basic means of understanding complex concepts of feelings and abstractions through mappings from more concrete and directly understandable domains. they further underline that the most fundamental values in a culture will be coherent with the metaphorical structure of the most fundamental concepts in the culture. in recent linguistic studies, focus has further been into getting a deeper understanding of the underlying cognitive processes of metaphors @xcite , as well as achieving consensus both monolingually and across languages of what constitutes a metaphor, often referred to as the metaphor identification procedure (mip) @xcite @xcite . other works examine how metaphor relates to other figurative figures of speech such as irony, sarcasm, and hyperbole @xcite , @xcite . in lexicography, conventionalised metaphors are typically described as specific word senses and most often also labelled explicitly in the dictionary as figurative/metaphor. in many cases the metaphor is also structurally related to its concrete sense in the form of a subsense/main sense relation. one example is the verb to splice in the oxford english dictionary (oed.com), where a subsense to the first sense of the verb is described as in various transferred and figurative uses: to unite, combine, join, mend. also svensk ordbok (swedish dictionary, svenska.se) marks figurative subsenses (as in fönster ('window') with the label äv. bildligt ('also figurative')). in other cases, the dictionaries simply mention the figurative meaning as part of the concrete sense description. in the danish dictionary (det danske sprog-og litteraturselskab, 2024), however, metaphors are almost always described as subsenses labelled 'metaphorical' or 'slang' making them thereby easy to identify and extract for our present study. further, recent wordnet studies suggest a lexical metaphor representation, called chainnet, where the link to the concrete meaning is highly explicit and where features from the concrete sense that are transferred to the metaphorical sense are described in a systematic way in terms of so-called feature transformations @xcite . in nlp, metaphors are also a topic of interest since understanding and representing them is one of the most challenging tasks to deal with in the field. in particular, it has been questioned to which extent llms generalise over the metaphorical meanings and represent the reference to the source domain, or whether they memorise them @xcite @xcite . knowledge graphs of metaphorical facts have further been studied as a means to represent the metaphor relations in the models in order to improve performance @xcite . ## conclusions and future work we have compiled a dataset of culture-specific danish metaphors supplemented with metaphors that are also found cross-culturally, or more specifically between danish and english. our aim was to examine how well the two leading chatbots on danish explain the metaphors and their reference to the source domain, and to which extent we could see a pattern of decline in quality of the explanations deriving from culture-specific expressions that do not have a parallel in english. our experiments confirm our hypothesis quite strongly. culture-specific metaphors are highly complex for the models to interpret, and the explanations indicate that erroneous language transfer from english takes place to a large extent, leading to strongly biased and/or hallucinated explanations. in particular, the models have problems in capturing the right sentiment of the metaphors, distorting thereby the specific danish 'tone of voice'. chatbots like chatgpt are currently rolled out throughout society, in particular through co-pilot enterprise, and people are using them for all kinds of tasks. in this context, the strong colouring towards english that we have documented in our work, indicates that the developing and maintaining of cultural and linguistic diversity is under strong pressure, and that the development might move very fast. this tendency is reinforced by the fact that the hallucinations are very wellformulated and on the surface convincing, meaning that only the highly experienced language user can dismiss garbled output. to extend our study, we would like to i) expand our danish dataset, ii) include metaphor studies for the other scandinavian language, and also iii) go deeper into the understanding of the inner wheels of the models with respect to where the tipping point is found between beneficial language transfer on the one hand and erroneous transfer that leads to cultural biases on the other. one way to proceed in improving the models (in addition to ensuring more danish training material in the first place) is via fine-tuning or retrieval augmented generation with use of knowledge graphs or other structured information sources. knowledge graphs can be compiled from culture-specific metaphor lists derived from existing dictionaries or corpora, or from wordnets enriched with feature transformations in a 'chainnet'-like fashion. all in all enrichments that could potentially lead to better and more culturally diverse language interpretation and generation.