Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding (2025)

Abstract: This paper presents a comprehensive evaluation of the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations. Although metaphor processing has gained significant attention in Natural Language Processing (NLP), previous research has been limited to single-dataset evaluations and specific task settings, often using artificially constructed data through lexical replacement. We address these limitations by conducting extensive experiments using diverse publicly available datasets with inference and metaphor annotations, focusing on Natural Language Inference (NLI) and Question Answering (QA) tasks. The results indicate that LLMs’ performance is more influenced by features like lexical overlap and sentence length than by metaphorical content, demonstrating that any alleged emergent abilities of LLMs to understand metaphorical language are the result of a combination of surface-level features, in-context learning, and linguistic knowledge. This work provides critical insights into the current capabilities and limitations of LLMs in processing figurative language, highlighting the need for more realistic evaluation frameworks in metaphor interpretation tasks. Data and code publicly available: https://github.com/elisanchez-beep/metaphorLLM

## introduction figurative language is a recurrent element in our daily communication. it reshapes our perception and understanding of knowledge, allowing us to better comprehend and transmit abstract concepts from a more concrete domain. @xcite defined these mental associations as conceptual mappings, which are verbalized through language into linguistic metaphors, subject of study of our work. the widespread use of metaphors in everyday language has boosted the popularity of research 1 https://github.com/elisanchez-beep/ metaphorllm figure 1 : example from impli @xcite dataset with one premise and two hypotheses. the premise contains the verb devour used metaphorically, equivalent to 'to read vividly'. note that the inference relation is affected by the metaphorical expression. on this type of figurative language within the field of nlp. large language models (llms, to refer to decoder-only models) are now widely available, not only for nlp researchers but also for all kinds of users in chatbot assistant forms. moreover, figurative language, metaphors specifically, are key for other nlp tasks, such as hate speech detection @xcite , political discourse analysis @xcite , or mental illness detection @xcite . for this reason, it is essential to critically assess the capabilities of llms and their applications, particularly their ability to understand complex cognitive-linguistic phenomena like metaphorical expressions. this work focuses specifically on metaphor interpretation. initial publications on this topic were based on statistics and machine learning @xcite @xcite . the introduction of transformer models @xcite ) marked a major milestone, providing a more accessible and open framework for evaluating and probing language models' abilities to process metaphorical expressions @xcite . although there has been some recent interest in researching metaphorical analogies or concep-tual metaphors @xcite @xcite , our work is centered on linguistic metaphors. on this particular topic, while interesting, previous work studying the capabilities of llms to understand linguistic metaphors has been hindered by several shortcomings. from a task perspective, the data used in these approaches has typically been developed through lexical substitution. this means that there is a metaphor in a premise and that a literal version, used as the hypothesis, is generated by replacing the metaphor with its literal sense @xcite @xcite . as a result, these resources are not representative of metaphor occurrence in natural language utterances and incorporate biases, namely lexical overlap, that may affect performance in the nli or qa tasks @xcite @xcite . moreover, most of the previous work tests llms only on a single dataset prepared ad-hoc for each specific experimentation. furthermore, the evaluation scenarios are limited to one setting or task. thus, there is a lack of comprehensive, crossdataset, and multi-prompt evaluation of the interpretation abilities of llms on diverse and natural language corpora. as a consequence, results have been rather mixed, with some studies claiming that llms achieve understanding over chance but below human performance @xcite while others conclude that llms can accurately establish entailment relations between the figurative and their literal counterparts @xcite . in order to address these issues, this work provides comprehensive experimentation to evaluate the ability of llms to understand and interpret metaphorical language across several datasets and tasks. more specifically: to tackle these questions, we gathered the most recent publicly available datasets with inference and metaphor annotations. then, we conducted comprehensive experiments on these data on evaluation scenarios based on nli and qa @xcite @xcite . the results and subsequent analyses led to the following key contributions: ## related work as mentioned in the previous section, metaphor interpretation has been formulated through other pivot tasks from natural language understanding (nlu), such as nli @xcite , qa @xcite or paraphrasing @xcite . initially, supervised and unsupervised deep learning techniques were employed @xcite , using datasets specifically designed for the task @xcite or other external resources with exploitable linguistic information @xcite . along with transformers' emergence, metaphor interpretation was approached as a sequence classification task to evaluate masked language models (encoders). @xcite evaluate bert's capability to generate literal substitutes by leveraging metaphor detection vuam @xcite and moh-x @xcite datasets. similarly, @xcite develop their evaluation corpus with conventional and novel metaphors to test if bert distinguishes between metaphor and nonsense @xcite . recent studies framed metaphor interpretation through nli or qa to evaluate llms such as gpt @xcite and llama @xcite . the work of coms , a et al. ( 2022 ) introduces a set of 300 metaphorical questions and paired implications to ask models whether these implications are true or false. in addition to english resources, @xcite publish a dataset with seven underrepresented languages with a high number of speakers to assess the impact of socio-cultural characteristics on model performance. datasets specifically designed to evaluate metaphor interpretation through inference are relatively scarce and based on lexical replacement @xcite . they usually include different annotations, such as metaphoricity, acceptability, or correctness of the paraphrases. however, for our study, we focus on datasets that include inference labels that are affected by metaphorical expressions. in the following, we will provide a more detailed overview of recent publications that present both metaphor and inference labels, which we will leverage for our multi-task, multiprompt, and cross-dataset evaluation @xcite of the capabilities of llms to understand metaphorical language. ## evaluation datasets in this section, we will describe the main features of the datasets used for the experiments. we collected these resources because they are labeled for nli and include metaphorical language. all of these corpora are manually validated. except for meta4xnli, the majority of them were developed through lexical replacement. thus, results might be biased by their templatic nature and lexical artifacts @xcite . in addition, we describe the process of generating adversarial literal examples to test whether correct inferences were due to the understanding of metaphorical expressions by llms or related to more surface-level features. we include the distribution of the data in impli @xcite this dataset is a compilation of 24k silver and 1.8k gold nli pairs with metaphorical and idiomatic language. in this work, we will use the 668 instances with metaphors from the gold subset. to develop gold pairs, @xcite collected metaphorical sentences from the vuam corpus @xcite , gutenberg poetry corpus (jacobs, 2018), and from @xcite . annotators were asked to rephrase the metaphorical sentence by removing the figurative expression. to create not-entailed hypotheses, annotators rewrote premises, adding elements to change their meaning but maintaining as much as possible the lexical overlap. flute @xcite this benchmark provides 9000 nli pairs with metaphor, simile, sarcasm, and idioms. regarding metaphors, they collected 750 metaphoric sentences from exist-ing datasets, namely, figurative-nli, impli, and @xcite . they prompted gpt-3 with metaphorical sentences to generate a literal paraphrase as entailment. for contradictions, they used the gpt-3 generated literal sentences and prompted the model to invert the sentence and contradict the metaphor itself. results were reviewed and post-edited when needed by annotators. in total, they obtained 1500 pairs, from which we leveraged the 248 pairs belonging to the gold test set. fig-qa @xcite the dataset follows the winograd schema format @xcite , wherein human annotators created paired sentences that share identical opening segments but conclude with contrasting metaphorical meanings. each sentence pair is accompanied by two corresponding hypotheses: one that represents an entailed paraphrase and another that is not entailed. for our experimental analysis, we used the 2188 nli pairs extracted from the development set, as the test set labels were not accessible. meta4xnli (sanchez-bayona and agerri, 2024) this parallel dataset includes nli instances for spanish and english. since other corpora are only available in english, we used only samples in this language. it is a compilation of existing nli datasets, xnli @xcite and esxnli @xcite . in contrast to the other datasets, it contains spontaneously generated natural language text for nli tasks which was subsequently annotated for metaphoricity by the meta4xnli project. our experiments use the test set with metaphorical sentences, that is, a total of 598 nli pairs. ## experimental setup in this section, we provide technical information about the experiments and the characteristics of each prompt formulation. we will also detail the settings used for the generation of the literal paraphrases. ## results we first report the results of zero-and few-shot experiments in all experimental settings with the original metaphor datasets (section 5.1, -met results in ## analysis of results firstly, to explore the factors contributing to the disparity in results between the datasets, we tried to capture some form of lexical overlap between premises and hypotheses via levenshtein distance @xcite , and average sentence length. these analyses aim to provide further insight into how sentence structure and similarity may influence the models' performance @xcite . secondly, we manually inspected some errors from the evaluation with adversarial literal paraphrases. all the examples used for quantitative and qualitative analyses come from the experimental setup that obtained the best average results, that is, cot prompt with qwen2.5-72b-instruct. ## concluding remarks the main aim of this work is to test the capabilities of llms to understand metaphorical language. in order to do so, we evaluate whether llms can predict the inferential relationship between a premise and a hypothesis when metaphorical expressions affect the inference. more specifically, we use multiple available datasets in english, some developed through lexical replacement and others with natural spontaneously generated utterances and framed the task as nli and qa. in addition, we performed comprehensive experimentation with various verbalizations and zero-and few-shot settings. also, we automatically developed a parallel version of the original datasets with literal paraphrases that served as adversarial examples. the results indicate that llms' performance is more influenced by features like lexical overlap and sentence length than by metaphorical content, demonstrating that any alleged emergent abilities of llms to understand metaphorical language are the result of a combination of surface-level features, in-context learning, and linguistic knowledge. through our experiments, we demonstrate that performance fluctuates remarkably depending on the dataset features, especially lexical overlap (a consequence of data created through lexical substitution) between premise and hypothesis, as well as sentence length. a higher overlap and shorter sentences boost the performance, while naturally occurring sentences, lower overlap, and shorter sentences result in poorer performance. moreover, llms with a smaller number of parameters show almost random performance in zero-shot settings, which can be easily improved with few-shot prompting, while models with more parameters display a more stable performance across prompts. furthermore, formulating the task as qa and providing few-shot examples enables superior performance, especially when combined with cot, which helps to outperform any other scenario, including some strong baselines. we hypothesize that this is due to the post-training of the instruct models @xcite . our manual error analysis shows that automatic generation of literal paraphrases requires exhaustive human evaluation, since models still include metaphors in the newly generated sentences. we argue that this behavior reveals llms' lack of ability to discriminate between metaphorical and literal expressions, although more research is required, perhaps with manually generated paraphrases in future work. we believe that this work provides critical insights into llms' current capabilities and limitations in processing figurative language, highlighting the need for more realistic evaluation frameworks in metaphor interpretation tasks.