How Are Metaphors Processed by Language Models? The Case of Analogies (2024)

Abstract: The ability to compare by analogy, metaphorically or not, lies at the core of how humans understand the world and communicate. In this paper, we study the likelihood of metaphoric outputs, and the capability of a wide range of pretrained transformer-based language models to identify metaphors from other types of analogies, including anomalous ones. In particular, we are interested in discovering whether language models recognise metaphorical analogies equally well as other types of analogies, and whether the model size has an impact on this ability. The results show that there are relevant differences using perplexity as a proxy, with the larger models reducing the gap when it comes to analogical processing, and for distinguishing metaphors from incorrect analogies. This behaviour does not result in increased difficulties for larger generative models in identifying metaphors in comparison to other types of analogies from anomalous sentences in a zero-shot generation setting, when perplexity values of metaphoric and non-metaphoric analogies are similar.

## introduction analogical reasoning is critical to deep language understanding, as it is a core mechanism of human generalization and creativity @xcite . analogical thinking includes figurativeness (e.g. the mind is a sponge.), in which humans naturally express relationships based on non-literal connections. traditionally, metaphors have been challenging to model from a computational perspective @xcite and in the context of nlp. this is due to their proteiform nature, conventional or creative, concise or structurally more complex. some limitations might have been lifted given the new wave of language models (lms) that have revolutionalised the field of nlp and beyond @xcite @xcite . indeed, recent studies on the last generation of large transformer-based lms show enhanced abilities to perform analogical reasoning @xcite , suggesting that models of a larger size may gain the ability to process complex analogies. as a conceptual innovation device, figurative analogies have also been studied in relation to the fluency, creativity and originality of students' writing @xcite . creative writing support tools specialising in metaphor generation have been @xcite . the emergence of llms as writing assistants has further highlighted the importance of understanding how metaphors are processed by lms, especially given some limitations pointed by their users related to the generation of poor metaphors and overly predictable endings, to name a few @xcite . motivated by the recent advances in language modeling and the need for understanding how lms process metaphors, we establish the following two research questions: research question 1 (rq1). how do language models distinguish metaphors from literal and anomalous sentences? in particular, we are interested in determining if the likelihood of metaphors compared to both literal and anomalous sentences is consistent across models. for this, we are also interested in analysing the differences among model families and, particularly, sizes. this research question is addressed in section 5. ## background in this section, we provide more details on the relation between analogies and metaphors, and discuss other terminology used across the paper. analogies. analogy is a type of similarity in which the same system of relations holds across different sets of elements @xcite . the analogies that we consider express parallels across pairs of concepts captured minimally through attributive structures a is-a b or more explicitly with comparisons of the form a is to b what c is to d. mapping conceptual structures to understand or create analogies comes naturally to humans, but it is generally challenging for computational models because it conveys implicit semantic attributes and relations. for example, understanding the statement ketchup is to tomato what guacamole is to avocado involves an internal representation of the relation x is made of mashed y. in two-word analogies, the relation of interest is implicit. for example, from the sentence his editing style was a chainsaw, one can reconstruct an implicit 4-term analogy: his editing style was to the text what a chainsaw is to a forest. 2 metaphors. within conceptual metaphor theory (cmt), a metaphor is defined as a mapping process between broad conceptual domains @xcite , which occur at the level of thought and manifests through language. in order to study the ability of models to identify metaphoric mappings, we experiment on linguistic expressions constrained in form. in this paper, a metaphor is defined as a word (or a set of related words), that can be understood through the prism of another distant word (or another paired set of related words), without relying on additional explicit context. we feed minimal metaphoric sentences that almost only contain the words forming mappings into the models, to gain a better understanding of how they are represented by the lms. according to @xcite , all metaphors mediate an analogy , but not all analogies are metaphors. the relation between metaphors and analogies has been much debated. researchers who refer to shared features and structural analogies as the basis of metaphors disagreed with some conceptual mapping theorists who have argued that similarity is not the basis for metaphors @xcite . @xcite and @xcite introduce a framework that intends to unify both views. the present study adopts this theoretical framework. metaphors are treated here as a species of analogies. more recently, @xcite proposed a taxonomy of analogies where the metaphors included in our dataset would be classified as semantic and pragmatic analogies, i.e. the two most complex types of analogies, which require good semantic representations, and sometimes pragmatic knowledge, to be processed accurately. among all analogies, we hypothesise that metaphors might be even harder to process, because they are more structurally variable than other types of analogy. the attribute and relation conveyed are partial matches. they can even violate structural consistency @xcite . according to @xcite , a good metaphor is one that involves two very different domains. it is not an absolute criterion, but good metaphors are often cross-domain (far) analogies, which adds to the complexity. another specificity of metaphors is that the mapping is not reversible @xcite , i.e., metaphors have directionality. for example the acrobat is a hippopotamus suggests a clumsy acrobat and the hippopotamus is an acrobat suggests a graceful hippopotamus. for these two reasons, lms may struggle to catch capture the parallelism between the concepts involved in a metaphor in comparison to other types of analogies. anomalies. semantic anomalies can resemble metaphors in the sense that they may eventually bring together concepts that are distant from each other. unlike metaphors, the two concepts do not share any obvious properties. for example, a chair is a syllogism can be considered to be an anomaly @xcite . fallacious analogies made of two word pairs in the a is to b what c is to d structures are constructed by mapping words that are not connected by the same relation. for example, having the first pair linked by a part of relation and the second pair by a made of relation. foot_2 ## related work automatic metaphor processing research has seen a garnered increased in recent years, partially due to the encouraging performance of language models on existing benchmarks @xcite . however, there have been almost no studies on metaphors in the context of analogies. ## experimental details: model selection and perplexity computation our aim in this paper is to evaluate a wide range of diverse lms in terms of architecture and size, which are presented below. ## language model representation of metaphoric analogies in this section, we aim to understand how lms identify metaphors in comparison to other types of analogies or literal statements, and how models can identify them from semantically anomalous sentences. to this end, we rely on three datasets containing sets of metaphoric and literal sentences, which are presented in section 5.1. following this, we rely exclusively on zero-shot experiments, first by computing perplexity scores (section 5.2) and then by studying the abilities of the models to identify metaphors by following instructions (section 5.3). ## conclusion in this paper, we have analysed the capabilities of lms to perceive and identify metaphors. using perplexity as a proxy to measure plausibility in lms, we observe that, in general, lms perceive metaphors as less likely, and are often perceived closer to anomalous sentences than literal ones. in general, lms struggle more often to distinguish metaphors from anomalous sentences even when instructed to do so, although this gap diminishes with newer and larger models. as a result of this finding, we also investigated whether these results would be reflected in how models can distinguish metaphors from anomalies in a wider context. the results show that, at least for the new generation of lm-based conversational agents, this does not appear to be as problematic. several follow-up questions remain unaddressed in spite of these findings. what is the role of metaphors in generative models? do lms generate (new) metaphors in the context of a conversation, or do they resort to existing expressions and literal sentences? in the context of computational linguistics and semantics, it would be interesting to better understand how metaphors are internally represented or encoded in this new generation of lms.