A howling success or a working sea? Testing what BERT knows about metaphors (2021)

Abstract: Metaphor is a widespread linguistic and cognitive phenomenon that is ruled by mechanisms which have received attention in the literature. Transformer Language Models such as BERT have brought improvements in metaphor-related tasks. However, they have been used only in application contexts, while their knowledge of the phenomenon has not been analyzed. To test what BERT knows about metaphors, we challenge it on a new dataset that we designed to test various aspects of this phenomenon such as variations in linguistic structure, variations in conventionality, the boundaries of the plausibility of a metaphor and the interpretations that we attribute to metaphoric expressions. Results bring out some tendencies that suggest that the model can reproduce some human intuitions about metaphors.

## introduction metaphor is a blooming affair. since the publication of @xcite metaphors we live by, it has been shown that metaphors represent a core cognitive mechanism, identified as a process that aids human beings in the comprehension of abstract concepts. metaphors could be described as a process to endow linguistic expressions with new meaning: a concept of a target domain a is understood in terms of a source domain b. metaphors are pervasive in language, and they are a complex phenomenon to describe and categorize. we can distinguish metaphors for their degree of conventionalization, their linguistic structure (e.g., "a is b", "a of b", etc.), and for the semantic effect they create @xcite , namely concretizing metaphors, animating and personifying metaphors, and synaesthetic metaphors. all these categories can tell us something about the degree of metaphoricity that an expression conveys. many psycholinguistic studies have been carried out to understand how metaphors are perceived by humans and to what extent a metaphor is recognized as such @xcite @xcite . a key aspect is surely the distinction between conventional metaphors (e.g., her lawyer is a shark) and novel (or creative, birdsell (2018)) metaphors (e.g., her mouth was a fountain of @xcite . these types of metaphors are processed differently by humans @xcite . moreover, the latter constitutes an open class, as we have the ability to create new metaphors and make sense out of them. however, this ability is not erratic, but it is ruled by cognitive mechanisms: for example, metaphors have been shown to be justified by analogies between conceptual domains. this ability has also clear boundaries: due to the same mechanisms we can evaluate the plausibility of a given expression, and distinguish creative meanings from nonsense expressions. for instance, even if both an ambassador is a peacock and an ambassador is a fish constitute semantic violations, we accept only the first as a plausible metaphor. another example are the sentences the wind was a howling wolf and the wind was a jumping seat @xcite . transformer language models such as bert @xcite have brought important improvements in metaphor-related tasks (see section 2). however, such models have been used only in application contexts, while analysis aimed at investigating directly what the models capture about this phenomenon has not been conducted. in the first part of our work, we tested whether bert predicts metaphors differing for linguistic structure and conventionality, with a particular focus on whether the model is able to identify the boundaries of metaphoric creativity. of particular relevance for the second aspect is the comparison of novel, unconventional metaphors with nonsense expressions. our ability with figurative language also allows us to assign new meanings to words when they are used metaphorically, independently from the conventionality of a metaphor. therefore, we expect to find information about these meanings in the model's representations of different types of metaphors. to test these aspects, we collected a dataset of conventional and creative metaphors, matched with control literal and nonsense sentences, evaluated by english native speakers for their degree of metaphoricity and semantic plausibility. we carried out two experiments. in experiment 1 we modeled the dataset of human judgments using bert as a language model, to evaluate whether its pseudo-log-likelihood values correlate with the human semantic plausibility scores. our results show that bert seems to distinguish the literal, metaphorical and nonsense expressions. in experiment 2, we used the landmark method introduced by @xcite to test the representation of metaphorical meanings in bert contextualized embeddings. despite limitations given by the size of our dataset, we show that some consistent trends about how the model processes metaphors can be identified by analyzing the model's representations. we observed that several factors such as the layer and the representations analyzed could influence the model's performance. to sum up, we collected various evidence related to what bert learns about metaphorical language. such results pave the way for future investigation, and can be of interest for those who use transformers language models in nlp applications or in metaphor-related tasks. ## related works the computational literature on metaphors has focused on two distinct tasks. metaphor identification involves deciding whether a sequence or a single word is an instance of a metaphor or not. on the other hand, metaphor interpretation concerns the description of the meaning of metaphorical expressions, and is typically cast as a paraphrasing @xcite or a classification @xcite task. even if some recent work has approached the interpretation task @xcite @xcite , much of the literature in the last years has been devoted to metaphor identification @xcite @xcite @xcite . the use of deep learning for this task has become widespread and has contributed to advance the state-of-the-art. while the most recent models proposed for this task differ with respect to the information they exploit, most of them use transformers neural language models like bert in order to obtain an initial representation of the processed sequence. this strategy is now very common and has led to general improvements in performance: four out of the six best systems that participated in the 2020 shared task on the vua corpus @xcite used it, and a system based only on bert and an additional layer outperformed many other systems which were based on explicit linguistic information @xcite . these results strongly suggest that models like bert already possess some knowledge that is relevant to the detection of metaphors. however, to the best of our knowledge, there is no study directly investigating what these models know about metaphors and if this knowledge shares some aspects with that of humans. the application of this last question to other aspects of language has characterized the field of study known as bertology @xcite . researchers in this field intrinsically evaluate bert and its variants on challenge sets (belinkov and glass, 2019), annotated datasets of various sizes that target specific linguistic phenomena. part of the literature on this subject made use of methods which we will adopt in our work. a first method is to study the probabilities assigned by the language model to the instances of a specific linguistic phenomenon, to establish whether the model is able to predict such phenomenon @xcite ; ettinger (2020) among others). this methodology can provide interesting insights when applied to figurative language. since these phenomena can be seen as exceptions to general linguistic rules, they require the model to apply some special ability, which is independent and often in contrast with the signals it has been exposed to during training. another possibility is to directly investigate the model's embeddings which are then used for tasks like metaphor identification. since these representations have been shown to be transferable to a high number of linguistic tasks @xcite , they must encode some sort of general knowledge about linguistic expressions. previous work @xcite ) hypothesizes that they encode knowledge about meaning, and therefore bert and its variants can be seen as distributional semantics models (dsms), @xcite . consequently, they applied the same methods used in distributional semantics to bert, for example comparing the model's representations via cosine similarity to see whether they reproduce human judgments of similarity. however, bert as a dsm has a crucial property which previous dsms lack: the ability to produce contextualized embeddings for each word token. as such, it can positively contribute to the modeling of the meaning of figurative expressions like metaphors, where words acquire new senses which are highly contextdependent. this is why bert can be compared with previous distributional models of metaphors, and the evaluation methods used for these models can be adapted to test bert. in this work, we use the landmark method proposed by @xcite , which tested whether distributional representations of metaphors reflect human intuitions about the meaning of the expressions they model. we will describe the method in more detail in section 3.2. ## discussion likelihood of metaphors with respect to other expressions. the likelihood scores that the models assign to metaphorical expressions lie halfway between those of literal and nonsense sentences, in line with human judgments of semantic plausibility (compare the first two plots in figure 4 .1 with the last one). most importantly, bert discriminates unconventional creative metaphors from nonsense sentences and seems to discriminate highly conventionalized metaphors from literal expressions, 1 these data are reported in the appendix due to space constraints even though the difference is not significant in the second case. this result raises some interesting questions that need to be further investigated. where does bert's ability to discriminate metaphorical expressions come from? how does bert know that, for example, the sentence a smile is an ambassador (@xmath0.02) is more plausible than the sentence a smile is a fishing man (@xmath1.89)? it is likely that the model has not received explicit training on this, since it has probably never or rarely encountered any of such expressions during learning. does this ability share some aspects with the human ability of producing novel metaphors? a possible answer is that the creative metaphors that are in our dataset are grounded on associations that manifest themselves more frequently in language. for example, the creative metaphor the sea whispered on the sand represents a personifying mapping which is expressed by other more frequent expressions such as whispering wind. the model might learn these mappings from training data in the form of associations between groups of words, and then extend them to similar constructions. an analysis of the model's representations, although it does not allow us to draw firm conclusions about bert capability to capture our intuitions about metaphorical meaning, brings out fairly clear trends with respect to the model's interpretation of metaphors, some of which are consistent with findings in the previous literature. general evaluation of bert. our second experiment reveals that in some cases bert representations of metaphorical expressions are significantly more similar to the metaphorical landmarks than to the literal ones. this result is not stable but varies considerably across configurations (see below). moreover, in both versions of experiment 2, bert is almost always above the glove baseline (as can be seen from the plots in figures 4.2 and 4.3). for metwords this is not surprising, since in that case we examined only the static vector of a word, which is obviously more similar to the literal than to the metaphoric landmarks. however, it is less obvious for metsentences, where the baseline is outperformed by 16% in some layers (bert classified correctly 16 more sentences out of 100). ## conclusion in this paper, we adopted different methods used in the literature on the analysis of neural language models to explore how bert captures various human evidence about a linguistic phenomenon that is relatively underexplored in this field, namely metaphors. in experiment 1, starting from the assumption that metaphorical sentences are more plausible than nonsense sentences (even if they are "deviant" from literal ones), we tested whether bert can make a distinction between literal, nonsense and metaphorical expressions with various degrees of conventionality. we show that bert can distinguish between these types of expressions, assigning them different degrees of plausibility in much the same way as humans do. in experiment 2, we wanted to test bert's ability to understand the figurative meaning of metaphorical sentences and the lexical meaning shifts these sentences activate, in comparison with their literal meaning. despite the limited size of our dataset, we can derive some trends. in particular, we observed that the shift is captured by the model's representations (without fine-tuning and when compared via cosine similarity) to a fair extent, especially when it can exploit more contextual information (in upper layers and at the sentence level). we take the results of this experiment as the starting point for future work aimed at investigating how the model can perform when it receives explicit training on this specific task.