An analysis of language models for metaphor recognition (2020)

Abstract: We conduct a linguistic analysis of recent metaphor recognition systems, all of which are based on language models. We show that their performance, although reaching high F-scores, has considerable gaps from a linguistic perspective. First, they perform substantially worse on unconventional metaphors than on conventional ones. Second, they struggle with handling rarer word types. These two findings together suggest that a large part of the systems’ success is due to optimising the disambiguation of conventionalised, metaphoric word senses for specific words instead of modelling general properties of metaphors. As a positive result, the systems show increasing capabilities to recognise metaphoric readings of unseen words if synonyms or morphological variations of these words have been seen before, leading to enhanced generalisation beyond word sense disambiguation.

## introduction metaphor is a type of figurative language where meaning transfer occurs via similarity between two conceptual domains. in examples 1 to 3, the metaphors attacked, bashed and ceasefire stem from a transfer from the domain war (or fight) to the domain argument @xcite . 1 (1) he attacked my argument. metaphoric instances can stem from such regular metaphoric patterns equating two domains habitually or conventionally. 2 however, they can also be novel/unconventional such as the famous emily dickinson metaphor hope is the thing with feathers, which does not correspond to a well-known metaphorical pattern. even within a metaphorical pattern such as argument is war, there are degrees of conventionality with example 1 being more conventional than examples 2 and 3. to a human, unconventional metaphors tend to be more noticeable. metaphor detection has been studied extensively in nlp in recent years (see @xcite for overviews). state-of-the-art approaches in metaphor detection build strongly on language models and word embeddings, with more than half of the participants in the 2020 shared task on metaphor detection @xcite using a variant of bert language models @xcite . evaluations on the standard metaphor recognition test sets report scores that creep up steadily, using such methods. we investigate whether these models really are able to learn general properties of metaphor. to do so and to go beyond word sense disambiguation, they should be able to (i) recognise conventional and unconventional metaphors (ii) be able to perform well on rarer word types that often follow the same metaphoric patterns as frequently seen ones and (iii) be able to generalise across synonyms and morphological variations of word types (such as making the inference from example 1 to 2). our contributions are as follows: ## results we use the standard vua-all-pos and vua-seq training/test splits. evaluation measures are precision, recall and f1 for the metaphoric class as well as accuracy on all target tokens. ## analysis we will now investigate (i) how well the current systems handle conventional vs. novel metaphors, (ii) if they can handle frequent and less frequent word types and (iii) what the influence of morphology as well as semantic similarity is on their capability to handle metaphoric usage of word types not seen in training. in our opinion, frequency and conventionality analysis are crucial to test whether the system mainly recognises frequently seen, word-specific meanings that might also be specified in dictionary entries (see the metaphors in example 4 in section 2.2) or whether it is able to generalise the concept of metaphor to word types not seen in training or newly occurring metaphoric transfers. we conduct the analysis on our reruns of gao and mao as well as bert on vua-seq and extend the analysis with the dm models on vua-all-pos. this includes the state-of-the art systems on both datasets as well as 3 different language models (and extensions). ## conclusion and future work we compared several recent models for metaphor recognition, which all build on language models and/or embeddings. we cast doubt on the capability of these models to actually learn general properties of metaphor as the models do not perform well on non-conventionalised metaphors and rarer word types. they excel on frequently seen word types with conventionalised metaphoric meanings, which is more akin to word sense disambiguation. however, they do show generalisation capabilities beyond word sense disambiguation for words unseen in training if morphological variations or synonyms have been seen in training. the latter finding suggests that metaphoric patterns that hold across morphological variations and synonyms might to a degree be learnt by current systems. to follow this up, in the future, we will extend current metaphor corpora by annotating them for metaphorical patterns (or what @xcite call conceptual metaphors) as well as existence of annotated metaphorical meanings in wordnet.