When is a Metaphor Actually Novel? Annotating Metaphor Novelty in the Context of Automatic Metaphor Detection (2024)

Abstract: We present an in-depth analysis of metaphor novelty, a relatively overlooked phenomenon in NLP. Novel metaphors have been analyzed via scores derived from crowdsourcing in NLP, while in theoretical work they are often defined by comparison to senses in dictionary entries. We reannotate metaphorically used words in the large VU Amsterdam Metaphor Corpus based on whether their metaphoric meaning is present in the dictionary. Based on this, we find that perceived metaphor novelty often clash with the dictionary based definition. We use the new labels to evaluate the performance of state-of-the-art language models for automatic metaphor detection and notice that novel metaphors according to our dictionary-based definition are easier to identify than novel metaphors according to crowd-sourced novelty scores. In a subsequent analysis, we study the correlation between high novelty scores and word frequencies in the pretraining and finetuning corpora, as well as potential problems with rare words for pre-trained language models. In line with previous works, we find a negative correlation between word frequency in the training data and novelty scores and we link these aspects to problems with the tokenization of BERT and RoBERTa.

## introduction most data for training and evaluating automatic metaphor detection systems contains binary labeling that only distinguishes between metaphoric and literal tokens. the distinction between novel and conventionalized metaphor has received some, albeit little attention in the context of annotating data for automatic metaphor detection @xcite @xcite . there is however still a lack of publicly available, large-scale annotations that make this distinction. so far, all existing work on novel metaphor in nlp has used crowdsourced anno-tations of metaphors from the vu amsterdam metaphor corpus (vuamc; @xcite . this lack of resources and research in general is problematic in several ways: @xcite suggest that considerable numbers of novel metaphors remain undetected by metaphor detection systems. what is more, the notion of when a metaphor can be considered novel varies in the literature. besides crowdsourced novelty scores, dictionary-based approaches are frequently used. for example, @xcite and @xcite consider a metaphor to be not conventionalized, if the sense in which it is used cannot be found in a dictionary. @xcite already argued for the use of corpus-based dicitionaries in metaphor analysis since they provide transparency and replicability, which would also be an advantage in annotation of metaphor novelty. do @xcite even stress the need to compare their novelty scores with dictionary entries. identifying novel metaphors is especially important for computational approaches to figurative language, since other forms of metaphor can often be easily captured by distributional approaches to meaning. for example, depending on the subsection for the respective register, between 33% and 45% of the prepositions in the vuamc were used in a metaphorical way @xcite . these are often temporal prepositions such as in july, which are seen as spatial words such as in transferred into a temporal sense. these highly conventionalized metaphors are not usually of interest in computational approaches to figurative language. in this study, we evaluate crowdsourced novelty scores by investigating how well they align with dictionary-based definitions of novel metaphor (rq1). we show that there are systematic discrepancies between these two definitions. based on our findings, we develop a new label for novel metaphor and use this label to evaluate current state-of-the-art metaphor detection systems specifically on novel metaphors (rq2). we will make these new annotations on metaphor novelty publicly available. we find that the systems appear to find a higher share of novel metaphors defined by our dictionary-based label, compared to a distinction based on the crowdsourced novelty score and a threshold. we link our results to the findings of @xcite and raise the concern that crowdsourced novelty scores may mainly trace the overall rarity of the words and should be replaced by deeper estimates of the unconventionality of metaphors in future work. ## methodology we systematically evaluate the crowdsourced novelty scores of do @xcite by comparing them with our own binary annotation of mrws that uses a linguistic definition of novel metaphor, according to which a metaphor is considered novel if the contextual meaning of the mrw is not included in a standard dictionary for the language under investigation. @xcite and @xcite used the macmillan dictionary @xcite as a primary lexical resource in their metaphor annotation. unfortunately, the online version of the macmillan dictionary has been shut down in july 2023. in our annotation, we thus use the longman dictionary of contemporary english (ldoce), which was used by @xcite as a secondary source and which is also corpus-based, as our main resource for checking if the contextual meaning of a word is represented in the dictionary, either as a sense description in the entry of a word or as entry for a fixed expression. dmip, introduced in the previous section, considers metaphor novelty (via the availability of sense descriptions in dictionaries) as one criterion for potential deliberateness of a metaphor. it was applied to all mrws in the vuamc by @xcite . unfortunately, the authors did not provide any further information on why they consider a metaphor potentially deliberate. nevertheless, @xcite present novel metaphors according to a dictionary-based definition as a subset of potentially deliberate mrws. this provides a good heuristic to find mrws that are potentially novel. we consequently check the contextual meaning in the dictionary for: all listed cases are manually checked by two annotators: a student assistant trained in metaphor annotation and the first author of this paper. if the contextual meaning of the mrw is not found in the dictionary, the mrw receives the label novel, otherwise it is considered conventionalized. after our additional dictionary-based novelty annotation, we conduct a survey of model performance in a similar fashion to @xcite . we conduct reruns of metaphor detection systems and then compare their performance. for this, we chose deepmet @xcite , because of its strong performance in the 2020 metaphor detection shared task. we moreover selected melbert @xcite since it achieves competitive performance with deepmet, while its architecture is more strongly motivated by linguistic theories on metaphor. here, the layer inspired by spv is particularly interesting, as already @xcite attempted to use spv for the detection of novel metaphor. for melbert we therefore test the entire architecture as well as both layers in isolation. finally, we considered the model used in @xcite , since it was designed with the specific goal of finding novel metaphors. we evaluate their model with both the joint objective as well as with only the metaphor detection task in training, in the following referred to as djokic (joint pred.) and djokic (met. only), respectively. the models are trained on the binary classification task (metaphoric vs. literal), with the data from the vuamc as in the 2020 metaphor detection shared task (including the same training-test splits) and the same hyperparameters as in the respective original papers. all models use the bert and roberta implementations from the huggingface transformer library @xcite 2020 ), we then look at the share of novel metaphors (both according to our definition and according to a novelty score threshold of 0.45) that was detected by the model. ## conclusion and future work we systematically compared the crowdsourced novelty scores of do @xcite with sense entries available in the dictionary. we evaluated state of the art systems for automatic metaphor detection on their performance on novel metaphor wrt. both score-based and dictionary-based labels. finally, we discussed these results by taking word frequency as well as the underlying subword tokenization of bert and roberta into account. we found that measuring novelty purely by crowdsourced scores and a set threshold ignores a wide range of metaphors for which no conventionalized sense descriptions are available, and in addition considers words used in actually conventionalized senses to be novel. as many theoretical approaches to metaphor use dictionaries as a tool for measuring the degree of conventionalization, we thus present new annotations of metaphor novelty for the widely used vuamc that are more in line with these theoretical concepts. we moreover argue that our dictionary-based annotations are more transparent, compared to crowdsourced scores, where the perception of annotators sometimes appears to diverge greatly and where the overall rarity of a word may have a great impact on the annotator's perception of novelty. our evaluation of metaphor detection models suggests that rare words may present a larger problem than words used in unusual contexts or with novel meaning. this is reflected in a higher percentage of recognized novel metaphors according to our linguistically grounded label, compared with novelty defined via crowdsourced scores only. moreover, we found that the underlying word representations of bert and roberta are often formed from sub-word units that do not reflect the actual morphology of derived words and consequently might not be fit for semantically complex tasks such as metaphor detection, especially when the words are used in unusual, novel contexts. one potential line for future research would be a closer look at the perception of annotators in crowdsourcing on whether unusual words have a higher tendency to be perceived as novel metaphors, even though they are used in a relatively conventionalized way. moreover, since the number of truly novel metaphors in the vua corpus is quite small, further data sets that contain a higher share of novel metaphors and, consequently, evaluation on these data sets is necessary to better judge the performance of metaphor detection systems. finally, we would like to propose to extend the task of automatic metaphor detection from a binary classification task to a three-way classification by further distinguishing between novel and conventionalized metaphors.