Improvements and Extensions on Metaphor Detection (2021)

Abstract: Metaphors are ubiquitous in human language. The metaphor detection task (MD) aims at detecting and interpreting metaphors from written language, which is crucial in natural language understanding (NLU) research. In this paper, we introduce a pre-trained Transformer-based model into MD. Our model outperforms the previous state-of-the-art models by large margins in our evaluations, with relative improvements on the F-1 score from 5.33% to 28.39%. Second, we extend MD to a classification task about the metaphoricity of an entire piece of text to make MD applicable in more general NLU scenes. Finally, we clean up the improper or outdated annotations in one of the MD benchmark datasets and re-benchmark it with our Transformer-based model. This approach could be applied to other existing MD datasets as well, since the metaphoricity annotations in these benchmark datasets may be outdated. Future research efforts are also necessary to build an up-to-date and well-annotated dataset consisting of longer and more complex texts.

## introduction today we are drowning in a sea of social media posts. metaphors serve as strong modifiers to the intentions and meanings of written texts. in the header sentence, the metaphorical use of the word "drown" in the sentence well expresses the worries of the speaker towards the large number of messages in social media, compared to the narrative version of the sentence, e.g. "there are a lot of messages on social media". as defined by @xcite , metaphors involve words used outside their familiar domains. for example, the word "sea" in the leading sentence literally means a large body of water, but it is used metaphorically as a modifier to the phrase "social media posts" to emphasize the abundance of messages in social media. similarly, people can "drown" in water, but not in messages. as shown in this example, metaphors are expressed by the context but not the aspect words themselves, and there are no limits to the number of the metaphorical parts of speech. metaphor detection (md) serves as a strong component in the natural language understanding (nlu) pipeline, since nlu models cannot correctly process the meaning of written text without understanding the metaphors in the content. md serves to aid the nlu models by figuring out the metaphorical parts of speech in each sentence. however, this is a difficult task since metaphors are carried out by long spans of text, not by the appearance of single words or phrases. existing algorithms and neural models are not able to encode long contexts without losing critical information related to metaphors. moreover, the lack of labeled data and the difficulties in labeling metaphorical texts are obstacles to md research as well. due to these issues, the research on md is still in an early stage and has not seen the improvements observed in other nlp tasks in recent years. to reduce the annotation difficulties, researchers have been simplifying md to a classification problem on the metaphoricity of one word or a word pair in each sentence. existing md benchmark datasets are almost all labeled in this manner. while the vua dataset @xcite extends md into a sequential labeling problem, it still limits the metaphorical parts-of-speech to be one per sentence. this setting alleviates the pressure of early md models which are based on handcrafted features @xcite @xcite @xcite . nonetheless, the limitation overly simplifies md and makes existing md sentence her husband often abuses alcohol. explanation to use excessively example abuse alcohol to verify our hypothesis, we fine-tune and evaluate a pre-trained bert @xcite model on all the md benchmark datasets. our model outperforms the previous state-of-the-art models with large margins, as expected. the evaluation results almost all exceed 90% in f-1 scores, suggesting that the existing md settings and datasets are too easy for deep transformer networks to solve. we also extend md to a classification task on the sentence level by removing the labels about the candidate metaphorical words. while the results slightly drop on two md datasets (0.32% and 3.44% in f-1 scores), they are still high, especially on trivial sentences. we believe it is time to expand md to include sentencelevel metaphoricity labeling and to be evaluated on longer, more complex texts. in the evaluations, we uncover flaws in the md benchmark datasets by analyzing the prediction errors our model makes. one example of the annotation errors is displayed in the contributions of this paper are three-fold. first, we report new state-of-the-art performances on three md benchmark datasets to display the power of pre-trained deep transformer networks on md. second, we identify and clean up the annotation errors in one of the md benchmark datasets through manual analysis and validation, which will be made publicly available. third, we believe that the current settings of md are overly simplistic for deep neural network models to solve, based on the evaluation performances of our model. thus, we extend md to a sentencelevel classification task and provide benchmark results on the three md datasets. our future research efforts will involve the construction of an md dataset with sentence-level annotations and longer and more complex texts. ## the metaphor detection task following @xcite , we apply both the sequential labeling and word-level classification settings of md in the experiments. also, we generalize the classification setting of md to the sentence level, disregarding the aspect labels. we describe the three settings of md as follows. for clarity, we use @xmath0, w 2 , ..., w k } to denote a sentence with k words. sequential labeling: given a sentence s, predict one label l i for each word w i indicating whether w i is metaphorical in the context. word-level classification: given a sentence s and an aspect word w @xmath1, with exceptions), predict the metaphoricity label l i associated with the aspect word. sentence-level classification: given a sentence s, predict whether s is metaphorical. the first two settings of md have been extensively studied in previous research. since metaphors are expressed by the linguistic expressions, attributing the metaphoricity of a sentence to an aspect word overly simplifies md. but annotating an md dataset with complex sentences under the sequential labeling setting is too difficult and costly. we provide the sentence-level classification formulation of md for higher annotation quality while avoiding annotating an md dataset on the token level. ## datasets we base our evaluations and discussions on three md benchmark datasets, namely moh @xcite , trofi @xcite , and lcc @xcite . the moh dataset contains sentences from wordnet @xcite examples and the other two corpora are collected from news articles. the average number of words in the moh dataset (7.40) is much lower than the trofi (29.65) and lcc (28.66) datasets. this makes the moh dataset the simplest among the three benchmark datasets. all three datasets provide one aspect word and a metaphoricity label for each sentence. the label is associated with the aspect word. the lcc dataset additionally provides the annotation about the target word of the aspect word in each sentence. different from the other two datasets, the lcc dataset annotates the metaphoricity scores of the aspect words in the set {-1, 0, 1, 2, 3}. in the experiments, we get rid of the -1 labels in the lcc dataset since it denotes uncertain annotations. we display one sample sentence from each dataset in the moh dataset is constructed with 1640 sentences, 410 out of which are annotated as metaphorical. the trofi dataset is made of 1592 literal sentences and 2145 non-literal ones. in the lcc dataset, 493 sentences are labeled as completely literal (0) while 1242, 1251 and 1838 sentences are annotated with metaphoricity scores of 1, 2, and 3, respectively. we perform 10-fold cross-validation on all the three benchmark datasets under the word-level classification, sentence-level classification and sequential labeling settings in the experiments for fairness. though there exist other benchmark md datasets as well, we choose to use the above three datasets intentionally. the vua dataset provides annotations for the sequential labeling setting of md. however, it is not publicly avail-able now so we cannot obtain the data. the tsv dataset @xcite is also widely used, but its training set contains only a list of adjectivenoun pairs without the context. despite the important role the aspect words play in md, the lack of context makes it improper to train or finetune deep transformer-based models on the tsv dataset. clues for the sentence-level metaphoricity prediction cannot be learned in the training process either. thus we do not take these two datasets into our evaluation. ## related work since md is originally defined as a classification task, most early researchers solve it with logistic regression or svm (support vector machine) classifiers. to use the information in the context, researchers concern much about the interrelations between the aspect words and the words closely related to them. thus pos (part of speech) tags and dependency paths are frequently used in md research. @xcite and @xcite group the grammatical relations between each pair of aspect word and its target word into clusters, and they use rules to find out metaphorical combinations. topical information is also a crucial clue to the domain information of a sentence so it is widely used in md. @xcite represent the domain distribution of a sentence with sentence lda. they then base their metaphoricity predictions on the similarities, differences and transition patterns between adjacent sentence pairs. it is interesting, though, that some words are regularly used metaphorically. the intrinsic characteristics of these words are often taken into account when solving md. @xcite assume that highly imaginable words are promising metaphorical words. they lookup the imaginability scores of the aspect words in the mrcpd lexicon @xcite and label the words with high imaginability scores as metaphorical. similarly, bracewell et al. (2014) also consider imaginability in predicting the metaphoricity of words. @xcite and @xcite instead use abstractness of the aspect words or the entire sentences as features in detecting metaphors. other word-based features include the wordnet features (e.g. synonyms and semantic categories) @xcite , the verbnet features (e.g. thematic roles) @xcite , the domains of the candidate-words' arguments @xcite , and the named entity information @xcite . jang et al. (2016) claim that metaphors reveal the emotional or cognitive features of the author, so they use the occurrence of words in the liwc lexicon (tausczik and pennebaker, 2010) to model the sentences in their research. some researchers do not agree with the wordlevel classification setting of md. instead, @xcite claim that every word in a sentence can be metaphorical or literal, making it unrealistic to list all the possible aspect words. they introduce the sequential labeling setting of md and apply crf (conditional random field) to solve it. researchers are actively studying md as a sequential labeling task, but a well-annotated dataset under this setting is difficult to obtain at least for now. not until the year of 2016 did natural language processing (nlp) researchers start to use neural networks in md. with the power of neural networks, more and more researchers begin to examine the use of longer-term context information in md. do do @xcite encode the aspect words with an mlp (multilayer perceptron) taking the vectorized word embeddings, pos features and positional features as inputs. they predict the metaphoricity of each aspect word by feeding their encodings into a logistic regression classifier. @xcite similarly use pretrained word embeddings to represent the aspect words, and they use svd (singular value decomposition) to gain sentence representation for classification. @xcite assume that the metaphoricity of a two-word phrase can be modeled with the cosine similarity between the aspect word embedding and the phrase embedding. to represent the aspect word and phrase, they slide a window of fixed size on the context and use ## model architecture the transformer networks have been overtaking the state-of-the-arts in the nlp field since their emergence. however, there have been very few works that have studied the usage of the transformer networks in md. edge, dankers et al. (2019) made the first and only attempt in applying bert @xcite , one of the most prevalent pre-trained transformerbased models, on md. they build an mlp or additional attention layers on top of bert to make metaphoricity predictions. in our point of view, however, combining bert with complex neural network architectures is a waste of its strength. the additional layers co-trained with bert are only exposed to the task-specific dataset which is much smaller than the bert training data. this makes it difficult to adapt bert to the classification layers. it is good enough to simply use a linear layer to resize the bert output to the prediction space. we specify the neural network architecture underlying bert in figure 1a and show in figure 1b the simple model architecture with which we are able to achieve the state of the art on three md benchmark datasets. our experiments are based on the pytorch implementation of the transformer networks by huggingface @xcite . ## experiments as is mentioned in previous sections, we fine-tune and evaluate bert models for classification and for sequential labeling on the trofi, moh and lcc datasets with 10-fold cross validation. in the experiments, we use the pre-trained bert-basecased model released by google. the model architecture is a 12-layer transformer model with 12 dataset id sentence label pred moh 1 the house looks north. metaphorical literal 2 the huge waves swallowed the small boat and it sank metaphorical literal shortly thereafter. 3 you must adhere to the rules. metaphorical literal 4 they adhere to their plan. literal literal trofi 5 at 9 p.m . , a doctor examines her and orders tests ./. non-literal literal 6 the study , which examined 50 people who were wearing lap belts during auto accidents , concluded non-literal literal that 32 would have " fared substantially better if they had been wearing a lap-shoulder belt . "/" 7 in order to focus federal resources on the ssc , its non-literal literal backers decided that isabelle had to die ./. ## analysis and discussions we manually inspect the predictions our model makes to analyze the causes of the prediction errors. our model performs much better on the trofi dataset than on the moh dataset, benefited from the abundant instances in each word group and the relatively balanced labels. however, we do not fully agree with the annotations either. the label for the word "examine" in sentence 5, for example, is metaphorical, though the usage of "examine" in this sentence well aligns with its literal meaning "test or examine for the presence of disease or infection". similarly, the "examine" in sentence 6 is used in its literal meaning "to question or examine thoroughly and closely", but it is labeled as metaphorical. since the trofi dataset is collected from news articles, abbreviations sometimes cause trouble in the evaluation as well. the name "isabelle" in sentence 7 can well denote a person without preliminary knowledge about ssc (superconducting supercollider) in the context. it is then understandable why our model predicts the sentence as using the verb "die" literally. in the future, we suggest adding the surrounding sentences in the context into the dataset to make md better defined and more appropriate for training deep neural network models. different from the moh and trofi datasets, the lcc dataset does not limit the source words to verbs. another difference is that the labels in the lcc dataset are metaphoricity scores. this makes the lcc dataset more difficult to solve. our model predicts 3 while the label is 2 for the word "form" in sentence 8, for example. possibly our model detects the metaphorical use of "copper" in the same sentence and decides to assign a higher metaphoricity score to the entire sentence. the prediction error of our model in sentence 9 is in a similar case. our model predicts a high score due to the synergy of the metaphorical words "example" and "invader". the annotations in the lcc dataset are sometimes controversial as well. the word "reduce" in sentence 10 perfectly matches the literal meaning "to cut down on", but is annotated as 1 (weakly metaphorical) in lcc, for example. on the other hand, since higher attention weights are put on the evidence for classification in transformer-based models, we examine the attention maps on the last self-attention layer generated by our model under the sentence-level classification setting to interpret the performance of our model. under the sentence-level classification setting, the predictions are made from the hidden states of the cls token. so we evaluate the attention scores of the cls token on all the other words in each sentence. the rule well applies to the word-level classification and sequential labeling settings, only with different tokens on which to base the predictions. to avoid duplication, we only display the attention heatmaps generated under the sentence-level setting in this paper. figure 2 displays the attention heatmaps on moh and trofi examples to reflect the influence of metaphorical polarity on the attention scores and figure 3 contains the heatmaps on lcc examples to show the effect of metaphorical intensities. in figure 2a , the subject "the good player", the verb "times" and the object "his swing" are all heavily attended, indicating the literal usage of the word "time" paired with the words "player" and "swing". quite on the contrary, the verb "visited" in figure 2b is very lightly attended compared to "he" and "illness" in the same sentence, which is a signal of the metaphorical use of the word "visited" in its context. the same pattern applies to the examples in trofi (figure 2d and 2c ) and lcc (figure 3a , 3b , 3c and 3d ) that the more heavily our model attends on an aspect word, the lower chance it is used metaphorically in the context. it is worth noting that when the sentences grow longer, the amount of potential aspect words also increases. the use of these aspect words can be literal or metaphorical at the same time, which benefits classifying the metaphoricity of the sentence as a whole. in figure 2a , for instance, the verb "hit" is used literally with the noun "ball" as well. but there are also cases where the multiple aspect words in one sentence hold different metaphoricities, e.g. the words "swallow" and "sink" in sentence 2 of ## conclusion and future work though difficult, md has been an important task in the nlp community. in this paper, we refined the definitions of md by defining a new task formulation. we also designed and evaluated a bert-based model on three md benchmark datasets. our model largely outperformed the previous state-of-the-art methods. through analysis of the prediction errors made by our model, we found that a large number of prediction errors can be attributed to the simplicity of the datasets and the annotation qualities. to validate this, we reannotated the moh dataset and manually verified the quality of our new annotations. we saw in the experiments that our model achieves very high accuracy on existing md benchmark datasets, meaning that they are becoming overly simple for deep neural networks. our future work will focus on collecting and annotating a new md dataset with more complex texts. regarding the prosperity of social media, we also plan to address the metaphor detection problem on informal text. we hope our work will attract more interest to md and we call for future contributions to solve the problem.