I llini M et: I llinois System for Metaphor Detection with Contextual and Linguistic Information (2020)

Abstract: Metaphors are rhetorical use of words based on the conceptual mapping as opposed to their literal use. Metaphor detection, an important task in language understanding, aims to identify metaphors in word level from given sentences. We present IlliniMet, a system to automatically detect metaphorical words. Our model combines the strengths of the contextualized representation by the widely used RoBERTa model and the rich linguistic information from external resources such as WordNet. The proposed approach is shown to outperform strong baselines on a benchmark dataset. Our best model achieves F1 scores of 73.0% on VUA ALLPOS, 77.1% on VUA VERB, 70.3% on TOEFL ALLPOS and 71.9% on TOEFL VERB.

## introduction metaphors are a form of figurative language used to make an implicit or implied comparison between two things that are unrelated @xcite . they are widely used in natural language, conveying rich semantic information which deviates from their literal meaning. for instance, in the sentence "tom has always been an early bird waking up at 5:30 a.m.", the phrase "early bird" does not mean a real animal but refers to someone doing something early. the ubiquity and subtlety of metaphors present challenges to language understanding in natural language processing. detecting metaphors is the first step towards metaphor understanding, which helps to uncover the meaning more accurately. metaphor detection has been used in a variety of downstream applications such as sentiment classification @xcite and machine translation @xcite . some existing approaches to metaphor detection rely on linguistic features such as lexicon based metaphor constructions, lexical abstractness and word categories in wordnet @xcite . most of these approaches either do not consider the contextual information or only focus on limited contexts. others use unigram based features regardless of their contexts @xcite , and still others identify metaphors in the limited context of subjectverb-object triples @xcite . we note that the contextual information is crucial for metaphor detection. as shown in metaphor: i think that we need to begin from facts and fix important data. literal: they couldn't fix my old computer, and i had to buy a new one. recent studies incorporating contextual information into metaphor detection include unsupervised approaches @xcite and supervised models @xcite . neural networks such as long short term memory (lstm) have achieved state-of-the-art performance in metaphor detection due to their ability to encode contextual information @xcite . recently proposed contextualized representation models that have been widely used include embeddings from language models (elmo) @xcite , representations from transformer (bert) @xcite and robustly optimized bert pretraining approach (roberta) @xcite . their use has shown dramatic improvements in the performance of several nlp tasks. these models are pretrained on large text corpora to encode rich contextualized knowledge into the semantic representation of words with deep network structures. in this paper, we build our metaphor detection model upon roberta to leverage its strength in capturing contextual information. in addition, we enhance the power of contextualized representation by using linguistic features. using external resources our model integrates features such as word concreteness, which serves to complement the contextual knowledge in roberta embeddings and aid the task of metaphor detection. this model was our contribution to the second shared task on metaphor detection @xcite . our best model achieves f1 scores of 73.0% and 77.1% on all words and verbs alone respectively for metaphors in the vua dataset. our model performance is 70.3% and 71.9% on all words and only verbs respectively in the toefl dataset. we make our implementation available for a wider exploration foot_0 . feature-based approaches. many recent works have explored the use of various linguistic features for automatic metaphor detection. these features include word abstractness @xcite , wordnet features, hypernyms and synonyms @xcite , syntactic dependencies and semantic patterns @xcite , word imageability @xcite as well as word embeddings @xcite . neural network models. deep learning models are becoming very popular in various downstream applications of natural language processing owing to the ability to train models in an endto-end manner without explicit feature engineering. approaches built upon neural networks have shown great successes in metaphor detection task as well. different structures of neural networks have been extensively studied to better encode semantic knowledge by capturing metaphorical patterns @xcite . sequential models such as recurrent neural networks (rnn) demonstrate strong performance in metaphor detection @xcite ). an lstm is applied to identifying metaphors together with a convolutional neural network (cnn) @xcite . the model utilizes pretrained word2vec word embeddings. another approach built on sequential models is a bidirectional lstm model augmented with pretrained contextualized embeddings @xcite . a recent work draws inspiration from linguistic theories, and proposes rnn hg and rnn mhca, which are variants of rnn models @xcite . assuming that pretrained glove embeddings carry literal meaning, the model rnn hg detects metaphors by comparing glove embeddings with the contextualized word representations learned by the rnn. the other variant, rnn mhca, integrates the multi-head attention mechanism in model hidden states, and enriches word representations with information from broader contexts. ## metaphor detection in this section, we will introduce our model design, training and prediction for metaphor detection. ## experiment we empirically evaluate our model for metaphor detection in this section. precision, recall and f1 score are the evaluation metrics used in our experiments. we report metaphor detection results on words of any pos tags as well as verbs alone in the test set provided by the shared task foot_1 . dataset. two datasets were used for the evaluation of metaphor detection -the vu amsterdam metaphor corpus (vua) @xcite and the toefl dataset @xcite . the sentences in these two datasets were manually annotated for the task of metaphor detection at the word level. the vua dataset was collected from the bnc in four genres including news, academic, fiction and conversation. it provides 12, 122 sentences for training, and 4, 080 sentences for testing. around the toefl dataset consists of essays written by non-native speakers of english. it contains 180 train essays with 2, 741 sentences and 60 test essays with 968 sentences. in the training partition, 7% of toefl tokens are metaphors. models. we report and compare the performance of the variants of our model. the methods we discuss in this work include: ## discussion in this section, we perform an analysis to explore the strengths and weaknesses of our model. since we did not have ground truth labels for the test instances, we divided the training data into train, dev and test sets with a ratio of 4 : 1 : 1 for the purpose of error analysis. we trained and tuned our best performing model on the resulting train and dev sets respectively, and evaluated it on the test set. model performance on different pos tags. we evaluate the model performance on words of different part-of-speech tags. on the vua dataset, determiners ("dt"), prepositions and conjunctions ("in") had the highest f1 scores, while adjectives ("jj") and plural nouns ("nns") got the lowest f1 scores among all words. on the toefl dataset, the best-performing words were adjectives ("jj") and verbs ("vb"). words ranking at the bottom were prepositions and conjunctions ("in") and nouns ("nn"). the nouns on both datasets received low f1 scores. one explanation is that nouns have a large vocabulary and often have multiple senses. as will be discussed in the error analysis below, the model may not make correct predictions if a noun and its different senses are not included in the training data. interestingly, prepositions and conjunctions in the vua dataset got good f1 scores while those in the toefl dataset got low scores. since toefl data was collected from written texts of non-native english learners, we conjecture that there is more variety in the usage of prepositions in toefl dataset. the corresponding noise may have made it harder for the model to generalize from the training set to the test set. error analysis. we look through the examples where our model made wrong predictions, and summarize the patterns of these error examples below. some metaphorical words in the test set do not occur in the training set. examples are "whip-aerials", "puritans", "half-imagined" and "pandora box". our model incorrectly classifies these words as non-metaphorical. ## conclusion in this paper, we introduced the illinimet system for the task of word-level metaphor detection. our model leveraged contextual and linguistic information by combining contextualized representation and external linguistic features. we adopted an ensemble approach to reduce the variance of predictions and improve the model performance. the empirical results showed the effectiveness of our model. we also performed an error analysis to gain insights into the model behavior.