Metaphor Detection Using Contextual Word Embeddings From Transformers (2020)

Abstract: The detection of metaphors can provide valuable information about a given text and is crucial to sentiment analysis and machine translation. In this paper, we outline the techniques for word-level metaphor detection used in our submission to the Second Shared Task on Metaphor Detection. We propose using both BERT and XLNet language models to create contextualized embeddings and a bi-directional LSTM to identify whether a given word is a metaphor. Our best model achieved F1-scores of 68.0% on VUA AllPOS, 73.0% on VUA Verbs, 66.9% on TOEFL AllPOS, and 69.7% on TOEFL Verbs, placing 7th, 6th, 5th, and 5th respectively. In addition, we outline another potential approach with a KNN-LSTM ensemble model that we did not have enough time to implement given the deadline for the competition. We show that a KNN classifier provides a similar F1-score on a validation set as the LSTM and yields different information on metaphors.

## introduction a metaphor is a form of figurative language that creates a link between two different concepts and conveys rich linguistic information @xcite . the complex information that accompanies a metaphorical text is often overlooked in sentiment analysis, machine translation, and information extraction. therefore, the detection of metaphors is an important task in order to achieve the full potential of many applications in natural language processing @xcite . the differences between a metaphorical text and a non-metaphorical text can be subtle and require specific domain information. for instance, in the phrase the trajectory of your legal career the word trajectory is used metaphorically. to identify this metaphor, both the meaning of the word in the context of the sentence and its literal definition must be recognized and compared. in this case, the word trajectory is used to describe the path of a legal career in the sentence, whereas its basic definition involves the path of a projectile. as a result of the ambiguity present in determining the basic meaning of a word, as well as whether it deviates significantly from a contextual use, detecting metaphors at a word-level can be challenging even for humans. additionally, the metaphor identification procedure used to label the datasets (mipvu) accounts for multiple kinds of metaphors @xcite . capturing implicit, complex metaphors may require different information than capturing direct, simple metaphors. this paper describes the techniques that we utilized in the second shared task on metaphor detection. the competition provided two datasets: a subset of ets corpus of non-native written english, which contains essays written by test-takers for the toefl test and was annotated for argumentation relevant metaphors, and the vu amsterdam metaphor corpus (vua) dataset, which consists of text fragments sampled across four genres from the british national corpus (bnc) -academic, news, conversation, and fiction. for each dataset, participants could compete in two tracks: identifying metaphors of all parts of speech (allpos) or verbs only (verbs). our final submission uses pretrained bert @xcite and xlnet @xcite transformer models, part-of-speech (pos) labels, and a two-layer bi-directional long short-term memory (bi-lstm) neural network architecture. bert and xlnet are used to generate contextualized word embeddings, which are then combined with pos tags and fed through the bi-lstm to predict metaphoricity for each word. by creating contex-tualized word embeddings using transformers, we hoped to capture more long-range interdependencies between words than would be possible using methods such as word2vec, glove, or fasttext. indeed, our model achieved f1-scores of 68.0% on vua allpos and 73.0% on vua verbs, improving upon results from the first shared task @xcite . on the toefl task, we achieved f1scores of 66.9% on allpos, and 69.7% on verbs. our scores placed 7th, 6th, 5th, and 5th respectively in the second shared task on metaphor detection @xcite . ## related works historically, approaches to automatic metaphor detection have focused on hand-crafting a set of informative features for every word and applying a supervised machine learning algorithm to classify words as metaphorical or non-metaphorical. previous works have explored features including pos tags, concreteness, imageability, semantic distributions, and semantic classes as characterized through sumo ontology, wordnet, and verbnet @xcite @xcite . deep learning methods have also been employed for automatic metaphor detection. in the first shared task on metaphor detection, the top three highest scoring teams all employed an lstm model with word embeddings and additional features @xcite . @xcite trained fasttext word embeddings on various native and non-native english corpora, and passed the sequences of embeddings to an bi-lstm. the highest-performing model from @xcite employed a bi-lstm on glove embeddings and concreteness ratings for each word. @xcite appended pos and semantic class information to pretrained word2vec word embeddings, and utilized a cnn in addition to a bi-lstm in order to better capture local and global contextual information. in all these cases, the word embeddings used are contextindependent: the same word appearing in two different sentences will nonetheless have the same embedding. thus, these embeddings may not be able to fully capture information about multi-sense words (for example, the word bank in river bank and bank robber), which is crucial for properly identifying metaphors. more recently, @xcite proposed two rnn models for word-level metaphor detection based on linguistic theories of metaphor identification. glove and elmo embeddings are used as input features that capture literal meanings of words, which are compared with the hidden states of bi-lstms that capture contextual meaning. we chose to explore transformer-based embeddings as an alternative way to capture contextual information. transformer-based models have shown state-ofthe-art results on a wide variety of language tasks, including sentence classification, question answering, and named entity recognition. these models rely on self-attention mechanisms to capture global dependencies, and can be used to generate contextualized word embeddings. we chose to examine the models bert, gpt2, and xlnet. these three models all achieve remarkable performances on various nlp tasks, but they capture long-distance relationships within the text in different ways. bert is an autoencoder model, consisting of a stack of encoder layers, and is able to capture bi-directional context using masking during training @xcite . gpt2 is an autoregressive model, consisting of a stack of decoder layers, and thus is only able to capture unidirectional context @xcite . xlnet is also autoregressive, but it captures bi-directional context by considering all permutations of the given words @xcite . each of these models has its advantages and disadvantages that are worth exploring in the context of metaphor detection. ## methodology our method for metaphor detection begins with generating contextualized word embeddings for each word in a sentence using the hidden states of pretrained bert and xlnet language models. next, those embeddings are concatenated together, pos tags for each word are appended to the embeddings, and a bi-lstm reads the features as input and classifies each word in the sentence. word embeddings due to limited metaphorannotated data, rather than training a transformer model on our downstream task, we instead opted to take a feature-based approach to generating contextualized word embeddings from pretrained transformer models. this idea was inspired by the approach to the token-level named entity recognition task described in @xcite , which used a number of strategies for combining hidden state representations of words from a pretrained bert model to generate contextualized word embeddings. we installed the python transformers library developed by huggingface @xcite , which includes a pytorch @xcite implementation of bert and several pretrained bert models. we opted to use the bert base uncased model, which consists of 12-layers, 768-hidden, 12-heads, and 110m parameters. for each line in the vua and toefl datasets, we use the bert tokenizer included in the transformers package to pre-process the text, then generate hidden-state representations for each word by inputting each line into the pretrained bert model. each token is given a 12x768 hidden-state representation from bert. we generate 768-dimension word embeddings by summing the values from each of the 12 hidden layers for each token. words out-of-vocab for bert are split into multiple tokens representing subwords. to generate embeddings for these words, embeddings are generated for each subword token, then averaged together. similarly, we installed the huggingface implementation of xlnet and used its pretrained xlnet base uncased model to generate embeddings for each word in the dataset using the same method as with bert. once both embeddings are generated, we con-catenate the bert and xlnet embeddings for each word to generate 1536-dimensional word embeddings. by combining word embeddings from multiple high-performing pretrained transformers, we are able to capture more contextual information for each word. additionally, we supplement these word embeddings with the pos tag for each word as generated by the stanford parser @xcite . pos tags were shown to improve metaphor detection in the 2018 metaphor detection shared task (leong et al., 2018), and we find a small improvement by including them here. ## conclusion in this paper, we describe the best performing model that we submitted for the second shared task on metaphor detection. we used bert and xlnet language models to create contextualized embeddings, and fed these embeddings into a bidirectional lstm with a sigmoid layer that used both local and global contextual information to output a probability. our experimental results verify that contextualized embeddings outperform previous state-of-the-art word embeddings for metaphor detection. we also propose an ensemble model combining a bi-directional lstm and a knn, and show promising results that suggest the two models encode complementary information on metaphors.