Character aware models with similarity learning for metaphor detection (2020)

Abstract: Recent work on automatic sequential metaphor detection has involved recurrent neural networks initialized with different pre-trained word embeddings and which are sometimes combined with hand engineered features. To capture lexical and orthographic information automatically, in this paper we propose to add character based word representation. Also, to contrast the difference between literal and contextual meaning, we utilize a similarity network. We explore these components via two different architectures - a BiLSTM model and a Transformer Encoder model similar to BERT to perform metaphor identification. We participate in the Second Shared Task on Metaphor Detection on both the VUA and TOFEL datasets with the above models. The experimental results demonstrate the effectiveness of our method as it outperforms all the systems which participated in the previous shared task.

## introduction metaphors are an inherent component of natural language and enrich our day-to-day communication both in verbal and written forms. a metaphoric expression involves the use of one domain or concept to explain or represent another concept @xcite . detecting metaphors is a crucial step in interpreting semantic information and thus building better representations for natural language understanding @xcite . this is beneficial for applications which require to infer the literal/metaphorical usage of words such as information extraction, conversational systems and sentiment analysis @xcite . the detection of metaphorical usage is not a trivial task. for example, in phrases such as breaking the habit and absorption of knowledge, the words breaking and absorption are used metaphorically to mean to destroy/end and understand/learn respectively. in the phrase, all the world's a stage, the world (abstract) has been portrayed in a more concrete (stage) sense. thus, computational approaches to metaphor identification need to exploit world knowledge, context and domain understanding @xcite . a number of approaches to metaphor detection have been proposed in the last decade. many of them use explicit hand-engineered lexical and syntactic information @xcite , higher level features such as concreteness scores @xcite and wordnet supersenses @xcite . the more recent methods have modeled metaphor detection as a sequence labeling task, and hence have used bilstm @xcite in different ways @xcite @xcite . in this paper, we use concatenation of glove @xcite and elmo @xcite vectors augmented with character level features using cnn and highway network @xcite . such a method of combining pre-trained embeddings with character level representations has been previously used in several sequence tagging tasks -part-of-speech (pos) tagging @xcite and named entity recognition (ner) @xcite , question answering @xcite and multitask learning @xcite . this inspires us to explore similar setting for metaphor identification as well. we propose two models for metaphor detection foot_0 with the input prepared as above -a vanilla bil-stm model and a vanilla transformer encoder @xcite model similar to @xcite (but without pre-training). to contrast the difference between a word's literal and contextual representation @xcite con-catenated the two before feeding into the softmax classifier. instead, we extend the idea of cosine similarity between two words in a phrase of signifying metaphoricity @xcite to similarity between the literal and contextual representations of a word and then feed this result into the classifier. finally, we participate in the second shared task on metaphor detection foot_1 on both the vu amsterdam metaphor corpus (vua) @xcite and toefl, a subset of ets corpus of non-native written english @xcite datasets with the above models and a vanilla combination of them. the combination of the models outperforms the winner @xcite of the previous shared task @xcite . ## related work previous metaphor detection frameworks include supervised machine learning approaches utilizing explicit hand-engineered features, approaches based on unsupervised learning and representation learning, and deep learning models to detect metaphors in an end-to-end manner. (köper and schulte im walde, 2017) determine the difference of concreteness scores between the target word and its context and use this to predict the metaphoricity of verbs in the vua dataset. @xcite combine vector space representations with features such as abstractness and imageability and word-net supersenses to model the metaphor detection problem in two syntactic constructions -subjectverb-object (svo) and adjective-noun (an). evaluating their approach on the trofi dataset @xcite , they achieve competitive accuracy. @xcite explore differences in compositional behaviour of a word's literal and metaphorical use in certain syntactic settings. using lexical, wordnet supersense features and pos tags of sentence tree, they train an svm using tree-kernel. @xcite use semantic classes of verbs such as orthographic unigram, lemma unigram, distributional clusters etc. to identify metaphors in the vua dataset. some of the methods for metaphor detection utilize unsupervised learning. @xcite train word embeddings on wikipedia dump and use wordnet compute a best-fit word corresponding to a target word in a sentence. the cosine similarity between these two words indicates the metaphoricity of the target word. @xcite compute word embeddings and phrase embeddings on wikipedia dump. they extract visual features from cnns using images from google images. next, multimodal fusion strategies are explored to determine metaphoricity. recently, approaches based on deep learning have been proposed. the first in this line is supervised similarity network by @xcite . they capture metaphoric composition by modeling the interaction between source and target domain by a gating function and then using a cosine similarity network to compute metaphoricity. they evaluate their method on adjective-noun, verb-subject and verb-direct object constructions on the moh @xcite and tsv @xcite datasets. more recently, the problem has been modeled as a sequence labeling task, in which at each timestep the word is predicted as literal or metaphoric. @xcite used word2vec @xcite , pos tags and word clusters as input features to a cnn and bilstm network. they compared inference using softmax and crf layers, and found softmax to work better. @xcite propose two models -a bilstm with dense layers before and after it and a recursive model for bigram phrase composition using fully-connected neural network. they also added concreteness scores to boost performance. @xcite fed glove and elmo embeddings into a vanilla bilstm followed by softmax. @xcite proposed models based on mip @xcite and svp @xcite ) linguistic theories and achieved competitive performance on vua, moh and trofi datasets. ## methodology in this paper we propose two architectures for metaphor detection based on sequence labeling paradigm -a bilstm model and a transformer encoder model. both the models are initialized with rich word representations. first, we describe the word representations, then, the similarity network, and subsequently the models (figure 1 ). ## results we first compare our method against the baseline systems which have the same experimental setting as ours on the vua test set -cnn-bilstm and bilstm-concat. we now compare our method with the other two baselines on a common experimental setting. we tune our hyperparameters in this setting due to difference in training and validation data. specifically, since training set is of smaller size, we increase dropout probabilities, and the dimension of pos embedding is reduced from 30 to 10. as shown in the scores on toefl dataset are lower than the vua dataset. this is due to the lesser number of training instances in toefl dataset. also, while we have higher recall on vua, on toefl we have higher precision. ## ablation study this section considers the performance of different components of our method in isolation and combination on the vua validation set unless otherwise specified. the reason for choosing validation set is because we were not able to evaluate some settings on the test set due to limited time and number of submissions. wherever we have test set results we report those as well. impact of character embeddings we first note the performances of vanilla bilstm and vanilla transformer models and a simple combination of them in we suspect that employing the baseline unigram features @xcite provided by the organizers instead of learned characterembeddings may be seen as a way to achieve the same goal. but our method is more robust in the sense that, we allow for learning of different ngram features of a word (including unigram itself). particularly, our method is helpful in cases where the target word has incorrect spelling, because we learn representations instead of using fixed precomputed features. impact of similarity network by a small margin. the above reasoning indicates towards similarity network as being an important component for detection of mip guided labeling of metaphors. in all the cases examined till now, transformer based models have higher precision than the bil-stm based models, and in 3 out of 4 cases of (vanilla, ce, sn, ce + sn), the combination has as even better precision than either of the individual models. in terms of f-score, bilstm based models score higher than transformer based ones in 2 cases (vanilla and ce), equal in ce + sn and lower in sn. impact of pos tags incorporation of pos tags proves to be beneficial. it improves the f-score of the last model in ## conclusion we proposed two metaphor detection models, a bilstm model based on prior work and a transformer model based on their success in nlp tasks. we augment these models with two components -character embeddings and similarity network to learn lexical features and contrast literal and contextual meanings respectively. our experimental results demonstrate the effectiveness of our method as we achieve superior performance than all the previous methods on vua corpus and toefl corpus. through an ablation study we examine the contribution of different parts of our framework in the task of metaphor detection. in our future work we would explore metaphor detection in a multi-task setting with semantically similar tasks such as word sense disambiguation and co-reference resolution. these auxiliary tasks may help to better understand the contextual meaning and reach of a word. for toefl dataset, future avenues would include strategies to exploit the metadata, and similarity measures more suitable for argumentation-relevant metaphors.