Metaphor Detection using Ensembles of Bidirectional Recurrent Neural Networks (2020)

Abstract: In this paper we present our results from the Second Shared Task on Metaphor Detection, hosted by the Second Workshop on Figurative Language Processing. We use an ensemble of RNN models with bidirectional LSTMs and bidirectional attention mechanisms. Some of the models were trained on all parts of speech. Each of the other models was trained on one of four categories for parts of speech: “nouns”, “verbs”, “adverbs/adjectives”, or “other”. The models were combined into voting pools and the voting pools were combined using the logical “OR” operator.

## introduction figurative language is common in everyday speech and generally easy for humans who are speaking the same language to interpret, yet machines have trouble with it, limiting our interaction with them. if a machine has trouble understanding our natural language, then it will have trouble interpreting our intentions and translating them correctly to another language or human. therefore, the goal of our research is to improve metaphor detection to facilitate the interpretation and translation of natural language in discourse. the second shared task on metaphor detection used the vrije university amsterdam metaphor corpus (vuamc) @xcite , which has been the most widely used database for training machines to detect metaphors. the metaphor labels in the vuamc are per word and indicate whether the word is a metaphor related word (mrw) or not. an mrw may be an indirect, direct, or implicit metaphor. the vuamc contains text from four sources: academic texts, newspapers, conversations, and fiction. each word was labeled using the mipvu procedure, with greater than 0.8 interannotator reliability @xcite . about 13% of the words in the vuamc are labeled as metaphor related words. the second shared task on metaphor detection with the vuamc demonstrated state of the art performance with the best performer achieving an f1 score of 0.769. on the same training and test sets, we were able to achieve an f1 score of 0.703, and when we randomly split the vuamc data around sentences vs. fragments (which may contain more than one related sentence), we were able to achieve an f1 score of 0.730. @xcite provide a summary of the results from all participants in the shared task. listed below are our major findings and contributions: next we present related work followed by a discussion of our approach. then we present a summary of our results followed by conclusions. we conclude with a brief discussion about our future research plan. stemle and onysko (2018) also split sentences into segments depending on a maximum sequence length. shorter sequences were padded. they used a layered model with input to a bidirectional lstm which provided input to a fully connected output layer that was activated by the softmax function. the length of the output equaled the length of the input. the output predicted whether each word from the input layer was a metaphor related word or not. they used a categorical cross-entropy loss function to address the imbalance of non-metaphor related words to metaphor related words. they experimented with word embeddings from various models that were pre-trained on corpora that varied in their language proficiency levels. on the vaumc shared task data sets for all parts of speech, they achieved 69.8% recall with 55.3% precision. leong et al. ( 2018 ) used logistic regression and random forest classifiers with lemmatized unigrams, generalized wordnet semantics, and difference in concreteness ratings between verbs/adjectives and nouns @xcite . during training, each class was weighted by the inverse of its frequency. for the optimization function, they used the f1-score. they achieved 69.6% recall with 51% precision on the vuamc shared task data sets for all parts of speech. @xcite trained an lstm on 300d glove embeddings. they also experimented with using part-of-speech information and features from the general inquirer, which worsened their results on the test data. @xcite presented an architecture that used a context encoder inspired by a bidirectional lstm. the output of the encoder was fed to a feature selection module to select features for the token word. they showed that re-weighting examples and using parts of speech, wordnet, and concreteness ratings improved the performance of their model. @xcite presented a crf sequence model that was trained using glove word embeddings and contextual information. @xcite used a hybrid model of bi-lstm and crf trained with word2vec embeddings for the token word and its lemma, 20d vectors representing the pos, and onehot vectors for whether the lemma and the token were the same, and whether the lemma was present in the token. other researchers have made progress in metaphor detection at the word level, but the results were reported for data sets other than the vuamc. @xcite used svms with tree kernels on syntactic features and achieved an f1-score of 75%. they built a corpus of 3872 labeled metaphors which they also released. @xcite presented results from using the theory of meaning to identify metaphors in a subset of the bnc, which they labeled themselves. they achieved an f1-score of 87%. (the vuamc is also a subset of the bnc.) @xcite used wordnet bigram counts to identify metaphors in the master metaphor list created by @xcite . they reported 58% accuracy, 70% precision, and 61% recall. ## method we designed and experimented with various rnn architectures using bidirectional lstms and attention mechanisms @xcite . the input was an 11-gram for each word in the training set (or test set during testing). each word was represented by an 11-gram and appeared at the center of the 11-gram. furthermore, each word in the 11-gram was represented by a 1,324 dimensional word embedding which was the result of concatenating a 1,024 dimensional elmo @xcite embedding with a 300 dimensional glove @xcite embedding, because preliminary testing revealed that elmo concatenated with glove resulted in better performance than either one of them alone. the 11-grams were from one sentence; i.e., they never extend across two sentences. padding was used if the center word was not in the context of exactly 5 words to the left or right, so the first word in a sentence would always have 5 pads to its left and the last word in a sentence would always have 5 pads to its right. five was chosen for the window size because it produced the best results in preliminary experiments. the output was a two-dimensional vector representing the probabilities for the center word being a metaphor or not. softmax was used to choose the highest probability. two architectures were used for our final results on the shared task. they are described below. the first architecture is a many-to-one bidirectional lstm with bidirectional attention (see figure 1 ). in this architecture, the outputs of the forward and backward lstm cells in the attention layer are concatenated only at the output for the center, target word, of the 11-gram. the expected output is a 1 or 0 for the center word in the 11-gram, depending on whether it is a metaphor related word figure 2: many-to-one fully-connected bidirectional lstm with bidirectional attention or not. each attention cell receives output from only one bidirectional lstm cell (vs. a fully connected architecture where the output of every bidirectional lstm cell is forwarded to every attention cell). see the difference between figures 2 and 1 for reference. the intuition behind the choice to forward only one cell's output per attention cell is that the attention cells are intended to process input in a sequential order (i.e., one word at a time). providing each step with the entire matrix of weights for all words from the bidirectional lstm seems to violate the design of the attention mechanisms. we tried both architectures and got better results with the architecture in which each attention cell receives output from only one bidirectional lstm cell. the second model is a many-to-many bidirectional lstm with bidirectional attention (see figure 3 ). the expected output is a 1 or 0 for each word in the 11-gram, depending on whether the word is a metaphor related word or not. however, in the trained model, only the output for the center word, w, is used to assign a prediction to w. the key difference between this model and the first model is that the many-to-many model updates its weights based on the performance of the model on the target word's context words, in addition to the performance of the model on target word. the performance feedback includes whether or not the target word is in the context (within a window of 5 words on either side) of another metaphor related word. this is also why we must split across sentences and not allow a sentence from the training set to also appear in the test set. the many-tomany model was chosen because its performance complements the first model (i.e., the many-to-one bidirectional lstm with bidirectional attention). voting among trained instances of the many-tomany model gives better precision, while voting among trained instances of the many-to-one model gives better recall. both have comparable f1 scores. another important note about the many-to-many model is that the output of the backward attention layer starting at the last word in the 11-gram, w 10 , is concatenated with the output of the forward attention layer for w 1 0; the output of the backward attention layer for w 9 is concatenated with the output of the forward attention layer for w 9 ; and so on until the output of the backward attention layer for w 0 is concatenated with the output of the forward attention layer for w 0 . we used a training batch size of 200. an adam optimizer was used with a learning rate 0.006 and a decay rate of 0.001. the loss function was categorical cross entropy. a dropout rate of 0.2 was used for the bi-lstm layer and a dropout of 0.1 was used for both the forward and backward attention. the hidden states for both the bi-lstm and attention layers were vectors of length 128. the output layer of each model used the softmax activation function. keras with a tensorflow backend was used for the implementation. we trained and tested our models independently on two data splits. for the first split, 25% of the data samples (11-grams from the vuamc) were randomly selected and held out for testing. among the remaining 75%, one-third of the samples were randomly selected and preserved, along with all of the positive samples (labeled 1) in the remaining two-third. the rest of the training samples were discarded to achieve a more balanced training set. (however, testing was always performed on the entire test set.) for the second split, we used the training and test sets from the second shared task on metaphor detection. for the shared task, the training and test sets were sampled by fragments, in which a fragment (e.g., a paragraph) may contain more than one sentence. we initially used one-third of the training samples, the same way we did with the first split, but then we tried using the entire training set and passed class weights to the adam optimizer to mitigate the imbalanced number of samples per class. the class weights were proportional to the percentage of samples in each class. we got better results using all of the samples in the training set. (we did not go back to the first split to train with all of the training samples and class weights, but we hypothesize that we'd get better results if we did.) first, we trained and tested our model using 300 dimensional glove vectors @xcite . next, we tried 1024 dimensional elmo vectors @xcite . finally, we used 1324 dimensional vectors from combining glove with elmo. in the last case, for each word, we simply concatenated the elmo vector representation for that word with the glove vector representation for that word, resulting in a vector of length 1324. we trained each architecture independently multiple times on all parts of speech and then on each of four categories for parts of speech: "nouns", "verbs", "adverbs/adjectives", or "other". we used the nltk toolkit @xcite to derive the parts of speech. ## results we achieved the best results with an ensemble of trained models. the ensemble consisted of five models per architecture trained on all parts of speech, and five models per architecture trained independently on each of four parts-of-speech categories: "nouns", "verbs", "adverbs/adjectives", or "other". the five models per architecture per partof-speech category (including "all parts of speech") were assembled into a voting pool, so there were ten models total per category. each set of ten models were combined into a voting pool and the voting pools were combined using the logical "or" operator. we also evaluated our many-to-one model with respect to the novelty scores provided by @xcite for the vuamc. on novel metaphors (i.e. metaphors with a novelty score of at least 0.5 from @xcite ) the manyto-one architecture found 52/77, or 67.5% in the shared task test set. ## conclusions and next steps we have described two model architectures and an ensemble approach for metaphor detection. we have shown that splitting the training and data sets by randomly sampling sentences rather than fragments (or paragraphs), which may contain more than one sentence, may lead to better results. we believe this may be because there are patterns of language in the test fragments that were not seen in the training fragments. allowing a model to train on some sentences from a fragment and then test on the other sentences in the same fragment may produce better results overall. we shared that when forwarding information from a bidirectional lstm to an attention layer, better performance can be obtained when each attention cell receives output from only one bidirectional lstm cell. finally, we have revealed that it is possible to get better performance from logically combining the outputs of an ensemble. in future work, we will continue improving metaphor detection with a focus on novel metaphors. according to @xcite , " @xcite conducted a corpus study of the use of metaphor in educational discourse for all parts of speech. she found that verbs account for around 50% of the data, the rest shared by nouns, adjectives, adverbs, copula constructions and multiword metaphors." about 43% of metaphors in the vuamc are verbs, while verbs are only 23% of all tokens in the vuamc database. however, do @xcite found that only about 24% of the novel metaphors are verbs and 41% are nouns. the rest are adjectives and adverbs. (other pos were not included.) therefore, future work in detecting novel metaphors may place a heavier weight on nouns (vs. verbs as has been the case with conventional metaphors).