Augmenting Neural Metaphor Detection with Concreteness (2020)

Abstract: The idea that a shift in concreteness within a sentence indicates the presence of a metaphor has been around for a while. However, recent methods of detecting metaphor that have relied on deep neural models have ignored concreteness and related psycholinguistic information. We hypothesis that this information is not available to these models and that their addition will boost the performance of these models in detecting metaphor. We test this hypothesis on the Metaphor Detection Shared Task 2020 and find that the addition of concreteness information does in fact boost deep neural models. We also run tests on data from a previous shared task and show similar results.

## introduction the automatic detection and processing of metaphor is an ongoing challenge for true deep semantic understanding of natural language text. metaphors often convey unrelated concepts to their literal meaning and the meaning of metaphor involves more than just its words meaning, but it incorporates the whole context with a wider knowledge of their conceptual domain. traditional methods of metaphor detection that do not make use of neural networks have used concreteness scores to improve metaphor detection @xcite . however, neural models that use distributional semantics (i.e. word embeddings: @xcite ) have shown promising and often state-ofthe-art results in a range of nlp tasks and have recently produced promising results in metaphor detection @xcite @xcite . these models, however, focus on the textual information provided by the word embeddings and do not further explore the use and effect of combining other lexical information. this paper reports the result of combining neural networks with a lexical resource for measuring concreteness for word-level metaphor detection. despite the success of deep neural models, we hypothesise that they do not have access to concreteness information with their structure. to test this, we explicitly add concreteness information to deep neural models and compare their performance with and without this information. our experiments show that deep neural models, like more traditional models, do benefit from concreteness information. ## related work early work, by @xcite on the use of concreteness to detect metaphor made use of the relatively small mrc psycholinguistic dataset @xcite for concreteness scores. their work uses a logistic regression model to detect the metaphoricity of adj-noun pairs in the trofi dataset @xcite . subsequently, @xcite made use of the same mrc dataset to detect subject-verb-obj metaphors from troifi dataset. they also train a supervised logistic regression classifier on english triples and test on a russian dataset. köper and schulte im walde (2017b) extend this work by using a significantly larger dataset @xcite of concreteness ratings and propagating the concreteness rating to phrases using word2vec @xcite . their experiments use @xcite 's logistic regression classifier on vuamc for verbs using ten-fold cross-validation process. the context that a word occurs in plays an important role in metaphor detection @xcite . words and phrases often convey very different meanings in different contexts. consider the phrase "cut down" in the sentence "she cut down his advances with her words." in ab-sence of the context, it is not clear that "cut down" is metaphorical. many supervised learning approaches, including those described above, utilise bag of words methods, thus focusing on sets of features which do not capture context. those that do consider context, do so only to a small extent, for example by focusing only on specific sentence constructs like adj-noun pairs @xcite or subj-verb-obj @xcite . given the importance of context and the power of neural models in capturing context, it was only natural to use deep neural models for metaphor detection. @xcite make use of deep neural networks to detect metaphor with significant success across multiple datasets including vuamc. in particular they use bidirectional long short term memory networks (bi-lstm) that capture relations in both directions for word-level metaphor classification with word-embeddings as input. other work on using concreteness and similar psycholinguistic features for metaphor detection include that by @xcite who combined concreteness with property norms to formulate representations. @xcite combine imageability scores with concreteness for crosslingual metaphor detection and dunn (2015) make use of abstractness. this paper reports the result of applying concreteness score to individual words in the tokenlevel metaphor classification for the metaphor detection shared task competition 2020. we build on @xcite 's sequence labelling network by adding concreteness scores to individual words. the arrival of deep neural networks has meant that psycholinguistic features are no longer explicitly considered and, as mentioned in section 1, we hypothesise that deep neural networks do not have access to this information. in this work, we show that this is the case and that access to this information improves the accuracy of deep neural networks by testing on multiple datasets. ## generalising concreteness scores we used the resource created by @xcite for concreteness scores. this is a list of about 40,000 english words rated for concreteness between 1 to 5 where 1 is most abstract and 5 is most concrete. as an illustration, "wood" has a rating of 4.85, "counterargument" a rating of 2.17 and "conventionalism" 1.18. before we can use concreteness scores for metaphor detection, we need a way of handling those words in our dataset that do not have corresponding concreteness scores in the concreteness lexical resource created by @xcite . the most obvious solution is to set the concreteness scores of these words to 0. however, the fact that a large number of words in our dataset do not have corresponding concreteness scores makes this impractical. to get around this, we use the concreteness values available to train a support vector machine. we use bert @xcite embeddings as features to the svm and the rounded up concreteness values as output classes. so as to use bert embeddings as input to an svm, we extract static, non-contextual bert embeddings. we choose to use bert, as opposed to static embedding like word2vec, due to bert's unique tokenizer that allows for the generation of embeddings for all words in our dictionary. we use the following hyperparameters for the svm: hidden layer sizes 100, activation identity, solver adam, alpha 0.0001, batch size auto, learning rate adaptive, learning rate init 0.001, powert 0.5, max iteration 200, shuffle true, random state none, tol 0.0001, verbose false, warm start false, momentum 0.9, nesterovs momentum true, early stopping false, validation fraction 0.1, beta1 0.9, beta2 0.999, epsilon 1e-08, and niter no change 10. ## neural metaphor detection with concreteness we use @xcite 's sequence labeling model as the baseline and modify it to include a concreteness rating as follows. for every input word x i we modify w i , the 300-d glove pretrained embedding for x i , with the concreteness class assignment c i of x i . this results in a 301-d representation [w i : c i ] for each of the input words. these representations of words are fed to the sequence labeling model, which consists of a bi-lstm which generates a contextual representation of each word. these are then fed to feedforward neural networks which predict a label for each word. figure 1 provides an illustration of the sequence labeling model, wherein the bi-lstm is represented by pink blocks and the blue blocks represent the feedforward neural networks. we also test appending the probabilities of each of the four concreteness classes output by the svm. in this case the 300-d pre-trained representation w i is concatenated with a vector p i of length four, where each digit represents the probability of this word belonging to the output class 1, 2, 3 or 4 respectively. this results in a 304-d representation [w i : p i ] for each word. this method of using the probability distribution is unlike previous methods that have used a single concreteness score. we use the concreteness scorers generated by our svm model even when a word and the corresponding concreteness score is included in the dataset provided @xcite and used as the training data for our svm. we find that the addition of probabilities is far more effective than the addition of a single score possibly because this provides more of a signal for the model to pick up on (4 features not 1). importantly, if pre-trained embeddings (in our case glove) contained concreteness information, the explicit addition of this information by means of appending it to the embeddings should not improve the performance of a well trained bi-lstm model as such models are capable of extracting relevant information from their input. an improvement in performance with the addition of concreteness information would imply that such information is not contained in the pre-trained embeddings we use. ## results the metaphor detection shared task allowed multiple submissions and we use this to evaluate different models, both with and without concreteness scores. we present this comparative analysis of our models first before describing our performance in section 5.2. we also test our models on the previous shared and present these results in section 5.3. ## analysis and discussion the training data from the vuamc dataset has 181,501 tokens, 19,177 of which are labeled metaphor with 162,324 labeled literal. an exploration of the results shows that the most frequently occurring words in the dataset are prepositions. the word of, for example, occurs 4,638 but is labeled as a metaphor only 151 times. with, on the other hand, appears 995 times and is labelled as a metaphor 620 times and up is labelled 137 times as metaphor out of 335 occurrences. in addition, prepositions also appear as part of phrases, such as in "some of the" making it harder still to classify them correctly. often, the meaning of a phrasal verb differs significantly from that meaning of its parts. additionally, concreteness of each of the individual parts is also different from that of the phrase. for example, in the sentence "the use of role play, incorporating previously discussed difficulties (i.e. homework assignment session 4) in real or set up situations provide an opportunity for testing these skills.", the overall meaning of the phrase set up is different from the meaning of set and up. additionally, we were able to successfully classified set as a metaphor, but failed to classify up as a metaphor in this context. each word meaning by itself differs from the meaning of the whole phrase. put means to place something physically, up means the position up and with mean accompanied by someone or something; however, these three together refer "to accept an unpleasant situation, something or someone (willingly or not)." as for their and their near context degrees of abstractness are as follows: the word to is correctly labelled as literal has rating equal to metaphor, put is correctly labelled as literal has rating 2 as its concreteness rating, followed by the word up that is correctly labelled as literal and has a rating of 3. next is the word with that is incorrectly classified as literal has concreteness of 2 and lastly, the word the is correctly classified as literal has concreteness of 1. since there is no drastic shift in concreteness or their senses, the model fails to spot the hits and labels them all as literal. of the 15,439 unique tokens in the dataset, 7527 tokens appear exactly once. for example, "there were others , but lucy never disclosed any of them to us" the word disclosed is labelled as metaphor but incorrectly classified as literal. there are two interpretations for this sentence. the metaphorical sentence talks about uncovering "people's identities" lucy knew when referencing others and them, or could literally talks about uncovering of "secrets" lucy hides, which are referenced by them and others. as for the concreteness ratings for the sentence's words, the rating range between 1 and 2, other than lucy that has rating of 4; therefore, we could say that the rating did not help to ## 0 2 although that is the position in law , the court emphasised that as a matter of sense a tenant should first complain to the landlord before exercising the right to prosecute . 1 1 2 ## conclusion and future work this paper reports the results of providing deep neural models with concreteness information by appending a measure of concreteness to word embedding for all content words. our hypothesis is that explicitly adding a concreteness rating to the word representation will boost the neural network performance in detecting metaphors as neural models do not have access to this information. we tested two representations of concreteness, one as a scale and the other is class probabilities using the vua allpos data from the second metaphor detection shared task 2020 and data from the first metaphor detection shared task 2018 and find that this information does boost performance in all cases. we plan on testing the effectiveness of incorporating other psycholinguistic information, such as imageability, into deep neural models so as to establish their impact on metaphor detection. we also intend to incorporate these features into contextual pre-trained models, such as bert @xcite as context is critical to identifying metaphor. in this current work, bert pre-trained representations were used only in training an svm and not in the bi-lstm that detects metaphor. we also intend to use more complex models to expand concreteness, imageability and other such features to a larger vocabulary. these models will be designed to perform classification better and also capture context so as to better identify the concreteness of words in context. finally, we intend to extend our work to include phrases a significant source of errors in this task.