Modality Enriched Neural Network for Metaphor Detection (2020)

Abstract: Metaphor as a cognitive mechanism in human’s conceptual system manifests itself an effective way for language communication. Although being intuitively sensible for human, metaphor detection is still a challenging task due to the subtle ontological differences between metaphorical and non-metaphorical expressions. This work proposes a modality enriched deep learning model for tackling this unsolved issue. It provides a new perspective for understanding metaphor as a modality shift, as in ‘sweet voice’. It also attempts to enhance metaphor detection by combining deep learning with effective linguistic insight. Extending the work at Wan et al. (2020), we concatenate word sensorimotor scores (Lynott et al., 2019) with word vectors as the input of attention-based Bi-LSTM using a benchmark dataset–the VUA corpus. The experimental results show great F1 improvement (above 0.5%) of the proposed model over other methods in record, demonstrating the usefulness of leveraging modality norms for metaphor detection.

## introduction metaphor is a common linguistic mechanism that encodes versatile ontological information accompanied by phenomena of e.g. domain transfer @xcite , concreteness change @xcite or semantic surprise @xcite etc. in general, metaphor usually involves certain concept transfer from one domain (source) to another (target), as demonstrated in the example 'sweet smile' (using the taste modality to describe the vision modality). it is a fundamental way for people to relate our physical and familiar social experiences to a multitude of other subjects and contexts that are more complex, implicit or less known @xcite . detecting metaphors in texts is essential for capturing the authentic meaning of a language unit (e.g. word, phrases or even sentences) when the language is beyond literal interpretations, which can benefit a lot of language processing applications, such as machine translation, dialogue systems and sentiment analysis @xcite . although detecting metaphors is subconsciously effortless for human, it is actually a great challenge to existing technologies due to the subtle semantic and ontological differences between metaphorical and non-metaphorical ones @xcite . this work proposes a modality-enriched deep learning model for metaphor detection based on the idea that the basic senses (touch, hearing, smell, taste, vision and interception) and action affectors (mouth/throat, hand/arm, foot/leg, head) indicated by the sensorimotor norms @xcite of words provide crucial information for metaphoricity inference. as a next step of @xcite which only adopts simple statistical models, we build a more complex model combining neural networks with modality information. for standard reference, we adopt the benchmark dataset of the first and second shared tasks of metaphor detection-the vua corpus @xcite 1 for experiments. the primary methods in metaphor detection, including the earliest attempts, mainly adopt supervised machine learning paradigm with training on pre-labelled datasets @xcite based on various features mentioned above. this can be attested by the recent system papers for the 1 st and 2 nd shared tasks of metaphor detection @xcite . although 90% of the systems adopt deep learning architectures @xcite @xcite , there are substantial linguist resources employed for system refinement, such as data with information of concreteness and imageability, semantic encoding of wordnet, framenet, verbnet, sumo ontology, property norms, syntactic dependency patterns, sensorial and vision-based information, etc. such external resources provide different linguistic cues for metaphoricity inference, demonstrating their effectiveness for metaphor detection to various degrees @xcite @xcite . to name a few advances, @xcite build up an ensemble model of rnns together with attention-based bi-lstms. @xcite adopt bert to obtain sentence embeddings, and then apply a linear layer with softmax to each token for metaphoricity predictions. @xcite use roberta to obtain word embeddings and concatenate it with linguistic features (e.g. wordnet, verbnet, pos, topicality, concreteness), and then feed them into a fully-connected feedforward network to make predictions. @xcite combines the concreteness of a word with its static and contextual embeddings as inputs into a deep multi-layer perceptron network for predicting metaphoricity. despite the above advances, metaphor detection remains a difficult task because the semantic and ontological differences between metaphorical and non-metaphorical expressions are often subtle and their perception may vary from person to person. existing methods show different strengths on detecting metaphors, yet each has its respective disadvantages, such as having generalization problems or lack association of their results with the intrinsic properties of metaphors. besides, the performance of metaphor detection in record is still unsatisfactory (0.6-0.8 f1). in this work, we propose an innovative deep learning model for metaphor detection combining the strengths of neural networks with sensorimotor norms. our work provides a new perspective for understanding metaphors as a linguistic mechanism accompanied by concept shifts indicated by the scores of sensorimotor norms. it also aims to testify the effectiveness of such scores encoded in words for the token-level metaphor detection task. ultimately, we hope to improve the task of metaphor detection in a consistent way. ## the modality enriched model in the model, words are processed with the integration of modality scores and word embeddings, as demonstrated in figure 2 below. we map the words to the sensorimotor norms and obtain the modality representations (64 dimension for each word). at the same time we get the word vectors (300 dimension for each word) using glove @xcite and then concatenate them as inputs to neural networks. for those words not mapped in the sensorimotor norms, we assigned three kinds of values, including all zeros, random values following normal distribution, as well as average scores of the sensory words in the corpus. in the end, we chose to use the average score for each dimension of the out-ofdictionary words due to optimization reasons. figure 2 : the modality enriched model bi-lstm layer produces a hidden status of each word in a sentence. we use these status to calculate an attention weight which will be multiplied with the output of bi-lstm layer. let @xmath0, h 2 .... h n ] that is produced by lstm, where d is the size of hidden layers and n is the length of the given sentence. the attention mechanism will generate an attention weight α. the final sentence representation is given by: we also add an additional linear layer. the final probability distribution is: let y be the target distribution for sentence, ŷ be the predicted metaphoricity distribution. train to minimize the cross-entropy error between y and ŷ for all sentences. finally, we get a probability distribution of 0-1 label to train the model and as the prediction result. ## conclusion and future work we presented a linguistically enhanced method for metaphor detection using modality features plus word embeddings on the basis of attention-based neural network, as a further evidence of wan et al. (2020)'s first proposal of utilizing conceptual norms for metaphor detection. the proposed model with access to modality scores achieves outperforming results that 1) are significantly higher than one without such information, and 2) show leading performances than most related works in record. the results of the study are in line with previous work, and strengthen the conclusion that conceptual modality is crucial to metaphors (relevant to both our understanding of metaphors and nlp). in future, we will adopt more advanced deep learning architecture and introduce modality using other paradigms rather than concatenating embedding with the hope to further enrich the content.