M ulti MET : A Multimodal Dataset for Metaphor Understanding (2021)

Abstract: Metaphor involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought, which makes understanding it challenging. As a means of cognition, metaphor is rendered by more than texts alone, and multimodal information in which vision/audio content is integrated with the text can play an important role in expressing and understanding metaphor. However, previous metaphor processing and understanding has focused on texts, partly due to the unavailability of large-scale datasets with ground truth labels of multimodal metaphor. In this paper, we introduce MultiMET, a novel multimodal metaphor dataset to facilitate understanding metaphorical information from multimodal text and image. It contains 10,437 text-image pairs from a range of sources with multimodal annotations of the occurrence of metaphors, domain relations, sentiments metaphors convey, and author intents. MultiMET opens the door to automatic metaphor understanding by investigating multimodal cues and their interplay. Moreover, we propose a range of strong baselines and show the importance of combining multimodal cues for metaphor understanding. MultiMET will be released publicly for research.

## introduction metaphor is frequently employed in human language and its ubiquity in everyday communication has been established in empirical studies @xcite @xcite . since @xcite introduced conceptual metaphor theory (cmt), metaphor has been regarded as not only a linguistic, but also a cognitive phenomenon for structuring human thought. individuals use one usually concrete concept in metaphors to render another usually abstract one for reasoning and communication. for example, in the metaphorical utterance "knowledge is treasure," knowledge is viewed in terms of treasure to express that knowledge can be valuable. according to cmt, metaphor involves the mapping process by which a target domain is conceptualized or understood in terms of a source domain. as a means of cognition and communication, metaphor can occur in more modes than text alone. multimodal information in which vision/audio content is integrated with the text can also contribute to metaphoric conceptualization @xcite . a multimodal metaphor is defined as a mapping of domains from different modes such as text and image, text and sound, or image and sound @xcite . for example, in figure 1 (a), the metaphorical message of fire in the sky is conveyed by a mapping between the target domain "sky" (sunset) and the source domain "fire" from two modalities. figure 1 (b) offers another example with the metaphor of lungs made from cigarettes so a relation is triggered between two different entities, lung and cigarette, with the perceptual idea that smoking causes lung cancer. the source domain "cigarette" comes from the image, while the target domain "lung" appears in both text and image. understanding multimodal metaphor requires decoding metaphorical messages and involves many cognitive efforts such as identifying the semantic relationship between two domains @xcite , interpreting authorial intent from multimodal messages @xcite , analyzing the sentiment metaphors convey @xcite , which might be difficult for computers to do. qualitative studies have investigated the interplay between different modes underlying the understanding of multimodal metaphors in communicative environments such as advertisements @xcite , movies @xcite , songs @xcite , and cartoons @xcite . in particular, with the development of mass communication, texts nowadays are often combined with other modalities such as images and videos to achieve a vivid, appealing, persuasive, or aesthetic effect for the audience. this rapidly growing trend toward multimodality requires a shift to extend metaphor studies from monomodality to multimodality, as well as from theory-driven analysis to data-driven empirical testing for in-depth metaphor understanding. despite the potential and importance of multimodal information for metaphor research, there has been little work on the automatic understanding of multimodal metaphors. while a number of approaches to metaphor processing have been proposed with a focus on text in the nlp community @xcite @xcite @xcite @xcite , multimodal metaphors have not received the full attention they deserve, partly due to the severe lack of multimodal metaphor datasets with their challenging and timeand labor-consuming creation. to overcome the above limitations, we propose a novel multimodal metaphor dataset (multimet) consisting of text-image pairs (text and its corresponding image counterparts) manually annotated for metaphor understanding. multimet will expand metaphor understanding from monomodality to multimodality and help to improve the performance of automatic metaphor comprehension systems by investigating multimodal cues. our main contributions are as follows: multimet will be released publicly for research. ## dataset analysis metaphor category. we analyzed the role of textual and visual modalities to detect metaphors. from figure 4 (a), we can see a complementary category among the three kinds of multimodal metaphors, which requires the interplay of textual and visual modality to understand the metaphorical meaning. it accounts for the largest proportion of metaphors, followed by the text-dominant and image-dominant categories. it shows the contribution of visual factors, which are similarly important in detecting metaphors. we therefore present a quantitative study of the role of textual and visual modalities in metaphor detection through human annotations and confirm the role and contribution of visuals in metaphor occurrence in natural language. author intent. figure 4 (b) shows that expressive and persuasive intentions occur most frequently in the metaphorical data. however, descriptive intention occurs most frequently in the non-metaphorical data. this suggests that on the one hand, we are more likely to use metaphorical expressions when expressing our feelings, expressing emotions, or trying to persuade others. on the other hand, we tend to use literal expressions to make relatively objective statements. sentiment. figure 4 (c ) shows that there are some differences in the distribution of sentiment between the metaphorical data and non-metaphorical data. in the non-metaphorical data, neutral sentiment accounted for the largest proportion of 51%, followed by positive sentiment (33%), strong positive sentiment (7%), negative sentiment (7%), and strong negative sentiment (2%). in the metaphorical data, positive sentiment accounted for the largest proportion of 42%, followed by neutral sentiments (39%), strong positive sentiment (8%), negative sentiment (8%), and strong negative sentiment (3%). it turns out that there are more non-neutral sentiments in metaphor expression than in nonmetaphorical expression, and that metaphors are more frequently used to convey sentiments. our findings accord with the results of previous studies on monomodal textual metaphors that metaphors convey more sentiments or emotions than literary text @xcite . we confirm the stronger emotional impact of metaphors than literary messages from a multimodal perspective. in positive sentiment, the most common words in the source domain are person, face, and flower; the most common words in the target domain are love, life, and success. in negative sentiment, heart, food, and smoke are the most common words in the source domain, and the world, disaster, and life are the most common words in the target domain. this shows that sentiment tendency can influence the category in the source and target domains to some extent. ## experiment for the dataset constructed for this paper, we propose three tasks and provide their baselines, namely multimodal metaphor detection, multimodal metaphor sentiment analysis, and multimodal metaphor author intent detection. we used the model shown in figure 5 to detect metaphors, metaphorical sentiments, and metaphorical intentions. for text input, we used a text encoder to encode the text and to get the feature vector of the text. this paper used two different methods to encode the text, namely the pre-trained bidirectional encoder representations from transformers (bert) model @xcite and bidirectional long-short term memory (bi-lstm) networks @xcite . similarly, for image input, we used an image encoder to extract image features. we used three different image pre-training models: vgg16 (simonyan and zisserman, 2014), resnet50 @xcite , and effi-cientnet @xcite . these methods have been widely used by researchers in feature extraction for various tasks. after obtaining the text feature vector and the image feature vector, we used four different feature fusion methods to combine the vectors, namely concatenation @xcite , elementwise multiply @xcite , element-wise add @xcite , and maximum @xcite . finally, we inputted the fusion vector into a fully connected layer and obtained the probabilities of different categories through the softmax activation function. figure 5 : multimodal model for integrating text and image data. ## conclusion this paper presents the creation of a novel resource, a large-scale multimodal metaphor dataset, multimet, with manual fine-gained annotation for metaphor understanding and research. our dataset enables the quantitative study of the interplay of multimodalities for metaphor detection and confirms the contribution of visuals in metaphor occurrence in natural language. it also offers a set of baseline results of various tasks and shows the importance of combining multimodal cues for metaphor understanding. we hope multimet provides future researchers with valuable multimodal training data for the challenging tasks of multimodal metaphor processing and understanding ranging from metaphor detection to sentiment analysis of metaphor. we also hope that multi-met will help to expand metaphor research from monomodality to multimodality and improve the performance of automatic metaphor understanding systems and contribute to the in-depth understanding and research development of metaphors. the dataset will be publicly available for research.