M ulti CMET : A Novel C hinese Benchmark for Understanding Multimodal Metaphor (2023)

Abstract: Metaphor is a pervasive aspect of human communication, and its presence in multimodal forms has become more prominent with the progress of mass media. However, there is limited research on multimodal metaphor resources beyond the English language. Furthermore, the existing work in natural language processing does not address the exploration of categorizing the source and target domains in metaphors. This omission is significant considering the extensive research conducted in the fields of cognitive linguistics, which emphasizes that a profound understanding of metaphor relies on recognizing the differences and similarities between domain categories. We, therefore, introduce MultiCMET, a multimodal Chinese metaphor dataset, consisting of 13,820 text-image pairs of advertisements with manual annotations of the occurrence of metaphors, domain categories, and sentiments metaphors convey. We also constructed a domain lexicon that encompasses categorizations of metaphorical source domains and target domains and propose a Cascading Domain Knowledge Integration (CDKI) benchmark to detect metaphors by introducing domain-specific lexical features. Experimental results demonstrate the effectiveness of CDKI. The dataset and code are publicly available.

## introduction metaphor plays an important role in human cognition and communication, and its ubiquity has been established by empirical studies showing the occurrence of metaphor about once in every three sentences in regular parlance @xcite . with the development of modern media, when compared to monomodal metaphors, multimodal metaphors are significantly increasing due to their vivid, attractive, and persuasive effects. a multimodal metaphor is defined as a mapping that conceptualizes one target domain in terms of another source domain from different modes, such as text and image, text and sound, or image and sound @xcite . figure 1 presents a compelling metaphorical representation depicting lungs constructed from cigarettes, symbolizing the connection between two distinct entities, namely the lung and the cigarette. this metaphor invokes the perceptual notion that smoking is a leading cause of lung cancer. the source domain, represented by the "cigarette" image, intertwines with the target domain, represented both textually and visually through the depiction of the "lung". understanding multimodal metaphor mainly focuses on identification of the underlying mapping of the two domains as well as the extraction of properties conveyed by metaphors @xcite . it also requires the decoding of implicit messages and the identification of the semantic relationship between target and source domains @xcite , so it involves cognitive efforts, which makes it significantly challenging for machines. in recent years, there has been a surge of interest in multimodal metaphor research within the field of natural language processing (nlp) @xcite @xcite . however, the study of multimodal metaphors in the chinese language is extremely limited. it is important to note that metaphorical linguistic and visual expressions may vary across languages and cultures, despite the existence of universal conceptual metaphors @xcite . for example, "human is animal" is a universal conceptual metaphor in which the attributes or behaviors of animals are used to describe human attributes or behaviors. however, various languages may pick up different metaphorical linguistic and visual expressions underlying the same or similar semantics to realize the metaphorical conceptualization. for example, to use a metaphor to describe the state of being drunk, english speakers choose newt or skunk as a source domain, such as "as drunk as a skunk", while chinese speakers would choose mud, as in "as drunk as mud". due to the variations in multimodal metaphors across different languages, there is an urgent need to construct multimodal metaphor datasets in languages other than english. this will enable a more comprehensive automatic understanding of metaphor. furthermore, previous research on multimodal metaphor understanding has been limited in its depth of analysis, primarily focusing on metaphor detection tasks while neglecting tasks related to deep metaphor comprehension. these tasks include differentiating various categories of source domains and target domains, as they significantly contribute to the emergence of metaphors and facilitate a more profound understanding of their underlying meanings.therefore, given the importance of domain classification knowledge, the establishment of a domain dictionary that includes domain categorization is crucial for the automatic comprehension of metaphors. to bridge the aforementioned research gap, we introduce the chinese multimodal metaphor dataset (multicmet). this dataset comprises textimage pairs extracted from chinese advertisements, encompassing textual slogans and their corresponding image counterparts. we annotate not only the occurrence of metaphor as in previous work, but also how metaphor arises in multimodality. specifically, we provide explicit reasoning for multimodal metaphors by annotating both source and target domains. the quality control and agreement analyses for multiple annotators are also presented. furthermore, we construct a domain lexicon which contains hierarchical domain categories based on the upper/lower relation in wordnet's set of nouns @xcite along with corresponding examples. we introduce three evaluation tasks to assess the capabilities of multimodal metaphor understanding: metaphor detection, sentiment analysis, and a novel task, namely domain classification, which focuses on classifying the source and target domains of metaphors-an area that has not been previously explored. we propose the cascading domain knowledge integration (cdki) benchmark, which leverages domain-specific features derived from the domain lexicon. during the evaluation phase, we employ multiple benchmarks with cdki to assess the performance of metaphor understanding tasks, showcasing the effectiveness of cdki. our contributions are as follows: ## dataset analysis refer to figure 5 demonstrates varying emotional distributions across different metaphors. in metaphorical data, positive and neutral emotions are evenly distributed, while negative emotions account for a smaller portion. this indicates that metaphorical expressions in advertising often correlate with positive or neutral attitudes. in non-metaphorical data, neutral emotions dominate the dataset, followed by positive emotions, with negative emotions being the least prevalent. this reflects the directness and clarity of non-metaphorical expressions, which are typically utilized to convey widely accepted information or depict everyday life scenarios. by comparing the statistical data of metaphorical and non-metaphorical expressions, we can observe that the proportion of neutral data in metaphorical expressions is lower than that in non-metaphorical expressions, suggesting that metaphorical expressions tend to encompass a richer emotional range. our findings are consistent with previous research suggesting that metaphors convey more emotions or feelings than simple prose (citron and goldberg, 2014). ## methodology taking into account the significance of domain knowledge in metaphor detection, we propose a benchmark called cascading domain knowledge integration (cdki) that aims to enhance the model's ability to detect metaphors by incorporating domain knowledge. the schematic diagram of cdki is illustrated in figure 6 . introducing domain knowledge into models poses a challenge due to inherent differences in how it is presented in text and images. textual domain knowledge is typically easier to extract, expressed explicitly through specific vocabulary and sentence structures. conversely, domain knowledge in images tends to be more implicit. in the following section, we will elaborate on how cdki addresses this disparity and achieves the extraction and integration of domain knowledge. regarding the text modality, we employ a segmentation technique to preserve multiple inherent domain-specific vocabularies in the textual data by extracting nouns. these nouns are then connected using the [sep] token to form k text , which serves as the domain knowledge for the text modality. regarding the image modality, we have constructed a cascaded domain word set to introduce and enhance the dependency relationships of domain knowledge within the images. specifically, the cascaded domain word set for an image consists of three parts: the macro-level domain word set, the micro-level domain word set, and the entity-level domain word set. these word sets are composed of vocabulary corresponding to the respective levels in the domain lexicon. taking the macro-level domain word set as an example, we will describe the construction method of the cascaded domain word set for images. by utilizing clip @xcite , we obtain the probabilities of each vocabulary in the macro-level of the domain dictionary appearing in the image. we then retain the top n vocabularies with the highest probabilities as the macro-domain word set set macro for that particular image. the construction method for the microdomain word set, set micro , and the entity-domain word set, set entity , follows the same approach. each set consists of p and q vocabularies, respectively. collectively, these three sets constitute the domain knowledge within the image,denoted as k image : lastly, the domain knowledge from the image and text modalities are combined to generate the final domain knowledge, denoted as k pair , for the image-text pairs: in addition, we conducted multiple sets of experiments to investigate the impact of different vocabulary sizes on the model results while keeping the text attributes fixed. the details of these experiments can be found in appendix ??. ultimately, we determined that the optimal values for n, p, and q are 2, 3, and 5, respectively. to incorporate the domain knowledge k pair into the model, we employ bert @xcite to convert the k pair into vectors, which serve as the domain feature input for the image-text pairs. we utilized bert and resnet50 @xcite to obtain textual and visual features respectively. subsequently, we utilize cross-attention mechanism @xcite to jointly model the domain features with the image and text features. this approach enhances the interaction of domain knowledge between different features. the resulting outputs are concatenated and fed into the softmax function to obtain the final result. 6 experiments 6.1 baselines in this section, we introduce the baselines used in our experiments. specifically, we adopted random, text-based, image-based, and multi-modal models as the baseline models and compared their performance with our proposed model. due to the fact that metaphors, irony, and hate speech fall under the category of figurative language, there exists a certain degree of similarity between these three tasks @xcite . thus, we selected these multi-modal methods as our control models. random: it denotes random predictions based on the data. resnet50 @xcite : it is a 50-layer convolutional neural network with residual connections. image features are extracted from its last convolutional layer's output vector. vgg16 @xcite : it enhances network depth and the expression capability of non-linear features by stacking consecutive 3x3 convolutional layers and max pooling layers. bi-lstm @xcite ): it's a bidirectional recurrent neural network that combines the outputs of forward and backward lstm to capture global context in a sequence. bert @xcite and roberta @xcite : we conducted the experiment using two widely-used chinese transformer-based pre-trained language models, namely bert-basedchinese and roberta-base-chinese, as baselines. res-bert @xcite : it uses resnet50 to extract image features and bert to encode text. then, the two sets of features are concatenated to form a fused feature vector, which is input into a fully connected layer for classification. vilbert @xcite and visualbert (li et al., 2019): they are both multimodal language models based on the transformer architecture. vil-bert utilizes a dual-stream attention mechanism to encode visual and textual information. visual-bert employs a self-attention mechanism to encode images and text in a single stream manner. hfm @xcite : the multimodal fusion model combines image, attribute, and text features using bi-lstm and mlp to generate a unified representation for ironic prediction tasks. ces @xcite : this method improves hate speech meme classifier accuracy by leveraging image-caption to enhance both multimodal and unimodal models. ## conclusion in conclusion, this paper addresses the limitations in research on multimodal metaphors beyond the english language and the lack of exploration in categorizing source and target domains in metaphors. it introduces the multicmet dataset, a large-scale multimodal chinese metaphor dataset consisting of text-image pairs from advertisements. the dataset includes manual annotations of metaphors, domain categories, and conveyed sentiments. additionally, a domain lexicon is constructed, providing hierarchical domain classifications and corresponding examples. we propose the cdki benchmark, which leverages domain-specific features for better metaphor understanding. this paper significantly contributes to the field of multimodal metaphor understanding by providing a comprehensive dataset, novel methodologies, and benchmark results. it opens up avenues for future research in exploring metaphors across different languages, enhancing metaphor comprehension, and addressing the cognitive mechanisms involved in metaphor processing. the availability of the dataset and code further encourages the research community to expand and improve upon the findings presented in this work.