Exploring Chain-of-Thought for Multi-modal Metaphor Detection (2024)

Abstract: Metaphors are commonly found in advertising and internet memes. However, the free form of internet memes often leads to a lack of high-quality textual data. Metaphor detection demands a deep interpretation of both textual and visual elements, requiring extensive common-sense knowledge, which poses a challenge to language models. To address these challenges, we propose a compact framework called C4MMD, which utilizes a C hain-of-Thought(CoT) method for M ulti-modal M etaphor D etection. Specifically, our approach designs a three-step process inspired by CoT that extracts and integrates knowledge from Multi-modal Large Language Models(MLLMs) into smaller ones. We also developed a modality fusion architecture to transform knowledge from large models into metaphor features, supplemented by auxiliary tasks to improve model performance. Experimental results on the MET-MEME dataset demonstrate that our method not only effectively enhances the metaphor detection capabilities of small models but also outperforms existing models. To our knowledge, this is the first systematic study leveraging MLLMs in metaphor detection tasks. The code for our method is publicly available at https://github.com/xyz189411yt/C4MMD .

## introduction metaphors are highly prevalent in our everyday expressions and writings, which can have a range of impacts on downstream tasks in natural language processing (nlp), such as semantic understanding @xcite , sentiment analysis @xcite and other tasks. in recent years, the rise of social media has sparked interest in multi-modal metaphors. as a result, several datasets for multi- ## related work early metaphor detection tasks were confined to a single modality and employed methods based on rule constraints and metaphor dictionaries @xcite @xcite . with the flourishing development in the field of nlp, machine learning-based methods @xcite and neural network-based methods @xcite have successively emerged. following the introduction of the transformer @xcite , methods based on pre-trained models gradually supplanted the former methods and became the current mainstream approach @xcite @xcite . @xcite have categorized current efforts into four main directions, namely additional data and feature methods @xcite @xcite , semantic methods @xcite @xcite @xcite , context-based methods @xcite , and multitask methods @xcite @xcite @xcite , where semantic methods and multitask methods have become the primary focus of recent research. as an emerging direction, numerous datasets across image and text modalities have emerged, primarily sourced from social media and advertisements, yielding extensive multilingual text-image modal data @xcite @xcite . unlike the aforementioned approaches that extract information from different modalities and directly merge them, we leverage llms employing the cot method to analyze features between modalities, aiding downstream models in cross-modal fusion. ## method we propose a novel framework called c4mmd using mllms to enhance metaphor detection. we first introduce the task definition(3.1) and the complete model architecture((3.2). after that, we elaborate on knowledge acquisition from mllms using the cot method(3.3) and the implementation of the downstream fusion module(3.4). finally, we provide a brief exposition of the training methodology (3.5). ## conclusion our study aimed to tackle the challenges of multimodal metaphor interpretation by leveraging advanced mllms. we designed a three-step method with cot-prompting to extract richer information from both images and text. augmented knowledge from mllms proved crucial in enhancing smaller models to grasp metaphorical features within each modality and in the fusion of modalities. this work not only advances multi-modal metaphor detection but also paves the way for future research exploring the potential of mllms in addressing complex language and vision challenges.