M 3 H op- C o T : Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thought (2024)

Abstract: In recent years, there has been a significant rise in the phenomenon of hate against women on social media platforms, particularly through the use of misogynous memes. These memes often target women with subtle and obscure cues, making their detection a challenging task for automated systems. Recently, Large Language Models (LLMs) have shown promising results in reasoning using Chain-of-Thought (CoT) prompting to generate the intermediate reasoning chains as the rationale to facilitate multimodal tasks, but often neglect cultural diversity and key aspects like emotion and contextual knowledge hidden in the visual modalities. To address this gap, we introduce a **M**ultimodal **M**ulti-hop CoT (M3Hop-CoT) framework for **M**isogynous meme identification, combining a CLIP-based classifier and a multimodal CoT module with entity-object-relationship integration. M3Hop-CoT employs a three-step multimodal prompting principle to induce emotions, target awareness, and contextual knowledge for meme analysis. Our empirical evaluation, including both qualitative and quantitative analysis, validates the efficacy of the M3Hop-CoT framework on the SemEval-2022 Task 5 (**MAMI task**) dataset, highlighting its strong performance in the macro-F1 score. Furthermore, we evaluate the model’s generalizability by evaluating it on various benchmark meme datasets, offering a thorough insight into the effectiveness of our approach across different datasets. Codes are available at this link: https://github.com/Gitanjali1801/LLM_CoT

## introduction in recent years, the proliferation of memes on social media platforms like facebook, twitter, and instagram has gained significant attention due to their widespread influence and potential to shape public discourse. while many memes are created for entertainment, some serve political or activist 1 codes are available at this link: https://github.com/ gitanjali1801/llm_cot purposes, often employing dark humor. misogynous memes foot_0 , however, stand apart by propagating hatred against women through sexist and aggressive messages on social media @xcite @xcite . these memes exacerbate sexual stereotyping and gender inequality, mirroring offline societal issues @xcite and have become a concerning issue @xcite @xcite . identifying misogynous memes is much more challenging than other memes, as the task demands an understanding of world knowledge and common sense @xcite . despite the challenges, developing deep learning models to classify such memes can provide sociological benefits, such as understanding hidden meanings, supporting humanities research, and raising awareness on a large scale @xcite . previous research has primarily focused on developing robust deep-learning models that learn cross-modal interactions (c.f. figure 1 (a) ) from scratch to identify these memes @xcite @xcite @xcite @xcite . however, learning complex multimodal interactions can be difficult with limited data @xcite . the advent of large language models (llms) offers a way to bridge this gap. although llms are highly adept at question-answering and reasoning tasks, they often overlook the cultural diversity of human reasoners, crucial for tasks demanding commonsense, contextual knowledge, and multimodal reasoning @xcite . however, the recent concept of chain-of-thought (cot) has demonstrated the potential of llms for multi-hop reasoning @xcite @xcite , showing that llms can perform chainstyle reasoning effectively with the right prompts. nonetheless, most cot reasoning studies focus primarily on the language modality @xcite , often overlooking multimodal contexts. analyzing memes is particularly challenging because their implicit meanings are not fully conveyed through text and images. in such a scenario, neglecting one modality in meme detection can negatively impact model performance. as depicted in figure 1 @xcite , if an llm can not only interpret emotions, such as anger or disgust from the text, "your job is simple. stay in kitchen," but also analyze the visual elements of the meme featuring a woman and a man, which would further enhance its ability to recognize emotions by considering their facial expressions and body language. this is crucial for identifying sexist stereotypes. moreover, the llm can determine if the meme targets women by evaluating both textual and visual modalities. furthermore, understanding the broader context, which encompasses societal and cultural discussions on gender roles and equality, is crucial, and this can also be achieved using llms. to achieve this, the ability to perform multi-hop reasoning chain-of-though(cot) (i.e., inferring emotion, target, and then understanding the context) is indispensable. by hierarchically considering key aspects of misogynous memes, such as emotions, targets, and contextual backgrounds, we can create general-purpose models that are in sync with human intent for real-world tasks like meme identification. our proposed work is motivated by the aforementioned discussion, where we introduce a deep learning-based framework named m3hop-cot (misogynous meme identification with multimodal multi-hop chain-of-thought) a mod-ular approach that leverages an llm as the "reasoning module" and operates over a given meme. in the m3hop-cot approach, we first extract the entity-object-relationship (eors) of the memeimage using a scene graph @xcite . subsequently, the meme text, image, and eors are fed into the multi-hop cot llm, enabling it to identify three crucial hidden cues for inferring the meme's rationales: (i) emotion, (ii) target, and (iii) context. m3hop-cot eliminates the need for external resources, also bridging the gap between the modalities by utilizing both textual and visual aspects of the meme in rational generation at zero cost. to ensure the weighted contribution of each reasoning step, we employ a hierarchical cross-attention mechanism that assesses the contribution of each rationale in decision-making. the main contributions of this work are summarized below: (i) this is the first study where we introduce multimodal llm in a cot manner to identify the misogynous memes. (ii) we introduce the m3hop (misogynous meme identification with multimodal multi-hop chain-of-thought) framework, where we utilize the meme text and eors of the meme-image as a prompt to the llm in a multi-hop cot manner, enabling it to identify three crucial rationales helpful to detect misogynous memes: (a) emotion, (b) target, and (c) context. (iii) our empirical evaluation, including both qualitative and quantitative analysis, validates the efficacy of the m3hop-cot framework on several datasets, highlighting its strong performance. ## dataset for our experiments, we employ two misogynous meme datasets: mami (semeval2022 task 5, subtask a) (in english) @xcite and mimic (in hindi-english code-mixed) @xcite (refer to ## results analysis in this section, we present the results of our comparative analysis, which examines the baseline models foot_1 , llm-based models, our proposed model, and their respective variations for misogynous meme identification tasks foot_2 . we use the macro-f1 (f1) score on both the dev and test sets as the preferred metrics to measure this. ## conclusion in conclusion, our work introduces a novel approach for detecting misogynous content in memes, leveraging the power of llms with cot reasoning. our proposed model, m3hop-cot, integrates multimodal information and employs a three-step reasoning process to effectively capture memes' emotional, target-oriented, and contextual nuances. by incorporating scene graphs, we enhance the model's ability to understand the visual aspects of memes. our results demonstrate that m3hop-cot outperforms existing sota models, significantly improving f1 scores on both dev and test sets. in the future, we could explore extending our approach to other forms of online content and integrating additional modalities to enhance the model's effectiveness.