M eme CLIP : Leveraging CLIP Representations for Multimodal Meme Classification (2024)

Abstract: The complexity of text-embedded images presents a formidable challenge in machine learning given the need for multimodal understanding of multiple aspects of expression conveyed by them. While previous research in multimodal analysis has primarily focused on singular aspects such as hate speech and its subclasses, this study expands this focus to encompass multiple aspects of linguistics: hate, targets of hate, stance, and humor. We introduce a novel dataset PrideMM comprising 5,063 text-embedded images associated with the LGBTQ+ Pride movement, thereby addressing a serious gap in existing resources. We conduct extensive experimentation on PrideMM by using unimodal and multimodal baseline methods to establish benchmarks for each task. Additionally, we propose a novel framework MemeCLIP for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model. The results of our experiments show that MemeCLIP achieves superior performance compared to previously proposed frameworks on two real-world datasets. We further compare the performance of MemeCLIP and zero-shot GPT-4 on the hate classification task. Finally, we discuss the shortcomings of our model by qualitatively analyzing misclassified samples. Our code and dataset are publicly available at: https://github.com/SiddhantBikram/MemeCLIP.

## introduction in recent years, the pervasive integration of social media platforms into everyday life has resulted in an exponential increase in the generation and dissemination of multimedia content. at the heart of this digital ecosystem lies the meme: a textembedded image imbued with humor, wit, and often, a subversive edge, which offers a medium through which individuals can express opinions, share experiences, and engage in online activism @xcite . with their ability to distill complex ideas into digestible units of communication, memes have emerged as a powerful medium for expressing both support and opposition toward socio-political events @xcite . however, with opinions being expressed freely, hate speech becomes prevalent, often directed towards individuals, organizations, and even marginalized communities @xcite , targeting them with vitriol and prejudice @xcite . particularly, the lgbtq+ movement stands as a prominent subject of online discourse, where memes serve as vehicles of both solidarity and resistance, reflecting the multifaceted dynamics of attitudes and perceptions within the community and beyond @xcite . in this context, the distinction between humor and harm becomes blurred, as memes straddle the line between satire and offense, challenging researchers and platforms alike to navigate the complexities of online content moderation @xcite . previous attempts that endeavored to suppress such content have resulted in the discriminative suppression of all lgbtq+ content @xcite , which can harm the awareness and acceptance of this community. thus, understanding the nuances of hate speech, opinions, and intended humor within memes becomes paramount for fostering an inclusive digital environment and combating online discrimination. to address these challenges, we introduce pridemm: a novel dataset comprising 5,063 textembedded images related to the lgbtq+ movement annotated with a multi-aspect schema encompassing four tasks: ularly challenging given the need for contextual understanding and the prevalence of ambiguity and subjectivity in them @xcite . accordingly, the multi-aspect nature of our dataset provides a more holistic view of the diverse themes usually expressed through memes. through pridemm, we aim to cultivate a more profound understanding of interactions on social media through memes and facilitate the development of multimodal content moderation methods to make the internet a safer space. we implement a range of baseline and stateof-the-art hate speech detection models to establish benchmarks for each task of pridemm. sample images from pridemm are illustrated in figure 1 alongside their annotation labels. we further propose memeclip, a novel framework that leverages the knowledge of the contrastive language-image pre-training (clip) model @xcite by using multiple lightweight modules for multimodal and multiaspect meme classification. we employ linear layers to effectively disentangle image and text representations in clip's multimodal embedding space. we utilize feature adapters to preserve the prior knowledge of clip and adapt its embedding spaces to the meme classification task while avoiding overfitting on smaller datasets. we further implement a cosine classifier alongside semantic-aware initialization @xcite to make it more robust to the class imbalances that may exist in datasets such as pridemm and harmeme @xcite that are representative of real-world data distributions. distinct from previous multimodal meme classification frameworks, memeclip is trained end-to-end in a single step and does not rely on extraneous models to create augmented data. our main contributions can be summarized as follows: ## dataset in this section, we describe various aspects of our dataset including data collection, annotation guidelines, and dataset statistics. our dataset comprises 5,063 text-embedded images that encompass memes, posters, and infographics relevant to the lgbtq+ movement. we only include images from 2020-2024 as this period saw an upsurge of social media content in this domain @xcite . this also allows our dataset to represent contemporary social media interactions through memes. note that by the term lgbtq+, we refer to all gender identities and sexual orientations inclusively. ## methodology in this section, we describe our proposed framework, memeclip, for multi-aspect meme classification. we utilize the vision-language model clip @xcite to create rich representations that effectively encapsulate the semantics of a meme. we add lightweight modules on top of clip to disentangle image and text representations, prevent overfitting, and make memeclip more robust to imbalanced data. figure 2 illustrates the overall architecture of memeclip. below, we describe each component of memeclip in detail. zero-shot clip. the vision-language model clip exhibits stellar zero-shot performance and transfer learning capabilities @xcite . clip is pre-trained on 400 million image-text pairs from the internet, enabling it to encode visual and textual data in a shared embedding space. the model consists of an image encoder e i and a text encoder e t . we freeze the weights of both encoders to preserve the valuable knowledge captured by them during pre-training. the unimodal image and text representations f i , f @xmath0: ## experimental results ## conclusion in this work, we release pridemm, a multimodal and multi-aspect dataset comprising 5,063 memes related to the lgbtq+ movement, addressing a serious gap in data resources in this domain. this dataset provides memes annotated across three aspects and one sub-class, allowing for greater flexibility in the establishment of ethical guidelines by social media policymakers. we further introduce memeclip, a lightweight yet effective framework to harness clip's knowledge for multimodal meme classification while not being reliant on extraneous models to create augmented data for training. with each module of memeclip, we tried to address pervasive problems in the domain of hateful image classification. specifically, we used a cosine classifier to counter class imbalances, which are common in multi-aspect datasets with many classes for each aspect. we utilize feature adapters to mitigate overfitting as datasets in this domain have a relatively small scale. we further use linear projection layers to dissociate the image and text modalities in the representations created by the pretrained clip encoders as the pre-training dataset of clip majorly consists of image-text pairs that convey the same overall meaning, which may not be the case with creative or sarcastic memes. we show that clip, one of the most basic multimodal models, when combined with lightweight additional modules, can compete with or outperform models that require extraneous models to create augmented data, or even vlms such as gpt-4. our work is grounded on the importance of a data-centric approach to solving problems in this domain, rather than creating larger frameworks and incorporating extraneous data. our dataset endeavors to foster a deeper understanding of online interaction, community-building, and social change, whereas our framework is a step toward effective content moderation that helps create an inclusive, diverse, and equitable digital environment for all.