M eme G uard: An LLM and VLM -based Framework for Advancing Content Moderation via Meme Intervention (2024)

Abstract: In the digital world, memes present a unique challenge for content moderation due to their potential to spread harmful content. Although detection methods have improved, proactive solutions such as intervention are still limited, with current research focusing mostly on text-based content, neglecting the widespread influence of multimodal content like memes. Addressing this gap, we present MemeGuard , a comprehensive framework leveraging Large Language Models (LLMs) and Visual Language Models (VLMs) for meme intervention. MemeGuard harnesses a specially fine-tuned VLM, VLMeme , for meme interpretation, and a multimodal knowledge selection and ranking mechanism ( MKS ) for distilling relevant knowledge. This knowledge is then employed by a general-purpose LLM to generate contextually appropriate interventions. Another key contribution of this work is the dataset, a high-quality, labeled dataset featuring toxic memes and their corresponding human-annotated interventions. We leverage ICMM to test MemeGuard , demonstrating its proficiency in generating relevant and effective responses to toxic memes. red Disclaimer : This paper contains harmful content that may be disturbing to some readers.

## introduction in today's digital world, memes serve as a universal language for expression and engagement. however, as they become a powerful tool for rapid information dissemination, they are increasingly weaponized for cyberbullying and spreading toxic content, posing a challenge to existing content moderation systems. these systems struggle to decipher memes' nuanced meanings, typically reacting figure 1 : an instance of the meme intervention task. @xcite rather than proactively mitigating the harm of offensive content. intervention represents a proactive approach to content moderation, going beyond simple detection to take preventive action against offensive content. interventions aim to mitigate the harmful effects of toxic content and foster a more positive and respectful online discourse. however, existing intervention research has primarily been limited to text-based content such as hate speech @xcite , and misinformation @xcite . the excessive focus on text-based content overlooks the prevalence of multimodal content, which are major contributors to the content ecosystem in social media platforms. this exacerbates the potential for multimodal toxicity, enabling misuse of such mediums. a representative example of intervening in case of a cyberbullying meme is displayed in figure 1 . large language models (llms) @xcite , and visual language models (vlms) @xcite have shown remarkable capability in understanding and generating humanlike text and multimedia content. this has led to their application in various tasks within the domain of content moderation, but predominantly for detection purposes. their ability to understand nuanced language and visual cues has been leveraged to detect toxic or harmful content, both in text @xcite @xcite and multimodal formats @xcite @xcite . some attempts have also been made to use these models for intervention generation tasks, but these efforts have been largely restricted to text-based content @xcite . in this landscape, the zeroshot learning @xcite of these models presents a unique advantage, particularly for tasks like meme intervention, which are characterized by data scarcity. while these models hold considerable promise for content moderation, they are not without their limitations. first and foremost, vanilla llms and vlms lack grounding in a knowledge base specific to the task at hand. this can lead to the generation of generic interventions that may fail to adequately address the bias, stereotype, assertions, and toxicity present in the meme content. in terms of visual content, while vlms have performed well on a variety of traditional visual-linguistic tasks, they often struggle when it comes to memes. the primary reason behind this is the unique nature of memes, which are highly contextual and often rely on a shared understanding of internet subcultures for their interpretation. this makes the accurate assimilation and analysis of memes a challenging task. finally, even when these models are grounded in a knowledge base for the task of meme intervention, there is a need to filter irrelevant and noisy information. without appropriate filtering, these models might end up incorporating irrelevant or misleading information into their interventions. in order to address these limitations inherent in llms and vlms for meme intervention, we developed a comprehensive framework called meme-guard. the development of memeguard was a multi-stage process designed to create a tool capable of understanding and effectively intervening in the spread of toxic memes. in the first stage, we developed a meme-aligned vlm (vlmeme), specifically fine-tuned to understand and interpret memes in all their complexity. this allowed our model to delve deeper into the content of memes. next, we utilized this meme-aligned vlm to identify various facets of the meme, such as the underlying toxicity, bias, stereotypes, and claims being made, which provides valuable insights into the meme's potential harm. to address the challenge of irrelevant knowledge, we then proposed a multimodal knowledge selection mechanism (mks). this mechanism retained only the most relevant knowledge for the intervention generation process. in the subse-quent stage, we utilized a general-purpose llm grounded on this refined knowledge to generate appropriate interventions. this model took the insights provided by the vlm and the ranked knowledge to create contextually relevant and effective responses to toxic memes. finally, to test the efficacy of our framework, we developed a high-quality labeled dataset featuring a variety of toxic and cyberbully memes with their corresponding human annotated intervention, intervening cyberbullying in multimodal memes (icmm) dataset. to summarize, we make the following main contributions: a novel task of meme intervention to combat the toxicity of cyberbullying memes. a novel dataset, icmm, to advance the research in this area. a novel framework, memeguard, that utilizes a meme-aligned vlm (vlmeme) to generate contextual information about the meme that is then used to generate the final intervention. hee et al. (2023) addressed the research gap by proposing hatred dataset annotated with the underlying hateful contextual reasons. moreover, sharma et al. (2023) proposed a new task that aims to identify evidence from a given context to explain the meme. jha et al. (2024) proposed multibully-ex a benchmark dataset for with multimodal rationales for codemixed cyberbullying memes. recently, hwang and shwartz (2023) also released a labeled dataset for the task of meme caption. intervention and counterspeech generation: list all the toxic element in the meme write an interventive content addressing the toxic element present in the meme. write an interventive filler addressing the criteria that meme violates. list all the criteria that meme violates while bestiality is unethical and warrants condemnation, it is derogatory to defame an entire country based on the actions of a limited group promoting it. we should strive to use language that is respectful and appropriate in all situations. interventive content interventive filler • diversity • inclusivity • respectful • empathy • equality • trust • fact-checking • ....... after sex selfies usa croatia germany serbia • gender • religion • nationality • physical ability • mental ability • stereotype • societal biases • ......... ocr text counterspeech can be regarded as a preferred remedy for combating hate speech as it simultaneously educates the perpetrators about the consequences of their actions and upholds the principles of free speech. recent studies have shown the remarkable effectiveness of social intervention on social media like twitter @xcite , and facebook @xcite . @xcite studied the conversations on twitter and found that some arguments between strangers lead to favorable changes in discourse. @xcite released the first counterspeech dataset with 13,924 manually annotated youtube comments, marked as counterspeech or not. due to the limited scalability and generalizability of socially intervened counterspeech datasets, expert-annotated datasets were developed @xcite for counterspeech in hate speech. ## icmm dataset to create icmm, we utilize multibully dataset @xcite , which includes 3222 bully and 2632 nonbully memes. we selected this dataset because it is the only openly available meme dataset on cyberbullying in a code-mixed setting. our annotation process focuses solely on bully memes to develop interventions against online cyberbullying. ## methodology problem formulation: the input consists of two entries: (1) the source content m, which is a meme potentially bearing toxic content; this multimedia data is composed of visual elements inherent to the image; (2) the ocr text x derived from the source content m, representing the textual elements embedded within the meme. the output is a natural language sequence y , which represents the intervention crafted to counter the toxic content of the meme m. this section presents the novel architecture of memeguard (figure 3 ), a multimodal, knowledgegrounded framework designed to mitigate toxic content in memes. for a comprehensive understanding, we partition our approach into three distinct modules: (i) toxic meme analysis mod-ule (ii) multimodal knowledge selection (mks), and (iii) intervention generation module. ## experimental results and discussion experimental setup: we leveraged following general purpose llms for intervention generation module: dolly foot_1 , llama @xcite , redpajama foot_2 , flan-t5 @xcite , and gpt3.5-turbo. details are mentioned in appendix ## conclusion and future work in this paper, we introduce icmm, the first meme dataset incorporating human-annotated interventions for 1000 cyberbullying memes, setting a new gold standard. the icmm dataset presents novel avenues for assessing the efficacy of interventions generated by generative models relative to human responses. we further put forth the memeguard framework, harnessing the power of large language models (llms) and visual language models (vlms) in the generation of meme interventions, supplemented by the introduction of a memefocused vision language model (vlmeme) and a multimodal knowledge selection mechanism (mks). in future research, we aim to focus on the development of more resilient and effective vision-language models for a better understanding of memes. additionally, we plan to expand this work to encompass toxic videos and reels.