M eme I nterpret: Towards an All-in-One Dataset for Meme Understanding (2025)

Abstract: Meme captioning, the task of generating a sentence that describes the meaning of a meme, is both challenging and important in advancing Computational Meme Understanding (CMU). However, existing research has not explored its decomposition into subtasks or its connections to other CMU tasks. To address this gap, we introduce MemeInterpret, a meme corpus containing meme captions together with corresponding surface messages and relevant background knowledge. Strategically built upon the Facebook Hateful Memes dataset, MemeInterpret is the last piece in a set of corpora that unifies three major categories of CMU tasks for the first time. Extensive experiments on MemeInterpret and connected datasets suggest strong relationships between meme captioning, its two proposed subtasks, and the other two key categories of CMU tasks: classification and explanation. To stimulate further research on CMU, we make our dataset publicly available at https://github.com/npnkhoi/MemeInterpret.

## introduction memes, which are user-created combinations of images overlaid with text, have become a prevalent means of online communication @xcite . they are created with a variety of purposes: while some memes are simply used to express personal opinions, other memes can be malicious, such as inciting hatred or spreading manipulative propaganda. as such, recent years have seen increasing interest within nlp in the emerging area of computational meme understanding (cmu), a term that we coined to refer to research on the automated comprehension of memes @xcite . while numerous cmu tasks have been proposed in the past few years, researchers have essentially adopted the same methodology for tackling each newly-proposed task: (1) annotate a corpus of memes with task-specific labels and (2) fine-tune a vision-language model (vlm) on the resulting corpus. while there is nothing inherently wrong with this methodology, it is not necessarily healthy for the long-term development of cmu research: as a field, existing cmu tasks are being tackled as if they have nothing to do with each other. in light of this concern, we believe we should start thinking about developing task-agnostic rather than task-specific representations of memes. the question, then, is: what task-agnostic representation can be shared by and therefore benefit a range of cmu tasks? as many meme-based nlp tasks require an understanding of the meaning of a meme (i.e., what the meme author tries to convey) rather than its form (i.e., how the meaning is conveyed through the visuals and text), a good starting point would be to experiment with using the meaning of a meme as its task-agnostic representation. 2 for this reason, we believe that it is important to examine an under-studied but important category of cmu tasks, meme interpretation, which is referred to as meme captioning by @xcite and intent description generation by @xcite . both tasks concern generating a description that captures the meaning of a meme. given the meme in figure 1a , the meme caption would be "this meme maliciously makes fun of the trauma of the vietnam war veterans". meme captioning could benefit a range of cmu tasks. for example, given the aforementioned meme caption, a hateful meme detection (hmd) system can easily determine that this meme is hateful. we emphasize, however, that the significance of meme captions lies in the fact that memes that have different forms but convey the same meaning are being mapped to the same meme caption. this helps reduce data sparsity and remove information irrelevant to the meaning. therefore, when training task-specific models for downstream cmu tasks, a model that takes meme captions (rather than the original memes) as input can potentially be finetuned on a smaller amount of task-specific data to achieve a given level of performance. meme captioning, however, is challenging for at least three reasons. first, for many memes, especially those that are malicious (e.g., the author's intent is to manipulate public opinion), the meaning of the meme is typically not explicitly stated and therefore can only be inferred. second, meme captioning relies on background knowledge (bk), which by definition is not explicitly stated in the meme either. take figure 1a as an example. to properly understand the terms "popcorn", "senior", and "vietnam veteran" and their relationships in the meme, one needs to possess the bk that (1) "the vietnam war was a highly traumatic period for us soldiers who are now old veterans mostly living in senior centers", and (2) "the sound of popcorn popping in the microwave can remind them of the gunfire sound during the war", but (3) "generally, that sound is not to be scared of because there is no apparent danger". while large language models (llms) possess lots of knowledge, what is challenging is the generation of the bk relevant to a given meme. third, even if a model can identify relevant bk, a successful meme captioner needs to address another challenging task that we refer to as surface message (sm) generation, which involves generating a sentence to (1) capture the details of the meme that are crucial for its interpretation by (2) combining the meme's image and text in a coherent manner. returning to figure 1a , a good sm for this meme is "it is an image of a 70-year-old caucasian man sitting in front of a window with a stunned face, wearing a hat that says 'vietnam veteran'. the author describes the image as 'me: puts bag of popcorn into the microwave. every one else at the senior center'." this sm captures the information on the image that enables the correct interpretation of the meme (e.g., the text "vietnam veteran" could have been easily ignored by a typical image captioner). furthermore, it merges the information from the text and the image in a coherent manner by resolving "me" to the author. 3 to advance research in meme captioning, we propose memeinterpret, a meme corpus in which each meme is manually annotated with its meme caption and, given the aforementioned challenges associated with meme captioning, its surface message and the relevant background knowledge. in previous work, each meme in memeinterpret is already labeled with whether they are hateful or not (a categorization task), and a subset of the hateful memes is additionally labeled with an explanation of why it is hateful (an explanation task). therefore, memeinterpret is the first meme corpus that bridges together three major categories of cmu tasks, namely categorization, explanation, and interpretation (see section 2). our empirical studies showed that surface message and background knowledge annotations significantly improved the performance on meme captioning. moreover, stronger meme captioning systems can advance the performance in meme classification and explanation. given these results, we envision that memeinterpret can spark a new avenue of research in cmu that involves studying how different categories of tasks interact with and possibly benefit from each other. ## related work in this section, we review three categories of tasks in cmu. for a more comprehensive overview of cmu research, see @xcite . memeinterpret, is composed of both hateful and non-hateful memes, memecap excludes hateful memes, which is a key weakness given the important role played by hateful memes in cmu @xcite , and does not annotate surface messages and background knowledge. explanation explanation tasks involve generating a description of why a meme should be assigned a particular label (e.g., sarcastic). for instance, if our task involves explaining why the meme in figure 1b is sarcastic, the explanation would be "languages used for people who fear long words are supposed to be short, but the medical name for the fear is actually long." so far, only two meme corpora have been produced for explanation tasks, namely, hatred @xcite and exhvv @xcite . caregorization categorization tasks involve classifying memes along various dimensions. these tasks can be broadly divided into two categories. the first group is composed of tasks that involve detecting malignity in memes, including offensiveness @xcite , trolls @xcite , hate @xcite , antisemistism @xcite , harm @xcite , and misogyny @xcite . the second group contains tasks that categorize memes along other dimensions such as types of persuasion techniques @xcite , types of figurative language (e.g., irony, allusion, irony, sarcasm, and contrast) @xcite , entities' roles (e.g., hero, villain, or victim) @xcite , emotions (e.g., humor) @xcite , and attacked target groups (e.g., religion, race, sex, nationality, and disability) @xcite . ## the memeinterpret dataset 3.1 meme source to assemble memeinterpret, we sample the memes from the facebook hateful meme dataset (fhm) @xcite . specifically, when assembling memeinterpret, we inherit the entire training set and the test "seen" set from fhm, which has 8500 and 1000 memes, respectively. 42% of the memes in memeinterpret are hateful. we chose fhm for two reasons. first, and perhaps most importantly, each meme in fhm was already labeled as hateful @xcite and a subset of the hateful memes were annotated with explanations of why the memes were hateful @xcite . therefore, our additional annotations on the memes sampled from fhm will enable us to study how the three categories of cmu tasksnamely interpretation (meme captioning), explanation (hatefulness explanation), and categorization (hateful meme detection) -interact with and possibly benefit from each other. second, the memes cover a variety of topics encountered in everyday life, including politics, race and ethnicity, sex and relationships, food and eating habits, animals and pets, historical events, social issues, religion and beliefs, lifestyle and daily life, stereotypes and prejudices, as well as hobbies and interests. ## empirical studies we conducted a broad range of experiments on memeinterpret to (1) gauge the usefulness of our annotations, and (2) empirically verify our hypotheses about meme captioning and its relationship with downstream cmu tasks. ## error analysis to gain further insights into the challenges in modeling meme interpretations, we analyzed the errors made by our best performing models, which are the fine-tuned bk generation model (row 4, ## conclusion and future work in light of the fact that existing cmu tasks are largely tackled independently of each other, we took the first step towards suggesting that this current research practice can possibly be reshaped by proposing a task-agnostic representation of memes based on meaning rather than form. subsequently, we (1) advocated the importance of advancing research on the under-studied task of meme captioning, (2) identified two challenging subtasks of meme captioning, background knowledge generation and surface message generation, and (3) proposed memeinterpret, which could spark research on studying the interactions of different categories of cmu tasks. extensive experiments showed that (1) the memeinterpret annotations are useful, as the fine-tuned results are better than the zeroshot results; (2) meme captioning could benefit other categories of cmu tasks, including classification and explanation tasks; and (3) memeinterpret could facilitate the development of a unified framework that can simultaneously address all three categories of cmu tasks. as the fine-tuned models still have a lot of room for improvement, we believe future work should include (1) the development of novel models that can better exploit our annotations, especially joint models that allow interaction of multiple cmu tasks and (2) an exploration of whether our annotations can benefit additional downstream cmu tasks.