M eme I ntent: Benchmarking Intent Description Generation for Memes (2024)

Abstract: While recent years have seen a surge of interest in the automatic processing of memes, much of the work in this area has focused on determining whether a meme contains malicious content. This paper proposes the new task of intent description generation : generating a description of the author’s intentions when creating the meme. To stimulate future work on this task, we (1) annotated a corpus of memes with the intents being perceived by the reader as well as the background knowledge needed to infer the intents and (2) established baseline performance on the intent description generation task using state-of-the-art large language models. Our results suggest the importance of background knowledge retrieval in intent description generation for memes.

## introduction memes, which are "amateur media artifacts, extensively remixed and recirculated by different participants on social media networks" @xcite , are typically created with an intent to perform some "action" @xcite . while many memes are intended to make a joke (where the author tries to make fun of a celebrity's weird accent, for instance), other memes may have malicious intentions. for instance, a meme author may seek to provoke fear (e.g., by conveying the message that vaccines contain microchips) or manipulate public opinions (e.g., by portraying hillary clinton as a corrupt politician before the 2016 presidential campaign with the goal of garnering support for donald trump). the core task in automated meme understanding, therefore, involves identifying the intent behind the creation of a meme. in this paper, an intent is defined as an action that the meme author does via the meme. for example, the meme in figure 1a intends to "mock * these authors contributed equally to this work. justin trudeau as a communist for being similar to fidel castro", while figure 1b "makes fun of donald trump's hypocrisy regarding his view on the severity of covid-19". as shown in these examples, intents are best represented in textual form. therefore, intent identification is naturally cast as a generation task, hence will be called intent description generation. automatically generating the intent description of a meme is by no means an easy task, for at least two reasons: first, background knowledge is often needed for proper interpretation of a meme. background knowledge refers to the knowledge that is not present in the meme but is needed to recognize the intent when combined with the information that is explicitly stated in the meme. there are different kinds of background knowledge, including historical knowledge (e.g., '"make america great again" is the slogan used by trump in his presidential campaigns'), general political ideologies (e.g., 'progressives favor stricter gun control policies'), or knowledge of the meme culture (e.g., 'the meme template drakeposting foot_0 funnily expresses an objection and an approval'), etc. for example, figure 1a requires the knowledge that castro is a communist leader and that there has been a fear of communism in the west since the cold war. figure 1b , on the other hand, does not require any special knowledge as all of the information necessary to understand the meme is presented at face value. second, in order to derive the intent, complex inference mechanisms may be needed to combine background knowledge with different pieces of information extracted from the image and text portions of the meme. those "combination" steps reflect how a human thinks, such as based on logos (i.e., logical reasoning), ethos (i.e., speaker's authority), or pathos (i.e., emotional appeal). some of these steps are not about logical reasoning, thus harder to automate @xcite . as an example of such inference mechanisms, consider figure 1a again. to arrive at the final intent, we first have to recognize that the person on the left is justin trudeau (canada's prime minister) and the person on the right is fidel castro (cuba's former leader). in addition, trudeau has his mouth open whereas castro has his mouth closed, which signifies that trudeau is speaking and castro is listening. when combining this information with the text "happy father's day", one can infer that trudeau either admits that castro is his father or simply likes castro enough to send his greetings to him. then, combining the background knowledge that castro was a staunch communist with the fact derived earlier that trudeau admires castro, the meme poster is trying to transfer the communist nature of castro to trudeau to damage trudeau's reputation. given the negative sentiment towards communism in the western public, the final intent is thus "mocking trudeau as a communist for being similar to castro". intent description generation, though challenging, is a task whose solution has both practical and theoretical significance. from a practical perspective, knowledge of the intent being perceived through the meme could be useful for other memerelated processing tasks. for instance, knowing what the intent is could facilitate the determination of whether a meme contains harmful content @xcite or the detection of persuasion techniques @xcite . theoretically speaking, being able to generate intents like humans requires that a machine read between the lines and achieve a deeper level of understanding of perceptual input, enabling machine perception to get one step closer to human perception. our contributions in this paper are four-fold. first, we propose the new task of intent description generation. second, we construct the first benchmark for intent description generation, called memeintent, which shows the background knowledge required for each meme and its final intent(s). third, we produce preliminary results on memeintent from two state-of-the-art language and visionlanguage models. finally, based on the experimental results, we justify the need for more careful treatments of background knowledge in meme processing. to stimulate future research in intent description generation for memes, memeintent has been made publicly available foot_1 . the rest of this paper is organized as follows. section 2 provides an overview of related work on automated meme processing. in section 3, we describe our intent description generation benchmark, memeintent. to get an idea of how challenging intent description generation is, we conduct experiments on memeintent, discussing our experimental setup in section 4 and showing preliminary evaluation results of two state-of-the-art large language models on memeintent in section 5. finally, we present our conclusions in section 6. ## related work there has been a recent surge of interest in meme processing. ## benchmark creation in this section, we will show details about memeintent. ## experimental setup with memeintent constructed, we now evaluate the performance of state-of-the-art models on intent generation for memes. the experiments were set up to answer the following research questions (rqs): rq1: what is the effect of adding background knowledge to the input on models' performance in intent description generation for memes? ## conclusion and future work we examined the novel task of generating the description of intents in memes, specifically by (1) constructing memeintent, a benchmark of memes with intents, and background knowledge and (2) producing baseline results on our dataset against which future models can be compared. our key findings suggest the importance of background knowledge treatments in intent description generation. to stimulate research on this task, we make our annotations publicly available. regarding future work, the experimental results w.r.t. the models and zero-shot vs. few-shot are inconclusive. therefore, more experimentation is needed to get a clearer picture. besides, we attempted to fine-tune llama on the training set of memecap and test on memeintent, but the result was not good. this seems to be a failure to generalize from one meme interpretation dataset to another. therefore, more efforts should be put into looking at the discrepancies between current datasets. chakraborty. 2023. what do you meme? generating explanations for visual semantic role labelling in memes. proceedings of the aaai conference on artificial intelligence, 37(8):9763-9771. shivam sharma, tharun suresh, atharva kulkarni, himanshi mathur, preslav nakov, md. shad akhtar, and tanmoy chakraborty. 2022. findings of the constraint 2022 shared task on detecting the hero, the villain, and the victim in memes. in proceedings of the workshop on combating online hostile posts in regional languages during emergency situations, pages 1-11, dublin, ireland. association for computational linguistics. shardul suryawanshi, bharathi raja chakravarthi, mihael arcan, and paul buitelaar. 2020a. multimodal meme dataset (multioff) for identifying offensive content in image and text. in proceedings of the second workshop on trolling, aggression and cyberbullying, pages 32-41, marseille, france. european language resources association (elra). shardul suryawanshi, bharathi raja chakravarthi, pranav verma, mihael arcan, john philip mccrae, and paul buitelaar. 2020b. a dataset for troll classification of tamilmemes. in proceedings of the wildre5-5th workshop on indian language data: resources and evaluation, pages 7-13, marseille, france. european language resources association (elra). sarah wiegreffe and ana marasovic. 2021. teach me to explain: a review of datasets for explainable natural language processing. 1. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. arxiv:1904.09675 [cs].