M eme C ap: A Dataset for Captioning and Interpreting Memes (2023)

Abstract: Memes are a widely popular tool for web users to express their thoughts using visual metaphors. Understanding memes requires recognizing and interpreting visual metaphors with respect to the text inside or around the meme, often while employing background knowledge and reasoning abilities. We present the task of meme captioning and release a new dataset, MemeCap. Our dataset contains 6.3K memes along with the title of the post containing the meme, the meme captions, the literal image caption, and the visual metaphors. Despite the recent success of vision and language (VL) models on tasks such as image captioning and visual question answering, our extensive experiments using state-of-the-art VL models show that they still struggle with visual metaphors, and perform substantially worse than humans.

## introduction web users frequently communicate their thoughts and feelings online using memes @xcite . memes are created by taking an existing widespread image and attaching new meaning to it by altering the text inside the image. for example, in figure 1 , tom cat is a metaphor for the person who posted the meme and the cats he is shaking hands with represent his two regular followers who always like his posts. this incongruity between the image and the text makes memes humorous @xcite . because of their complementary nature, interpreting the meaning of a meme requires understanding both the visual and text modalities. moreover, memes are often posted on social media platforms along with additional text, such as "one of them is my alt" in fig. 1 , which is further needed to understand the meme. recently, there is a surge of vision and language (vl) models (e.g. @xcite @xcite . vl models have shown remarkable capabilities in generating detailed and accurate descriptions of images in both zero-shot and in-context setups. such models are first pretrained on language-only and vision-only datasets, and then trained on tasks such as image captioning and visual question answering, where the redundancy between the vision and language is used to embed them in a shared space. for example, the majority of image captions in existing datasets describe what is depicted in the image, at most adding subjective interpretations or inferences about the story behind the image @xcite . in contrast, there is little work on visual metaphors to date @xcite . in this paper, we are investigating whether vl models can successfully interpret memes. we propose the task of meme captioning, in which models are presented with a meme along with its title (e.g. the title of the post containing the meme), and is tasked with generating a concise caption describing the meaning of the meme. this task goes beyond object recognition and language understanding. it is challenging due to the metaphorical role of the visual content of the meme @xcite . for example, in fig. 1 , the model needs to recognize that tom cat is merely a metaphor for the meme poster, and that handshaking signals appreciation. the literal content of the image, such as tom or the handshake, should not be part of the meme caption. recognizing and interpreting such metaphors involve detecting facial expressions, the tone expressed in the texts, making commonsense inferences, and more @xcite . to that end, we collected a meme captioning dataset memecap, containing 6,384 memes along with their captions. each meme is also annotated with the literal image description (e.g. "tom cat is shaking hands with two small cats and smiling"), and the visual metaphors (e.g. tom is a metaphor for the meme poster). we establish comprehensive baseline performances with recent large-scale vl models, in various training setups (e.g. zero-shot, few-shot, finetuning), and inputs (i.e. meme, title, literal image captions, and metaphors). human evaluation of the generated captions shows that models are far from humans in captioning memes. in particular, models tend to ignore important visual or textual elements, and instead, repeat the text inside the meme or make up fake elements. our findings merit future research on this task. foot_0 2 background ## the memecap dataset the overall data collection and annotation process is illustrated in figure 2 . we collected memes (sec 3.1) and crowdsourced their captions (sec 3.2). we present the data splits and statistics in sec 3.3. ## experimental setup we report the performance of various baselines on memecap. all models are tasked with generating … this is a meme with the title "every damn time". the image description is "a scary looking monster". the following text is written inside the meme: "what time did you go to bed last night?\n me: early, why". ## results we evaluated the performance of the various models with both automatic metrics (sec 5.1) and human evaluation (sec 5.2). we show that the vision and language modalities are complementary through ablation tests (sec 5.3). ## conclusion we present memecap, the first meme captioning dataset. memecap is challenging for the existing vl models, as it requires recognizing and interpreting visual metaphors, and ignoring the literal visual elements. the experimental results using state-ofthe-art vl models indeed show that such models are still far from human performance. in particular, they tend to treat visual elements too literally and copy text from inside the meme. our work opens up interesting future research on recognizing visual metaphors, interpreting them with respect to a textual context, and generating meme captions that are complete with respect to both modalities without creating fake elements.