M eme MQA : Multimodal Question Answering for Memes via Rationale-Based Inferencing (2024)

Abstract: Memes have evolved as a prevalent medium for diverse communication, ranging from humour to propaganda. With the rising popularity of image-focused content, there is a growing need to explore its potential harm from different aspects. Previous studies have analyzed memes in closed settings - detecting harm, applying semantic labels, and offering natural language explanations. To extend this research, we introduce MemeMQA, a multimodal question-answering framework aiming to solicit accurate responses to structured questions while providing coherent explanations. We curate MemeMQACorpus, a new dataset featuring 1,880 questions related to 1,122 memes with corresponding answer-explanation pairs. We further propose ARSENAL, a novel two-stage multimodal framework that leverages the reasoning capabilities of LLMs to address MemeMQA. We benchmark MemeMQA using competitive baselines and demonstrate its superiority - ~18% enhanced answer prediction accuracy and distinct text generation lead across various metrics measuring lexical and semantic alignment over the best baseline. We analyze ARSENAL’s robustness through diversification of question-set, confounder-based evaluation regarding MemeMQA’s generalizability, and modality-specific assessment, enhancing our understanding of meme interpretation in the multimodal communication landscape.

## introduction memes offer an accessible format for impactful information dissemination for everyone without conventional dependencies of proper formatting or formal language. it provides an easy opportunity for novice content creators and seasoned professionals to propagate information that may * denotes equal contribution 1 caution: potentially sensitive content included; viewer discretion is requested. sometimes be harmful to the general audience, especially in the age of internet virality. previous work has explored aspects such as harmfulness in various forms, such as hate speech @xcite , cyber-bullying @xcite , and offensive languages @xcite , of memes, typically in a black-box setting. memes, with their appealing format and influential nature on social media, necessitate the modeling of complex aspects like harmfulness, targeted social groups, and offensive cues to assess their narrative framing and ensure online content safety. their growing prevalence as a key medium for information dissemination poses significant societal challenges. a question-answering setup, particularly open-ended or instruction/response formats, offers a user-friendly method for probing models about the potential harmfulness of memes and understanding their responses. this approach enhances model interpretability and serves as an effective tool for content moderation. in this work, we explore contextualized semantic analysis of memes by introducing a novel multimodal task, mememqa (c.f. fig. 1 ), which is formulated as follows: given a meme and a structured question about the semantic role assigned to various entities, (a) deduce the correct answer entity from a set of multiple options, while also, (b) generating succinct explanations towards the answer. building on the work of @xcite , we explore the narrative framing of entities like well-known individuals and political figures in online memes. this research is especially important during critical events like elections or pandemics, where the risk of spreading harmful content such as hate speech and misinformation increases, highlighting the need for effective moderation. we adopt terms like 'hero @xcite to analyze memes' intentions of victimization, glorification, and vilification. our goal is to deepen the understanding of these memes and contribute to making social media safer. the mememqa framework is designed to assist social media users and fact-checkers in evaluating the harmfulness of memes, enabling them to ask questions and receive accurate, informed responses. analyzing memes in mememqa is complex due to their nuanced meanings that demand advanced reasoning, including common sense, and cultural understanding. for instance, the meme in fig. 1 could simultaneously highlight the role of immigrants in america and promote the libertarian party. to correctly answer the question "who is lauded in this meme?", it's essential to grasp the meme's key themes and the implied message about immigrants enriching diversity, which directly glorifies them. therefore, "immigrants" is the most suitable answer in this context, rather than "libertarian party" or "america", which, despite being referenced positively, would lead to an incorrect conclusion. in summary, we introduce a new task for answering and explaining multiple-choice questions about political memes, creating a dataset (mememqacorpus) with 1, 880 questions for 1, 122 memes using exhvv dataset @xcite . we benchmark mememqacorpus with various unimodal and multimodal baselines, including recent multimodal llms, and propose arsenal, a novel modular approach that leverages multimodal llm reasoning capabilities. arsenal includes rationale, answer prediction, and explanation generation modules. we analyze and compare the performance of arsenal against these baselines, highlighting its strengths and limitations. our contributions are summarised as follows foot_0 : this section provides a concise coverage of prominent studies on meme analysis, while also reviewing contemporary works within the domain of visual question answering. finally we consolidate our assessment of the current state-of-the-art in multimodal llms. ## the mememqacorpus dataset current meme datasets typically encompass either categorical labels @xcite @xcite or their associated explanations @xcite . although conventional visual question answering (vqa) @xcite frameworks exist, they lack the nuanced complexity of memes. these include tasks like detection, segmentation, conditional multimodal modeling (such as caption generation, visual question answering, and multiple-choice vqa), and strong visual-linguistic integration (e.g., setups similar to ms coco for question-answering that focus on common-sense and objective reasoning) @xcite . while these areas present their distinct challenges and mark a significant line of inquiry within the intersecting realms of computer vision and natural language processing (multimodality), they fall short of addressing the complexities of multimodal reasoning, abstract idea representation, and the nuanced use of language mechanisms like puns, humor, and figures of speech, etc. these elements are often integral to memes. this oversight has generally curtailed the effectiveness of existing multimodal approaches @xcite in capturing the nuanced complexities inherent to memes. to address this gap, we introduce mememqacorpus, a novel dataset designed to emulate free-form questioning and multiple-choice answering. given the overwhelming diversity of possible question-answer pairs for the multifarious phenomena presented in memes, we supplement exhvv @xcite , an existing multimodal dataset consisting of natural category-specific role-label synonym (r') "in this meme?" ## the arsenal model prior to exploring an effective design towards addressing mememqa, we analyze different prompting configurations using meme-based inputs to determine the optimal strategy. this section begins by outlining the optimal prompting strategy, then details the structural aspects of arsenal. ## experiments in our study, arsenal is rigorously tested against various models, with results averaged over five runs. the mememqa task involves two components: answer prediction and explanation generation, each evaluated using different metrics. answer prediction is measured for accuracy due to entity imbalance and open-ended nature in the exhvv dataset. explanation quality is assessed against exhvv ground truth using metrics like bleu-1, bleu-4, rouge-l, meteor, chrf, and bertscore. baseline comparisons span uni-modal (text, image) and multi-modal settings. additionally, arsenal's robustness is evaluated through diverse question types, confounding-based tests, and multimodal and error analyses. ## benchmarking mememqa in the closed-form visual question answering domain, we benchmark against models like the multimodal vilt, which achieves a fine-tuned accuracy of 0.43. llm-based models like minigpt4 and gpt3.5 show low answer prediction accuracy in both zero-shot (0.32) and visual descriptionbased fine-tuning (0.28) for the former, and 0.28 for gpt3.5. these models' explanations lack specificity, as indicated by minigpt4's bleu-1 score of 0.12 and rouge-l score of 0.16 post-fine-tuning. despite their detailed and reasonable reasoning, they fall short in standard evaluations due to their excessive length and nonspecific content. however, bertscore values of 0.771 for minigpt4 (ft) and 0.837 for llava-based models suggest a reasonable coherence with the memes in question. our primary comparison is to models leveraging the mm-cot model in various prompt and input settings. the utility of the ocr text is proven by the 8% drop in accuracy on eliminating the ocr text from the input. the addition of generic lectures (l) also improves the model's performance, with a 2% increment in answer prediction accuracy. introducing a contextual rationale using zero-shot inferencing using an llm such as llava presents qualitative improvements in the explanation generation quality of the mm-cot model. it is also worth noting, that the mm-cot model underperforms in understanding memes compared to the new model, which excels in accuracy and explanation due to its rationale generation module, offering a deeper contextual grasp of meme content. ## robustness analysis a key factor that is expected to characterize the efficacy of a model for a task like mememqa, is it's robustness to variations within the question/answer formulation. this is also critical due to the resultant variability within the llm's generated responses (salinas and morstatter, 2024). to this end, we examine arsenal's performance in comparison to other contemporary baselines, by (a) question diversification, and (b) confounding analysis (c.f. ## conclusion and future work this study introduced mememqa, a task that involves multimodal question answering for imagetext memes, delving into their intricate visual and linguistic layers. utilizing recently opensourced llms, especially their multimodal adaptations, we tackled the challenge of complex, nontrivial multimodal content. through a new dataset, mememqacorpus, we assessed systems' reasoning in assigning semantic roles to meme entities via question-answering and contextualization based objectives. our experiments showcased the efficacy of the proposed two-stage training framework, arsenal, while leveraging existing language models and multimodal llms, to outperform the stateof-the-art by a remarkable 18% accuracy gain. this study reveals the potency and limitations of multimodal llms, enabling the scope for sophisticated setups embracing diverse questions, domains, and emotional nuances conveyed through memes. ultimately, our findings steers future exploration and the development of comprehensive systems dedicated to deciphering memetic phenomena. our future aim is to create sophisticated, multiperspective sets for mememqa, moving beyond standard qna towards an optimal multimodal solution. llava's generation capacity. for some cases, llava's rationales seem to be mining the inductive biases due to the co-occurrences of disparate keywords while being influenced by llm's pretraining corpus and web data, exhibited mostly for missing-modality and high inter-modal incongruity. an example for the latter shown in fig. 8 (c.f. appendix i) illustrates how biased inference by llava dilutes arsenal's output due to inaccurate contextualization, whereas mm-cot deduces the answer accurately, possibly due to standardized definitions being used instead of llm-based rationales.