MEMEX : Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization (2023)

Abstract: Memes are a powerful tool for communication over social media. Their affinity for evolving across politics, history, and sociocultural phenomena renders them an ideal vehicle for communication. To comprehend the subtle message conveyed within a meme, one must understand the relevant background that facilitates its holistic assimilation. Besides digital archiving of memes and their metadata by a few websites like knowyourmeme.com, currently, there is no efficient way to deduce a meme’s context dynamically. In this work, we propose a novel task, MEMEX - given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. At first, we develop MCC (Meme Context Corpus), a novel dataset for MEMEX. Further, to benchmark MCC, we propose MIME (MultImodal Meme Explainer), a multimodal neural framework that uses external knowledge-enriched meme representation and a multi-level approach to capture the cross-modal semantic dependencies between the meme and the context. MIME surpasses several unimodal and multimodal systems and yields an absolute improvement of 4% F1-score over the best baseline. Lastly, we conduct detailed analyses of MIME’s performance, highlighting the aspects that could lead to optimal modeling of cross-modal contextual associations.

## introduction social media has become a mainstream communication medium for the masses, redefining how we interact within society. the information shared on social media has diverse forms, like text, audio, and visual messages, or their combinations thereof. a meme is a typical example of such social media artifact that is usually disseminated with the flair of sarcasm or humor. while memes facilitate convenient means for propagating complex social, cultural, or political ideas via visual-linguistic semiotics, they often abstract away the contextual de-meme source: redditr/historymemes ## related work this section briefly discusses relevant studies on meme analysis that primarily attempt to capture a meme's affective aspects, such as hostility and emotions. besides these, we also review other popular tasks to suitably position our work alongside different related research dimensions being explored. meme analysis: several shared tasks have been organized lately, a recent one on detecting heroes, villains, and victims from memes @xcite , which was later followed-up via an external knowledge based approach in @xcite and further extended towards generating explanations in @xcite . other similar initiatives include troll meme classification @xcite and memeemotion analysis via their sentiments, types and intensity prediction @xcite . notably, hateful meme detection was introduced by @xcite and later followed-up by @xcite . significant interest was garnered as a result of these, wherein various models were developed. a few efforts included fine-tuning visual bert @xcite , and uniter @xcite , along with using detectron-based representations @xcite for hateful meme detection. on the other hand, there were systematic efforts involving unified and dual-stream encoders using transformers @xcite , vil-bert, vlp, uniter @xcite @xcite , and lxmert @xcite for dual-stream ensembling. besides these, other tasks addressed anti-semitism @xcite , propaganda techniques @xcite , harmfulness @xcite , and harmful targets in memes @xcite . visual question answering (vqa): early prominent work on vqa with a framework encouraging open-ended questions and candidate answers was done by @xcite . since then, there have been multiple variations observed. @xcite classified the answers by jointly representing images and questions. others followed by examining cross-modal interactions via attention types not restricted to co/soft/hard-attention mechanisms @xcite @xcite , effectively learning the explicit correlations between question tokens and localised image regions. notably, there was a series of attempts toward incorporating common-sense reasoning @xcite @xcite . many of these studies also leveraged information from external knowledge bases for addressing vqa tasks. general models like updn @xcite and lxmert @xcite explicitly leverage non-linear transformations and transformers for the vqa task, while others like lmh @xcite and ssl @xcite addressed the critical language priors constraining the vqa performances, albeit with marginal enhancements. ## mcc: meme context corpus due to the scarcity of publicly-available large-scale datasets that capture memes and their contextual information, we build a new dataset, mcc (meme context corpus). the overall dataset curation was conducted in three stages: (i) meme collection, (ii) content document curation, and (iii) dataset annotation. these stages are detailed in the remaining section. ## methodology in this section, we describe our proposed model, mime. it takes a meme (an image with overlaid text) and a related context as inputs and outputs a sequence of labels indicating whether the context's constituting evidence sentences, either in part or collectively, explain the given meme or not. as depicted in fig. 2 , mime consists of a text encoder to encode the context and a multimodal encoder to encode the meme (image and text). to address the complex abstraction requirements, we design a knowledge-enriched meme encoder (kme) that augments the joint multimodal representation of the meme with external common-sense knowledge via a gating mechanism. on the other hand, we use a pre-trained bert model to encode the sentences from the candidate context. we then set up a meme-aware transformer (mat) to integrate meme-based information into the context representation for designing a multilayered contextual-enrichment pipeline. next, we design a meme-aware lstm (ma-lstm) that sequentially processes the context representations conditioned upon the meme-based representation. lastly, we concatenate the last hidden context representations from ma-lstm and the meme representation and use this jointly-contextualized meme representation for evidence detection. below, we describe each component of mime in detail. context representation: given a related context, c consisting of sentences [c 1 , c 2 ...c n ], we encode each sentence in c individually using a pre-trained bert encoder, and the pooled output corresponding to the [cls] token is used as the context representation. finally, we concatenate the individual sentence representation to get a unified context representation h c 2 r n⇥d , with a total of n sentences. knowledge-enriched meme encoder: since memes encapsulate the complex interplay of linguistic elements in a contextualized setting, it is necessary to facilitate a primary understanding of linguistic abstraction besides factual knowledge. in our scenario, the required contextual mapping is implicitly facilitated across the contents of the meme and context documents. therefore, to supplement the feature integration with the required common sense knowledge, we employ conceptnet @xcite : a semantic network designed to help machines comprehend the meanings and semantic relations of the words and specific facts people use. using a pre-trained gcn, trained using concept-net, we aim to incorporate the semantic characteristics by extracting the averaged gcn-computed representations corresponding to the meme's text. in this way, the representations obtained are common sense-enriched and are further integrated with the rest of the proposed solution. to incorporate external knowledge, we use con-ceptnet @xcite knowledge graph (kg) as a source of external commonsense knowledge. to take full advantage of the kg and at the same time to avoid the query computation cost, we use the last layer from a pre-trained graph convolutional network (gcn), trained over conceptnet @xcite . we first encode meme m by passing the meme image m i and the meme text m t 10 to an empiri-cally designated pre-trained mmbt model @xcite , to obtain a multimodal representation of the meme h m 2 r d . next, to get the external knowledge representation, we obtain the gcn node representation corresponding to the words in the meme text m t . this is followed by average-pooling these embeddings to obtain the unified knowledge representation h k 2 r d . to learn a knowledge-enriched meme representation ĥm , we design a gated multimodal fusion (gmf) block. as part of this, we employ a meme gate (g m ) and the knowledge gate (g k ) to modulate and fuse the corresponding representations. here, w m and w k 2 r 2d⇥d are trainable parameters. meme-aware transformer: a conventional transformer encoder @xcite ) uses self-attention, which facilitates the learning of the inter-token contextual semantics. however, it does not consider any additional contextual information helpful in generating the query, key, and value representations. inspired by the context-aware selfattention proposed by @xcite , in which the authors proposed several ways to incorporate global, deep, and deep-global contexts while computing self-attention over embedded textual tokens, we propose a meme-aware multi-headed attention (mha). this facilitates the integration of multimodal meme information while computing the selfattention over context representations. we call the resulting encoder a meme-aware transformer (mat) encoder, which is aimed at computing the cross-modal affinity for h c , conditioned upon the knowledge-enriched meme representation ĥm . conventional self-attention uses query, key, and value vectors from the same modality. in contrast, as part of meme-aware mha, we first generate the key and the value vectors conditioned upon the meme information and then use these vectors via conventional multi-headed attention-based aggregation. we elaborate on the process below. given the context representation h c , we first calculate the conventional query, key, and value vectors q, k, v 2 r n⇥d , respectively as given below: here, n is the maximum sequence length, d is the embedding dimension, and w q , w k , and w v 2 r d⇥d are learnable parameters. we then generate new key and value vectors k and v , respectively, which are conditioned on the meme representation ĥm 2 r 1⇥d (broadcasted corresponding to the context size). we use a gating parameter 2 r n⇥1 to regulate the memetic and contextual interaction. here, u k and u v constitute learnable parameters. we learn the parameters k and v using a sigmoid based gating mechanism instead of treating them as hyperparameters as follows: finally, we use the query vector q against k and v , conditioned on the meme information in a conventional scaled dot-product-based attention. this is extrapolated via multi-headed attention to materialize the meme-aware transformer (mat) encoder, which yields meme-aware context representations h c/m 2 r n⇥d . meme-aware lstm: prior studies have indicated that including a recurrent neural network such as an lstm with a transformer encoder like bert is advantageous. rather than directly using a standard lstm in mime, we aim to incorporate the meme information into sequential recurrencebased learning. towards this objective, we introduce meme-aware lstm (ma-lstm) in mime. ma-lstm is a recurrent neural network inspired by @xcite that can incorporate the meme representation ĥm while computing cells and hidden states. the gating mechanism in ma-lstm allows it to assess how much information it needs to consider from the hidden states of the enriched context and meme representations, h c/m and ĥm , respectively. fig. 2 shows the architecture of ma-lstm. we elaborate on the working of the ma-lstm cell below. it takes as input the previous cell states c t 1 , previous hidden representation h t 1 , current cell input h ct , and an additional meme representation ĥm . besides the conventional steps involved for the computation of input, forget, output and gate values w.r.t the input h ct , the input and the gate values are also computed w.r.t the additional input ĥm . the final cell state and the hidden state outputs are obtained as follows: the hidden states from each time step are then concatenated to produce the unified context representation ĥ c/m 2 r n⇥d . prediction and training objective: finally, we concatenate ĥm and ĥ c/m to obtain a joint contextmeme representation, which we then pass through a feed-forward layer to obtain the final classification. the model outputs the likelihood of a sentence being valid evidence for a given meme. we use the cross-entropy loss to optimize our model. ## baseline models we experiment with various unimodal and multimodal encoders for systematically encoding memes and context representations to establish comparative baselines. the details are presented below. unimodal baselines: • bert @xcite : to obtain text-based unimodal meme representation. ## experimental results this section presents the results (averaged over five independent runs) on our thematically diversified test-set and performs a comparison, followed by qualitative and error analysis. for comparison, we use the following standard metrics -accuracy the last row shows the absolute improvement of mime over mmbt (the best baseline). e-f: early fusion and v-bert: visualbert. ## conclusion this work proposed a new task -memex that aims to identify evidence from a given context to explain the meme. to support this task, we also curated mcc, a novel manually-annotated multimodal dataset encompassing a broad range of topics. after that, we benchmarked mcc on several competitive systems and proposed mime, a novel modeling framework that utilizes knowledge-enriched meme representation and integrates it with context via a unique multi-layered fusion mechanism. the empirical examination and an extensive ablation study suggested the efficacy of mime and its constituents. we then analyzed mime's correct contextual mapping heuristics, juxtaposed with its limitations, suggesting the possible scope of improvement.