Hate- CLIP per: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features (2022)

Abstract: Hateful memes are a growing menace on social media. While the image and its corresponding text in a meme are related, they do not necessarily convey the same meaning when viewed individually. Hence, detecting hateful memes requires careful consideration of both visual and textual information. Multimodal pre-training can be beneficial for this task because it effectively captures the relationship between the image and the text by representing them in a similar feature space. Furthermore, it is essential to model the interactions between the image and text features through intermediate fusion. Most existing methods either employ multimodal pre-training or intermediate fusion, but not both. In this work, we propose the Hate-CLIPper architecture, which explicitly models the cross-modal interactions between the image and text representations obtained using Contrastive Language-Image Pre-training (CLIP) encoders via a feature interaction matrix (FIM). A simple classifier based on the FIM representation is able to achieve state-of-the-art performance on the Hateful Memes Challenge (HMC) dataset with an AUROC of 85.8, which even surpasses the human performance of 82.65. Experiments on other meme datasets such as Propaganda Memes and TamilMemes also demonstrate the generalizability of the proposed approach. Finally, we analyze the interpretability of the FIM representation and show that cross-modal interactions can indeed facilitate the learning of meaningful concepts. The code for this work is available at https://github.com/gokulkarthik/hateclipper

## introduction multimodal memes, which can be narrowly defined as images overlaid with text that spread from person to person, are a popular form of communication on social media @xcite . while most internet memes are harmless (and often humorous), some of them can represent hate speech. @xcite . while the memes on the left column are hateful, the ones in the middle are non-hateful image confounders, and those on the right are non-hateful text confounders. given the scale of the internet, it is impossible to manually detect such hateful memes and stop their spread. however, automated hateful meme detection is also challenging due to the multimodal nature of the problem. research on automated hateful meme detection has been recently spurred by the hateful memes challenge competition @xcite held at neurips 2020 with a focus on identifying multimodal hateful memes. the memes in this challenge were curated in such a way that only a combination of visual and textual information could succeed. this was achieved by creating non-hateful "confounder" memes by changing only the image or text in the hateful memes, as shown in figure 1 . in these examples, an image/text can be harmless or hateful depending on subtle contextual information contained in the other modality. thus, multimodal (image and text) machine learning (ml) models are a prerequisite to achieve robust and accurate detection of such hateful memes. in a multimodal system, the fusion of different modalities can occur at various levels. in early fusion schemes @xcite @xcite , the raw inputs (e.g., image and text) are combined and a joint representation of both modalities is learned. in contrast, late fusion approaches @xcite , learn end-to-end models for each modality and combine their outputs. however, both these approaches are not appropriate for hateful memes because the text in a meme does not play the role of an image caption. early fusion schemes are designed for tasks such as captioning and visual question answering, where there is a strong underlying assumption that the associated text describes the contents of the image. hateful memes violate this assumption because the text and image may imply different things. we believe that this phenomenon makes the early fusion schemes non-optimal for hateful meme classification. in the example shown in the first row of figure 1 , the left meme is hateful because of the interaction between the image feature "skunk" and the text feature "you" in the context of the text feature "smell". on the other hand, the middle meme is non-hateful as "skunk" got replaced by "rose" and the right meme is also non-hateful because "you" got replaced by "skunk". thus, the image and text features are related via common attribute(s). since modeling such relationships is easier in the feature space, an intermediate fusion of image and text features is more suitable for hateful meme classification. the ability to model relationships in the feature space also depends on the nature of the extracted image and text features. existing intermediate fusion methods such as concatbert @xcite pretrain the image and text encoders independently in a unimodal fashion. this could result in the divergent image and text feature spaces, making it difficult to learn any relationship between them. thus, there is a need to "align" the image and text features through multimodal pretraining. moreover, hateful meme detection requires faithful characterization of interactions between fine-grained image and text attributes. towards achieving this goal, we make the following contributions in this paper: ## related work the hateful memes challenge (hmc) competition @xcite established a benchmark dataset for hateful meme detection and evaluated the performance of humans as well as unimodal and multimodal ml models. the unimodal models in the hmc competition include: image-grid, based on resnet-152 (he et al., 2016) features; image-region, based on faster rcnn (ren et al., 2017) features; and text-bert, based on the original bert (devlin et al., 2018) features. the multimodal models include: concat bert, which uses a multilayer perceptron classifier based on the concatenated resnet-152 (image) and the original bert (text) features; mmbt (kiela et al., 2019) models, with image-grid and image-region features; vilbert @xcite ; and visual bert @xcite . a late fusion approach based on the mean of image-region and text-bert output scores was also considered. all the above models were benchmarked on the "test seen" split based on the area under the receiver operating characteristic curve (auroc) @xcite ) metric. the results indicate a large performance gap between humans (auroc of 82.65 foot_0 ) and the best baseline using visual bert (auroc of 75.44). the challenge report @xcite , which was released after the end of the competition, showed that all the top five submissions @xcite @xcite ; love the way you smell today clip image encoder clip text encoder p i1 p i2 ... p in p t1 p t2 ... p tn p t1 p i1 ... ... p t1 p in ... ... ... ... ... ... ... ... p tn p i1 ... ... p tn p in image projection layers text projection layers p t1 p i1 p t1 p i2 ... p t1 p in p t2 p i1 ... ... p tn p in pre output layers output layer frozen clip image encoder trainable trainable input flatten explicit cross-modal interaction f i p i p t f t feature interaction matrix (fim) align-fusion variant cross-fusion variant lippe et al., 2020; sandulescu, 2020) achieve better auroc than the baseline methods. this improvement was achieved primarily through the use of ensemble models and/or external data and additional input features. for example, zhu (2020) used a diverse ensemble of vl-bert @xcite , uniter-itm @xcite , villa-itm @xcite and ernie-vil @xcite with additional information about entity, race, and gender extracted using cloud apis and other models. this method achieved the best auroc of 84.50 on the "test unseen" split. @xcite extended the hmc dataset with fine-grained labels for protected category and attack type. protected category labels include race, disability, religion, nationality, sex, and empty protected category. attack types were labeled as contempt, mocking, inferiority, slur, exclusion, dehumanizing, inciting violence, and empty attack. zia et al. (2021) used clip @xcite encoders to obtain image and text features, which were simply concatenated and passed to a logistic regression classifier. separate classification models were learned for the two multilabel classification tasks -protected categories and attack types. momenta @xcite also uses representations generated from clip encoders, but augments them with the additional feature representations of objects and faces using vgg-19 (simonyan and zisserman, 2014) and text attributes using distilbert @xcite . furthermore, momenta uses cross-modality attention fusion (cmaf), which concatenates text and image features (weighted by their respective attention scores) and learns a cross-modal weight matrix to further modulate the concatenated features. momenta reports performance only on the harmeme dataset @xcite . although bilinear pooling (tenenbaum and freeman, 2000) (outer product) of different feature spaces has shown improvements for different multimodal tasks @xcite @xcite , it is not well experimented with multimodally pretrained (aligned feature space) encoders like clip or for the hateful meme classification task. ## methodology our objective is to develop a simple end-to-end model for hateful meme classification that avoids the need for sophisticated ensemble approaches and any external data or labels. we hypothesize that there is sufficiently rich information available in the clip visual and text representations and the missing link is the failure to model the interactions between these feature spaces adequately. hence, we propose the hate-clipper architecture as shown in figure 2 . in the proposed hate-clipper architecture, the image i and text t are passed through pretrained clip image and text encoders (whose weights are frozen after pretraining) to obtain unimodal features f i and f t , respectively. we use trainable projection layers: note that clip is pre-trained using contrastive learning on 400 million image-text pairs from the internet. this multimodal pretraining encourages similarity between the feature spaces of the image and its corresponding text caption. however, in the dataset used for pretraining, the image and text pairs usually convey the same meaning, which is not always the case in hateful memes. therefore, to better model the semantic relationship between the image and text feature spaces of memes, we further add trainable projection layers at the output of the clip image and text encoders. the main purpose of projection layers is not to ensure same dimensionality for both text and image embeddings, but to achieve better alignment between the text and image spaces. while clip is already trained to align the two spaces at a high-level, this needs to be further finetuned for the specific task/dataset at hand. instead of finetuning the entire clip model using small datasets, it is more prudent to add projection layers and only learn these projection layers based on the given datasets. these projection layers map the unimodal image and text features f i and f t to the corresponding image projection p i and text projection p t , respectively. the projection layers are designed such that both p i and p t have the same dimensionality n. the use of customized trainable projection layers after the clip encoders is one of the key differences between the proposed architecture and the one used in @xcite . ## interpretability to determine the interpretability of the feature interaction matrix (fim), we employ the following simple approach. first, we compute a n 2 -dimensional binary trigger vector for each hateful meme, where a value of 1 indicates that the specific element in the fim r is salient for determining if the given input belongs to the hateful class. these trigger vectors are then clustered into groups using a kmeans clustering algorithm. we manually examine these clusters to determine if most samples within a cluster have a common underlying pattern. to compute the trigger vector, we first reset the feature interaction matrix r to zero values and evaluate the gradient of the loss function for the non-hateful class with respect to r. let @xmath0. each element in the d matrix represents the direction (positive/negative) of the corresponding element in r matrix towards the hateful class. we then bifeaturing hateful examples with all the original text in this place would be distasteful; hence the meme text is masked with the exception of few words that are required for discussion. however, the reader can choose to look at the non-censored memes in the appendix narize the d matrix by setting all elements in the top-20 and bottom 20 percentiles (based on magnitude) to value 1 and assigning 0 values to all the other elements. then, for each hateful meme (i, t) in the training set, we perform one forward pass through the hate-clipper and compute the memespecific fim r. again we binarize the r matrix by setting all elements in the top-10 and bottom-10 percentiles (based on magnitude) to value 1 and assigning 0 values to all the other elements. finally, the trigger matrix t is computed for each meme as the element-wise (hadamard) product of binarized d and r matrices, i.e., @xmath1. this trigger matrix is then flattened to obtain the trigger vector corresponding to a meme. we apply k-means clustering algorithm available in scikit-learn @xcite on the trigger vectors to group the hateful memes. samples from the resulting groups of memes are shown in figure 4 . from figure 4 , we observe that clusters 5, 7, and 11 contain memes related to the same concept. it is interesting to note that hate-clipper is able to produce similar features for the same concept expressed in different modalities. for example in cluster 5, which is characterized by the concept 'death', we can see some memes representing 'death' only in images (b) and other memes representing the same concept only in text (a, d, e). furthermore, note that the memes (f) and (g), under the same cluster, do not directly relate to death, but the meme texts could hint toward death related events (blow -> blast; popcorn sounds -> bullet sounds) in different contexts. with the clustered memes, we can also identify the positions in fim r, which get activated for the matching concepts. however, some of the clusters are ambiguous. for example, cluster 13 has memes from different concepts. also, when the clusters have less than 3 memes or greater than 10 memes, they exhibit greater diversity in terms of the underlying concepts and are not useful for the explanation. ## conclusion in this work, we emphasized the need for intermediate fusion and multimodal pretraining for hateful meme classification. we proposed a simple end-to-end architecture called hate-clipper using explicit cross-modal clip representations, which achieves the state-of-the-art performance quickly in 14 epochs with just 4 trainable layers (1 image projection, 1 text projection, 1 pre-output, and 1 output) in the hateful memes challenge dataset, moreover, our model does not require any additional input features like object bounding boxes, face detection, text attributes, etc. we also demonstrated similar performance in multi-label classification based on the propaganda memes dataset. finally, we performed preliminary studies to evaluate the interpretability of cross-modal interactions.