Caption Enriched Samples for Improving Hateful Memes Detection (2021)

Abstract: The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.

## introduction multimodal transformers, including visual-bert @xcite , vilbert @xcite , and uniter @xcite are currently the state of the art methods in tasks that involve both text and images, such as visual question answering @xcite and image captioning @xcite . one particular example is that of the "hateful memes" challenge @xcite , in which the task is to classify whether a given meme is hateful or not. by construction (and similar to real-world memes), the classifier has to apply complex reasoning that involves a multimodal text-image analysis from both image and text. the prevalence of sarcasm adds another layer of difficulty to both machines and humans. @xcite results indicate that the human accuracy is only about 85%. recent state-of-the-art models perform considerably less accurately, achieving up to 64.73% accuracy. in this work, we adopt an off-the-shelf caption generator, and show that combining such captions as a third input (in addition to the image and the overlayed text) can effectively improve the performance of both multimodal and unimodal models. two techniques are used: continued pre-training of the baseline model with the additional augmented captions, and using the added information as part of the fine-tuned model used for the hateful memes task. ## related work transformer-based language models (lms) have shown significant performance gains in various natural language understanding tasks, such as textsimilarity and language inference @xcite . these models are usually pre-trained on the masked language model (mlm) objective followed by a task-specific finetuning process @xcite . recently, vision and language tasks have gained a lot of traction, where transformer-based multi-modality models were introduced, showing great promise in solving multimodality problems, such as visual reasoning @xcite , image caption generation @xcite and more. in @xcite , the authors introduced the hateful memes challenge -a dataset of 10,000 memes, each associated with an image and text. each meme sample image-text pair is binary labeled as hateful or not hateful. the dataset is split into 8,500 train memes, 500 and 1,000 validation and test samples, respectively. the test labels are not public and are used for leaderboard scoring of models. the dataset was artificially constructed based on real-world memes. to ensure that the dataset requires multimodal reasoning, image and text confounders were crafted to enrich the dataset with contrastive examples. confounders samples are memes that appear in two variations: same image with different texts that have different labels or vice versa (see fig. 1 ). in @xcite , the authors show that both unimodal models (such as bert @xcite ) and multimodal models (e.g. visualbert @xcite ) seem to produce relatively poor performance, that is significantly inferior to the human accuracy (reported to be 84.7%). this can be attributed to the task difficulty, which requires an advanced reasoning process, where some samples are found to also confuse humans. the task of caption generation has been shown to yield promising results in various works, such as the show and tell @xcite , show tell and attend @xcite , and more. among the different caption generator techniques, the neural image caption (nic) @xcite has been shown to yield promising results on the various caption tasks. the nic model is an encoderdecoder network, trained to generate captions from images. we use this generator to augment the hateful memes dataset, by associating each meme image-text pair with an additional auto-generated caption. in our work, we adopt the implementation of the ibm image caption generator 2 . @xcite show that continued pre-training of a pre-trained language model on a given dataset at hand improves the performance of the fine-tuning procedure. in a similar spirit, we demonstrate that for the hateful meme task, it can be beneficial to continue pre-training on text-pairs of meme text and augmented captions. ## method the hateful memes challenge is a binary classification problem in which each meme sample @xmath0, c) consists of an image in some image domain m and an associated text in a language domain c. in both cases, we employ an off-theshelf image-caption generator g : @xmath1, which receives an image from domain m and returns a caption from c. every sample thus becomes x * = (m, c, c * ), where c * = g(m). we next present the training strategy, termed caption enriched samples (ces), in either the unimodal or the multimodal settings. in the former, the network classifier considers only text input, and in the latter image-text inputs. ## results in this section, we evaluate and report the performance of ces applied to the following models: bert and roberta are the bert base @xcite and roberta base @xcite models, fine-tuned on the meme text using a standard fine-tuning approach. visualbert is another multimodal transformer that was pre-trained on image-text inputs, by aligning elements of the text and regions in the image through cross-attention and self-attention operations @xcite . in contrast to vilbert, this model employs a single-stream architecture, embedding image regions and language tokens through the same transformer blocks. for the image-caption generator g we employ the ibm maxcaption model @xcite for generating captions for all memes in the dataset. the length of the generated captions is at most 15 words (see fig. 2 ). in tab.1, we report the performance of the above models applied with and without our ces strategy. as can be seen, ces is shown to improve the performance of all models by a sizeable margin. specifically, for the bert and roberta unimodal models, we observe that ces improves the test performance by 8.6% and 10%, respectively. this can be attributed to the fact that in ces, the unimodal receives a textual representation of the image (the generated caption c * ), which is a proxy to the image itself. in the multimodal settings, we observe that ces improves the performance of vilbert and visualbert by 2.1% and 3.6%, respectively. a similar table reporting the accuracy scores can be found in the appendix. fig. 2 depicts two representative samples demonstrating the effectiveness of ces applied to the vi-sualbert model. as can be seen in the figure, the ces-visualbert model was able to correctly predict the right label for both images, while the baseline visualbert model was confused. ## summary detecting hateful memes is a challenging task that requires complex image-text reasoning. in this work, we introduce ces, a training approach for both multimodal and unimodal models that leverages an off-the-shelf image-caption generator to enhance the accuracy of hateful meme classifiers. our results indicate that augmented captions are highly beneficial for transformer-based models, which seem to be able to effectively attend between the augmented caption, meme text, and image.