Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset (2021)

Abstract: Hateful memes pose a unique challenge for current machine learning systems because their message is derived from both text- and visual-modalities. To this effect, Facebook released the Hateful Memes Challenge, a dataset of memes with pre-extracted text captions, but it is unclear whether these synthetic examples generalize to ‘memes in the wild’. In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that ‘memes in the wild’ differ in two key aspects: 1) Captions must be extracted via OCR, injecting noise and diminishing performance of multimodal models, and 2) Memes are more diverse than ‘traditional memes’, including screenshots of conversations or text on a plain background. This paper thus serves as a reality-check for the current benchmark of hateful meme detection and its applicability for detecting real world hate.

## introduction hate speech is becoming increasingly difficult to monitor due to an increase in volume and diversification of type @xcite . to facilitate the development of multimodal hate detection algorithms, facebook introduced the hateful memes challenge, a dataset synthetically constructed by pairing text and images @xcite . crucially, a meme's hatefulness is determined by the combined meaning of image and text. the question of likeness between synthetically created content and naturally occurring memes is both an ethical and technical one: any features of this benchmark dataset which are not representative of reality will result in models potentially overfitting to 'clean' memes and generalizing poorly to memes in the wild. thus, we ask the question: how well do @xcite represent memes found in the real world? we use pinterest memes (pin) as our example of memes in the wild and explore differences across three aspects: to compare text modality content, we examine the most frequent n-grams and train a classifier to predict a meme's dataset membership based on its text. ## background the majority of hate speech research focuses on text, mostly from twitter (waseem and hovy, 2016; @xcite @xcite . text-based studies face challenges such as distinguishing hate speech from offensive speech @xcite and counter speech @xcite , as well as avoiding racial bias @xcite . some studies focus on multimodal forms of hate, such as sexist advertisements @xcite , youtube videos @xcite , and memes @xcite @xcite . while the hateful memes challenge @xcite encouraged innovative research on multimodal hate, many of the solutions may not generalize to detecting hateful memes at large. for example, the winning team zhong (2020) exploits a simple statistical bias resulting from the dataset generation process. while the original dataset has since been re-annotated with fine-grained labels regarding the target and type of hate @xcite , this paper focuses on the binary distinction of hate and non-hate. ## discussion the key difference in text modalities derives from the efficacy of the ocr extraction, where messier captions result in performance losses in text bert classification. this forms a critique of the way in which the hateful memes challenge is constructed, in which researchers are incentivized to rely on the pre-extracted text rather than using ocr; thus, the reported performance overestimates success in the real world. further, the challenge defines a meme as 'a traditional meme' but we question whether this definition is too narrow to encompass the diversity of real memes found in the wild, such as screenshots of text conversations. when comparing the performance of unimodal and multimodal models, we find multimodal mod-els have superior classification capabilities which may be because the combination of multiple modes create meaning beyond the text and image alone @xcite . for all three multimodal models (concat bert, late fusion, and mmbt-grid), the score for fb memes with ground truth captions is higher than that of fb memes with ocr extracted text, which in turn is higher than that of pin memes. finally, we note that clip's performance, for zero-shot and linear probing, surpasses the other models and is stable across both datasets. limitations despite presenting a preliminary investigation of the generalizability of the fb dataset to memes in the wild, this paper has several limitations. firstly, the errors introduced by ocr text extraction resulted in 'messy' captions for pin memes. this may explain why pin memes could be distinguished from fb memes by a naïve-bayes classifier using text alone. however, these errors demonstrate our key conclusion that the preextracted captions of fb memes are not representative of the appropriate pipelines which are required for real world hateful meme detection. secondly, our pin dataset relies on noisy labels of hate/non-hate based on keyword searches, but this chosen heuristic may not catch subtler forms of hate. further, user-defined labels introduce normative value judgements of whether something is 'offensive' versus 'funny', and such judgements may differ from how facebook's community standards define hate @xcite . in future work, we aim to annotate the pin dataset with multiple manual annotators for greater comparability to the fb dataset. these ground-truth annotations will allow us to pre-train models on pin memes and also assess transferability to fb memes.