F ig M emes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes (2022)

Abstract: Real-world politically-opinionated memes often rely on figurative language to cloak propaganda and radical ideas to help them spread. It is not only a scientific challenge to develop machine learning models to recognize them in memes, but also sociologically beneficial to understand hidden meanings at scale and raise awareness. These memes are fast-evolving (in both topics and visuals) and it remains unclear whether current multimodal machine learning models are robust to such distribution shifts. To enable future research into this area, we first present FigMemes, a dataset for figurative language classification in politically-opinionated memes. We evaluate the performance of state-of-the-art unimodal and multimodal models and provide comprehensive benchmark results. The key contributions of this proposed dataset include annotations of six commonly used types of figurative language in politically-opinionated memes, and a wide range of topics and visual styles.We also provide analyses on the ability of multimodal models to generalize across distribution shifts in memes. Our dataset poses unique machine learning challenges and our results show that current models have significant room for improvement in both performance and robustness to distribution shifts.

## introduction memes are transmittable units of culture that evolve fast @xcite , which use images and/or text to convey opinions. figurative language foot_0 such as metaphors or sarcasm is often used in memes (see figure 1 ) to persuade, enhance the impact of ideas, and help these ideas spread @xcite . this is especially true for internet memes, which rely on figurative language to spread propaganda @xcite , and support violent, discriminatory or radical ideology @xcite . for instance, the transmission of dehumanized metaphors for women may help perpetuate negative beliefs about women's roles in modern society @xcite , and the usage of rhetorical devices can attract new audiences to radicalized groups that fight against refugees @xcite . figurative language classification has been challenging in machine learning @xcite @xcite as the task requires understanding world knowledge and commonsense. despite its difficulty, the task of developing machine learning models for classifying figurative language in politically-opinionated memes (memes that relate to politics or share views on controversial topics) can provide sociological benefits. these benefits include understanding hidden meanings, aiding humanities research, and raising awareness at scale. yet, there is a lack of appropriate datasets for studying figurative language in these memes (see as politically-opinionated memes evolve quickly based on the latest news and cultural references, it also remains unclear whether current multimodal models for memes classification are robust to such distribution shifts. for example, a model trained in early 2019 may perform poorly when tested on memes referencing covid, or one trained on photo-like memes may perform poorly when tested on hand sketches. earlier work such as @xcite has neglected this problem. despite recent growth in classification tasks for memes @xcite @xcite , there has been limited work on assessing a model's ability to: 1) identify and understand figurative language in politically-opinionated memes, 2) generalize to a data distribution different from the training data, e.g., across different topics or visual categories. to bridge these research gaps, we first present a novel and challenging multi-label memes dataset called figmemes (figurative language identification in memes). our proposed task is to identify the type of (one or more) figurative language used in a meme. then, we conduct analyses by splitting the dataset to create distribution shifts (within each modality, e.g. text or image), aimed at assessing generalization of memes classification models. the figmemes dataset contains 5141 politicallyopinionated memes collected from the 4chan /pol/ (the politically incorrect) board, posted between january 2017 and december 2021. unlike previous work, we do not build the dataset by hand-picking topics using keywords (hence less inductive biases). the dataset covers a wide range of topics and visual categories. we provide comprehensive benchmark results using several state-of-the-art unimodal and multimodal models. the best multimodal model achieves an average f1 score of 46.69% on the proposed dataset, which highlights the difficulty of this task and significant room for improvement in future research. our further analyses show that 1) making use of external knowledge, understanding commonsense, and reasoning are important for figurative language understanding in memes, 2) further development of multimodal models for memes that are robust to distribution shifts and datasets for testing such models are needed. in summary, our contributions are: ## distribution shifts in memes and analysis settings to evaluate a machine learning model's ability to generalize across distribution shifts, we simulate distribution shifts in individual modalities (i.e., image or text) separately. the images and extracted text are grouped into different categories (visual categories and topic clusters) and used to create different distribution-shifted evaluation settings. ## baseline models we provide results of baseline models for the proposed multimodal multi-label classification task. our selection covers state-of-the-art unimodal textand image-only models, and multimodal models with and without multimodal pre-training. text-only: we use bert @xcite we concatenated the text and image representations from clip for classification (clip-mm). we experimented with both full fine-tuning (-ft) and linear probing (-lp) with the rn50x4 model. vinvl @xcite ) is a multimodal transformer that utilizes image features from a faster r-cnn-based (ren et al., 2015) objectdetection model. 13 bert+clip is a multimodal transformer initialized with bert weights. for the image input, we used the penultimate pre-pooling feature maps from clip (rn50x4) and applied an adaptive maxpooling to reduce the feature size to 6 × 6. these features were flattened and concatenated with input text tokens as input to the transformer, with the cnn component frozen during training. clip-mm-ood (distribution shifts analysis only). lp-ft @xcite is a recently developed training strategy for improved out-ofdistribution generalization in image classification tasks. this method consists of a 2-step tuning process that performs linear probing followed by finetuning. we applied this training method to clip and refer to it as clip-mm-ood. we train all models with binary cross-entropy loss (each label was treated as a separate binary classification task) and re-weighted all positive instances to address class imbalances. the detailed hyperparameters and discussions of hyperparameter search methods are given in appendix c. 13 we used the top 36 regions as input for the model. to evaluate the performance of the models, we report averaged (macro) f1 scores (averaged over 4 runs) in our experiments. the averaged f1 is computed by taking the classification task as 6 binary classification tasks. ## baseline results and discussions overall, text-only models performed poorly on this task, with the worst f1 scores in the anthropomorphism/zoomorphism category. vision is more important than expected. notably, image-only models performed significantly better than text-only models in all categories, especially in allusion, exaggerations, anthropomorphism/zoomorphism and contrast. for example, both convnext-ft and clip-cnn-ft visiononly models achieved significantly higher f1 scores in exaggerations (7.96 points and 12.92 points higher) and anthropomorphism/zoomorphism (12.14 points and 26.62 points higher) than the best text-only model. we hypothesize that both of these categories may be easy to identify with a vision module, as in our data the exaggerations category contains a sizable number of memes with visual reference to minority stereotypes. similarly, the anthropomorphism/zoomorphism category often contains memes that use animal characters in an image. consequently, better image representations would be critical to perform well on these memes. in addition, a better language model (deberta) does not seem to improve results significantly. clip-based models achieve better results. in general, clip-based models achieved better results than other models for both unimodal (image only) and multimodal inputs, with clip-cnn-lp (unimodal) achieving the best f1 score. the clip model being contrastively trained over a large corpus from the internet may explain the superior results. it is also interesting to note that the bert+clip model outperformed the multimodal pre-trained vinvl model (which uses bert weights). this could further indicate that clip provides a better image representation than objectdetector-based features for meme-like tasks. ## distribution shift analysis as memes evolve fast based on the latest news and cultural references, in this section we aim to analyze models trained on different memes classification tasks to understand 1) how models perform under data distribution shifts on figmemes (proposed); and 2) do the findings also translate to other memes datasets? we use three previously mentioned datasets ( §4.1) in this analysis with the following multimodal models: vinvl, bert+clip, clip-mm-ft, and clip-mm-ood. in general, we found the models perform better when trained and evaluated on data from the same distribution (id) than evaluated on data that are ood, see figure 3 and figure 4 . however, there are anomalies between datasets. we further observed that clip-mm-ood does not close the gaps between id and ood evaluation results as it did in image-only tasks @xcite , which vinvl bert+c. c.-mm-ft c.-mm-ood model 0 10 20 30 40 f1 id artistic ood artistic id real ood real (a) figmemes vinvl bert+c. c.-mm-ft c.-mm-ood model 0 10 20 30 40 50 f1 id artistic ood artistic id real ood real (b) memotion 2 vinvl bert+c. c.-mm-ft c.-mm-ood model 0 10 20 30 40 50 f1 id artistic ood artistic id real ood real (c) mami figure 4: image-based in-distribution versus out-of-distribution evaluation results. id artistic means train and test on artistic visual style data. ood artistic means train on artistic and test on all except artistic style data. similar for id/ood real. c. is clip. models trained with figmemes are less sensitive to text-based distribution shifts. we do not observe any significant drop in averaged f1 scores for figmemes between id and ood evaluation (figure 3a ) across all evaluated multimodal models. in fact, poor ood test results mainly came from the ood test set without any text (see appendix e.2 for results per ood test set). this is important since an ideal multimodal model should not only be able to utilize features from each input modality, but also maintain performance on a task when an input mode is missing (and the nature of the task is unchanged). figure 4a shows that models trained on fig-memes are more sensitive to visual category shifts, which aligns with our previous findings. we also found that training with memes that are artistic in style creates a harder transfer scenario. id/ood performance gaps are dataset/taskdependent. contrary to findings on figmemes, models evaluated on the memotion 2 dataset do not show much difference between id/ood results in the visual category shifts. we suspect that this is due to the extreme imbalance of the dataset. 14 on the other hand, mami (misogyny classification) is quite sensitive to distribution shifts in both text and images (with the largest drop of 14.11 and 20.09 in f1, respectively). 15 the most prominent drop is when training on memes in the real style, but testing on memes from other styles (especially when tested on memes in the artistic style, see based on these results, we recommend that future work on multimodal datasets perform analyses on distribution shifts and identify potential failure modes, as well as provide splits for evaluating these failures. datasets should help facilitate the development of machine learning models that are robust to distribution shifts, which also have implications for their real-world applicability. ## conclusions in this paper, we introduce a novel multi-label memes dataset -figmemes, for classifying figurative language in politically-opinionated memes. the dataset contains annotated memes covering a wide range of figurative language and topics over a period of 5 years. we provide comprehensive benchmarks using state-of-the-art unimodal and multimodal machine learning models and show that the dataset poses unique challenges to existing models. our error analysis and distribution shifts analysis reveal the limitations of state-of-the-art models and the need to develop robust models that understand world knowledge and commonsense. our dataset presents unique challenges and opportunities for future research in robust multimodal models that can benefit computer science research and humanities research in political sciences and sociology.