Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis (2022)

Abstract: Existing self-supervised learning strategies are constrained to either a limited set of objectives or generic downstream tasks that predominantly target uni-modal applications. This has isolated progress for imperative multi-modal applications that are diverse in terms of complexity and domain-affinity, such as meme analysis. Here, we introduce two self-supervised pre-training methods, namely Ext-PIE-Net and MM-SimCLR that (i) employ off-the-shelf multi-modal hate-speech data during pre-training and (ii) perform self-supervised learning by incorporating multiple specialized pretext tasks, effectively catering to the required complex multi-modal representation learning for meme analysis. We experiment with different self-supervision strategies, including potential variants that could help learn rich cross-modality representations and evaluate using popular linear probing on the Hateful Memes task. The proposed solutions strongly compete with the fully supervised baseline via label-efficient training while distinctly outperforming them on all three tasks of the Memotion challenge with 0.18%, 23.64%, and 0.93% performance gain, respectively. Further, we demonstrate the generalizability of the proposed solutions by reporting competitive performance on the HarMeme task. Finally, we empirically establish the quality of the learned representations by analyzing task-specific learning, using fewer labeled training samples, and arguing that the complexity of the self-supervision strategy and downstream task at hand are correlated. Our efforts highlight the requirement of better multi-modal self-supervision methods involving specialized pretext tasks for efficient fine-tuning and generalizable performance.

## introduction the overwhelming scale of digital mutation constantly transpiring over the web is "creating the illusion of reality, addressing the viewer, and representing a convoluted space" @xcite . almost every social activity affects or is affected by an online entity, sometimes even disturbing social harmony, influenced by a prominent surge of multimodal harmful, abusive and hateful online content. therefore, it is imperative to explore solutions towards automatic mediation of online activities that pre-dominantly involve multi-modality. recently, there has been a defining resurgence of advancements in multi-modal ai, albeit slowly. existing self-supervision strategies for visuallinguistic applications involve different pretext tasks like masked language modeling (mlm) @xcite , masked region modeling (mrm) @xcite , word-region alignment (wra) @xcite , and image-text matching (itm) @xcite , which inherently presume visuallinguistic grounding @xcite . as a consequence, the large-scale datasets like ms coco @xcite , conceptual captions (cc) @xcite , wikipedia-based image text (wit) @xcite and laion-400m @xcite , curated towards the required pre-training, are either mostly generic in nature or represent a greater degree of visual-semantic association between the image and text pairs. moreover, the required multi-modal datasets are rather challenging to create, as they often require multidimensional and fine-grained manual annotations for a large volume of multi-modal data. these frameworks have demonstrated impressive pre-training schemes for addressing downstream multi-modal tasks like visual question answering (vqa), image captioning (ic), visual commonsense reasoning (vcr), etc. @xcite . still, there is significant room for improvement in terms of their generalizability. for instance, besides masked language modelling (mlm), state-of-the-art multi-modal models like visual bert, vilbert and lxmert are pretrained wrt pretext tasks like sentence-image prediction @xcite , masked multi-modal learning, multi-modal alignment prediction @xcite and detected-label classification @xcite , which presume aspects like availability of multiple semantically grounded sentences corresponding to an image and visual-semantic object and pixel-level annotations for the images. these requirements constrain modeling aspects for multi-modal content like memes. although such approaches address the issue of scale and crossmodal alignment in terms of common-sense reasoning extremely well, they tend to fall short on performance for complex multi-modal tasks like meme analysis @xcite . this is because memes do not represent strong visual-linguistic grounding and solicit sophisticated multi-modal fusion along with contextual knowledge integration. this paper presents the design and evaluation of efficient multi-modal frameworks that do not rely upon large-scale dataset curation and annotation and can be pre-trained using the datasets from the wild. also, the pre-training employed is optimally designed toward learning enriched multimodal representations, which can be further used for addressing downstream tasks like meme analysis in a label-efficient manner. our contributions, as enlisted below, are three-fold: multi-modal pre-training: recently, @xcite proposed a simple yet effective multi-modal system with specialized convolution layers at the beginning of the encoder and a textual decoder as a follow-up. other recent similar works include dall-e @xcite , a zero-shot, generative scalable transformer that models multi-modal information in an auto-regressive manner and is conditioned on a textual query. this is followed by clip, a contrastive learning-based model @xcite , which is pre-trained on 400 million image-text pairs collected from different webbased resources. the primary objective of such efforts is to learn multi-modal embedding space jointly. however, the datasets used to pre-train are too generic to capture complex semantics. in this work, we intend to examine such constraints and their impact on the performance of multi-modal systems. studies on memes: although the recent past has witnessed an overwhelming amount of research related to memes, especially for topics like online hate, harm, offense, abuse, etc. @xcite , still, there are a wide array of meme related tasks, that are yet to be addressed. kolawole (2015) explored the classification task on a small dataset and with a linear svm on lowlevel descriptors, leveraging only visual information. significant efforts have been invested towards meme generation by representing the meme image and the catchphrase in the same vector space using a deep neural network @xcite , leveraging pre-trained inception-v3 network-based feature extraction. this was further explored in @xcite for caption generation and rule-based classification. the human assessment in this study outperformed random choices. the quality, however, was below-par as compared to human-produced memes. efforts are solicited wherein richer and more meaningful content modeling is achieved towards solving tasks that conventional multi-modal approaches cannot. ## dataset pretraining: to address generalizability towards an array of such topics, we employ the mmhs150k dataset @xcite as our primary data source for pre-training our proposed systems. it consists of 150k multi-modal (images + text) tweets spanning over four hate-inclined topics racism, sexism, homophobia, and religious extremism. moreover, the images in the dataset represent diversity with the presence of memes, morphed images, satirical art, etc. besides this, to ensure that our pre-training dataset reasonably represents the content type we would evaluate as part of downstream tasks, we also add the memes from the training split of the facebook's hateful memes dataset @xcite , that we reserve exclusively for our pre-training. training and evaluation: we employ three datasets (hateful memes, harm-p, and memotion) and five different tasks (hate detection, harmfulness detection, sentiment analysis, emotion classification, and emotion class quantification) to demonstrate the efficacy of our proposed approaches. the harm-p dataset belongs to the harmeme task @xcite and consists of 3552 memes annotated with two labelsharmful or not-harmful. the memotion dataset @xcite has approx. 8k memes and defines three subtasks foot_2 sentiment analysis (positive/negative), emotion classification (humour/sarcasm/offense/motivational), and emotion class quantification (slightly/mildly/very). although these datasets are based on memes or multimodal content, their objectives are different and have varying complexities. 3 . we leverage a dataset that represents the raw, unprocessed large-scale corpus of multi-modal information, specifically emphasizing different types of hate speech. we acknowledge that a labeling scheme initially accompanies the dataset (mmhs150k). however, we do not utilize that information either during the pre-training stage or during the task-specific fine-tuning stage. this is also represented in the form of proposed loss functions, which do not utilize source data labels but solely rely on the intermediate neural representations, hence self-supervised. also, the underlying presumption for utilizing such a dataset (mmhs150k) in a self-supervised way is based on the fact that the original dataset owners collected it using a pre-defined set of database keywords @xcite , and this is all that one would need to do to obtain such a dataset at scale towards pre-training the models proposed. also, no explicit annotation process is required for pre-training mm-simclr and ext-pie-net. now, as for the taskspecificity, we already showcase the performances of the fully supervised systems that utilize finetuning of the models, pre-trained using a generic dataset. we propose the frameworks that, if pretrained using a "domain-oriented" dataset that can be easily obtained, without any special annotations, can quickly and in a label-efficient way adapt to related downstream tasks. ## proposed solution we propose two methods: mm-simclr and ext-pie-net, that utilize adaptations of popular contrastive and triplet loss formulations for learning multi-modal embedding space. proposed solutions also encapsulate specialized multi-modal pretext tasks suited toward joint multi-modal representation learning. before describing the proposed solutions, we first review the two-loss formulations below. ## impact of label-efficient supervision during fine-tuning towards assessing the label-efficient setup, we compare the performances over incremental supervision. we also analyze their temporal training behavior. as can be observed from fig. 2a , ext-pie-net converges efficiently to 0.3565 f1 score with just 10% (600) training samples, as compared to mm-simclr which converges to 0.3511 f1 score after learning from 50% (3000) of the labeled samples. this highlights the capacity of a sophisticated ssl regime to learn better representations for a complex setup for the sent task compared to a slightly simpler model mm-simclr. a similar pattern can be observed for emot task in fig. 2b . ext-pie-net is observed to achieve an overall better f1 score of 0.7547, which is better than mm-simclr and outperforms all other results. although the optimal performance of simclr is reasonably at-par or even better for sent and emot tasks compared to the baseline and the previous best results, there is barely any active convergence visible within the plots depicted in fig. 2 for it. this is obvious considering the incomplete information that an image-only based uni-modal system would learn for the downstream task. vse is observed to yield 3.02% and 7.98% improvement over the sl baseline. still, it fails to register an impressive performance compared to the increment of 12.52% and 17.52% for the two tasks, respectively, by vse++. these observations can also be correlated with the training performance (c.f. fig. 3 ), wherein the performance curves are depicted for a total of 100 epochs across four different label-efficiency scenarios. for primary assessment, we showcase smoothed curves overlaid on unsmoothed ones towards observing global and local trends. 6 fig. 3 presents a clear depiction of progressive learning for all the supervision configurations evaluated in case of ext-pie-net for the sent and emot tasks (c.f. fig. 3 ) is given. on the other hand, the training curves for mm-simclr show saturated learning for tasks sent and emot respectively (c.f. fig. 3 ). delineating on the performance trend observed in the emot-q task earlier, neither ext-pie-net nor simclr shows definite convergence, as we consider the incremental supervision depicted in fig. 2c . whereas, mm-simclr is observed to show stable, yet non-incremental growth in performance reporting the best overall f1 score of 0.3318 (c.f. for harmeme task, the incremental supervision (c.f. fig. 4 ) exhibits incremental performance with the increase in the amount of supervision during fine-tuning. notably, the final f1 score of 0.814 obtained by the mm-simclr model is on just 50 % (1510) of the actual training set. this demonstrates the efficacy and generalizability of the pre-training via strategies adopted in this work. also, the progressive convergence observed at 50% supervision, as shown in fig. 4 for mm-simclr, demonstrates the generalizability of the proposed approach. this also suggests the importance of having smaller architectures with sophisticated fusion strategies to solve the task at hand effectively. ## discussion the observations made from the results obtained for the downstream evaluation suggest interest-6 for further reference, unsmoothed training curves are also included and discussed separately in app. c. ing trends. since memotion dataset involves multi-class, multi-label and multi-level hierarchical granularity due to the natural distribution of such realistic dataset, either ensembling-based approaches are observed to yield better results or, there are strong variations observed in the performance trends across the three memotion tasks @xcite . the results reported as part of ## conclusion this work empirically examined various selfsupervision strategies to learn effective representations that help solve multiple multi-modal downstream tasks in a label-efficient setting. we propose two strategies for this -(i) mm-simclr: a multimodal contrastive loss formulation that factors in the loss terms for image modality and the multimodality in a joint manner, and (ii) ext-pie-net: a joint formulation of weighted modality-specific hinge loss terms, combined with the contrastive loss that is computed between a pair of representations, obtained using symmetric multi-modal fusion. extensive analysis over 2 datasets and 5 tasks demonstrate how domain-aware self-supervised pre-training, using a multi-modal dataset, that can be directly obtained from the wild (www) in raw form, can be leveraged to perform label-efficient multi-modal adaptation, leading to competitive, even superior performance gains for some scenarios. the performances observed for the proposed methods indicate task-dependent efficacies. mm-simclr being a lighter model is observed to perform better on emot-q and harmeme tasks, having a lower level of granularity to be modeled. whereas ext-pie-net performs better on sent and emot tasks, which require modeling a higher abstraction level for the target categories. despite exhibiting interesting performance within label-efficient evaluation settings, the objectives addressed in this work can further benefit from extensive analysis and evaluation towards obtaining a broader understanding of the generalizability of the proposed methodology.