DD - TIG at Constraint@ ACL 2022: Multimodal Understanding and Reasoning for Role Labeling of Entities in Hateful Memes (2022)

Abstract: The memes serve as an important tool in online communication, whereas some hateful memes endanger cyberspace by attacking certain people or subjects. Recent studies address hateful memes detection while further understanding of relationships of entities in memes remains unexplored. This paper presents our work at the Constraint@ACL2022 Shared Task: Hero, Villain and Victim: Dissecting harmful memes for semantic role labelling of entities. In particular, we propose our approach utilizing transformer-based multimodal models through a VCR method with data augmentation, continual pretraining, loss re-weighting, and ensemble learning. We describe the models used, the ways of preprocessing and experiments implementation. As a result, our best model achieves the Macro F1-score of 54.707 on the test set of this shared task.

## introduction memes are getting popular as a communication tool on social media platforms for expressions of opinions and emotions, conveying a subtle message through multimodal information from both images and texts. however, memes are increasingly abused to spread hate instigate social unrest and therefore seem to be a new form of expression of hate speech on online platforms @xcite . automatic hateful memes detection is difficult since it primarily requires context and external knowledge to understand online speech, which sometimes can be very short and contains nuanced meaning @xcite . a new type of challenging task has been introduced by the hateful memes challenge @xcite proposed by facebook ai to leverage machine learning models to address hateful memes detection problems, which can only be solved by joint reasoning and un-derstanding of visual and textual information @xcite . in previous studies, researchers focus on binary classification problems, labelling a meme as hateful or non-hateful based on image and text features @xcite . moreover, the relationships of entities in memes remain unexplored, and the task of role labelling of entities in hateful memes can be more sophisticated. the constraint@acl2022 shared task: hero, villain and victim: dissecting harmful memes for semantic role labelling of entities offers us a perspective on this issue @xcite . this task aims to promote the detection and classification of glorified, vilified or victimized entities within a meme. the shared dataset concerns memes from us politics domains and covid-19. covid-19-related online hostile content especially demands to be detected as early as possible after their appearance on social media. in this paper, we present our work on this task. specifically, mainstream multimodal models of transformer-based architecture are applied through a visual commonsense reasoning (vcr) method, with the leverage of continual pretraining to fit models with our dataset. then, data augmentation and loss re-weighting are implemented to improve the performance of models. the predictions from variant models are combined in a machine learning method to produce final results. ## related work hateful memes understanding and reasoning is a vision and language task. current state-of-theart vision-language machine learning models are based on the transformer architecture @xcite . multimodal models learn the joint visual and textual representations through selfsupervised learning that utilize large-scale unlabelled data to conduct auxiliary tasks @xcite , including masked language modelling based on randomly-masked sub-words, masked region prediction and image-text matching. among these models, there are two prevalent approaches: singlestream and dual-stream @xcite . in single-stream architecture, the representations of two modalities are learned by a single transformer encoder. particularly, the text embeddings @xmath0, w 2 , w 3 , • • • , w l } and image features @xmath1, o 2 , o 3 , • • • , o k } are concatenated together as @xmath2, added some special embeddings to indicate position and modalities, and fed into a transformer-based encoder. there are many implementations in singlestream models, such as visualbert @xcite , uniter @xcite , oscar @xcite . in dual-stream models, the image and text features are first sent to two independent encoders. then two features are separately fed into crossmodal transformer layers, where the query vectors are from one modality while the key and value vectors are from another. they are responsible for exchanging the information and aligning the semantics between the two modalities l and v . the formula of cross-modal transformer layers is represented as follows. following each cross-attention layer, there is also a layer computing the self-attention of each modality independently. features are combined at the end of the model. several dual-stream models have been proposed in former studies, such as lxmert @xcite , ernie-vil @xcite , de-vlbert @xcite , vilbert @xcite , 3 task definition given the image and transcribed text of a meme, the role of a certain entity in this meme will be determined as hero, villain, victim or other, which can be interpreted as a multi-class classification task. the official evaluation measure for the shared task is the macro-f1 score for the multi-class classification. ## data composition the dataset provided in this task is a combination of memes from covid-19 and us politics domain. every sample in the train and validation set contains an image, a transcription of texts and a list of entities with annotated labels. the shared task organizers provide the definitions for each class foot_1 : we present the distribution of entities' roles in hero villiain victim other donald trump 47 560 68 708 coronavirus 3 68 12 661 joe biden 22 183 17 587 barack obama 39 90 28 488 mask ---326 work from home ---272 2020 -35 -167 democratic party -161 24 115 figure 1: roles distribution of most frequent entities 5 system descriptions 5.1 preparation for visual feature preprocessing, we use the pretrained mask-rcnn model provided in the detectron2 framework foot_2 to obtain the object detection based region feature embedding @xmath3, o 2 , • • • , o k ] of images. detectron2 is proposed by facebook ai with state-of-the-art detection and segmentation algorithms. specifically, 50 boxes of 2048 dimensions region-based image features are extracted for every meme. for the text transcriptions, we make the content lower-case and remove punctuation and stopwords with nltk library @xcite . ## results and discussion in meanwhile, visualbert-large, uniter-large, and oscar-large models shows improvements in performance through continual pretraining, which can be interpreted as domain adaptation on our dataset. ensemble learning remarkably raises our score by 3.5 than the best single model, which achieves the best score for our submission in this task. ## conclusion in this paper, we have exploited a vcr approach to tackle the role labelling of entities in hateful memes, which is a novel task in multimodal understanding and reasoning. four popular transformerbased multimodal models, visualbert, uniter, oscar, and ernie-vil are applied as base models while strategies like loss re-weighting and data augmentation are implemented during the training of models. then, continual pretraining is taken for domain adaptation and achieves better performance. ensemble learning of variant models achieves the impressive macro f1-score of 0.5470 on the final (unseen) test set.