Logically at the Constraint 2022: Multimodal role labelling (2022)

Abstract: This paper describes our system for the Constraint 2022 challenge at ACL 2022, whose goal is to detect which entities are glorified, vilified or victimised, within a meme . The task should be done considering the perspective of the meme’s author. In our work, the challenge is treated as a multi-class classification task. For a given pair of a meme and an entity, we need to classify whether the entity is being referenced as Hero, a Villain, a Victim or Other. Our solution combines (ensembling) different models based on Unimodal (Text only) model and Multimodal model (Text + Images). We conduct several experiments and benchmarks different competitive pre-trained transformers and vision models in this work. Our solution, based on an ensembling method, is ranked first on the leaderboard and obtains a macro F1-score of 0.58 on test set. The code for the experiments and results are available at https://bitbucket.org/logicallydevs/constraint_2022/src/master/

## introduction the rapid rise in the amount of harmful content being spread online is becoming a major societal challenge, with still unknown negative consequences. large resources have been invested by many actors in the field of social media to shield users from harmful content. it is imperative to understand in a systematic way how information is spread, and be able to scalably monitor existing narratives and flag hateful ones circulating using technology. one way this is done is using entity recognition coupled with entity sentiment @xcite . the former technique is to support osint(open source intelligence) analysts in understanding who or what are the subjects of discussion, and the latter automates the process of analysing if they are coupled with positive or negative feelings, in order to assist with understanding the stance of online users on specific topics. efforts to tackle this challenge were mainly focused on english-language text-based data formats such as articles @xcite . however, the complexity of content being posted online has drastically increased over time, and the challenge of harmful content detection now extends to multimedia, including memes @xcite . the emergence and proliferation of memes on social media have made their analysis a crucial challenge to understand online interactions. a point can also be made about the study of entities sentiment online, as the polarising portrayal of famous (or infamous) personalities or institutions often give rise to inflammatory views and content. extracting insights from memes is a novel field and still has a lot of opportunities for growth. the multimodality of text and image adds a layer of complexity which contains more information, but is also harder to extract. indeed each modality needs to understand their intrinsic properties but also capture cross-modal semantic understanding @xcite . this paper delves into the field of multimodal semantic role labelling, a new task with particular challenges. examples of the multimodal dataset @xcite used to tackle this problem and provided as part of the constraint competition are presented in figure 1 . the first sample shows a meme image displaying two politicians from opposite parties separated on two sides of the image, with text around them, as well as the associated json line input with the extracted text from the image (also known as optical character recognition or ocr), as well as the entities' mentioned labelled roles. in this case, all entities are referenced in the text of the image. in the second sample, however, we notice that not all are mentioned in the text, and visual information is needed to classify all entities. depending on the textual information in the image, textual role classification is insufficient as some memes' underlying message requires understanding of the visual information it contains, especially with the use of humour and sarcasm often associated with the format. the work done in this competition aims at finding unique and effective ways of tackling harmful meme classification as seen in the current social media space. an algorithm is designed for the task of role labelling for memes using a twin model (and ensemble) method. this siamese network is constructed by combining the output of pre-trained state-of-the-art (sota) models for both the visual components in the form of a cnn (efficientnet-b7 @xcite ) and for textual components using a transformer (deberta @xcite ). the feature outputs obtained from both branches are then combined to obtain a final solution. data analysis and investigation into potential bias in the dataset are also conducted to contextualise the task and present the difficulties of curating accurate multimodal datasets aimed at tackling the task for data in the wild @xcite . in this paper, an overview of past work in the field is presented (section 2), followed by a deep dive into the problem statement as well as the method followed to respond to it (section 3), then data analysis (section 4). experiments ran are presented in section 5, with results and discussion in section 6, and finally conclusion (section 7). ## related work there have been some work done with respect to semantic role labelling in text. the idea of absa(aspect based sentiment analysis) works along the same line. hence, utilisation of de-berta has provided the sota results (silva and marcacini) due to the disentangled attention improving the focus more on the positional embeddings rather than just based on the word embeddings. hence, improved results were also obtained in various snli task for this algorithm @xcite .they are nowadays very popular in natural language processing (nlp) as they usually get sota for a variety of nlp tasks such as classification, sentiment analysis, named entity recognition, translation, question answering, etc. classifying memes into relevant classes is a field that has got much more interest over the past few years. the facebook hateful meme competition @xcite was a very publicised initiative to try and augment the field's capabilities. the task was a binary classification of hateful/not hateful meme based on a dataset curated by meta. the winning solutions all comprised of ensembles of multimodal models. the memotion competitions @xcite are another example of work done in the meme space. this time, the classification was based on sentiment (positive, negative, neutral), as well as the strength of the sentiment and the underlying aim of the meme (satirical, humour or harmful). multimodal models here also obtained the top scores. multimodal models have seen a change over the past few years from twin networks like siamese @xcite to models pretrained on multiple multimodal tasks such as image captioning and visual question answering using transformers @xcite . object detection is used in these models to extract image features thanks to pre-trained two-staged detectors faster r-cnn model ( ren et al., 2015) ), or single-stage detectors (yolo v3 @xcite ). inspired by bert @xcite , models such as uniter @xcite and visualbert @xcite use a transformer architecture to jointly encode text and images, while lxmert @xcite and vilbert @xcite innovated by splitting their architectures in two, where a different transformer is applied to images and text individually before the features are combined by a third transformer. oscar (object-semantics aligned pre-training )(( li et al., 2020) ) add in the text input the class objects detected from the images by a faster r-cnn detector called object tags. the use of object tags in images as anchor points, significantly ease the learning of alignments during the pretraining. these models' effectiveness are demonstrated through their sota results on different multimodal dataset tasks such as nlvr2. this can be attributed to the models' increased capability to understand cross-modal correlations. however, these models are only as good as the data they've been pretrained on, which will present a challenge for the use case of the competition tackled in this paper. another point is that the architectures of the textual streams of these models are a few years old (such as bert) and inferior to the current sota (deberta). ## dataset the ocred text was obtained by running the google ocr api on the images, which in some examples leads to imperfect text detection or extraction. these two issues materialise in the form of either poorly clustered text paragraphs into the appropriate text boxes, meaning sentences from two separate paragraphs would be concatenated together midway through, but also through more basic spelling mistakes. another point relevant to meme analysis is the presence of sub images inside each image. an image might itself contain two separate images which tell a different story, often contrasting between sentiments of entities in each, such as in figure 4 . a big challenge with this task of entity classification is detecting where the entity is mentioned whether in the ocr or in the image. ## results and discussion just the simple experiment classifying all entities as "other" yielded 0.21 f1 score. we experimented with various models starting with just the textbased model, further adding image signals to using the image embeddings and finally a fully imageand-language based multimodal model to evaluate the model architecture efficiency in predicting a low resource multimodal problem. here are some observations :-(1) unimodal -we can see the difference in results moving from "deberta-v3-small" to "deberta-v3-large" in ## conclusion we described our participation in the con-straint 2022 shared task on "detecting the hero, the villain, and the victim in memes" with the implementation of various models. ensemble model based system outperforms all the models on val set and test set. a challenge in this task is the low resource of data available for training models. hence, transfer learning provides the best results. the best performing model in this competition combines the simple averaging of vilt, roberta large, deberta large, naive multimodal and deberta xlarge models. the ensemble seems to perform the best as the data size is small and we use a large model to allow for better transfer learning, this ultimately leads to some overfit of models but applying the averaging improves the results, like the boosted trees systems. we found that there were two major challenges with the problem :-(i) the entities were sometimes not present in the image or the text. (ii) the size of data required to learn this implicit learning was not sufficient. this ultimately undermines the performance of our deep learning architecture. creating a dataset for real-word multimodal problems, particularly the natural language inference problem of role labelling is challenging @xcite . we appreciate the work by the con-straint 2022 organizers, yet, a more elaborate and extensive data would make this dataset more suitable for benchmarking. as an emergent research field, we hope our extensive model analysis and proposed solutions can act as baseline and inspire further work.