Combining Language Models and Linguistic Information to Label Entities in Memes (2022)

Abstract: This paper describes the system we developed for the shared task ‘Hero, Villain and Victim: Dissecting harmful memes for Semantic role labelling of entities’ organised in the framework of the Second Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (Constraint 2022). We present an ensemble approach combining transformer-based models and linguistic information, such as the presence of irony and implicit sentiment associated to the target named entities. The ensemble system obtains promising classification scores, resulting in a third place finish in the competition.

## introduction the exponential growth of social media such as twitter, facebook or youtube has created a variety of novel ways to communicate. this daily exposure to other users' opinions and comments has become a constant in many people's lives. unfortunately, this new way of freely communicating online has also given a forum to people who want to denigrate others because of their race, color, gender, sexual orientation, religion, etc., or to spread fake news and disinformation. the automatic processing of this user generated text by means of natural language processing (nlp) techniques may contribute to an effective analysis of public opinion, but also to the automatic detection of this harmful online content. one very popular mode of expression on social media today are internet memes. memes are often used for entertainment purposes, but they are also used for online trolling, because of their potential for spreading provocative and attentiongrabbing humor @xcite . they have been described both as speech acts @xcite and performative acts, involving a conscious decision to either support or reject an ongoing social discourse @xcite . their multi-modal nature, composed of a mixture of text and image, makes them a very challenging research object for automatic analysis. research has already been proposed to automatically process harmful memes in various downstream tasks. a related shared task was proposed by @xcite , who organized the hateful memes challenge, where systems were developed to detect hate speech in multimodal memes. most systems participating to the task applied fine-tuning of state-of-the-art transformer methods, such as supervised multimodal bitransformers @xcite , vilbert @xcite and @xcite to classify memes as being hateful or not. this paper presents our system developed to classify entities as hero, villain, victim or other, in memes about two controversial topics provoking a lot of hate speech and disinformation, namely the presidential election in the us and the covid-19 pandemic spreading. to tackle the task, we incorporated both transformer-based embeddings as well as linguistic information (implicit entity connotations and irony detection labels) into our classifier. the remainder of this paper is organized as follows. section 2 introduces the shared task and data sets, whereas section 3 describes the information sources and ensemble system we developed to label named entities in memes. section 4 lists the experimental results and provides a detailed analysis and discussion. section 5 ends with concluding remarks and indications for future research. the research described in this paper was carried out in the framework of the constraint 2020 shared task: hero, villain and victim: dissecting harmful memes for semantic role labeling of entities @xcite . given a meme and an entity, systems have to determine the role of the entity in the meme, namely: the task is conceived as a multi-class classification task, which has to be analyzed from the meme author's perspective. ## system description we approached the meme entity labeling task as a multi-class classification task, where a category is predicted for all entities occurring in the meme. to this end, an ensemble classifier is built combining probability scores output by various transformerbased language models and linguistic information assigning implicit sentiment to the entities and detecting irony in the meme text. we first give an overview of all different information sources incorporated in the feature vector (section 3.1), and then describe the ensemble method combining the various information sources into a feature vector for classification (section 3.2). ## experimental results a first set of experiments was carried out to assess the classification performance of the different language models. in this case, the classifier is trained and evaluated on feature vectors containing similarity scores for the four different labels. the first three lines of for a second set of experiments, we created an ensemble system containing various combinations of the mcqa language model probability scores per label, together with the implicit sentiment feature for the target entity. the best performing ensemble appeared to be a combination of the twitterxlm-roberta, covid-bert and bert-tweet similarity scores per label, together with the implicit sentiment features, resulting in the best performance scores on the held-out test set, viz. a macro f1score of 0.5514. combining this ensemble system with the irony detection and fasttext word vector features resulted in a lower f-score (0.5495) and precision (0.5201), but in a higher recall score (0.6045). apart from challenges posed by the data set itself, such as noise in the ocr text, very skewed class distribution, or spelling mistakes in the target entities 4 , our error analysis revealed some other trends in wrongly predicted named entity labels. 4 mistakes like "dr. dr. anthony fauci" and "valdimir puitin". first, it is clear that labeling entities in memes is a very hard task. systems have to both understand the ocr text, but also correctly process the picture that sometimes contains crucial information. as we only incorporate text processing features in our ensemble system, a lot of the erroneous predictions are caused because of lacking visual information to correctly interpret the picture of the meme, as illustrated by figure 6 . in addition, some memes require a lot of common sense or factual/news knowledge. as an example, we can refer to figure 7 , where the entity melania trump had to be labeled as "villain", but was predicted by the system as "other". it is impossible, however, to interpret this meme correctly without knowing that donald trump's wife, melania, took center stage on the first day of the republican national convention, and was accused of the fact that a portion of her speech plagiarized michelle obama. ## conclusion in this paper, we describe the system proposed for the constraint 2022 shared task on labeling entities in memes as hero, villain, victim or other. to tackle the task, we built an ensemble classi-fier combining the output predictions of various transformer-based language models with implicit sentiment features for the target entities, irony predictions on the ocr text and fasttext word vectors. the best performing system combines the predictions of three different language models with the implicit sentiment feature, obtaining a macro f1score of 55%. as the data set was very skewed, we obtained much better results for the "other" class than for the other three labels. especially for the hero class, only represented by 3% of the training entities, classification appeared to be challenging (f1-score of 33%). the analysis of the results showed there is still a lot of room for improvement. in future research, we plan to integrate visual information into our ensemble system, as it is clear that we lacked this information to properly address this multimodal task. in addition, we will investigate other ways to set up the multiple choice qa system, in order to construct better sentences containing the target entities. finally, the system would also benefit from more semantic information, in order to model entities that are now not explicitly mentioned in the ocr text. it would, for instance, be interesting to semantically link an ocr text line talking about brexit with the entity uk government. this would allow to inject some common sense into the meme classification system.