Detecting Propaganda Techniques in Memes (2021)

Abstract: Propaganda can be defined as a form of communication that aims to influence the opinions or the actions of people towards a specific goal; this is achieved by means of well-defined rhetorical and psychological devices. Propaganda, in the form we know it today, can be dated back to the beginning of the 17th century. However, it is with the advent of the Internet and the social media that propaganda has started to spread on a much larger scale than before, thus becoming major societal and political issue. Nowadays, a large fraction of propaganda in social media is multimodal, mixing textual with visual content. With this in mind, here we propose a new multi-label multimodal task: detecting the type of propaganda techniques used in memes. We further create and release a new corpus of 950 memes, carefully annotated with 22 propaganda techniques, which can appear in the text, in the image, or in both. Our analysis of the corpus shows that understanding both modalities together is essential for detecting these techniques. This is further confirmed in our experiments with several state-of-the-art multimodal models.

## introduction social media have become one of the main communication channels for information dissemination and consumption, and nowadays many people rely on them as their primary source of news @xcite . despite the many benefits that social media offer, sporadically they are also used as a tool, by bots or human operators, to manipulate and to mislead unsuspecting users. propaganda is one such communication tool to influence the opinions and the actions of other people in order to achieve a predetermined goal @xcite . warning: this paper contains meme examples and words that are offensive in nature. propaganda is not new. it can be traced back to the beginning of the 17th century, as reported in @xcite @xcite , where the manipulation was present at public events such as theaters, festivals, and during games. in the current information ecosystem, it has evolved to computational propaganda @xcite , where information is distributed through technological means to social media platforms, which in turn make it possible to reach well-targeted communities at high velocity. we believe that being aware and able to detect propaganda campaigns would contribute to a healthier online environment. propaganda appears in various forms and has been studied by different research communities. there has been work on exploring network structure, looking for malicious accounts and coordinated inauthentic behavior @xcite @xcite . in the natural language processing community, propaganda has been studied at the document level @xcite , and at the sentence and the fragment levels @xcite . there have also been notable datasets developed, including (i) @xcite , which consists of document-level annotation labeled with four classes (trusted, satire, hoax, and propaganda); (ii) qprop @xcite , which uses binary labels (propaganda vs. non-propaganda), and (iii) ptc @xcite , which uses fragment-level annotation and an inventory of 18 propaganda techniques. while that work has focused on text, here we aim to detect propaganda techniques from a multimodal perspective. this is a new research direction, even though large part of propagandistic social media content nowadays is multimodal, e.g., in the form of memes. memes are popular in social media as they can be quickly understood with minimal effort @xcite . they can easily become viral, and thus it is important to detect malicious ones quickly, and also to understand the nature of propaganda, which can help human moderators, but also journalists, by offering them support for a higher level analysis. figure 1 shows some examples of memes foot_0 and propaganda techniques. example (a) applies transfer, using symbols (hammer and sickle) and colors (red), that are commonly associated with communism, in relation to the two republicans shown in the image; it also uses name calling (traitors, moscow mitch, moscow's bitch). the meme in (b) uses both smears and glittering generalities. the one in (c) expresses smears and suggest that joe biden's campaign is only alive because of mainstream media. the examples in the second row show some less common techniques. example (d) uses appeal to authority to give credibility to a statement that rich politicians are crooks, and there is also a thought-terminating cliché used to discourage critical thought on the statement in the form of the phrase "we know", thus implying that the clintons are crooks, which is also smears. the above examples illustrate that propaganda techniques express shortcuts in the argumentation process, e.g., by leveraging on the emotions of the audience or by using logical fallacies to influence it. their presence does not necessarily imply that the meme is propagandistic. thus, we do not annotate whether a meme is propagandistic (just the propaganda techniques it contains), as this would require, among other things, to determine its intent. our contributions can be summarized as follows: textual content most research on propaganda detection has focused on analyzing textual content @xcite @xcite . @xcite developed the tshp-17corpus, which uses document-level annotation and is labeled with four classes: trusted, satire, hoax, and propaganda. tshp-17 was developed using distant supervision, i.e., all articles from a given news outlet share the label of that outlet. the articles were collected from the english gigaword corpus and from seven other unreliable news sources. among them two were propagandistic. they trained a model using word n-gram representation with logistic regression and reported that the model performed well only on articles from sources that the system was trained on. @xcite developed a new corpus, qprop , with two labels: propaganda vs. non-propaganda. they also experimented on tshp-17 and qprop corpora, where for the tshp-17 corpus, they binarized the labels: propaganda vs. any of the other three categories. they performed massive experiments, investigated writing style and readability level, and trained models using logistic regression and svms. their findings confirmed that using distant supervision, in conjunction with rich representations, might encourage the model to predict the source of the article, rather than to discriminate propaganda from non-propaganda. similarly, @xcite developed a corpus with 1.3k arguments annotated with five fallacies, including ad hominem, red herring, and irrelevant authority, which directly relate to propaganda techniques. a more fine-grained propaganda analysis was done by @xcite . they developed a corpus of news articles annotated with 18 propaganda techniques. the annotation was at the fragment level, and enabled two tasks: (i) binary classification -given a sentence in an article, predict whether any of the 18 techniques has been used in it; (ii) multi-label multi-class classification and span detection task -given a raw text, identify both the specific text fragments where a propaganda technique is being used as well as the type of the technique. on top of this work, they proposed a multi-granular deep neural network that captures signals from the sentence-level task and helps to improve the fragment-level classifier. subsequently, a system was developed and made publicly available @xcite . multimodal content previous work has explored the use of multimodal content for detecting misleading information @xcite , deception @xcite , emotions and propaganda @xcite , hateful memes @xcite @xcite , antisemitism @xcite and propaganda in images @xcite . @xcite proposed models for detecting misleading information using images and text. they developed a corpus of 500,000 twitter posts consisting of images labeled with six classes: disinformation, propaganda, hoaxes, conspiracies, clickbait, and satire. then, they modeled textual, visual, and lexical characteristics of the text. @xcite explored multilingual multimodal content for deception detection. they had two multi-class classification tasks: (i) classifying social media posts into four categories (propaganda, conspiracy, hoax, or clickbait), and (ii) classifying social media posts into five categories (disinformation, propaganda, conspiracy, hoax, or clickbait). multimodal hateful memes have been the target of the popular "hateful memes challenge", which the participants addressed using fine-tuned state-ofart multi-modal transformer models such as vil-bert @xcite , multimodal bitransformers @xcite , and @xcite to classify hateful vs. not-hateful memes @xcite . @xcite explored different early-fusion multimodal approaches and proposed various methods that can improve the performance of the detection systems. our work differs from the above research in terms of annotation, as we have a rich inventory of 22 fine-grained propaganda techniques, which we annotate separately in the text and then jointly in the text+image, thus enabling interesting analysis as well as systems for multi-modal propaganda detection with explainability capabilities. ## propaganda techniques propaganda comes in many forms and over time a number of techniques have emerged in the literature @xcite @xcite @xcite @xcite . different authors have proposed inventories of propaganda techniques of various sizes: seven techniques @xcite , 24 techniques weston (2018), 18 techniques @xcite , just smear as a technique @xcite , and seven techniques @xcite . we adapted the techniques discussed in @xcite , @xcite and (abd kadir and sauffiyan, 2014), thus ending up with 22 propaganda techniques. among our 22 techniques, the first 20 are used for both text and images, while the last two appeal to (strong) emotions and transfer are reserved for labeling images only. below, we provide the definitions of these techniques, which are included in the guidelines the annotators followed (see appendix a.2) for more detal. in some cases, the support is built based on preconceived judgements. ## dataset we collected memes from our own private facebook accounts, and we followed various facebook public groups on different topics such as vaccines, politics (from different parts of the political spectrum), covid-19, gender equality, and more. we wanted to make sure that we have a constant stream of memes in the newsfeed. we extracted memes at different time frames, i.e., once every few days for a period of three months. we also collected some old memes for each group in order to make sure we covered a larger variety of topics. ## experimental results rows 6-8 present results for multimodal fusion models. the best one is bert + resnet-152 (+2 points over fasttext + resnet-152). we observe that early fusion models (rows 7-8) outperform late fusion ones (row 6). this makes sense as late fusion is a simple mean of the results of each modality, while early fusion has a more complex architecture and trains a separate multi-layer perceptron for the visual and for the textual features. we can also see that both mid-fusion models (rows 7-8) improve over the corresponding textonly ones (rows 3-5). finally, looking at the results in rows 9-11, we can see that each multimodal model consistently outperforms each of the unimodal models (rows 1-8). the best results are achieved with vilbert cc (row 10) and visual-bert coco (row 11), which use complex representations that combine the textual and the visual modalities. overall, we can conclude that multimodal approaches are necessary to detect the use of propaganda techniques in memes, and that pretrained transformer models seem to be the most promising approach. ## conclusion and future work we have proposed a new multi-class multi-label multimodal task: detecting the type of propaganda techniques used in memes. we further created and released a corpus of 950 memes annotated with 22 propaganda techniques, which can appear in the text, in the image, or in both. our analysis of the corpus has shown that understanding both modalities is essential for detecting these techniques, which was further confirmed in our experiments with several state-of-the-art multimodal models. in future work, we plan to extend the dataset in size, including with memes in other languages. we further plan to develop new multi-modal models, specifically tailored to fine-grained propaganda detection, aiming for deeper understanding of the semantics of the meme and of the relation between the text and the image. a number of promising ideas have been already tried by the participants in a shared task based on this data at semeval-2021 @xcite , which can serve as an inspiration when developing new models.