S em E val-2021 Task 6: Detection of Persuasion Techniques in Texts and Images (2021)

Abstract: We describe SemEval-2021 task 6 on Detection of Persuasion Techniques in Texts and Images: the data, the annotation guidelines, the evaluation setup, the results, and the participating systems. The task focused on memes and had three subtasks: (i) detecting the techniques in the text, (ii) detecting the text spans where the techniques are used, and (iii) detecting techniques in the entire meme, i.e., both in the text and in the image. It was a popular task, attracting 71 registrations, and 22 teams that eventually made an official submission on the test set. The evaluation results for the third subtask confirmed the importance of both modalities, the text and the image. Moreover, some teams reported benefits when not just combining the two modalities, e.g., by using early or late fusion, but rather modeling the interaction between them in a joint model.

## introduction internet and social media have amplified the impact of disinformation campaigns. traditionally a monopoly of states and large organizations, now such campaigns have become within the reach of even small organisations and individuals @xcite . such propaganda campaigns are often carried out using posts spread on social media, with the aim to reach very large audience. while the rhetorical and the psychological devices that constitute the basic building blocks of persuasive messages have been thoroughly studied @xcite @xcite , only few isolated efforts have been made to devise automatic systems to detect them @xcite @xcite . warning: this paper contains meme examples and wording that might be offensive to some readers. thus, in 2020, we proposed semeval-2020 task 11 on detection of persuasion techniques in news articles, with the aim to help bridge this gap @xcite . the task focused on text only. yet, some of the most influential posts in social media use memes, as shown in figure 1 , 1 where visual cues are being used, along with text, as a persuasive vehicle to spread disinformation @xcite . during the 2016 us presidential campaign, malicious users in social media (bots, cyborgs, trolls) used such memes to provoke emotional responses @xcite . in 2021, we introduced a new semeval shared task, for which we prepared a multimodal corpus of memes annotated with an extended set of techniques, compared to semeval-2020 task 11. this time, we annotated both the text of the memes, highlighting the spans in which each technique has been used, as well as the techniques appearing in the visual content of the memes. based on our annotations, we offered the following three subtasks: subtask 1 (st1) given the textual content of a meme, identify which techniques (out of 20 possible ones) are used in it. this is a multilabel classification problem. subtask 2 (st2) given the textual content of a meme, identify which techniques (out of 20 possible ones) are used in it together with the span(s) of text covered by each technique. this is a multilabel sequence tagging task. subtask 3 (st3) given a meme, identify which techniques (out of 22 possible ones) are used in the meme, considering both the text and the image. this is a multilabel classification problem. a total of 71 teams registered for the task, 22 of them made an official submission on the test set and 15 of the participating teams submitted a system description paper. they performed massive experiments, investigated writing style and readability level, and trained models using logistic regression and svms. their findings confirmed that using distant supervision, in conjunction with rich representations, might encourage the model to predict the source of the article, rather than to discriminate propaganda from non-propaganda. the study by @xcite also proposed a corpus with 1.3k arguments annotated with five fallacies, including ad hominem, red herring, and irrelevant authority, which directly relate to propaganda techniques. a more fine-grained propaganda analysis was done by @xcite , who developed a corpus of news articles annotated with the spans of use of 18 propaganda techniques, from an invetory they put together. they targeted two tasks: (i) binary classification -given a sentence, predict whether any of the techniques was used in it; and (ii) multi-label multi-class classification and span detection task -given a raw text, identify both the specific text fragments where a propaganda technique is being used as well as the type of technique. they further proposed a multigranular gated deep neural network that captures signals from the sentence-level task to improve the performance of the fragment-level classifier and vice versa. subsequently, an automatic system, prta, was developed and made publicly available @xcite , which performs fine-grained propaganda analysis of text using these 18 fine-grained propaganda techniques. ## persuasion techniques scholars have proposed a number of inventories of persuasion techniques of various sizes @xcite @xcite . here, we use an inventory of 22 techniques, borrowing from the lists of techniques described in @xcite , @xcite and (abd kadir and sauffiyan, 2014). among these 22 techniques, the first 20 are applicable to both text and images, while the last two, appeal to (strong) emotions and transfer, are reserved for images. below, we provide a definition for each of these 22 techniques; more detailed instructions of the annotation process and examples are provided in appendix a. ## dataset the annotation process is explained in detail in appendix a, and in this section, we give a just brief summary. we collected english memes from our personal facebook accounts over several months in 2020 by following 26 public facebook groups, which focus on politics, vaccines, covid-19, and gender equality. we considered a meme to be a "photograph style image with a short text on top of it", and we removed examples that did not fit this definition, e.g., cartoon-style memes, memes whose textual content was strongly dominant or non-existent, memes with a single-color background image, etc. then, we annotated the memes using our 22 persuasion techniques. for each meme, we first annotated its textual content, and then the entire meme. we performed each of these two annotations in two phases: in the first phase, the annotators independently annotated the memes; afterwards, all annotators met together with a consolidator to discuss and to select the final gold label(s). the final annotated dataset consists of 950 memes: 687 memes for training, 63 for development, and 200 for testing. while the maximum number of sentences in a meme is 13, the average number of sentences per meme is just 1.68, as most memes contain very little text. note that the number of instances for subtasks 1 and 3 differs, and in some cases by quite a bit, e.g., for smears, doubt, and appeal to fear/prejudice. this shows that many techniques cannot be found in the text, and require the visual content, which motivates the need for multimodal approaches for subtask 3. note also that different techniques have different span lengths, e.g., loaded language and name calling are about 2-3 words long, e.g., violence, mass shooter, and coward. however, for techniques such as whataboutism, the average span length is 22 words. figure 2 shows statistics about the distribution of the number of persuasion techniques per meme. note the difference for memes without persuasion techniques between figures 2a and 2c : we can see that the number of memes without any persuasion technique drastically drops for subtask 3. this is because the visual modality introduces additional context that was not available during the text-only annotation, which further supports the need for multimodal analysis. the visual modality also has an impact on memes that already had persuasion techniques in the text-only phase. we observe that the number of memes with only one persuasion technique in subtask 3 is considerably lower compared to subtask 1, while the number of memes with three or more persuasion techniques has greatly increased for subtask 3. 5 evaluation framework ## participants and results below, we give a general description of the systems that participated in the three subtasks and their results, with focus on those ranked among the top-3. appendix c gives a description of every system. ## conclusion and future work we presented semeval-2021 task 6 on detection of persuasion techniques in texts and images. it was a successful task: a total of 71 teams registered to participate, 22 teams eventually made an official submission on the test set, and 15 teams also submitted a task description paper. in future work, we plan to increase the data size and to add more propaganda techniques. we further plan to cover several different languages. to collect the data for the dataset, we used facebook, as it has many public groups with a large number of users, who intentionally or unintentionally share a large number of memes. we used our own private facebook accounts to crawl the public posts from users and groups. to make sure the resulting feed had a sufficient number of memes, we initially selected some public groups focusing on topics such as politics, vaccines, covid-19, and gender equality. then, using the links between groups, we expanded our initial group pool to a total of 26 public groups. we went through each group, and we collected memes from old posts, dating up to three months before the newest post in the group. out of the 26 groups, 23 were about politics, us and canadian: left, right, centered, antigovernment, and gun control. the other 3 groups were on general topics such as health, covid-19, pro-vaccines, anti-vaccines, and gender equality. even though the number of political groups was larger (i.e., 23), the other 3 general groups had a higher number of users and a substantial amount of memes.