S em E val-2020 Task 11: Detection of Propaganda Techniques in News Articles (2020)

Abstract: We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about Span Identification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250 teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles.

## introduction propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda. it can hide in news published by both established and non-established outlets, and, in the internet era, it has the potential of reaching very large audiences @xcite @xcite . propaganda is most successful when it goes unnoticed by the reader, and it often takes some training for people to be able to spot it. the task is way more difficult for inexperienced users, and the volume of text produced on a daily basis makes it difficult for experts to cope with it manually. with the recent interest in "fake news", the detection of propaganda or highly biased texts has emerged as an active research area. however, most previous work has performed analysis at the document level only @xcite or has analyzed the general patterns of online propaganda @xcite . semeval-2020 task 11 offers a different perspective: a fine-grained analysis of the text that complements existing approaches and can, in principle, be combined with them. propaganda in text (and in other channels) is conveyed through the use of diverse propaganda techniques @xcite , which range from leveraging on the emotions of the audience -such as using loaded language or appeals to fearto using logical fallacies -such as straw men (misrepresenting someone's opinion), hidden ad-hominem fallacies, and red herring (presenting irrelevant data). some of these techniques have been studied in tasks such as hate speech detection @xcite and computational argumentation @xcite . figure 1 shows the fine-grained propaganda identification pipeline, including the two targeted subtasks. our goal is to facilitate the development of models capable of spotting text fragments where propaganda techniques are used. the task featured the following subtasks: subtask si (span identification): given a plain-text document, identify those specific fragments that contain at least one propaganda technique. (this is a binary sequence tagging task.) subtask tc (technique classification): given a propagandistic text snippet and its document context, identify the propaganda technique used in that snippet. (this is a multi-class classification problem.) figure 1 : the full propaganda identification pipeline, including the two subtasks: span identification and technique classification. a total of 250 teams registered for the task, 44 of them made an official submission on the test set (66 submissions for both subtasks), and 32 of the participating teams submitted a system description paper. the rest of the paper is organized as follows. section 2 introduces the propaganda techniques we considered in this shared task. section 3 describes the organization of the task, the corpus and the evaluation measures. an overview of the participating systems is given in section 4, while section 5 discusses the evaluation results. related work is described in section 6. finally, section 7 draws some conclusions, and discusses some directions for future work. ## propaganda and its techniques propaganda comes in many forms, but it can be recognized by its persuasive function, sizable target audience, the representation of a specific group's agenda, and the use of faulty reasoning and/or emotional appeals @xcite . the term propaganda was coined in the 17th century, and initially referred to the propagation of the catholic faith in the new world @xcite . it soon took a pejorative connotation, as its meaning was extended to also mean opposition to protestantism. in more recent times, the institute for propaganda analysis @xcite proposed the following definition: propaganda. expression of opinion or action by individuals or groups deliberately designed to influence opinions or actions of other individuals or groups with reference to predetermined ends. recently, bolsover and howard (2017) dug deeper into this definition identifying its two key elements: (i) trying to influence opinion, and (ii) doing so on purpose. propaganda is a broad concept, which runs short for the aim of annotating specific propaganda fragments. yet, influencing opinions is achieved through a series of rhetorical and psychological techniques, and in the present task, we focus on identifying the use of such techniques in text. whereas the definition of propaganda is widely accepted in the literature, the set of propaganda techniques considered, and to some extent their definition, differ between different scholars @xcite . for instance, @xcite considers seven propaganda techniques, whereas @xcite lists at least 24 techniques, and the wikipedia article on the topic includes 67. 1 below, we describe the propaganda techniques we consider in the task: a curated list of fourteen techniques derived from the aforementioned studies. we only include techniques that can be found in journalistic articles and can be judged intrinsically, without the need to retrieve supporting information from external resources. for example, we do not include techniques such as card stacking @xcite , since it would require comparing multiple sources. note that our list of techniques was initially longer than fourteen, but we decided, after the annotation phase, to merge similar techniques with very low frequency in the corpus. a more detailed list with definitions and examples is available online 2 and in appendix c, and examples are shown in ## doubt can the same be said for the obama administration? 6 appeal to fear/prejudice a dark, impenetrable and "irreversible" winter of persecution of the faithful by their own shepherds will fall. ## evaluation framework the semeval 2020 task 11 evaluation framework consists of the ptc-semeval20 corpus and the evaluation measures for both the span identification and the technique classification subtasks. we describe the organization of the task in section 3.3; here, we focus on the dataset, the evaluation measure, and the organization setup. ## 4 participating systems in this section, we focus on a general description of the systems participating on both the si and the tc subtasks. we pay special attention to the most successful approaches. the subindex on the right of each team represents their official rank in the subtasks. appendix a includes brief descriptions of all systems. ## 7 conclusion and future work we have described semeval-2020 task 11 on detection of propaganda techniques in news articles. the task attracted the interest of a number of researchers: 250 teams signed up to participate, and 44 made submissions on the test dataset. we received 35 and 31 submissions for subtask si and subtask tc, respectively. overall, subtask si (segment identification) was easier and all systems managed to improve over the baseline. however, subtask tc (technique classification) proved to be much more challenging, and some teams could not improve over our baseline. in future work, we plan to extend the dataset to cover more examples as well as more propaganda techniques. we further plan to develop similar datasets for other languages.