“Don’t discuss”: Investigating Semantic and Argumentative Features for Supervised Propagandist Message Detection and Classification (2021)

Abstract: One of the mechanisms through which disinformation is spreading online, in particular through social media, is by employing propaganda techniques. These include specific rhetorical and psychological strategies, ranging from leveraging on emotions to exploiting logical fallacies. In this paper, our goal is to push forward research on propaganda detection based on text analysis, given the crucial role these methods may play to address this main societal issue. More precisely, we propose a supervised approach to classify textual snippets both as propaganda messages and according to the precise applied propaganda technique, as well as a detailed linguistic analysis of the features characterising propaganda information in text (e.g., semantic, sentiment and argumentation features). Extensive experiments conducted on two available propagandist resources (i.e., NLP4IF’19 and SemEval’20-Task 11 datasets) show that the proposed approach, leveraging different language models and the investigated linguistic features, achieves very promising results on propaganda classification, both at sentence- and at fragment-level.

## introduction propaganda represents an effective, even though often misleading, communication strategy to promote a cause or a viewpoint, for instance in the political context @xcite @xcite . different communication means can be used to disseminate propaganda, i.e., textual documents, images, videos and oral speeches. the ability to effectively identify and manifestly label such kind of misleading and potentially harmful content is of primarily importance to restrain the spread of such information to avoid detrimental consequences for the society. in this paper, we tackle this challenging issue @xcite ) by proposing a textual propaganda detection model. more precisely, we address the following research questions: (i) how to automatically identify propaganda in textual documents and further classify them into finegrained categories?, and (ii) what are the linguistic distinctive features of propaganda text snippets? the contribution of this paper consists not only in proposing a new effective neural architecture to automatically identify and classify propaganda in text, but we also present a detailed linguistic analysis of the features characterising propaganda messages. our work focuses on the propaganda detection and classification task, casting it both as a binary and as a multi-class classification task, and we address it both at sentence-level and at fragment-level. we investigate different architectures of recent language models (i.e., bert, roberta), combining them with a rich set of linguistic features ranging from sentiment and emotion to argumentation features, to rhetorical stylistic ones. the extensive experiments we conducted on two standard benchmarks (i.e., the nlp4if'19 and semeval'20-task 11 datasets) show that the proposed architectures achieve satisfying results, outperforming state-ofthe-art systems on most of the propaganda detection and classification subtasks. an error analysis discusses the main sources of misclassification. furthermore, we analysed how the most relevant features for propaganda detection impact the finegrained classification of the different techniques employed in propagandist text, revealing the importance of semantic and argumentation features. ## related work in the last years, there has been an increasing interest in investigating methods for textual propaganda detection and classification. among them, @xcite present a system to organize news events according to the level of propagandist content in the articles, and introduces a new corpus (qprop) annotated with the propaganda vs. trustworthy classes, providing information about the source of the news articles. @xcite present the benchmark of the shared task nlp4if'19 foot_0 on fine-grained propaganda detection. as a follow up, in 2020 semeval proposed a shared task (t11) @xcite reducing the number of propaganda categories with respect to nlp4if'19, and proposing a more restrictive evaluation scheme. to evaluate the proposed approach, we rely on these two standard benchmarks, i.e., the nlp4if'19 and semeval'20 datasets. the most recent approaches for propaganda detection are based on language models that mostly involve transformer-based architectures. the approach that performed best on the nlp4if'19 sentence-level classification task relies on the bert architecture with hyperparameters tuning without activation function @xcite . @xcite focused first on the pre-processing steps to provide more information regarding the language model along with existing propaganda techniques, then they employ the bert architecture casting the task as a sequence labeling problem. the systems that took part in the semeval 2020 challenge -task 11 represent the most recent approaches to identify propaganda techniques based on given propagandist spans. the most interesting and successful approach @xcite proposes first to extend the training data from a free text corpus as a silver dataset, and second, an ensemble model that exploits both the gold and silver datasets during the training steps to achieve the highest scores. notice that most of the most performing recent models heavily rely on transformer-based architectures. in this paper, we also rely on language model architectures for the detection and classification of propaganda messages, empowering them with a rich set of features we identified as pivotal in propagandist text from the computational social science literature. in particular, @xcite discusses how emotional markers and affect at wordor phrase-level are employed in propaganda text, whilst @xcite show that the most effective technique to extract sentiment for the 3 propaganda detection as a classification task @xcite define the fine-grained propaganda detection task as two subtasks, with different granularities: i) sentence-level classification task (slc), which asks to predict whether a sentence contains at least one propaganda technique, and ii) fragment-level classification task (flc), which asks to identify both the spans and the type of propaganda technique. in the following example, "in a glaring sign of just how stupid and petty things have become in washington these days, manchin was invited on fox news tuesday morning to discuss how he was one of the only democrats in the chamber for the state of the union speech not looking as though trump killed his grandma." the span "stupid and petty" carries some propagandist bias, and is labeled as "loaded language" , "not looking as though trump killed his grandma" is considered as "exaggeration and minimisation", and "killed his grandma" is "loaded language". according to the slc task, the whole sentence should be classified as a propaganda message given that it contains at least one token (e.g., "stupid and petty") considered as such. as previously introduced, current systems address these tasks relying on word embedding models (e.g., bert-embedding) and standard features (e.g., pos, name-entity, n-grams), as representations to feed various rnn architectures @xcite . recently, the language model bert @xcite has been widely utilized to optimize the performances of classification tasks, but there is still room for improvement, in particular when applied to propaganda detection @xcite . in this work, we experiment with multiple architectures and language models to classify propagandist messages on both sentence and fragmentlevel. prior to that, we conduct a detailed investigation of linguistic and argumentation features to capture propaganda strategies. ## feature analysis propaganda strategies generally involve specific targets to be stimulated by the message. to better study such techniques from a computational point of view, we investigate a set of features that we assume to play a role in propaganda. ## sentence-level classification in the following, we describe the experiments we carried out to address the propaganda detection task type.php, https://www.thefreedictionary.com/list-ofpronouns.htm at sentence level, investigating different architectures and leveraging both recent language models and the features that proved to play a role in propaganda messages. for the evaluation, we used the two available datasets for propaganda detection: i) the nlp4if'19 data set @xcite (293 articles for training and 101 for testing); and ii) the data from semeval'20 t11 @xcite (371 articles for training and 75 in the development set). ## fragment-level classification in this section, we address the task of fragmentlevel classification, meaning that both the spans and the type of propaganda technique should be identified in the sentences. again, to test the proposed methods, we use both nlp4if'19 and se-meval'20 t11 datasets. however, in the two challenges, the flc task was evaluated according to different strategies, explained in the following. ## concluding remarks in this paper, we proposed a new neural architecture combined with state-of-the-art language models and a rich set of linguistic features for the detection of propaganda messages in text, and their further classification along with standard propaganda tech-niques. despite the boost in accuracy we achieved on two standard benchmarks for propaganda detection and classification (∼ 10% of f1 scores on sentence-level classification and on specific propaganda techniques on fragment-level classification), this task remains challenging, in particular regarding the fine-grained classification of the different propaganda classes. the state-of-the-art results on this subtask require further improvement to actually embed these solutions in real-world systems. future work goes in this direction, with the aim to improve the performance both of the disinformation detection task and of the classification of propaganda techniques. moreover, we are currently investigating the propaganda classes we discussed in this paper in the context of political debates, with the aim of building a fallacy detection systems that relies on the identification of propagandist messages in political speeches.