Interpretable Propaganda Detection in News Articles (2021)

Abstract: Online users today are exposed to misleading and propagandistic news articles and media posts on a daily basis. To counter thus, a number of approaches have been designed aiming to achieve a healthier and safer online news and media consumption. Automatic systems are able to support humans in detecting such content; yet, a major impediment to their broad adoption is that besides being accurate, the decisions of such systems need also to be interpretable in order to be trusted and widely adopted by users. Since misleading and propagandistic content influences readers through the use of a number of deception techniques, we propose to detect and to show the use of such techniques as a way to offer interpretability. In particular, we define qualitatively descriptive features and we analyze their suitability for detecting deception techniques. We further show that our interpretable features can be easily combined with pre-trained language models, yielding state-of-the-art results.

## introduction with the rise of the internet and social media, there was also a rise of fake @xcite , biased @xcite , hyperpartisan @xcite , and propagandistic content @xcite . in 2016, news got weaponized, aiming to influence the us presidential election and the brexit referendum, making the general public concerned about the dangers of the proliferation of fake news @xcite @xcite @xcite . there ware two reasons for this. first, disinformation disguised as news created the illusion that the information is reliable, and thus people tended to lower their barrier of doubt compared to when information came from other types of sources. figure 1 : comparison of propaganda prediction interpretability using existing methods. our proposed method helps users to interpret propaganda predictions across various dimensions, e.g., is there a lot of positive/negative sentiment (can signal the use of loaded language, which appeals to emotions), are the target sentence and the document body related to the title, does the sentence agree/disagree with the title, etc. each symbol in the top bar chart represents an information source for propaganda detection. second, the rise of citizen journalism led to the proliferation of various online media, and the veracity of information became an issue. in practice, the effort required to fact-check the news, and its bias and propaganda remained the same or even got more complex, compared to traditional media, since the news was re-edited and passed through other media channels. propaganda aims to influence the audience with the aim of advancing a specific agenda @xcite . detecting it is tricky and arguably more difficult than finding false information in an article. this is because propagandistic articles are not intended to simply make up a story with objective errors, but instead use a variety of techniques to convince people, such as selectively conveying facts or appealing to emotions @xcite . while many techniques are ethically questionable, we can think of propaganda techniques as rhetorical expressions that effectively convey the author's opinion ( @xcite . due to these characteristics, propagandistic articles are often produced primarily for political purposes (but are also very common in commercial advertisement), which directly affect our lives, and are commonly found even in major news media outlets, which are generally considered credible. the importance of detecting propaganda in the news has been recently emphasized, and research is being conducted from various perspectives @xcite @xcite . however, while previous work has done reasonable job at detecting propaganda, it has largely ignored the question of why the content is propagandistic, i.e., there is a lack of interpretability of the system decisions, and in many cases, there is a lack of interpretability of the model as well, i.e., it is hard to understand what the model actually does even for its creators. interpretability is indispensable if propaganda detection systems are to be trusted and accepted by the users. according to the confirmation bias theory @xcite , people easily accept new information that is consistent with their beliefs, but are less likely to do so when it contradicts what they already know. thus, even if a model can correctly predict which news is propagandistic, if it fails to explain the reason for that, people are more likely to reject the results and to stick to what they want to believe. in order to address this issue, we propose a new formulation of the propaganda detection task and a model that can explain the prediction results. figure 1 compares the coverage of the explanations for pre-existing methods vs. our proposal. our contributions can be summarized as follows: given a document d that consists of n sentences @xmath0, each sentence should be classified as belonging to one of 18 propaganda techniques or as being non-propaganda. the exact definition of propaganda can be subtly different depending on the social environment and the individual's growth background, and thus it is not surprising that the propaganda techniques defined in the literature differ @xcite @xcite @xcite . the techniques we use in this paper are shown in this granular labeling went beyond our scope and we had to restructure the data. first, we divided the data into sentences. second, in order to reduce the complexity of the task, we changed the multi-label setup to a multi-class one by ignoring duplicate labels and only allowing one technique per sentence (the first one), breaking ties at random. as a result, we obtained 20,111 sentences labeled with a non-propaganda class or with one of 18 propaganda techniques. based on this data, we built a system for predicting the use of propaganda techniques at the sentence level, and we provided the semantic and the structural information related to propaganda techniques as the basis of the results. ## proposed method our method can detect the propaganda for each sentence in a document, and can explain what propaganda technique was used with interpretable semantic and syntactic features. we further propose novel features conceived in the study of human behavioral characteristics. more detail below. ## experimental results in this section, we present our experimental setup for interpretable propaganda detection and the evaluation results from our experiments. specifically, we perform three sets of experiments: (i) in section 4.1, we quantitatively analyze the effectiveness of the features we proposed in section 3; (ii) in sections 4.2 and 4.3, we compare our feature-based model to the state-of-the-art model described in @xcite using the experimental setup from that paper; (iii) in section 4.4, we analyze the performance of our model with respect to each of the 18 propaganda techniques. ## related work research on propaganda detection has focused on analyzing textual content @xcite @xcite @xcite . @xcite developed the tshp-17 corpus, which uses document-level annotation with four classes: trusted, satire, hoax, and propaganda. they trained a model using word n-gram representation and reported that the model performed well only on articles from sources that the system was trained on. @xcite developed the qprop corpus with two labels: propaganda vs. non-propaganda. they also experimented on tshp-17 and qprop corpora, where for the tshp-17 corpus, they binarized the labels: propaganda vs. any of the other three categories. similarly, @xcite developed a corpus with 1.3k arguments annotated with five fallacies, including ad hominem, red herring, and irrelevant authority, which directly relate to propaganda techniques. moreover, @xcite studied the connection between hyperpartisanship and propaganda. a more fine-grained propaganda analysis was proposed by @xcite , who developed a corpus of news articles annotated with 18 propaganda techniques which was used in two shared tasks: at semeval-2020 @xcite and at nlp4if-2020 @xcite . subsequently, the prta system was released @xcite , and improved models were proposed, addressing the limitations of transformers @xcite . the prta system was used to perform a study of covid-19 disinformation and associated propaganda techniques in bulgaria @xcite and qatar @xcite . finally, multimodal content was explored in memes using 22 fine-grained propaganda techniques @xcite , which was also used in a semeval-2021 shared task @xcite . ## conclusion and future work we proposed a model for interpretable propaganda detection, which can explain which sentence in an input news article is propagandistic by pointing out the propaganda techniques used, and why the model has predicted it to be propagandistic. to this end, we devised novel features motivated by human behavior studies, quantitatively deduced the relationship between semantic or syntactic features and propaganda techniques, and selected the features that were important for detecting propaganda techniques. finally, we showed that our proposed method can be combined with a pre-trained language model to yield new state-of-the-art results. in future work, we plan to expand the dataset by creating a platform to guide annotators. the dataset will be updated continuously and released for research purposes. 3 we also plan to release an interpretable online system, with the aim to foster a healthier and safer online news environment.