S oc C og C om at S em E val-2020 Task 11: Characterizing and Detecting Propaganda Using Sentence-Level Emotional Salience Features (2020)

Abstract: This paper describes a system developed for detecting propaganda techniques from news articles. We focus on examining how emotional salience features extracted from a news segment can help to characterize and predict the presence of propaganda techniques. Correlation analyses surfaced interesting patterns that, for instance, the “loaded language” and “slogan” techniques are negatively associated with valence and joy intensity but are positively associated with anger, fear and sadness intensity. In contrast, “flag waving” and “appeal to fear-prejudice” have the exact opposite pattern. Through predictive experiments, results further indicate that whereas BERT-only features obtained F1-score of 0.548, emotion intensity features and BERT hybrid features were able to obtain F1-score of 0.570, when a simple feedforward network was used as the classifier in both settings. On gold test data, our system obtained micro-averaged F1-score of 0.558 on overall detection efficacy over fourteen propaganda techniques. It performed relatively well in detecting “loaded language” (F1 = 0.772), “name calling and labeling” (F1 = 0.673), “doubt” (F1 = 0.604) and “flag waving” (F1 = 0.543).

## introduction propaganda is studied in a wide range of social sciences disciplines, including social psychology, political science, media and mass communication, as well as advertising and marketing @xcite @xcite . as jowett and o'donnell (2018) put it, propaganda is a "deliberate, systematic attempt to shape perceptions, manipulate cognitions, and direct behavior to achieve a response that furthers the desired intent of the propagandist". to achieve the agenda, propagandists may use various influence techniques such as loaded emotive language and flag waving. such techniques are centered on influencing the audiences' opinions and behaviors through psychological and rhetorical tricks in order to reach its purpose, such as promoting a particular politician or product in political or marketing campaigns. the ability to automatically detect propaganda has important societal implications. for news management, propaganda detection may help publishers to quickly identify news articles that may be subjected to propagandistic characteristics that severely deviate from journalism principles. for the general public, such tools may raise awareness for social media users to stay alert of potential propagandistic content, which often may leverage non-obvious psychological tricks, and potentially mitigate the propagation of such content. we participated in task 11 on the detection of propaganda techniques in news articles @xcite , in particular the technique classification task (task tc), a multi-class classification task that aims to classify each identified text segment with the existence of a collection of fourteen propaganda techniques @xcite . appendix a provides a summary and a distribution analysis on this task. this text segment-based ground truth data presents an advancement to this line of study with an ability to allow an algorithm to not only identify the existence of propaganda, but also to name the specific techniques. 1. "stop those refugees; they are terrorists" ["appeal to fear-prejudice"] 2. "the best of the best" ["exaggeration,minimisation"] 3. "entering this war will make us have a better future in our country" ["flag waving"] 4. "a lone lawmaker's childish shouting" ["loaded language"] 5. "republican congressweasels" ["name calling,labeling"] 6. "make america great again!" @xcite to extract the sentence-level emotional salience features in the news segments, we leveraged gupta and yang (2018)'s work which trains a collection of svm-based algorithms, named as crystalfeel foot_0 , which detects the intensities of five emotion dimensions present in a given text message, including the sentiment valence, joy, anger, fear and sadness @xcite . as the key purpose of propaganda is to influence or persuade the audiences, our main design hypothesis is that sentence-level emotional salience features will help to characterize a few most commonly used propaganda techniques that involve a degree of emotional connotations in their language manifestations. ## correlation analysis to gain an exploratory understanding on the usefulness of the emotional salience features, we performed bivariate correlation analyses between each of the propaganda ground truth labels for the 1,043 text segments in the development set and the emotion intensity scores derived from crystalfeel. results indicate interesting patterns: "loaded language", "flag waving", "slogans", "appeal to fearprejudice", and a total of twelve propaganda techniques are significantly correlated with at least one of the emotion intensity scores ( * * @xmath0.01, * @xmath1.05, @xmath2, 043). most notably, "loaded language" is negatively correlated with valence intensity (τ = -0.224**) and joy intensity (τ = -0.089**), but is positively correlated with anger intensity (τ = 0.181**), fear intensity (τ = 0.140**) and sadness intensity (τ = 0.243**). the "slogans" technique has a similar correlational pattern. in contrast, "flag waving" has the exact opposite pattern, where it is positively correlated with valence intensity (τ = 0.182**) and joy intensity (τ = 0.099**), but is negatively correlated with anger intensity (τ = -0.179**), fear intensity (τ = -0.168**) and sadness intensity (τ = -0.167**). the "appeal to fear-prejudice" technique has a similar pattern. two propaganda techniques, "black-and-white fallacy" and "causal oversimplification", are not found to be correlated with any emotion intensity scores. noted that these techniques also have less occurrences in the dataset (gold @xmath3; see appendix a) and are not conceptually associated with emotional connotation or emotional appeal by definition. the results showed initial support to our main design intuition, which also implies that the emotional saliences based system is likely to be effective in detecting emotions-associated (but not non-emotionsassociated) propaganda techniques. ## system overview following the the exploratory analysis, we proceed to design a predictive system named as "soccogcom". our soccogcom system is designed to determine the specific propaganda technique used in a given text segment from news articles. the possible techniques are based on a range of fourteen possibilities which are defined in the official semeval 2020 task 11 description paper @xcite . figure 1 depicts the system architecture. input layer: a training example consists of the span of text that contains a propaganda technique: features extraction: for every input text segment, our system extracts the following features: the extracted features for each text segment include five dimensions of emotion intensity features. 3. liwc features: word-level psycholinguistic features from the liwc lexicon foot_3 (pennebaker et al., 2015) (l f ). we obtained 73 extracted features that represent psycholinguistic characteristics of a piece of text that may involve a propaganda technique. fusion layer: crystalfeel and liwc features obtained above are first concatenated and a dense layer is applied over the concatenated vector to obtain a feature vector, h f , of dimension d @xmath4. this is done in order to project the features extracted from crystalfeel and liwc to a similar latent space as that of bert features. here, the extracted features, b f and h f , are simply concatenated to form the representation: z @xmath5; h f ] of dimension d @xmath6. a dense layer with 256 dimensions is applied over z f . after this, the final representation -o f is obtained by applying a dropout layer @xcite with dropout rate of 0.5. output layer: the system employs a fully-connected layer with softmax activation where the fused representation o f is fed. ## features experiments and results for data pre-processing, we used keras tokenizer to split the text into word tokens. the sentences are cleaned to remove unwanted characters and double spaces are replaced with single space. we conducted the features experiments using the standard training and development datasets provided in the official tc task, based on the system set up described in section 3. hyper-parameters are tuned using a held out validation data: 10% of the training data. to optimize the parameters, we use adam optimizer @xcite with an initial learning rate of 1e first, we evaluated the effects of using bert features and emotional salience features from crystalfeel outputs alone. bert only obtained micro-averaged f1 score of 0.5484, showing strong performance in comparison to a simple baseline using logistic regression. crystalfeel features achieved 0.5234, which shows fair performance given this is a low-dimensional features set. when combined, bert and crystalfeel features achieved better results, with micro-averaged f1 score of 0.5701 than the individual settings. we also assessed classic word-level psycholinguistics features based on liwc lexicons. the bert + liwc condition didn't converge, as the loss didn't decrease and was fluctuating a lot. adding liwc onto the hybrid bert + crystalfeel, i.e., the bert + crystalfeel + liwc condition, obtained micro-averaged f1 score of 0.5626, indicating that additional word-level psycholinguistics features do not appear to improve over the bert + crystalfeel condition. we tested albert + crystalfeel too, and they did not match the results obtained from bert + crystalfeel condition. based on the experiment results, we used the best-performing hybrid features sets (bert + crystalfeel) for our system results submission for the gold test set. after we submitted our results, we experimented a new condition where context features were added to the bert + crystalfeel condition. for context, we extracted features using 3 words before and after the target text segment. the results showed improvement (micro-averaged f1 @xmath7.5824).