UNTL ing at S em E val-2020 Task 11: Detection of Propaganda Techniques in E nglish News Articles (2020)

Abstract: Our system for the PropEval task explores the ability of semantic features to detect and label propagandistic rhetorical techniques in English news articles. For Subtask 2, labeling identified propagandistic fragments with one of fourteen technique labels, our system attains a micro-averaged F1 of 0.40; in this paper, we take a detailed look at the fourteen labels and how well our semantically-focused model detects each of them. We also propose strategies to fill the gaps.

## introduction this paper describes the unt linguistics contribution to the propaganda evaluation (propeval) shared task at semeval-2020 @xcite . our system uses non-neural classification methods to address the two linked subtasks: 1) detection of variable-length english-language news article spans that employ propagandistic rhetorical techniques; and 2) 14-way labeling of identified spans. the system focuses on semantically-oriented features, including word embeddings, sentiment/valence analysis, and named entity features. the amount of text available in our vast global landscape, both content created by companies for public use and user-generated content, has grown far beyond the reach of human moderation. the prevalence of content with misleading rhetorical techniques, often promoting partisan political or social agendas, calls for automated detection and labeling of propaganda in textual media. currently, systems like this are rare (for example, proppy @xcite ), and to our knowledge there are none in wide use. widely-available, browser-based, automated, interactive detection of online propaganda could reduce the quantity of that propaganda and promote the formation of a better-informed public, acting as a bulwark against misleading content in real time. such systems could influence rhetorical trends in the presentation of online news to skew away from the unnecessarily sensational and toward the factual. our system approaches the task of propaganda detection primarily from a semantic perspective, following the intuition that propagandistic text is different from non-propagandistic text in ways that are linguistically nuanced. the semantic information encoded in word and sentence embeddings, for example, has been shown to capture such nuances @xcite . we focus primarily on the subtask 2 (technique classification) and made an official submission for this task only. our subtask 1 system achieved a macro-averaged f1 score of 0.349, but only after the end of the official submission period. our system for subtask 2 came in 27th out of 32 teams on the test set (@xmath0.391) and 34th out of 42 teams on the development set (@xmath1.409). detailed results and error analysis appear below. subtask 1: propagandistic span identification. subtask 1 requires the identification of propagandistic spans from raw text. we treat this as a sequence labeling task, using a conditional random field (crf, @xcite ) classifier and a beginning/inside-propaganda vs. not-propaganda labeling approach. in other words, the first token of a propagandistic span is labeled b-p, the subsequent tokens of that same span are labeled i-p, and all other tokens are labeled n. crfs naturally incorporate contextual factors when identifying spans, which in this case vary in length from one or two tokens to multi-sentence chunks. for example, we convert the following sentence (propagandistic span underlined) to the labeling shown below: manchin says democrats acted like babies at the sotu (video) personal liberty poll exercise your right to vote becomes [b-p, i-p×5, n×10]. 2 results for this three-way labeling task were very bad (see discussion in section 4). in a second approach, we reduce the granularity to the sentence-fragment level, breaking the text into chunks by splitting at label boundaries and at sentence boundaries; recall that the input texts are usually multi-sentence documents. for labeling, we collapse the distinction to simply propaganda (p) or not-propaganda (n), such that the example above would be labeled as the sequence: [p×6, n×10]. final features for subtask 1 appear in the left-hand side of subtask 2: technique classification. for subtask 2, we use a logistic regression classifier to label individual propagandistic fragments with one of fourteen technique labels. see ## models and experimental setup for data preprocessing, we use spacy's foot_2 nlp pipeline to get part-of-speech, dependency, named entity, and word embedding features. specifically, we use glove embeddings @xcite . for sentiment features, we access each token's valence, arousal, and dominance values from the vad lexicon. classifiers are trained and evaluated using scikit-learn 4 with default parameter settings, and all code has been released on github. 5 in the first phase of development, we use a 70/30 split of the 371 articles in the training set. in a later phase, we use all 371 articles for training, and we evaluate the system on the development set. results in ## results and analysis subtask 1. for subtask 1, we implemented two systems, one with b-p/i-p/n labeling, and the other with binary labeling. see results in the b-p/i-p/n system struggles with the variability of the target spans' length and content. the label n is by far the most frequent in the training data, and the system is especially unlikely to correctly label b-p tokens. this underlabeling of b-p tokens consequently leads to underlabeling of i-p tokens, since without a correctly identified b-p, subsequent i-p tokens cannot be identified. feature analysis reveals that the most informative feature is the token's word embedding; 10 out of the 30 most informative features are individual vector elements. another important set of features concerns the token's left-hand context: both the bag-of-words before the target token and the bow's word embedding (bow beginning and semantic left) account for 13 out of 30 most informative features. these results are contrasted with their right-hand counterparts (bow end and semantic right), which when measured similarly account for only 3 out of 30 informative features. it seems that the semantic content of tokens used in propagandistic rhetoric differs to some degree from tokens in non-propagandistic fragments. additionally, it seems that the left-hand context is more important for signaling propaganda than the right-hand context. this could be due to the fact that it is easier for the system to detect a continuation of propagandistic speech (i.e., i-p tokens) than it is to detect the critical b-p tokens. this is possibly a result of the highly-skewed distribution. very"), meaning that the system's enthusiasm for the i-plabel results in many misclassifications of what are actually non-propagandistic tokens. the subtask 2 classifier overwhelmingly predicts labels from a small subset (see figure 1 for details), often incorrectly labeling fragments as loaded language or doubt. when examining the gold-labeled loaded language fragments, we see high diversity in these segments, both semantically and in terms of length. there are many one-and two-word fragments, and many others that run for multiple sentences. loaded language seems to be a sort of super-category, overlapping with many other categories. many fragments that are labeled with other techniques, such as exaggeration/minimization (e.g. "their dream of independence is all but dead"), also use high-affect language that elicits an emotional response in the reader, making it easy to miss pragmatic subtleties and mislabel fragments containing loaded language as belonging to the loaded language category. a similar effect is perhaps in play in the misclassification of many fragments as doubt: almost all of the (bandwagon; reductio ad hitlerum) fragments were labeled as doubt. this could be due to similar sentiment readings across the two labels: both labels are likely to use low-valence (negative) and high-arousal (intense) lexemes. they are also likely to use many named entities of similar categories, given that the goal of both techniques is to cast aspersions on an individual or population. future iterations. one significant shortcoming of our subtask 1 implementation is its inability to address the skewed distribution of labels in the training data. at the token level, there is a marked imbalance between the number of propagandistic tokens and the number of non-propagandistic tokens. roughly 50,000 tokens are labeled as b-p and i-p in the training articles, while almost 12 million tokens are labeled n. specifically, only .004% of tokens in the training data are propagandistic. the ratio is likely to be similarly skewed in development and testing data. to address this problem in the future, we will try downsampling the n class. some of the label combinations seem to obfuscate critical characteristics of the data: for example, the combination of the previously-separate labels of bandwagon and reductio ad hiterlum. these two techniques are not similar in presentation; the reductio ad hitlerum fragments are semantically distinct from the bandwagon fragments in that they tend to contain entities known for having been terrorists or terrorized, such as adolf hitler, the jews, the palestinians, isis, henry viii, and the soviet union. bandwagon fragments do not share this characteristic. since the combination of labels seems to have been undertaken for balancing purposes instead of true similarity, our system would in the future separate these combined labels. finally, the features our system utilizes are not focused around distinguishing between the pragmatic subtleties inherent to each task. a future round of feature engineering will revolve around finding the differences between frequently misclassified labels and their predicted counterparts, and modeling these to achieve greater coverage among all tasks. ## conclusion semantic information such as word embeddings, as we have demonstrated, can be used to some small effect to detect and label propagandistic rhetoric in news articles. however, relying upon semantic and sentiment information alone is far from a complete approach to this complex problem. in the future, we will downsample the skewed subtask 1 data to maximize our classifier's performance. for subtask 2, we will create a more granular labeling taxonomy and will seek to create semantic-and discourse-level features that distinguish the more specific technique labels (e.g., thought-terminating clichés) from the more general (e.g., loaded language). taking these next steps will ideally help to advance the automatic detection of propaganda in news articles.