Team D o N ot D istribute at S em E val-2020 Task 11: Features, Finetuning, and Data Augmentation in Neural Models for Propaganda Detection in News Articles (2020)

Abstract: This paper presents our systems for SemEval 2020 Shared Task 11: Detection of Propaganda Techniques in News Articles. We participate in both the span identification and technique classification subtasks and report on experiments using different BERT-based models along with handcrafted features. Our models perform well above the baselines for both tasks, and we contribute ablation studies and discussion of our results to dissect the effectiveness of different features and techniques with the goal of aiding future studies in propaganda detection.

## introduction propagandist news articles are misleading in nature and aim at biasing their audience towards a particular point of view by using psychological and rhetorical techniques, including loaded language, name calling, repetition, exaggeration, minimization, etc. with the rapid growth in the number of online sources of information and the speed with which information spreads online, manual flagging of propagandist news articles has become untenable, leading to an ongoing need for new research on methods for identifying these articles automatically to mitigate the negative influence they might have on users. until very recently, most of the work in this area has focused on article-level detection @xcite . however, in 2019, @xcite published a corpus of english news articles with individual spans of propaganda annotated that addresses the problem at a more granular level. this corpus is used in shared tasks at nlp4if-2019 @xcite and at semeval-2020 @xcite . the 2020 shared task is a modified version of the prior year's task and includes two subtasks 1 : 1. span identification (si): given a plain-text document, identify those specific fragments which contain at least one propaganda technique. this is a binary sequence tagging task. 2. technique classification (tc): given a text fragment identified as propaganda and its document context, identify the applied propaganda technique in the fragment. we present our models 2 for both tasks alongside discussions of our results and ablations. ## background automatic fact-checking and propaganda detection in news articles have attracted considerable attention in recent years @xcite @xcite . many of these works focus on identifying propaganda at the article level. @xcite created a corpus of news articles with 4 different news types: trusted, satire, hoax and propaganda. they compared the language of these different types and reported classification results using lstm, max-ent, and naive bayes models. @xcite experimented on this corpus with a maximum entropy classifier to discriminate propagandist from non-propagandist articles. da san martino et al. (2019b) formulated the problem of detecting propaganda techniques in text and released a new corpus for the problem. they also presented a multi-granularity neural network for this task that outperformed several baselines. this dataset was used in the nlp4if-2019 shared task @xcite with 2 subtasks: fragment-level classification (flc) and sentence-level classification (slc). multiple systems using novel approaches were submitted. @xcite designed multi-granularity and multi-tasking neural architectures which could jointly detect sentence-and fragment-level propaganda. they also report results on different ensemble schemes. @xcite presented an ensemble of the transformer version of the universal sentence encoder @xcite and bert. many other teams also incorporated bert-base or bert-large with a linear layer or lstm @xcite @xcite . @xcite tested a model that makes indirect use of bert, but ultimately attained slightly higher performance using a logistic regression model incorporating tf-idf vectors and emotion features, among others. several other teams also experimented with handcrafted features such as part-of-speech tags, character ngrams, punctuation frequency, sentence length variance, average word length and sentiment features @xcite @xcite . the datasets for both 2019 subtasks are imbalanced. almost all teams made some attempt to address this issue. methods included undersampling, data augmentation techniques such as oversampling and synonym insertion, dropping words @xcite @xcite or adjusting the threshold for prediction probabilities @xcite . here, we describe our efforts for both subtasks of semeval 2020 task 11. for si, we experimented with 3 different model architectures and combinations of features like bert embeddings, part of speech (pos) tags, and named entities. for tc, our best model is an lstm with bert embeddings plus similar features. we were able to enhance the performance of this model via language model finetuning and data augmentation. ## conclusion we investigated several models and combinations of features to identify propaganda spans in text, and classify the techniques used within the span. for the span identification task, we found that our lstmbased model combining bert predictions with our original features gives the highest f1 score of 39.2% on the dev set. among the features we used, named entities and pos tags were found to be the most useful for this task. for the technique classification task, when using pre-trained bert, we found data augmentation helps the lstm model, however, when we finetuned bert, we got better results without using the augmented data for training. our best model for this task got an f1 score of 54.8% on the dev set. the top teams in the semeval 2020 shared task achieved an f1 score of 51.6% and 62.1% on the test set for the si and tc task respectively. these scores represent significant improvements over the baselines. nonetheless, we believe that there is still room for improvement on this challenging task, and we hope that our results, ablations, discussions, and the release of our models will help the research community in further studies.