C yber W all E at S em E val-2020 Task 11: An Analysis of Feature Engineering for Ensemble Models for Propaganda Detection (2020)

Abstract: This paper describes our participation in the SemEval-2020 task Detection of Propaganda Techniques in News Articles. We participate in both subtasks: Span Identification (SI) and Technique Classification (TC). We use a bi-LSTM architecture in the SI subtask and train a complex ensemble model for the TC subtask. Our architectures are built using embeddings from BERT in combination with additional lexical features and extensive label post-processing. Our systems achieve a rank of 8 out of 35 teams in the SI subtask (F1-score: 43.86%) and 8 out of 31 teams in the TC subtask (F1-score: 57.37%).

## introduction propaganda is defined as "the deliberate and systematic attempt to shape perceptions, manipulate cognitions, and direct behavior to achieve a response that furthers the desired intent of the propagandist" @xcite . with the advent of rapid dissemination of news articles through online social media, automatic detection of biased or fake reporting has become more crucial than ever before. this paper describes our participation in both of the subtasks offered by @xcite in the semeval 2020 shared task for the detection of propaganda techniques in news articles. the span identification (si) subtask is a binary classification problem to discover propaganda at the token level, and the technique classification (tc) subtask involves a 14-way classification of propagandistic text fragments. to address the si task, we combine token-level bert embeddings and linguistic features with bidirectional lstms and data post-processing methods. to address the tc task, we use bert sequence embeddings and linguistic features to train a feed-forward neural network before post-processing is applied. while top-scoring teams in a similar shared task @xcite focus primarily on leveraging the performance of the pre-trained, context-dependent language model bert, we find that further encoding of linguistic features offers a meaningful boost over both bert and glove word embeddings alone. majority voting and span merging in the si task, as well as pre-and post-processing techniques to account for frequent occurrences of the repetition technique in the tc task result in further performance increases. along with our best-performing model, we provide an extensive exploration into various embedding, feature and classifier combinations. we release our code at github.com/cicl-iscl/cyberwalle-propaganda-detection. ## background before the introduction of the 2019 shared task on propaganda detection, approaches to propaganda recognition in news generally focused on classification at the article level. @xcite compare the language of "trusted news articles" from a well-known corpus with stories from "unreliable news sites". the publicly-available, feature-based tool proppy, released by @xcite , ranks articles by their likelihood of containing propagandist content. these previous systems have been prone to misclassification and lack of explainability, however, due to the assumption that articles from news sources deemed propagandist will always contain propaganda. the release of the shared task on fine-grained propaganda detection by da san martino et al. (2019a) sparked the creation of numerous new detection systems, resulting in 14 system papers published. in contrast to this year's tasks, the 2019 participants encountered sentences, rather than tokens, in the binary classification round, and were asked to differentiate 18, rather than 14 classes, in the multiclass classification round. the highest-scoring teams in that task integrate the language representation model bert (bidirectional encoder representations from transformers) into their systems. bert, developed by @xcite , was shown to reach state-of-the-art performance on eleven natural language processing tasks at the time of its release. the base version of the model for english contains 110 million parameters over 12 layers, produces embeddings of size 768 and encodes semantic information on a sub-token level. a number of the previous year's teams use linguistic features in addition to bert or other word embeddings. gupta et al. (2019) ## dataset the corpus used is an extension of the corpus described in @xcite . the articles, pulled from around 50 news outlets, have been annotated into specific techniques at the fragment level; several classes with few instances are then condensed under single labels, such as in whataboutism, straw men, red herring. an example of a fragment containing an instance of the class loaded language: not loaded language looking as though trump killed his grandma . descriptions of each class can be found @xcite . class imbalance exists strongly in the dataset. a minority of sentences are assessed to contain propaganda, and fragments that do contain propaganda are classed as only 3 of the 14 labels around 60% percent of the time. it is important to note that some of the techniques can be identified on the fragment level (e.g. name calling, flag-waving), while others often require a broader context of the discourse at hand to be discovered (e.g. repetition, red herring). ## conclusion while fair results on propaganda detection can be achieved with bert embeddings alone, we further improve the performance on this task through the addition of linguistic features and pre-and postprocessing techniques. our error analysis indicates that majority voting across several runs additionally increases the f1-score and stability of the model. future work can proceed in different directions. in our experiments for the si subtask, we applied majority voting across different runs of the same model, but it could be beneficial to use majority voting to combine predictions from the high-recall glove-based model with the high-precision bert-based model. another potential source of improvement for either subtask is retraining the bert model, similarly to @xcite and @xcite . finally, training a joint system for both subtasks, as @xcite have done, may result in more accurate predictions.