S koltech NLP at S em E val-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection (2020)

Abstract: This paper presents a solution for the Span Identification (SI) task in the “Detection of Propaganda Techniques in News Articles” competition at SemEval-2020. The goal of the SI task is to identify specific fragments of each article which contain the use of at least one propaganda technique. This is a binary sequence tagging task. We tested several approaches finally selecting a fine-tuned BERT model as our baseline model. Our main contribution is an investigation of several unsupervised data augmentation techniques based on distributional semantics expanding the original small training dataset as applied to this BERT-based sequence tagger. We explore various expansion strategies and show that they can substantially shift the balance between precision and recall, while maintaining comparable levels of the F1 score.

## introduction propaganda is one of the primary indicators of false news. therefore, the task of detecting and classifying propaganda is an important one in the field of fake news detection. @xcite presented a new dataset for propaganda detection. the main advantage of this dataset is the markup at the level of individual text fragments. in addition to binary markup (is/is not propaganda), there is also a multiclass markup, which includes 18 different propaganda classes. using this dataset, the span identification (si) task of detection of propaganda techniques in news articles task requires to determine specific text fragments which contain at least one propaganda technique. for instance, in the sentence "manchin says democrats acted like babies at the sotu (video) personal liberty pollexercise your right to vote" the part from the 34-th through the 40-th character (i.e., word "babies") belongs to the name-calling and labeling class, so it should be marked as propaganda. this paper describes the solution by "skoltechnlp" team which took part in the si task of the competition and achieved a score f 1 = 0.34 on the test set evaluation. our solution is based on bert @xcite that is specially pretrained for the sequence tagging task. however, such architectures usually require large datasets for fine-tuning: for instance, @xcite consists of 976 articles in the training set, in comparison with 371 articles in the training set for the propaganda detection task. therefore, we conducted a study of the dataset expansion with several augmentation techniques. we built a number of strategies, performing investigation of various combinations of such parameters as: the size of the enlarged dataset, models for word representations, words' part of speech, and class for substitution. all codes for reproducing the results are openly available online. 1 the rest of the paper is organized as follows. section 2 covers several previous approaches to this problem. section 3.1 presents our final solution and describes model architecture used, preprocessing, and implementation details. section 4 provides the description and the results of the dataset expansion experiments. finally, section 5 concludes this report. since in the previous competition the participant with successful solutions focused more on pre-trained contextualized models, we also decided in our approach to focus on such models, bert in particular. moreover, as data augmentation applications were used in previous works in propaganda detection @xcite and also have shown significant results in other fields like computer vision @xcite , it seems promising to continue research in this direction. ## method we approach the problem as named entity recognition (ner) problem with two classes -inside and outside of propaganda. since models for such a task usually use token classification and the markup was made on a character level, firstly, we made a preprocessing step that converts char-level markup into token-level markup. at the post-processing step, we did the reverse transformation of the markup. the pipeline of our final solution is presented in figure 1 . ## data augmentation we hypothesized that relatively low results obtained by the baseline model could be due to the reasons that, (i) one the one hand the semantics of the phenomenon at hand is complex and (ii) on the other hand, the training dataset is too small for even fine-tuning. therefore we decided to perform automatic augmentation on the provided dataset to try reaching better generalization and more stable training of the baseline model. ## conclusion we presented the solution of "skoltechnlp" team for the span identification task in the semeval-2020 task 11 competition. our final solution is based on the bert masked language model, specially pretrained for the ner task, which showed strong performance out-of-the-box with respect to the baseline. in addition, we investigated various strategies for the dataset augmentation on the public set of our best model, trained on the expanded text datasets. unfortunately, this approach did not give a significant increase of the f1 score. however, it was shown that the proposed strategies can substantially improve precision if words from the target "propaganda" class are expanded and improve recall if substitutions for the neutral class is used to generate new training examples. therefore, the developed expansion methods could be useful for shifting "sweet spot" of a classification model between precison and recall maintaining similar f1 level. as future work, more careful search on hyperparameters for augmentation can be considered. for example, in this work we just manually selected the ratio of dataset increase (x2, x5, x10), however, this parameter can be searched on the scale of natural numbers. the same can be done with the ratio of words selected for substitution. we selected 70% ratio from our own conclusions of the diversity of the new contexts obtained, but this number can also vary. another aspect not covered in the work is the imbalance of classes. in this case, expanding the data to balance examples in both classes, as well as redistributing class weights, can be useful for obtaining a more stable model.