UMSIF oreseer at S em E val-2020 Task 11: Propaganda Detection by Fine-Tuning BERT with Resampling and Ensemble Learning (2020)

Abstract: We describe our participation at the SemEval 2020 “Detection of Propaganda Techniques in News Articles” - Techniques Classification (TC) task, designed to categorize textual fragments into one of the 14 given propaganda techniques. Our solution leverages pre-trained BERT models. We present our model implementations, evaluation results and analysis of these results. We also investigate the potential of combining language models with resampling and ensemble learning methods to deal with data imbalance and improve performance.

## introduction propaganda techniques are used to promote or publicize a particular political cause or point of view, especially of a biased or misleading nature @xcite . to achieve its desired outcome, psychological and rhetorical techniques are frequently used. while initially people tend to agree with propaganda messages due to their misuse of logic and/ or arousal of emotional response, they later change their opinion and realize arguments are not convincing. indeed, for maximum effect propaganda techniques are intended to go unnoticed. it is therefore important to detect propaganda in initial stages and identify the specific propaganda techniques used. by successfully detecting and classifying propaganda, people can look at information more rationally and logically. this paper describes the solutions towards the techniques classification (tc) task of the semeval 2020 task 11 "detection of propaganda techniques in news articles" competition. the task requires classifying textual fragments that relate to at least one of the 14 given propaganda techniques. our solutions leverage the pre-trained bert @xcite based classifier. moreover, fine-tuning allows us to conveniently adjust its pre-trained bottom layer weights on the given propaganda detection corpus shared by the organizers. among the 46 participating teams, our submission is ranked 17. the rest of the paper is organized as follows. in section 2 we give the problem definition. in section we present the propaganda dataset, in section 4 we describe our solution and elaborate on the architectural and implementation details of the model used, in section 5 we present results achieved by our best submission and analyze of these results, and finally in section 6 we conclude with directions for future work. ## problem definition given a document and a propaganda-related text excerpt from the document, the task is to identify the specific propaganda technique present in the text excerpt from the 14 propaganda classes available @xcite . text excerpts can be overlapping and reflect multiple propaganda techniques simultaneously, therefore the propaganda identification task needs to be approached as a multi-label multiclass classification problem. in practice, for cases when multiple propaganda techniques are used in the same text fragment, the organizers created as many copies of the respective text fragment equal to the number of propaganda techniques present in the fragment. this allows us to approach the problem as a single-label multi-class machine learning classification problem. ## dataset the data shared with us by the competition organizers contains 8,982 plain text articles retrieved from the newspaper3k foot_1 library and divided into training/ validation/ testing sets. the training set contains propaganda labels, while for the validation and testing articles we need to predict the correct labels. in each article in the dataset is made up of a title specified on the first line, followed by an empty second line, and the article content starting from third line onwards, one sentence per line. in figure 1 we include an example of a training set article from the propaganda detection corpus for which the propaganda labels are given. for instance, the term "babies" on the first line denotes name calling, labeling type of propaganda. on the fourth line, "stupid and petty" is loaded language propaganda and "not looking as though trump killed his grandma" is an instance of exaggeration and minimisation. in figure 2 we include an example of a test article from the propaganda detection corpus for which we need to identify the propaganda techniques present. ## method we address propaganda detection as a multi-class text classification problem, and our solution relies on the pre-trained bert @xcite model. we choose bert since it is already pre-trained on large amounts of data and has demonstrated strong performance in a multitude of natural language processing tasks including text classification. we fine-tune bert-base on the textual fragments delimited by start and end indices from articles in the training set for which we know the correct labels. we apply basic text pre-processing techniques, such as lowercasing and tokenization. we also convert the textual fragments into a format compatible with bert by adding special tokens to mark the start ([cls]) and end ([sep]) of each sentence, pad sentences with special token @xcite to ensure all have the same length of 64 tokens, and differentiate real tokens from padding tokens using a mask list which contains 0 for padding tokens and 1 for all other tokens. in figure 3 we include an example of a propaganda textual fragment after pre-processing. not all 14 classes are equally represented in the propaganda dataset, and to alleviate data imbalance issues we use oversampling and undersampling techniques. we oversample the number of examples for minority classes, and undersample the number of examples for majority classes; please see figure 4 . when oversampling, we resample each class containing less than 400 samples to 400 samples by bootstrap sampling, i.e random sampling with replacement. when undersampling, we resample every class for which the number of samples is greater than 400 to 400 samples also by bootstrapping. we choose 400 as the number of samples per class given there are 6,129 propaganda samples in total in the training data and the goal is to have all 14 propaganda classes equally represented. the data obtained after the resampling process is used in combination with bagging (quinlan and others, 1996). a total of nine bert based models are assembled on different subsets of the training data with randomly sampled examples. each training subset contains 400 examples per propaganda class, therefore each bert bagging model is trained on 5,600 training examples (14 classes x 400 examples/@xmath0,600 total examples). in figure 4 we present the architecture of bert models used with bagging. we implement our solution in pytorch and use huggingface (wolf et al., 2019) library. we train the bert-base-uncased model with 12 transformer (vaswani et al., 2017) layers and 110 million parameters. we chose model hyper-parameters by doing grid search and for our submission we use the following values: max sequence length set to 64, batch size equal to 32, a learning rate of 2e-5 and adam (kingma and ba, 2014) as the optimization algorithm. we use 4 fine-tuning epochs with 10-fold cross validation on the training data set provided by competition organizers. we optimize for the f1 score in our evaluation. ## result and analysis in what follows we present results for our final bert-based submission which ranks 17 among 46 participating teams. our final run achieves 0.5729 validation f1-score and 0.5514 testing f1-score. comparing our submission with the performance of other teams in the competition, we observe that submissions ranked 4-17 achieve comparable f1-scores to ours (team 4 reports 0.589 f1 score), and that teams ranked 1-3 achieve higher f1 scores than us (0.617 f1 score and beyond). next we assess the performance of our run at finer granularity for each propaganda technique. in table in ## conclusion and future work we have presented our solution for the semeval 2020 task 11 "detection of propaganda techniques in news articles" -techniques classification task. we find that fine-tuning the pre-trained bert-base model on the propaganda detection corpus provides a solid approach given all external knowledge it incorporates. we aim to improve on that by using ensemble methods such as bagging and training multiple instances of bert on different subsets of the training data assembled with oversampling and undersampling. we find that bert with bagging improves f1-scores for classes underrepresented in the training data. in future work we plan to investigate systematic ways to tune the model hyperparameters when bagging and combine the pre-trained bert language model with other ensemble learning methods, such as boosting and stacking. in addition, we plan to explore the application of more advanced oversampling and undersampling methods such as smote @xcite .