Inno at S em E val-2020 Task 11: Leveraging Pure Transfomer for Multi-Class Propaganda Detection (2020)

Abstract: The paper presents the solution of team ”Inno” to a SEMEVAL 2020 task 11 ”Detection of propaganda techniques in news articles”. The goal of the second subtask is to classify textual segments that correspond to one of the 18 given propaganda techniques in news articles dataset. We tested a pure Transformer-based model with an optimized learning scheme on the ability to distinguish propaganda techniques between each other. Our model showed 0:6 and 0:58 overall F1 score on validation set and test set accordingly and non-zero F1 score on each class on both sets.

## introduction modern society is experiencing an overflow of different kinds of information. thousands of news media are publishing a giant number of articles, blogposts, twits and other kind of content every day and even every hour. people who want to stay informed about global situation are often not able to distinguish which source is reliable and provides relevant content. therefore, such topics as automatic fake news detection, sentiment analysis and propaganda detection are gaining increasing attention. the authors of (jowett and o'donnell, 2018) defined propaganda in the most neutral sense as dissemination or promotion of a particular idea. this paper provides our solution to semeval 2020 task 11 "detection of propaganda techniques in news articles". the task consists of two subtasks: (i) a propaganda sample span identification and bounding and (ii) a classification of these spans into one of the 18 classes (or techniques). some techniques in the second subtask were merged into one class to get 14 classes for prediction in the end. all the techniques, as well as the other details about the task, are described by the task organizers in @xcite . in this paper we present a solution to the second subtask. the suggested system 1 consists of a transformer-based classifier with additional learning optimization techniques such as undersampling, cost-sensitive learning and context addition, aimed at robustness. this approach allowed us to achieve 0.6 and 0.58 overall f1 scores with non-zero f1 score for each class on validation set and test set accordingly. we took the 7th place out of 31 teams on the second subtask. ## background one of the first classifications of propaganda techniques was proposed in 1936 by clyde r. miller @xcite after a presidential election in the usa. this categorization consists of description of 7 propaganda techniques: "name calling", "glittering generalities", "transfer", "testimonial", "plain folks", "card stacking" and "band wagon". since then a lot more techniques have been identified, but this classification is still relevant (some of the techniques have been included in the semeval 2020). probably, the most promising state-of-art strategies of dealing with propaganda and filtering it from sentimentally neutral texts is the one of viewing the task as a text classification problem. some of the research efforts in propagandist texts classification are described further in this section. ## system overview considering the top-performing systems of the previous challenge, we decided to use transformer-based model as a solution baseline -mainly focusing on bert @xcite and roberta @xcite . however, since a lot of new models have been published since the previous challenge, we also tested several different transformer-based models, including distilbert @xcite , albert @xcite , xlnet @xcite . after calculating f1 scores for all of them, we chose the one with the highest performance -roberta. we also decided to use a model pretrained on a huge dataset and fine-tune it on the given dataset by adding a fully-connected classification layer. this strategy is highly suitable for our case because the training dataset consisted of 371 articles and contained 6129 propaganda samples in total for all classes, which is not enough for training a good-performing transformer model from scratch, but is reasonable for fine-tuning. the following sections describe the techniques we used for optimizing the robustness of our model. ## experimental setup and results to choose between different transformer-based models we trained each one of them on a single epoch and compared their validation scores. the result of this evaluation are shown in according to as the next step, we tuned the hyperparameters of the model including the number of epochs, batch size and maximum sequence length. this step allowed us to improve f1 score for 2 points and provided us with a ready-to-run model so that we could proceed to optimization techniques described in previous section.