UPB at S em E val-2020 Task 11: Propaganda Detection with Domain-Specific Trained BERT (2020)

Abstract: Manipulative and misleading news have become a commodity for some online news outlets and these news have gained a significant impact on the global mindset of people. Propaganda is a frequently employed manipulation method having as goal to influence readers by spreading ideas meant to distort or manipulate their opinions. This paper describes our participation in the SemEval-2020, Task 11: Detection of PropagandaTechniques in News Articles competition. Our approach considers specializing a pre-trained BERT model on propagandistic and hyperpartisan news articles, enabling it to create more adequate representations for the two subtasks, namely propaganda Span Identification (SI) and propaganda Technique Classification (TC). Our proposed system achieved a F1-score of 46.060% in subtask SI, ranking 5th in the leaderboard from 36 teams and a micro-averaged F1 score of 54.302% for subtask TC, ranking 19th from 32 teams.

## introduction propagandistic texts can be perceived as deliberate acts of spreading ideas with the self-serving purpose of influencing, distorting, and manipulating the opinions of other individuals or groups @xcite . propagandistic texts use several techniques to achieve these goals -for example, using emotionally charged statements to persuade the audience not on the merits of the argument, but based on the visceral reaction to it. thus, early detection and the labeling of such texts is of utmost importance to lessen the negative impact on the society and diminish the spread of such news through online social media. semeval-2020 task 11 proposes the detection of propagandistic text sequences in english news articles, including a span detection subtask (subtask si), followed by a fine-grained propaganda classification into 14 classes (subtask tc) @xcite . the competition was divided into three phases. participants had access to the training set with full golden labels and a dev set without the golden labels. during the development phase, participants could submit the results on the dev set for both the si and tc subtasks, and receive an instant score. in contrast, the test set submissions received the scores only at the end of the competition. our solution builds upon bert @xcite , a pre-trained transformer-based @xcite language model composed of several (12 or 24) multi-head attention blocks. the pre-trained model provided by google was further pre-trained to create a domain-specific model using masked language modeling on a corpus consisting of news articles. this approach was used in narrow domains, like scientific papers @xcite or bio-medical data @xcite , with significant performance gain in domain-specific tasks. the rest of the paper is organized as follows. section 2 presents work related to the propaganda detection problem. section 3 provides details on the dataset, its processing, together with model implementation details. afterwards, we discuss the achieved results in section 4, followed by conclusions in the last section. in addition, the shared tasks of previous editions of semeval tackled the issue of hyperpartisan news @xcite with remarkable results. approaches ranged from traditional nlp models like support vector machines, k-nearest neighbor, naive bayes, logistic regression @xcite , n-grams @xcite to transformer-based approaches @xcite . recently, da san martino et al. (2019) ## results tables 1 and 2 present the results of the si subtask. we can observe that all bert-based models surpassed bilstm-crf with a large margin on the development set. also, the split did not impact the bert model. however, our improved bertp model could gain better precision by splitting the articles into larger chunks, while maintaining the same recall as for the sentence level split. our best model (i.e., bertp-crf, ps) achieved a precision of 58.612% and a recall of 37.936%, totaling a f1-score of 46.06% and reaching the 5th place from 36 teams on the leaderboard. we experimented with several article splitting strategies for the subtask tc, on a sentence or paragraph basis, but the best performance was achieved by taking only the propagandistic span as training data. the additional context did not support a better classification (see ## conclusions this paper introduces our bert-based model for both propaganda span detection, as well as propaganda technique classification. further pretraining of the bert model using large domain-specific corpora improves propaganda detection and classification performance. experimenting with different text splitting methods has shown that even though for the original bert model these have no significant influence, our bertp model benefits from the larger context window. the imbalanced dataset proved to be the largest challenge for the classification task. an ensemble with multiple checkpoints in the bert pretraining process obtained better classification results for underrepresented classes. in our future work we will analyze the impact of adding meta-features from named entity recognition on the performance of the propaganda span detection. also, experiments with distilled versions of bert, or bert models with a reduced number of attention heads, are envisioned.