TTUI at S em E val-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles (2020)

Abstract: In this paper, we describe our approaches and systems for the SemEval-2020 Task 11 on propaganda technique detection. We fine-tuned BERT and RoBERTa pre-trained models then merged them with an average ensemble. We conducted several experiments for input representations dealing with long texts and preserving context as well as for the imbalanced class problem. Our system ranked 20th out of 36 teams with 0.398 F1 in the SI task and 14th out of 31 teams with 0.556 F1 in the TC task.

## introduction propaganda is purposeful information aiming to influence an audience to persuade an ideological or political agenda. propagandistic messages use psychological and rhetorical techniques to hide their intention and can be delivered through materials such as articles, books, images, videos, etc. there have been multiple studies addressing propaganda detection using machine learning systems @xcite @xcite . @xcite organized the nlp4if-2019 shared task on fine-grained propaganda detection extended from their previous work, which has a similar framework to this shared task. in semeval-2020 task 11: detection of propaganda techniques in news articles, the organizers provided a propaganda techniques corpus (ptc) manually annotated by six professional annotators in 446 news articles from 48 news outlets @xcite . this shared task is divided into two sub-tasks. the first sub-task is span identification (si) to identify the begin offset and end offset of propaganda fragments in text. this binary sequence tagging task is evaluated by the f1 score of the normalized length of the intersection between manually annotated and predicted spans. the second one is technique classification (tc) to label the name of the technique of a given propaganda fragment. in this sub-task, a system is required to classify the specific propaganda fragment into one of the 14 given classes. the original classes were 18 but the organizers merged five underrepresented techniques into two classes to resolve a low proportion of some techniques. this multi-class classification task is evaluated based on the micro-average f1 score. both sub-tasks share the same dataset and only the last submission on the test set is evaluated per sub-task even though there are no limitations of the number of submissions. a more detailed description of this shared task can be found @xcite . in this paper, we describe our approaches toward dealing with these problems. two state-of-the-art transformer-based language models, bidirectional encoder representations from transformers (bert) @xcite and a robustly optimized bert pretraining approach (roberta) @xcite , were used in this shared task. we fine-tuned multiple pre-trained models and merged the models using an average ensemble. we also conducted some experiments to find out which input representations preserve context better. our final performance on the test set ranked in 20th out of 36 with 0.398 f1-score for the si task and in 14th out of 31 with 0.556 for the tc task. ## conclusion we presented a description of our systems to identify propaganda fragments in news articles. the average ensemble models of bert and roberta performed better than when they were used independently in both sub-tasks. to deal with long articles in these models, we observed chunking sentences with overlapping 4 previous sentences showed the best performance among other segmentation methods. as a conclusion, our work showed competitive results in this shared task but still has limitations. training with the development dataset for the testing model should be treated carefully due to its difficulty in selecting an effective model. and predicting minority classes was another challenge in achieving high performance.