Aschern at S em E val-2020 Task 11: It Takes Three to Tango: R o BERT a, CRF , and Transfer Learning (2020)

Abstract: We describe our system for SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. We developed ensemble models using RoBERTa-based neural architectures, additional CRF layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. We achieved sizable improvements over baseline fine-tuned RoBERTa models, and the official evaluation ranked our system 3rd (almost tied with the 2nd) out of 36 teams on the span identification subtask with an F1 score of 0.491, and 2nd (almost tied with the 1st) out of 31 teams on the technique classification subtask with an F1 score of 0.62.

## introduction the proliferation of disinformation online, commonly known as "fake news", has given rise to a lot of research on automatic fake news detection. however, most of the efforts have focused on checking whether a piece of information is factually correct, and little attention has been paid to the propaganda techniques that malicious actors use to spread their message. semeval-2020 task 11 @xcite aims to bridge this gap. it focused on detecting the use of propaganda techniques in news articles, foot_0 creating a dataset that extends @xcite , and offering two subtasks: below, we describe the systems we built for these two subtasks. at the core of our systems is roberta @xcite , a pre-trained model based on the transformer architecture @xcite . however, we improved over roberta by adding extra layers in the neural network architecture, and we further added some post-processing steps. we further applied transfer learning between the two subtasks, and finally, we combined different models into an ensemble. 2 ## related work the dataset for the task comes from @xcite , which used a bert-based model with multi-task learning and a gated architecture; the system can be tried online @xcite . there was also a related previous task on fine-grained propaganda detection @xcite , where the participants used transformer-style models, lstms and ensembles @xcite @xcite . some approaches further used non-contextualized word embeddings, e.g., based on fasttext and glove @xcite , or handcrafted features such as liwc, quotes and questions @xcite . for the fragment-classification subtask (a combination of two subtasks of the current semeval) the lstm-crf @xcite or bilstm-crf @xcite models were applied besides bert @xcite . moreover, some efforts have been made to increase the size of the dataset using unsupervised language model pre-training @xcite . finally, there is a recent survey on computational propaganda detection @xcite . ## our systems in this section, we provide a general overview of our systems for the two subtasks. for both subtasks, we trained ensembles based on roberta with some postprocessing. ## experimental setup data we experimented with the training, the development and the test datasets provided for semeval-2020 task 11, which contain 371, 75, and 90 news articles with 6,128, 1,063 and 1,790 spans, respectively. we randomly selected 20% of the training dataset for local evaluation when developing our models. ## conclusion and future work we described the system we developed for the semeval-2020 task 11 on detection of propaganda techniques in news articles. we developed ensembles of roberta-based neural architectures, additional crf layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. we achieved sizable improvements over baseline roberta models, and the official evaluation ranked our system 3 rd (almost tied with the 2nd) out of 36 teams on the span identification subtask, and 2 nd (almost tied with the 1st) out of 31 teams on the techniques classification subtask. in future work, we plan to explore other neural architectures such as t5 @xcite and @xcite . we further want to explore transfer learning from other tasks such as argumentation mining @xcite and offensive language detection @xcite .