P suedo P rop at S em E val-2020 Task 11: Propaganda Span Detection Using BERT - CRF and Ensemble Sentence Level Classifier (2020)

Abstract: This paper explains our teams’ submission to the Shared Task of Fine-Grained Propaganda Detection in which we propose a sequential BERT-CRF based Span Identification model where the fine-grained detection is carried out only on the articles that are flagged as containing propaganda by an ensemble SLC model. We propose this setup bearing in mind the practicality of this approach in identifying propaganda spans in the exponentially increasing content base where the fine-tuned analysis of the entire data repository may not be the optimal choice due to its massive computational resource requirements. We present our analysis on different voting ensembles for the SLC model. Our system ranks 14th on the test set and 22nd on the development set and with an F1 score of 0.41 and 0.39 respectively.

## introduction in contemporary times, fake news and propaganda have gained a lot of traction. a contributing factor to these problems is the easy dissemination of information on social media and various alternative news outlets on the internet which house a vast repository of content which is tough to effectively moderate. propaganda is often used to promulgate news articles or content that is misleading. in conjunction, fake news not only contrives hysteria and spreads lies, in extreme cases it leads to physical violence @xcite . most of the workaround propaganda detection has been limited to document-level classification @xcite @xcite . in the past, shared tasks such as the nlp4if 2019 have dealt with sentence level classification (slc) and fragment level classification (flc) of propaganda @xcite . fine-grained propaganda techniques provide a more suitable method of detecting propaganda because its classification provides the reasoning behind why an instance has been flagged as propaganda. the semeval shared task 11 makes progress in this aspect with its two tasks namely, span identification (si) that has the objective of finding propaganda spans, and technique classification (tc) that labels the propaganda technique employed in a propaganda span @xcite . in this paper, we have focused on the span identification task that involves character level tagging of propaganda spans in text. to achieve this, we used ensemble transformer-based architectures to first perform slc which is followed by token level tagging of spans of only propaganda sentences. these are fed to the bert-crf span identification model, predictions of which are later processed to obtain the character level tagging of propaganda fragments. in addition to this, we carry out various experiments to deal with the class imbalance in the provided data corpus and obtain a generic model that can be employed to detect propaganda fragments in any text. the remaining paper is organised as follows. section 2 gives a background on existing work on propaganda detection, the task we worked on and the novelty of our approach. section 3 provides the rationale behind the system setup and the system's working. following this, section 4 describes the experimental setup to expatiate the basis on which we conducted our experiments. the next section, section 5, provides the results of all our enlisted experiments and our final system along with the final model configurations that can be used to replicate our results. section 6 finally concludes the paper and presents our error analysis. ## related work the span identification task aims at providing a more fine-grained analysis of propaganda in text. owing to the massive importance of regulating the quality of content being circulated among the populace, researchers have explored several techniques for achieving sequence labelling and sentence level classification, both of which provide an important background for the si task. the task organizers provide a corpus of 550 news articles that are labelled against character level offsets of the propaganda spans within the articles. it is a binary sequence tagging task. the annotation is done manually. the example below explains it clearly where character offsets from 19 to 40 contains a propaganda span "nefarious connections". for sentence level classification, rashkin (2017) used an lstm model and presented a comparison of its performance with naive bayes and maximum entropy models. da san martino (2019) used multi-granularity bert for fine-grained propaganda detection. work by @xcite demonstrated the use of lstms in sequence tagging. to further this work and leverage the learning from both the future and past inputs in a sequence, @xcite discussed the use of bi-lstms. recently, @xcite proposed bilstm-crfs which promise bidirectional comprehension whilst making use of the sentence level tag information mapped by the crf layer. crf's efficacy was further demonstrated by @xcite who reported higher f1 scores in ner with four different languages without leveraging any knowledge specific to those languages. ## system overview we adopt a two-step method of detecting propaganda spans by first performing slc and then detecting spans in sentences which have been flagged as propaganda sentences. comparing the results of span identification: with and without slc -we observe an f1 score improvement of nearly 0.13 using the former method. ## results in this section, we present the results of all the experiments discussed in section 4 which justify our choice of the proposed model and its configurations. the metrics used in the results are the same metrics used to evaluate the task results for the leaderboard foot_1 . in as observed in as observed, the xlnet-roberta ensemble produced the best f1 score for the si task and hence it was employed in our final model pipeline. we also attempted to use the bert -large for the task but achieved an f1 of 0.35. additionally, we analyzed several hyperparameters including sets of higher learning rates such as that of 1e-4 (produced f1 -0.38), more number of training epochs such as 20 epochs producing an f1 score of 0.372 and smaller batch sizes such as 8 which resulted in an f1 score of 0.377. having analysed all these results -our proposed model's configurations were decided as shown in in the future, we intend to explore more advanced and efficient transformer models including t5 and reformer respectively.