NTUAAILS at S em E val-2020 Task 11: Propaganda Detection and Classification with bi LSTM s and ELM o (2020)

Abstract: This paper describes the NTUAAILS submission for SemEval 2020 Task 11 Detection of Propaganda Techniques in News Articles. This task comprises of two different sub-tasks, namely A: Span Identification (SI), B: Technique Classification (TC). The goal for the SI sub-task is to identify specific fragments, in a given plain text, containing at least one propaganda technique. The TC sub-task aims to identify the applied propaganda technique in a given text fragment. A different model was trained for each sub-task. Our best performing system for the SI task consists of pre-trained ELMo word embeddings followed by residual bidirectional LSTM network. For the TC sub-task pre-trained word embeddings from GloVe fed to a bidirectional LSTM neural network. The models achieved rank 28 among 36 teams with F1 score of 0.335 and rank 25 among 31 teams with 0.463 F1 score for SI and TC sub-tasks respectively. Our results indicate that the proposed deep learning models, although relatively simple in architecture and fast to train, achieve satisfactory results in the tasks on hand.

## introduction propaganda is the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends @xcite . it is often combined with misinformation and fake news and all together have the potential to polarise public opinion, to promote violent extremism and hate speech. to deal with this problem, automatic identification and categorisation of propaganda, fake news and hyperpartisan content has been heavily addressed in recent research, mainly in the article level @xcite @xcite . this task is a follow-up to the nlp4if shared task on fine-grained propaganda detection @xcite and aims to produce models capable of spotting propaganda techniques in text fragments and then categorise them in one or more of 14 propaganda techniques. the first sub-task is called span identification (si) and its goal is to spot propaganda fragments given a plain text document. the evaluation metric for this task is a modified f1 measure appropriate for taking into account partial matching between the spans. detailed information about this metric can be found in @xcite . the second sub-task which is called technique classification (tc) consists of text fragments as inputs, labelled as one or more of 14 propaganda classes such as name calling and loaded language. the goal is to classify the fragments to one or more of those techniques. due to the fact that the distribution of the techniques in gold labels is rather imbalanced, results were evaluated with the micro-averaged f1 measure. in the current paper, we propose two novel deep learning architectures to deal with the two subtasks of the competition. in addition, we compare our deep learning methods with classic machine learning methods such as logistic regression provided with word embeddings pre-trained on large external corpora. transfer learning architectures showed promising results in various natural language processing (nlp) tasks @xcite @xcite , so there was a strong intuition to try this kind of architectures in this demanding task as well. the rest of the paper is organised as follows. section 2 describes the previous related research work in the field of propaganda and fake news detection. section 3 describes the data and the system overview. section 4 analyses the results and the errors of the methods proposed. finally, section 5 draws conclusions and suggests future work and improvements in the existing one. ## related work a very similar task was shared previously in 2019 called fine-grained propaganda detection. this was a part of natural language processing for internet freedom (nlp4if'19) workshop and included to sub-tasks, sentence level classification (slc) and fragment level classification (flc). the dataset used is a subset for this task's dataset and flc was basically a combination of si and tc in terms of including both detection and classification of text fragments. this was the first time propaganda and fake news detection was addressed in a fragment level. previous work in propaganda and hyperpartisan news have been tackled mostly in the article level @xcite . in hyperpartisan news detection task at semeval 2019 two datasets were used. one relatively small labelled manually and one large corpus suitable for deep learning methods that was labelled using distant supervision, assuming that all articles from a given news outlet share the label of that outlet. @xcite created a corpus of news articles labelled as propaganda, trusted, hoax or satire with distant supervision as well. apparently, this method introduces noise, and in the hyperpartisan news detection task a lot of participating teams claimed that using the large corpus resulted in worse performance than not using it at all. ## conclusion and future work in this paper, we presented methods which combined transfer learning and recurrent neural networks capable of detecting and classifying propaganda fragments in news articles. in the competition various techniques and architectures were explored and the results show that deep learning methods improve performance as was initially expected. for future work, we plan to explore different methods such as bert based sequence tagger and to experiment with techniques that will help to tackle the class imbalance problem, mainly in the tc task.