HOMADOS at S em E val-2021 Task 6: Multi-Task Learning for Propaganda Detection (2021)

Abstract: Among the tasks motivated by the proliferation of misinformation, propaganda detection is particularly challenging due to the deficit of fine-grained manual annotations required to train machine learning models. Here we show how data from other related tasks, including credibility assessment, can be leveraged in multi-task learning (MTL) framework to accelerate the training process. To that end, we design a BERT-based model with multiple output layers, train it in several MTL scenarios and perform evaluation against the SemEval gold standard.

## introduction fine-grained propaganda detection is a new approach to tackling online misinformation, highlighting instances of propaganda techniques on the word level. these techniques are used in textual communication in order to encourage certain beliefs, but instead of straightforward presentation of arguments, they rely on psychological manipulation, logical fallacies or emotion elicitation. there are general-purpose natural language processing (nlp) methods that could be used for automatic detection of such text fragments. the challenge here is that they require large amounts of training data, which are laborious to produce. however, propaganda techniques are often related to other misinformation challenges, for which large datasets do exist, e.g. credibility assessment or fake news detection. in the present study we aim to investigate how this connection can be used in the multi-task learning (mtl) framework. we show how the performance of multi-label token-level propaganda detection within shared task 6 at semeval-2021 can be improved by building neural architectures that are also trained to solve other tasks: singlelabel propaganda detection from semeval-2020 and document-level credibility assessment based on a fake news corpus. we check different mtl scenarios (parallel and sequential) and show which aspects of the model benefit the most from this approach. ## problem statement we participate in semeval-2021 task 6 (,,detection of persuasion techniques in texts and images"), subtask 2 @xcite , where the goal is to identify all propaganda techniques within a provided fragment of text. specifically, given a character sequence c 0 , c 1 , . . . , c n , we aim to produce a set of annotations {(b 0 , e 0 , t 0 ), (b 1 , e 1 , t 1 ), . . . , (b k , e k , t k )}, where each triple consists of the character offsets of the span it covers (0 ≤ b @xmath0. we can see it as a multi-label sequence classification task @xcite , where each character (or token) can be assigned from 0 to 20 labels. ## related work propaganda has been observed in text for a long time, but the problem of automatic detection of such techniques was posed just recently. initially, a lack of word-level datasets confined the analysis to document-level classification, e.g. based on stylometric features @xcite . classification on the word level became possible with the dataset @xcite released for the ,,fine-grained propaganda detection" shared task at the nlp4if 2019 workshop @xcite . the corpus includes 550 news articles annotated with propaganda techniques on the word level. among the submissions, the best performing models were based on word embeddings and pretrained lan-guage models, such as bert @xcite . to tackle the data sparsity problem, participants employed various over-sampling methods or trained their models on auxiliary data. similar objectives were pursued at semeval 2020 task 11, consisting of two subtasks: binary sequence tagging task and multi-class classification task. the majority of tasks' participants based their solutions on the transformers architecture @xcite and word embeddings, combining them with other neural architectures (e.g. cnns or lstms) or models such as crf and logistic regression. ensemble models were able to achieve satisfactory results when performing both tasks jointly. ## results ## conclusion in this work we explore how detection of propaganda techniques in text of memes can be facilitated using external data in multi-task learning framework. the results show that the auxiliary tasks indeed influence the results, both in terms of accelerating the learning process and changing the set of recognised techniques. nevertheless, these modifications do not offer clear advantages over the basic bert-based solution. we hypothesise this is because the link between main and auxiliary tasks is not strong enough to deliver benefits through multi-task learning. additionally, propaganda is rarely a straightforward phenomenon and different techniques may require tailored approaches. we treat this effort as a preliminary study and aim to further investigate mtl's relevance in detecting propaganda by extending the auxiliary tasks portfolio with corpora reflecting other related issues, such as hate speech or hyperpartisan language.