AP att at S em E val-2023 Task 3: The Sapienza NLP System for Ensemble-based Multilingual Propaganda Detection (2023)

Abstract: In this paper, we present our approach to the task of identification of persuasion techniques in text, which is a subtask of the SemEval-2023 Task 3 on the multilingual detection of genre, framing, and persuasion techniques in online news. The subtask is multi-label at the paragraph level and the inventory considered by the organizers covers 23 persuasion techniques. Our solution is based on an ensemble of a variety of pre-trained language models (PLMs) fine-tuned on the propaganda dataset. We first describe our system, the different experimental setups we considered, and then provide the results on the dev and test sets released by the organizers. The official evaluation shows our solution ranks 1st in English and attains high scores in all the other languages, i.e. French, German, Italian, Polish, and Russian. We also perform an extensive analysis of the data and the annotations to investigate how they can influence the quality of our systems.

## introduction due to its diffusion on social media platforms, information is being shared in real-time in the modern digital world, in particular following the covid-19 pandemic. the detection of propaganda has emerged as an important research topic with the recent interest in fake news @xcite . in today's information age, anyone can take advantage of the diffusion potential of internet to spread propaganda surreptitiously, and this is in fact done by activist groups, businesses, religious institutions, the media, and even ordinary people, reaching vast audiences. many different strategies are used to "hide" propaganda messages inside standard text, ranging from appealing to the audience's emotions to utilizing logical fallacies. semeval-2023 task 3 @xcite ) subtask 3 offers a different way to investigate this problem: given a news article selected from a media source that could potentially spread disinformation, the system is tasked to identify the persuasion techniques adopted in each paragraph. this is a multi-label task at the paragraph level and also multilingual, with news articles in 6 different languages: english, italian, polish, german, french and russian. in our approach, we adopt transfer learning and, instead of using one single pre-trained language model, we create an ensemble of multiple state-ofthe-art architectures. to output the final prediction two different approaches are used. for all the languages except english it is possible to use a maximum of two models, so for every ensemble we take the probabilities of these models and average them to make the final prediction. in english, instead, where it is possible to use more than two models for the ensemble, we do not average but weight the predictions of each model. we notice that some models have better results on a subset of classes, so we assign them a higher weight on the corresponding classes in the final output. our contribution is summarized as follows: the remainder of this paper is organized as follows: in section 2 we review some recent work in the current literature. in section 3 we provide the details of our method, while in section 4 we describe the experimental setup. in section 5 we demonstrate the effectiveness of the proposed approach by showing and discussing its results. we provide conclusions in section 6. ## related work propaganda detection is a crucial topic for research in the field of natural language processing because it may be useful for additional applications like spotting fake news on social media. @xcite used the representation power of graphs to detect strange behaviours of groups of people. they noticed that propaganda techniques are applied when there are strange patterns on the adjacency matrix and in the spectral subspaces of the corresponding graph. @xcite tried to capture the main style of propaganda with a graph-based three-stage pipeline: they first built a conversation graph about a topic, then they partitioned the graph to identify potential controversies and finally they measured the amount of controversy from features of the graph. @xcite developed a technique to determine the "degree" of propaganda in a piece of writing and integrated their model into a website that groups recent articles about the same event based on their propagandistic content. with a similar approach @xcite implemented a system, called protect, to automatically detect propagandist messages and classify them along with the propaganda techniques employed. protect allows users to input text and retrieve the propagandist spans in the message as output. they used the roberta plm and then performed a post-processing step to automatically join tokens labelled with the same propaganda technique into the same textual span @xcite focused on more in-depth analysis, putting forward a more theoretical paradigm for identifying and categorizing propaganda campaigns. they first created a corpus of manually annotated news articles at the fragment level and then they proposed a suitable evaluation measure for this task. a major limit of all the above cited previous work is their focus on the english language. with this work we aim to study propaganda techniques in a multilingual setup by employing ensemble methods. ## results we first report results on the english language. the introduction of weighting the predictions improved our results. as we can see from for the other languages we obtained good results in german and polish, as we can see from the performance and the corresponding ranking shown in ## conclusion in this paper, we examined the ability of transformer-based language models, and in particular weighting the predictions of multiple models, to carry out the task of detecting propaganda techniques in different languages. we showed how the class imbalance of a dataset can influence the performance of a model on this task. future work includes examining more effective methods for solving the previous issue and also optimizing the value of the weights on the dev set to improve the performance. for the russian language both roberta and bert were pretrained on the taiga corpus. for the polish language bert was pretrained on polish wikipedia, a polish parliamentary corpus, opus @xcite and oscar @xcite . for the french language bert was pretrained on the french wikipedia corpus. roberta was pretrained on the french wikipedia corpus and on cc-news. finally for the german language bert was pretrained on the german wikipedia and openlegal-data corpora, as distilbert.