On Experiments of Detecting Persuasion Techniques in P olish and R ussian Online News: Preliminary Study (2023)

Abstract: This paper reports on the results of preliminary experiments on the detection of persuasion techniques in online news in Polish and Russian, using a taxonomy of 23 persuasion techniques. The evaluation addresses different aspects, namely, the granularity of the persuasion technique category, i.e., coarse- (6 labels) versus fine-grained (23 labels), and the focus of the classification, i.e., at which level the labels are detected (subword, sentence, or paragraph). We compare the performance of mono- verus multi-lingual-trained state-of-the-art transformed-based models in this context.

## introduction nowadays, readers of online content are exposed more than ever to manipulation, disinformation and propaganda, which can potentially influence their opinion on relevant topics, such as, e.g., elections, health crises, migration crises, military conflicts, etc. thus, the analysis of online media landscape is essestial in order to get a deeper insight on the presented narratives around certain topics across countries, to detect and identify manipulation attempts and to enchance users' media literacy. as a result, in the recent years, one could observe an ever-growing trend of research on automated methods supporting the detection of potentially deceptive and manipulative content, on narrative extraction, and on tools for comparative analysis of online media of different political orientations. in this paper, we present the results of some preliminary experiments on the detection of persuasion techniques in online news in polish and russian. in order to perform our experiments, we exploit the datasets used in the semeval 2023 shared task 3: detecting the genre, the framing, and the persuasion techniques in online news in a multilingual setup @xcite , whose one specific subtask focuses on the detection of persuasion techniques at paragraph level in nine lan-guages, including, i.a., polish and russian, which, to the best of our knowledge, constitutes the first ever annotated resource for persuasion technique detection for these languages at intra-document level. while the aforementioned shared task revolves solely around paragraph-level detection and classification of persuasion techniques using a taxonomy of 23 techniques, in our work, we focus on the evaluation with different settings: (a) the granularity of the data after aggregating the results of the classifier: fine-grained (23 labels) versus coarse-grained (6 labels); and (b) the focus of the classification, i.e., at which level the labels are aggregated: subword, sentence, and paragraph level. the main drive behind the inclusion of these different dimensions in the evaluation is to gain a better understanding about the usability of automated persuasion technique detection for practical applications. the primary focus is to compare the performance of mono-versus multi-lingual-trained state-of-the-art transformed-based models in this context. the rest of this paper is organized as follows. first, we report on related work in section 2. next, the persuasion technique detection task and the underlying taxonomy is introduced in section 3. subsequently, in section 4 we report on the carried out experiments, including the description of the dataset, evaluation methodology, models explored, the results, and some rudimentary error analysis. we end up with the conclusions and future outlook in section 5. ## related work the work on automated detection of persuasion techniques in text is related to work on propaganda detection. the work in the latter area initially focused on document-level analysis and predictions. for example, @xcite reports on prediction of four classes (trusted, satire, hoax, and propaganda) of documents, whereas @xcite developed a corpus of documents tagged either as propaganda or nonpropaganda) and further investigated writing style and readability level. in parallel to the above, other research work focused on the detection of specific persuasion techniques in text. @xcite presented a corpus with 1.3k arguments annotated with 5 fallacies that directly relate to propaganda techniques. a more fine-grained analysis was done by @xcite , who developed a corpus of english news articles labelled with 18 propaganda techniques at span and sentence level, and proposed a deep learning-based solutions for this task. improved models were proposed addressing the limitations of transformers by @xcite , whereas the topic of interpretable propaganda detection was addressed by @xcite . somewhat related is also the work on detection of use of propaganda techniques in memes @xcite , the relationship between propaganda and coordination @xcite , and work studying covid-19 related propaganda in social media @xcite . @xcite reported on the creation of annotated text snippet dataset with logical fallacies for covid-19 domain and evaluation or ml-based approaches using this corpus. @xcite presents three-stage evaluation framework of detection, coarse-grained, and fine-grained classification of logical fallacies through adapting existing evaluation datasets, and evaluate various state-of-the-art models using this framework. @xcite proposed the task of logical fallacy detection and a new dataset of logical fallacies found in climate change claims. noteworthy, all the persuasion techniques and logical fallacy taxonomies introduced in the aforementioned research works do, in principle, overlap to a very high degree, but are structured differently, and different naming conventions are used. a comprehensive survey on computational propaganda detection is presented in @xcite . various shared tasks related to persuasion technique detection were organized in the recent years. for instance, semeval-2020 task 11 on detection of persuasion techniques in news articles @xcite focused on the detection of persuasion techniques (at text span level) in news articles. the nlp4if-2019 task on fine-grained propaganda detection task proposed a similar-in-nature task with a taxonomy of 18 persuasion techniques. the semeval-2021 task 6 on detection of persuasion techniques in texts and images focused on the detection of propaganda techniques deployed in memes, and used a taxonomy of 22 techniques @xcite . finally, wanlp'2022 @xcite shared task centred around the detection of 20 propaganda techniques in arabic tweets @xcite , while the semeval 2023 shared task 3: detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup @xcite has a subtask revolving around the detection of persuasion techniques at paragraph level in nine languages, including: english, french, georgian, german, greek, italian, polish, russian, spanish. the work on persuasion detection for polish and russian is scarce, and focused mainly on the analysis of the use of persuasion techniques, not their detection. for instance @xcite studies the use of linguistic cues defined as persuasive linguistic tricks (plt) in social media (sm) marketing communication. @xcite studied the use of propaganda techniques in the russian news in the context of the russian military intervention in ukraine in 2014, and also explored nlp-based models for their automated detection, however, this is done at document level, i.e., classification of articles into persuasive and non-persuasive ones, which is different from our work which is at the intra-document level. to our best knowledge, the resources used in the context of the semeval 2023 shared task 3 @xcite constitute the only resource for persuasion technique detection for polish and russian at text span level, on top of which we carry out our research reported in this paper. ## persuasion technique detection persuasion techniques are tools and strategies used by individuals to influence others' opinions or to motivate them to undertake or support some action or adopt new behaviour(s). in order to perform our set of experiments, we exploit the persuasion techniques taxonomy from the semeval 2023 shared task 3: detecting the genre, the framing, and the persuasion techniques in online news in a multilingual setup @xcite , which is an extended version of the the taxonomy introduced @xcite . at the top level, there are 6 coarse-grained types of persuasion techniques: these six types are further subdivided into 23 fine-grained techniques. figure 1 gives an overview of the two-tier taxonomy and a short definition of all fine-grained techniques. the persuasion technique detection is a multiclass multi-label classification task. some examples of persuasion techniques for polish and russian are provided in figure 2 . ## experiments we explore the performance of state-of-the-art transformer-based models for the task at hand, on the two languages of interest, namely, polish and russian, and the effect of cross-lingual transfer learning using multi-lingual models. specifically, we compared the performance of mono-lingual models with the current multi-lingual model xlm-roberta @xcite , we measured attack on reputation name calling or labelling: a form of argument in which loaded labels are directed at an individual, group, object or activity, typically in an insulting or demeaning way, but also using labels the target audience finds desirable. guilt by association: attacking the opponent or an activity by associating it with a another group, activity or concept that has sharp negative connotations for the target audience. casting doubt: questioning the character or personal attributes of someone or something in order to question their general credibility or quality. ## conclusions and future work in this paper we reported on some preliminary experiments on the detection of persuasion techniques in online news in polish and russian, using a taxonomy of 23 persuasion techniques, and considering different evaluations scenarios: fine-versus coarsegrained classification, the text-structure level at which the labels are detected (subword, sentence, or paragraph). the comparison of mono-and multilingual-trained state-of-the-art transformed-based models revealed the superiority of the latter in most evaluation settings, however, given the complexity of the task, there is significant space for improvement. in our future research we envisage to: (a) enlarge the pool of transformer-based models for inclusion in the evaluation to get a more complete picture of the phenomena observed so far, (b) explore whether and how to exploit data augmentation @xcite to boost the performance of the low-populated persuasion technique classes, and (c) investigate different pre-trained models for the task, like models fine-tuned on multi-lingual питались они в кафе[casting doubt]not correct: and not a technique но болгарское правительство удивило своих граждан [casting doubt] correct единый подход воспитания и образования [appeal to values] not correct в то же время, отмечает селиванов, в всу осознают, что значительная часть населения украины не будет поддерживать страну [flag waiving] almost correct: would have been correct without the negation, otherwise it is both casting doubt and appeal to popularity развернутая сша и их союзниками пропагандистская кампания о «российской агрессии» против украины преследует провокационные цели, тем самым поощряя власти в киеве к саботажу минских соглашений [casting doubt] correct есть миша кавелашвили, который всегда был верен принципам и был бойцом [appeal to values] correct ранее прокуратура санкт-петербурга направила в суд иск о признании блокады ленинграда геноцидом [loaded language] correct на них денег в казне вечно не хватает [questioning reputation] correct схожим образом высказалась премьер новой зеландии джасинда ардерн [casting doubt] not correct: and not a technique сделать ставку на дальнейший развал россии, то есть российской федерации [causal oversimplification] almost correct: it is rather an instance of false dilemma qa @xcite or nli @xcite corpora to investigate their performance on thought coherent classes (like simplification or distraction families).