Unleashing the Power of Discourse-Enhanced Transformers for Propaganda Detection (2024)

Abstract: The prevalence of information manipulation online has created a need for propaganda detection systems. Such systems have typically focused on the surface words, ignoring the linguistic structure. Here we aim to bridge this gap. In particular, we present the first attempt at using discourse analysis for the task. We consider both paragraph-level and token-level classification and we propose a discourse-aware Transformer architecture. Our experiments on English and Russian demonstrate sizeable performance gains compared to a number of baselines. Moreover, our ablation study emphasizes the importance of specific types of discourse features, and our in-depth analysis reveals a strong correlation between propaganda instances and discourse spans.

## introduction the widespread of disinformation and information manipulation in various domains, such as politics, economics, and health (e.g., , has led to an increased demand for fact-checking and propaganda detection. to tackle this, several datasets have been created to analyze online media and assist in system development @xcite . one of the recent competitions in this field is semeval 2023 task 3 @xcite , which introduces a multilingual dataset with six languages. this paper specifically addresses the most challenging task in the competition, namely persuasion techniques detection. for propaganda detection, the most recent effective approaches employ encoder-based transformers as their backbone models, with some minor task-specific or dataset-specific modifications @xcite @xcite . however, these approaches have aimed mostly to achieve the highest quality or competition scores. john hancock, the first signer of the declaration of independence, said, attribution continue steadfast and, with a proper sense of your dependence on god, nobly defend those rights in contrast, we adopt a broader perspective and aim not only to develop a qualitative approach, but also to gain interpretation and understanding that can facilitate progress in propaganda detection. to this end, we analyze the linguistic structure of the texts by examining the discourse features. among the various types of discourse representations available, the rhetorical structure theory (rst) suggested by @xcite was selected for our analysis due to its widespread usage and availability of high-quality open-source parsers for multiple languages. according to this theory, a text can be represented as a tree structure, where the nodes correspond to text spans and are connected by discourse relations such as elaboration, joint, and condition. figure 1 illustrates an example of a discourse tree constructed for a real media text, with propaganda spans annotated. notably, the "attribution" discourse span aligns precisely with the "appeal to authority" propaganda span. it is reasonable to observe this alignment, and further examples can be found in the dataset. however, establishing a definitive set of rules for matching between discourse and propaganda is quite challenging. the primary objective of our research is to enhance the effectiveness of a transformer-based approach for propaganda detection by integrating discourse information. this allows not only to improve the quality of the model, but also to evaluate the relationship between discourse and propaganda. our contributions can be summarized as follows: ## related work general view a fine-grained propaganda analysis was proposed by @xcite , who developed a corpus of news articles annotated with 18 propaganda techniques, considering separately the task of technique spans detection and classification. subsequently, the dataset and its extentsions were employed in several shared tasks, such as semeval-2020 @xcite and semeval-2023 @xcite . transformer-based approaches have become common in solving the propaganda detection task, treating it as a token-or text classification problem @xcite . recent studies have utilized bert-based models, such as bert @xcite , roberta @xcite , deberta @xcite , albert @xcite , or their ensemble @xcite . @xcite considered xlm-roberta and extended the base approach by using a contrastive formulation of the loss function. another approach proposed by @xcite introduced a bert-based hierarchical model that combines token classification and multilabel token classification tasks. we also focus on transformer-based approaches, but our analysis goes beyond the typical modifications associated with task formulation or base dataset characteristics. instead, we additionally investigate the nature of propaganda in terms of discourse structure in order to enhance the interpretability of our approach. discourse in previous studies, the effectiveness of discourse-aware measures has been demonstrated in evaluating the quality of machine translation @xcite . we also consider some discourse-based characteristics, but incorporate them as features in our neural approach. similarly, @xcite enhanced extractive summarization by combining discourse-based representations with bert embeddings. the fact-checking task is closely related to propaganda detection, as both involve analyzing the reliability of information. previous research has shown that discourse integration techniques have been effective for this task. for instance, @xcite developed the multitask model that incorporated discourse tree reconstruction as an auxiliary loss. @xcite utilized pre-constructed discourse trees and encoded them using a recursive neural network. regarding interpretable approaches, @xcite conducted a study on classification-based methods for propaganda detection. nevertheless, their analysis primarily concentrated on syntactic and sentiment features, neglecting discourse features. to fill this gap, our work investigates the relationship between discourse trees and propaganda spans, and underscores the importance of specific discourse features in enhancing the efficacy of neural approaches. finally, an analogy can be drawn with the study conducted by @xcite , which suggested multi-task learning with propaganda identification as the main task and metaphor detection as an auxiliary task. ## datasets in our experiments, we utilized the dataset provided by the semeval-2023 task 3 subtask 3 competition @xcite , which is devoted to the persuasion technique detection task. this dataset is regarded as the most relevant and comprehensive publicly available resource. it encompasses news articles in six different languages, each of which has been annotated by experts using a set of 19 labels. articles have been pre-partitioned into train, validation, and test sets in competition. we employed these sets and compared our results with the best approaches suggested during the competition. to investigate the discourse structure, we employed pre-trained discourse parsers. while this approach has certain limitations, such as potential errors in the parsers, it provides a universally applicable approach that does not depend on human resources. we have identified publicly available parsers with mit license for two languages: english @xcite and @xcite ; and compared results for these languages. in this research, we utilize transformers, eliminating the need for any specialized preprocessing techniques. the only preprocessing was to remove all non-ascii characters from the articles. ## implementation details we fine-tuned the base-sized deberta-v2 @xcite for english and the base-sized xlm-roberta @xcite for russian. these models have 184m and 125m parameters respectively. the maximum sequence length was set to 256 in all cases: we selected this value by analyzing the training set. the models were trained on batches of size 16, with a learning rate of 3e-5, for 20-40 epochs. for all other hyper-parameters, we used the default values. regarding the hyper-parameters related to the discourse features, we selected @xmath0. we trained each model (setting) on a tesla v100 32g gpu for approximately two hours. ## discussion this section aims to elucidate the importance of incorporating discourse structure in the propaganda detection task and investigate the interpretable correlation between propaganda spans and specific types of discourse features. ## conclusion and future work in this paper, we investigated the efficacy of discourse-enhanced transformers in the context of the propaganda detection task. specifically, we examined two different settings, namely paragraph and token classification, using the english and russian subsets of the semeval-2023 dataset. we suggested a modification of the base transformer architecture to incorporate discourse features. our experimental results indicated that discourse information substantially enhances the performance of the base models. we conducted a comprehensive analysis to determine the relative importance of each type of discourse feature. furthermore, our findings revealed a strong correlation between propaganda instances and discourse spans. we believe that this research contributes to the advancement of propaganda detection algorithms and provides valuable insights into the role of discourse in propagandistic texts. future work can focus on investigating additional types of discourse features, neural architecture modifications, as well as exploring the generalizability of the suggested approach.