HTE at A r AIE val Shared Task: Integrating Content Type Information in Binary Persuasive Technique Detection (2023)

Abstract: Propaganda frequently employs sophisticated persuasive strategies in order to influence public opinion and manipulate perceptions. As a result, automating the detection of persuasive techniques is critical in identifying and mitigating propaganda on social media and in mainstream media. This paper proposes a set of transformer-based models for detecting persuasive techniques in tweets and news that incorporate content type information as extra features or as an extra learning objective in a multitask learning setting. In addition to learning to detect the presence of persuasive techniques in text, our best model learns specific syntactic and lexical cues used to express them based on text genre (type) as an auxiliary task. To optimize the model and deal with data imbalance, a focal loss is used. As part of ArabicNLP2023-ArAIEval shared task, this model achieves the highest score in the shared task 1A out of 13 participants, according to the official results, with a micro-F1 of 76.34% and a macro-F1 of 73.21% on the test dataset.

## introduction in an era marked by the proliferation of information via digital platforms, separating fact from fiction has become an increasingly difficult task, nearly impossible to achieve manually. news and social media platforms are effective tools for disseminating information, but they also serve as breeding grounds for propaganda, misinformation, and manipulation. propaganda messages can be used to influence people's opinions, beliefs, and behaviours by appealing to their emotions or by using persuasive techniques and arguments that may sound convincing but are based on faulty logic and thus invalid. to combat this, persuasive technique detection has emerged as an important component in the fight against deceptive content. most studies in this field focus on one genre of textual content for detecting persuasive techniques @xcite @xcite @xcite . handling multi-genre text has received little attention. in this paper, we concentrate on the automatic detection of persuasive techniques in tweets and news. we propose various transformer-based systems for detecting persuasive techniques that implicitly and explicitly utilize content type to enhance detection in multi-genre text. as part of the arabicnlp2023-araieval shared task @xcite , the task in which we participate (task 1a) involves a collection of arabic tweets and news paragraphs annotated to indicate the presence or absence of persuasive content. the rest of the paper is organized as follows: section 2 gives an overview of related work. in section 3, we present the data used. the proposed system is described in section 4. in section 5, we provide the details of our experiments, and then the results for our official runs are presented in section 6. in section 7, a discussion of the results is presented. we conclude the paper in section 8. ## related work over the past few years, there has been an increase in concern about the spread of opinion-shaping news and misinformation, particularly in the context of critical events such as covid-19, elections, and conflicts. as a result, identifying propaganda content and persuasive techniques has gained more importance. research on propaganda content detection has targeted various media platform contents, including news @xcite , memes @xcite , and tweets @xcite @xcite . with the introduction of transformer models @xcite , the detection of propaganda and fake news has seen significant improvement in performance. some works relied solely on real-world data to fine-tune a pre-trained language model for propaganda and persuasive technique detection @xcite , whereas others combined it with synthetically augmented data @xcite . the ensemble approach was also investigated, in which various combined pre-trained language models are fine-tuned in a vanilla setting @xcite , or by using adapters @xcite . one major limitation of all the preceding works is that they focus on one type of media platform content at a time. in this paper, we investigate the detection of persuasive techniques from multigenre text extracted from tweets and news articles. the task is made more difficult by the differences in writing styles and contexts in both texts. in line with the assumption proposed by @xcite , which affirms that sentence representations incorporating information about writing style tend to exhibit better generalization than word-level representations in news propaganda detection, we delve into the integration of content type (genre) information within transformer-based models for the detection of persuasive techniques. ## data our data comes from the proposed dataset for persuasion technique detection as part of the araieval 2023 shared task 1a. 2 no additional data was used. each entry in the data file is composed of three fields: text referring to the textual content, type referring to the genre of text: tweet or news, and label referring to the presence or absence of persuasive technique in the text: true or false. when comparing persuasive tweets and news paragraphs, the context used to express news paragraphs is twice as long as the context used to express tweets, with the average length of news being 211 characters compared to 100 characters for tweets. we expect that the length of the context will influence the syntactic and lexical cues used to express persuasive techniques in news paragraphs and tweets. we should point out that no text preprocessing was done on the dataset for training. ## system overview the system's goal is to determine whether a multigenre (a tweet or news paragraph) snippet contains persuasive content. our proposed system (cf. figure 1 ) is made up of : (1) a transformer-based encoder enc i @xcite that encodes the input texts into a fixed-size contextualized vector, (2) followed by a feature injection layer (f eat) that concatenates the content type vector with the input vector, and (3) two parallel classifiers c, each of which is made up of a fully-connected layer, a dropout layer, and an activation layer, and perform two different tasks: each task calculates one loss, and optimizing the model optimizes the sum of the two losses. because the two tasks share the same encoder, the auxiliary task can help the main task learn additional specific syntactic and lexical cues for tweets or news content used to express persuasive arguments. according to recent research, jointly learning common characteristics shared across multiple tasks can have a significant impact on nlp classification performances as it enhances the performance of the main task by incorporating other related tasks, making it easier to combine information from multiple resources @xcite @xcite @xcite . ## experimental setup we experiment with different configurations for the proposed system components. we evaluated two transformer-based encoders: marbert (noted m ) (abdul-mageed et al., 2021) and arabert (noted a) @xcite , noting enc i with @xmath0, a}. because f eat directly injects the text type into the system as a feature and c aux learns to predict it, it is not possible to enable both f eat and c aux in a model, thus enabling one disables the other. the resulted systems from various configurations are shown below: -enc m +c main and enc a + c main , that we consider as baseline models, that perform a classical binary classification based on the input vector, in a monotask setting without any additional features. a cross-entropy loss (noted ce) is used to optimize the systems. we also experiment with a ## results in general, we found that explicitly or implicitly incorporating content type information into the system could improve overall results for both mar-bert and arabert encoders when either crossentropy or focal loss was used. for example, both enc m +f eat+c main and enc m +c aux +c main beat enc m +c main , with almost + 2% and + 1% on micro-f1. among the twelve evaluated system configurations, enc a +c main +c aux optimized using a focal loss represents our best performing one during the development phase, achieving an increase of nearly 3% above baselines. this model was submitted for official ranking on the test dataset for task 1a, and the obtained micro-f1 is 0.7634, which is the highest score on the leaderboard for this task. ## discussion the test dataset contains 503 inputs, of which 119 are classified incorrectly. out of these, 103 are paragraphs (87%) and 16 are tweets (13%). a closer examination of the confusion matrices per content type for the best-performing system configuration (cf. figure 3 for tweet text and figure 2 for news paragraphs) reveals that the majority of tweet misclassifications (75%) involve non-persuasive content being mistaken for per-models ce focal enc m +c main 0.8301 0.8263 enc a +c main 0.8108 0.8301 enc m +f eat+c main 0.8533 0.8417 enc a +f eat+c main 0.8108 0.8571 enc m +c main +c aux 0.8378 0.8610 † enc a +c main +c aux 0.8147 0.8340 ## conclusion in this paper, we present our experiments and findings on the detection of persuasive techniques in multi-genre texts, which encompass tweets and news paragraphs. this research was part of the arabicnlp2023-araieval task1a shared task, focusing on identifying persuasive techniques through binary classification. our team proposed a system based on fine-tuning a transformer-based model to assess the impact of integrating content type information on persuasive technique detection. we experimented with two different approaches to information integration: implicitly, by adding an additional learning objective to the model, or explicitly, as an additional feature. we evaluated two pre-trained language models for arabic and optimized the system using two different loss functions: cross-entropy and focal loss to address data imbalance. our highest scores on the development dataset were achieved by the marbert model, trained using focal loss, for both persuasive technique detection and content type detection. as a result, our model secured the first position in the arabicnlp2023-araieval task1a shared task during the test phase. our future work will involve exploring data augmentation techniques to address data imbalance and integrating multiple pre-trained language models. finally, our results indicate that our system faces challenges in identifying persuasive content in news paragraphs. to pinpoint the causes of misclassifications, a deeper investigation into incorrectly classified sentences is warranted.