UL & UM 6 P at A r AIE val Shared Task: Transformer-based model for Persuasion Techniques and Disinformation detection in A rabic (2023)

Abstract: In this paper, we introduce our participating system to the ArAIEval Shared Task, addressing both the detection of persuasion techniques and disinformation tasks. Our proposed system employs a pre-trained transformer-based language model for Arabic, alongside a classifier. We have assessed the performance of three Arabic Pre-trained Language Models (PLMs) for sentence encoding. Additionally, to enhance our model’s performance, we have explored various training objectives, including Cross-Entropy loss, regularized Mixup loss, asymmetric multi-label loss, and Focal Tversky loss. On the official test set, our system has achieved micro-F1 scores of 0.7515, 0.5666, 0.904, and 0.8333 for Sub-Task 1A, Sub-Task 1B, Sub-Task 2A, and Sub-Task 2B, respectively. Furthermore, our system has secured the 4th, 1st, 3rd, and 2nd positions, respectively, among all participating systems in sub-tasks 1A, 1B, 2A, and 2B of the ArAIEval shared task.

## introduction social media platforms have transformed into significant spaces where people communicate and collect information from various sources. however, along with this positive shift, a significant amount of false, misleading, and harmful content has also emerged. this includes various forms of problematic content like misinformation, disinformation, and malinformation in the form of spreading propaganda, conspiracy theories, rumors, hoaxes, fake news, false statements, hate speech, cyberbullying, and among others @xcite @xcite @xcite @xcite . furthermore, the surge in online communication platforms has also made it more important to understand how people try to persuade each other. the persuasion detection task involves the identification and analysis of communication strategies aimed at influencing individuals' beliefs or actions. it encompasses recognizing techniques such as emotional appeals, logical reasoning, and rhetorical devices in various forms of content @xcite . propaganda, a subset of persuasive communication, refers to the deliberate dissemination of information, often with a biased or misleading intent, to manipulate opinions or behaviors. it involves employing well-defined psychological and rhetorical methods to sway audiences @xcite . several shared tasks have been organized for the detection of propaganda techniques in text and memes. this includes the nlp4if-2019 shared task on fine-grained propaganda detection @xcite , semeval-2020 task 11 on detection of persuasion techniques in news articles @xcite , and semeval-2021 task 6 on detection of persuasion techniques in texts and images @xcite . in addition to detecting propaganda techniques, another intriguing task is to identify misleading content within social media. this aims to uncover various forms of disinformation, such as hate speech, offensive language, rumors, and spam @xcite @xcite . most of the previously mentioned research works have primarily focused on the english language. therefore, there is a noteworthy need to develop such methods for the arabic language, which is spoken by a considerable number of people globally, with an estimated 372 to 446 million speakers worldwide. with the aim of bridging this language gap, @xcite have presented a description of three arabic tasks that were offered as part of the third edition of the checkthat! lab at clef 2020. it focused on false information propagated on arabic social media, particularly on twitter. furthermore, @xcite have run a shared task on detecting propaganda techniques in arabic tweets as part of the wanlp 2022 work-shop. more recently, hasanain et al. (2023) have introduced the araieval shared task that includes two tasks: (i) persuasion techniques detection (sub-task 1a and sub-task 1b), and (ii) disinformation detection (sub-task 2a and sub-task 2b) in the arabic language. in this paper, we present our submitted system for the araieval shared task @xcite , where we tackle both the tasks of detecting persuasion techniques and identifying disinformation. our system utilizes a deep learning model that comprises a transformer-based pretrained language model (plm) encoder designed for the arabic language, coupled with a classifier. the classifier consists of a dropout layer followed by a linear layer. to encode text inputs, we have evaluated the performance of three arabic plms: arbertv2, marbertv2, and arabert-large @xcite @xcite . during the model training process, we have explored the following training objectives: our system is evaluated using the weightedaverage precision and recall as well as the micro and macro f1 score. it has achieved micro-f1 scores of 0.7515, 0.5666, 0.904, and 0.8333 on the test sets of sub-task 1a, sub-task 1b, sub-task 2a, and sub-task 2b, respectively. furthermore, our system has secured the 4th, 1st, 3rd, and 2nd positions, respectively, among all participating systems in the corresponding sub-tasks of the araieval shared task. it is worth mentioning that the best results have been obtained using the ar-bert sentence encoder for both sub-task 1b and sub-task 2a. while, for sub-task 1a and sub-task 2b, the best performance has been achieved using the marbertv2 encoder. ## data the araieval shared task @xcite comprises two tasks: persuasion techniques detecting (sub-task 1a and sub-task 1b), as well as disinformation detection (sub-task 2a and sub-task 2b) in arabic. for disinformation detection, the araieval organizers provide data for the following two sub-tasks: for the input texts encoding, we have explored the performance of three existing bert-based arabic plms, including arbertv2, marbertv2, and arabert-large @xcite @xcite . these plms have been trained on large arabic textual corpora, covering both modern standard arabic and dialectal arabic, using the masked language modeling objective function. as shown in figure 1 , given an input text of length m, the plm's tokenizer split it into n sub-words and append the [cls] and [sep ] special tokens, representing the start and end of the input sequence, to the tokenized text ([cls], w 1 , w 2 , w 3 , ..., w n , [sep ]). then, the bert-based encoder is fed with the tokenized text and outputs the contextualized word embedding h [cls] , h 1 , h 2 , h 3 , ..., h n , h @xcite . finally, the pooled output of the [cls] token is passed to the classifier to predict the class label of the input text. ## experiments and results in this section, we present the experiment settings and the obtained results for each sub-task. ## discussion the obtained results have shown that the training objective and the text encoder have a significant impact on our models' performance. the overall results demonstrate the effectiveness of the plms encoders that are pre-trained on large text corpora from the same domain as the target downstream tasks (marbertv2). a straightforward path of future research work is to investigate the performances of other state-of-the-art arabic plms and other training objectives that deal with the class imbalance problem. ## conclusion in this paper, we have introduced our submitted system to the araieval shared task for persuasion techniques and disinformation detection in arabic. our system uses a deep learning model that consists of a transformer-based pre-trained language model (plm) encoder for the arabic language and a classifier. for the model training, we have explored several training objectives and assessed the performance of three arabic plms. on the official test set, our system has obtained micro-f1 scores of 0.7515, 0.5666, 0.904, and 0.8333 for sub-task 1a, sub-task 1b, sub-task 2a, and sub-task 2b, respectively. besides, it has been ranked in the 4th, 1st, 3rd, and 2nd positions among all participating systems in sub-task 1a, sub-task 1b, sub-task 2a, and sub-task 2b, respectively.