A r AIE val Shared Task: Persuasion Techniques and Disinformation Detection in A rabic Text (2023)

Abstract: We present an overview of the ArAIEval shared task, organized as part of the first ArabicNLP 2023 conference co-located with EMNLP 2023. ArAIEval offers two tasks over Arabic text: (1) persuasion technique detection, focusing on identifying persuasion techniques in tweets and news articles, and (2) disinformation detection in binary and multiclass setups over tweets. A total of 20 teams participated in the final evaluation phase, with 14 and 16 teams participating in Task 1 and Task 2, respectively. Across both tasks, we observe that fine-tuning transformer models such as AraBERT is the core of majority of participating systems. We provide a description of the task setup, including description of datasets construction and the evaluation setup. We also provide a brief overview of the participating systems. All datasets and evaluation scripts from the shared task are released to the research community. We hope this will enable further research on such important tasks within the Arabic NLP community.

## introduction social media has become one of the predominant communication channels for freely sharing content online. with this freedom, misuse has emerged, turning social media platforms into potential grounds for sharing inappropriate posts, misinformation, and disinformation @xcite @xcite . malicious users can disseminate disinformative content, such as hate-speech, rumors, and spam, to gain social and political agendas or to harm individuals, entities and organizations. such content can inflame tension between different groups and ignite violence among their members, making early detection and prevention essential. previous successful attempts to address such kinds of problems at a large scale over arabic content include offensive and hate speech detection shared tasks @xcite . social media content designed to promote hidden agendas is not limited to disinformation. in the past years, propaganda has been widely used as well, to influence and/or mislead the audience, which became a major concern for different stakeholders, social media platforms and government agencies. news reporting in the mainstream media also exhibits a similar phenomenon, where a variety of persuasion techniques @xcite are used to promote a particular editorial agenda. to address this problem, the research area of "computational propaganda" has emerged aimed at automatically identify such techniques in textual, visual and multimodal (e.g., memes) content. @xcite curated a set of persuasion techniques, such as loaded language, appeal to fear, straw man and red herring. the focus of the work was mainly on textual content (i.e., newspaper articles). following this prior work, in 2021, @xcite organized a shared task on propaganda techniques in memes. these efforts mainly focused on english. to enrich the arabic ai research, we have organized a shared task on detection of fine-grained propaganda techniques for arabic, which attracted many participants @xcite . following the success of our previous shared tasks @xcite @xcite , and given the great interest from the community in further pushing research in this domain, this year we organize the arabic ai evaluation (araieval) shared task covering the following two tasks: (i) persuasion technique detection over tweets and news articles, and (ii) disinformation detection over tweets. this edition of the shared task has attracted wide participation. the task was run in two phases: (i) the development phase with 38 registrations, and 14 teams submitting their systems; and (ii) the evaluation phase with 25 registrations, and 20 teams submitting their systems. in the remainder of this paper, we define each of the two tasks, describe the arabic evaluation datasets that were manually constructed, and provide overview of participating systems and their official scores. ## task 1: propaganda detection the goal of this task is to identify the persuasion techniques present in a piece of text. it targets multi-genre content, including tweets and paragraphs from news articles, as persuasion techniques are commonly used within these domains. the task is organized into two subtasks. ## task 2: disinformation detection this task targeted tweets and was organized into two subtasks, as discussed below. ## conclusion and future work we presented an overview of the araieval shared task at the arabicnlp 2023 conference, targeting two shared tasks: (i) persuasion technique detection, and (ii) disinformation detection. the task attracted the attention of many teams: a total of 25 teams registered to participate during the evaluation phase, with 14 and 16 teams eventually making an official submission on the test set for tasks 1 and 2, respectively. finally, 17 teams submitted a task description paper. task 1 aimed to identify the propaganda techniques used in multi-genre text snippets, including tweets and news articles, in both binary and multilabel settings. on the other hand, task 2 aimed to detect disinformation in tweets in both binary and multiclass settings. for both tasks, the majority of the systems fine-tuned pretrained arabic language models and used standard pre-processing. several systems explored different loss functions, while a handful of systems utilized data augmentation and ensemble methods. given the success of the task this year, we plan to run a future edition with an increased data size, and with wider coverage of domains, countries, and arabic dialects. we are also considering implementing a multi-granularity persuasion techniques detection setting.