PD - AR at A r AIE val Shared Task: A BERT -Centric Approach to Tackle A rabic Disinformation (2023)

Abstract: This work explores Arabic disinformation identification, a crucial task in natural language processing, using a state-of-the-art NLP model. We highlight the performance of our system model against baseline models, including multilingual and Arabic-specific ones, and showcase the effectiveness of domain-specific pre-trained models. This work advocates for the adoption of tailored pre-trained models in NLP, emphasizing their significance in understanding diverse languages. By merging advanced NLP techniques with domain-specific pre-training, it advances Arabic disinformation identification.

## introduction disinformation is the deliberate creation and spreading of false or misleading information that can cause public harm or generate profit for organizations that participate in such practices @xcite . the consequences of disinformation can be significant, affecting political decisions @xcite , manipulating public opinion, or even inciting violence. detecting disinformation can be challenging because it can appear similar to real information and spread very quickly. additionally, creators are constantly evolving their methods, making it more difficult to detect their content. while disinformation detection in english has received much attention, the nuances, dialectical variations, and morphological richness of arabic present unique challenges that have not been comprehensively addressed. the araleval foot_0 shared task 2: disinformation detection @xcite aims to encourage further exploration of disinformation detection in arabic content. it includes two sub-tasks: (a) to categorize whether a given tweet is disinformative, modelled as a binary classification task, and (b) detecting the fine-grained disinformation class for a tweet, modelled as a multiclass classification task with labels indicating the subtype of disinformation contained -hate speech, offensive, rumour or spam. bert-based @xcite models have been shown to be successful in understanding the context behind language and benefit from being able to transfer learned knowledge to various tasks. due to these advantages, such models are very good in text classification tasks. we have, therefore, utilised bert-based models for the shared task which has been pre-trained over arabic text. we hypothesized that such pre-trained models will better understand arabic text than a bert model that has been pre-trained over english text. however, there are certain challenges when we are dealing with text that is code-mixed. tweets usually contain texts that contain code-mixed text which may prove to be difficult to work with. ## related work techniques used for disinformation detection include manually or automatically analyzing the content of a piece of information to identify features that are associated with disinformation, analyzing the social media activity around a piece of information to identify patterns that suggest it is being spread as disinformation and verifying the claims made in a piece of information using external knowledge @xcite . in the domain of fake news detection, significant work has already been done which are covered by many seminal survey works on fake news detection such as @xcite @xcite @xcite @xcite . given that the surveys encompass research endeavors concerning fake news, encompassing both misinformation and disinformation detection, we shall employ these terms interchangeably within this section. however, with the advent of transformer models @xcite , the prospect of training neural network models on languages beyond english has become increasingly prominent. the application of transformer models to arabic text is highly promising. these models, trained on extensive text data, can excel in arabic nlp tasks. they can grasp sentiment nuances, crucial for sentiment analysis, and enhance translation accuracy in challenging arabic-english translation tasks. for information retrieval, understanding arabic query and document semantics is vital, where transformers show exceptional performance. this advancement has significantly improved arabic nlp across domains. consequently, a plethora of research studies focusing on arabic fake news detection has emerged, many of which have been reviewed in prominent surveys like those by @xcite @xcite . other related works also include shared task results on propaganda @xcite and detection and reasoning of tweets @xcite . the covid-19 pandemic also led to a range of research works focusing on arabic misinformation regarding the pandemic. the work by @xcite introduces a dataset for misinformation detection, covering various topical categories influenced by covid-19, and presents benchmarking results for tweet-level verification. @xcite examines the scale of arabic covid-19 disinformation, identifying prominent topics related to violations of civil liberties, vaccine-related conspiracies, and calls for action. @xcite presents a machine learning-based system for detecting misinformation in arabic tweets related to covid-19 vaccination, achieving promising performance. the work by @xcite introduces a comprehensive dataset annotated with fine-grained misinformation classes and situational information, and presents baseline results using various classifiers. in contrast to the aforementioned work, this system paper investigates the effectiveness of a finetuned bert model in binary and multi-class classification of disinformation work, thereby capturing a broader aspect of disinformation regarding arabic twitter data. ## data for both the sub-tasks, we used the training and development sets for the competition since the test set labels were part of the competition. however, after the competition the test set labels were also released which is why in this paper we are including the details of the whole dataset for both the sub-tasks. for subtask 2a, the details of the dataset are shown in for the subtask 2b, the details of the dataset is shown in ## system in this section,we will first describe the system architecture and then discuss the implementation of the system. ## results since the test file with gold standard annotations was available for both sub-tasks, we have evaluated our model on this file. the results for both the sub-tasks are shown below in these results substantiate the efficacy of our approach, highlighting its robustness and suitability for the given tasks. the superior performance of our system showcases the importance of specialized pre-trained models, in enhancing the accuracy and effectiveness of domain specific natural language processing tasks. ## discussion in this section, we delve into the discussion of the results obtained from our experiments with various bert-based models. our results, as illustrated in ## conclusion in this study, we have presented a comprehensive analysis of our system's performance in addressing the task of arabic disinformation. the results of our evaluation unequivocally illustrate the superiority of our system over various baseline approaches, including those based on generic and multilingual transformer models. notably, our system's outstanding performance in both sub-tasks underscores the significance of such language-specific pre-trained models in enhancing the precision and utility of natural language processing applications. furthermore, the superiority of our system, even when compared to camelbert, a model pretrained on arabic text, highlights the importance of considering the specific nuances of data sources. in our case, a pre-trained model on twitter-specific arabic text data proved to be an advantageous choice, particularly for tasks involving twitter data, where the writing style is distinct due to character limitations.