A ra D etector at A r AIE val Shared Task: An Ensemble of A rabic-specific pre-trained BERT and GPT -4 for A rabic Disinformation Detection (2023)

Abstract: The rapid proliferation of disinformation through social media has become one of the most dangerous means to deceive and influence people’s thoughts, viewpoints, or behaviors due to social media’s facilities, such as rapid access, lower cost, and ease of use. Disinformation can spread through social media in different ways, such as fake news stories, doctored images or videos, deceptive data, and even conspiracy theories, thus making detecting disinformation challenging. This paper is a part of participation in the ArAIEval competition that relates to disinformation detection. This work evaluated four models: MARBERT, the proposed ensemble model, and two tests over GPT-4 (zero-shot and Few-shot). GPT-4 achieved micro-F1 79.01% while the ensemble method obtained 76.83%. Despite no improvement in the micro-F1 score on the dev dataset using the ensemble approach, we still used it for the test dataset predictions. We believed that merging different classifiers might enhance the system’s prediction accuracy.

## introduction approximately 66% foot_0 of individuals in the middle east utilize social media to seek out daily news. the rise of rapid development in social media and online communication, such as chat platforms @xcite , have emerged as prevalent means to facilitate the widespread dissemination of disinformation at an unprecedented pace. disinformation is the phenomenon that refers to how individuals or groups can be deceived or manipulated by false or misleading information. as disinformation spreads gradually, it can boost existing biases, polarize viewpoints, and hinder constructive dialogue, compromising the collaborative spirit essential for a healthy democracy. this far-reaching phenomenon can affect opinion decision-making and can threaten different foundations of democratic societies by eroding public trust in different institutions and planting seeds of divisions among communities @xcite @xcite . to detect disinformation and prevent it from spreading, modern methods use transformer-based architectures that are trained specifically on arabic text and are available in public, such as arabert @xcite and marbert @xcite . this paper outlines our participation in the disinformation detection (task 2) of the araieval shared task @xcite . we introduce a method incorporating three distinct classifiers: the marbert pre-trained language model (plm) and both zero-shot and few-shot models. our objective is to improve the accuracy of disinformation identification in tweets by adopting a majority voting ensemble strategy. the subsequent sections are structured as follows: section 2 reviews prior studies; section 3 describes our proposed method; section 4 details our experimental result; and finally, we conclude with a summarisation of our main findings. ## related work nowadays, various types of disinformation have swiftly disseminated across social media platforms and digital news outlets, each possessing distinct attributes and objectives to deceive and influence people. due to the simplicity of sharing data online, it has become challenging to differentiate between trustworthy information and fake ones @xcite . many research studies have been con-ducted to detect disinformation. in this section, we provide a concise overview of the recent research on disinformation. in their study about dis-information detection, bahurmuz n. et al. 2017 used two transformers, arabert and marabert. the proposed paradigm removes all the non-textual, non-linguistic url features to get a real dataset. two sampling techniques have been used to solve the unbalanced dataset. transformers models were trained by fine-tuning the hyper-parameters using freezing model techniques. marabert shows better performance results @xcite . in 2021, al-yahya m et al. examined various neural networks and transformer models for arabic fake news detection. the experiments were conducted by using document and word embedding to test multiple neural network models like cnn, rnn, and gru, then compared to the transformers like (arabert v1, arabert v2, arelectra, qarib, arbert, and marbert. qarib obtained high accuracy scores compared to the limitation of small data size, repeated tweets, and noisy tweets that do not belong to any class @xcite . albalawi. r. et al. 2022 proposed a model that relies on textual visual features to detect disinformation. they used mrabert for text feature extraction, while resnet-50 was used to extract image features. the text and visual features were combined and input into one multi-modal classifier to detect rumors from non. early fusion of features achieved an accuracy of 0.85. the efficiency of the proposed model could not outperform the textbased models in accuracy. this is due to the size of the dataset (r. m. @xcite . @xcite worked on collecting a dataset related to covid-19 disinformation news from twitter; this dataset was the first arabic covid-19 dataset comprising about 6.7k tweets. word cloud has been used to obtain crucial words to analyze both real and fake news. they also prepared a version of arabert trained based on covid-19 tweets known as arabert-cov19. to reach a more accurate result than previous models, authors have been dependent on preparing and labeling the collected dataset manually. @xcite . hate speech and fake news can work together as a powerful weapon against society; for example, an article claiming that a particular group of people is planning to commit violence can justify hate speech against that group. this can lead to real-world violence, as seen in cases such as the rohingya genocide in myanmar @xcite . a study by researchers at the university of southern california's information sciences institute found that 20 percent of tweets containing hate speech were also fake @xcite .; therefore, ameur m. et al. (2021) used fine-tuned two pretrained models, arabert cov19" and "mbert cov19. the work aimed to build a model that can detect fake news about covid-19 and hate speech simultaneously (ameur and aliane, 2021). ## methodology the proposed system is composed of three distinct models. raw tweets were prepared and preprocessed as inputs to the models, as outlined in section 3.1. sections 3.2, 3.3, and 3.4 explain the three models incorporated using an ensemble technique, as clarified in section 3.4. ## result given the nature of the shared tasks, we conducted our initial experiments on the development dataset and accordingly selected the bestperforming method for delivering predictions on the test dataset. the organizers of this shared task have shared an annotated dataset sourced from pre-proce ssing arabert qarib marbert no 68.16% 69.04% 68.27% yes 68.64% 69.22% 69.79% twitter @xcite . this dataset is noteworthy for being one of the most extensive publicly accessible arabic datasets focused on disinformation content foot_1 . it contains 14126 tweets as a training set and 2115 tweets as a development set. specifically, we compared three pre-trained models as well as the gpt models in zero-shot and view-shot settings. we used the official evaluation measure adopted by the organizers (micro-f1). additionally, we used two experimental settings: zero-shot and few-shot prompting strategies. due to the cost of using such models, we used the preprocessed text based on previous experiments' findings. in future work, we will study the impact of pre-processed text for gpt models on arabic usergenerated text extracted from social media. finally, after studying and analyzing the performance of the models, we decided to take advantage of each one of them using a majority voting approach to merge three classifiers: marbert, fewshot and zero-shot prompting strategies. although the micro-f1 score on the dev dataset was not improved using the ensemble approach, we used it to deliver predictions of the test dataset. we hypothesized that the system's ability to generalize by combining different classifiers may balance out better classification prediction. ## conclusion disinformation on social media may be biased in the society's collective opinion. consequently, this may lead to social abuse action. accordingly, social media needs an apparatus to help people reveal false claims. this study used three arabic transformers for comparison (arabert, marbert, qarib).