D etective R edasers at A r AIE val Shared Task: Leveraging Transformer Ensembles for A rabic Deception Detection (2023)

Abstract: This paper outlines a methodology aimed at combating disinformation in Arabic social media, a strategy that secured a first-place finish in tasks 2A and 2B at the ArAIEval shared task during the ArabicNLP 2023 conference. Our team, DetectiveRedasers, developed a hyperparameter-optimized pipeline centered around singular BERT-based models for the Arabic language, enhanced by a soft-voting ensemble strategy. Subsequent evaluation on the test dataset reveals that ensembles, although generally resilient, do not always outperform individual models. The primary contributions of this paper are its multifaceted strategy, which led to winning solutions for both binary (2A) and multiclass (2B) disinformation classification tasks.

## introduction the spread of disinformation across social media platforms presents an omnipresent challenge that transcends modalities, manifesting in text, audio, and images @xcite . within the sphere of text, disinformation is not exclusive to one language but spans many languages and dialects. its impact permeates several topics, including politics, entertainment, sports, and finance. in our study, we direct our efforts to detecting disinformation in arabic on twitter as part of the shared task 2: arabic deception detection, at araieval, arabicnlp 2023 @xcite . from a research standpoint, arabic has been receiving an increasing amount of attention addressing several key problems @xcite . investigations into disinformation in arabic can offer valuable insights into unique linguistic and cultural aspects that influence the dissemination and impact of false information within arabicspeaking communities. in a social aspect, the consequences of disinformation across all global communities are profound. specifically, arabic is spoken by hundreds of millions of people worldwide and serves as a linguistic backbone for critical geopolitical regions. while also not exclusive to arabic, disinformation can impact democratic processes and public health @xcite . research in this critical domain has real-world implications that can influence policy decisions, governance, and public well-being. arabic itself presents its own set of complexities; it is a rich language featuring intricate word formations and variations @xcite . this makes the language both highly derivational, meaning words can be formed from root words in various ways, and inflectional, indicating that the form of words can change to convey different meanings. these linguistic traits add an extra layer of difficulty to the already challenging task of disinformation detection. shared task 2 comprises two separate sub-tasks. task 2a is a binary classification challenge requiring us to categorize whether a given tweet is disinformative. task 2b, on the other hand, is a more nuanced multiclass classification task, where the objective is to identify fine-grained disinformation classes such as hate speech, offensive content, rumors, or spam @xcite . with this task, there are several open problems due to phenomena including code-switching @xcite ), short texts, and lack of grammatical structure in tweets. these issues lead to the deterioration of the effectiveness of conventional analytical tools. code-switching refers to the practice of switching between languages within a conversation, or text. tweets tend to mirror the linguistic styles and variations spoken by individuals hailing from a particular region. for example, moroccan tweets contain moroccan darija mixed with french, english, or spanish. this phenomenon can occur for various reasons, including cultural exchange, or historical factors such as colonization. in addressing disinformation detection in arabic, our multi-faceted strategy begins with specialized preprocessing, including handling codeswitching and incorporating tweet elements like hashtags and urls, which previous literature often neglects @xcite . we then utilize large language models, specifically @xcite , and experiment with a softvoting ensemble to improve performance. while effective, these large models are computationally expensive; we seek to mitigate this through optimization pipelines, which in turn add their own computational overhead. ## dataset and tasks in the araieval shared task at arabicnlp 2023, participants are presented with two main tasks: task 1 focuses on persuasion technique detection, while task 2 aims at disinformation detection. each of these primary tasks are further divided into two sub-tasks. our research specifically concentrates on task 2, which consists of sub-task 2a and sub-task 2b. in sub-task 2a, the goal is to classify tweets as either disinformative or not, a binary classification problem. for sub-task 2b, we must identify specific types of disinformation within a tweet, which involves a multiclass classification framework. the fine-grained labels that we consider include hate speech, offensive language, rumors, and spam @xcite . tables 1 and 2 represent the class distributions and total size of the training, validation, and testing sets, for task 2a and 2b, respectively. ## system for tasks 2a and 2b, our approach adopts a specialized methodology using comprehensive preprocessing which deals with code-switching and emoji conversion. after which an intensive search for optimal large language models and hyperparameters is performed. our decisions of which models to utilize were based on performance on the validation set. the arabert-covid19 model @xcite surfaced as the best fit for task 2a. this model, an enhancement of the original arabertv02, has been further refined through finetuning on 1.5 million multi-dialect arabic tweets. these tweets, sourced from the extensive arabic twitter dataset @xcite , specifically focused on covid-19. conversely, for task 2b, we utilize arabertv02-twitter, which was pretrained on approximately 60 million tweets spanning various arabic dialects. subsequently, we employ a soft voting ensemble method, integrating five arabertv02-twitter models that have been optimized. while each model maintains identical hyperparameters and architecture, they differ solely in terms of random initialization. for this process, we utilized the torchensemble library foot_0 . we optimize both arabertv02-twitter and arabert-covid19 models leveraging the optimization framework optuna @xcite . by the deadline for task 2, only two optimized models were evaluated: the arabert-covid19 model for task 2a and the arabertv02-twitter ensemble for task 2b. to ensure the best performance in regards to our target metric, "micro f1", we explored a variety of models. our initial model candidate list included the following: a) arabertv02-twitter (antoun et al., 2020) b) arabert-covid19 @xcite ## results and discussion the top two candidate models, identified through hyperparameter optimization, were arabert-covid19 for task 2a and arabertv02-twitter for task 2b. these selections represented the only results submitted by the task deadline. our results presented in it's also important to highlight that we did not fine-tune the ensemble's hyperparameters, which could have contributed to its less-than-optimal performance against the single models. this supports the idea that ensemble methods, while often robust, require task-specific validation. in future work, optimization techniques specifically for ensembles and not just the individual models may prove to be beneficial, such as an varied amount of classifiers in the ensemble or different weighting techniques. the exploration of additional preprocessing techniques to better handle code-switching could also be a beneficial avenue. our results reiterate the importance of nuanced model selection, especially given the challenges posed by binary and multiclass classification tasks. our findings also pave the way for future work focused on improving computational efficiency and generalization capabilities of disinformation detection models. ## conclusion in this study, we tackled the nuanced problem of disinformation detection in arabic, a language fraught with complexities like code-switching and dialectal variations. we combined meticulous preprocessing with hyperparameter-optimized arabert models, effectively achieving first-place performance in both binary and multiclass deception detection tasks at araieval 2023. a notable insight from our empirical analysis is that individual models occasionally outperform ensembles, indicating the need for careful model selection. our results not only validate our comprehensive approach but also invite further research into optimizing ensemble methods and addressing the challenges associated with code-switching and dialectal variations in arabic text. future work should look at refining these ensemble strategies and explor-ing additional preprocessing techniques, as we aim to create universally effective tools for countering disinformation.