Mavericks at A r AIE val Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion (2023)

Abstract: In this paper, we highlight our approach for the “Arabic AI Tasks Evaluation (ArAiEval) Shared Task 2023”. We present our approaches for task 1-A and task 2-A of the shared task which focus on persuasion technique detection and disinformation detection respectively. Detection of persuasion techniques and disinformation has become imperative to avoid distortion of authentic information. The tasks use multigenre snippets of tweets and news articles for the given binary classification problem. We experiment with several transformer-based models that were pre-trained on the Arabic language. We fine-tune these state-of-the-art models on the provided dataset. Ensembling is employed to enhance the performance of the systems. We achieved a micro F1-score of 0.742 on task 1-A (8th rank on the leaderboard) and 0.901 on task 2-A (7th rank on the leaderboard) respectively.

## introduction in today's digital age, numerous platforms aid people in reaching out to the world. however, some individuals resort to disinformation and persuasion techniques to influence people, keeping in mind a certain biased agenda, which can have negative societal effects. disinformation @xcite is an intentional effort to disseminate malicious, manipulative, and misleading information for espionage. the propagation of incorrect information can be deleterious to an individual, an organization, or a nation. hence, disinformation detection has become imperative to catch false reports and avoid social upheaval. persuasion is the act of changing someone's convictions, views, or conduct through interaction or exchange. persuasion techniques can be employed to propagate propaganda @xcite and influence the behavioral patterns of the targeted audience. persuasion can * equal contribution be done via textual mediums such as news articles and tweets. social media can act as a key instrument to proliferate persuasive content as well as disinformation among the masses. with advancements in science and technology, specifically in the domain of machine learning, machines are now capable of detecting persuasion techniques as well as disinformation from the given data. however, the detection techniques have certain limitations. the tactics used to spread disinformation constantly evolve, and the sheer volume is immense. understanding context and intent is another challenge when it comes to detecting persuasion and disinformation. detecting and countering these instruments of influence across multiple languages and cultural contexts can be daunting. this paper demonstrates our work on task 1 -persuasion technique detection and task 2 -disinformation detection @xcite . we intend to examine whether the given multigenre textual snippets contain persuasive content in task 1 and classify whether the given tweet @xcite is disinformation or not in task 2. our approach highlights the use of various transformerbased models for binary classification on the given arabic data. ensemble-based techniques have also been employed to yield better results. ## related work in the pre-internet era, traditional media analysis, fact-checking, and investigative journalism were employed to detect disinformation and persuasion techniques. with the emergence of the internet, keyword-based approaches and sentimental analysis techniques found their groove in detecting fake news and persuading content. an analysis of linguistic features @xcite , lexical patterns @xcite , and rhetorical structures @xcite was used for this purpose. further advancements in text analysis (pérez-rosas et al., 2017) proved fruitful for this task. although mitigating disinformation and persuasion techniques had been a difficult undertaking, machine learning techniques @xcite showed promise to address this issue. supervised machine learning methods @xcite such as support vector machine(svm), xgboost, and naive bayes have been used for this purpose. (iyer and sycara, 2019) stated that the detection of persuasive tactics in a text can be automated using unsupervised learning. network analysis methods @xcite such as centrality measures can be used to identify coordinated behavior. this technique can help to highlight the propagation of disinformation or persuasive content over social networks. subsequently, deep learning techniques @xcite ) also contributed to fine-tuning the results. the utilization of word embeddings and convolutional neural networks (cnns) for recognizing persuasion at an early stage also fueled the prevention of social engineering attacks @xcite . multiple state-of-the-art systems such as lstms @xcite are also used to detect fake news. further research revealed that transfer learning approaches like bert produced more promising results than other cutting-edge nlp techniques @xcite . ensembling techniques @xcite were utilized to further enhance the results by integrating various approaches into a single one. hybrid architectures like combining bert with a recurrent neural network (rnn) @xcite or a combination of parallel cnns with @xcite achieved a significant score. recent developments suggest that ai approaches such as explainable ai (xai) @xcite are being experimented with for the task of disinformation and persuasion technique detection. in this paper, we present our approach, which encompasses the utilization of transformer-based models for classification. variations of bert are used to develop an ensemble-based system for the given classification tasks. ## data the dataset provided for task 1 -persuasion technique detection comprises multigenre text snippets, which are either tweets or news paragraphs. the training data has 2427 samples of such snippets, the development data has 259 samples, and the testing data has 503 samples. the training data contains features such as the id, text, label, and type. each snippet in the training dataset is labeled as either 'true' or 'false' based on the presence of persuasion techniques in the given sample. this task falls under the category of binary classification. the dataset provided for task 2 -disinformation detection comprises tweets. the training data has 14147 (14126 non-null) samples of such tweets, the development data has 2111 samples, and the testing data has 3729 samples. the training data contains features such as the id, text, and label. each tweet in the training dataset is labeled as either 'disinfo' or 'no-disinfo' based on the content in every sample. this task falls under the category of binary classification. the provided dataset is preprocessed using regular expressions to remove irrelevant strings such as "@user", "link" and "rt" to reduce the noise. ## system this shared task discusses the problems of disinformation and persuasion detection. these problems come under the umbrella of classification problems for which transformer-based models have been widely used and have achieved impressive performance. thus, we have utilized several transformerbased models and ensembling methods in our research as shown in figure 1 . the models are trained for 10 epochs with a learning rate of 1e-5, a batch size of 32, and the adamw optimizer. the methodologies have been briefly discussed in the section below. ## ensembling ensembling is a technique that combines the results of various models to generate the eventual intended result of the system. statistical as well as non-statistical methods are used for this purpose. ensembling is useful as it helps generate results that are better than the results given by the individual models. amongst several methods leveraged for ensem-bling, we observed that the "hard voting" ensemble technique proved to be the most efficient and accurate. in hard voting, the majority vote or the "mode" of all the predictions is selected as the final prediction. it helps improve the robustness of the system and minimizes the variance in the results. ## results we discuss the results of our experiments for tasks 1-a and 2-a in this section. in the post-evaluation phase (after submission of the official scores), out of the various models we experiment with for the given task, marbertv2 outperforms araelectra-basediscriminator and emerges as the best model with a micro f1 score of 0.876 on development dataset. this can be attributed to the large size of the tweetbased training corpus. it boosts the ensemble scores to the 0.869 on development dataset and the 0.751 on test dataset. ## conclusion in this paper, we compared the performance of several transformer-based models on the tasks of persuasion technique detection and disinformation detection. for the final submission, amongst the individual models, it is observed that the araelectrabase-discriminator achieved the best performance for task 1-a. this model was able to achieve a micro f1 score of 0.742. likewise, arabertv0.2-twitter-base achieved the best results for task 2-a and the final system yielded a micro f1 score of 0.901. hard voting-based ensembling is used for our final systems to improve performance whilst also generating stable predictions. in the future, with the availability of better computational resources, we can enhance the system's performance by training it for longer and by using larger models. moreover, we can experiment with other suitable ensembling techniques to gauge their effectiveness.