A ra BERT Model for Propaganda Detection (2022)

Abstract: Nowadays, the rapid dissemination of data on digital platforms has resulted in the emergence of information pollution and data contamination, specifically mis-information, mal-information, dis-information, fake news, and various types of propaganda. These topics are now posing a serious threat to the online digital realm, posing numerous challenges to social media platforms and governments around the world. In this article, we propose a propaganda detection model based on the transformer-based model AraBERT, with the objective of using this framework to detect propagandistic content in the Arabic social media text scene, well with purpose of making online Arabic news and media consumption healthier and safer. Given the dataset, our results are relatively encouraging, indicating a huge potential for this line of approaches in Arabic online news text NLP.

## introduction people are moving away from traditional media and toward digital content in today's landscape, and with trust in traditional media at an all-time low of 32% (according to a gallup inc. poll), it's no surprise that people are turning to alternative sources for news. furthermore, social media has recently evolved into a major source of news content, giving rise to the "fake news" phenomenon (s. @xcite , in which a large amount of false information circulates, often with malicious intent (p. @xcite . the associated propaganda, which is almost always present in fake news, is an important but often overlooked feature of such destructive content (s. @xcite . the primary goal of propaganda is to influence the opinions of target individuals through language manipulation (d. @xcite . there are over 313 million people worldwide who speak arabic, with roughly 90% of them getting their news from the internet and online content. furthermore, the prevalence of "fake news" in online content, as well as its amplification by social platforms, poses a number of serious challenges to society (d. @xcite . propaganda techniques, for example, obfuscation, black or white fallacy, loaded language, name calling, straw man, red herring, whataboutism, and others, can pose grave threats to society, economy, democracy, health, journalism, the environment, and a variety of other areas. while there have been recent studies that developed machine learning models to detect fake news in a variety of languages (n. @xcite , the lack of research into arabic is, to say the least, concerning. propaganda detection in arabic is a collaborative effort (f. @xcite to combat fake news by developing models for identifying propaganda techniques in arabic social media text. so far, recent efforts to detect propaganda in news items around the world @xcite have addressed this as a fine-grained problem of finding it within fragments, and as a result, transformer-based embeddings work reasonably well in such detection approaches. as a result, in this article, we attempt to achieve the goal of our contributions by following the flow: ## system the model used for training is the arabert model's second version (w. antoun et al., 2020). arabert is a powerful, cutting-edge transformer-based model for arabic language understanding that has the same configuration as the base bert model: twelve encoder blocks, twelve attention heads, seven hundred and sixtyeight hidden dimensions, five hundred and twelve maximum sequence lengths, and a total of approximately one hundred and ten parameters. the model included pre-trained embeddings that had been trained on approximately seventy million sentences from various sources. the model was trained in the google collaboratory using a tesla v100-sxm2-16gb gpu. the training set was divided into batches of sixteen each, with a gradient accumulation step of two, and an evaluation batch size of one hundred and twenty-eight. the optimization algorithm used is the adam optimizer, with an epsilon value of 10 -8 , a learning rate of 0.00002, and iterates over twenty-five epochs. the overall training process took about five minutes, thanks to gpu parallelization. the model generates 18 probabilities, each of which corresponds to a propaganda technique, including no technique. we used the following methodology to determine the output labels for each tweet based on those probabilities. we took the top five predictions from each tweet and discarded the rest. this is due to the fact that no tweet had more than 5 labels in the original data. we had to optimize the threshold that will be used to filter out low probabilities from the top five. after some experimentation, we noticed that the most similar distribution was obtained with an optimal threshold of 0.35, assuming that the label distributions in the training, testing, and validation sets were the same. we considered that if one of the remaining labels was 'no-technique', the corresponding tweet's labels would be all labels with a probability greater than that of the no-technique label. otherwise, the tweet will be labeled as notechnique. we merged the three labeled datasets before predicting the labels of the unlabeled datasets and submitting the samples after training, validating, and testing on the datasets and reaching the optimal configuration. ## results because the competition organizers set the micro-f1 score as the primary metric to evaluate the performance of the models, we used it to examine our model performance. this is primarily due to the fact that our task is simply to maximize the number of correct predictions made by the classifier, and no class is more important than the other. it is worth noting that we were able to achieve a micro-f1 of 0.61 while using data augmentation and attempting to optimize the classification layer weights. due to other deadlines, this result was not submitted. a micro-f1 score of 0.578 on the evaluation dataset was a very promising result that will be improved using the methods described in the following section. ## discussion given that the micro-f1 score on the training set is around 0.88, it is clear that the training data is overfit. the used model is cutting-edge and does a good job of capturing the data's complexity. however, the true issue is that the dataset is not representative enough for the model to generalize outside of it. to improve the model's performance outside of the training set, the first step would be to add a regularization to the model, such as l2 regularization, to reduce overfitting. however, given the small dataset size, this may not be so promising. the second optimization approach would be to augment the existing data by performing some transformation on the documents, such as random insertion, random deletion, and word swapping. the latter method will increase data diversity at a lower cost than collecting brand new labeled data. a third approach that could be used is to use class weights to compensate for class imbalances by penalizing misclassification by infrequent classes (flagwaving, for example) more than that of more abundant classes. given enough time, the most time-consuming but effective thing that could be done is to collect or manually label more propaganda-related tweets so that we have more representative data for real-world tasks. ## conclusion in this paper, we presented a transformer-based model that serves as a contribution framework to identify propaganda types in arabic text social media content (tweets basically), by highlighting the propaganda strategies utilized (such as obfuscation, black or white fallacy, loaded language, name calling, straw man, red herring, whataboutism, and others). with a micro-f1 score of 0.578 and given the relatively small dataset, the model appears promising, and we are confident that performance improvements can be expected with a more balanced and richer dataset. we intend to improve the model in the future by focusing more on the labeled dataset and expanding it by either applying careful, wellstructured augmentation to some data or by developing a platform to assist annotators in labeling data. this ensures that the model is constantly updated and improved. furthermore, we intend to conduct extensive research on various aspects of propaganda in order to develop a general propaganda detection system, thereby broadening the scope of our work in relation to the existing platform, with the goal of making the online arabic journey healthier and safer.