PTUK - HULAT at A r AIE val Shared Task Fine-tuned Distilbert to Predict Disinformative Tweets (2023)

Abstract: Disinformation involves the dissemination of incomplete, inaccurate, or misleading information; it has the objective, goal, or purpose of deliberately or intentionally lying to others aboutthe truth. The spread of disinformative information on social media has serious implications, and it causes concern among internet users in different aspects. Automatic classification models are required to detect disinformative posts on social media, especially on Twitter. In this article, DistilBERT multilingual model was fine-tuned to classify tweets either as dis-informative or not dis-informative in Subtask 2A of the ArAIEval shared task. The system outperformed the baseline and achieved F1 micro 87% and F1 macro 80%. Our system ranked 11 compared with all participants.

## introduction nowadays, social media has advanced to the point that it can compete with traditional media. the freedom of user participation could have negative consequences @xcite . disinformation is one of the side effects of this intentionally aiming to mislead the truth that could affect negatively people in many fields like politics, and health, among others. spreading fake news can lead to misunderstanding, harm individuals or groups, damage reputation, or even influence public opinion and decisionmaking, @xcite . thus, automatic detection of these kinds of data is a very important issue. for a while, it seemed so easy to detect disinformation data by domain experts or fact-checkers, but with daily huge propagation data in social media, more resources are needed to automate and speed up the process of detection of this kind of information. arabic language is one of the languages spoken in the world, with 422 million people including na-tive and non-native speakers. it is the official language in 22 countries with at least 30 distinct dialects. however, there are three categories: classical arabic (ca), modern standard arabic (msa), and dialectal arabic (da) @xcite . ca is the original arabic language that has been used for over 1,500 years, and it is usually used in most arabic religious texts. msa is one of the official languages of the united nations and is widely used in todays arabic newspapers, letters, and formal meetings, which are also focused on by researchers. da is spoken arabic used in informal daily communication. with the recent advanced improvements in the natural language processing (nlp) field and the evolution of large language modeling which is based on transformers architecture @xcite the development of arabic language solutions in the nlp field has evolved. to mention some previous efforts devoted to creating arabic datasets to train and test systems, the arabic factchecking and stance detection corpus @xcite contains 422 claims: 219 false claims from verify foot_0 , and 203 true claims from reuters. all these claims were made about the war in syria and related middle east political issues. @xcite describes an arabic corpus of 342 rumors and 3,000 no rumors about death personalities. automatic disinformation classification is the most straightforward way of disinformation analysis. some previous work has been developed in the arabic language such as harrag and djahli (2022) that explored convolutional neural networks (cnns) for fact-checking using @xcite to evaluate the proposal obtaining an accuracy averaged from 0.886 to 0.898. a more recent work nassif et al. (2022) evaluated a transformerbased classifier to recognize fake news using arabic word embeddings. authors reported a performance accuracy of 98% using models such as quaribert-bse and arabic-bert among others. the remainder of this article is organized as follows: the task definition is described in section 2. section 3 describes the data set that was used, then an explanation of the baselines and the proposed system are given in section 4. system evaluation is introduced in section 5. finally, conclusions are given in section 7. ## task definition text classification is a machine-learning process that assigns a document to one or more predetermined categories based on its content abdulghani and abdullah (2022). it is a key problem in nlp, with applications ranging from sentiment analysis to email routing to offensive language detection, spam filtering, and language identification. disinformation classification is a form of text classification that is normally identified as binary classification @xcite . given a set of labeled contexts that will be modeled as a feature f, the task aims to predict whether f is disinformative or not. ## data the organizers of the araieval shared task @xcite released a data set that aims to categorize a tweet whether it is a disinformative or not. these shared tasks represent continuous works on the arabic language after @xcite . the data set is extracted from the twitter website by twarc package @xcite . these tweets were extracted by using the word corona in arabic in february and march 2020. each sample in the data sets is composed of three fields, the id which represents the sample identifier, the text field which includes the tweet text, and the label which represents the annotated label for the text either "disinfo" or "no-disinfo". three separate data sets were released in two phases. the training and the developing data set were released in the first phase, containing 14,147 and 2,115 samples respectively. then the test data set was released in the second phase with 3729 samples. ## system baseline: in order to familiarize the participants with the task, the organizers provided two baselines in the code repository, random and majority baselines. proposed system: pre-trained transformerbased architectures have recently proven to be particularly efficient at language modeling and understanding when trained on a large enough corpus. bidirectional encoder representations from transformers (bert) @xcite is one of these models that gains the attention of the researchers due to its ability to predict words considering left and right context sides. two model sizes are released for bert as modeling language goals, both of them depend on encoder architecture, bert large and bert base . the main difference between them is the number of encoders. bert base consists of a stack of 12 encoders, on the other hand, bert large consists of a stack of 24 encoders. in addition, they differ in the number of hidden units (768, 1,024) and attention heads @xcite . despite the notable results of pre-trained bert models, it has a drawback which makes it very slow due to its parameter numbers @xcite . so, the distillation process, which is known as a compression technique in which a small model (the student) is trained to mimic the behavior of a bigger model (the teacher) or an ensemble of models @xcite is produced to deal with this issue. based on the current resources for nlp, 90% of the worlds population speaks languages that do not benefit from recent language technologies due to the lack of resources @xcite . arabic nlp is among these languages that still need more interest to make it mature @xcite . cross-language transfer is considered the main technique used for addressing the lack of resources in the target language, in which higher resource language models are adapted to the low resource language. the cross-lingual transfer could be achieved in two ways, the first one is by using a trained single high-resource language model, or the second way is by using multiple languages with varying amounts of resources. the idea behind these strategies is that the lower-resource language benefits from the model's learning of language invariant features from a huge amount of data in the high-resource language. thus, to overcome the low resources problems for the arabic language and the slowness of the bert model, in our proposed model we used a distilled multilingual version of bert which was released by @xcite . distilbert is a multilingual model that is trained in 104 different languages including the arabic language from the wikipedia website. thus, the distilbert model has 6 layers, 768 dimensions, and 12 heads, totalizing 134m parameters. in the following sub-sections, the description of two phases, development, and evaluation, will be described in detail. ## results the system performance was evaluated by using f1 macro, and f1 micro; micro and macro averages are aggregation methods for the f1 score, a metric that is used to measure the performance of classification machine learning models. f1 score is calculated per class, which means that if you want to calculate the overall f1 score for a dataset with more than one class you will need to aggregate in some way. micro f1 score is the normal f1 formula but calculated using the total number of true positives (tp), false positives the macro f1 score is the unweighted mean of the f1 scores calculated per class. it is the simplest aggregation for the f1 score. the formula for the macro f1 score is therefore: in the training phase, the system achieved 81% f1 micro and 72% f1 macro. the system achieved 87% f1 micro and 80% f1 macro on the testing data set. to get further, the f1 score was computed per class; for the "disinfo" class the system achieved 68% , and "no-disinfo" class, the proposed model achieved 92% f1 score. ## discussion in this work, a distilled multilingual version of bert was fine-tuned to predict disinformative tweets that are extracted from the social media the data set which are provided by the organizers is imbalanced and this affects the results as shown in the result section. the proposed system failed to predict 494 samples in total, 346 samples related to "disinfo" class which is the minority class in the data set. on the other hand, 148 samples that were labeled with "no-disinfo" were predicted false from the data set. based on our in-depth failure analysis, we found that the system failed to predict correctly the examples containing english words in arabic letters such as " ‫"فولو‬ which means "follow" in english. another reason of failure is that some users repeat some characters in some words to express their emotions such as ‫."طفلللل"‬ ## conclusion in this work, we described our proposed system to classify arabic tweets as either disinformative or not. distilbert's multilingual model was finetuned on the task dataset. the system overcomes the baselines and achieves f1 micro 87% and f1 macro 80% on the testing data set. the arabic language is the official language of 22 countries and it is spoken by over 422 million people, but more efforts are needed to get benefits of the recent nlp technologies. writing a foreign language in arabic letter should be taken into account to improve the proposed model, in addition to using repeated characters to express emotions.