R00 at NLP 4 IF -2021 Fighting COVID -19 Infodemic with Transformers and More Transformers (2021)

Abstract: This paper describes the winning model in the Arabic NLP4IF shared task for fighting the COVID-19 infodemic. The goal of the shared task is to check disinformation about COVID-19 in Arabic tweets. Our proposed model has been ranked 1st with an F1-Score of 0.780 and an Accuracy score of 0.762. A variety of transformer-based pre-trained language models have been experimented with through this study. The best-scored model is an ensemble of AraBERT-Base, Asafya-BERT, and ARBERT models. One of the study’s key findings is showing the effect the pre-processing can have on every model’s score. In addition to describing the winning model, the current study shows the error analysis.

## introduction social media platforms are highly used for expressing and delivering ideas. most people on social media platforms tend to spread and share posts without fact-checking the story or the source. consequently, the propaganda is posted to promote a particular ideology to create further confusion in understanding an event. of course, it does not apply to all posts. however, there is a line between propaganda and factual news, blurred for people engaged in these platforms @xcite . and thus, social media can act as a distortion for critical and severe events. the covid-19 pandemic is one such event. several previous works were published for using language models and machine learning techniques for detecting misinformation. authors in @xcite presented a twitter data set for covid-19 misinformation detection called "arcov19-rumors". it is an extension of the "arcov-19" @xcite , which is a data set of twitter posts with "propagation networks". propagation networks refer to a post's retweets and conversational threads. other authors in @xcite performed an exploratory study of covid-19 misinformation on twitter. they collected data from twitter and identified misinformation, rumors on twitter, and misinformation propagation. authors in @xcite presented ct-bert, a transformer-based model pre-trained on english twitter data. other works that used deep learning models to detect propaganda in news articles @xcite . the nlp4if @xcite shared-task offers an annotated data set of tweets to check disinformation about covid-19 in each tweet. the task asked the participants to propose models that can predict the disinformation in these tweets. this paper describes the winning model in the shared task, an ensemble of arabert-base, asafya-bert, and arbert pre-trained language models. the team r00's model outperformed the other teams and baseline models with an f1-score of 0.780 and an accuracy score of 0.762. this paper describes the dataset and the shared task in section 2. the data preprocessing step is presented in section 3. the experiments with the pre-trained language models are provided in section 4. finally, the proposed winning model and methodology are discussed in section 5. ## dataset the data provided by the organizers @xcite of tweets, which are posts from the twitter social media platform "twitter.com". the posts are related to the covid-19 pandemic and have been annotated in a "yes or no" question style annotation. the annotator was asked to read the post/tweet and go to an affiliated weblink (if the tweet contains one). for each tweet, the seven main questions that were asked are: for each question, the answer can be "yes" or "no". however the questions two through five depend on the first question. if the first question (verifiable factual claim) is answered "no", questions two through five will be labeled as "nan". "nan" is interpreted as there's no need to ask the question. for example, for the following tweet: this tweet is not a verifiable factual claim. therefore asking whether it's false information or is in need of verification is unnecessary. moreover, our model modified the values to be " no" for all text samples with labels annotated as "nan". task our team participated in the arabic text shared task. the arabic data set consists of 2,536 tweets for the training data, 520 tweets for the development (validation) data, and 1,000 tweets for the test data. it has been observed that the label distribution in the training data is unbalanced, as shown in figure 1 . ## data pre-processing social media posts can contain noisy features, particularly the special characters (#, @, emojis, weblinks, etc..). many elements within arabic text can act as distortions for the model. we tokenize the arabic text 1 , and for each sequence of tokens, we remove stop-words, numbers, and punctuation from the text. we also remove any non-arabic terms in the text. stemming and segmentation are two common pre-processing operations done in arabic natural language processing. however, we do not apply them here, except in the case of arabert, where segmentation was applied. ## fine-tuning pre-trained language models we approach the problem as a multi-label classification problem. for each label in a text sample, the label's value can be one (yes) or zero (no). in the training phase, we load the pre-trained language model (along with its corresponding tokenizer) and stack a linear classifier on top of the model. this section describes the pre-trained arabic language models that have been used in the study. the hyperparameters' fine-tuning is also detailed in this section in addition to the experiments' results. ## ensemble pre-trained language models to maximize the scores, we resort to ensembling some of the models we fine-tuned on the data set. ensemble models are known to improve accuracy strategy through experimentation and for each label, train a model that performs well on that label and save it for an ensemble. then, train a model that generally performs well on all labels (relative to the models at hand) and save it for an ensemble. after collecting several models, ensemble these models through various combinations. and for each ensemble, record the combination and its score (performance on validation data). choose the best performing ensemble. weighted-average our approach for an ensemble is to take the weighted average of each's model predictions for each sample. each model produces a vector of probabilities (whose length is equal to the number of labels) for each tweet. we take the weighted average point-wise and then apply a 0.5-threshold to decide if a label is one (yes) or zero (no). we suggest using the weighted average rather than a normal average with equal weights to give higher confidence to the generally performing model as opposed to the less generally performing one. the intuition is that you would want the model to be the deciding factor in predicting better overall performance. the models with the lesser weights are merely there to increase the models' confidence in predicting some labels. the optimal weights for an ensemble are obtainable through experimentation. as a hyperparameter, they can be tuned. ## conclusion this paper described the winning model in the nlp4if 2021 shared task. the task aimed to check disinformation about covid-19 in arabic tweets. we have ensembled five pre-trained language models to obtain the highest f1-score of 0.780 and an accuracy score of 0.762. we have shown the performances of every pre-trained language model on the data set. we also have shown some of the models' performances on each label. moreover, we have demonstrated the confusion matrix for the ensemble model. we have illustrated that a pre-trained model on twitter data (asafya-bert in section 4.1) will perform better relative to a model that hasn't.