NARNIA at NLP 4 IF -2021: Identification of Misinformation in COVID -19 Tweets Using BERT weet (2021)

Abstract: The spread of COVID-19 has been accompanied with widespread misinformation on social media. In particular, Twitterverse has seen a huge increase in dissemination of distorted facts and figures. The present work aims at identifying tweets regarding COVID-19 which contains harmful and false information. We have experimented with a number of Deep Learning-based models, including different word embeddings, such as Glove, ELMo, among others. BERTweet model achieved the best overall F1-score of 0.881 and secured the third rank on the above task.

## introduction rapid propagation of social media has revolutionized the way information is consumed by general public. the ability of web platforms, such as twitter, instagram and facebook, to quickly and broadly disseminate huge volumes of information has encouraged any user to be a (super) conduit of information. this can be helpful for problem solving in stressful and uncertain circumstances. however, this has also raised serious concerns about the disability of naive internet users in distinguishing truth from widespread misinformation. as the world reacts to the covid-19 pandemic, we are confronted with an overabundance of virusrelated material. some of this knowledge may be misleading and dangerous. the wildfire of fake news in the times of covid-19 has been popularly referred to as an 'infodemic' by the who chief. also, in literature, we see terms such as 'pandemic populism' and 'covidiocy' @xcite . distorted facts and figures formed by drawing false equivalence between scientific evidence and uninformed opinions and doctored videos of public figures have flooded the online space since the onset of covid. in order to ensure safety and well * * joint first author being of online information consumers, it is crucial to identify and curb the spread of false information. twitter should mark content that is demonstrably inaccurate or misleading and poses a serious risk of damage (such as increased virus transmission or negative impacts on public health systems). hence, developing and improving classification methods for tweets is need of the hour. in the present work, fighting with covid19 infodemic dataset @xcite comprising english tweets about covid-19 has been utilised for identifying false tweets. many deep learning models have been trained to predict several properties of a tweet as described in section 3. the rest of the paper is organized as follows. section 2 discusses related research work. section 3 describes the dataset and section 4 describes the language models we have used for our predictions. sections 5 and 6 report the results of the experiments we conducted for the different language models and the error analysis respectively. finally, in section 7, we discuss future work that can be done in this area and conclude our paper. ## related work classification of tweets has been studied widely by many researchers. most of the methods use traditional machine learning classifiers on the features extracted from individual tweets, such as pos, unigrams, bigrams. @xcite built a naive bayes classifier for detecting sentiment of tweets. they considered lemmas, polarity lexicons, and multiword from different sources and valence shifters as input features to the classifier. in recent times, the advancement of deep learning approaches (e.g., neural networks and transformer-based pre-trained language models like bert and gpt) have taken precedence over feature-based classifiers (e.g., naive-bayes, svm, among others). classification problems have primarily been tackled in two ways -feature based and by fine-tuning of parameters. feature based approaches use word-embeddings, such as word2vec @xcite , glove @xcite , elmo @xcite , and feed them into some deep learning model to perform downstream task. on the other hand, parameter fine-tuning based approach fine tunes all the pre-trained parameters on downstream tasks. we have experimented with both these approaches. recently, language models such as bert @xcite , pre-trained on large amount of unlabelled data and fine tuned on downstream task, have given state-of-the-art results in numerous nlp tasks. bertweet @xcite ) is one such model which is pre-trained on english tweets. it has been found that bertweet outperforms other state-of-the art language models, e.g roberta, xlm-r @xcite with respect to several nlp tasks, viz. text classification, ner etc. this motivates us to use bertweet based approach for this task. ## dataset description the dataset used in this task contains english tweets, and the corresponding labels (which are mainly "yes"/"no"), that are the answers to the following questions: as per the dataset specifications, q2 to q5 will be nan if and only if q1 is "no". further, q1, q6 and q7 are never supposed to be nan. if there are some instances where this condition is violated, we have dropped the corresponding tweets (independently for all the questions) during training or validation. finally, for the final predictions, we first obtain the predictions for q1, and the tweets are checked for the labels q2 to q5 only when q1 is "yes". the given dataset has 869 tweets in the train dataset. we randomly split the dataset for training and in-sample validation purposes, with the splits having 695 and 174 tweets respectively (80 -20 split). for validation, we are given a dev dataset with 53 tweets. the test dataset on which we submit our final predictions contains 418 tweets. ## model description a vast number of language models have been developed in the last decade. we used a number of them to solve the given problem, and they are described in the following subsections. ## performance evaluation this section describes the evaluation scheme, followed by the results obtained for the different models. ## error analysis for q1, only three tweets in the dev data (452, 490, 492: all having a verifiable claim) are predicted wrong by our model. similarly, for q2, three examples (491, 498, 500), which also have a positive label (denoting that the tweet appears to contain false information), have been predicted wrong while for q4, four examples (461, 462, 484, 485), all having negative labels (denoting that the claim made in the tweet cannot be harmful to the society), are predicted wrong by our model. some of these tweets (as described above) can be found in ## conclusion and future work we implemented five models, described in section 4, and showed that the bertweet based models outperforms the rest. however, apart from the dependence of q2 to q5 on q1 (refer section 3), we have assumed all questions to be independent. but, by the definitions of questions (section 3), it is evident that q4 & q6 and q5 & q7 have some dependence on each other. this can be seen in the dataset labels as well, because q4 & q6 have the same label for 87.6% of the tweets. similarly, q5 and q7 have the same label 83.3% of the times. since correlation does not imply causation, this property can be further explored to see if there is some dependence between the labels, which can possibly be incorporated in the model to improve the predictions for q4 to q7. moreover, in this work, we have not experimented with multi-class classification techniques, which can be further explored for a possible improvement.