Transformers to Fight the COVID -19 Infodemic (2021)

Abstract: The massive spread of false information on social media has become a global risk especially in a global pandemic situation like COVID-19. False information detection has thus become a surging research topic in recent months. NLP4IF-2021 shared task on fighting the COVID-19 infodemic has been organised to strengthen the research in false information detection where the participants are asked to predict seven different binary labels regarding false information in a tweet. The shared task has been organised in three languages; Arabic, Bulgarian and English. In this paper, we present our approach to tackle the task objective using transformers. Overall, our approach achieves a 0.707 mean F1 score in Arabic, 0.578 mean F1 score in Bulgarian and 0.864 mean F1 score in English ranking 4 ^{th} place in all the languages.

## introduction by april 2021, coronavirus(covid-19) pandemic has affected 219 nations around the world with 136 million total cases and 2.94 million deaths. with this pandemic situation, a rapid increase in social media usage was noticed. in measures, during 2020, 490 million new users joined indicating a more than 13% year-on-year growth @xcite . this growth is mainly resulted due to the impacts on day-to-day activities and information sharing and gathering requirements related to the pandemic. as a drawback of these exponential growths, the dark side of social media is further revealed during this covid-19 infodemic @xcite . the spreading of false and harmful information resulted in panic and confusions which make the pandemic situation worse. also, the inclusion of false information reduced the usability of a huge volume of data which is generated via social media platforms with the capability of fast propagation. to handle these issues and utilise social media data effectively, accurate identification of false information is crucial. considering the high data generation in social media, manual approaches to filter false information require significant human efforts. therefore an automated technique to tackle this problem will be invaluable to the community. targeting the infodemic that occurred with covid-19, nlp4if-2021 shared task was designed to predict several properties of a tweet including harmfulness, falseness, verifiability, interest to the general public and required attention. the participants of this task were required to predict the binary aspect of the given properties for the test sets in three languages: arabic, bulgarian and english provided by the organisers. our team used recently released transformer models with the text classification architecture to make the predictions and achieved the 4 th place in all the languages while maintaining the simplicity and universality of the method. in this paper, we mainly present our approach, with more details about the architecture including an experimental study. we also provide our code to the community which will be freely available to everyone interested in working in this area using the same methodology foot_0 . ## related work identifying false information in social media has been a major research topic in recent years. false information detection methods can be mainly categorised into two main areas; content-based methods and social context-based methods @xcite . content-based methods are mainly based on the different features in the content of the tweet. for example, @xcite find that highly credible tweets have more urls, and the textual content length is usually longer than that of lower credibility tweets. many studies utilize the lexical and syntactic features to detect false information. for instance, @xcite find that the part of speech (pos) is a distinguishable feature for false information detection. @xcite find that some types of sentiments are apparent features of machine learning classifiers, including positive sentiments words (e.g., love, nice, sweet), negating words (e.g., no, not, never), cognitive action words (e.g., cause, know), and inferring action words (e.g., maybe, perhaps). then they propose a periodic time-series model to identify key linguistic differences between true tweets and fake tweets. with the word embeddings and deep learning getting popular in natural language processing, most of the fake information detection methods were based on embeddings of the content fed into a deep learning network to perform the classification @xcite . traditional content-based methods analyse the credibility of the single microblog or claim in isolation, ignoring the high correlation between different tweets and events. however, social contextbased methods take different tweets in a user profile or an event to identify false information. many studies detect false information by analyzing users' credibility @xcite or stances @xcite . since this shared task is mainly focused on the content of the tweet to detect false information, we can identify our method as a contentbased false information identification approach. ## data the task is about predicting several binary properties of a tweet on covid-19: whether it is harmful, whether it contains a verifiable claim, whether it may be of interest to the general public, whether it appears to contain false information, etc. @xcite . the data has been released for three languages; english, arabic and bulgarian foot_1 . following are the binary properties that the participants should predict for a tweet. i verifiable factual claim: does the tweet contain a verifiable factual claim? ii false information: to what extent does the tweet appear to contain false information? iii interest to general public: will the tweet have an effect on or be of interest to the general public? v need of verification: do you think that a professional fact-checker should verify the claim in the tweet? vi harmful to society: is the tweet harmful for the society? vii require attention: do you think that this tweet should get the attention of government entities? the main motivation for our architecture is the recent success that the transformer models had in various natural language processing tasks like sequence classification @xcite @xcite , token classification @xcite , language detection (jauhiainen et al., 2021), word context prediction (hettiarachchi and ranasinghe, 2020a, 2021) question answering @xcite etc. apart from providing strong results compared to rnn based architectures @xcite , transformer models like bert @xcite provide pretrained multilingual language models that support more than 100 languages which will solve the multilingual issues of these tasks @xcite @xcite . transformer models take an input of a sequence and outputs the representations of the sequence. there can be one or two segments in a sequence which are separated by a special token [sep] @xcite . in this approach we considered a tweet as a sequence and no [sep] token is used. another special token [cls] is used as the first token of the sequence which contains a special classification embedding. for text classification tasks, transformer models take the final hidden state h of the [cls] token as the representation of the whole sequence @xcite . a simple softmax classifier is added to the top of the transformer model to predict the probability of a class c as shown in equation 1 where w is the task-specific parameter matrix. in the classification task all the parameters from transformer as well as w are fine tuned jointly by maximising the log-probability of the correct label. the architecture of transformer-based sequence classifier is shown in figure 1 . ## experimental setup we considered the whole task as seven different classification problems. we trained a transformer model for each label mentioned in section 3. this gave us the flexibility to fine-tune the classification model in to the specific label rather than the whole task. given the very unbalanced nature of the dataset, the transformer models tend to overfit and predict only the majority class. therefore, for each label we took the number of instances in the training set for the minority class and undersampled the majority class to have the same number of instances as the minority class. we then divided this undersampled dataset into a training set and a validation set using 0.8:0.2 split. we mainly fine tuned the learning rate and number of epochs of the classification model manually to obtain the best results for the development set provided by organisers in each language. we obtained 1e -5 as the best value for learning rate and 3 as the best value for number of epochs for all the languages in all the labels. the other configurations of the transformer model were set to a constant value over all the languages in order to ensure consistency between the languages. we used a batch-size of eight, adam optimiser (kingma and ba, 2014) and a linear learning rate warm-up over 10% of the training data. the models were trained using only training data. we performed early stopping if the evaluation loss did not improve over ten evaluation rounds. a summary of hyperparameters and their values used to obtain the reported results are mentioned in appendix - arabert recently language-specific bert based models have proven to be very efficient at language understanding. arabert (antoun et al., 2020 ) is such a model built for arabic with bert using scraped arabic news websites and two publicly available arabic corpora; 1.5 billion words arabic corpus @xcite and osian: the open source international arabic news corpus @xcite . since arabert has outperformed multilingual bert in many nlp tasks in arabic ( antoun ## results when it comes to selecting the best model for each language, highest f1 score out of the evaluated models was chosen. due to the fact that our approach uses a single model for each label, our main goal was to achieve good f1 scores using light weight models. the limitation of available resources to train several models for all seven labels itself was a very challenging task to the team but we managed to evaluate several. as depicted in as shown in ## conclusion we have presented the system by infominer team for nlp4if-2021-fighting the covid-19 infodemic. we have shown that multiple transformer models trained on different labels can be successfully applied to this task. furthermore, we have shown that undersampling can be used to prevent the overfitting of the transformer models to the majority class in an unbalanced dataset like this. overall, our approach is simple but can be considered as effective since it achieved 4 th place in the leader-board for all three languages. one limitation in our approach is that it requires maintaining seven transformer models for the seven binary properties of this task which can be costly in a practical scenario which also restricted us from experimenting with different transformer types due to the limited time and resources. therefore, in future work, we are interested in remodeling the task as a multilabel classification problem, where a single transformer model can be used to predict all seven labels.