Can Multilingual Transformers Fight the COVID -19 Infodemic? (2021)

Abstract: The massive spread of false information on social media has become a global risk especially in a global pandemic situation like COVID-19. False information detection has thus become a surging research topic in recent months. In recent years, supervised machine learning models have been used to automatically identify false information in social media. However, most of these machine learning models focus only on the language they were trained on. Given the fact that social media platforms are being used in different languages, managing machine learning models for each and every language separately would be chaotic. In this research, we experiment with multilingual models to identify false information in social media by using two recently released multilingual false information detection datasets. We show that multilingual models perform on par with the monolingual models and sometimes even better than the monolingual models to detect false information in social media making them more useful in real-world scenarios.

## introduction by june 2021, the coronavirus(covid-19) pandemic has affected 219 nations around the world with 176 million total cases and 3.81 million deaths. the nature of the virus caused many governments to implement lockdown in their countries. as a result, many people started spending more time at home during the pandemic and started using social media more, providing an unexpected boost to engagement on these platforms @xcite . as a drawback of these exponential growths, social media has become a conduit for spreading both rumours and deliberate misinformation, and many perpetrators are deploying sites such as facebook, twitter, youtube, and whatsapp to create a sense of panic and confusion. on the other hand, the general public can not completely ignore the information seen in social media due to the fact that the centers for disease control and prevention, the world health organisation (who), numerous journals, government and other health care organisations are regularly posting guidance across a host of platforms. therefore, rather than completely disregarding information seeing in social media, accurate identification of false information is crucial @xcite . considering the high data generation in social media, manual approaches to filter false information require significant human efforts. therefore an automated technique to tackle this problem will be invaluable to the community. in the light of this many shared task has been organised to tackle the false information detection in social media @xcite leading to implement various machine learning models which can identify false information automatically @xcite . however, most of these approaches build languagespecific models trained specifically on a particular language. given the fact that most of the social media platforms are massively multilingual, maintaining fake news identification models for each language would not be feasible. one machine learning model that can work across many languages would be invaluable to the community. in this research, we explore multilingual models for false information detection. we experiment with two recently created datasets that target two different aspects in false information detection which also covers 5 languages; arabic, bulgarian, english, spanish and turkish. we show that multilingual models based on pretrained transformer models perform on par with the language-specific models trained for each language on both aspects in false information detection making them more feasible in real-world applications. ## related work false information detection identifying false information in social media has been a major research topic in recent years. according to literature, mainly, there are two types of methods for false information detection as social context-based methods and content-based methods @xcite . social context-based methods use different properties in user profiles such as user's credibility @xcite or stances @xcite while the content-based methods use different features in the content of posts such as certain keywords, number of urls and the length of textual content to detect false information. however, due to ethical considerations, most of the social media platforms do not allow to release datasets with details which can be used to identify users of the posts. therefore social context-based methods have not been popular in recent research as contentbased methods. due to the nature of datasets we use for this research, we also focused on contentbased methods. content-based methods mainly focus on different features of post contents. for example, @xcite found that highly credible tweets have more urls and lengthy textual contents than low credible tweets. also, many studies utilise lexical and syntactic components of the content as useful features. for instance, @xcite found part of speech (pos) as a distinguishable feature for false information detection. similarly, @xcite found that some types of sentiments including positive words (e.g. love, nice, sweet), negating words (e.g. no, not, never), cognitive action words (e.g. cause, know) and inferring action words (e.g. maybe, perhaps) as apparent features for a periodic time-series model to identify key linguistic differences between true and fake tweets. with the recent popularity gained by embedding and deep learning-based approaches in natural language processing, there was a tendency to use deep neural networks powered by content embeddings to perform false information classification too @xcite . later, with the introduction of transformers @xcite , there was a tendency to involve large pretrained transformer models also @xcite @xcite . however, all of these models were trained specifically on a single language making them less useful in real scenarios where we need to process multilingual data. ## data for this research, we used two recently released datasets on false information identification. we mainly considered two factors when selecting datasets; the dataset should be annotated in multiple languages and it should have been annotated very recently. the first dataset (nlp4if) which was released for nlp4if shared task; fighting the covid-19 infodemic is about predicting several binary properties of a tweet on covid-19 such as whether it is harmful, whether it contains a verifiable claim, whether it may be of interest to the general public and whether it appears to contain false information @xcite . the data has been released for three languages; english, arabic and bulgarian foot_0 . seven labels were targeted by this dataset. the first label was verifiable factual claim: does the tweet contain a verifiable factual claim?. we only considered this label for our research as this is directly related to false information detection and this label had the most annotated data out of the seven labels. false information detection using this label can be considered as a binary text classification task. the second dataset (clef2021) that we considered was released for clef2021 checkthat-lab @xcite task 1: check-worthiness estimation of the tweets @xcite . given a tweet, the participants need to predict whether it is worth fact-checking. this task is directly related to the first label of the nlp4if dataset. however contrast to the binary classification in the previous task, the models in this task need to predict a continuous value between 0-1 that reflects the worthiness to perform fact-checking. the dataset has been annotated in five languages; arabic, bulgarian, english, spanish and turkish @xcite promoting multilingual research. ## architecture the main motivation for our architecture is the recent success that the transformer models had in various natural language processing tasks including text classification @xcite , word sense disambiguation (hettiarachchi and ranasinghe, 2020a, 2021), language identification @xcite etc. apart from providing strong results compared to rnn based architectures @xcite , transformer models like bert @xcite provide pretrained multilingual language models that support more than 100 languages which will solve the multilingual issues of these tasks @xcite . transformer models take an input of a sequence and output the representations of the sequence. there can be one or two segments in a sequence which are separated by a special token [sep] @xcite . in this approach we considered a tweet as a sequence and no [sep] token is used. another special token [cls] is used as the first token of the sequence which contains a special classification embedding. for text classification tasks, transformer models take the final hidden state h of the [cls] token as the representation of the whole sequence @xcite . a simple softmax classifier is added to the top of the transformer model to predict the probability of a class. for the text regression tasks, a fully-connected layer is added on top of the [cls] token. the fully-connected layer will have a single output neuron which predicts the target. for both tasks, all the parameters of the transformer model as well as w are fine-tuned jointly by maximising the log-probability of the gold truth. ## experimental setup we trained a transformer model for each dataset mentioned in section 3. given the very unbalanced nature of the datasets, the transformer models tend to overfit and predict only the majority class. therefore, for each label, we took the number of instances in the training set for the minority class and undersampled the majority class to have the same number of instances as the minority class. we then divided this undersampled dataset into a training set and a validation set using the 0.8:0.2 split. we mainly fine-tuned the learning rate and the number of epochs of the classification model manually to obtain the best results for the validation set. we obtained 1e -5 as the best value for the learning rate and 3 as the best value for the number of epochs for both datasets. the other configurations of the transformer model were set to a constant value over all the experiments in order to ensure consistency between them. we used a batch size of 8, adam optimiser and a linear learning rate warm-up over 10% of the training data. the models were trained using only training data. we performed early stopping if the evaluation loss did not improve over 10 evaluation rounds. the implementation was done using huggingface transformer implementation @xcite . a summary of hyperparameters and their values used to obtain the reported results are mentioned in for multilingual experiments, we first combined data instances from all the languages of each task which left us with two large multilingual false information identification datasets. then we trained the transformer models on that combined datasets. as the multilingual pretrained transformer model, we used the bert-multilingual-cased @xcite model. ## results in as can be seen in the results multilingual models outperformed monolingual models in arabic, bulgarian, spanish and turkish languages while performing on par with english. similar to the results of the previous dataset, these multilingual models are very competitive with the best systems submitted to each of the languages. ## conclusion in this paper, we explored multilingual models for false information identification using two recently created datasets. in our experiments, we observed that multilingual models built using powerful pretrained multilingual transformers perform on par or sometimes even better than the monolingual models. these results are consistent with both datasets and across five languages. findings in this paper would be valuable when building realworld applications for false information identification where maintaining separate machine learning models for each language would be more expensive and chaotic. as future work, we would like to expand this research into more transformer models and more languages. we would like to experiment with how the multilingual transformer models with the crosslingual concepts like xlm-roberta would perform in multilingual false information identification. furthermore, we would explore zero-shot and few-shot learning with multilingual models which would be beneficial to low resource language where the training data is scarce.