Detecting Multilingual COVID -19 Misinformation on Social Media via Contextualized Embeddings (2021)

Abstract: We present machine learning classifiers to automatically identify COVID-19 misinformation on social media in three languages: English, Bulgarian, and Arabic. We compared 4 multitask learning models for this task and found that a model trained with English BERT achieves the best results for English, and multilingual BERT achieves the best results for Bulgarian and Arabic. We experimented with zero shot, few shot, and target-only conditions to evaluate the impact of target-language training data on classifier performance, and to understand the capabilities of different models to generalize across languages in detecting misinformation online. This work was performed as a submission to the shared task, NLP4IF 2021: Fighting the COVID-19 Infodemic. Our best models achieved the second best evaluation test results for Bulgarian and Arabic among all the participating teams and obtained competitive scores for English.

## introduction automatic detection of misinformation online is a crucial problem that has become increasingly necessary in recent years. misinformation is shared frequently online, especially via social media platforms which generally do not filter content based on veracity. the ongoing covid-19 pandemic has highlighted the importance of creating tools to automatically identify misinformation online and to help stop the spread of deceptive messages. during a global health crisis, misinformation can cause tremendous damage. a recent study polled americans about beliefs in covid-19 misinformation, and found that many conspiracy theories were believed by a substantial percentage of participants. for example, 20% of participants believed that the pandemic is a ruse "to install tracking devices inside our bodies." such conspiracy theories, when shared widely online, could influence people to make choices based on those beliefs which can jeopardize their own health and safety as well as others around them. there has been recent work in the nlp community aimed at identifying general misinformation on social media @xcite and particularly covid-19 misinformation @xcite . most of this prior work has focused on data in english. in this paper we address the problem of cross-lingual identification of covid-19 misinformation. there is a severe data shortage of high quality datasets that are labeled for misinformation in multiple languages. because of this, we need to develop models of deception and misinformation that can leverage large amounts of training data in a source language, such as english, and generalize to new target languages. some prior work on misinformation and deception detection has been applied to a cross-cultural setting @xcite and recently to a cross-lingual setting @xcite . whereas previous approaches have focused on single task models, in this work we train four different multitask models and demonstrate their performance in cross-lingual settings for identifying misinformation in social media. two of these models are based on bert @xcite . we show that even with no training data in the target language, the multilingual bert based model can obtain 0.685 f1 in english, 0.81 f1 in bulgarian and 0.672 f1 in arabic. we used the tweet data provided for the fighting the covid-19 infodemic shared task @xcite . 1 the data was created by answering 7 questions about covid-19 for each tweet @xcite . questions include: q1 -does the tweet contain a verifiable factual claim? q2 -does the tweet appear to contain false information? each question has a yes/no (binary) annotation. however, the answers to q2, q3, q4 and q5 are all "nan" if the answer to q1 is no. the data includes tweets in three languages: english, bulgarian and arabic. an example of an english tweet from the dataset along with its 7 labels is shown below. tweet: anyone else notice that covid-19 seemed to pop up almost immediately after impeachment failed? labels: q1 yes, q2 yes, q3 yes, q4 yes, q5 no, q6 yes, q7 no the training, development and test data sizes for each of the three languages are shown in ## methodology the task is to predict 7 properties of a tweet about covid-19 misinformation based on the corresponding 7 questions mentioned in section 2. we use four multitask learning models for this task as described below. ## experiments we performed cross lingual experiments in different setups as outlined below. we are interested in gauging the cross-lingual ability of prediction models when trained on data from one language (source language) and tested on data from another language (target language). we compare 4 experimental conditions, varying the amount of source language and target language data that is used for training. the four conditions are described below. zero shot in this condition, we only used the source language training data to train the model. the model does not see any training data in the target language and we only evaluated the model on the target language dev set. the advantage of the zero shot setup is that it enables us to evaluate the prediction models in conditions when no training data is available in the target language. few shot in this setup, we considered all the source language training data combined with x training samples from the target language. we set x to 50, 100, 150 and 200. we select x samples from the complete target language training data uniformly at random to simulate the few shot setup. for arabic, the number of training data samples is 198 (without using additional data, see target in this setup, we used only the target language training data for training. this setup enables us to evaluate the models when there is availability of training data in the target language, and compare monolingual with cross-lingual classification. ## conclusion in this paper, we described hunter speechlab's submission to the shared task, nlp4if 2021: fighting the covid-19 infodemic. we explored the cross-lingual generalization ability of multitask models trained from scratch (logistic regression, transformer encoder) and pre-trained models (english bert, m-bert) for deception detection. we found that even without using any training samples in bulgarian and arabic (zero shot), m-bert achieves impressive scores when evaluating on those languages. in some cases, using just a few training samples in the target language achieves results equal or better than using all the training data in the target language. our best systems are based on english bert for english and multilingual bert for bulgarian and arabic. we obtained competitive evaluation test scores on all the three languages, especially bulgarian and arabic for which we obtained second best scores among all participating teams. in future work we will further explore the cross-lingual generalization ability of bert based models in detecting false or deceptive information.