Studies Towards Language Independent Fake News Detection (2021)

Abstract: We have studied that fake news is currently one of the trending topic and it causes problem to many people and organization. We use COVID19 domain and 7 languages to work on. We collect our data from twitter. We build two types of model one is language dependent and other one is language independent. We get better result in language independent model for English, Hindi and Bengali language. Results of European languages like German, Italian, French and Spanish are comparable in both language dependent and independent model.

## introduction in our day-to-day world, we humans mostly generate unstructured data and mostly the textual data in a large scale. while utilizing information grounded in such textual data, our efforts become useless when we stuck in handling fake news or misinformation. we observe three main different types of news -legitimate news, fake news and satirical news and all these news differ with respect to two parameters; authenticity and intent. if authenticity of news is not verifiable or false and its intent is to mislead the readers, then it is known as fake news. on the other hand, if authenticity of the news is verifiable or true and its intent is to convey or spread the authenticity to the readers then it is known as legitimate news. finally, if authenticity of news is not verifiable or false and its intent is oriented towards entertainment, then such type of news is known as satirical news. in the present approach, we have considered only fake and non-fake news. here we use linguistic features for detecting fake news. we tried to detect fake news in language dependent and independent both ways. we have also checked which features are more important than others in detecting fake news for a particular language. our remaining paper is divided into different sections. in section 2 we will see some studies or previous works related to fake news detection. in section 3 we will see our dataset, upon which we performed this research work. after that in next section we will discuss methods or model architecture. in section 5 we discussed about result and error analysis and lastly in section 6 we draw a conclusion and discussed about future works. ## related work misinformation is currently one of the balmy topics of last six to seven years. in this fake news field particularly many researches are already been executed and many are currently also going. researchers suggested many different ways of detecting fake news. if we generalize them, then we can come to a conclusion that we can detect a news fake or not based news content or social context. these are the only two generalize way of detecting fake news. we can call these two waysa) news content model, and b) social context model. again there are two divisions of news content model based upon which we can detect a fake newsa) knowledge based detection, and b) style based detection. for knowledge based method everything is depend on a knowledge base that we extracted from the news. after creating that knowledge base we have to studies towards language independent fake news detection soumayan bandhu majumder @xcite and dipankar das 1 compare it with some reliable source to check its authenticity. in style based method we mainly focus on linguistic features and based on that we predict the news. like news content model social context model also divided in two categories, based on which we can detect a fake newsa) propagation based techniques, and b) credibility based techniques. in case of propagation based method, we have to find propagation path of the news on the social media and have to track the original source of the news. but for credibility based method, we have to find the various relationship between news article and users, publishers, posts, shares, comments etc. several researchers have explored the area into different ways. however, these above mentioned techniques have one problem that is they can detect the fake news of a particular language and particular domain. but, one of the important issues is that whether we can detect it in a language independent fashion or not. therefore, in the present attempt, one of our aims is to detect the fake news in language independent way. moreover, none of the above mentioned approaches deal with less computerized and less resourced language like bengali. here, in the present task, we have developed dataset as well as explored the detection techniques for bengali along with english, hindi and other european languages. ## dataset preparation undoubtedly, the term fake news comes into our mind while we think of social media. thus, we aimed to collect data from a social media like twitter. we also collected newspaper data from different languages like english, hindi and bengali. already there is a popular multilingual fake news dataset present in covid19 domain, but this dataset does not contain german and bengali languages and secondly size of our dataset is much larger than this dataset. in case of newspaper data, we crawled sentences (mostly news) from various web sources and manually labeled them. we crawled bengali sentences from 'abp ananda 1 ' and hindi sentences from 'abp news 2 ' and 'aajtak 3 ', as well as 'twitter 4 '. in case of collecting data for european languages, we considered the data available in the clef shared task by participation. in addition, we have collected covid19 twitter data from 15th march, 2020 to 15th may 2020 of 7 different languages (german, italian, french, spanish, english, hindi and bengali). we collect our covid19 related data from twitter using tweet-scrapper library. it has been observed in preliminary study that fake news datasets are always skewed because the frequency of real news data are much more than fake news data. thus, we tried to maintain a similar ratio of fake and real news in each of the languages. however, we were able to collect very less number of fake tweets in bengali and hindi in covid19 domain. thus, in order to train the models for bengali and hindi, we had to add more data from other domain as well. when we started collecting our dataset from twitter, we have the following tweet filtering feature options like 'tweet_id ', 'hashtags', 'noun_count', 'adj_count', 'verb_count', 'pron_count', 'adv_count', 'other_pos', 'sentiment', 'tags', 'tags_org', 'tags_per', 'tags_loc', 'tags_misc'. for annotating our dataset, we considered the help of factcheck.com. the detail statistics of the dataset are shown in ## language dependent classification in order to investigate the roles of language to detect fake news, here each of the models and its input features are exclusive to that particular language. we employ different types of machine learning algorithms and also ensemble them in order to achieve better results by exploring the benefits of individual machine learning classifiers. here we first extracted some features from the tweets. we here don't use twitter specific features because we want our experiment to be on general purpose, instead of twitter specific fake news. therefore, we use different types of open source libraries to extract different types of features from the text. here, we have mainly conducted experiment upon 7 different languages. among these 7 languages we collected our covid19 related data purely in 5 languages and for other two languages (bengali and hindi), we added data from other domain as the covid19 data of these two languages were very less in number. from each text or tweet, we extracted a couple of features for different languages using different libraries. for english and european languages, we used spacy foot_2 and polyglot library foot_3 . like for french we have to download 'fr_core_news_sm' (which is exclusively for french). finally, we had to install spacytextblob library foot_4 . for hindi language, we use nltk library foot_5 and for bengali language, we use bengali-nlp library @xcite . we extracted 17 different features (e.g., 'word count', 'char count', 'word density', 'punctuation count', 'title word count', 'upper case word count ', 'noun count', 'verb count', 'adjective count', 'pronoun count', 'adverb count', 'other pos', 'sentiment', 'tags_loc', 'tags_misc', 'tags_per', 'tags_org'). classification algorithms are of three types, such asbinary classification, multiclass classification and multi-label classification. here, we have used binary classification algorithms, because our output is between any one of the fake and real class. we have used logistic regression, knn, svm, random forest, xgboost, ensemble learning and naïve bayes to accomplish our goals. here we take logistic regression based model as our baseline model. ## language independent classification here, we have conducted experiments to see if we can detect fake news in language independent way or not. we here mainly focused on multilingual bert model. this bert model is already pre-trained on some different corpus. therefore, we will first fine tune this multilingual bert model with our different language corpora, then we received a fixed length embedded output through this model for each of the tweets, then we pass this output through the two layers of artificial neuron network of different unit size and at last we pass that output through the sigmoid layer. ## result comparison after doing both language dependent and independent fake news detection, now we will compare both results. table3: result comparison between two models here, we compare our results based upon two parameters. firstly, for all over performance we take f1 score as our parameter. secondly, we take recall of fake class as our second parameter. we take this second parameter because we want to see how many of fake news are correctly predicted as fake news. here, we give more emphasize on fake data over real or neutral data. in the above mentioned table, recall is for fake class. for english language, our independent model gives the better result in both parameters. for french, german, italian and spanish results of dependent and independent models are comparable in both parameters. for hindi language, our independent model gives better results in case of both the parameters. it has also been observed that the bengali language independent model gives better result in f1 score, but dependent model gives better result in recall value. ## conclusion the present work deals with some of the modern day topics like fake news and covid19. we first collect our data using various sources like twitter, newspaper and shared task. we analyse fake news in multilingual aspect and check how each model and language performs differently in each scenario. though our data in hindi and bengali is very less but still we got some good results in some model. in future if we can get more data in hindi and bengali then we can build more concrete models upon these two languages. here we also learn the basic architecture and concepts of bert model which is one of the most popular pre-trained models of nlp. we build a language independent model using bert multilingual which supports many languages. so with our collected data we just fine-tune bert model with each language data. it produces some astonishing results. though our data is less but it still gives very good results. in english, hindi and bengali language our language independent model outperformed most of the language dependent models. in case of european languages like german, french, italian and spanish both language dependent and language independent model performs very good and their results are comparable. one thing we should mention that in case of german language our language independent model predicted all real news correctly and only four fake news wrongly, which quite astonishing.