Leveraging Expectation Maximization for Identifying Claims in Low Resource I ndian Languages (2021)

Abstract: Identification of the checkable claims is one of the important prior tasks while dealing with infinite amount of data streaming from social web and the task becomes a compulsory one when we analyze them on behalf of a multilingual country like India that contains more than 1 billion people. In the present work, we describe our system which is made for detecting check-worthy claim sentences in resource scarce Indian languages (e.g., Bengali and Hindi). Firstly, we collected sentences from various sources in Bengali and Hindi and vectorized them with several NLP features. We labeled a small portion of them for check-worthy claims manually. However, in order to label rest amount of data in a semi-supervised fashion, we employed the Expectation Maximization (EM) algorithm tuned with the Multivariate Gaussian Mixture Model (GMM) to assign weakly labels. The optimal number of Gaussians in this algorithm is traced by using Logistic Regression. Furthermore, we used different ratios of manually labeled data and weakly labeled data to train our various machine learning models. We tabulated and plotted the performances of the models along with the stepwise decrement in proportion of manually labeled data. The experimental results were at par with our theoretical understanding, and we conclude that the weakly labeling of check-worthy claim sentences in low resource languages with EM algorithm has true potential.

## introduction misinformation has taken over the internet and it is now a well-recognized problem all over the world. governments, news agencies, information security people all are trying to fight this menace with the helping hands from the researchers of social networks, natural language processing, data science and many more. with the exponential rise in in-formation or news, making automated systems of fact-checking becomes a necessity. researchers have made significant achievements in this field. but most of the research is in high resource languages like english. however, misinformation or fake news doesn't stop in the high resource language. fake news is spreading far and wide in low resource languages as well, like in the indian languages of @xcite . a lot more research needs to be done in this arena and our present work we have contributed towards it. it is a common practice in machine learning to use a semi supervised approach to weakly label data when labeled data is scarce @xcite . on the other hand, filtering out check worthy claim sentences has been attempted by many teams as mentioned in @xcite , @xcite . while @xcite has made a generic and large system, @xcite worked on twitter feeds. however, the problem of identifying claim sentences in indian languages has not been attempted much and thus doesn't have a lot of labeled data. so here, we use the expectation maximization (em) @xcite , a semi supervised algorithm to assign weakly labels to a lot of unlabeled data in indian languages. suggested by @xcite , automated fact checking is a two-fold task: 1) identification of check-worthy sentences, and 2) checking their trustworthiness based on some reliable sources. as checking the trustworthiness or veracity is a very resource intensive and computationally expensive job, identification or filtering out the checkworthy sentences becomes very important. in this work, we have used the semi supervised expectation maximization (em) @xcite unlabeled data is a common practice in machine learning and nlp. but, using em or similar algorithms to weakly label check-worthy sentences has rarely been tried, and has certainly never been attempted in the context of indian languages. the rest of the sections are organized as follows. section 2 gives the approaches in developing the datasets while section 3 and section 4 describe the algorithms of expectation maximization (em) and gmm, respectively. the experimental results are shown in section 5 along with important observations. finally, section 6 concludes the draft. ## data set preparation we prepared our own data for this work. we crawled sentences (mostly news) from various web sources and manually labeled some of them. we crawled bengali sentences from 'abp ananda' ( https://bengali.abplive.com/ ) and hindi sentences from 'abp news' ( https://www. abplive.com/ ) and 'aajtak' ( https://aajtak. intoday.in/ ), as well as 'twitter' (https:// twitter.com/?@xmath0. we manually labeled a portion of the data (about 1500 for bengali, and about 900 for hindi). there were two classes. class 1 refers to check-worthy claims, whereas class 0 refers to not-check-worthy sentences(that is, the rest of the sentences). only the sentences, which is a fact checkable claim, with a clear context, are put in class 1. for example the sentence ('it is to be mentioned that in 2014, rahul defeated smriti irani by 1 lakh 6 thousand votes in amethi constituency.') is considered a checkworthy claim and hence would be labeled as class 1. whereas the sentence ('the court summoned him on thursday for this issue') is not considered a check-worthy claim and hence would be labeled as class 0. we extracted tf-idf scores, unigram, bi-gram and part of speech tags, as features for both the languages. it has to be mentioned that the fitting of data in a multivariate gaussian mixture model is a very computationally expensive job. experimentation by using all the thousands of features in our gaussian mixture model (19907 for bengali, and 27928 for hindi) was not possible in our available hardware setup. therefore, we used principal component analysis (pca) to reduce the dimensionality of the data to 1000. for bengali we retained 92.9% data whereas in hindi we retained 96.6% data. ## em -semi-supervised algorithm it is described in @xcite that em is an iterative optimization method to estimate some unknown parameters θ, given measurement data u. however, we are not given some "hidden" nuisance variables j, which needs to be integrated out. in particular, we want to maximize the posterior probability of the parameters θ given the data u, marginalizing over j: @xcite describes it from a statistical point of view in detail and @xcite states its proof. the em algorithm seeks to find the maximum likely estimation of the marginal likelihood by iteratively applying the two steps namely expectation and maximization. the em algorithm needs a mixture model to represent the data. here, we use 'multivariate gaussian mixture models' to represent the data. a gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of gaussian distributions with unknown parameters. 'multivariate gaussian mixture models' are gaussian mixture models in a multivariable or multidimensional vector space. ## experimental setup firstly, we separated out the validation and the test sets from the manually labeled data with a train-validation-test split of 60%-20%-20% in ratio. then, we made the training set using (strongly/human) labeled training data plus some of the unlabeled data while the validation and the test set consist of entirely labeled data. in each step of expectation maximization, we fit the training set to a gaussian mixture model with a chosen number of gaussians and assign the weak labels to the unlabeled data. then, we measured the efficiency of the gaussian mixture model by training a logistic regression component and evaluated on the validation set. we repeat this step and select the gmm which gives the best results on the validation set. at last, we report the performance of the selected gmm on the test set by modeling various machine learning models using human labeled data plus weakly labeled data. in addition, we have also explored this for various proportions of human (strongly) and weakly labeled data, and observed the performances with respect to the change in proportion. the details of each step are given in the following subsections as pseudo-code. (note that we denote check-worthy claim sentences as class '1' and rest sentences as class '0') (shown in figure 2 ). ## experiments and evaluation we considered various gaussian mixture models, with different numbers of gaussian and selected the model which has the best performance on the validation data. we enlist the precision, recall, and the f1 score of the models on the validation data for both the classes and both the languages in ## conclusion in the present work, we have attempted to filter out check-worthy claims, which is the first step of fact-checking. since there is a lot of information, and little intelligent labour available for manually tagging check-worthy sentences, we have used a semi-supervised algorithm to quickly tag a lot of check-worthy sentences by achieving a satisfied level of accuracy. we used the semi supervised expectation maximization algorithm with gaussian mixture models, to weakly label the unlabeled data. we crawled a lot of data from the web and weakly labelled them using the expectation maximization algorithm. we trained some classification models with this weakly labeled data and observed the results on human annotated data. we find that the performance is very close to the performance by training with manually labeled data. therefore, we can conclude that our semi supervised algorithm works well in the task of identifying fact checkable sentences. we expect the efficiency will increase in future with more complicated models and more labeled data. for the expectation maximization algorithm, computation power is also a concern. it is difficult to fit gaussian mixture models with high dimensions with currently available hardware. however we did proof that this approach works for identifying a lot of check-worthy claims with fair accuracy in indian languages.