A New Approach to Claim Check-Worthiness Prediction and Claim Verification (2020)

Abstract: The more we are advancing towards a modern world, the more it opens the path to falsification in every aspect of life. Even in case of knowing the surrounding, common people can not judge the actual scenario as the promises, comments and opinions of the influential people at power keep changing every day. Therefore computationally determining the truthfulness of such claims and comments has a very important societal impact. This paper describes a unique method to extract check-worthy claims from the 2016 US presidential debates and verify the truthfulness of the check-worthy claims. We classify the claims for check-worthiness with our modified Tf-Idf model which is used in background training on fact-checking news articles (NBC News and Washington Post). We check the truthfulness of the claims by using POS, sentiment score and cosine similarity features.

## introduction today we live in a world where falsehood seems to reflect everywhere be it in administration, sports, entertainment sector and even in the education field. many popular and influential personalities seem to be vulnerable in keeping their words. the opinions and comments they make, their claims keep changing frequently. therefore we can not blindly rely on present news. during the 2016 us presidential campaign, people came to realize how fake news could be spread in mainstream news channels and social media. @xcite reported the influence of fake news on social media during the election. they showed that about 171 million tweets were made during the election among which 25% were fake or extremely biased. many journalists started investigation into identifying the actual truth. however, it was a time consuming and tedious task to do the work manually. this problem gave rise to the concept of automatic fact and claim checking. research has been going on since then to effectively tackle this problem which has proved to be a very challenging problem. typically for the claim verification task, relevant evidence related to the claims is collected first and then the claim is compared with the evidence to know the actual fact. @xcite did this with the help of knowledge graph taken from wikipedia. we propose a suitable rule based approach with the help of feature engineering for the task. our work consists of two tasks, first we extract the claims which are check-worthy and then we verify the truthfulness of these check-worthy claims. we carried out our experiments on the dataset of the fact checking master (preslav nakov and martino., 2018) shared task, organized in clef-2018, which deals with fact checking on the u.s. presidential debate articles of 2016. extraction of check-worthy claims is carried out in two processes i.e. supervised and unsupervised approach. in unsupervised approach, the checkworthy claims are extracted with pos tags and k-means clustering algorithm. dataset related to claims can be generated from any conversation with the help of this method. in supervised approach, we have collected some fact checking news articles (nbc news and washington post) for background training and a modified tf-idf model is created to classify the claims whether check-worthy or not. cosine similarity, sentiment scores and pos tags are also used here. on comparing with the original labels, this model gives an accuracy of 98.6% when passed along with gbm. claim verification is performed by comparing the classified check-worthy claims with the fact checking news articles to verify their truthfulness. pos tagging, cosine similarity are used to search for the explanations of the claims from the arti-cles. with all these features together we make a hypothesis for the final classification. these two tasks are consecutively done in a work @xcite previously. else there are many works to influence on individual approaches of the model. the paper is divided into many sections, section 2 describes related works, dataset in section 3, features and proposed methodology in section 4 and 5 respectively, results in section 6 and conclusion is given in section 7. ## related works fact-checking has become a trending topic recently. @xcite provides a survey on fake news research and their study focuses on fake news from four perspective -the false knowledge it carries, its writing style, its propagation patterns, and the credibility of its creators and spreaders. pepa gencheva and koychev (2017) extracted the check-worthy claims by comparing them with some popular news articles and verified the truthfulness of the claims using support vector machines (svm) and feed-forward neural networks (fnn). mihai surdeanu and manning (2010) used conditional random field (crf) for legal claim identification. firstly they used optical character recognition (ocr) to convert pdf documents into text and then they used four types of crf architectures for the actual task. @xcite reported a work on fake news identification in which sentiment scores and tf-idf are used as features to build a majority voting model with four classifiers -gradient boosting, random forest, extra tree and xgboost classifiers to identify fake news. @xcite showed that false news has various emotional patterns to mislead the readers and with the emotional sentiment features they propose an lstm model for the classification task. takatsuka. (2016 ## dataset we carried out our experiments on the dataset of the fact checking master (preslav nakov and martino., 2018) shared task, organized in clef-2018. the dataset contains stated claims of the u.s. presidential debates (2016) with a total of 1,403 sentences in the first presidential debate and 1,303 sentences in the second presidential debate. there are four speakers -holt (host of the first presidential debate), cooper (host of the second presidential debate), hillary clinton and donald trump. all the claims made by mr. trump and ms. clinton in these debates can be sensed manually since all of the claims show some actions happened in the past, or any comments or actions from the opponent in the past. observing the patterns, we analysed the prominent features and developed our models. ## results and discussion we calculate the values of accuracy, precision, recall and f-score for each model proposed in the paper. confusion matrices are also shown for result visualization for both the debate articles. all the results are analysed below. ## conclusion and future work the method which we use gives a very good result in check-worthiness classification, and the approach we propose here is a modified version of tf-idf for using background fact checking article. this can help us to do further research in background training. other approach we have used is unsupervised approach, where we studied every feature very carefully and built a model. though this model needs further modification but it can be used to generate new data on claims from any debate, conversation or interview. so it can contribute to the field of dataset creation. now for claim truthfulness, we first extracted the proper explanations from the articles, then applied our own threshold for classification. this approach works good for the class true and false, but for half true, it needs tuning and modification. further we can apply our modified tf-idf model in other researchers' works and check the efficiency. the model and hypothesis for truthfulness verification needs further improvement. we will try to add deep learning methods also. it can be concluded that our proposed model is very versatile and can be used in other fields as well.