Claim extraction from text using transfer learning. (2020)

Abstract: Building an end to end fake news detection system consists of detecting claims in text and later verifying them for their authenticity. Although most of the recent works have focused on political claims, fake news can also be propagated in the form of religious intolerance, conspiracy theories etc. Since there is a lack of training data specific to all these scenarios, we compiled a homogeneous and balanced dataset by combining some of the currently available data. Moreover, it is shown in the paper that how recent advancements in transfer learning can be leveraged to detect claims, in general. The obtained result shows that the recently developed transformers can transfer the tendency of research from claim detection to the problem of check worthiness of claims in domains of interest.

## introduction the advent of social media and mobile-based messaging applications has led to a rapid spread of fake news and misinformation, which are having serious consequences in real-world scenarios. manual factcheckers are often overwhelmed by the number of sources that need to be verified. facebook has released some statistics in its regular enforcement reports, as follow: the automated fact-checking process can help to mitigate this problem. the very first step of a factchecking system is to identify claims from the text, which can then be verified by querying a knowledge base or by converting it into a question and looking for suitable answers using a crawler @xcite . since identifying claims is the very first step of a fact-checking pipeline, the performance on this task has a major impact on the result of an end to end fact-checking system. as the performance of the state-of-the-art deep learning models are highly dependent to the availability of labelled data, in this paper, we introduce a novel dataset to assist in training models for the task of claim detection. moreover, we investigate the performance of some of the recent transformers based language models (e.g., bert) on the proposed data. the paper is organized as follow. some of the recently proposed models and datasets for the task of claim detection are presented in section 2. section 3 contains a brief introduction to transfer learning and the state-of-the-art models. the proposed approaches for compiling the dataset and detecting claims, and also the baseline model are described in details in section 4. ## related work since fact-checking and fake news detection is time-consuming and effort-full tasks, in many systems, claim detection and argument mining are considered as the preliminary modules that provide input for fact-checking module. claimbuster, which is introduced in @xcite , tries to rank political claims in debates based on their check-worthiness using supervised models. to this end, a labelled dataset of spoken sentences by presidential candidates is constructed. each sentence in the dataset is given one of the three possible labels; it is not a factual claim; it is an unimportant factual claim; it is an important factual claim. among various classification method source of training data model used precision recall f1 claimbuster hand annotated us presidential debates svm 79% 74% 76% claimrank popular fact checking organizations fnn 93% 65% 77% full fact crowd sourced annotations logreg 88% 80% 83% logically annotated news articles google use 90% 89% 89% another recent approach to claim detection is claimrank @xcite . the authors claim that although the system originally trained on political debates, it works for any text, e.g. interviews and regular news articles. they compiled a dataset by taking the outputs of fact-checking of a political debate, published by nine reputable organizations simultaneously. models were created to predict if the claim would be highlighted by at least one or by a specific organization. the modelling is done with a large variety of features from both the individual sentence and the wider context around it. to classify claims and rank them on their check worthiness, a two hidden layered neural network is trained. adler and boscaini-gilroy (2019) used google's universal sentence encoder to obtain the sentence embedding and passed it through a logistic regression layer to get the final classification. their model has trained on a dataset based on news articles created by them. a detailed comparison of the previously proposed methods is presented in ## transfer learning neural networks need large scale datasets to be trained efficiently. they are difficult to apply where the available data is sparse. the imagenet moment @xcite where fine-tuning, a model trained on a large dataset could be applied in various applications with limited availability of data, created a breakthrough in computer vision. language modelling was seen as the most appropriate task to model this achievement by imagenet in natural language processing (nlp). when trained for language modelling, neural networks capture the basic structure and understanding of language and can thus be fine-tuned on downstream tasks with limited data. many attempts have been made in creating a sophisticated model trained on a large dataset that can be fine-tuned easily on a downstream task. ulmfit (howard and ruder, 2018) was one of the earliest models to achieve this through an awd lstm @xcite . elmo @xcite used a bi-directional lstm architecture with character level encoding of features to avoid out of vocabulary errors. bert @xcite used the recently developed transformer architecture to perform language modelling. transformers have the advantage of being able to train faster due to possibility of parallelization. distilbert model (sanh et al.) was created by a method called model distillation, and it is 40% smaller, and 60% faster than bert and it retains 97% of its performance, making it more deployment friendly. in our experiments, bert and dis-tilbert have been used to extract claims from text. ## proposed approach in this section we present the applied approaches for compiling the dataset and detecting claims. moreover, the baseline model is also described in this section. ## conclusion and future work in this work, we have presented our publicly available dataset and quantified the performance of bert and distilbert in detecting claims in general. these are one of the most advanced transfer learning methods available and can provide highly accurate results with fewer data. the transfer learning based pre-training of these models helped it to achieve high evaluation scores despite having been trained with a fraction of the available dataset. until now, fake news detection has been thought of as a two-step process consisting of detecting claims and verifying them. the first part can be further subdivided into detecting claims and sorting them according to the check worthiness of a claim. other research domains such as argument mining would benefit from this since they would want to sort claims according to argumentative claims rather than check worthiness. researchers can thus decide themselves the type of claim to filter out. we will address the problem of check worthiness in our future work.