Claim Check-Worthiness Detection as Positive Unlabelled Learning (2020)

Abstract: As the first step of automatic fact checking, claim check-worthiness detection is a critical component of fact checking systems. There are multiple lines of research which study this problem: check-worthiness ranking from political speeches and debates, rumour detection on Twitter, and citation needed detection from Wikipedia. To date, there has been no structured comparison of these various tasks to understand their relatedness, and no investigation into whether or not a unified approach to all of them is achievable. In this work, we illuminate a central challenge in claim check-worthiness detection underlying all of these tasks, being that they hinge upon detecting both how factual a sentence is, as well as how likely a sentence is to be believed without verification. As such, annotators only mark those instances they judge to be clear-cut check-worthy. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning that finds instances which were incorrectly labelled as not check-worthy. In applying this, we out-perform the state of the art in two of the three tasks studied for claim check-worthiness detection in English.

## introduction misinformation is being spread online at ever increasing rates @xcite and has been identified as one of society's most pressing issues by the world economic forum @xcite . in response, there has been a large increase in the number of organizations performing fact checking @xcite . however, the rate at which misinformation is introduced and spread vastly outpaces the ability of any organization to perform fact checking, so only the most salient claims are checked. this obviates the need for being able to automatically find check-worthy content online and verify it. the natural language processing and machine learning communities have recently begun to address the problem of automatic fact checking @xcite @xcite @xcite @xcite . the first step of automatic fact checking is claim check-worthiness detection, a text classification problem where, given a statement, one must predict if the content of that statement makes "an assertion about the world that is checkable" @xcite . there are multiple isolated lines of research which have studied variations of this problem. figure 1 provides examples from three tasks which are studied in this work: rumour detection on twitter @xcite , check-worthiness ranking in political debates and speeches @xcite @xcite , and citation needed detection on wikipedia @xcite . each task is concerned with a shared underlying problem: detecting claims which war-rant further verification. however, no work has been done to compare all three tasks to understand shared challenges in order to derive shared solutions, which could enable improving claim checkworthiness detection systems across multiple domains. therefore, we ask the following main research question in this work: are these all variants of the same task, and if so, is it possible to have a unified approach to all of them? we answer this question by investigating the problem of annotator subjectivity, where annotator background and expertise causes their judgement of what is check-worthy to differ, leading to false negatives in the data @xcite . our proposed solution is positive unlabelled conversion (puc), an extension of positive unlabelled (pu) learning, which converts negative instances into positive ones based on the estimated prior probability of an example being positive. we demonstrate that a model trained using puc improves performance on english citation needed detection and twitter rumour detection. we also show that by pretraining a model on citation needed detection, one can further improve results on twitter rumour detection over a model trained solely on rumours, highlighting that a unified approach to these problems is achievable. additionally, we show that one attains better results on political speeches check-worthiness ranking without using any form of pu learning, arguing through a dataset analysis that the labels are much more subjective than the other two tasks. the contributions of this work are as follows: ## methods the task considered in this paper is to predict if a statement makes "an assertion about the world that is checkable" @xcite . as the subjectivity of annotations for existing data on claim check-worthiness detection is a known problem @xcite , we view the data as a set of positive and unlabelled (pu) data. in addition, we unify our approach to each of them by viewing wikipedia data as an abundant source corpus. models are then trained on this source corpus using variants of pu learning and transferred via fine-tuning to the other claim check-worthiness detection datasets, which are subsequently trained on as pu data. on top of vanilla pu learning, we introduce positive unlabelled conversion (puc) which relabels examples that are most confidently positive in the unlabelled data. a formal task definition, description of pu learning, and explanation of the puc extension are given in the following sections. ## experimental results to what degree is claim check-worthiness detection a pu learning problem, and does this enable a unified approach to check-worthiness detection? in our experiments, we progressively answer this question by answering the following: 1) is pu learning beneficial for the tasks considered? 2) does pu citation needed detection transfer to rumour detection? 3) does pu citation needed detection transfer to political speeches? to investigate how well the data in each domain reflects the definition of a check-worthy statement as one which "makes an assertion about the world which is checkable" and thus understand subjectivity in the annotations, we perform a dataset analysis comparing the provided labels of the top ranked checkworthy claims from the puc model with the labels given by two human annotators. in all experiments, we report the mean performance of our models and standard deviation across 15 different random seeds. additionally, we report the performance of each model ensembled across the 15 runs through majority vote on each sample. ## discussion and conclusion in this work, we approached claim checkworthiness detection by examining how to unify three distinct lines of work. we found that checkworthiness detection is challenging in any domain as there exist stark differences in how annotators judge what is check-worthy. we showed that one can correct for this and improve check-worthiness detection across multiple domains by using positive unlabelled learning. our method enabled us to perform a structured comparison of datasets in different domains, developing a unified approach which outperforms state of the art in 2 of 3 domains and illuminating to what extent these datasets reflect a general definition of check-worthy. future work could explore different neural base architectures. further, it could potentially benefit all tasks to consider the greater context in which statements are made. we would also like to acknowledge again that all experiments have only focused on english language datasets; developing models for other, especially low-resource languages, would likely result in additional challenges. we hope that this work will inspire future research on check-worthiness detection, which we see as an under-studied problem, with a focus on developing resources and models across many domains such as twitter, news media, and spoken rhetoric.