Generating Fact Checking Briefs (2020)

Abstract: Fact checking at scale is difficult—while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem. However, despite good intentions, contributions from volunteers are often error-prone, and thus in practice restricted to claim detection. We investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs. We investigate passage-based briefs, containing a relevant passage from Wikipedia, entity-centric ones consisting of Wikipedia pages of mentioned entities, and Question-Answering Briefs, with questions decomposing the claim, and their answers. To produce QABriefs, we develop QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. To train its components, we introduce QABriefDataset We show that fact checking with briefs — in particular QABriefs — increases the accuracy of crowdworkers by 10% while slightly decreasing the time taken. For volunteer (unpaid) fact checkers, QABriefs slightly increase accuracy and reduce the time required by around 20%.

## introduction fact checking is a challenging task. it requires deep knowledge of a claim's topic and domain, as well as an understanding of the intricacies of misinformation itself. checking a single claim can take professional fact checkers 15 minutes to one day @xcite . volunteers on the other hand are not considered accurate enough; with access to a search engine, @xcite report crowdsourced fact check accuracies of around 58%. this result corroborates earlier reports foot_0 by fact checking websites which attempted to engage volunteers, but reported success only for claim detection, which is considered a much simpler task @xcite . this is problematic, both from the perspective of using crowdsourced fact checking to combat misinformation and from the perspective of helping individuals fact check themselves. one path for scaling fact checking could be through full automation, taking a claim as input and producing a verdict @xcite . existing work has framed fact checking as classification, often supported by evidence @xcite @xcite . however, due to the limitations of existing automated solutions, practitioners prefer solutions that improve efficiency in reaching a verdict, instead of approaches to the complete process @xcite . in this work, we propose briefs to increase the accuracy and efficiency of fact checking ( figure 1 ). by generating fact checking briefs, our models aim to provide evidence a human fact checker would find useful. we investigate several approaches, including returning wikipedia passages that relate to the claim, and an entity linking approach that shows information about mentioned entities. crucially, we introduce qabriefs -a set of relevant questions and their answers (see figure 2 ). to learn how to produce qabriefs and create training data, we use crowdsourcing to gather such briefs based on existing fact checks. we create qabriefdataset, a collection of about 10,000 qabriefs with roughly 3 question and answer pairs each. we introduce qabriefer, a novel model that performs structured generation via claim-conditioned question generation and open domain question answering. each question is used to identify evidence using a search engine. finally, a pretrained question answering model is finetuned to generate answers and produce the full brief. in experiments with crowdworkers, qabriefs improve accuracy by 10% compared to using only a search bar while reducing the time a fact check takes. for volunteer fact checkers, accuracy is improved by 4% and the process is 20% faster compared to using a search bar. using qabriefs from human annotators leads to the largest improvement, followed by briefs generated by qabriefer and other proposed forms of briefs. this suggests that briefs are a promising avenue for improving crowdsourced fact checking. further, qabrief-dataset can be used to develop models capable of answering challenging, real world questions. ## briefs for fact checking fact checkers must comprehend each part of a claim, which requires gathering information about a wide range of concepts-a precise definition of a term, how a politician voted, or the exact contents of a bill. such knowledge is available in many sources: knowledge bases, statistical reports, or on the internet. we introduce the notion of briefs to provide relevant information to fact checkers-as if briefing them before fact checking-and explore three possible forms: passage briefs, entity briefs, and question answering briefs. we show how they can be constructed with modern nlp approaches. ## qabrief dataset to train and evaluate models to generate qabriefs, we collect a dataset of questions based on claims, together with answers to those questions found on the open web. crucially, annotators first read the article from a fact checking website that describes how the claim was checked, and then decompose the process into questions, for which answers are provided. the claims for the dataset are sourced from existing fact checking datasets, specifically datacommons 2 and multifc @xcite . the annotator instructions are in the appendix and examples are shown in figure 3 . while numerous question generation and answering datasets exist, none of them focuses on using questions and answers to combat misinformation. qabriefdataset focuses on this real world problem, with each question grounded in a claim that was actually fact checked. further, existing datasets are quite different from our usecasefor example, many datasets are based on wikipedia, but fact checkers find evidence from other sources. many datasets have short answer spans, but our questions are complex, so require longer answers. ## qabrief model the automatic generation of qabriefs presents numerous modeling challenges. generating such a brief is a hierarchical process: writing the questions, and then conditioned upon the questions, searching the web and writing the answers. while many question answering datasets exist, questions in qabriefdataset are grounded on real claims that were fact checked. the diversity of the claims renders reusing questions across claims unlikely to work, thus precluding the use of retrieve-and-rank approaches @xcite . unlike previous question generation models @xcite @xcite that generate based on an answer, we treat question generation closer to structured planninglaying out the format for the entire brief. in contrast to most question answering datasets, the length of the answers in qabriefdataset are long-form @xcite . for example, the average answer in squad @xcite is four words long, while the average answer in qabriefdataset is forty. further, datasets such as squad, natural questions @xcite , and hotpotqa @xcite are built from wikipedia, while qabriefs uses the web. in this section, we describe qabriefer (see figure 5 ). for each claim, the question generation model is used to generate multiple questions. for each question, an evidence document is retrieved using a search engine. we take the top search hit as the evidence and retrieve the text from commoncrawl foot_3 . finally, the generated question and retrieved evidence document is provided to the question answering model to generate an answer. ## experimental setup our main question is whether briefs can increase the accuracy and efficiency of fact checking. we focus on human evaluation with both crowdworkers and volunteers fact checking claims. ## results we show in human evaluations that fact checking efficiency and accuracy are improved with briefs. ## related work previous work in nlp has focused on claim veracity. it has been treated as a classification problem (wang, 2017), often using stance detection @xcite . the fever challenge @xcite proposed providing provenance for a decision along with classification, and various approaches developed combine information retrieval with stance detection or question answering @xcite . question generation and answering has been considered in the context of fever (jobanputra, 2019) -the focus was on eliciting the right answer from a question answering system rather than improving the accuracy and efficiency of human fact checkers. however, fever is based on modified wikipedia sentences, not real world claims, which are arguably more difficult. to address this @xcite considered the claims fact checked by the website snopes, but used the reports accompanying them as evidence instead of finding the evidence directly. @xcite and @xcite used search engines, but without ensuring that they provide evidence supporting/refuting the claim instead of being related to it or that they were not fact checking reports. finally, @xcite used responses on social media for rumour verification, but did not address evidence finding. various work studies how to improve the fact checking process. analysis shows accuracy can improve by providing feedback @xcite , additional time @xcite , tooling @xcite , or training @xcite . these works are complementary to ours -we provide support in the form of briefs. studies emphasize that current solutions for fully automated fact checking face various challenges @xcite that must be addressed with interdisciplinary research @xcite . developing tools to aid human-in-the-loop fact checking has received increasing attention, from nlp to human-computer interaction and psychology, often with positive results when tested with journalists @xcite and professionals @xcite . ## discussion while our experiments show a generally positive impact of briefs for human fact checking, it is important to put them into a broader perspective. briefs for professional fact checkers crowdworkers and professional fact checkers perform different tasks under very different circumstances. professionals often investigate alternative interpretations and produce an explanation of their process in an article. they often have years of experience and must check a variety of claims. consequently, we do not claim that briefs will make a difference in their work. nevertheless, qabriefs can provide insights into the fact checking process. as the qabrief dataset was created using professional fact checking articles describing how a claim was checked, by decomposing a claim into multiple components, we can encourage a more structured fact checking process. biases introduced by briefs while briefs can increase accuracy, they can introduce biases. we found that providing a qabrief increased confidence -many submitted their fact check based on the brief alone, without the search bar. figure 8 (left) displays that around 45% of crowdworkers did not use the search bar when given the gold qabrief, even though accuracy without the search bar is reduced. briefs aid accuracy and efficiency, but are not fully sufficient to produce a verdict. ## conclusion we propose the concept of fact checking briefs, to be read before performing a fact check. crucially, we develop qabriefer and release the accompanying qabriefdataset, to create qabriefs. we show in extensive empirical studies with crowdworkers and volunteers that qabriefs can improve accuracy and efficiency of fact checking.