QAC heck: A Demonstration System for Question-Guided Multi-Hop Fact-Checking (2023)

Abstract: Fact-checking real-world claims often requires intricate, multi-step reasoning due to the absence of direct evidence to support or refute them. However, existing fact-checking systems often lack transparency in their decision-making, making it challenging for users to comprehend their reasoning process. To address this, we propose the Question-guided Multi-hop Fact-Checking (QACheck) system, which guides the model’s reasoning process by asking a series of questions critical for verifying a claim. QACheck has five key modules: a claim verifier, a question generator, a question-answering module, a QA validator, and a reasoner. Users can input a claim into QACheck, which then predicts its veracity and provides a comprehensive report detailing its reasoning process, guided by a sequence of (question, answer) pairs. QACheck also provides the source of evidence supporting each question, fostering a transparent, explainable, and user-friendly fact-checking process.

## introduction in our age characterized by large amounts of both true and false information, fact-checking is not only crucial for counteracting misinformation but also plays a vital role in fostering trust in ai systems. however, the process of validating real-world claims is rarely straightforward. unlike the simplicity of supporting or refuting a claim with a single piece of direct evidence, real-world claims often resemble multi-layered puzzles that require complex and multi-step reasoning to solve @xcite @xcite @xcite . as an example, to verify the claim "sunlight can reach the deepest part of the black sea.", it may be challenging to find direct evidence on the web that 1 qacheck is public available at https://github.com/ xinyuanlu00/qacheck . a recorded video is at https://www. youtube.com/watch?@xmath0: black sea has a maximum depth of 2,212 meters. a2: sunlight does not penetrate water below 1,000 meters. q1: what is the greatest depth of the black sea? q2: how far can sunlight penetrate water? claim: sunlight can travel to the deepest part of the black sea. we introduce the question-guided multi-hop fact-checking (qacheck) system, which addresses the aforementioned issues by generating multi-step explanations via question-guided reasoning. to facilitate an explainable reasoning process, qacheck manages the reasoning process by guiding the model to self-generate a series of questions vital for claim verification. our system, as depicted in figure 2 , is composed of five modules: 1) a claim verifier that assesses whether sufficient information has been gathered to verify the claim, 2) a question generator to generate the next relevant question, 3) a question-answering module to answer the raised question, 4) a qa validator to evaluate the usefulness of the generated (q, a) pair, and 5) a reasoner to output the final veracity label based on all collected contexts. qacheck offers enough adaptability, allowing users to customize the design of each module by integrating with different models. for example, we provide three alternative implementations for the qa component: the retriever-reader model, the flan-t5 model, and the gpt3-based reciterreader model. furthermore, we offer a user-friendly interface for users to fact-check any input claim and visualize its detailed question-guided reasoning process. the screenshot of our user interface is shown in figure 4 . we will discuss the implementation details of the system modules in section 3 and some evaluation results in section 4. finally, we present the details of the user interface in section 5. and conclude and discuss future work in section 6. ## related work fact-checking systems. the recent surge in automated fact-checking research aims to mitigate the spread of misinformation. various factchecking systems, for example, tanbih foot_0 @xcite , prta foot_1 (martino et al., 2020), and whatthewikifact foot_2 @xcite predominantly originating from wikipedia and claims within political or scientific domains, have facilitated this endeavor. however, the majority of these systems limit the validation or refutation of a claim to a single document, indicating a gap in systems for multi-step reasoning @xcite . the system most similar to ours is quin+ @xcite , which demonstrates evidence retrieval in a single step. in contrast, our qacheck shows a question-led multistep reasoning process with explanations and retrieved evidence for each reasoning step. in summary, our system 1) supports fact-checking realworld claims that require multi-step reasoning, and 2) enhances transparency and helps users have a clear understanding of the reasoning process. explanation generation. simply predicting a veracity label to the claim is not persuasive, and can even enhance mistaken beliefs @xcite . hence, it is necessary for automated fact-checking methods to provide explanations to support model predictions. traditional approaches have utilized attention weights, logic, or summary generation to provide post-hoc explanations for model predictions @xcite @xcite @xcite . in contrast, our approach employs question-answer pair based explanations, offering more human-like and natural explanations. ## system architecture figure 2 shows the general architecture of our system, comprised of five principal modules: a claim verifier d, a question generator q, a question-answering model a, a validator v, and a reasoner r. we first initialize an empty context @xmath1. upon the receipt of a new input claim c, the system first utilizes the claim verifier to determine the sufficiency of the existing context to validate the claim, i.e., d(c, c) → {true, false}. if the output is false, the question generator learns to generate the next question that is necessary for verifying the claim, i.e., q(c, c) → q. the questionanswering model is then applied to answer the question and provide the supported evidence, i.e., a(q) → a, e, where a is the predicted answer, and e is the retrieved evidence that supports the answer. afterward, the validator is used to validate the usefulness of the newly-generated (q, a) pair based on the existing context and the claim, i.e., v(c, {q, a}, c) → {true, false}. if the output is true, the (q, a) pair is added into the context c. otherwise, the question generator is asked to generate another question. we repeat this process of calling @xmath2. in this case, the reasoner module is called to utilize the stored relevant context to justify the veracity of the claim and outputs the final label, i.e., r(c, c) → {supported, refuted}. the subsequent sections provide a comprehensive description of the five key modules in qacheck. ## performance evaluation to evaluate the performance of our qacheck, we use two fact-checking datasets that contain complex claims requiring multi-step reasoning: hover @xcite and feverous @xcite , following the same experimental setthe bold text shows the best results for each setting. the evaluation results are shown in ## user interface we create a demo system based on flask foot_4 for verifying open-domain claims with qacheck, as shown in figure 4 . the qacheck demo is designed to be intuitive and user-friendly, enabling users to input any claim or select from a list of pre-defined claims (top half of figure 4 ). upon selecting or inputting a claim, the user can start the fact-checking process by clicking the "submit" button. the bottom half of figure 4 shows a snapshot of qacheck's output for the input claim "lars onsager won the nobel prize when he was 30 years old". the system visualizes the detailed question-guided reasoning process. for each reasoning step, the system shows the index of the reasoning step, the generated question, and the predicted answer to the question. the retrieved evidence to support the answer is shown on the right for each step. the system then shows the final veracity prediction for the original claim accompanied by a comprehensive rationale in the "prediction with rationale" section. this step-by-step illustration not only enhances the understanding of our system's fact-checking process but also offers transparency to its functioning. qacheck also allows users to change the underlying question-answering model. as shown at the top of figure 4 , users can select between the three different qa models introduced in section 3.3, depending on their specific requirements or preferences. our demo system will be opensourced under the apache-2.0 license. ## conclusion and future works this paper presents the qacheck system, a novel approach designed for verifying real-world complex claims. qacheck conducts the reasoning process with the guidance of asking and answering a series of questions and answers. specifically, qacheck iteratively generates contextually relevant questions, retrieves and validates answers, judges the sufficiency of the context information, and finally, reasons out the claim's truth value based on the accumulated knowledge. qacheck leverages a wide range of techniques, such as incontext learning, document retrieval, and questionanswering, to ensure a precise, transparent, explainable, and user-friendly fact-checking process. in the future, we plan to enhance qacheck 1) by integrating additional knowledge bases to further improve the breadth and depth of information accessible to the system @xcite , and 2) by incorporating a multimodal interface to support image @xcite , table @xcite , and chart-based fact-checking @xcite , which can broaden the system's utility in processing and analyzing different forms of data.