GP roof T : A Multi-dimension Multi-round Fact Checking Framework Based on Claim Fact Extraction (2024)

Abstract: In the information era, the vast proliferation of online content poses significant challenges, particularly concerning the trustworthiness of these digital statements, which can have profound societal implications. Although it is possible to manually annotate and verify the authenticity of such content, the sheer volume and rapid pace of information generation render this approach impractical, both in terms of time and cost. Therefore, it is imperative to develop automated systems capable of validating online claims, ensuring that users can use the wealth of information available on the Internet effectively and reliably. Using primarily ChatGPT and the Google search API, GProofT fact checking framework generates question-answer pairs to systematically extract and verify the facts within claims. Based on the outcomes of these QA pairs, claims are subsequently labeled as Supported, Conflicted Evidence/Cherry-Picking, or Refuted. Shown by extensive experiments, GProofT Retrieval generally performs effectively in fact-checking and makes a substantial contribution to the task. Our code is released on https://github.com/HKUST-KnowComp/GProofT.

## introduction with the chaotic nature of information on the internet, it appears to be challenging to determine the credibility of claims on the web. this poses difficulties on large language models (llms) such as @xcite to conduct fact checking as the hallucination @xcite @xcite of them can produce seemingly feasible but fake information. though time-consuming and tedious when performed manually, fact-checking is rather crucial to ensure the trustworthiness of information, especially for the fact-sensitive industry such as journalism and science. in the explosion of information, it's far from * first three authors make equal contribution to this paper. adequate to solely rely on manual check to eliminate the rumor and misinformation, while remain difficult to be detected simply with commonsense knowledge @xcite @xcite . therefore, it's pivotal to develop a trustworthy automatic process to complete factchecking efficiently and accurately. recent advancements in llms have showcased remarkable performance in tasks involving text comprehension and generation @xcite @xcite . however, the application of llms in automatic fact-checking has remained a persistent challenge, undergoing continuous exploration and development @xcite . current llms can only memorize the knowledge embedded in their pretrain data, which makes them struggle with fact-checking when the event is out of their pretrain corpus, namely, out of their knowledge domain. under this circumstance, it is necessary and crucial to incorporate real-time online search engine to provide llms with real-time facts information to assist its reasoning. however, the chaotic nature of internet could imply that the knowledge provided from the search engine could result in misinformation to the llms, hindering its reasoning process. hence, a consistent framework for multi-dimension, multi-round fact checking needs to be proposed to generate stable and trustworthy fact checking result. to solve the limitation, we propose gprooft fact checking framework to crawl and analyze web evidence based on google search api and chatgpt. as demonstrated in figure 1 , for each given textual claim, we incorporate three stages to retrieve the pertinent evidence from the internet and a final step to attribute a label based on the retrieved evidence. as suggested in the shared task, our retrieved evidence is in the format of qa pair. the retrieval procedure includes claim split, question generation, answer generation and expansion. more information could be found in section 3.1. overall, claim: the united states of america and its western allies have been using their media outlets to publish articles based on fabricated information under allegations of noncompliance with the chemical weapons convention. q: what are the chemical weapons and their destruction (chemical weapons convention or cwc) evidentia: a new report from the international atomic energy agency (iaea) shows that iran has not been complying with its nuclear deal with world powers, des? a: chemical weapons and on their destruction (chemical weapons convention or cwc) q: what are the chemical weapons and their destruction (chemical weapons convention or cwc) evidentia: a new report from the international atomic energy agency (iaea) shows that iran has not been complying with its nuclear deal with world powers, des? a: chemical weapons and on their destruction (chemical weapons convention or cwc) q: what are the chemical weapons and their destruction (chemical weapons convention or cwc) evidentia: a new report from the international atomic energy agency (iaea) shows that iran has not been complying with its nuclear deal with world powers, des? a: chemical weapons and on their destruction (chemical weapons convention or cwc) averitec irrelevant questions meaningless answer q: have the united states of america and its western allies been using their media outlets to publish articles based on fabricated information? a: no, because the text does not provide evidence of fabricated information being published by the united states of america and its western allies. q: did allegations of noncompliance with the chemical weapons convention occur? a: yes, because the text contains mentions of investigations regarding possible non-compliance with the chemical weapons convention. ## system overview in this section, we would introduce the gprooft fact checking framework and elaborate our benchmark setup. ## experiments in this section, we will elaborate the data processing flow and the evaluation setting we adopted in the experiments. ## analysis in this section, we conduct error analysis and case study to further investigate the strengths and potential drawbacks of our framework. furthermore, a imbalance prediction analysis is attached in appendix a to serve as a analysis on prediction distribution of our framework. ## conclusion in this paper, we introduced gprooft, a multidimension, multi-round fact-checking framework designed to improve the efficacy and accuracy of validating online claims by leveraging llms and web evidence retrieval. through extensive experi-ments, our approach demonstrated superior performance compared to baseline models, particularly in the critical task of evidence retrieval. moreover, our framework requires less human labor involved in evidence checking which means it could be easily scale up when there is a huge amount of fact checking workload, improving the efficiency. apart from such advantages, our framework also encounter challenges such as duplicated subclaims and biased claim splits, indicating areas for further improvement. furthermore, refining the claim decomposition process and enhancing the handling of conflicting evidence will be crucial steps in advancing automated fact-checking systems. our work contributes to the ongoing efforts to develop reliable, scalable, and automated tools for ensuring the trustworthiness of online information. as demonstrated in