F act G enius: Combining Zero-Shot Prompting and Fuzzy Relation Mining to Improve Fact Verification with Knowledge Graphs (2024)

Abstract: Fact-checking is a crucial natural language processing (NLP) task that verifies the truthfulness of claims by considering reliable evidence. Traditional methods are labour- intensive, and most automatic approaches focus on using documents as evidence. In this paper, we focus on the relatively understudied fact-checking with Knowledge Graph data as evidence and experiment on the recently introduced FactKG benchmark. We present FactGenius, a novel method that enhances fact- checking by combining zero-shot prompting of large language models (LLMs) with fuzzy text matching on knowledge graphs (KGs). Our method employs LLMs for filtering relevant connections from the graph and validates these connections via distance-based matching. The evaluation of FactGenius on an existing benchmark demonstrates its effectiveness, as we show it significantly outperforms state-of- the-art methods. The code and materials are available at https://github.com/SushantGautam/FactGenius.

## introduction fact-checking is a critical task in natural language processing (nlp) that involves automatically verifying the truthfulness of a claim by considering evidence from reliable sources @xcite . this task is essential for combating misinformation and ensuring the integrity of information in digital communication @xcite . traditional fact-checking is performed by domain experts and is a labour-intensive process. automatic factchecking systems have been introduced to address this, but most of them work with textual data as evidence sources @xcite . recent advancements in large language models (llms) have shown promise in enhancing factchecking capabilities @xcite . llms, with their extensive pre-training on diverse textual data, possess a vast amount of embedded knowledge @xcite . however, their outputs can sometimes be erroneous or lacking in specificity, especially when dealing with complex reasoning patterns required for fact-checking. external knowledge, such as knowledge graphs (kgs) @xcite , can aid in factchecking. in this paper, we propose factgenius, a novel approach that combines zero-shot prompting of llms with fuzzy relation-mining techniques to improve reasoning on knowledge graphs. specifically, we leverage dbpedia @xcite , a structured source of linked data, to enhance the accuracy of fact-checking tasks. our methodology involves using the llm to filter potential connections between entities in the kg, followed by refining these connections through levenshtein distance-based fuzzy matching. this two-stage approach ensures that only valid and relevant connections are considered, thereby improving the accuracy of fact-checking. we evaluate our method using the factkg dataset @xcite , which comprises 108,000 claims constructed through various reasoning patterns applied to facts from dbpedia. our experiments demonstrate that factgenius significantly outperforms existing baselines @xcite , particularly when fine-tuning roberta @xcite as a classifier, achieving superior performance across different reasoning types. in summary, the integration of llms with kgs and the application of fuzzy matching techniques represent a promising direction for advancing factchecking methodologies. our work contributes to this growing body of research by proposing a novel approach that effectively combines these elements, yielding significant improvements in fact-checking performance. ## literature review fact-checking has become an increasingly vital aspect of natural language processing (nlp) due to the proliferation of misinformation in digital communication @xcite . traditional approaches to fact-checking have typically relied on manually curated datasets and rule-based methods. while these methods are effective in controlled environments, they often struggle with scalability and adaptability to new types of misinformation @xcite . the labor-intensive nature of these methods also poses significant challenges in rapidly evolving information landscapes @xcite . to address challenges in understanding machinereadable concepts in text, factkg introduces a new dataset for fact verification using claims, leveraging knowledge graphs (kgs) to encompass diverse reasoning types and linguistic patterns. this approach aims to enhance the reliability and practicality of kg-based fact verification @xcite . similarly, the fact extraction and verification (fever) dataset @xcite pairs claims with wikipedia sentences that support or refute them, providing a benchmark for fact-checking models. the authors employed a combination of natural language inference models and information retrieval systems to assess claim veracity. the gear framework @xcite ) improves fact verification by using a graph-based method to aggregate and reason over multiple pieces of evidence. this approach surpasses previous methods by enabling more interactive and effective use of evidence. recent advancements in large language models (llms) have demonstrated considerable potential for enhancing fact-checking processes @xcite . llms are pretrained on vast and diverse corpora @xcite , allowing them to generate human-like text and possess a broad knowledge base (choi and ferrara, 2024). however, despite their impressive capabilities, llms can sometimes produce erroneous outputs or lack the specificity required for complex fact-checking tasks @xcite . this issue becomes particularly evident when intricate reasoning and contextual understanding are necessary to verify claims accurately @xcite . several studies have explored the integration of llms with external knowledge sources to improve their performance in fact-checking tasks @xcite . the incorporation of knowledge graphs into factchecking frameworks has also garnered attention. kgs, such as dbpedia @xcite , provide structured and linked data that can enhance the contextual understanding of llms. knowledge graphs have been used to improve various nlp tasks by providing additional context and relationships between entities, as demonstrated by initiatives for knowledge-aware language models @xcite and kg-bert @xcite . approximate string matching, also called fuzzy string matching, is a technique used to identify partial matches between text strings @xcite . fuzzy matching techniques @xcite have been applied to enhance the integration of llms and kgs @xcite . the levenshtein distance-based similarity measure @xcite ) is one such technique that helps identify strings with approximate matches, which can be useful for finding relevant connections between entities by accommodating minor discrepancies in data representation. this approach has been beneficial in refining the outputs of llms, ensuring that only valid and contextually appropriate connections are considered @xcite . our proposed method, factgenius, builds on these advancements by combining zero-shot prompting of llms with a fuzzy relation-mining technique to improve reasoning over kgs. this methodology leverages dbpedia as a structured source of linked data to enhance fact-checking accuracy. by using llms to filter potential connections between entities and refining these connections through fuzzy matching, factgenius aims to address the limitations of existing factchecking models. ## preliminaries a knowledge graph (kg) g is a set of triples (s, r, o), with s, @xmath0, where e is the set of entities, and r is the set of relations connecting those entities. a kg can be viewed either as a set of triples or as a graph with nodes in e and edge labels in r. hence, when we discuss the 1-hop neighborhood of a certain entity e, we refer to the set of entities connected to e through an edge in this graph. for a triple (s, r, o), we consider s to be connected to o through an edge labeled as r, while we consider o to be connected to s through an edge labeled as ∼r, where ∼r denotes the inverse relation of r. we consider natural language sentences in their intuitive sense. given a claim in natural language c, a kg g with entities e, and a set of entities relevant to the claim e c , the fact verification with kg evidence task is to predict whether the claim c is supported or not according to the evidence in g. ## methodology we introduce the factgenius system for the fact verification with kg evidence task. our system has two main components: a graph filtering component that selects the relevant kg evidence for the input claim, and a classifier component that uses this evidence together with the claim to predict whether the claim is supported or not. factgenius leverages the capabilities of a large language model (llm) to filter the set of triples in the input graph g. more concretely, an llm is used in a zero-shot setting to select the relevant relations from the 1-hop neighborhood of the entities e c associated with claim c. since the output of llms can be erroneous, the triples are further validated against the unfiltered set using fuzzy matching techniques. finally, the classifier, which can be fine-tuned over pre-trained models like bert @xcite or roberta @xcite , or a zero-shot llm, determines whether the claim is supported or refuted. the overall pipeline is shown in figure 1 . ## experiments to evaluate the performance of our proposed methods, we conducted a series of experiments comparing different strategies for fact-checking on the factkg @xcite ) benchmark. ## discussion the enhanced performance of factgenius, particularly in conjunction, existence, and negation reasoning, can be attributed to its innovative combination of zero-shot prompting using large language models (llms) and fuzzy text matching on knowledge graphs. the evidence-based filtering approaches revealed significant findings. the stage-b validation approach improves accuracy compared to stage-a, although the model shows only moderate performance improvement in multi-hop reasoning. this suggests that more advanced techniques may be necessary to handle the complexity of multi-hop reasoning effectively. the two-step approach of filtering and validating connections proved to be especially effective. in the first step, the llm narrows down potential connections based on the context provided by the claim, significantly reducing the search space. the second step refines these connections through fuzzy matching, ensuring that only the most relevant and accurate ones are retained. our comparative study confirmed the importance of both steps, with the second step being particularly beneficial for conjunction and negation reasoning tasks. while fine-tuned llm models, such as bert and roberta, generally outperformed the zeroshot llama3-instruct model and claim-driven relation filtering, the increased graph complexity in dbpedia-full compared to dbpedia-light limited the gains from fine-tuning. this limitation can be attributed to the input token restrictions of bert and roberta, which truncate inputs after 512 tokens. truncation is more likely with the larger dbpedia-full graph, potentially excluding relevant information, thereby reducing the effectiveness of evidence-based filtering. additionally, the similar performance between stage-a and stage-b relation mining in the full graph setting suggests that the added complexity of stage-b does not yield better accuracy, likely due to these input constraints. these observations underscore the need for architectural adaptations or preprocessing methods to more effectively handle larger datasets. as llm inference is a crucial component of this framework, we employed vllm (vllm project, 2024) to enable rapid inference using a single nvidia a100 gpu. in our experiments, the llm inference speed was approximately 15 queries per second, including retries in case of failure. this rate is feasible, especially as llm inference continues to be optimized with the latest technologies. embedding llm in this framework has proven to be a sound decision. ## conclusion in this paper, we introduced factgenius, a novel method that combines zero-shot prompting of large language models with fuzzy relation mining to improve reasoning on knowledge graphs. this approach addresses several key challenges in traditional fact-checking methods. first, the integration of llms allows for the leveraging of extensive pre-trained knowledge in a zero-shot setting. second, the use of fuzzy text matching with levenshtein distance ensures that minor discrepancies in entity names or relationships do not hinder the relationship selection process, thus improving robustness. our experiments on the factkg dataset demonstrated that factgenius significantly outperforms traditional fact-checking methods and existing baselines, particularly when fine-tuning roberta as a classifier. the two-stage approach of filtering and validating connections was crucial for achieving high accuracy across various reasoning types. the findings from this study suggest that utilizing llms for kg evidence retrieval holds great promise for advancing fact-checking capabilities. future work could explore applying this approach to other domains and datasets, as well as incorporating additional structured data sources to further enhance performance.