Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals (2024)

Abstract: Despite recent success in natural language processing (NLP), fact verification still remains a difficult task. Due to misinformation spreading increasingly fast, attention has been directed towards automatically verifying the correctness of claims. In the domain of NLP, this is usually done by training supervised machine learning models to verify claims by utilizing evidence from trustworthy corpora. We present efficient methods for verifying claims on a dataset where the evidence is in the form of structured knowledge graphs. We use the FactKG dataset, which is constructed from the DBpedia knowledge graph extracted from Wikipedia. By simplifying the evidence retrieval process, from fine-tuned language models to simple logical retrievals, we are able to construct models that both require less computational resources and achieve better test-set accuracy.

## introduction as the volume of information generated continues to grow, so does the risk of misinformation spreading, which has made automatic fact verification a crucial task in nlp @xcite @xcite . traditionally, fact verification has been tackled in journalism by experts manually researching topics and writing articles about their findings. some specific websites dedicated to this approach are factcheck.org and politifact.com. however, it is time-consuming and labor-intensive, and is not able to follow the pace of information created in digital media @xcite . one of the most popular datasets for fact verification is the fact extraction and verification (fever) dataset @xcite . it consists of claims supported by a corpus of wikipedia articles. models trained on the dataset need to extract the relevant evidence and use it to classify claims as supported, refuted or not enough information. despite its popularity, several issues have been discovered. due to the manual construction of claims, the structure of the language is inherently biased with respect to the classes, and therefore it is possible to achieve good performance without using the evidence at all @xcite . it has also been shown that models trained on fever experience a significant drop in performance when the factual evidence is changed in a way that influences the validity of claims @xcite . these issues can be improved by accordingly adjusting the validation and test dataset to contain less biased data @xcite , but we believe it is important to develop models on other datasets as well. a less studied approach to process evidence is by structured data. in many real-world examples, data is available in large structured databases, rather than unstructured articles. this is relevant for domains such as social networks, logistics, management systems and database systems. the dataset tabfact @xcite was created with this intent, consisting of claims with tabular evidence extracted from wikipedia. this paper aims to increase the performance of models trained on the factkg dataset @xcite , a dataset created for fact verification with structured evidence in the form of knowledge graphs (kgs). the claims are created with evidence from dbpedia @xcite , a large kg extracted from wikipedia. a kg consists of nodes and edges linked together to represent structural concepts. nodes represent entities, such as persons, things or events, and edges represent relations, conveying how entities are related, as shown in figure 1 . for instance, a node can be the company meyer werft, and since it is located in the city papenburg, they are connected with the edge location. we refer to meyer werft, location, papenburg as a knowledge triple. since the task of fact verification with kgs re-figure 1 : an example claim from factkg @xcite . the claim can be verified or refuted based on the dbpedia kg @xcite . this is figure 1 from @xcite . the textual finetuning serves as a simple and conventional method, while the qa-gnn can handle graph based data efficiently and is more specifically constructed for the task of interest. in contrast, the llm prompting displays how well general purpose language models can perform on the task. it does not require any further training and does not use any evidence. therefore, it will serve as a baseline and give insight to how difficult the task is. our main contribution is that we increase the accuracy and computational efficiency of models trained on factkg. by utilizing efficient subgraph retrieval methods, we are able to increase the testset accuracy from 77.65% @xcite to 93.49%. to the best of the authors' knowledge, this is the best performance achieved so far on this dataset. additionally, our models train quicker, taking only 1.5-10 hours, compared to the 2-3 days spent on the benchmark model from @xcite , reported by the authors. the code and documentation used for this article can be found at https://github.com/tobias-opsahl/ fact-or-fiction . ## discussion we were able to train better and more efficient models by simplifying the subgraph retrieval methods, both by using a fine-tuned bert and a slightly modified qa-gnn model. while the qa-gnn models trained the fastest, the fine-tuned bert clearly performed the best, significantly outperforming the benchmark models. this suggests that the simple logical subgraph retrievals worked better than the complex trained approaches in previous work. we suggest that the performance gain in the claim-only benchmark was due to slightly better hyperparameters. all of the models performed better the bigger the subgraphs were. this suggests that the model architectures are able to utilize the relevant parts of the subgraphs, without needing an advanced subgraph retrieval step. this is emphasized by our fine-tuned bert model achieving a 93.49% test set accuracy by only using the single-step subgraphs, while the gear model from @xcite , which trained two language models to perform graph retrieval, achieved a 77.65% test-set accuracy. when examining the precision and recall metrics in a limitation of our subgraph retrieval methods is that they never include nodes that are more than one step away from an entity node, while the trained approach from @xcite is dynamic and may include more. this might make the hypothesis that the simple subgraph retrieval methods will perform worse on multi-hop claims than the dynamically trained, however, we see the exact opposite behavior. the best bert and qa-gnn models score 80.32% and 74.72% at the multihop claims respectively, while the dynamic gear model scores 68.84%, even lower than the models not using the subgraphs at all. we do however see that the best performing bert model clearly performs the worst on the multi-hop claims compared to the other claim types, indicating that even bigger subgraphs might be beneficial. while the sample size for the chatgpt metrics were small, it does indicate that non-fine-tuned llms can achieve adequate few-shot performance compared to a fine-tuned claim-only bert. the performance does not seem to be substantially compromised when the amount of questions prompted increases, so with a bigger access to computational resources, it might be possible to prompt the full test-set at once. the removal of fine-tuning greatly improves the ease of use if one only needs to verify a few claims. while we are hesitant to make any conclusion with the small sample size, we believe that the results serve as an approximate benchmark of how difficult the dataset is. ## conclusion and future work our results show that with simple, yet efficient methods for subgraph retrieval, our models were able to improve fact verification with knowledge graphs with respect to both performance and efficiency. the fine-tuned bert model with singlestep subgraphs clearly achieves the best performance, while the qa-gnn models are more efficient to train. this indicates that complex models can work well with simple subgraph retrieval methods. since the single-step subgraphs mostly contain information not relevant for the claims, the models are themselves able to filter away irrelevant information, and complex subgraph retrieval methods may hence not be necessary for accurate fact verification. additionally, since the best performing model performed the poorest with multi-hop claims, future research could explore simple subgraphs retrieval methods allowing for bigger depths than one. additionally, future work should also be directed towards running similar experiments on other datasets. we also encourage researchers that have access to bigger computational resources to further explore the performance of llms for fact verification. a core limitation of our chatgpt prompting was the inability to use the full test-set, and we consider this crucial for further development. we also think it would be especially interesting to make llm and kg hybrid models. since our results indicate that simple single-step subgraph retrievals outperform more complex methods, a promising path of future research would be to simply use both the claims and the single-step subgraphs as input to the llm. if possible, the llm could also be fine-tuned on task: determine the truth value (true or false) of the following claims based on information verifiable from wikipedia, as represented in the dbpedia knowledge graph. provide your answers without using real-time internet searches or code analysis, relying solely on your pre-trained knowledge. instructions: output format: format your responses as a list in python. each entry should be a tuple, formatted as (claim number, answer, explanation). example claims: 1. the atatürk monument is located in izmir, turkey, where the capital is ankara. 2. yes, eamonn butler's alma mater is the university of texas system! 3. i have heard 300 north lasalle was completed in 2009. 4. the band clinton gregory created an album in the rock style. ... example output: [ (1, true, "the atatürk monument is indeed located in izmir, and the capital of turkey is ankara."), (2, false, "eamonn butler did not attend the university of texas system; he is a british author and economist whose educational background does not include this institution."), (3, true, "300 north lasalle in chicago was indeed completed in 2009."), (4, false, "clinton gregory is primarily known as a country music artist, not rock."), ... ] here are the actual claims you should answer: the dataset. we also encourage future work to create fully reproducible results with llms, which we were unable to do.