Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs (2024)

Abstract: Despite progress in automated fact-checking, most systems require a significant amount of labeled training data, which is expensive. In this paper, we propose a novel zero-shot method, which instead of operating directly on the claim and evidence sentences, decomposes them into semantic triples augmented using external knowledge graphs, and uses large language models trained for natural language inference. This allows it to generalize to adversarial datasets and domains that supervised models require specific training data for. Our empirical results show that our approach outperforms previous zero-shot approaches on FEVER, FEVER-Symmetric, FEVER 2.0, and Climate-FEVER, while being comparable or better than supervised models on the adversarial and the out-of-domain datasets.

## introduction fact-checking is the task of assessing the truthfulness of a claim, and is well-studied across multiple disciplines. traditionally, journalists perform such a task manually, which is time-consuming. more recently, automated fact-checking systems have become of interest due to the explosion of (mis)information on social media @xcite . in the nlp community, fact-checking is typically defined as a task consisting of three stages: claim detection, evidence retrieval, and claim verification @xcite . in particular, verdict prediction assumes the evidence is retrieved from sources such as wikipedia or the web, and aims to predict the verdict of a claim given the retrieved evidence, often as a three-way classification task @xcite : supports, refutes, and nei (not enough info). recent work (dehaven and scott, 2023) has achieved strong results on canonical datasets like fever @xcite , mostly relying on supervised approaches. however, concerns have been expressed on whether these models learn language's and the task's nuances or merely leverage embedded biases and dataset idiosyncrasies. this argument @xcite gains empirical weight when such high-performing models are tested against adversarial fact-checking datasets such as fever-symmetric @xcite and fever 2.0 @xcite . their underperformance @xcite in these adversarial benchmarks exposes a lack of model robustness. the narrative of this vulnerability extends to outof-domain contexts as well. a pertinent example is the climate-fever dataset-a platform for verifying real-world climate claims @xcite . supervised models, despite their commendable performance on the original fever dataset, suffer performance degradation when evaluated on climate-fever. additionally, earlier zero-shot fact-checking approaches @xcite hinge on synthetic data creation for training purposes. while this data emanates from factual evidence, it largely adheres to the domain boundaries of the originating dataset. such inherent domain confinement curtails the model's capacity for broader generalization. in this work we propose a zero-shot method utilizing semantic triples and knowledge graphs in conjunction with pretrained natural language inference (nli) models, and does not require training data for parameter learning. in particular, we propose to extract triples from the claim and the evidence texts to form knowledge graphs and fill potential gaps in the evidence using a universal schema model @xcite on wikidata and wikipedia. crucially, our method refrains from utilizing any annotated or synthetic training data, sidestepping the pitfalls of biases and dataset artifacts that can inadvertently be encoded into models. additionally, by decomposing the original claim into triples, our method can harness the pre-trained nli model's strengths more effectively. both design choices position our approach to exhibit greater robustness when subjected to adversarial and out-of-domain evaluations. as shown in figure 1 , we follow a two-stage verification process: triple-level and claim-level. for triple-level, we employ nli models pre-trained without further fine-tuning on any fact-checking training dataset, hence a zero-shot setting. for claim-level verification, we design a simple rulebased system relying on the triple verification. in figure 1 , the process involves extracting claim and evidence sentences to generate triples. subsequently, the universal schema is applied to obtain additional triples. the nli model is then employed to assign triple-level labels, resulting in 2 supports, 1 refutes, and 1 nei foot_0 . finally, a rule-based system is utilized to derive claim-level verification. in this example, since one claim triple is refuted, the entire claim is considered refuted. note that we are able to use the "gap" triples filled by the universal schema model to retrieve better evidence. for example, <manning, member_of, @xmath0. however, such a triple is missing from the evidence triple extraction because the word member is not mentioned in the evidence. instead, <manning, professor_of, @xmath1. therefore, with the universal schema model, <manning, mem-ber_of, @xmath2, prof_of, @xmath3, and the gap is filled. we evaluate our approach on the fever @xcite , fever-symmetric @xcite , fever 2.0 @xcite , and climate-fever @xcite datasets. our findings show that our system consistently outperforms zero-shot nli model baselines by a margin of approximately 2.5 percentage points and beats the previous zero-shot approach by around 3 percentage points on fever-symmetric. notably, in contrast to state-of-the-art supervised methods (dehaven and scott, 2023), our approach exhibits robustness on both adversarial datasets. when evaluated on the out-of-domain climate-fever dataset, our method outperforms the supervised method by a margin exceeding 10 percentage points. ## related work recent advances in natural language processing have highlighted significant challenges associated with supervised learning models. a prominent concern is the models' tendency to learn datasetspecific biases, often at the expense of genuine linguistic understanding. for instance, @xcite demonstrated the effectiveness of a claim-only model that classifies each claim in isolation, without the need for associated evidence. the high performance achieved by their system over the baseline can be attributed to the idiosyncrasies inherent in the dataset's construction. similarly, @xcite highlighted the vulnerability of several fever systems, observing significant performance declines under adversarial conditions with simple rule-based perturbations. in other tasks such as nli, previous works @xcite examined the susceptibility of neural models to such spurious correlations, revealing a troubling propensity for models to exploit unintended, data-specific heuristics. taken together, these findings suggest that annotation artifacts within datasets contain discernible patterns. such vulnerabilities underscore the necessity for more rigorous evaluation mechanisms, thus motivating the introduction of several adversarial factchecking evaluation datasets @xcite . @xcite presented the first work to investigate zero-shot fact verification, where they proposed a framework named question answering for claim generation (qacg). from any given evidence, qacg generates supports, refutes, and nei claims. a classifier is then trained using the generated claims instead of annotated claims, hence a zero-shot setting. to generate claims, qacg first produces qa pairs using a question generator fine-tuned on the processed squad dataset @xcite . next, a qa-to-claim model is fine-tuned on the qa2d dataset @xcite , which converts each qa pair into a declarative sentence. however, their experiments are limited, using only the gold evidence to evaluate various zero-shot methods, which is not practical in a real-world setting. also, unlike their work, where training is still performed using the generated training data, our approach does not require any training for claim verification. knowledge graphs have long been investigated in nlp, where the first discussions of a graphical knowledge representation can date back to the 50s @xcite . since then, many nlp researchers have tried to integrate knowledge graphs into various nlp tasks, notably language models with knowledge graphs @xcite @xcite and many downstream tasks such as question answering @xcite ) and text classification @xcite . for fact-checking specifically, @xcite proposed to use knowledge graphs to verify simple natural language claims, considering fact-checking as a special case of link prediction. their method uses the subject and object of the claim and then finds the shortest path between the two entities. if the claim is true, there should be such a shortest path (or an edge); otherwise, there should be no shortest path (nor edge). while the fact that a simple ## methodology as introduced, the verdict prediction step of claim verification is to predict a label @xmath4, refutes, nei} given a claim c and its corresponding evidence e, indicating if c is supported, refuted, or cannot be verified by e. while we do not use any training data (manually or automatically labeled), we assume a human-annotated development set is available for fine-tuning hyperparameters of our system. in keeping with prior research, we use the same set of notations and extend it to include triples. ## results and discussion our main results are presented in finding 1 our zero-shot approach exhibited enhanced robustness against adversarial perturbations and manifested notable out-of-domain effectiveness in contrast to supervised approaches. as shown in by abstaining from using training data, our approach intuitively circumvents these issues and offers a more robust approach to fact-checking. specifically, despite exhibiting lower performance than supervised systems on the original fever dataset, our models achieved highly competitive scores on the fever-symmetric dataset, trailing the state-of-the-art by only approximately 2 percentage points. we attribute this positive outcome to our system's utilization of nli models, which already demonstrate outstanding performance on this adversarial dataset. the results obtained on the fever 2.0 dataset align with fever-symmetric and further strengthen our conclusions. on climate-fever, the supervised approach delineated by @xcite achieved an accuracy of 38.78% and an f1 score of 32.85%. in comparison, our introduced zero-shot methodology showcased enhanced results, achieving an accuracy rate of 46.71% and an f1 score of 45.71%, which demonstrated a notable generalization ability. these findings suggest that our zero-shot method offers a promising avenue for improved performance in out-of-domain tasks. finding 2 our system, utilizing triple-level inference, consistently improves over the baseline results irrespective of the nli model used. ## conclusion we introduced a novel zero-shot fact-checking method, translating claims and evidence into semantic triples with external knowledge graphs. this method surpasses other zero-shot baselines, impressively without direct fever dataset training. its resilience is evident, avoiding the typical performance dips seen in supervised models on adversarial datasets like fever-symmetric and fever 2.0. also on the climate-fever dataset, our approach outshines even supervised counterparts, highlighting its generalization prowess. augmented by pretrained nli models, our system's robustness is further emphasized. as future steps, we aim to hone model interpretability, examine diverse knowledge graphs, and test our method's versatility on other fact-checking datasets.