Zero-Shot Fact Verification via Natural Logic and Large Language Models (2024)

Abstract: The recent development of fact verification systems with natural logic has enhanced their explainability by aligning claims with evidence through set-theoretic operators, providing faithful justifications. Despite these advancements, such systems often rely on a large amount of training data annotated with natural logic. To address this issue, we propose a zero-shot method that utilizes the generalization capabilities of instruction-tuned large language models. To comprehensively assess the zero-shot capabilities of our method and other fact verification systems, we evaluate all models on both artificial and real-world claims, including multilingual datasets. We also compare our method against other fact verification systems in two setups. First, in the zero-shot generalization setup, we demonstrate that our approach outperforms other systems that were not specifically trained on natural logic data, achieving an average accuracy improvement of 8.96 points over the best-performing baseline. Second, in the zero-shot transfer setup, we show that current systems trained on natural logic data do not generalize well to other domains, and our method outperforms these systems across all datasets with real-world claims.

## introduction in the context of fact-checking, fact verification (fv) is a process of verifying whether a textual hypothesis holds, based on retrieved evidence. while many improvements have been made in this field due to the recent rapid growth in nlp @xcite @xcite , fv systems often employ pipelines with black-box components that hide the underlying reasoning. one line of research attempts to improve explainability with attention-based methods @xcite and post-hoc summarizations @xcite . however, these approaches do not provide faithful justifications -explanations that accurately reflect the model's decision-making process and the data it used @xcite . in contrast, systems such as naturalli @xcite and proofver @xcite provide faithful justifications by expressing semantic relations between claim/evidence pairs. modeling these logical relations and their aggregation explicitly with natural logic (natlog) allows for the accurate processing of phenomena such as double-negation and has resulted in more accurate and robust fact-checking systems. however, a limitation of natural logic-based fv systems is that they require large amounts of training data annotated with entire natural logic proofs. for example, proofver @xcite was trained on 145k instances artificially obtained from structured knowledge bases such as ppdb @xcite and wikidata @xcite . while recent work @xcite attempts to alleviate this issue by proposing a few-shot learning method trained on as few as 32 instances, human annotation of even a small number of proofs can be impractical and expensive, as it requires substantial linguistic knowledge and familiarity with natural logic. moreover, few-shot systems require additional training data in order to generalize effectively to new domains, further increasing the costs. to this end, we propose zero-natver foot_0 , a zeroshot fact verification approach for constructing natural logic proofs that leverages prompting and question-answering with instruction-tuned large language models (llms). zero-natver's proof generation process is illustrated in figure 1 . first, a claim is chunked into smaller units of information. then, the units are aligned to relevant parts of the evidence, and natural-logic operators are assigned refuted oliver twist, a novel by dickens, was published before the pickwick papers. the pickwick papers, published in 1836, is a novel by charles dickens that follows the humorous adventures and misadventures of samuel pickwick and his companions in the pickwick club. oliver twist, published in 1837, is a novel by charles dickens that tells the story of an orphan boy who... to each claim-evidence pair. lastly, the proofs are executed on a finite state automaton (dfa) as defined in natural logic inference, producing the verdict. ## related work natural logic @xcite and naturalli (angeli and manning, 2014), composes full inference proofs that operate directly on natural language, capable of expressing more complex logical relationships between claim and evidence, such as double-negation. @xcite trained natural logic inference systems for fact verification, achieving competitive performance while remaining faithful and more explainable than its entirely neural counterpart. while these neural-symbolic approaches require substantial training data to perform well, @xcite explored natural logic inference in a few-shot setting by casting natural logic operators into a question-answering framework, subsequently making use of the generalization capabilities of instruction-tuned language models. although our work also considers question-answering, we further expand on this approach, addressing prediction calibration issues frequently encountered in a zero-shot setting @xcite . other neuro-symbolic reasoning systems for fv use simple logical rules to aggregate veracity information on a claim's components to provide simple faithful explanations @xcite @xcite . however, these rules lack the expressiveness of natural logic and thus cannot inherently model more complex phenomena such as double negation. previous work on zero-shot fv is limited and largely relies on the generation of weakly supervised training samples and on knowledge of the target domain @xcite . @xcite observe that typical fv systems fail when transferred to unseen domains in a zeroshot setting and propose a data augmentation technique to improve generalizability. moreover, none of the aforementioned zero-shot methods produces (faithful) explanations. in a few-shot setting, several recent works have explored the use of large language models that produce explanations alongside the verdict. @xcite define a reasoning program consisting of a sequence of subtasks to verify complex claims. @xcite propose chain-of-thought prompting complemented by action operations to support the model's reasoning and its explanation generation. @xcite propose to edit rationales generated via chain-ofthought prompting by querying knowledge sources. unlike our work, these approaches still rely on in-context examples. in the first two steps, zero-natver segments c into several chunks (sec. 3.1) and aligns each such chunk with relevant information from e (sec. 3.2). this process results in a sequence of l claimevidence alignment pairs @xmath0, a 2 , ..., a l . as part of this alignment process, we also generate alignment explanations that are parsed for supporting/refuting signals. these signals are used in the third stage of the pipeline where zero-natver determines semantic relations of aligned pairs in terms of natural logic. thus, it generates a sequence of natural logic operators @xmath1, o 2 , ..., o l , which correspond to alignment pairs in a (sec. 3.3). fi-nally, o is used in the last stage to traverse a deterministic finite automaton (dfa), which determines the claim's veracity. the following sections describe each step in more detail. ## conclusion we have presented zero-natver, a zero-shot fact verification method grounded in natural logic. our method leverages the generalization capabilities of instruction-tuned llms and generates faithful justifications for proofs without relying on training data annotated with natural logic. we have evaluated zero-natver in two zero-shot setups, outperforming our baselines on most datasets. the ablation study shows the importance of individual design choices, and our comparison with the direct non-natlog approach shows that natural logic provides competitive performance while providing explainability via faithful justifications. we hope that the methods and analyses presented here enable further progress toward improving the efficiency and explainability of fact verification systems.