QA - N at V er: Question Answering for Natural Logic-based Fact Verification (2023)

Abstract: Fact verification systems assess a claim’s veracity based on evidence. An important consideration in designing them is faithfulness, i.e. generating explanations that accurately reflect the reasoning of the model. Recent works have focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim with its evidence via set-theoretic operators. However, these approaches rely on substantial resources for training, which are only available for high-resource languages. To this end, we propose to use question answering to predict natural logic operators, taking advantage of the generalization capabilities of instruction-tuned language models. Thus, we obviate the need for annotated training data while still relying on a deterministic inference system. In a few-shot setting on FEVER, our approach outperforms the best baseline by 4.3 accuracy points, including a state-of-the-art pre-trained seq2seq natural logic system, as well as a state-of-the-art prompt-based classifier. Our system demonstrates its robustness and portability, achieving competitive performance on a counterfactual dataset and surpassing all approaches without further annotation on a Danish verification dataset. A human evaluation indicates that our approach produces more plausible proofs with fewer erroneous natural logic operators than previous natural logic-based systems.

## introduction automated fact verification is concerned with the task of identifying whether a factual statement is true, with the goal of improving digital literacy @xcite . a typical fact verification system consists of an evidence retrieval and a claim verification component (i.e. the judgement of whether a claim is true). the latter is typically implemented as a neural entailment system @xcite , which is not transparent in regards to its underlying reasoning. while efforts have been made to improve their explainability, for instance via highlighting salient parts of the evidence @xcite , or generating summaries @xcite , there is no guarantee that the explanations are faithful, i.e. that they accurately reflect the reasoning of the model @xcite . contrarily, proof systems like naturalli @xcite , perform natural logic inference as proofs, and are faithful by design. their transparent reasoning empowers actors to make informed decisions about whether to trust the model and which parts of the prediction to dispute @xcite . @xcite constructed a natural logic theorem prover for claim verification, using an autoregressive formulation with constrained decoding. however, an important limitation of this approach is its dependence on substantial resources for training, relying on large datasets for claim verification, and structured knowledge bases like the para-phrase database @xcite , word-net @xcite , and wikidata @xcite . however such manually curated resources are typically accessible for high-resource languages, thus limiting its applicability. to this end, we propose qa-natver: question answering for natural logic-based fact verification, a natural logic inference system that composes a proof by casting natural logic into a question answering framework. as illustrated in figure 1 , a proof is a sequence of steps, with each step describing the semantic relation between a claim span and an evidence span via a set-theoretic natural logic operator (natop), and this sequence of natops determines the veracity of the claim following a deterministic finite state automaton (dfa). qa-natver predicts natops using operator-specific boolean questions (cf. s rice born in new jersey r rice born in new orleans the current veracity state and mutation operator determine the transition to the next state, via a fine state automaton (dfa). starting at s , the span anne rice is mutated via the equivalence operation (≡), resulting in s , according to the dfa. the inference ends in r , indicating the claim's refutation. we use question-answering to predict the natops, taking advantage of the generalization capabilities of instruction-tuned language models. was born and the evidence born is ascribed the equivalence natop (≡), which we predict with questions such as is "was born" a paraphrase of "born"?. this formulation enables us to make use of instruction-finetuned language models, which are powerful learners, even with limited supervision @xcite . since the input format to our question-answering formulation constrains the context to the aligned claim-evidence spans, we generate claim-evidence alignments between overlapping spans of varying length, and individually predict the natop for each pair of aligned spans. to select the best proof over all possible proofs, we combine the answer scores to the questions associated with each proof. in a few shot setting with 32 training instances on fever, qa-natver outperforms all baselines by 4.3 accuracy points, including proofver, loren @xcite , and a state-of-the-art few-shot classification system, t-few @xcite . by scaling the instruction-tuning model from bart0 to flan-t5, we achieve a score of 70.3 ± 2.1, closing the gap to fully supervised models, trained on over 140,000 instances, to 8.2 points. on an adversarial claim verification dataset, symmetric fever @xcite , qa-natver scores higher than few-shot and fully supervised baselines, including proofver, demonstrating the robustness of our approach. in a low-resource sce-nario on a danish fact verification dataset (nørregaard and derczynski, 2021, danfever) without further annotation for training, our system outperforms all baselines by 1.8 accuracy points, highlighting the potential of the question-answering formulation for low-resource languages. an ablation study indicates the benefits of qa-natver's question-answering formulation, outperforming chatgpt (openai, 2022) (over 430x the size of bart0) prompted with in-context examples to predict natops by 11.9 accuracy points. finally, we show in a human evaluation that qa-natver improves over previous natural logic inference systems in terms of explainability, producing more plausible proofs with fewer erroneous natops. 1 2 related work natural logic @xcite operates directly on natural language, making it an appealing alternative to explicit meaning representations such as lambda calculus, since the translation of claims and evidence into such representations is error-prone and difficult to decode for nonexperts. natlog @xcite proposes the use of natural logic for textual inference, which has then subsequently been extended by @xcite into the nat-figure 2 : qa-natver's proof construction. we first chunk the claim and the evidence, and align them at multiple granularity levels. we then assign a natural logic operator to each aligned claim-evidence span using questionanswering. finally, we select the proof by combining the answer scores to the questions associated with the proof. uralli proof system. with the surge of pre-trained language models, multiple works have attempted to integrate natural logic into neuro-symbolic reasoning systems @xcite . in particular, proofver, a natural logic inference system specifically designed for fact verification, achieves competitive performance yet remains faithful and more explainable than its entirely neural approaches @xcite . @xcite propose an alternative framework of logical reasoning, evaluating the veracity of individual claim spans (or of atomic facts in @xcite ) and determining the overall truthfulness by following a simple list of logical rules. @xcite use a similar list of logical rules but aggregate the outcomes with a neural network component. however, all previous approaches have in common that they require substantial training data to perform well, limiting their use to resource-rich languages and domains. casting a natural language problem to a questionanswering setting has previously been explored in a variety of tasks such as relation classification @xcite , and semantic role labeling @xcite . for fact verification, in particular, previous works have considered formulating it as a question generation task, decomposing a claim into relevant units of information to inquire about, followed by question answering to find relevant answers in a large knowledge base @xcite ). yet, these works do not consider aggregating nor constructing proofs from the answers to the questions. finally, work on few-shot claim verification is limited. @xcite explore using a perplexity score, however, their approach is constrained to binary entailment, i.e. either supported or refuted. @xcite explore active learning in combination with pet @xcite , a popular prompt-based few-shot learning method, and @xcite and @xcite generate weakly supervised training data for zero-shot claim verification. however, none of the aforementioned methods produces (faithful) explanations. ## method given a claim c and a set of k evidence sentences e, the task of claim verification is to predict a veracity label ŷ and to generate a justification for the selected label. qa-natver is a system that returns a natural logic inference proof which consists of a sequence of relations @xmath0, . . . , m l , each of them specifying a relation between a claim and evidence span and a natop operator o. 2 the sequence of operators @xmath1, . . . , o l is then the input to a deterministic finite state automaton that specifies the veracity label ŷ = dfa(o) (c.f. figure 1 ). the proof p itself serves as the justification for the predicted label ŷ. qa-natver constructs its proofs following a three-step pipeline, illustrated in figure 2 : multigranular chunking of the claim and its evidence sentences and the alignment between claim-evidence spans (sec. 3.1), assignment of natops to each aligned pair using question answering (sec. 3.2), and a proof selection mechanism over all possible proofs by combining the answer probabilities to the questions associated with the proof (sec. 3.3). ## conclusion this paper presented qa-natver, a natural logic inference system for few-shot fact verification that frames natural logic operator prediction as question-answering. we show that our approach outperforms all baselines while remaining faithful. human evaluation shows that qa-natver produces more plausible proofs with fewer erroneous natural logic operators than the state-of-the-art natural logic inference system, while being trained on less than a thousandth of the data, highlighting qa-natver's generalization ability. future work looks at extending the capability of natural logic inference systems to more complex types of reasoning, including arithmetic computations.