Diagnosing the First-Order Logical Reasoning Ability Through L ogic NLI (2021)

Abstract: Recently, language models (LMs) have achieved significant performance on many NLU tasks, which has spurred widespread interest for their possible applications in the scientific and social area. However, LMs have faced much criticism of whether they are truly capable of reasoning in NLU. In this work, we propose a diagnostic method for first-order logic (FOL) reasoning with a new proposed benchmark, LogicNLI. LogicNLI is an NLI-style dataset that effectively disentangles the target FOL reasoning from commonsense inference and can be used to diagnose LMs from four perspectives: accuracy, robustness, generalization, and interpretability. Experiments on BERT, RoBERTa, and XLNet, have uncovered the weaknesses of these LMs on FOL reasoning, which motivates future exploration to enhance the reasoning ability.

## introduction recently, transformers-based @xcite language models (lms), such as bert @xcite and roberta @xcite , have achieved great success on natural language understanding (nlu). however, there are growing concerns about whether lms can truly understand natural language or not. tasks with complex reasoning have provided evidence that lms lack expected reasoning abilities @xcite . even if neural models can make correct predictions, they tend to make decisions through spurious statistical correlations rather than reasoning abilities @xcite @xcite . therefore, an increasing number of studies have focused on diagnosing specific reasoning abilities of state-of-the-art lms @xcite . first-order logical (fol) reasoning is one of the most widely used reasoning forms in natural language @xcite , which has a simple paradigm consisting of combinations of seven fundamental logics (fols, including conjunction ∧, disjunction ∨, negation ¬, implication →, equation ≡, universal quantifier ∀, and existential quantifier ∃) with simple propositions @xcite . nevertheless, whether lms can truly make fol reasoning is still inconclusive in nlp @xcite . as a result, we propose a systematic diagnostic method for fol reasoning by proposing a novel benchmark, named logical natural language inference (logicnli). the proposed benchmark follows three principles: 1) it includes abundant logical expressions covering all seven fols and their commonly used combinations in texts; 2) the instances of the benchmark conform to natural language; 3) it introduces as little commonsense as possible to prevent the targeting fol reasoning and commonsense inference from being entangled with each other @xcite . according to the principles, logicnli is an nli-style dataset @xcite , including triplets of facts, rules, and a statement. the objective is to determine the logical relation @xcite ) between the premise (facts and rules) and its corresponding hypothesis (statement) by fol reasoning shown in figure 1 . in practice, we have introduced an additional logical relation, "paradox", to represent the situation where the hypothesis and its negative proposition can be logically entailed to the premise simultaneously based on different reasoning paths (bottom of figure 1 ). this novel logical relation forces the model to search at least two reasoning paths to infer the authenticity of two opposing propositions, thereby effectively avoiding spurious correlations caused by dataset bias. ## task definition in this section, we introduce how the task on log-icnli is defined. on the basis, we also exhibit how fol reasoning is embodied in logicnli. we first define elements in logicnli: its negative expression ¬s are independent conditioning on p (¬s ⊥ s|p ). the logical relations include "entailment", "contradiction", "neutral", and "paradox", whose conditions are shown in equation 1, where means syntactic consequence. logicnli includes more than 30k instances consisting of facts, rules, a statement to be judged, proofs, the reasoning path, and the label (shown in figure 2 ). for each instance, it requires a multihop fol reasoning process to reason out the final answer. to simplify the reasoning process, we set two limitations: 1) only considering the reasoning from cause to effect; 2) neglecting the true meanings of predicates. therefore, logicnli is more suitable for benchmarking the specific (fol) reasoning ability instead of serving as a comprehensive nlu task. as a result, we leave open the question of how lms perform in real reasoning scenarios with fols because it is difficult to disentangle multiple influencing factors. logicnli also provides four kinds of test sets that correspond to four diagnostic abilities in diagnosis, including total accuracy, robustness to irrelevant information, more-hop generalization, and proof-based traceability. specifically, we attempt to answer the following questions relevant to the fol reasoning ability based on these evaluations: q1: do models truly perform fol reasoning automatically in diverse scenarios? q2: do reasoning results accord with reasonable logic? accuracy, robustness, and generalization are adopted to answer q1 from different conditions. accuracy is the most common in-domain evaluation that measures the overall performance of lms. compared with accuracy, the robustness test offers a scenario that increases/decreases non-proof sentences. as robustness does not change the reasoning process, it can be regarded as an in-domain evaluation. the generalization test offers a scenario that increases the reasoning hop and therefore increases proofs, so it is an out-of-domain evaluation. traceability test is introduced to answer q2 by validating the whole reasoning process according to the proofs. ## conclusion in this paper, we propose a diagnostic method to diagnose lms' fol reasoning ability. this method introduces a novel proposed benchmark, logicnli, that disentangles the fol reasoning from commonsense inference. specifically, it includes four evaluations to measure the fol reasoning ability from different perspectives. results on three lms show that although some lms (roberta) own a certain interpretable fol reasoning ability, they still cannot make sensible fol reasoning like humans. detailed analysis motivates us to enhance specific reasoning abilities or explore new methods to make neural models understand more refined logic.