Neutralizing Bias in LLM Reasoning using Entailment Graphs (2025)

Abstract: LLMs are often claimed to be capable of Natural Language Inference (NLI), which is widely regarded as a cornerstone of more complex forms of reasoning. However, recent works show that LLMs still suffer from hallucinations in NLI due to attestation bias , where LLMs overly rely on propositional memory to build shortcuts. To solve the issue, we design an unsupervised framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias. To measure bias reduction, we build bias-adversarial variants of NLI datasets with randomly replaced predicates in premises while keeping hypotheses unchanged. Extensive evaluations show that our framework can significantly reduce hallucinations from attestation bias. Then, we further evaluate LLMs fine-tuned with our framework on original NLI datasets and their bias-neutralized versions, where original entities are replaced with randomly sampled ones. Extensive results show that our framework consistently improves inferential performance on both original and bias-neutralized NLI datasets.

## introduction natural language inference (nli) has long been recognized as a foundational understanding task in language understanding with various downstream applications @xcite @xcite . it assesses the understanding ability of models by requiring them to determine if a given premise logically entails a hypothesis. recently, with the rise of llms, the field of nli has witnessed significant advancements @xcite @xcite . these models, pre-trained on vast amounts of text data, have been claimed to capture inferential relations between statements enabling reasoning, positioning them as state-of-the-art tools for nli. ## experimental setup to evaluate the effectiveness of our eg-enhanced models, we conduct two experiments: (1) examining whether eg-enhanced models reduce attestation bias in §5, and (2) assessing their inferential performance in nli benchmarks in §6. both evaluation experiments follow the same settings, including the evaluation datasets and baselines, as detailed in the following sections 3 . ## experiment 1: attestation bias reduction to evaluate the effectiveness of our method in reducing attestation bias, we measure attestation bias by comparing estimated probabilities of predicting entail conditioned on whether the hypothesis is predicted attested or not. however, in original nli dataset entailments may coincidentally refer to attested facts, which could lead to spurious correlation between inference and attestation scores, making it difficult to ## experiment 2: evaluate capability of inference eg-enhanced llms exhibit a notable reduction in attestation bias. to further investigate how this attestation bias reduction affects their capability of inference, we evaluate their general inferential performance on original nli tasks. however, attestation biases and reported data leakage @xcite @xcite often lead to an overestimation of llms performance on original nli dataset, where llms rely on memorization to form reasoning shortcuts rather than genuinely reasoning, making it challenging to accurately evaluating their true inference capabilities. to address these challenges, we introduce a method to generate bias-neutralized datasets from original nli datasets for evaluation. we neutralize biases by replacing the original entities with others of the same type, generating new inference data that includes counterfactual statements while preserving original entailment labels. we then evaluate our eg-enhanced llms on both original and biasneutralized nli test sets, enabling a more accurate assessment of their true inferential capabilities. ## errors analysis as shown in ## conclusion in nli tasks, attestation bias in llms leads to false positives, as models rely on memorization rather than reasoning from the given premise, ultimately undermining their robustness and accuracy in inference. to address this, we propose an unsupervised method for constructing logically consistent counterfactual egs to fine-tune llms, enhancing their robustness against attestation bias. experimental results show that our method reduces attestation bias and enhances inference by learning counterfactual egs, enabling models to learn predicate entailment while without introducing artifacts, thereby minimizing dependency on memorized patterns. our method improves robustness and effectiveness of models in practical applications while providing a more objective evaluation of inference capability across llms.