Explicit Inductive Inference using Large Language Models (2024)

Abstract: Large Language Models (LLMs) are reported to hold undesirable attestation bias on inference tasks: when asked to predict if a premise P entails a hypothesis H , instead of considering H ‘s conditional truthfulness entailed by P , LLMs tend to use the out-of-context truth label of H as a fragile proxy. In this paper, we propose a pipeline that exploits this bias to do explicit inductive inference. Our pipeline uses an LLM to transform a premise into a set of attested alternatives, and then aggregate answers of the derived new entailment inquiries to support the original inference prediction. On a directional predicate entailment benchmark, we demonstrate that by applying this simple pipeline, we can improve the overall performance of LLMs on inference and substantially alleviate the impact of their attestation bias.

## introduction large language models (llms) are claimed to possess implicit inductive reasoning ability through pre-training: from the massive examples they memorized, they draw inference rules and encode them latently so that they can apply these rules to do reasoning at test time. however, recently @xcite has pointed out that llms are severely affected by an attestation bias when performing inference tasks. given the question of whether premise p entails hypothesis h with few-shot examples, an llm's prediction is deeply bound to the hypothesis' outof-context truthfulness, instead of its conditional truthfulness entailed by the premise. when the hypothesis h is attested in an llm's world knowledge (the llm believes h to be true), the llm is likely to predict the entailment to be true, regardless of the premise. as a result, llms suffer a significant performance drop when the entailment labels disagree with the attestation label of hypothesis h. although this is a severe problem limiting llms' performance on non-attested inferences, we argue that with careful design, this bias can instead be exploited to improve llm performance on inference tasks. we notice a statistically true conclusion: given an entailment inquiry p |= h, the attestation bias is harmful only when the premise p is not attested. if we control p to always be attested, then p |= h will naturally share the same truth label with h on a distributional basis, which dissolves the negative effects of llms' attestation bias. applying this idea, we propose a simple yet effective explicit inductive inference pipeline with llms. as illustrated in figure 1 , the core idea is to transform a premise into a set of attested alternatives by replacing the arguments, and to aggregate the llm's predictions on these derived inquiries to support answering the original question. we test our pipeline with two llms on levy/holt (levy and dagan, 2016; holt, 2019), a difficult directional predicate inference dataset, and further analyze the influence of our pipeline against the models' attestation bias. the results show that our pipeline can improve not only llm's overall performance on predicate inference, but also their robustness against the attestation bias. to summarize our contribution, we propose an easy-to-use inference pipeline that 1) improves llms' performance on predicate inference, 2) substantially alleviates negative effects of the llms' attestation bias, and 3) uses llms' own generation capability without requiring external knowledge. ## related work llms accumulate a bias towards factual knowledge by encoding massive facts during pre-training @xcite @xcite . @xcite pointed out that llms suffer from an attestation bias on inference tasks as a result. note that the effect of attestation bias is similar to that of the hypothesisonly baseline @xcite , but while the former is a bias from pre-training, the latter originates from dataset artifacts in supervised learning. in other tasks, previous work has mitigated the bias towards attestation by introducing counterfactual examples @xcite @xcite or replacing argument entities with their type labels @xcite . in this paper, we go one step further to show that in an inference task, we should instead encourage the models to generate factual alternative examples. the idea of aggregating multiple versions of llms' outputs has been explored in prior work. @xcite encourage llms to generate multiple reasoning paths for one question, while @xcite embody one question with multiple prompts. in contrast, our method creates semantically different alternative questions, which serve as extra evidence for an original inquiry. ## conclusions we propose an explicit inductive pipeline exploiting the attestation bias of llms to do more robust predicate inference. with experiments on the directional levy/holt dataset and its attestation-biassplit subsets, we have shown that our baseline gains a significant improvement over llm's primary inference performance, and substantially reduces the performance loss caused by llms' attestation bias. without external knowledge, eidi use llms' own generation to exploit their attestation bias. our results suggest that although biases of llms are usually undesirable obstacles, in some scenarios they may be tapped for good with careful design. we advocate for similar ideas to be applied to other tasks to improve llm performance in future work.