Improve Rule Retrieval and Reasoning with Self-Induction and Relevance R e E stimate (2025)

Abstract: This paper systematically addresses the challenge of rule retrieval, a crucial yet underexplored area. Vanilla retrieval methods using sparse or dense retrievers to directly search for relevant rules to support downstream reasoning, often suffer from low accuracy. This is primarily due to a significant semantic gap between the instantiated facts in the queries and the abstract representations of the rules. Such misalignment results in suboptimal retrieval quality, which in turn negatively impacts reasoning performance. To overcome these challenges, we propose Self-Induction Augmented Retrieval (SIAR), a novel approach that utilizes Large Language Models (LLMs) to induce potential inferential rules that might offer benefits for reasoning by abstracting the underlying knowledge and logical structure in queries. These induced rules are then used for query augmentation to improve retrieval effectiveness. Additionally, we introduce Rule Relevance ReEstimate (R ^3 ), a method that re-estimates the relevance of retrieved rules by assessing whether the abstract knowledge they contain can be instantiated to align with the facts in the queries and the helpfulness for reasoning. Extensive experiments across various settings demonstrate the effectiveness and versatility of our proposed methods.

## introduction with the advancement of pre-training @xcite and prompting techniques @xcite , large language models (llms) @xcite @xcite have made significant progress in their understanding, reasoning, and decision-making capabilities @xcite . rules can generate new knowledge from existing information (or make decisions based on observed situations), which can * kang is the corresponding author. enhance these abilities further @xcite . the de facto approach of rule-based reasoning typically involves summarizing generalized rules from past experiences by llms, then retrieving the relevant rules based on the descriptions of downstream tasks or feedback from the observed environment, and finally using the retrieved rules to assist in reasoning or decision-making @xcite @xcite @xcite . unfortunately, existing research has primarily focused on rule generation @xcite and application @xcite , neglecting the development of the rule retrieval techniques. furthermore, rule retrieval plays a crucial role in real-world scenarios. for example, in legal scenar-ios, one must retrieve the relevant laws based on the crime to make a judgment @xcite , and in medical settings, domain-specific rules must be retrieved based on symptoms to assist in diagnosis @xcite . the above highlights the urgency of exploring rule retrieval area. the rules discussed in this paper are referential rules @xcite , which typically manifest as the derivation of one set of facts from another. in natural language, they are usually expressed in the form "if premise, then conclusion," whereas in formal language, they are often represented as "premise ⇒ conclusion." as illustrated in figure 1 (a) , the query and the corresponding golden passage typically share some keywords or similar semantics in traditional knowledge retrieval scenarios @xcite , making direct matching between the query and passage feasible. however, in rule retrieval scenarios, the following characteristics of the query and golden rule present significant challenges: (1) the facts in the query are instantiated and specific. (2) the facts contained in the rule are composed of variables and predicates, where the variables typically have an abstract, conceptual type, and the predicates represent relationships between different variables, which might not be expressed in the query explicitly. (3) the gap between the query (concrete, implicit) and the rule (abstract, explicit) leads to a semantic misalignment. (4) there is an explicit derivation in the rule, but not in the query. for example, as shown in figure 1 (b) , "environmental law in california mandates recycling" is the fact in the query. in the golden rule, the entity "california" corresponds to "region z", "environmental law" corresponds to "legislation y", and this fact implies "legislation y applies to region z". furthermore, the rule incorporates the inferred conclusion "person x needs to obey legislation y". existing methods typically overlook the aforementioned characteristics and apply traditional retrieval techniques directly during the rule retrieval phase. as demonstrated in figure 1 (c ), whether using sparse retrieval or dense retrieval, relying on vanilla retrieval to assist in reasoning often results in varying degrees of performance degradation compared to reasoning without rules. this demonstrates the inadequacy of traditional retrieval methods in rule-based scenarios. in fact, if the retrieved rules are irrelevant or contain noise, the reasoning will be distracted. as the size of the rule base increases, rule retrieval will become the bottleneck for downstream task performance. in contrast, when the golden rule is directly provided to aid the llm in its reasoning process, performance enhancement of 31.54% and 23.67% can be witnessed in the 7b and 72b models respectively. the performance gap highlights the importance of rules in supporting reasoning and the necessity of accurate rule retrieval. to this end, this paper proposes self-induction augmented retrieval (siar). siar leverages self-induction to summarize and abstract the facts presented in the query and hypothesize potential inferential relationships to generate a potential rule. this newly generated rule is then used as the new query, or combined with the original query to form a new query for retrieval. specifically, we utilize few-shot in-context learning to prompt the llm to produce a self-induced rule. our theoretical insight is as follows: if we consider the query set and the rule set as belonging to different semantic subspaces, where the former is characterized by instantiated, concrete facts and the latter by abstract, conceptual knowledge. we hypothesize that these two subspaces are nearly non-overlapping. the role of self-induction is to project the query as much as possible into the rule subspace, enabling the query to better match rules that share similar underlying logic during retrieval. although siar can improve the ranking of the golden label in the retrieved rule list, the limited inductive capabilities of llms still make it challenging to handle more difficult queries. moreover, the retriever can only evaluate the semantic similarity instead of the helpfulness of the rule for the query. therefore, building on siar, we propose rule relevance reestimate (r 3 ), which utilizes the llms to estimate the relevance of the retrieved rule list. r 3 evaluates whether each rule can be applied to the current query for better reasoning and reranks the list based on the relevance estimation. we conduct experiments on two synthetic datasets as well as one real-world dataset. compared to direct retrieval, siar achieves significant improvements in both retrieval and reasoning performance, demonstrating its effectiveness in extracting and summarizing the knowledge and logic embedded in queries to assist retrieval. building on siar, r 3 further enhances both retrieval and reasoning performance, proving that llms can reliably assess the relevance between queries and rules, thereby improving the quality of rule retrieval. moreover, siar and r 3 consistently improve performance across different settings, including varying rule formats (natural and formal language), different types of retrievers (sparse and dense retrievers), and llms of different parameter scales. this demonstrates the generalizability of our proposed methods. the contributions of this paper are as follows: ## preliminary in the rule reasoning scenario, we have a rule library @xmath0, which incorporates inferential rules offering benefits for new knowledge induction and decision-making. we use the query q to retrieve relevant rules from this library, and the retrieved rules are concatenated with the query as context. this combined input is then fed into the llm to perform reasoning. we name this workflow as retrieve-then-reason. in the retrieval phase, we employ either sparse retrieval or dense retrieval. the former uses bm25 to compute the similarity between the query and the rules, while the latter leverages a pre-trained encoder to map both the query and the rules into a shared vector space, where their similarity is computed using the cosine function. the top-k rules are then returned based on the similarity ranking. to improve retrieval speed, we pre-build and cache the index of r. ## method in the aforementioned retrieve-then-reason paradigm, our method primarily focuses on im-proving retrieval quality by aligning the semantics between the query and the rule and re-assessing the relevance of retrieved rules. the former technique is inserted before the retrieval stage and the latter one is inserted between retrieval and reasoning. furthermore, we aim to positively impact the overall reasoning performance. our method is based on prompting the llms without any training, which is versatile and cost-efficient. ## experiment we select two synthetic datasets, clutrr @xcite and ulogic @xcite , as well as a real-world dataset, cail2018 @xcite from rulebench for our evaluation. we report recall@1, recall@5, recall@10 for retrieval results and match for reasoning results. we use gpt-4o (openai, 2024) and qwen2.5 @xcite series (7b and 72b) as the tested llms. due to space limitation, we put the entire experiment setting in appendix b. ## ablation study we perform more ablation experiments to explore more influencing factors. we use the ulogic dataset and test six methods: vanilla prompt, vanilla prompt + r 3 , siar (w/ si), siar (w/ si + input), siar-r 3 (w/ si), and siar-r 3 (w/ si + input). among them, "vanilla prompt + r 3 " refers to retrieving using the original query and then directly performing r 3 . due to space limitations, this method was not presented in the previous section. moreover, we put the result table of section 5.2 and section 5.3 in appendix e. ## conclusion this paper introduces self-induction augmented retrieval (siar) and rule relevance re-estimate (r 3 ) to address the challenges of rule retrieval in complex reasoning tasks. these techniques significantly enhance retrieval accuracy by llms to induce abstract inferential rules and assess the relevance of retrieved rules to queries. siar and r3 offer promising solutions for overcoming the semantic misalignment issues in traditional retrieval techniques, paving the way for more effective rulebased reasoning in real-world applications.