IDEA : Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abduction (2025)

Abstract: While large language models (LLMs) have been thoroughly evaluated for deductive and inductive reasoning, their proficiency in holistic rule learning in interactive environments remains less explored. We introduce RULEARN, a novel benchmark to assess the rule-learning abilities of LLM agents in interactive settings. In RULEARN, agents strategically interact with simulated environments to gather observations, discern patterns, and solve complex problems. To enhance the rule-learning capabilities for LLM agents, we propose IDEA, a novel reasoning framework that integrates the process of **I**nduction, **De**duction, and **A**bduction. The IDEA agent generates initial hypotheses from limited observations through abduction, devises plans to validate these hypotheses or leverages them to solve problems via deduction, and refines previous hypotheses through induction, dynamically establishing and applying rules that mimic human rule-learning behaviors. Our evaluation of the IDEA framework, which involves five representative LLMs, demonstrates significant improvements over the baseline. Furthermore, our study with human participants reveals notable discrepancies in rule-learning behaviors between humans and LLMs. We believe our benchmark will serve as a valuable and challenging resource, and IDEA will provide crucial insights for the development of LLM agents capable of human-like rule learning in real-world scenarios. Our code and data have been released at: https://github.com/KaiyuHe998/RULEARN_IDEA.

## introduction one major pillar of human intelligence is the ability to discern and apply rules. we identify patterns, form hypotheses, and iteratively refine them through interaction with the environment-a process that traditionally involves three stages: abduction, deduction, and induction. according to charles peirce @xcite , rule learning begins with an explanatory hypothesis (abduction), followed by iterative experiments (deduction) and hypothesis refinement (induction) (see figure 1 ). this interdependent process underpins human rule learning in the real world, yet recent studies often isolate these stages in noninteractive settings @xcite @xcite . to simulate the full complexity of human rule learning, three essential principles must be met: an interactive environment that encourages dynamic experimentation, a fine-grained action space that enables flexible and precise inputs for nuanced hypothesis testing, and the presence of unknown rules that force agents to infer, test, and revise hypotheses based solely on sparse observations. by integrating these three key principles, we introduce rulearn, which features 300 high-quality, manually created puzzles with hidden rules set in a text-based environment, where agents begin exploration without any prior knowledge of the underlying rules. rulearn simulates human-like rule learning-compelling agents to experiment dynamically, utilize fine-graind actions for detailed hypothesis testing, and infer rules from limited data. successfully solving the puzzles requires the agent to strategically select actions, efficiently gather pattern-revealing observations, and accurately reason from them to infer the hidden rules. figure 2 : a simplified puzzle in the rulearn benchmark and the idea agent's workflow (in real puzzles, agents have fewer initial observations and more complex rules). the agent generates an initial hypothesis through abduction, develops an exploration plan via deduction, and refines its hypothesis using induction. for example, the idea agent first hypothesizes that the password is the number of the blue paintings, tests this by entering code 003, and adjusts its strategy based on the feedback. rulearn presents substantial challenges, as agents must rely on observations generated by their chosen actions to discern rules; without clear patterns emerging from their exploration, the agents are likely to fail. rulearn consists of three types of environments to evaluate the rule-learning ability in different scenarios: @xcite the function operator: determining the coefficients of mathematical functions defined by hidden expressions. agents can assign various values to the input variables and observe the outputs, using this information to hypothesize the function's form. the challenge lies in efficiently selecting input values that reveal the underlying structure and accurately computing the coefficients based on limited observations. (2) the escape room: deciphering the passcode to exit an escape room. a hidden rule determines how the objects in the room infer the passcode digits. agents interact with these objects to gather clues and input passcodes into the door. based on feedback, agents formulate hypotheses and infer the relationship between the objects and the passcode. (3) the reactor: synthesizing target strings using a reactor with a hidden string-combining rule. agents need to experiment with different inputs and analyze outputs to deduce the reactor's transformation rule and achieve the desired outcome. to tackle the challenge in rulearn, we introduce idea, a novel reasoning framework that integrates the process of induction, deduction, and abduction. the idea agent employs these rea-soning processes iteratively to explore the environments, learn rules, and achieve goals. in the abduction phase, the idea agent generates an initial hypothesis from limited observations. during the deduction phase, the idea agent creates and executes plans to attempt objectives or test its hypothesis. in the induction phase, the idea agent refines its hypothesis based on new observations, enhancing their accuracy and robustness. this iterative cycle enables the llm agent to continually improve the learned rules through environmental feedback. an overview of how the idea agent solves puzzles in rulearn is shown in figure 2 . we evaluate idea on five popular llms-gpt-3.5-turbo, gpt-4o, gemma-7b, llama3-8b, and llama3-70b-observing roughly a 10% improvement in success rates compared to the baseline. without hypothesis guidance, the baseline agent tends to choose direct, uninformed actions that fail to uncover the hidden rules. in contrast, the idea agent reduces repeated actions by 30.2%, obtains more diverse observations, and better understands the underlying rules. to further investigate their rule-learning capabilities, we compare llm performance with that of 50 human participants. although idea narrows the gap, llms still face challenges: (1) inefficient exploration in unfamiliar environments, resulting in insufficient evidence to reveal rules; (2) difficulty in deducing valid plans to verify current hypotheses and guide future exploration; and (3) reluctance to correct initial hypotheses when confronted with contradictory observations. these findings provide important insights into improving llm agents to achieve more efficient rule learning in complex scenarios. ## related works agents powered by large language models (llms) have shown notable progress in understanding complex tasks @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . recent work examines different reasoning processes (abduction, deduction, induction) in llms @xcite @xcite @xcite , but typically in isolation. as a result, their comprehensive rule-learning abilities in interactive settings remain underexplored. current reasoning tasks are hindered by inadequate benchmarks that either rely on qa datasets like hotpot-qa @xcite and trivia-qa @xcite )-which lack active information gathering-or by coarse-grained interactive environments such as textworld @xcite and @xcite that limit agents to high-level actions (e.g., go to, open), impeding complex, experiment-driven rule discovery. this is in stark contrast to real-world rule learning, which requires active evidence gathering, experimentation, and iterative refinement. moreover, many studies use static, non-interactive scenarios where llms receive all information upfront @xcite @xcite , failing to capture the dynamic nature of real-world learning. even recent efforts @xcite that integrate interactivity treat information gathering, rule generation, and application as distinct phases, undermining the development of agents capable of seamlessly integrating these elements. we claim that to fully capture rule-learning ability in the real world, three criteria should be met: (i) interactive environment: the environment must be interactive so agents learn from interaction rather than passively receiving data; (ii) finegraind action space: the action space must be fine-grained, since existing benchmarks offer only coarse actions (e.g., go to, open) that prevent agents from performing the detailed experiments needed to test hypotheses @xcite ; and (iii) unknown-rule: the target rules must be unknown, as in scienceworld @xcite most test cases rely on knowledge llms have already mastered during pretraining. to address this gap, our proposed rulearn fulfills all three requirements by providing a fully interactive environment, offering a fine-grained action space in which agents submit arbitrary strings that our system parses to deliver character-level feedback, and introducing new, manually crafted puzzles whose rules are not familiar to llms. in turn, idea equips agents to manage the interdependent processes of information gathering, hypothesis generation, and validation within a unified framework that closely mimics human rule-learning behavior. ## the rulearn benchmark we develop three puzzle sets-function operator, escape room, and reactor-each consisting of 100 unique, manually created puzzles of varying complexity, with each set reflects a different realworld rule-learning scenario. unlike existing finegrained interactive environments, which are predominantly found in the robotics domain and offer significantly fewer tasks @xcite @xcite , rulearn is the first text-based environment providing such fine-grained interactions specifically for language agents. the function operator. this puzzle type simulates scenarios where systemic theories or established knowledge (e.g., mathematics) are applicable for efficiently testing hypotheses. the agent interacts with a set of univariate multi-term equations involving integer parameters from [0,9] and elementary functions of the variable x, selected from f (x) ∈ {x 0 , x 1 , x 2 , sin(x), 1 x , |x|, -x}. the agent is provided with the number of functions, the presence of specific parameters in each function (the exact numerical values of these parameters are unknown and represented by letters), and the types of elementary functions involved in the current puzzle. the goal of the agent is to deduce the values of these parameters. for example, in one puzzle, the ground truth is f 1 (x) = a sin(x) + b × 1 identical symbols represent the same letter, while different symbols denote different letters. each puzzle operates under one specific rule. the middle insertion rule inserts the shorter string into the longer string; if the length of the longer string is odd, the shorter string is inserted just to the right of the center. if both strings are of equal length, the second string is inserted into the middle of the first string. the prefix replacement rule retains the prefix of the longer string and concatenates it with the shorter string, dropping the tail of the longer string results in two output strings. there are two special cases where the strings are simply concatenated in order. ## the idea agent we introduce idea, a novel reasoning framework that integrates the process of induction, deduction, and abduction to learn rules in interactive environments. the idea agent consists of the following components: goal(g), action space(a), memory(m), hypothesis(h), and plan(p), which are elaborated in upon beginning to explore a puzzle, we initialize the agent memory with an initial observation of the environment. the agent's goal is initialized with the objective of the puzzle, e.g., synthesize a target string for a reactor puzzle. the agent's action space is initialized as the set of interactive actions defined by the puzzle, such as choosing two strings and running the reactor, as well as establish the initial hypothesis (abductive action), devises a plan to validate or leverage hypothesis (deductive action), and refining the current hypothesis (inductive action). the idea agent begins with an abductive action to generate an initial hypothesis, followed by a deduction step to create a new plan. based on this plan, the agent interacts with the environment. upon receiving feedback from the environment as a new observation, the agent may take an inductive action to refine the hypothesis or perform another interaction with the environment. deductive action is invoked to adjust the plan every time the hypothesis changes. this reasoning loop continues until the puzzle is solved or a maximum number of steps is reached. after each step, the results are appended to the agent's memory, including interaction outcomes and any modifications to the hypothesis or plan. we provide a simplified algorithm demonstrating how the idea agent operates in algorithm 1. specifically, at each step, we prompt the llm to reflect on the information recorded in the idea agent's components to make decisions and take actions. detailed prompts for each type of action are available in appendix a.5. more detailed implementation of the agent can be found in appendix a.3.1. similar to real-life scenarios, when agents solve tasks in rulearn puzzles, they do not know the outcomes in advance. consequently, it is challenging to decide when to refine or change their hypothesis and plans, as well as what interactive actions to take to gather patternrevealing observations. a detailed example of the idea agent solving the reactor puzzle is provided in figure 3 . m.add("new hypothesis and plan", h, p) 8: if a is interactive action then 11: result ← execute_action(a, g, h, p, m) 12: m.add(result) 13: m.add("refined hypothesis and plan", h, p) 18: goal of the agent in the current puzzle. action space(a) set of actions the agent can take, including abductive action, deductive action, inductive action, as well as the set of interactive actions defined by the puzzle. memory(m) set of natural language strings to record all interaction results till the current step. plan(p) generated plans to guide future actions. ## fine-grained human evaluation to compare llm agent rule-learning with human processes, we conducted a fine-grained evaluation of rule-learning trajectories at every reasoning step. computer science graduate students assessed the hypotheses and plans generated by both the idea agent and human participants during the abduction, deduction, and induction stages on a randomly selected 50% subsample of puzzles. abduction stage. in this stage, agents formulate an initial hypothesis based on early observations-sometimes even guessing the ground truth rule in simpler puzzles. figure 5(a) indicates that llms such as gpt-4o correctly identify the rule during abduction about 30% of the time, while only around 10% of human participants generate a correct hypothesis under initial uncertainty. this discrepancy suggests that deviations in rule learning between llms and humans emerge as early as the abduction stage. llms tend to process every word of the prompt and produce a hypothesis even when unsure, whereas humans generally refrain from formalizing hypotheses under uncertainty. deduction stage. after establishing an initial hypothesis-or each time the agent refines its hypothesis-the agent derives a plan to validate it or attempt the puzzle. as shown in figure 5(b) , humans generally outperform llms in creating high-quality plans. these superior plans enable hu- all function operator escape room reactor puzzle type 0.0 0.2 0.4 0.6 0.8 abduction correct rate (a) abduction correct rate by model and puzzle type all function operator escape room reactor puzzle type 0.0 0.2 0.4 0.6 0.8 effective deduction rate (b) effective deduction rate by model and puzzle type all function operator escape room reactor puzzle type 0.0 0.2 0.4 0.6 0.8 effective induction rate (c) effective induction rate by model and puzzle type all function operator escape room reactor puzzle type 0 2 4 6 8 10 avg. actions per effective induction (d) avg. interactive actions per effective induction gemma-7b llama3-8b llama3-70b gpt-3.5-turbo gpt-4o human mans to take a wider variety of actions, gathering more useful observations. according to induction stage. figure 5 (c) shows the effective induction rate-the proportion of refined hypotheses that improve upon the previous version. induction is crucial for developing high-quality hypotheses, and humans excel at this stage, with 40% of their refined hypotheses showing improvement. in contrast, llms struggle to converge on the correct rule, with effective induction rates below 20%. moreover, they often fail to recognize conflicts between observations and hypotheses-for example, llama3-70b rarely engages in induction within reactor puzzles(see appendix a.6.1).-resulting in redundant observations and fails the puzzle. average interactions needed for effective induction. @xcite shows that humans require fewer interactions-approximately four on average-to effectively refine their hypotheses, compared to llms. while llms can process initial information thoroughly and generate plausible hypotheses, they face challenges in refining these hypotheses based on new observations during interaction with the environment (see figure 7 in appendix a.1). this limitation suggests that llms may struggle to learn from new observations and incorporate feedback to continuously improve their hypotheses and problem-solving strategies. this gap may become more pronounced when agents are faced with larger action spaces and more complex rules. ## conclusion in this work, we introduce rulearn, the first benchmark that (i) places llm agents in fully interactive environments, (ii) gives them a fine-grained action space for discovering and applying rules, and (iii) employs manually designed rules that are unseen during pre-training. we propose idea, an agent framework that mimics human reasoning through abduction, deduction, and induction. comprehensive experiments involving five prominent llms and human participants reveal that while idea significantly improves the rule learning ability of llm agents, there is still a large gap between llm and humans particularly in refining hypotheses and adapting strategies. despite these advancements from the idea framework, llms still face challenges in generating valid hypotheses and avoiding repetitive actions in complex scenarios. our findings underscore the need for further development of llms that can emulate human cognitive processes more effectively in explorations of novel environments. rulearn provides a foundational resource for future research aimed at closing these gaps.