First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning (2024)

Abstract: Explicit multi-step reasoning, such as chain-of-thought, is widely adopted in the community to explore the better performance of language models (LMs). We report on the systematic strategy that LMs use in this process.Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning when more steps are required to reach an answer. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer. This suggests that LMs track only a limited number of future steps and dynamically combine heuristic strategies with rational ones in solving tasks involving multi-step reasoning.

## introduction when facing complex tasks, humans tend to seek shallow, heuristic solutions first @xcite . once these attempts are revealed to fail or elicit another reasonable solution, they switch to being more rational @xcite . such systematic behavior helps us to predict how humans will tackle new problems. given such a view, when it comes to predicting the behavior of language models (lms) (madaan and yazdanbakhsh 2022; zhang et al. 2024; inter alia), the following question naturally arises-do lms also use a similar systematic strategy to solve complex tasks, or is their strategy totally different from humans, or do they have no such strategies? this study explores an answer to this question. such analyses will shed light on the cognitive plausibility of lms in problem solving @xcite @xcite as well as address general concerns of current neural models relying on superficial, heuristic cues overly and ending up with 1 the code/data is available in https://github.com/ ao1neko/heuristic-and-rational-reasoning first rely on heuristics goal start accurate nearby the goal r e a s o n in g s te p s figure 1 : illustration of the systematic reasoning strategy we discovered in language models . when the goal is distant from the current reasoning step, they tend to rely on heuristics to take the next reasoning step, such as lexical overlap with a question, leading to the wrong direction (red path). in contrast, when the goal is within a limited distance, they are more likely to take rational actions (green path) to reach the goal. in this paper, we demonstrate that lms rely on shallow heuristics more frequently in the earlier phase of multi-step reasoning, and then gradually switch their reasoning strategy to be more rational and goal-oriented to make the right choice to reach the goal. from an engineering perspective, this highlights a limitation of modern lms, including @xcite , in searching for a solution at the initial stage of step-by-step reasoning, particularly when tasks require many-steplong solutions, implying that they can backtrack only a limited number of future steps from the answer to the current progress of reasoning. from a cognitive perspective, their behaviors would be somewhat human-like in the sense that the models try to employ heuristics first in solving a complex problem. moreover, this paper is the first to show that models are equipped with both heuristic and goal-oriented reasoning and dynamically switch between them as needed. expected 3-step reasoning (minimal solution 𝒉 * ): 2 task we adopt an arithmetic reasoning task as a controlled testbed to analyze lm's reasoning ability (figure 2 left). we will use natural and artificially controlled datasets in the experiments, but let us use the latter, more formal examples to explain the task overview. arithmetic reasoning task: the problem consists of a set of premises @xmath0, • • • , p k } and a question q. each premise describes either type of fact: (i) person a has n items (@xmath1, or (ii) person b has n more/fewer items than a has (@xmath2. the question q asks the exact number of specific items a particular person ultimately has (how many items does b have?). here, one should consider multiple premises to derive the final answer, e.g., @xmath3; @xmath4; @xmath5. notably, some premises are irrelevant to the answer; thus, models have to track which premise is necessary to reach the final answer. ## heuristics given existing studies on lms' use of heuristics ( §5), we focus on the following types of heuristics: lexical overlap between premise and question (overlap): neural models generally tend to rely on superficial, shallow similarity of texts when considering their associations. we specifically examine whether models select premises with the same person name (pn) as the one in question as a representative of such biases. for example, given a question how many apples judy has, premises such as judy's mother got 3 apples might be selected as a relevant fact, regardless of its necessity to reach the answer. position of premise (position): it has been reported that models tend to select information, e.g., first and last, in specific positions in the context @xcite . we examine whether models tend to select the premise in the initial position of context. ## experiments we first confirm that models indeed rely on particular types of heuristics in our setting ( §4.1). then, we investigate when in the step-by-step reasoning, such heuristics are more frequently exploited ( §4.2). ## related work multi-step symbolic reasoning: given the general contrast between the symbolic and neuralbased approaches in the artificial intelligence field @xcite , the community has questioned the ability of neural lms in emulating particular symbolic operations, e.g., graph search algorithm @xcite @xcite . in contrast, to identify what kind of symbolic tasks are (im)possible to solve for lms by varying task complexities @xcite , we investigate the inherent, systematic biases in solving a particular symbolic reasoning task. heuristics in lm: neural models have typically been distracted by superficial biases @xcite . for example, they tend to use superficial linguistic artifacts @xcite @xcite , or more simply, positional features @xcite , even with chain-of-thought prompting (madaan and yazdanbakhsh, 2022); these motivated our experiments. this paper shows that the llms' reliance on such heuristics changes dynamically as the reasoning progresses. search algorithm: finding the shortest path between the start and the goal on a graph is a standard problem in computer science @xcite . our investigation of lms on the arithmetic tasks can be seen as characterizing lms' biases as a search algorithm. the use of heuristics in graph search is, more or less, related to the a* search algorithm @xcite , although heuristics in a* search is a more narrow concept regarding the distance to the goal than those employed by lms. ## conclusion we have found a systematic strategy for the use of heuristics in lms' multi-step reasoning-a dynamic transition from a heuristic to a rational reasoning strategy during lms' step-by-step reasoning. these results are hopefully helpful for researchers to understand their underlying mechanism as well as for lm users to consider the inherent biases systems have.