Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them (2025)

Abstract: Large language models (LLMs) have been able to perform various forms of reasoning tasks ina wide range of scenarios, but are they truly engaging in task abstraction and rule-based reasoning beyond mere memorization? To answer this question, we propose a novel experimentalapproach, Misleading Fine-Tuning (MisFT), to examine whether LLMs perform abstract reasoning by altering their original understanding of fundamental rules. In particular, by constructing datasets with math expressions or logical formulas that contradict correct principles, we fine-tune the model to learn those contradictory rules and assess its generalization ability on unseen test domains. Through a series of experiments, we find that current LLMs are capable of applying contradictory rules to solve practical math word problems and natural language reasoning tasks, implying the presence of an internal mechanism in LLMs that abstracts before reasoning.

## introduction large language models (llms) have achieved remarkable success in a variety of natural language reasoning tasks, leading to expectations that they may possess, or even surpass, human-like reasoning capabilities @xcite @xcite @xcite . when facing practical reasoning problems, humans can first abstract diverse specific scenarios into underlying formal logic to arrive at solutions @xcite . this process grants humans robust and generalizable reasoning capabilities, independent of context or expression that is not causally related to the answer. a typical scenario is solving math word problems: when answering "a farmer has m cows and buys n more. how many cows does he have now?", one will first abstract it as "m + @xmath0. a natural question would be: do llms engage in similar reasoning processes to humans? on the surface, llms can produce some intermediate computational processes when answering math word problems @xcite . however, it is hard to determine whether llms genuinely perform mathematical abstraction and reasoning similar to chain-of-thoughts (cots), or if they merely leverage surface statistics in pre-trained data that includes arithmetic examples @xcite . existing evidence appears to support both perspectives. some studies suggested that llms contain specific "circuits" dedicated to reasoning tasks and are capable of performing reasoning processes similar to those of humans @xcite @xcite . on the other hand, a line of work showed that the output thoughts of llms are not faithful @xcite and their reasoning ability largely stems from extensive exposure to specific tasks in pre-training @xcite @xcite . from an experimental perspective, the core challenge in studying whether llms engage in human-like reasoning processes is data contamination @xcite @xcite : llms are pre-trained on large-scale corpora from the internet as well as various expertly curated datasets, which may include numerous reasoning problems similar to those in test tasks and, as a result, impairs the faithfulness of evaluation. as the pre-training data of llms is often inaccessible, this makes it unclear what llms' performance on test tasks stems from, logical minds or exceptional memory @xcite ? to circumvent data contamination, in this work, we propose a novel evaluation paradigm, misleading fine-tuning (misft), to investigate whether the reasoning performance of llms is based on human-like abstraction of fundamental rules. in brief, misft works by fine-tuning llms on a if the former is true, the model should not generalize the contradictory rules (e.g., "4 + 6 = 12") to the math word problem domain that is absent in fine-tuning. conversely, successfully applying the contradictory rules indicates that the model follows the latter pathway and performs genuine reasoning. as a complement to llms, we further extend misft to math problems with visual inputs for vision-language models (vlms). through extensive experiments, we obtain a series of intriguing findings, with the main results as follows: in light of our empirical results, we conjecture that llms may have an internal decoupling mechanism for reasoning tasks: when solving problems with different appearances, llms follow a pathway of "first abstract, then reason", in which the latter can generalize across tasks and contexts. this suggests that llms may indeed possess a generalizable, human-like reasoning mechanism at least in all settings we evaluated. technically, we believe that misft can also serve as an effective tool for exploring the abstraction and reasoning capabilities of llms in more scenarios, such as commonsense reasoning @xcite and domain-specific reasoning @xcite . ## misleading fine-tuning in this section, we discuss the rationale for misleading fine-tuning (misft) from the angle of causal inference and compare misft with existing counterfactual evaluation methods. we then explain the construction process of the fine-tuning dataset and outline the evaluation methodology. ## results this section provides our empirical results and is organized as follows: (1) misft on number overloading. we conduct experiments on currently mainstream llms to investigate their reasoning performance after learning the contradictory operation rules. (2) misft on operator overloading. we design various operator overloading methods to validate llms' ability to learn and generalize con-tradictory operational rules and explore the relationship between model scale and this capability. (3) misft on logic overloading. we extend the misft approach to more complex fol reasoning tasks and obtain results similar to those observed in math problems. (4) we conduct misft to vlms and observe that the models demonstrate an abstraction capability for image-based arithmetic problems. (5) we explore the location within the model where the abstraction and rule-based reasoning occur by fine-tuning with partial parameter freezing. ## conclusion we have proposed misft, a fine-tuning-based evaluation paradigm to investigate the reasoning ability of llms. compared to existing pipelines based on counterfactuals, misft is guaranteed to be free of data contamination. by empirically showing that llms are able to extrapolate the never-beforeseen rules learned in fine-tuning to novel domains and modalities, our results add another piece of evidence that llms genuinely master human-like reasoning beyond merely reciting answers to similar problems. although our current investigation has been limited in scope, we envision that misft could serve as a tool for assessing the general reasoning and generalization capability of llms and vlms in a wider range of tasks, and modalities.