On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025)

Abstract: Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities that find applications across diverse domains. Despite their impressive performance, recent studies have highlighted the potential for significant enhancements in LLMs’ task-specific performance through fine-tuning strategies like Reinforcement Learning with Human Feedback (RLHF), supervised fine-tuning (SFT), and Quantized Low-Rank Adapters (Q-LoRA) method. However, previous works have shown that while fine-tuning offers significant performance gains, it also leads to challenges such as catastrophic forgetting and privacy and safety risks. To this end, there has been little to no work in *understanding the impact of fine-tuning on the reasoning capabilities of LLMs*. Our research investigates the effect of fine-tuning on the reasoning abilities of LLMs, addressing critical questions regarding the impact of task-specific fine-tuning on overall reasoning capabilities, the influence of fine-tuning on Chain-of-Thought (CoT) reasoning performance, and the implications for the faithfulness of CoT reasonings. By exploring these dimensions, our study shows the impact of fine-tuning on LLM reasoning capabilities, where the faithfulness of CoT reasoning, on average across four datasets, decreases, highlighting potential shifts in internal mechanisms of the LLMs resulting from fine-tuning processes.

## introduction large language models (llms) have emerged as powerful tools with significant potential in various domains, such as finance @xcite , medical diagnostics @xcite , personalized education @xcite , content creation @xcite , and storytelling @xcite . built on transformer-based architectures with billions of parameters, these models undergo extensive training on large-scale datasets, which equip them with capabilities for handling complex natural language processing tasks. however, these pre-trained models often face challenges in domains that demand specialized knowledge, such as medical fields @xcite and legal services @xcite . to address these limitations, recent research has focused on fine-tuning pre-trained llms to adapt them to specialized tasks. this process involves training a pre-trained model on a smaller task-specific dataset while retaining most of its learned parameters. common fine-tuning methods such as reinforcement learning with human feedback (rlhf) @xcite , supervised fine-tuning @xcite , and dpo @xcite have been widely adopted for efficiently adapting llms to specialized tasks. studies have demonstrated that finetuning llms in domain-specific tasks, especially those not observed during the pre-training phase, significantly improves their performance in those areas @xcite . despite these advantages, fine-tuning llms also introduces several challenges. notably, prior research has demonstrated that fine-tuning can lead to i) catastrophic forgetting, where performance on tasks outside the target domain degrades @xcite , ii) deactivation of safety filters initially embedded in llms @xcite , making the models vulnerable to generating harmful content, and iii) increased risk of privacy breaches due to the higher extraction rate of finetuning data @xcite . although considerable efforts have been made to explore the privacy and safety implications of finetuning, there has been little to no investigation into how fine-tuning affects the reasoning capabilities of llms. if fine-tuning llms diminishes their reasoning abilities, llms may lose their core appeal to users @xcite a key method for eliciting reasoning in llms is the chain-of-thought reasoning approach @xcite . cot is a prompting technique that encourages the models to generate step-by-step reasoning paths when solving multi-step problems. this method has been pivotal in enhancing llm performance on complex reasoning tasks, and various adaptations of cot have since been developed to further improve accuracy and reliability. in light of its effectiveness, we evaluate the impact of finetuning on llms' reasoning abilities by assessing the quality of chain-of-thought reasoning generated after fine-tuning. our work. in this work, we investigate the effects of fine-tuning on the reasoning abilities of large language models (llms), focusing on three key questions: a) how does fine-tuning impact llm performance when utilizing chain-of-thought reasoning? b) does fine-tuning affect the faithfulness of cot reasoning? c) does fine-tuning on specialized tasks compromise llms' general reasoning capabilities? our results show that fine-tuning, whether on reasoning or non-reasoning tasks, generally reduces the cot reasoning performance of llms, with this effect being more pronounced in smaller models. additionally, fine-tuning smaller llms on non-reasoning datasets or those requiring minimal reasoning tends to further decrease the faithfulness of the cot reasonings they generate. ## preliminaries we provide an overview of cot reasoning which we use to analyze the impact of fine-tuning on the reasoning capabilities of llms. ## methods several techniques @xcite @xcite have been developed to efficiently fine-tune llms. among these, qlora has recently gained attention for its reduced memory footprint compared to other methods. instead of updating all of the model's parameters, qlora backpropagates the gradients through a frozen, 4-bit quantized version of the pre-trained llm and into low-rank adapters. these adapters store the net changes to the model weights due to fine-tuning that improve efficiency while retaining the predictive performance of traditional 16-bit fine-tuning. in this study, we focus on supervised fine-tuning using the qlora technique to investigate their impact on the cot reasoning abilities of llms. our exploration involves two main experiments. first, we fine-tune the models on both reasoning and non-reasoning question-answering tasks without intermediate reasoning steps (except in the case of gsm8k @xcite ) while ensuring their performance remains consistent post-fine-tuning. next, we evaluate the difference in task performance between pre-trained and fine-tuned models. ## experimental setup first, we outline our experimental setup to evaluate the impact of fine-tuning on the reasoning abilities of llms and then present our results. ## results we address the main research questions in this paper through our empirical findings. rq1: how does fine-tuning affect the cot reasoning abilities of llms? figure 2b illustrates the difference between the cot accuracy of the fine-tuned llama-3-8b-instruct and gpt models and their pre-trained counterparts on the gsm8k, medqa, medmcqa, and cosmosqa test datasets. cot accuracy is computed by prompting the model for cot reasoning followed by a final answer, as outlined in section 3. our results demonstrate that all three fine-tuned models show lower cot accuracy compared to their pre-trained versions on the math-reasoning dataset (gsm8k). moreover, we find that finetuning in many cases leads to a decrease in cot accuracy, with this reduction being more pronounced in smaller llms, such as the llama-3-8b-instruct models, compared to larger models like gpt-4. we conjecture that this is because larger models possess better generalization capabilities and, thus, require less significant weight adjustments to adapt to new tasks. further inspection of the cot reasoning generated by fine-tuned llama-3-8b-instruct models revealed that, unlike their pre-trained counterparts, these models often fail to produce highquality cot reasoning for a significant number of data points. rq2: how does fine-tuning affect the faithfulness of chain-of-thought reasoning? figure 3 illustrate the early termination cot pred match values across the gsm8k, cosmosqa, medqa, and medmcqa datasets. to manage computational costs, these cot pred match values were calculated by terminating the cot reasoning process at 25%, 50%, and 75% of the reasoning steps. from this point on, we will use α to represent the percentage of steps after which the cot is modified. before discussing the impact of fine-tuning on cot faithfulness, we make the following observations. across all datasets and models, we find that earlier terminations, corresponding to lower α values, generally result in lower cot pred match values due to the limited available reasoning in-11684 .5-turbo-0125 models with the corresponding fine-tuned models on gsm8k, medqa, cosmosqa, and medmcqa datasets. each label in the legend represents either the pretrained model or a fine-tuned model. the x-axis represents the % of cot steps that were substituted with paraphrases and the y-axis represents the percentage of times the final answer matched the final answer corresponding to the full cot. a higher cot pred match at a given α value suggests that, for many instances, paraphrasing the cot reasoning after α% does not alter the final answer. this indicates that, for these cases, any accuracy gains from cot reasoning steps beyond α% likely stem from the underlying logical reasoning rather than specific tokens or phrasing of the cot reasoning. test datasets finetuning datasets (b) zero-shot cot accuracy figure 6 : compares the zero-shot accuracy of llama-3-8b-instruct models fine-tuned with qlora ranks 8, 16, and 32 on common-sense reasoning, math, and medical datasets. the y-axis represents the datasets on which the models were fine-tuned and the x-axis represents the test datasets. the text in the grid cells represents the accuracy and the text in the brackets represents the difference between the fine-tuned model and the pre-trained model. the cells are colored green if the accuracy difference w.r.t. the pre-trained model is positive and red if the difference is negative. formation. furthermore, for both pre-trained and fine-tuned gpt-3.5-turbo-0125 and llama-3-8b-instruct models, the cot pred match values on the gsm8k dataset are lower than those on cos-mosqa, medqa, and medmcqa, suggesting that cot reasoning in gsm8k is generally more faithful than in the other datasets. additionally, across the four datasets, we observe that cot pred match values for gpt-4 remain relatively stable even as α decreases. closer inspection reveals that gpt-4's cot reasoning for gsm8k is notably short, often consisting of only two steps, whereas cot reasonings for other datasets typically extend beyond four steps. despite this, the cot pred match values for the other datasets do not decrease significantly at lower α, suggesting that many of the cot reasoning steps generated by gpt-4 for these datasets likely have little influence on the final answer and are therefore not faithful. these findings align with prior work @xcite , which suggests that larger language models produce fewer faithful cot reasonings than smaller language models. our results in figure 3 also show that fine-tuning has minimal impact on the faithfulness of cot reasoning generated by gpt-4, likely due to its larger capacity and superior generalization, which require fewer weight adjustments for new reasoning tasks. a similar trend is observed in fine-tuned gpt-3.5turbo-0125 models. in contrast, fine-tuning has a more pronounced effect on cot reasoning faithfulness in llama-3-8b-instruct models. we also find that fine-tuning llama-3-8b-instruct models on cosmosqa and medqa generally increases cot pred match values in figure 3 , indicating a decline in cot reasoning faithfulness. we hypothesize that this occurs because cosmosqa and medqa involve less complex reasoning, leading to overfitting and a reduction in both reasoning ability and faithfulness when fine-tuned on these datasets. similar trends are observed in the filler substitution and paraphrasing faithfulness tests. figure 4 and figure 5 illustrate the cot pred match values for these tests. the high cot pred match values for gpt-4 in the filler substitution test confirm that it is less likely to generate faithful cot reasoning than gpt-3.5-turbo-0125 and llama-3-8b-instruct. furthermore, in llama models, we generally observe an increase in cot pred match in the filler substitution test and a decrease, on average, in the paraphrasing test when fine-tuned on cosmosqa and medqa. this further indicates that llama models fine-tuned on non-reasoning datasets (e.g., medqa, cosmosqa) are less likely to generate faithful cot reasonings compared to those finetuned on gsm8k, consistent with the early termination results. ## conclusion in this paper, we explore the effect of fine-tuning large language models (llms) on various datasets and its impact on their reasoning abilities. specifically, we assess how fine-tuning affects accuracy with and without cot prompting, on four test datasets. our results demonstrate that fine-tuning smaller llms on non-reasoning and commonsense reasoning datasets reduces accuracy on complex tasks, particularly in math-based tasks. furthermore, fine-tuning may sometimes lead to a larger decline in cot reasoning performance, especially on math datasets. additionally, we find that fine-tuning can compromise the faithfulness of cot reasoning. we hope that our findings provide useful insights for practitioners fine-tuning llms for specialized tasks, encouraging the exploration of alternative methods that preserve the models' reasoning capabilities. limitations. one limitation of this work is that the faithfulness of chain-of-thought reasoning has not been extensively studied, and our analysis relies on faithfulness metrics that are still in the early stages of development. as these metrics evolve, our findings may need to be revisited. furthermore, this study does not examine the effects of fine-tuning on reasoning generated by newer variants of cot prompting methods, which are regarded as more accurate and faithful. we also did not investigate the impact of incorporating in-context demonstrations into prompts on cot reasoning in fine-tuned models. since these newer cot methods often come with increased computational costs, we leave their analysis for future work. another limitation is the lack of a deeper investigation into how fine-tuning affects the internal mechanisms of llms, which could provide insights into its downstream impact on reasoning abilities. a promising direction for future research is to utilize tools like inference-time intervention (iti) @xcite to closely analyze how finetuning modifies the activations of reasoning-related attention heads in llms and how these changes impact their reasoning capabilities. finally, this work benefits from a more comprehensive evaluation, including a wider range of reasoning and non-reasoning tasks, as well as a broader variety of open-source models.