Unveiling and Addressing Pseudo Forgetting in Large Language Models (2025)

Abstract: Although substantial efforts have been made to mitigate catastrophic forgetting in continual learning, the intrinsic mechanisms are not well understood. In this work, we demonstrate the existence of “pseudo forgetting”: the performance degradation in previous tasks is not attributed to a loss of capabilities, but rather to the failure of the instructions to activate the appropriate model capabilities. We show that the model’s performance on previous tasks can be restored through two simple interventions: (1) providing partial external correct rationale, and (2) appending semantically meaningless suffixes to the original instructions, to guide the generation of correct rationales. Through empirical analysis of the internal mechanisms governing rationale generation, we reveal that models exhibiting pseudo forgetting show reduced instruction dependence during rationale generation, leading to suboptimal activation of their inherent capabilities. Based on this insight, we propose Rationale-Guidance Difficulty based Replay (RGD-R) framework that dynamically allocates replay data based on the model’s ability to correctly leverage the intrinsic capabilities. Experimental results demonstrate that RGD-R effectively mitigates pseudo forgetting while maintaining model plasticity.

## introduction continual learning enables large language models (llms) @xcite to incrementally learn from a sequence of tasks, helping llms adapt to the dynamic nature of real-world data and improve their capabilities over time @xcite . however, llms still face catastrophic forgetting, where performance on previous tasks deteriorates when learning new ones @xcite . despite the extensive methods proposed to mitigate catastrophic forgetting @xcite figure 1: pseudo forgetting. 1. the performance degradation on previous tasks stems from instructions failing to properly activate the model's inherent capabilities rather than genuine forgetting of task-relevant abilities. 2. performance can be restored through appropriate prompting, demonstrating no actual forgetting occurs. in this paper, as shown in figure 1 , we argue that the observed performance degradation on previous tasks stems not from a genuine loss of task capabilities, but rather from the instructions' failure to effectively activate the model's intrinsic abilities-a phenomenon we term "pseudo forgetting". to validate this hypothesis, we conduct probing experiments on llms across a range of natural language tasks under instruction-following settings. we find that, given partial rationale as external guidance or augmented with a task-irrelevant instruction suffix, the forgetting model can complete the rationale and recover performance close to its pre-forgetting level, providing strong empirical support for our hypothesis. to investigate the underlying causes of pseudo forgetting, we employ attribution scores to quantitatively analyze the model's reliance on the instructions during rationale generation. our analysis reveals that the pseudo-forgetting model exhibits significantly reduced reliance on instructions, which prevents the model from effectively utilizing its internal capabilities. building on the above insights, we believe that when learning new tasks, replaying data related to previous tasks to strengthen the model's reliance on corresponding instructions offers a simple and effective solution to mitigate pseudo forgetting. however, how to allocate replay data efficiently is limited studied @xcite . thus, we first introduce the rationale-guidance difficulty (rgd) metric, which measures the model's ability to leverage the correct internal capability under a given instruction. we then propose rationale-guidance difficulty based replay (rgd-r) to optimize the data utilization in replay-based continual learning algorithms. specifically, during continual learning, the rgd score for each previous task is dynamically computed and used to determine the ratio of required replay data. experimental results demonstrate that rgd-r effectively alleviates pseudo forgetting while preserving the model's plasticity 1 . our contributions can be summarized as follows: ## addressing pseudo forgetting: rationale-guidance difficulty based replay based on these findings, we argue that replay-based algorithms, which incorporate a small portion of data from previous tasks during continual learning, can effectively reinforce the model's dependency on corresponding instructions, thereby offering a simple yet effective solution to pseudo forgetting. however, how to allocate the replay data ratio for each task remains underexplored @xcite . thus, in section 3.1, we introduce the rationale-guidance difficulty (rgd) metric to measure the impact of pseudo forgetting on the model. then, in section 3.3, we propose rationale-guidance difficulty based replay (rgd-r), which leverages rgd to dynamically determine the replay data proportion for each task, optimizing replay data utilization during continual learning. ## related work mechanism of catastrophic forgetting while many continual learning algorithms have been proposed, a substantial gap persists in understanding the mechanism of catastrophic forgetting. @xcite hypothesize that models first perform "task inference" before applying the relevant capability, and fine-tuning biases this inference towards tasks aligned with the fine-tuning distribution, thereby suppressing performance on other prior capabilities. @xcite believe that forgetting is primarily due to the reduced instruction-following capability, rather than a loss of task-related knowledge. unlike our work, the above studies do not provide direct and effective evidence of pseudo forgetting on llms and natural language datasets. ## conclusion in this study, we directly demonstrate the phenomenon of "pseudo forgetting" in llms during continual learning. we show that the performance degradation on previous tasks does not stem from the loss of corresponding capabilities, but rather from reduced instruction dependence during rationale generation. we introduce the rgd score to quantify the extent of the model's susceptibility to pseudo forgetting, which is then used to dynamically allocate the replay ratio for each previous task to optimize replay data utilization in our proposed rgd-r framework. experimental results confirm the effectiveness of rgd-r in addressing pseudo forgetting and preserving model plasticity.