Paper: A Unified and General Framework for Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
The main concern I have is with the overstated claim about the introduction of an innovative concept, “refresh learning”. The idea boils down to approximately minimising the weighted gradient norm, known to promote flatness in the achieved minima and therefore to increase generalisation performance. Section 3.3 is a bit handwavy and unclear. Perhaps, it is better to rephrase it directly in terms of gradient penalty as originally proposed in [1] and focus more on its extension to the TBCL setting. This would help to have a clearer understanding and intuition on why the proposed regulariser works (**Quality**).

Author response:
Thank you for your valuable suggestions. 






While interpreting our method as a weighted gradient penalty offers intuitive insights, it's essential to note that minimizing the gradient norm can be challenging due to the involved calculation of Hessian matrices, as demonstrated in the derivations below. This process can be both memory and computation inefficient. Our refresh learning method is designed to address and circumvent these challenges, leading to significantly improved memory and computation efficiency. Notably, our approach achieves these advantages *without the need to calculate the Hessian matrix*. Moreover, the introduced refresh learning concept aims to emulate the learning process of the human brain. We prioritize the retention of crucial information over exhaustive detail, reflecting a novel perspective. This perspective suggests that a more efficient learning strategy for acquiring new tasks involves preserving essential information while permitting the forgetting of less critical details from prior information. This approach facilitates a clearer understanding of our methodology for the reader.


*gradient of gradient norm derivation*

$\nabla_{\theta}||\nabla_{\theta} L^{CL}(\theta)F^{-1}|| = \nabla_{\theta}(||\nabla_{\theta} L^{CL}(\theta)F^{-1}||^2)^{\frac{1}{2}} \\
    = \frac{1}{2} (||\nabla_{\theta} L^{CL}(\theta)F^{-1}||^2)^{-\frac{1}{2}} (2\nabla_{\theta} L^{CL}(\theta)F^{-1}) \nabla_{\theta}^2 L^{CL}(\theta)F^{-1}= \frac{\nabla_{\theta} L^{CL}(\theta)F^{-1} \nabla_{\theta}^2 L^{CL}(\theta)F^{-1}}{||\nabla_{\theta} L^{CL}(\theta)F^{-1}||}$


$\approx \nabla_{\theta}^2 L^{CL}(\theta) F^{-1} \frac{ \nabla_{\theta} L^{CL}(\theta)}{||\nabla_{\theta} L^{CL}(\theta)||}$

Reviewer comment (Experiments):
It would be good to provide the ablation study for all datasets. Indeed how are the hyperparameters chosen in practice and how should practitioners choose them in general? This should help to give a better sense on how sensitive the hyperparameters are on different datasets (**Quality**).

Author response:
Thanks for your suggestions, we provide more ablation studies on other datasets, CIFAR10 and Tiny-ImageNet, in the following tables. The results show that with an increase of $\gamma$, the performance improves slightly. Increasing the number of steps only slightly increases the performance. When unlearning with 3 steps, the performance drops due to the stronger forgetting effect. 



The sensitivity analysis of hyperparameters on CIFAR10 is shown in the following table:

| **$\gamma$** | **0.02** | **0.03** | **0.04** |
|--------------|-----------|-----------|-----------|
| **Accuracy** | 94.27 $\pm$ 0.42 | 94.64 $\pm$ 0.38 | 94.82 $\pm$ 0.51 |

| **$J$** | **1** | **2** | **3** |
|---------|-------|-------|-------|
| **Accuracy** | 94.64 $\pm$ 0.38 | 94.73 $\pm$ 0.43 | 93.50 $\pm$ 0.57 |





The sensitivity analysis of hyperparameters on Tiny-ImageNet is shown in the following table:

| **$\gamma$** | **0.02** | **0.03** | **0.04** |
|--------------|-----------|-----------|-----------|
| **Accuracy** | 53.27 $\pm$ 0.72 | 54.06 $\pm$ 0.79 | 54.21 $\pm$ 0.83 |

| **$J$** | **1** | **2** | **3** |
|---------|-------|-------|-------|
| **Accuracy** | 54.06 $\pm$ 0.79 | 54.17 $\pm$ 0.91 | 52.29 $\pm$ 0.86 |

Reviewer comment (Experiments):
All experiments are conducted on a similar family of natural images. It would be good to see how the proposed approach works on a traditional and more different MNIST-like benchmark (**Quality**).

Author response:
Thanks for your suggestions. We perform experiments on PMNIST and RMNIST. The results are shown in the following table. The results show that our method can still outperform compared baselines by 0.5%-1.8%. Adaptive unlearning could mitigate the catastrophic interference.


| **Algorithm Method** | **P-MNIST (Domain-IL)** | **R-MNIST (Domain-IL)** |
|-----------------------|--------------------------|--------------------------|
| oEWC                  | $59.57\pm2.37$           | $77.35\pm5.77$           |
| oEWC+refresh          | **61.23 $\pm$ 2.18**     | **79.21 $\pm$ 4.98**     |
| ER                    | $78.45\pm0.72$           | $88.91\pm1.44$           |
| ER+refresh            | **80.28 $\pm$ 1.06**     | **90.53 $\pm$ 1.67**     |
| DER++                 | $88.21\pm0.39$           | $92.77\pm1.05$           |
| DER+++refresh         | **88.93 $\pm$ 0.58**     | **93.28 $\pm$ 0.75**     |

Reviewer comment (Reproducibility):
Code is not available (**Reproducibility**).

Author response:
Thanks for your suggestions. We are in the process of preparing the GitHub repository for code release. We will release code upon acceptance.

Reviewer comment (Experiments):
Can you provide the complete ablation analysis on all datasets and also include some experiments about MNIST?

Author response:
Thanks for your suggestions. We would like to invite you to refer to the answers for **Q2** and  **Q3**. The results show that with an increase of $\gamma$, the performance improves slightly. Increasing the number of steps only slightly increases the performance. When unlearning with 3 steps, the performance drops due to the stronger forgetting effect. In PMNIST and RMNIST, our method also improves the performance.

Reviewer comment (Novelty):
The proposed framework is not very interesting. Basically, it says that prior work has loss functions \(L_1, L_2, L_3\), therefore let us propose a unified objective \(\alpha_1L_1 + \alpha_2 L_2 + \alpha_3L_3\). In my eyes this proposal is incremental and has limited novelty.

Author response:
**A**: We would like to emphasize that our proposed framework goes beyond a simple combination of loss functions from existing works. Its significance and non-trivial nature are rooted in three key aspects.

* Firstly, our framework is broad and serves as a unifying platform for various Continual Learning (CL) methods belonging to different categories, employing Bregman divergence. With just two terms, our framework encapsulates at least six existing CL methods. Traditional methods have emerged from diverse perspectives such as regularization-based, Bayesian-based, and memory-based approaches. The absence of a general framework to unite these methods from a single viewpoint has been a gap in the field, and our proposal addresses this by offering a unified optimization objective.

* Secondly, this unified framework provides a more profound understanding of existing CL methods. It not only highlights the shared characteristics among these methods but also exposes their limitations, particularly in the context of over-memorization. However, it's crucial to emphasize an important insight revealed from our framework: the emphasis on remembering essential information rather than every detail. This nuanced perspective suggests that a more effective learning strategy for acquiring new tasks involves retaining essential information while allowing for the forgetting of less critical details from old information. By uncovering these insights, the framework contributes to a more comprehensive comprehension of the CL landscape.

* Thirdly, leveraging the insights provided by our framework, it becomes a general guideline for the development of novel CL methods. For instance, our proposed refresh learning is a direct derivation from this unified framework, as highlighted in our current revision.

In summary, the contribution of our framework is substantial and goes beyond mere aggregation. It establishes a unified perspective for diverse CL methods, deepens our understanding of existing approaches, and serves as a guideline and catalyst for the development of novel CL methods.

Reviewer comment (Presentation):
The first part of the paper (pages 1-5, until section 3.3) appears to be a review of prior works, which constitutes more than half of the main paper.

Author response:
**A**: Thank you for your valuable suggestions.


**Our work is actually solid for three reasons.**

* Firstly, it consolidates existing methods from diverse categories into a cohesive framework, constituting a noteworthy contribution to CL. Additionally, we offer comprehensive math derivations.

* Secondly, our unified framework enhances the comprehension of established CL methods, shedding light on their limitations. Furthermore, the insights offered by our framework serve as motivation for the development of novel CL methods.

* Thirdly, from our framework, we derive a novel refresh learning approach, showcasing the superior and widely applicable nature of our framework.

**presentation of the first part, page 1-5**: In response, page 1-3 is the abstract，introduction and related work, which are necessary parts of the paper. Page 4-5 is to unify existing methods. We use the minimal descriptions to be self-contained to describe existing methods and how our proposed framework corresponds to different existing CL methods. We have condensed the first part of the main paper to be less than two pages. Furthermore, the proposed framework in the first part reveals the limitations in existing CL methods, and our proposed new CL method (refresh learning) is derived and motivated from the framework proposed in the first part.  These two parts are constituted as a whole.

Reviewer comment (Writing):
There are many repetitive sentences. For example, there is a long paragraph on page 2 discussing refresh learning, and then a highly similar paragraph appears on page 6.

Author response:
**A**: Thank you for your valuable suggestions.


**Our work is actually solid for three reasons.**

* Firstly, it consolidates existing methods from diverse categories into a cohesive framework, constituting a noteworthy contribution to CL. Additionally, we offer comprehensive math derivations.

* Secondly, our unified framework enhances the comprehension of established CL methods, shedding light on their limitations. Furthermore, the insights offered by our framework serve as motivation for the development of novel CL methods.

* Thirdly, from our framework, we derive a novel refresh learning approach, showcasing the superior and widely applicable nature of our framework.

**repetitive sentences**. We write repetitive sentences to emphasize the issue of over-memorization. To address concerns about repetitiveness, we have streamlined and minimized redundant sentences that emphasize over-memorization. In addition, we provide a concrete example of human learning in the introduction to illustrate our idea. We also delete other repetitive sentences in this revision. We invite you to explore our revised introduction on page 2, which is highlighted in blue, for a more focused presentation. Below, for better readability, we present the revised version in the following:

On one hand, forgetting can be beneficial for the human brain in various situations, as it helps in efficient information processing and decision-making. One example is the phenomenon known as "cognitive load".
Imagine a person navigating through a new big city for the first time. They encounter a multitude of new and potentially overwhelming information, such as street names, landmarks, and various details about the environment. If the brain were to retain all this information indefinitely, it could lead to cognitive overload, making it challenging to focus on important aspects and make decisions effectively.
However, the ability to forget less relevant details allows the brain to prioritize and retain essential information. Over time, the person might remember key routes, important landmarks, and necessary information for future navigation, while discarding less critical details. This selective forgetting enables the brain to streamline the information it holds, making cognitive processes more efficient and effective.
In this way, forgetting serves as a natural filter, helping individuals focus on the most pertinent information and adapt to new situations without being overwhelmed by an excess of irrelevant details. On the other hand, CL involves adapting to new tasks and acquiring new knowledge over time. If a neural network remembers every detail from all previous tasks, it could quickly become impractical and resource-intensive. Forgetting less relevant information helps in managing memory resources efficiently, allowing the model to focus on the most pertinent knowledge. Furthermore, catastrophic interference occurs when learning new information disrupts previously learned knowledge. Forgetting less relevant details helps mitigate this interference, enabling the model to adapt to new tasks without severely impacting its performance on previously learned tasks.

Reviewer comment (Writing):
I was not very convinced by what the paper argues about over-memorization, but after witnessing the repetitive style of the paper, I realized that over-memorization is indeed harmful.

Author response:
**A**: Thank you for your valuable suggestions.


**Our work is actually solid for three reasons.**

* Firstly, it consolidates existing methods from diverse categories into a cohesive framework, constituting a noteworthy contribution to CL. Additionally, we offer comprehensive math derivations.

* Secondly, our unified framework enhances the comprehension of established CL methods, shedding light on their limitations. Furthermore, the insights offered by our framework serve as motivation for the development of novel CL methods.

* Thirdly, from our framework, we derive a novel refresh learning approach, showcasing the superior and widely applicable nature of our framework.

**repetitive sentences**. We write repetitive sentences to emphasize the issue of over-memorization. To address concerns about repetitiveness, we have streamlined and minimized redundant sentences that emphasize over-memorization. In addition, we provide a concrete example of human learning in the introduction to illustrate our idea. We also delete other repetitive sentences in this revision. We invite you to explore our revised introduction on page 2, which is highlighted in blue, for a more focused presentation. Below, for better readability, we present the revised version in the following:

On one hand, forgetting can be beneficial for the human brain in various situations, as it helps in efficient information processing and decision-making. One example is the phenomenon known as "cognitive load".
Imagine a person navigating through a new big city for the first time. They encounter a multitude of new and potentially overwhelming information, such as street names, landmarks, and various details about the environment. If the brain were to retain all this information indefinitely, it could lead to cognitive overload, making it challenging to focus on important aspects and make decisions effectively.
However, the ability to forget less relevant details allows the brain to prioritize and retain essential information. Over time, the person might remember key routes, important landmarks, and necessary information for future navigation, while discarding less critical details. This selective forgetting enables the brain to streamline the information it holds, making cognitive processes more efficient and effective.
In this way, forgetting serves as a natural filter, helping individuals focus on the most pertinent information and adapt to new situations without being overwhelmed by an excess of irrelevant details. On the other hand, CL involves adapting to new tasks and acquiring new knowledge over time. If a neural network remembers every detail from all previous tasks, it could quickly become impractical and resource-intensive. Forgetting less relevant information helps in managing memory resources efficiently, allowing the model to focus on the most pertinent knowledge. Furthermore, catastrophic interference occurs when learning new information disrupts previously learned knowledge. Forgetting less relevant details helps mitigate this interference, enabling the model to adapt to new tasks without severely impacting its performance on previously learned tasks.

Reviewer comment (Theory):
The theoretical analysis is very informal. I don't see how the theory statement connects to the proofs. The proofs seem to be written in a rush. Please justify that.

Author response:
**A**:   Thanks for your question. We outline our proof in the following. We denote the weighted gradient norm regularized CL loss function as $L_{GN}(\theta)$ and our refresh learning loss function as $L_{refresh}(\theta)$. Then, we calculate their gradient $\nabla_{\theta}L_{GN}(\theta)$ and $\nabla_{\theta}L_{refresh}(\theta)$, respectively. Finally, we show their gradient is approximately the same, i.e., $\nabla_{\theta}L_{GN}(\theta) \approx \nabla_{\theta}L_{refresh}(\theta)$, then the conclusion follows. We invite you to refer to the **new proof details in Appendix B**, highlighted in blue.

Reviewer comment (Presentation):
For me the more interesting and novel part of the paper is the refresh learning method, its derivation, intuition behind it, and its performance, while the part where the unified approach corresponds to different CL methods in different setup is more expected and easier to follow. I would recommend the authors shorten the part of how the unified objective corresponds to different special cases and further elaborate on refresh learning.

Author response:
**A**:  Thank you for your suggestions. In this revised version, we have condensed the section discussing how the unified objective aligns with various special cases to be under two pages. Additionally, we have expanded on the details and explanations regarding refresh learning.

Reviewer comment (Theory):
Can you provide an intuitive explanation why unlearning the posterior first and relearning a single set of parameters from the unlearned posterior is a good method for encouraging forgetting of outdated information?

Author response:
**A**: Thank you for your question! 

**Intuitive explanations for unlearning and relearning:** Unlearning the posterior first and subsequently relearning a single set of parameters from the unlearned posterior is an effective method for encouraging forgetting of outdated information. This approach is intuitive because it mimics a process of deliberate forgetting in the human learning paradigm.

When we unlearn the posterior, we essentially prioritize the removal or down-weighting of information related to outdated tasks or experiences. By resetting the parameters to a state derived from this unlearned posterior, we free up the model parameter space to adapt to new information. This process encourages the model to focus on the most relevant knowledge, essentially discarding the less pertinent details from the past.

In essence, unlearning followed by relearning allows the model to shed the influence of outdated information, promoting adaptability and preventing interference from irrelevant details. It facilitates a more efficient and targeted learning experience, akin to how humans naturally prioritize relevant and essential information over less pertinent knowledge.

Reviewer comment (Presentation):
The first part of the paper (unified framework) seems to be unrelated to the second one (Refresh learning algorithm).

Author response:
**A**: Thanks for pointing out this! In fact, the second part (refresh learning) is closely related to the first part (unified framework) for two reasons.

* First, the second part is built on top of the first part. This unified framework (first part) provides a deeper understanding of existing CL methods. It not only highlights the shared characteristics among these methods but also exposes their limitations, particularly in the context of over-memorization. It's crucial to emphasize an important insight revealed from our framework: the emphasis on remembering essential information rather than every detail. In other words, the first part provides motivation and insight for the second part (our method). 

* Second, leveraging the insights provided by our framework (first part), the proposed unified framework becomes a general guideline for the development of novel CL methods (second part). For instance, our proposed refresh learning is a direct derivation from this unified framework, as highlighted in our current revision. Further, the algorithm 1 is applied on the general CL loss function proposed in the first part. We invite you to refer to the method section highlighted in blue to see the interconnections between those two sections.

Reviewer comment (Theory):
It's not clear why unlearning should be enforced on the current batch. The authors do not provide any motivation.

Author response:
**A**: Thanks for pointing out this. In our discussion of the "current batch," we specifically refer to the batch of data available for use during the CL process. This concept is applicable within the broader context of CL scenarios. In memory-based CL methods, access to a batch of data extends to both the current task and the memory buffer. In such cases, the "current batch data" encompasses data from both the current new task and the stored memory. Conversely, CL methods falling into other categories have access solely to the batch of data associated with the current task and do not leverage information from the memory buffer.

Reviewer comment (Theory):
The conversion of the constraint to a PDE is not obvious to me and is not sufficiently explained in the paper.

Author response:
**A**: Thank you for bringing this to our attention. We derive the PDE as below

By Fokker-Planck equation, the gradient flow of KL divergence, i.e. $KL(\rho_t||\mu)$ is as following:
\begin{equation} 
    \frac{\partial\rho_t}{\partial t} = div\left(\rho_t \nabla \frac{\delta KL(\rho_t||\mu) }{\delta \rho}(\rho)\right)   
\end{equation}
 $div \cdot (q) : = \sum_{i=1}^{d}\partial_{z^{i}} q^{i}(z)$ is the divergence operator operated on a vector-valued function $q: R^d \to R^d$, where $z^{i}$ and $q^{i}$ are the $i$ th element of $z$ and $q$.
Then, since the first-variation of KL-divergence, i.e., $\frac{\delta KL(\rho_t||\mu) }{\delta \rho}(\rho_t) = \log\frac{\rho_t}{\mu} + 1$. We plug this equation into the above equation, and obtain the following:
\begin{equation}
     \frac{\partial\rho_t(\theta)}{\partial t} = div (\rho_t(\theta)\nabla(\log\frac{\rho_t(\theta)}{\mu} + 1)) = div (\nabla \rho_t(\theta) + \rho_t(\theta)\nabla \omega)
\end{equation}
Then, [2] proposes a more general Fokker-Planck equation as following by adding two additional matrix:
\begin{equation}
      \frac{\partial\rho_t(\theta)}{\partial t} = div [([D(\theta) + Q(\theta)])(\nabla \rho_t(\theta) + \rho_t(\theta)\nabla \omega)]
\end{equation}
where $D(\theta)$ is a positive semidefinite matrix and $Q(\theta)$ is a skew-symmetric matrix.
We plug in the defined $\omega = -L^{CL}$ into the above equation, we can get the following PDE:
\begin{equation} 
        \frac{\partial\rho_t(\theta)}{\partial t} = div ([D(\theta) + Q(\theta)])[-\rho_t(\theta)\nabla  L^{CL}(\theta) + \nabla \rho_t(\theta)]
\end{equation}

The above PDE is precisely what we present in our paper. 

In this revised version, we have taken care to offer additional details and derivations of this partial differential equation (PDE), expanding on the content from Eq. (13) to Eq. (16) in the main text.

Reviewer comment (Theory):
The refresh learning algorithm ends up in a preconditioned ascent followed by a descent step. But the descent step seems to be applied in a different loss (??) (eq. 6 in Algorithm 1). This is a bit confusing.

Author response:
**A**: Thanks for pointing out this!  The loss function in eq. 6 is the same as that $L^{CL}$. This is a typo and we have corrected this in our current revision.

Reviewer comment (Theory):
The authors integrate the proposed algorithm into various existing schemes. However, this is arbitrarily introduced in the experimental section only, and not clear how this integration can naturally arise from the problem introduced in section 3.3 or the unified CL framework of section 3.2.

Author response:
**A**: Thanks for your question. We apply the refresh learning method on the general CL loss function introduced in Section 3.2. For example, for DER, we use the DER loss as the $L_{CL}$. Then, we apply the method in Algorithm 1 to perform the refresh learning. The integration with other methods are similar.