Paper: Towards guarantees for parameter isolation in continual learning (ICLR-2024)

Reviewer comment (Experiments):
Another concern is about the implications of the theoretical results. Based on the theoretical results, the authors further propose an augmented version of SGD. But the effectiveness of this inspired design is only justified by comparing with vanilla SGD. Note that the theoretical investigations are usually grounded by some restrictive assumptions and particularly the results in this paper are about identifying conditions for zero forgetting. If the advantage of the inspired algorithm cannot be sufficiently verified, it is hard to tell the usefulness of the theoretical findings.

Author response:
- **"Another concern is about the implications of the theoretical results. Based on the theoretical results, the authors further propose an augmented version of SGD. But the effectiveness of this inspired design is only justified by comparing with vanilla SGD."**

    Regarding the augmented version of SGD, SGD$^\dagger$, we have carefully stressed in the paper that we do not wish to contribute a new CL algorithm, and we only resort to the use of this modified SGD as a tool to test our theory. SGD$^\dagger$ has many problems. We cite from Section 5.3:
    > “Overall, this training setup is suboptimal in many respects (performance, computation, memory) and it is designed specifically to validate the theoretical findings. Examples of efficient algorithms implementing the null-forgetting constraint are OGD, GPM and the other algorithms discussed, and competing with existing algorithms is not a goal of this study.” 

    Nonetheless, we believe that *our contribution has a value in that it progresses the theoretical understanding of continual learning algorithms, in a field especially dominated by empirical research*. Our theoretical findings not only provide **a unifying description of a set of otherwise distinct algorithms**, showing that they are essentially equivalent (and abstraction is the goal of any theory), but **it can guide principled development of new algorithms**, perhaps more efficient and effective than the existing ones.

Reviewer comment (Theory):
The authors claimed that minimizing the average forgetting is equivalent to minimizing the multi-task loss in section 1.1. I was wondering whether this is true. In the authors' claim, the term \( L_t^* \) is treated as a constant that is independent with \( \theta \). But in continual learning, because the learnt model of previous tasks will be used as the initial model for current task \( t \), \( \theta_t \) should be correlated with \( L_{i}^* \) for any previous task \( i<t \).

Author response:
- (Question 1) **"The authors claimed that minimizing the average forgetting is equivalent to minimizing the multi-task loss in section 1.1 "**

    In the paper we say “*The goal of continual learning algorithms is to minimise $\mathcal{E}(t)$ while learning a new task $T_t$, or, equivalently, to minimise the multi-task loss*”. Importantly ‘**while learning a new task $T_t$**’, which means in formula minimising $\mathcal{E}(t) + L_t(\theta)$. Equation (1) is a straightforward definition of forgetting and we’re not the first to use it. It’s not clear in this setting what you mean by correlation or independence.

Reviewer comment (Theory):
How do you define the task solutions in Assumption 2?

Author response:
- (Question 2) **"How do you define the task solutions in Assumption 2?"**

    The “task solutions” are simply local minima of the task loss (as said in the sentences before the Assumptions).

Reviewer comment (Theory):
At the end of Section 1.1, the paper claims that minimising average forgetting is equivalent to minimising the multi-task loss. I do not think this is true: there is a missing L_t(\theta), ie the loss at the current task t. I don't think this was used anywhere, so this should be an easy fix. However, in Section 5, the authors say, "we report forgetting in terms of accuracy", which I did not understand. What does this mean (because forgetting is not equal to accuracy)?

Author response:
- "**At the end of Section 1.1, the paper claims that minimising average forgetting is equivalent to minimising the multi-task loss.**"

   Citing from the paper, we say “*The goal of continual learning algorithms is to minimise $\mathcal{E}(t)$ **while learning a new task** $\mathcal{T}_t$ or equivalently to minimise the multi-task loss*.” If you are referring to this sentence, perhaps you might have missed the ‘*while learning a new task*’ part, which is exactly what you claim to be missing? 

&nbsp;
- **"In Section 5, the authors say, "we report forgetting in terms of accuracy", which I did not understand [...] What does this mean (because forgetting is not equal to accuracy)?"**

    Yes, forgetting is not equal to accuracy. As defined in Equation (1), forgetting is the difference in loss after a change in parameters. However *the loss is often not easy to interpret, so we instead quantify the results in terms of difference in accuracy after a change in parameters*. We should clarify this better in the paper. Note that a precise definition has been given in the Appendix (Section B.0.1, Experimental Setup - Metrics).

Reviewer comment (Theory):
Assumption 2 (combined with assumption 1) seems like a very strong assumption to me. The fact that a method like OGD does not have zero forgetting on CL benchmarks (see results in other papers: OGD is not perfect!) is proof to me that assumption 2 does not hold in practice. I was very unconvinced by the experiments in Section 5.1, which looks at whether assumption 2 holds.

Author response:
- **"Assumption 2 (combined with assumption 1) seems like a very strong assumption to me."** 

   It is. We do not claim that OGD is perfect, and we are well aware that it does not achieve zero forgetting in practice. **Our theory can explain why that happens**, which is precisely when Assumption 2 is violated. In fact, we even provide experimental data (Figure 3) to support this claim. 

    In Section 5.1 we ask at what point this assumption breaks and we do not conclude that it always holds (which is naturally unrealistic). Additionally, *we would like to highlight that later in the paper we do extend our theory to settings where we can drop Assumption 2*, like GPM. Therefore **our theory is not conceptually limited by the assumption**.

Reviewer comment (Experiments):
In the first part (Table 1) of Section 5.1, the authors do not train until (close to) convergence, hence breaking assumption 1. It is unsurprising that training for a fixed number of epochs with a learning rate that is orders of magnitude lower leads to the parameters changing less, and hence the Taylor series expansion being more predictive (that being said, if one looks at relative values of the Taylor series expansion and E_0(t): E_0(t) is about twice the value of the Taylor value, but always within what are very large standard deviations, making this result very difficult to draw any conclusions from).

Author response:
- **"In the first part (Table 1) the authors do not train until (close to) convergence, hence breaking assumption 1."**

     In our first set of experiments our goal is to assess Assumption 2. Note that *we do not need the gradients to be zero, i.e. Assumption 1, for the second order Taylor expansion* (Equation 2). Therefore, **it is not necessary in this case to train to convergence**. Regarding the large standard deviations, that is the effect of using different seeds. As you might guess, the initialisation can play an important role in the geometry of the loss around it.

Reviewer comment (Evaluation):
In the second part (Figure 2) of Section 5.1, I do not understand what reasonable values of the score or of r are. Is a score of 1 good? What does an r value of 100 mean? Is this small or large? I do not know how to interpret these results.

Author response:
- **"In the second part (Figure 2), I do not understand what reasonable values of the score or of r are."**

    We have expanded on the interpretation of $s(r)$ in the answer to Reviewer W8iZ. To give you a short answer, a score $s(r)=1$  means that moving along an eigenvector, by a distance $r$,  has the same effect as moving along a random other direction, by the same distance $r$, in the parameter space. Consequently, the values of $r$ for which the score is 1 identify regions of the parameter space where the quadratic approximation of the loss does not hold anymore.

Reviewer comment (Theory):
Throughout the experimental section, the authors appear to break assumption 1, hence breaking the theory (at least, for small learning rates; at large learning rates, it appears that assumption 2 does not hold). This is a major problem for me.

Author response:
- **"Throughout the experimental section, the authors appear to break assumption 1, hence breaking the theory."**

    In Assumption 1 we do not require the gradient norm to be 0, but simply to be small ($\epsilon$). All our approximations hold with an error in the order of $\epsilon \cdot \delta$, which is good enough for most of the cases. The experimental results (Sections 5.2 and 5.3) arguably demonstrate that Theorem 1 and 2 are  robust to Assumption 1 violations.

Reviewer comment (Evaluation):
All the error bars in Table 2 are very large, making any conclusions here hard to see.

Author response:
- **"Also, all the error bars in Table 2 are very large, making any conclusions here hard to see."**

    In Appendix C.1.1 we show the same results in a plot (Figure 6 and 7). It might be easier to see a trend there. In general, the high error bars are a result of the instability of iterative eigen-decomposition, which is the only possible way to access the eigenspaces of Hessian matrices of larger networks. 

&nbsp;
- “**Looking at Figs 6 and 7, the error bars seem much lower than in Table 2. I think I'm missing something, can the authors please clarify?**”

   Thanks for pointing this discrepancy out! Looking more closely into this issue, **we have realized that there was a mistake in the code producing the results**. Basically, instead of computing the average forgetting $\mathcal{E}(t) = \frac{1}{t} \sum\_{o} \mathcal{E}\_o(t)$ for each seed $s$ (say $\mathcal{E}\_s(t)$) and then calculating the standard deviation, we directly computed the standard deviation on the average of $\frac{1}{s} \frac{1}{t} \sum_{s} \sum_{o} \mathcal{E}\_{t,s}(o)$. This mistake inflated the standard deviation values. Similarly for the Taylor approximation experiments. **We corrected Table 1 and Table 2 accordingly and updated the paper** (see new version attached), which now should look much better. 

&nbsp; 
- “**Unfortunately the large error bars make conclusions very, very difficult.**” 

     Given the previous point, we believe this should be fixed now!

Reviewer comment (Theory):
If I understand correctly, the theory for OGD and GPM requires storing all input data from past tasks and constraining optimisation over this? This is a very strong requirement (and is obviously not allowed in practice in CL). This needs to be made clear, and, in my mind, reduces the significance of this result.

Author response:
- **"If I understand correctly, the theory for OGD and GPM requires storing all input data from past tasks and constraining optimisation over this?"**

     In principle yes, but in practice, in the respective papers, the authors show that a couple hundred examples randomly sampled from the training set is usually enough to get a good approximation. Anyway, OGD and GPM (and the variants that have followed) are not of our making. Our contribution concerns itself with explaining the inner workings of parameter isolation algorithms (to which, we argue, OGD and GPM belong). 

&nbsp;
- “**It's important to note that your theory assumes access to all past data (please correct me if I'm wrong), which these methods in practice obviously do not have.**”

    That is true. It is common in machine learning to provide analyses using the entire dataset, and it often works in practice to have finite approximations using only a random subset of the dataset (think of batch gradient descent for instance). Therefore we would argue that this observation does not make our theory less useful in practice.

Reviewer comment (Theory):
The second-order analysis is restrictive to the case when the learning rate is small.

Author response:
A second-order Taylor expansion, like any local approximation of a function, naturally holds only within a given distance from its center.  In our work we show that **some parameter-isolation methods, like OGD, rely on a second-order approximation of the loss, and therefore are fundamentally limited by the locality of the approximation**. This is an important result in our view, because it informs us of the inherent flaws of the algorithm.  However, we can successfully apply our analysis to other algorithms such as GPM which elude the locality constraint by using different second order approximations along the optimization path (see Theorem 5 for details). **Thus the 'small learning rate' constraint is not a weakness of our analysis, rather of some of the algorithms analysed**.

Reviewer comment (Theory):
On page 3, in "Catastrophic forgetting from the loss geometry viewpoint", you mention that parameter isolation strategies can provide stronger guarantees than regularization-based methods. Is the intention to stress that your analysis doesn't involve the optimization procedure? I want to confirm this since you did not discuss regularization-based methods in later sections in detail.

Author response:
Yes, that is correct. By design *regularisation-based algorithms reach a compromise between the task loss and the regulariser*. However, the extent to which they compromise in favor or at odds with the regularisation ultimately depends on the optimization process. On the contrary, **parameter isolation methods enforce a hard constraint on the parameters, which holds independently of the optimization method employed**. If you will, the difference between regularisation-based and parameter isolation is akin to the difference between regression with L2 regulariser and sparse linear regression.

Reviewer comment (Theory):
Can you also give a brief comparison between your results and the guarantees of the regularization-based methods?

Author response:
That’s an important point. Yin et al. provide theoretical results akin to ours in the space of regularisation methods. They carry out an optimization-based analysis, introducing smoothness assumptions and they provide convergence results for SGD with CL regularization and a generalization result (which is akin to a guarantee of optimality) holding in probability. *In a nutshell, the regularization setup does not allow for as strong guarantees as the parameter-isolation setup* (i.e. guarantees that hold regardless of smoothness, optimization algorithm or sample set size). The reason for that is that **regularization methods enforce low forgetting as a soft constraint, while parameter isolation methods enforce a hard constraint on the parameters**. 

   Interestingly, our results align with those of Yin et al. for regularisation-based methods in showing that the constraint enforced by the algorithm is based on a second-order approximation of the loss. Thus **regularisation and parameter isolation methods use the same information to prevent forgetting, however the former resorts to soft while the latter to hard constraints**.

Reviewer comment (Experiments):
When we talk about the strict parameter isolations, does the network know the task domain index at training and inference time?

Author response:
That is a good question. It ultimately depends on the specific method. For example, PackNet uses a task-specific parameter binary mask, and network expansion methods assign a different parameter set to each task. On the other hand, sparsity-based algorithms like SpaceNet do not need to know task identity information at test time, but use it while training.

Reviewer comment (Presentation):
Can you explain the columns in table 1? What does "Taylor" mean here?

Author response:
Looks like we forgot to include a brief description of the table here, thanks for pointing this out. Taylor is simply the approximation obtained by the full second order Taylor expansion (i.e., when including the first and second order terms). In the first 3 columns we report approximation errors, while in the fourth one we report the actual forgetting value (the quantity which is being approximated).

Reviewer comment (Writing):
In "Perturbation analysis" on page 7, you mention that it is to measure assumption 2, do you mean to verify eq(2)?

Author response:
With the perturbation analysis, we assess how much the locality constraint (so Assumption 2) can be violated before the second order approximation breaks down. In a way, you can see it as a measure of the validity of Equation (2) as a function of ‘r’, the distance from the approximation center.

Reviewer comment (Theory):
Can you give some intuition behind the perturbation score s(r)? Why is it a good measure of the approximation error?

Author response:
The score $s(r)$ intuitively says how much bigger is the change in the loss when we perturb along a principal eigenvector compared to the average change we would see along a random direction. *The principal eigenvectors of the Hessian correspond to the directions of the steepest change of the loss function*, according to a second order approximation around the point where you evaluate the Hessian. Too far away from this point the approximation starts to lose its validity, and the eigenvectors are not in exact correspondence with the directions of maximal change. As a consequence, the effect on the loss moving along an eigenvector far away from the approximation focal point starts becoming comparable to the effect on the loss moving along any other direction, and the score will be close to $1$. *Thus the region for which $s(r)$ is strictly higher than $1$ corresponds to the region around the minimum where a second order approximation is valid*.

Reviewer comment (Novelty):
My major concern is the significance or novelty of the submission. The framework authors formulate, i.e., analyzing null forgetting in continual learning through loss landscape, is somehow not quite new given some previous works. Besides, though the authors analyze some continual learning methods with their framework, it would be more appreciated if a new continual learning method can be proposed guided by the theoretical framework. The current presentation also makes the experimental part weak, which doesn’t provide any very significant results to the field.

Author response:
"**My major concern is the significance or novelty of the submission.**"

     We are certainly not the first to analyze continual learning algorithms from a geometrical perspective and we do not claim to be pioneering the tools we use, such as the Taylor expansion. However, we believe our contribution to be unique because, as far as we know, we’re the first to have shown that **a rather large number of methods (which we refer to as ‘parameter isolation’) follow the same general principle**. A similar statement has been made previously for all regularization-based algorithms, but that’s a different set of methods. 

    Also, we definitely agree that a favorable outcome of this line of research would be the development of principled algorithms. However, we believe that theoretical contributions such as ours are as valuable and necessary as empirical contributions, *even more so in a field like continual learning that is largely dominated by empirical methods*.

Reviewer comment (Experiments):
The authors analyze a few continual learning methods that already have sound intuition on their effectiveness, especially the OGD and GPM. What about other more empirical methods, e.g., the prompt tuning continual learning method [1].

Author response:
"**The authors analyze a few continual learning methods that already have sound intuition on their effectiveness, especially the OGD and GPM.**"

     Our theory applies to any method which, implicitly or explicitly, partitions the parameter space between tasks, e.g. through a linear projection (OGD), parameter freezing (PackNet), sparsity constraints, etc… So *it could in principle also be applied to Wang et al., if the parameters used to learn the sequence of tasks were also partitioned*. We do not cover all the algorithms in the parameter isolation family in our paper, because that would be unfeasible within the space limits, and it would be pointless given that new algorithms are proposed all the time. However, *the examples we propose are meant to provide an intuition of how to apply the same strategy to other methods*. In particular, **for OGD and GPM (although they are ‘intuitively sound’) the connection to parameter isolation is not so clear**, as shown by the extensive proofs backing the result. On the other hand, for ‘strict parameter isolation’ methods (Section 4.3) we do not go into the proof detail, because the link to parameter isolation is obvious. We would like to stress that in the respective papers, the authors of OGD and GPM do not establish any theoretical guarantees for the effectiveness of their methods, and that we are the first to obtain them.

Reviewer comment (Writing):
What does $\xi_{\tau}(t)$ in Theorem 2 mean?

Author response:
Regarding your question: *$\mathcal{E}\_\tau(t) $ , is the forgetting of task $\tau$ after learning task $t$* (see Section 1.1, after Equation (2)). In the specific case of Theorem 2, we identify a constraint by which $\mathcal{E}\_\tau(t)=0$ for all tasks $\tau<t$.