Paper: Primal-Dual Continual Learning: Stability and Plasticity through Lagrange Multipliers (ICLR-2024)

Reviewer comment (Theory):
The convergence of $\mathbf{\lambda}$ is highly sensitive to the setting of the forgetting tolerance $\epsilon$, the number of tasks $T$, and the hardness of the tasks, which will affect the memory partition.

Author response:
Sensitivity of $\lambda$

We believe that the sensitivity of $\lambda$ with respect to the forgetting tolerance $\epsilon$ and the hardness of the tasks is indeed a *positive* aspect of our approach. As mentioned in the general response, the regularization parameter associated to each rehearsal loss is meant to be adaptive. For instance, if the forgetting tolerance $\epsilon_k$ associated to task $k$ is relaxed, the effective difficulty of this task is reduced, decreasing its associated dual variable and leading to a smaller buffer partition. This adaptivity is the insight leveraged to partition the buffer and select samples.

Reviewer comment (Evaluation):
At every timestep, the memory partition changes. Not just the problem mentioned in the discussion exists, where the optimal partition size of a previous task grows at the current timestep. For the tasks that have a smaller size at the current timestep, it needs to reselect the samples to store, which would cause additional computation costs.

Author response:
Re-computation of buffer partition.

Indeed, the buffer partition is computed at every iteration, which creates an additional cost with respect to fixed partitions. However, the computational overhead of running the Sequential Quadratic Programming algorithm is small (see Appendix A.8). More importantly, as mentioned in the previous response, the flexibility of recomputing the buffer partition after every iteration allows us to dynamically assess the relative difficulty of the observed tasks, which is a key positive aspect of the proposed approach.

As mentioned in Section 7, there are two non-efficient ways of addressing the problem of dual variable underestimation. Namely, one can fill the empty portion of the buffer with either augmented samples from the previously selected ones or samples from the current task, whose dataset is entirely available.

Reviewer comment (Experiments):
From a practical perspective, this paper proposes three “novel aspects” compared to the standard experience replay approach commonly used in continual learning: {1} the weighing of the replayed losses relative to the loss on the current task is determined by the Lagrangian dual variables (rather than, as is currently done in continual learning, either by a hyperparameter or as a function of how many tasks have been seen so far), {2} the selection of samples to be stored in the buffer at the task level (buffer partition), and {3} the selection of samples to be stored in the buffer at the sample level. However, it seems only the impact of the last two aspects are evaluated empirically. There is no direct comparison to assess the effect of {1}. Specifically, a comparison between "standard ER" and the proposed approach without buffer partition at task level or individual sample selection would strengthen the work.

Author response:
Indeed, the effect of adaptive weights for replay losses (as opposed to fixed hyperparameters) is evaluated only indirectly, along with the buffer partition strategy. We agree with the reviewer in that adding the explicit comparison against a primal-dual method that uses the same buffer partition as ER is pertinent (we refer to it as PD w. Reservoir). We provide the preliminary results of this comparison here (CIL setting) and will add the full experiments to the revised version. The main conclusion appears to be that adaptive weights alone provide an improvement with respect to fixed ones. Indeed, in the case of SpeechCommands with a buffer size of 2000, virtually all the gains are explained by the adaptivity of the task-loss weights. However, in many cases, this does not explain the performance improvement entirely. 


| Test Accuracy  | SeqMNIST |       | SpeechCommands  |       |
|-------------|-------|-------|-------|-------|
| Buffer Size | ER   | PD w. Reservoir   | ER   | PD w. Reservoir   |
| 400         | 70.28 | 82.1 | 45.75 | 49.43 |
| 2000        | 87.02  | 88.97 |  68.54 | 72.28 |

Reviewer comment (Experiments):
It is unclear whether the comparisons to assess the effect of {2} (buffer partition) are direct. For example, in Figure 1 (and similarly in Figure 4), when comparing “PDCL” with “Reservoir”, it is not specified how the replayed losses are weighed in the “Reservoir” case. Are they weighed the same way as in “PDCL” or differently? This should be clearly described. If they are weighed differently, then Figure 1 does not isolate the effect of buffer partition.

Author response:
Regarding the baseline Experience Replay, we use the standard implementation from (Chaudhry et al, 2019). At each iteration, a batch of samples from the buffer and a batch of samples from the current task are merged. This larger batch is used to perform a forward pass and to compute the gradient of the loss with respect to $\theta$. Thus, the relative weight associated to each task loss is proportional to the number of samples of that task available in the buffer. For instance, when using a uniform partition, the relative weights are also uniform. 

Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P.K., Torr, P.H. and Ranzato, M.A., 2019. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486.

Reviewer comment (Reproducibility):
The way the paper describes the difference between task- and class-incremental learning suggests that the models are trained in these two scenarios in the same way, with differences only in evaluation at test time. Clarification is needed on whether the experiments were indeed implemented this way, as there should potentially be training differences. For instance, with task-incremental learning the model only needs to discriminate between classes from the current task, whereas with class-incremental learning it must also learn that current samples are not from classes in previous tasks. More details are needed on how the difference between task- and class-incremental learning was implemented.

Author response:
We follow the formulation of class and task incremental learning of (Boschini et al. 2022), as implemented by the continual learning library Mammoth (https://github.com/aimagelab/mammoth). In this implementation, only the evaluation procedure is affected. The reviewer is correct in that this implementation provides an approximation to the TIL setting since it does not affect the training procedure. When evaluating the model in the TIL setting, only the relevant classes are taken into account. In CIL, however, the predictor can select any of the previously observed categories. We will clarify this distinction in the revised version. We will add the citation to the mentioned reference (van de Ven et al., 2022) and clarify this in the revised version. 

Boschini et al, 2022. Class-Incremental Continual Learning into the eXtended DER-verse

Reviewer comment (Novelty):
When discussing the distinction between task- and class-incremental learning, the paper should cite the original work (van de Ven et al., 2022; https://www.nature.com/articles/s42256-022-00568-3).

Author response:
We follow the formulation of class and task incremental learning of (Boschini et al. 2022), as implemented by the continual learning library Mammoth (https://github.com/aimagelab/mammoth). In this implementation, only the evaluation procedure is affected. The reviewer is correct in that this implementation provides an approximation to the TIL setting since it does not affect the training procedure. When evaluating the model in the TIL setting, only the relevant classes are taken into account. In CIL, however, the predictor can select any of the previously observed categories. We will clarify this distinction in the revised version. We will add the citation to the mentioned reference (van de Ven et al., 2022) and clarify this in the revised version. 

Boschini et al, 2022. Class-Incremental Continual Learning into the eXtended DER-verse

Reviewer comment (Presentation):
Minor issue – at the top of p9, a reference is made to Figure 9, but it appears Figure 4 might be intended.

Author response:
Thank you very much for pointing these out. We will correct the formatting issues and appendix references in the revised version.

Reviewer comment (Presentation):
Minor issue – in the reference list, the paper Buzzega et al. (2020) is included twice.

Author response:
Thank you very much for pointing these out. We will correct the formatting issues and appendix references in the revised version.

Reviewer comment (Presentation):
Minor issue – for several papers in the reference list, no venue is included (e.g., Gentile et al., 2022, among others).

Author response:
Thank you very much for pointing these out. We will correct the formatting issues and appendix references in the revised version.

Reviewer comment (Presentation):
Minor issue – there are several formatting issues with in-text citations in the Appendix.

Author response:
Thank you very much for pointing these out. We will correct the formatting issues and appendix references in the revised version.

Reviewer comment (Theory):
The paper defers the discussion on the *strong* concavity constant $c$ to Appendix A.5. However, I think this hides several important dependencies. For example, (1) It intrinsically assumes the usage of (might be a large amount of) weight decay to induce strong concavity of the objective function; (2) The loss function should be $G$-smooth, and the sub-optimality bound in Theorem 4.2 turns out to depend quadratically on $G$; (3) The constraint Jacobian (question: what is it exactly?) must be full rank, and the sub-optimality bound depends quadratically on the inverse of the minimum singular value of this matrix, which can be arbitrarily large. For these, I think the paper should be more clear and honest on several hidden dependencies.

Author response:
- On the estimation of dual variables.

Since we only have access to samples (not probability distributions) and we optimize over model parameters (not functions), in practice, we solve the empirical problem $(\hat{D}_t)$ instead of $P_t$. This corresponds to analyzing estimation and approximation errors of constrained learning, as done by e.g., (Vapnik 1999) or (Shai-Shalev Schwarz 2014) for supervised learning. For constrained learning, these errors were characterized, to a certain extent, in (Chamon et al. 2023). Theorem 4.2 bridges the gap between the optimal dual variables of $(P_t)$ and $(\hat{D}_t)$.

For the distance between these dual variables to be bounded, the dual function $g_u$ can not be flat. Observe that $g_u$ is the minimum of a family of affine functions; thus, it is concave, irrespective of whether the primal problem is convex. For it to be strongly-concave, we require three extra assumptions. Namely, strong convexity and smoothness of the loss and full-rankness of the constraint Jacobian. 

The first two (strong convexity and smoothness) are widely used to prove convergence results for gradient descent algorithms even in the convex setting and are satisfied by typical losses used in ML when, e.g., the hypothesis class $\mathcal{F}$ is compact. Note that we require convexity of the objective with respect to the functionals, but not model parameters, which holds for both mean squared error and cross-entropy loss. 

Although harder to guarantee, Assumptions 3.7 is a constraint qualification called LICQ (Linear Independence Constraint Qualification), which is ubiquitous in constrained optimization (Bertsekas 1995). The independence of the columns of $D_f \ell (f^*)$ reflects that, at the optimum, the constraints are not redundant and that they contribute in different ways to reach the optimal point. Although this is probably the stricter assumption, we believe that it can be lifted, being replaced by a stronger version of strict feasibility, but this is not immediate. With these assumptions, we can guarantee that $g_u$ is not linear, and that the optimal dual variables of $(P_t)$ and $(\hat{D}_t)$ are not too far away. 

Although improvements to this bound could be obtained, we leave them for future work. We believe that the current version of the bound theoretically grounds the use of dual variables as sensitivity indicators.

Reviewer comment (Theory):
Is $P^{\star}_t (\cdot)$ a convex function? I think this should be clarified in order to use the notion of sub-differential.

Author response:
- Regarding Theorem 3.2/5.1 and the sub-differential $\partial P^*_t(\epsilon_k)$. 

In this setting, the perturbation function $\partial P^{\star}\_t(\epsilon_k)$ is a convex function of the constraint upper bounds $\epsilon$. This is a well-known result from convex optimization (see e.g., (Bonnans & Shapiro, 1998) or (Rockafellar, 1997, Theorem 29.1)), and we agree with the reviewer in that it is pertinent to mention it before Theorem 3.2. We will also update the notation of the subdifferential to $P^*_{\epsilon_k}(\epsilon)$ as suggested to avoid confusion.

Reviewer comment (Writing):
I think the notation “$\partial P^{\star}_t (\epsilon_k)$” is quite confusing. I would like to suggest the notation like “$\partial_{\epsilon_k} P^{\star}_t (\epsilon)$” where $\epsilon = (\epsilon_1, …, \epsilon_t)$.

Author response:
- Regarding Theorem 3.2/5.1 and the sub-differential $\partial P^*_t(\epsilon_k)$. 

In this setting, the perturbation function $\partial P^{\star}\_t(\epsilon_k)$ is a convex function of the constraint upper bounds $\epsilon$. This is a well-known result from convex optimization (see e.g., (Bonnans & Shapiro, 1998) or (Rockafellar, 1997, Theorem 29.1)), and we agree with the reviewer in that it is pertinent to mention it before Theorem 3.2. We will also update the notation of the subdifferential to $P^*_{\epsilon_k}(\epsilon)$ as suggested to avoid confusion.

Reviewer comment (Theory):
In a higher level of discussion, does the paper ever require such a **local** sensitivity result to give a motivation?

Author response:
One could motivate the buffer partition approach without this sensitivity result. Note that dual variables accumulate the slacks (task-level constraint violations) over the continual learning procedure. Thus, it is not surprising that they carry information about task difficulty and that they can be used to manage the buffer. However, Theorem 3.2 provides a complete description of the induced task difficulty metric. Theorem 3.2 tells us that dual variables capture the minimum rate of degradation of the performance on the current task, when tightening the forgetting requirement in a previous task. This translates into a principled measure of how hard maintaining the performance in a past task is, relative to other observed tasks. Thus, we believe that Theorem 3.2 is a key component of this paper. We also provide an experimental interpretation of Proposition 5.1 (see Figure 3.b).

Reviewer comment (Theory):
Around Assumption 2.3, it would be great if the authors put some citations on universal approximation results for neural networks, which explains (with examples) the richness of (modern) machine learning model parametrization.

Author response:
- Other comments 

We agree with the reviewer regarding the lack of comments on how the proof of Theorem 3.2 relates to the proof of Proposition 5.1 (i.e., sample-level sensitivity). These modifications are very minor since the only difference between Theorem 3.2 (task-level sensitivity) and Proposition 5.1 (sample-level sensitivity) is the dimensionality of the input to $P^*_t(\epsilon)$. We will add these comments to the revised version. We will also fix the typos mentioned in the revised version and add citations on universal approximation results for neural networks (e.g., Hornik 1989).

Hornik, K., Stinchcombe, M. and White, H., 1989. Multilayer feedforward networks are universal approximators. Neural networks, 2(5), pp.359-366.

Reviewer comment (Theory):
Despite the theoretical result, the proposed algorithm does not fit the online continual learning scenario because the process "fill buffer" is done after visiting samples in line 11 of Algorithm 1.

Author response:
- On the availability of samples during training. 

The notation used could have led to some confusion regarding the availability of samples during the execution of the Primal-Dual iterations and buffer partition. As explained in section 2, at iteration $t$, the following are available:

1. Samples associated to task $t$.

2. Samples from tasks $1, \dots, t-1$ that where stored in the buffer: $\mathcal{B}\_t = \cup_{k=1}^{t-1} \mathcal{B}\_t^k$  where  $\mathcal{B}_t^k$ denotes the subset of the buffer allocated to task $k$ at iteration $t$.

This is the standard memory-based continual learning setting [1], where one can store a subset of the samples from previous tasks, revisiting them at future iterations. We will clarify this in the algorithm.

[1] GM van de Ven, Three types of incremental learning, 2022.

Reviewer comment (Theory):
It seems that this contradiction occurs because Algorithm 1 needs to access the information of $\lambda_k$ at the end of each task to compute the optimal buffer size. However, we should have at least the upper bound of the buffer size for the current task $k$ to save encountering samples in the online stream.

Author response:
- On the computation of $\mathbf{\lambda}$ at each iteration.

Indeed, at each iteration $t$, we undertake the dual problem $\hat{D}_t$, from which we obtain a vector of dual variables $\mathbf{\lambda}$. This means that we assess the relative difficulty of the observed tasks and compute a new buffer partition at each iteration. However, we do not require access to the entire dataset of a previous task to compute this partition. Instead, it is estimated using the set of samples $\mathcal{B}_t$ stored in the buffer. As mentioned in section 7, this can give rise to the issue of dual variable underestimation.

Reviewer comment (Theory):
The loss landscape on the parameter $\theta$ is non-convex, as the authors stated in Section 3. The local-optimal setting for a given local minimal point and the Lagrange multiplier do not guarantee remarkable performance in the empirical result. The existing heuristic methods based on constrained optimization, such as A-GEM, have already shown remarkable performance in more complex benchmarks, such as split-CIFAR100 and split-MiniImagenet.

Author response:
- On the sub-optimality of $\theta^*$.

Indeed, the loss landscape on the parameter $\theta$ is non-convex, creating a duality gap. In other words, the value of the dual problem $\hat{D}^{\star}_t$ need not be a good approximation of the value of the primal problem $P^\star_t$. However, as mentioned in section 3, this was tackled in (Chamon et al, 2022, Theorem 1):

$$ | P^\star_t - \hat{D}^{\star}_t | \leq (M\nu+\zeta)(1+ \Delta ) \text{,}$$ where $\Delta= \text{max} ( \Vert \tilde{\lambda}^* \Vert, \Vert \hat{\lambda^*} \Vert )$ and $\zeta = \text{max} \zeta_i(N_i, \delta)$ (see section 4.2 for the definition of $\zeta_i$ ).

Constrained Learning with Non-Convex Losses, 2022. Luiz F. O. Chamon, Santiago Paternain, Miguel Calvo-Fullana, and Alejandro Ribeiro.

Reviewer comment (Evaluation):
The reported metric is not standard in continual learning. Can the authors report the experimental result in terms of the average test accuracy and FWT?

Author response:
- On the reported metric.

We report the Top-1 Error, which is simple 100-Average Test Accuracy. We report this both in the Task Incremental Setup and Class Incremental Setup, both of which are standard in the continual learning literature. For completeness, we will add accuracy to the revised version in the Additional Experiments Appendix.

Reviewer comment (Experiments):
The constrained optimization-based CL baselines, such as GEM and A-GEM, should be included in the experiment section to analyze the novelty of the proposed method. Is there any reason why the authors do not contain these algorithms?

Author response:
- On the baselines. 

We agree with the reviewer that A-GEM is a relevant baseline. We attach the results of A-GEM on SpeechCommands here and will include them on the rest of the datasets in the camera-ready version. A-GEM appears to perform worse than PDCL on the Task Incremental setting (~2%), and slightly better on the Class Incremental setting.

|             | A-GEM |       | PDCL  |       |
|-------------|-------|-------|-------|-------|
| Buffer Size | TIL   | CIL   | TIL   | CIL   |
| 400         | 90.07 | 55.75 | 92.94 | 54.73 |
| 2000        | 92.5  | 72.54 | 94.79 | 72.36 |