Paper: Continual Learning with Orthogonal Weights and Knowledge Transfer (ICLR-2024)

Reviewer comment (Theory):
Problem (2) in the introduction section is questionable for OG. It is unclear what are the properties of important weight and important gradients, which the authors do not provide sufficient explanation. The authors also do not clearly explain the problem of overconsumption of network capacity, and why the performance drop in Figure 10 comes from this reason.

Author response:
The important weights and gradients are the weights and gradients with larger weights and gradients that determine the accuracy of a learning task. We clarified the concepts and provided the explanation for Figure 10 (corresponding to Figure 6 in the revised paper). Please see our resubmitted/revised paper.

Reviewer comment (Presentation):
I do not think it is a good practice to organize the paper such that important information such as pseudocode for the main algorithm is deferred to the appendix, especially in section 4 and 5. This problem is severe enough such that the text is very difficult to follow without checking the reference. Key experimental results are not shown in the main text. This essentially gains unfair advantage to exceed the page limit.

Author response:
Many thanks for your valuable comments! In our revised paper, we have put some important content into the main text. For example, Figure 1 (a)-(b) (Sec.4), Theorem 5 (Sec.3.2), ablation experiments of task similarity metric SDM (embedded in Table 3), etc., and have discussed them more to facilitate understanding.

Reviewer comment (Writing):
In the paragraph following Eq.1, what is the notation “/” mean? What are $W_t/W_{t-1}$ and $t/t-1$?

Author response:
The notation "/" means 'or'. Thus $W_t$/$W_{t-1}$ and $t$/$t-1$ represent that task $t$ or $t-1$ corresponds to its OIWS' weights $W_t$ or $W_{t-1}$ of the model.

Reviewer comment (Experiments):
Is there a way to measure how close the weights are orthogonal to each other? The ablation experiments do not indicate whether using OIWS indeed improves the performance by achieving orthogonality.

Author response:
Let the OWIS-based weights of the learning task $t-1$ and $t$ be $W_{t-1}$ and $W_t$ respectively, then the dot product of $W_{t-1}$ and $W_t$ can represent the degree of orthogonality between them. If they are orthogonal to each other, the dot product value is zero; the smaller the positive value is, the closer they are; the larger the negative value is, the greater the negative correlation between them. In our ablation experiments (see Table 3), it is clearly shown that without deploying the OIWS-based TIL strategy in STIL (denoted by STIL(-OIWS)), on all test datasets, the accuracy (ACC\%) of STIL decreases, which indicates that some forgetting has occurred. Therefore, the OIWS-based strategy plays an important role in STIL's resistance to forgetting.

Reviewer comment (Theory):
There seems to be confusion about some fundamental concepts and pitfalls with the proposed approach. Specifically, for each task, the model lies in some (sub)space. In the literature, the orthogonal projection method is proposed to update the model weights and learn the model for each task accordingly. This paper instead proposes "gradient space," which does not make much sense. Gradient descent is used to update weights in the model space, so it is unclear what "gradient space" refers to. The gradient in each step is computed using input data and is stochastic; therefore, would gradient space be random?

Author response:
**Q1**: What does gradient space refer to?

**Response**: Given a DNN model, we define the space of the input data as the Input Space denoted by $\mathbf{S}_{in}$, and the space of the gradient span as the Gradient Space $\mathbf{S}_g$. Please refer to the gradient space definition shown in paragraph 2, Section 3.1.

**Q2**: The gradient in each step is computed using input data and is stochastic; so would gradient space would be random?

**Response**: In the SGD algorithm used in deep learning, ``random'' means that it **randomly extract a batch of samples** from the training dataset of a learning task for model training. Since the data from the training dataset for each task are i.i.d (independently and identical distributed), their gradient space would not be random but come from the same gradient space.

Reviewer comment (Theory):
The paper claims that the proposed weight orthogonal method can eliminate catastrophic forgetting completely, which appears overly ambitious. For instance, when backward knowledge transfer takes place, it would change the weights for previous tasks. It is unclear how to ensure weight orthogonality is maintained in such cases.

Author response:
**Q3**: ... How to ensure the weight orthogonality all the way?

**Response**: Yes, you are right. However, in the backward knowledge transfer, the proposed STIL adopts the constraints in Eq.(8) to carry out both forward/backward knowledge transfer and forgetting prevention. Although we cannot guarantee zero forgetting in backward knowledge transfer, the constraint tries to maximally reduce it.

Reviewer comment (Theory):
For Theorem 1, it is not clear why prior OG-based methods cannot guarantee zero catastrophic forgetting. What if the modified weights spanned by the orthogonal gradient to the previously learned tasks happen on the whole weight space, instead of its own weight subspace?

Author response:
**Q4**: For Theorem 1, it’s not clear ...

**Response**: Section 3.1 has shown that the gradient space and the weight space of the model are two different spaces, although they are related. Therefore, modifying the weights in a gradient orthogonal manner does not guarantee that the modified weights of the current learning task are orthogonal to the weights of the previously learned task, as proven in Theorem 1. Only the weight orthogonality can ensure no CF.

Reviewer comment (Evaluation):
The proposed STIL seems complex, as it has two steps for learning each new task and four steps to calculate the similarity between tasks. The memory and time cost need to be justified. For example, since the proposed STIL adopts the GPM method to obtain knowledge bases with additional computation and memory cost (e.g., data replay), why can the proposed STIL have less time and memory cost compared to GPM for some datasets (e.g., CIFAR100 Sup, 5-dataset as shown in Table 6)? How do the authors define memory in this work? Does it include both stored bases and replay data?

Author response:
**Q5**: ... why the proposed STIL can have less time and memory compared to GPM for some datasets? ...

**Response**: The reason why time and space are better than GPM on the datasets you mentioned above is due to the optimization we made in the implementation of the system. GPM loads in all its stored information during its computation, while our STIL loads only the information needed for computation on demand. In addition, during model training, we added the processing of early stop of model convergence, that is, once the convergence of model training/learning is determined, the training is immediately ended, unlike GPM, which always needs to be iteratively completed, which have been explained in our revised Appendix D.2.

Reviewer comment (Evaluation):
The experimental results compared to CUBER need to be justified. As shown in Table 1, compared to CUBER, STIL shows better accuracy but higher forgetting, which is inconsistent with the claims.

Author response:
**Q6**: The experimental results compared to CUBER need to be justified...

**Response**: Although CUBER achieves positive BWT with its good BKT on some datasets, it is not difficult to see that its BWT oscillates between positive and negative values. And more importantly, when learning a new task, not when backward knowledge transfer, its CF prevention and FKT were inferior to our STIL, which results in its performance was not as good as ours in the end. Please see the detailed experimental results (tasks 3-5's ACC performance) shown in Figure 8 in Appendix.

Reviewer comment (Experiments):
The proposed method seems to only be applicable to task-incremental learning. It is unclear how it would work for class-incremental learning.

Author response:
**Q7**: The proposed method seems only can be used to Task-incremental learning. How about class-incremental learning?

**Response**: We believe that the proposed method STIL can be applied to class-incremental learning (CIL) in those CIL methods that use a task-incremental learning method to fully overcome forgetting and a task label prediction to identify the correct task for each test sample.

Reviewer comment (Writing):
Regarding the definition of f(slashed zero, t): 1) The function f(slashed zero, t) is defined as “when learning t, it does not use the knowledge of any task.” Does “any task” mean any prior tasks? 2) How do the authors calculate the accuracy of f(slashed zero, t) as shown in Eq(4)?

Author response:
**Q8**: The definition of $f (\emptyset, t)$: 1) the function $f (\emptyset, t)$ is defined as "when learning t, it does not use the knowledge of any task". Does the "any task" mean any prior tasks? 2) How do the authors calculate the accuracy of $f (\emptyset, t)$ as shown in Eq.(4).

**Response**: 1) "any task" means any prior similar task. 2) For the $f (\emptyset, t)$ calculation in Eq.(4), we built a separate model for the task using a duplicated network with randomized initial parameters and no training, which clearly has no knowledge transfer and no forgetting involved. Please refer to footnote 3 in this paper.

Reviewer comment (Experiments):
When discussing task similarities in Section 4.2, the authors consider top-k (i.e., k = 2) prior tasks. Will the k values affect performance for different datasets?

Author response:
**Q9**: When discussing task similarities in Section 4.2, the authors consider top-k (i.e. k = 2) prior tasks. Will the $k$ values affect the performance for different datasets?

**Response**: Our experimental results show that $k =2$ is good for all datasets. That is why we used the same $k=2$ in all our experiments.

Reviewer comment (Presentation):
Some of the formulas lack accompanying figures and textual explanations.

Author response:
Thanks for your valuable comment! We have followed your suggestion and incorporated some crucial parts of the appendix, e.g., some figures and textual explanations, into the main body of our revised paper. Please see Figure 1 (a) - (b) and paragraph 1 of Sec. 4 in the revised paper.

Reviewer comment (Presentation):
Certain formula derivations and images from the supplementary materials are essential and should be incorporated into the main body of the text.

Author response:
Thanks for your valuable comment! We have followed your suggestion and incorporated some crucial parts of the appendix, e.g., some figures and textual explanations, into the main body of our revised paper. Please see Figure 1 (a) - (b) and paragraph 1 of Sec. 4 in the revised paper.

Reviewer comment (Theory):
In Eq.(9), two constraint conditions are imposed. Can convergence be achieved during the optimization process, or will it increase the optimization difficulty, potentially leading to optimization failure?

Author response:
As discussed in the second paragraph below Eq.(9) (corresponding to Eq.(8) of the revised paper), to avoid extensive computations in the constrained optimization and non-convergence in Eq.(8), considering the orthogonality of these two constraints and the contrastive loss, we convert Eq.(8) with the constraints as a new bi-objective loss Eq.(9), which eliminates the above issues that you are concerned about.

Reviewer comment (Theory):
This paper introduces the method of orthogonal weights to mitigate catastrophic forgetting, but why does forgetting still occur, and what is the underlying reason?

Author response:
If the proposed method STIL only deals with forgetting without knowledge transfer (KT), then the orthogonal weights introduced in this paper can eliminate CF. However, as our STIL also carries out forward and backward KT operations, the backward KT of STIL would modify the weights of previous similar tasks resulting in slightly negative BWT on some datasets. Overcoming this issue is a further research direction.

Reviewer comment (Presentation):
Some crucial parts of the appendix should be incorporated into the main body of the text, such as Figure 2. Furthermore, it needs to be appropriately formatted to avoid potential misinterpretation.

Author response:
Thanks for your valuable comment! We have followed your suggestion and incorporated some crucial parts of the appendix into the main body in our revised paper. For example, we have put some important content that was originally in the appendix: Figure 1 (a)-(b) (Sec. 4), Theorem 5 (Sec. 3.2), ablation experiments of task similarity metric SDM (embedded in Table 3), etc., into the main body of the paper.

Reviewer comment (Presentation):
The proposed Self-Adaptive Task Incremental Learning (STIL) should be placed before the introduction of weight orthogonality in the paper.

Author response:
Since the proposed method STIL in this paper is based on the concept of Orthogonal Important Weight Subspace (OIWS), it is difficult to explain/discuss our STIL method without introducing the concept of OIWS first. That's why our paper content is organized this way.

Reviewer comment (Theory):
Theorem 1 seems a trivial proposition, which states that the weight vector can be spanned by the gradient vectors and the weight subspaces of each task are orthogonal. However, it cannot guarantee to explain the property of catastrophic forgetting in the final logit layer of DNN, which is directly connected to the output prediction.

Author response:
**Response**: The proposed method STIL is the method of updating/learning the neural network weights from the first layer of the network to the last logical layer with the following Theorem 1 (see Eq.(7). The property of overcoming CF in the last logit layer of DNN in Theorem 1 has been verified by the experimental results in Tables 1-2.

Reviewer comment (Theory):
The proposed $\mathcal{L}_{sim}$ is based on the constrained optimization, which cannot be applied to the nonconvex domain (most deep learning problems) directly. In addition, the orthogonality of weight matrix cannot explain the orthogonality of the hidden layers with non-linear activation, so the author should provide more empirical evidence why this proposed loss facilitates knowledge transfer.

Author response:
**Response**: So far, there is no evidence that the proposed $\mathcal{L}_{sim}$ cannot be applied to the nonconvex domain (...) directly. The neural network architecture with nonlinear activation layer (ReLU) used in our experiments, such as AlexNet and ResNet-18, all work well with our proposed method STIL. Unlike the existing methods, this paper provides a total of 11 various CL task sequences to fully evaluate the effectiveness of our method. Please see the experiment results in Tables 1-2 and Table 11 in Appendix D.4.

Reviewer comment (Novelty):
The similarity detection based algorithm seems not novel enough to explain the task distribution distance by measuring on the online data stream. The threshold $\theta$ is too heuristic without any empirical explanation.

Author response:
**Response**: We proposed the task similarity detection method based on the task distribution Wasserstein distance for the first time, and the experimental results (see Table 3) show that this method is effective and accurate. For the theoretical basis and empirical explanation of the value of threshold $\theta$, refer to its explanation in Eq. (11). For its experimental values in different datasets, refer to Table 4 in Appendix B.

Reviewer comment (Novelty):
The paper appears to be a simple combination of the existing gradient projection memory method and the heuristic similarity-based update rule.

Author response:
**Response**: There is misunderstanding. We aim to to overcome the three main weaknesses of the existing OG-based methods. This paper gives a novel and effective solution, please refer to Section 1 Introduction. Specifically, we first theoretically showed that the existing OG-based methods can only mitigates CF but cannot eliminate CF (Theorem 1). Then, we introduce the new concept OIWS and propose an OIWS-based TIL strategy, which can solve the problems (1) and (2) (see Theorems 1-3, Lemma 1, Table 1 and Figure 6 in Appendix). Based our theoretical research for KT, we give the bounds that can lead to negative FKT and BKT. Moreover, we propose a new task similarity detection method based on only direct reuse of the knowledge already learned in the current network and a new cross-entropy loss with constraints along with its optimal solution, which efficiently and effectively solve the problem (3) (see Table 2).

Reviewer comment (Theory):
Although updating schemes among orthogonal weight subspaces can be an interesting work, this paper does not theoretically or empirically provide what happens in the non-linear activation layer.

Author response:
**Response**: The proposed OIWS-based method works only on each parameter or weight layer of the neural network (see Eq. (7)), and the gradient calculation and updating of the nonlinear activation is taken care of by optimizer SGD of the deep machine learning platform.

Reviewer comment (Evaluation):
The proposed method should have achieved the best performance in both metrics, test accuracy and BWT, simultaneously if the proposed pipeline successfully works to handle catastrophic forgetting and knowledge transfer simultaneously.

Author response:
**Response**: The ACC performance of a TIL method is determined by its abilities to overcome forgetting, FKT, and BKT. The proposed STIL method is superior to existing strong baselines in resisting forgetting, FKT, and BKT. Please refer to the experimental results shown in Tables 1-2.

Reviewer comment (Evaluation):
The method seems to be an extension of GPM, but the discussion on the comparison in the experimental section is not enough. For example, why STIL without OIWS slightly performs better than GPM is not addressed.

Author response:
**Response**: In our revised paper, we have highlighted the main difference between our STIL and the method GPM in the resubmitted/revised paper (see paragraph 2 in Sec. 5.2). Briefly, as GPM has no explicit KT mechanism or OIWS-based strategy for CF prevention,  our STIL achieves a better performance than GPM.