Paper: IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
Continual learning in the context of the used baselines is a mature field with many existing works. However, the method does provide SOTA results across the board to warrant a contribution that offers a significant performance boost.

Author response:
We are delighted to find that the reviewer recognizes the performance boost achieved by IMEX-REG across all the continual learning scenarios, especially in low buffer regimes. We further conducted several analyses of our method that emphasize on the robustness in real world scenarios (Section 5) and other characteristics such as stability-plasticity tradeoff, model calibration (Appendix C).

Reviewer comment (Novelty):
Theoretical justifications of the paper are not novel and mostly are reiterating previous results. Doing so is OK but does not offer any new theoretical contributions.

Author response:
We do not claim that the theoretical insights in our paper are novel. Rather these are provided to support our hypothesis that forms the base for our design choices. For example, Conjecture 1 provides insights that CRL and cross-entropy learning share a common hypothesis class that is essential for the proposed novel regularization strategy. This hypothesis is later validated empirically in the Experimental Results (4.2) section. We believe that providing these theoretical justifications strengthen the confidence of the readers in our hypotheses.

Reviewer comment (Experiments):
It is important to study the effect of \alph, \beta, and \lambda on the performance. How the user should tune them? A study should be offered for this purpose. If the performance is sensitive with respect to the values of these hyperparameters, then it is essential to provide a solution for selecting the optimal values.

Author response:
We perform grid search to find the optimum hyperparameter and Table 6 in Appendix shows the hyperparameters used for our experiments.  We agree with the reviewer that a comparative study on hyperparameter tuning would provide more insights into IMEX-Reg’s performance. Table 7 in the revised manuscript provides hyperparameter tuning experiment results. Due to limited computational capacity, we will include an exhaustive study in the final revision.

In Table 7 in the revised manuscript, we can see that the model is fairly robust to the choice of hyperparameters and still outperforms the next best baseline across all the hyperparameters searched. However the model shows a higher decrease in performance as $\lambda$ value increases. Higher the $\lambda$, higher the restriction on the model to preserve old knowledge, thus limiting its ability to learn new information. In such cases, we can look at the task-wise performance of the model or stability-plasticity trade-off (Appendix C.2) to find the optimum balance between preserving learnt knowledge and adapting to novel information.

Reviewer comment (Experiments):
There are other common settings to study CL using CIFAR100, e.g., using 20 tasks each with 5 classes. I think adding experiments for these settings is also helpful to demonstrate how well the method scales when there are more tasks.

Author response:
Here we provide the results for  Seq-CIFAR100 with 20 tasks and buffer size 200. 

Method| Top-1 Acc (%)
-|-
ER|8.49
DER++|10.96
CLS-ER|25.98
OCDNet|27.24
IMEX-REG|28.11
As can be seen, IMEX-REG holds its performance as compared to other baselines in even longer task sequences. In the final revision, we intend to provide results with an average of multiple random initializations.

Reviewer comment (Reproducibility):
The parameters of the shared model and classifier are not defined.

Author response:
Section 3 and Figure 1 both describe the parameters associated with shared model and classifier.  Our CL model Φθ = {f, g, g′, h} consists of a shared backbone f, a linear classifier g, an MLP classifier projection g′, and a projection head h. The classifier g represents all the classes that belong to all the tasks, and the projection head h captures the embeddings of the ℓ2-normalized representation.

Reviewer comment (Writing):
Eq.1 is not clear to me. What is the actual network for f and g? Why is h not used in Eq.1?

Author response:
Section 4.1 provides information on particulars of f and g. We use Resnet-18 as as a backbone f and an MLP classifier as g. In Equation 1, we keep the information generic as any other backbone or classifier can be used in its place.

Reviewer comment (Writing):
In the text below Eq.2, you said z = h(f(.)). However, z is not defined in Eq.2. The input and output patterns for the models f, g and h are unclear.

Author response:
We define z = h(f(.)) in the text below Equation 2. z represents any arbitrary 128-dimensional ℓ2-normalized projection from CRL head h.

Reviewer comment (Theory):
Why introduce the existing Conjecture 1? Does this theory connect with your actual design?

Author response:
We introduce Conjecture 1 to provide more intuition behind our choice of auxiliary task for multi-task learning in a CL setting. Assuming that the tasks in MTL share a common hypothesis class, sharing representations across tasks primarily benefits tasks with limited training samples. Given semantic preserving augmentations, both CRL and supervised learning capture similar discriminative features. Through Conjecture 1, we hypothesize that both CRL and cross-entropy share a common hypothesis class and sharing representations across these tasks especially benefits CL under low buffer regimes.

Reviewer comment (Theory):
Why the classifier can create the function spaces?

Author response:
Deep learning classifiers create rich function spaces by leveraging hierarchical representation, non-linear activation functions, and automatic feature learning. The layered structure allows them to capture progressively abstract features, while non-linearities enable modeling complex relationships. Automatic feature learning adapts to data intricacies, and parameter sharing fosters generalization. With end-to-end learning and substantial model capacity, deep neural networks efficiently represent intricate patterns in a data-driven manner, making them adept at handling diverse and high-dimensional datasets.

Reviewer comment (Novelty):
The proposed approach is based on the existing technology, and the overall novelty is small.

Author response:
We respectfully disagree with the reviewer's assessment that novelty is limited. As acknowledged by us both in the paper, Contrastive Representation Learning (CRL) and consistency/representation regularization using Exponential Moving Average (EMA) have been previously used in the CL literature. There are works that employ consistency regularization, such as CLS-ER [1], and both consistency regularization and CRL, such as OCDNet [2]. We compare and contrast such methods (CLS-ER and OCDNet) in Tables 1 and 2. As can be seen from our empirical evaluation, IMEX-Reg clearly outperforms these methods across different scenarios.

Our novel contribution mainly lies in leveraging desirable traits of learning in the unit hypersphere through explicit classifier regularization in the function space. To the best of our knowledge, there are no works that leverage traits of learning in the unit hypersphere in CL. To this end, we align the geometric structures within the classifier hypersphere with those of the projection head hypersphere to compensate for weak supervision under low buffer regimes. Both through empirical evaluation (Tables 1 and 2) and also through an ablation study (Table 3), we highlight the effect of our novel contribution on several benchmark datasets. In addition, Section 5 on model characteristics uncovers additional benefits emanating from our novel contribution.

Reviewer comment (Writing):
The methodology section is hard to follow. A lot of notations are not defined clearly and the proposed approach is not novel enough.

Author response:
We understand the reviewer's concern regarding the complexity of the methodology section. We would like to emphasize that the intricacies stem from the incorporation of contrastive learning, exponential moving average, and a novel regularization technique that leverages the unique attributes of learning within the unit hypersphere through explicit classifier regularization in the function space. While we acknowledge that the methodology section may seem intricate, these details are intentionally included to provide a comprehensive understanding of our approach. Additionally, we would like to express that the perceived lack of novelty might be influenced by the level of experience in the field. Our approach integrates advanced concepts, and we understand that these may be less familiar to those less experienced in the specific domain. We are open to providing additional clarifications or explanations to enhance the readability of the methodology section for a broader audience.

Reviewer comment (Novelty):
The contribution of the paper is merely incremental as using contrastive learning and regularization on function space have been extensively studied before [1, 2, 3]. Imposing a regularization on the contrastive projected outputs along with the classification outputs improves the performance is not surprising.

Author response:
We thank the reviewer for relevant references with respect to contrastive learning in CL. The references [1,2,3,8] use contrastive learning in their learning objective. As we have repeatedly mentioned in the paper, we do not claim any novelty emanating from contrastive learning objective. We are very much aware of the competing literature that employs Exponential Moving Average (EMA) and Contrastive Representation Learning (CRL) in CL. Specifically, CLS-ER [7] employs EMA while Co2L [2] employs CRL. Going one step ahead, OCDNet [6] is the closest prior art that employs both CRL and EMA.  

We respectfully disagree with the idea that imposing regularization on the contrastive projected outputs implicitly improves classification performance. Our novel contribution mainly lies in leveraging desirable traits of learning in the unit hypersphere through explicit classifier regularization in the function space. To the best of our knowledge, there are no works that leverage traits of learning in the unit hypersphere in CL. To this end, we align the geometric structures within the classifier hypersphere with those of the projection head hypersphere to compensate for weak supervision under low buffer regime.

We carefully curate the list of baselines based on their learning objectives: contrastive learning methods, EMA based methods and approaches  that use both contrastive learning and EMA. We note that these are highly cited works in CL that are published in reputed conferences. We compare and contrast such methods (CLS-ER, CO2L, OCDNet) in ‘Related Works’, and in experimental evaluation in Tables 1 and 2. We also compare with OCDNet in terms of task-recency bias, robustness against natural and adversarial corruptions in Section 5. Table 3 highlights the contribution of IMEX-Reg in Seq-CIFAR100 for a buffer size of 200. As can be seen from our experimental results and analysis, IMEX-Reg clearly outperforms these competing methods that entail EMA and CRL. This further emphasizes that our novel loss brings additional benefits not just in terms of performance but also in terms of robustness and less bias toward recent tasks.

Reviewer comment (Experiments):
The method introduces many hyper-parameters (e.g., alpha, beta, lambda in Eq.7), but there was no study of how these hyper-parameters affect the model performance.

Author response:
As IMEX-Reg entails multiple learning objectives, it introduces several hyperparameters. Table 6 provides the hyperparameters for used IMEX-Reg to reproduce the results reported in Table 1. We agree with the reviewer that a comparative study on hyperparameter tuning would provide more insights into IMEX-Reg’s performance. Table 7 in the revised manuscript provides  hyperparameter tuning experiment results. Due to limited computational capacity, we  will include an exhaustive study in the final revision.

Reviewer comment (Experiments):
The authors argue that their method has advantages over existing replay methods. However, rehearsal-free methods such as [3, 4] already significantly outperform the proposed method. For instance, [3] achieves 87.8% accuracy on Seq-CIFAR10 and 47.1% on Seq-TinyImageNet without the need to save any samples.

Author response:
The method in [3] explores possibilities of integrating the existing parameter isolation based continual learning (CL) method HAT or Sup with the strong OOD detection method CSI. As the reviewer rightly pointed out, these approaches attain accuracy higher than the existing replay-based methods. The improvement in performance can be attributed to several aspects of training: HAT and Sup entail parameter isolation during training. That inherently assumes task boundary information during training. It is a stricter constraint as task boundaries are not always accessible in real world CL scenarios. Secondly, these methods involve two stage training wherein the backbone is trained in the first stage and is kept frozen in the second stage. An additional rotation prediction and averaging is done to train a classifier in the second stage. 

As discussed in Table 2, Generalized Class-IL (GCIL) exposes the CL model to a more challenging and realistic learning scenario by using probabilistic distributions to sample data from the CIFAR100 dataset in each task. The aforementioned methods would fail in such scenarios due to the necessity of accessing task boundary information. As having the features lie on the unit-hypersphere leads to several desirable traits, we would like to re-emphasize that  our novelty lies in leveraging activation correlations in the unit-hypersphere of the CRL to guide the classifier towards generalization. Although the above methods attain higher accuracy, they do not address the problem we have at hand.

Reviewer comment (Experiments):
Based on a paper under review, [5] achieves 74+% in Seq-CIFAR10 with a buffer size of 200 in ResNet-18. The authors may compare their method with [5].

Author response:
We thank the reviewer for suggestions on the related work. We will include this in the final revision.