Paper: YoooP: You Only Optimize One Prototype per Class for Non-Exemplar Incremental Learning (ICLR-2024)

Reviewer comment (Presentation):
Overall, I find the title may be unsuitable for the current manuscript. The proposed method utilizes prototypes to construct the training loss, while the embedding and classifiers are always optimizable throughout the learning process. Hence, not only “prototypes” are optimized, but also the backbone and classifiers are optimized for the current task.

Author response:
**A1:** Thanks for your suggestion! We will change the title to
"OPT: Optimizing Prototypes for Non-Exemplar Class-Incremental Learning" if the reviewer thinks it is appropriate.

Reviewer comment (Presentation):
Some illustrations are unclear. For example, there is no X coordinate in Figure 1, making it hard to figure out the memory usage of different methods. Corresponding explanations are also needed to show why other methods like PASS and FeTrIL require a much larger memory size. These methods also save the class prototype to generate instances, and the memory gap between them should be illustrated.

Author response:
**A2:** We have added the X coordinate for Figure 1 and also explained the "memory footprint" in the caption in our revised manuscript.

Additionally, we have explained the memory gap between our method and existing studies in the revised version. As described in the caption of Figure 1 and detailed in Appendix D, the term "memory size" denotes the memory footprint of our method and other approaches when computing prototypes for each task during training. Unlike other prototype-based methods that necessitate saving the embeddings of the entire class to generate the class mean prototype, YoooP generates optimized prototypes with a smaller memory footprint based on a mini-batch attentional mean-shift algorithm. That is why our approach requires a smaller memory footprint than the existing methods.

Reviewer comment (Presentation):
The experimental results should be reorganized. The current results in the main paper only focus on the Base-0 setting, while I also see the results in supplementary that some Base-50 results are also available. It would be better to reorganize these results to contain both settings since Base 50 is also common in today’s CIL.

Author response:
**A3:** Per your suggestion, we have moved the experimental results under half-base 5 phases and half-base 10 phases settings to the main paper.

Reviewer comment (Experiments):
It requires further experimental analysis, e.g., the influence of hyper-parameter gamma in Eq. 11.

Author response:
**A6:** We have explored the impact of hyper-parameter $\gamma$ on CIFAR-100 with zero-base 10 phases setting, as illustrated in the following table. It can be observed that the hyper-parameter $\gamma$ is not very sensitive, and increasing the weight of knowledge distillation within a certain range can be beneficial for YoooP/YoooP+. We can see that weight values between 25 to 40 result in excellent performance.

Table 2: Influence of hyper-parameter $\gamma$ on model performance
| $\gamma$ | 15 | 20 | 25 | 30 | 35 | 40 |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| YoooP+ (Avg) | 60.69\% | 60.44\% | 61.18\% | 61.93\% | 61.02\% | 61.30\% |
| YoooP+ (last)| 41.21\% | 42.17\% | 44.48\% | 47.17\% | 45.79\% | 46.51\% |
|   |      |      |        |       |       |     |

Reviewer comment (Novelty):
The method is incremental as some prototype-based methods already exist. The paper made some improvements. But the improvements can also be obtained by contrastive learning as the proposed idea is very similar to that of contrastive learning.

Author response:
**A1:** Our approaches bring substantial advancements over existing prototype-based methods. Unlike existing prototype-based approaches that only calculate the **mean prototype** for each class, we propose a novel attentional mean-shift algorithm for optimizing the prototype per class. The proposed method helps to learn a more compact and informative prototype. To our best knowledge, we are the first to study prototype optimization in CIL. Moreover, we develop a novel prototype augmentation technique based on a high-dimensional rotation matrix to generate high-quality synthetic data. In contrast, existing studies only add Gaussian noise into saved prototypes for data generation. The evaluation results demonstrated our methods can significantly outperform the baselines on multiple datasets under different settings. Thus, other reviewers, such as Reviewer #ZsAp, gcwm, and EwLK, recognize the novelty of our proposed methods.

In addition, we would like to highlight the distinction between our method and contrastive learning. Contrastive learning is known as self-supervised learning for pre-training. It aims to learn the representations of input data through the comparison of sample pairs, rather than learning a signal from individual data samples sequentially. This comparison involves both positive pairs (comprising "similar" inputs) and negative pairs (involving "dissimilar" inputs) [1]. In contrast, our methods introduce a novel attentional mean-shift algorithm for optimizing the prototype for each class. We then apply contrastive loss to further enhance the representativeness of each class's prototype. These two components ensure that the learned prototypes are more effectively representative. Note that the optimized prototypes can not be directly obtained through contrastive learning. Rather, the proposed attentional mean-shift algorithm plays a critical role here, though the contrastive loss is helpful.

**References**

[1] Contrastive representation learning: A framework and review. (Ieee Access 2020)

Reviewer comment (Theory):
The paper didn’t say where the forgetting occurs. My understanding is that the technique simply learns and saves the prototype for each class. Each task is learned independently. Then there is no forgetting.

Author response:
**A2:** As mentioned in our manuscript, catastrophic forgetting [1] refers to deep neural networks forgetting the acquired knowledge from the previous tasks disastrously while learning the current task. In other words, forgetting will occur when neural networks learn new tasks in sequence. In the context of Class-Incremental Learning (CIL), each task is learned independently and sequentially using **a single shared neural network**. Since all the tasks share the same network, learning a new task can overwrite the network weights learned for the previous tasks [2]. This leads to forgetting, i.e., accuracy drop on previous tasks. 

**References**

[1] Catastrophic interference in connectionist networks: The sequential learning problem. (Psychology of learning and motivation 1989, p109-165)

[2] Prototype Augmentation and Self-Supervision for Incremental Learning. (CVPR2021)

Reviewer comment (Reproducibility):
The inference procedure is not discussed. If the algorithm saves only the class prototype for each class, how does it get the feature representation for each test sample for cosine similarity computation in prediction?

Author response:
**A3:** In Class-Incremental Learning (CIL), the inference for each task is performed similarly to a standard classification task. Specifically, as depicted in Figure 2, we use the backbone, denoted as $F(;\theta)$, to extract embeddings from the test samples. These embeddings are then fed into the classifier $G(;w)$ to estimate the probability distribution over all classes. Conversely, the saved prototypes are merely utilized during the model training phase to avoid forgetting.

Reviewer comment (Evaluation):
The average accuracy in (Rebuffi et al. 2017) is average incremental accuracy (AIC). Based on AIC, your reported accuracy results are low. Please also report the last accuracy, i.e., after learning the last task and compare with the above methods.

Author response:
**A4:** Thanks for your suggestion! Our average accuracy means the average incremental accuracy (AIA) in our submission! Per your suggestion, we already updated the name of this metric in the revised manuscript.


Note that our proposed models are **non-exemplar CIL methods** that do not store real samples for replay but only store prototypes, while the iCaRL is an **exemplar-based method** that needs to store and replay real samples. Thus, iCaRL requires a much larger memory budget than our methods. Moreover, prior studies [3][4][5] have pointed out that exemplar-based CIL may suffer from extra risk of privacy leakage and large memory costs. In reality, non-exemplar CIL is much more challenging than exemplar CIL as mentioned by the Reviewer gcwm. Therefore, a direct comparison between our method and iCaRL seems unfair.

Nevertheless, we already compared our methods with iCaRL [1] on the TinyImageNet dataset under different settings in Figure 10 in Appendix F. We report the last accuracy in the following table. It can be observed that our methods still outperform the iCaRL in terms of average incremental accuracy.

Table 1: Evaluation of TinyImageNet.
|  TinyImageNet  | zero-5 last | zero-10 last | Half-5 last | Half-10 last |
|  ----  | ----  | ---- | ---- | ---- |
| * YoooP | 43.93\% | 37.67\% | 54.67\% | 45.22\% |
| * YoooP+ | **47.44**\% | **44.06\%** | **57.02\%** | **49.97\%** |
| iCaRL-CNN | 34.65\% | 21.64\% | 23.39\% | 16.04\% |
| iCaRL-NME | 44.33\% | 31.96\% | 33.10\% | 25.27\% |
| BiC | 47.41\% | 18.22\% | 36.41\% | 25.98\% |
| WA | 47.96\% | 30.94\% | 38.45\% | 29.07\% |
|      |         |      |      |       |

**References**

[1] Incremental Classifier and Representation Learning. (CVPR2017)

[2] Feature Translation for Exemplar-Free Class-Incremental Learning. (WACV2023)

[4] Prototype Augmentation and Self-Supervision for Incremental Learning. (CVPR 2021)

[5] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. (CVPR 2022)

[6] Feature Translation for Exemplar-Free Class-Incremental Learning. (WACV 2023)

Reviewer comment (Experiments):
The experimental datasets and baselines are too few. More baselines with or without using exemplars should be compared as saving some data is not an issue. [a] is a non-exemplar based method. 
(1) Kim et al. (2022). A theoretical study on solving continual learning. NeurIPS.  
(2) Wang et al. (2022). Beef: Bi-compatible class-incremental learning via energy-based expansion and fusion. ICLR.  
(3) Wu et al. (2019) Large scale incremental learning. CVPR.  
(4) Buzzega et al. (2020) Dark experience for general continual learning: a strong, simple baseline. NeurIPS.

Author response:
**A5:** Per your suggestion, we added a new experiment evaluating our proposed methods on another large ImageNet-100 dataset. The comparison results are illustrated in the table below. Our methods (YoooP and YoooP+) still significantly outperform the baselines in terms of average incremental accuracy on the new ImageNet-100 dataset.

Table 2: Evaluation of ImageNet-100.
|  ImageNet-100  | Half-10 Avg | Half-10 last |
|  ----  | ----  | ---- |
| * YoooP | 75.97\% | 68.96\% |
| * YoooP+ | 77.64\% | 70.54\% |
| * PASS | 64.3\% | 53.54\% |
| * FeTril | 72.55\% | 64.06\% |
| * SSRE | 60.23\% | 51.20\% |
| * LWF | 47.35\% | 30.53\% |
| iCaRL-CNN | 48.46\% | 36.29\% |
| iCaRL-NME | 59.04\% | 48.69\% |
| BiC | 50.69\% | 31.06\% |
| WA | 61.04\% | 46.24\% |
|     |       |

Regarding more baselines, the proposed models are **non-exemplar CIL methods** that do not store real samples for replay but only store prototypes, while the baselines you suggested are **exemplar-based methods** that need to store and replay real samples. Thus, comparing non-exemplar methods with the latest exemplar methods could be deemed unfair. Moreover, prior studies [1][2][3] have pointed out that exemplar-based CIL may suffer from extra risk of privacy leakage and large memory costs. Nonetheless, we still compared our method with a few exemplar-based methods like BiC [4], iCaRL [5], and WA [6] in Figure 11 in Appendix G. We can see that the proposed YoooP and YoooP+ can achieve comparable results under the zero-base settings, and even outperform some exemplar-based methods under the half-base settings.

Concerning the previous study [7], they utilize task ID information during training, which deviates from the established definition of Class-incremental learning (CIL). As outlined in [8] and [9], CIL, true to its name, operates without predefined task boundaries. Consequently, CIL methods typically lack access to any task ID information during both training and inference phases. This implies that it's impractical to train a model to predict the task ID of input samples during testing or inference in CIL scenarios. Unlike prior work [7], we only access the class label rather than task ID information during model training, which aligns with the methodology of prior works [1],[2],[3],[4],[5],[6]. Therefore, we have opted to exclude that particular work [7] from our comparative analysis.

**References (continue)**

[1] Prototype Augmentation and Self-Supervision for Incremental Learning. (CVPR 2021)

[2] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. (CVPR 2022)

[3] Feature Translation for Exemplar-Free Class-Incremental Learning. (WACV 2023)  

**References (continue for the above A5)**

[4] Large scale incremental learning. (CVPR 2019)

[5] Incremental Classifier and Representation Learning. (CVPR 2017)

[6] Maintaining Discrimination and Fairness in Class Incremental Learning. (CVPR 2020)

[7] A theoretical study on solving continual learning. (NeurIPS2022)

[8] Class-incremental learning: survey and performance evaluation on image classification. (TPAMI 2022)

[9] Deep Class-Incremental Learning: A Survey

Reviewer comment (Evaluation):
Please give the efficiency results. The method seems to be quite slow.

Author response:
**A6:** Table 3 compares the efficiency across different methods using the TinyImageNet dataset, all under identical computing environments and settings (i.e. zero-base with 10 phases). Compared to all baselines, our methods require lower GPU memory usage, lower Multiply–Accumulate Operations (MACs) [1], and shorter training time per epoch. 

Table 3: Efficiency, One GPU NVIDIA RTX A6000, BS=256, M=10^6, G=10^9, T=10^12, 
| Tiny, 0-10 | MACs | Parameters | MAX GPU-Memo | Time/Epoch| Epochs |
| -------- | -------- | -------- | -------- | -------- | -------- |
| YoooP+ | 571.26 G | 11.17 M | 9.86 GB | 21.07 s | 60 |
| YoooP| 571.26 G | 11.17 M | 9.53 GB | 15.60 s | 60 |
| PASS | 2.28 T | 11.18 M | >50 GB | 40.64 s | 101 |
| FeTril | 571.26 G | 11.17 M | 10.72 GB | 349.15 s | Task 0: 200, Others: 50 |
| SSRE | 633.31 G | 12.40 M | 13.23 GB | 50.17 s | 101 |
| LwF | 571.26 | 11.17 M | 9.53 GB | 16.90 s | Task 0: 200, Others: 250 |
| IL2A | 2.74 T | 11.17 M | >50 GB | 97.4 s | 101 |
|       |         |          |         |        |       |

**References**

[1] https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation

Reviewer comment (Reproducibility):
The main weakness of this work is that after reading the main paper the reader won't be able to reproduce and know exactly how the method works. The crucial part is in the appendix, i.e. Algorithm 1, where you see that we have S steps inside each epoch to calculate L_t, R iterations to calculate prototypes p_k, main loss - Arcface is mentioned in the implementation details and it hyper-parameter sigma etc.

Author response:
**A1:** To reproduce our experimental results, we have uploaded the source code in the Supplementary Material. Additionally, we will make the source code publically available on GitHub upon publication. By the way, our source code is not based on an existing framework like PyCIL.

Regarding parameters $S$ steps and $R$ iterations in Algorithm 1, they are determined by the ratio of Dataset Size to Mini-batch size. More specifically, $S$ is the number of total steps per epoch for each task. For the number of steps $R$ used to update the prototypes in each task, we simply set $R=S$ in our experiments. We have added the explanations of these two parameters in Appendix A in our revised version. 

For the Arcface loss hyperparameter $\delta$, we already conducted an ablation study in Appendix C.5 and consistently used $\delta=0.25$ across all experiments with various datasets. Per your suggestion, we have introduced the Arcface loss in the main paper.

Reviewer comment (Theory):
Some equations give even a wrong intuition about how the loss is computed, e.g. Eq.4, where the first component is presented as being over the whole dataset, while in practice it is within S minibatches.

Author response:
**A2:** Per your suggestion, we added some explanations of Eq.4 in the updated version as follows.

where $\ell(\cdot,\cdot)$ is a classification loss, and $i\in [n_t]:y_i=k$ is replaced by $i\in B$ when training by mini-batch.

Reviewer comment (Reproducibility):
Some crucial hyper-parameters are not provided in the main paper, or not provided at all — number of S steps, R iterations.

Author response:
**A3:** Per your suggestion, we have introduced the important hyper-parameters in the main paper. Regarding parameters $S$ steps and $R$ iterations, they are actually determined by the ratio of Dataset Size to Mini-batch size. Please refer to **A1** above.

Reviewer comment (Experiments):
Multiple things are combined in the YoooP method — three loss components, where feature distillation is quite strong regularization preventing forgetting but at the same time lowering plasticity. Additionally, mixing this with model interpolation with beta = 0.6 and freezing the classifier. The ablation is not clear about every single participant. It seems like the model interpolation is crucial here, and the big gain is for YoooP+.

Author response:
**A4:** The classification loss $L_{t,C}$ and the knowledge distillation loss $L_{t,KD}$ in YoooP are commonly used in recent Non-exemplar CIL methods[1],[2],[3]. These two components are also adopted by the baselines. In addition, the partial freezing of the classifier is also used in existing work LwF[1]. So any improvement over the baselines is a result of the proposed $L_{t,P}$ loss.

Accordingly, we do an ablation study on the newly proposed components, including model interpolation (MI), prototype optimization \(P\), and prototype augmentation (PA) in YoooP+. As illustrated in Figure 7 in Appendix C.1, we can see that YoooP(-P), which means without prototype optimization ($L_{t,P}$), will result in much lower accuracy, 51.50\%, than that of YoooP(-MI), 54.82\%. It thus suggests that prototype optimization is more important than MI. However, MI is also important since the accuracy of YoooP(-MI) will be lower compared to YoooP, 57.66\%. 

Additionally, we conducted an additional ablation study about MI on YoooP+. The evaluation results on CIFAR-100 with zeros-base 10 phases setting are reported in Table 1. Removing MI from YoooP+ reduces the accuracy by 3.91\% while removing PA from YoooP+ (i.e. YoooP) reduces the accuracy more, 4.27\%. Thus, prototype augmentation (PA) in YoooP+ is more important than MI.

Table 1: Additional Ablation Study
| CIFAR-100 | zero-10 Avg | zero-10 Last |
| -------- | -------- | -------- |
| YoooP | 57.66\% | 42.49\% |
| YoooP (-MI) | 55.42\% | 37.2\% |
| YoooP+ (-MI) | 58.02\% | 41.2\% |
| YoooP+ (YoooP +PA) | 61.93\% | 47.17\% |
| | | |

**References**

[1] Learning without Forgetting. (ECCV2016)

[2] Prototype Augmentation and Self-Supervision for Incremental Learning. (CVPR2021)

[3] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. (CVPR2022)

Reviewer comment (Presentation):
In the first figure there are missing units for the memory size. It is the first figure that is supposed to give some motivation, but currently it raises more questions. Additionally, SSRE to my knowledge has the growing part and then compression. How is it possible that it is in line with FeTrIL, PASS, IL2A?

Author response:
**A5:** Thanks for your suggestion! We have updated the units for the memory size in Figure 1.

As described in the caption of Figure 1 and detailed in Appendix D, the term 'memory size' denotes the memory footprint of our method and other approaches when computing prototypes for each task during training.

For SSRE, our reported memory size does not refer to the GPU memory during model training. SSRE, IL2A, and PASS compute the mean value of all embeddings per class. Thus, they have the same memory footprint (same line in Figure 1) and incur an expensive memory footprint for storing their embeddings. In contrast, our proposed mini-batch attentional mean-shift method computes prototypes in a mini-batch manner. This significantly reduces the memory footprint required for prototype generation.

Reviewer comment (Experiments):
The method is not evaluated with ImageNet-100. This dataset would shed more light on how the method performs on larger images and how to compare with the other baselines.

Author response:
**A6:** Per your suggestion, we have evaluated our methods on ImageNet-100. The results are reported in the following table, where "*" denotes non-exemplar methods, "-last" means the average accuracy of the last task. IL2A is not included in the comparison because we do not have enough GPU memory to run their model on ImageNet-100. Our methods (YoooP and YoooP+) still significantly outperform the baselines in terms of average accuracy on the new dataset.

Table 2: Evaluation on ImageNet-100.
|  ImageNet-100  | Half-10 Avg | Half-10 last |
|  ----  | ----  | ---- |
| * YoooP | 75.97\% | 68.96\% |
| * YoooP+ | 77.64\% | 70.54\% |
| * PASS | 64.3\% | 53.54\% |
| * FeTril | 72.55\% | 64.06\% |
| * SSRE | 60.23\% | 51.20\% |
| * LWF | 47.35\% | 30.53\% |
| iCaRL-CNN | 48.46\% | 36.29\% |
| iCaRL-NME | 59.04\% | 48.69\% |
| BiC | 50.69\% | 31.06\% |
| WA | 61.04\% | 46.24\% |
| | | |

Reviewer comment (Evaluation):
There is a logic issue in the reasoning on Page 7, before the statement: "Therefore, we can draw a conclusion that the proposed methods can outperform the non-exemplar baselines.":

    While SSRE and FeTrIL have lower average forgetting than our methods, their prediction accuracy drops rapidly in the initial tasks, resulting in low accuracy in the final task, as shown in Figure 4. In reality, the lower forgetting in SSRE and FeTrIL is attributed to the sharp drop in accuracy in the early tasks and the limited learning progress in the subsequent tasks.

That is true about SSRE and FeTrIL, and I agree. They mostly fight with forgetting. That's why they are good in base-half setting (see Tiny and FeTrIL in your appendix, YoooP is lower there). But how can you conclude the stated claim after these two sentences? Overall, in base0 setting your methods present better accuracy. But to conclude this, the behavior of SSRE and FeTrIL doesn't matter here.

Author response:
**A7:** We already updated the logic meaning on Page 7 as follows.

According to Table 1 and Figure 4, both SSRE and FeTrIL have lower forgetting while the prediction accuracy drops rapidly in the initial tasks. A lower forgetting in this case (with lower accuracy) indicates that the model is not improving or learning so such performance is not preferred. In contrast, the proposed YoooP and YoooP+ reach higher average incremental accuracy while achieving slightly higher or comparable forgetting compared to other non-exemplar baselines. This indicates a better stability-plasticity trade-off, implying that the model learns quickly on new tasks while incurring a minimal cost in forgetting. Therefore, it can be concluded that the proposed methods outperform the non-exemplar baselines.

Reviewer comment (Experiments):
In Sec 4.3, the statement "This is because YoooP+ can form a compact cluster for the samples in each class via prototype optimization and create high-quality synthetic data from the original distribution of cosine similarity to constrain **the boundary of old tasks.**" could benefit from an experiment to show this effect.

Author response:
**A8:** Our experiments in **Figure 6** verify that our method can achieve better classification boundaries for old tasks. As shown in Figure 6, the light gray points represent the distribution of data from old tasks. A comparison between Figure 6(a) and Figure 6(b) reveals that the data from previous tasks are still clustered into several groups using our method, exhibiting a clear boundary with the clusters of current task data. This pattern is also evident in Figure 6\(c\). The high-quality synthetic data effectively constrains the boundary of old tasks when training the current tasks, enabling the current tasks to establish a clear boundary with previous tasks. In contrast, the baselines like PASS (illustrated in Figure 6(d), (e), (f)), without our proposed prototype augmentation in YoooP+, will make the boundary between the current task data and previous task data gradually become less clear and their classes may even overlap with each other.

Reviewer comment (Reproducibility):
Overall, it is quite interesting work, but YoooP combines so many things, and it is not clear how each of them contributes. After reading the main paper you cannot reproduce it; you can have the idea how it works, but the appendix is necessary and still not enough. The paper should be rewritten paying attention to all the hyper-parameters, mentioning Arcface loss with sigma, and including a good ablation. Maybe some insights or comparisons with PASS can be moved to the appendix to make space for more crucial information/ablations.

Author response:
**A1:** To reproduce our experimental results, we have uploaded the source code in the Supplementary Material. Additionally, we will make the source code publically available on GitHub upon publication. By the way, our source code is not based on an existing framework like PyCIL.

Regarding parameters $S$ steps and $R$ iterations in Algorithm 1, they are determined by the ratio of Dataset Size to Mini-batch size. More specifically, $S$ is the number of total steps per epoch for each task. For the number of steps $R$ used to update the prototypes in each task, we simply set $R=S$ in our experiments. We have added the explanations of these two parameters in Appendix A in our revised version. 

For the Arcface loss hyperparameter $\delta$, we already conducted an ablation study in Appendix C.5 and consistently used $\delta=0.25$ across all experiments with various datasets. Per your suggestion, we have introduced the Arcface loss in the main paper.

**A4:** The classification loss $L_{t,C}$ and the knowledge distillation loss $L_{t,KD}$ in YoooP are commonly used in recent Non-exemplar CIL methods[1],[2],[3]. These two components are also adopted by the baselines. In addition, the partial freezing of the classifier is also used in existing work LwF[1]. So any improvement over the baselines is a result of the proposed $L_{t,P}$ loss.

Accordingly, we do an ablation study on the newly proposed components, including model interpolation (MI), prototype optimization \(P\), and prototype augmentation (PA) in YoooP+. As illustrated in Figure 7 in Appendix C.1, we can see that YoooP(-P), which means without prototype optimization ($L_{t,P}$), will result in much lower accuracy, 51.50\%, than that of YoooP(-MI), 54.82\%. It thus suggests that prototype optimization is more important than MI. However, MI is also important since the accuracy of YoooP(-MI) will be lower compared to YoooP, 57.66\%. 

Additionally, we conducted an additional ablation study about MI on YoooP+. The evaluation results on CIFAR-100 with zeros-base 10 phases setting are reported in Table 1. Removing MI from YoooP+ reduces the accuracy by 3.91\% while removing PA from YoooP+ (i.e. YoooP) reduces the accuracy more, 4.27\%. Thus, prototype augmentation (PA) in YoooP+ is more important than MI.

Table 1: Additional Ablation Study
| CIFAR-100 | zero-10 Avg | zero-10 Last |
| -------- | -------- | -------- |
| YoooP | 57.66\% | 42.49\% |
| YoooP (-MI) | 55.42\% | 37.2\% |
| YoooP+ (-MI) | 58.02\% | 41.2\% |
| YoooP+ (YoooP +PA) | 61.93\% | 47.17\% |
| | | |

**References**

[1] Learning without Forgetting. (ECCV2016)

[2] Prototype Augmentation and Self-Supervision for Incremental Learning. (CVPR2021)

[3] Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning. (CVPR2022)

Reviewer comment (Novelty):
Out of curiosity, why in the introduction when you introduce the class-incremental learning setting do you refer to three quite recent works of Zhu and Zhou (2021-2022)? This scenario of CL training has existed much longer than that.

Author response:
**A9:** We have cited some earlier works on class-incremental learning in the Introduction in our revised version.

Reviewer comment (Experiments):
In the ablation of beta (Fig.8(a)), why do we see different starting points for the first task? Beta is the interpolation that will take place after the first task.

Author response:
**A10:** According to Equation 12, we update the parameters $\theta$ based on $\beta$. The different $\beta$ in our ablation study result in different starting points, $\theta_1$, for each case.

Reviewer comment (Reproducibility):
Why not provide the source code in the supplementary? Without it, I think the results will be hard to reproduce. I guess you use an existing framework for your work — looking at the results, it seems to be PyCIL. It would be good to give credit to the authors and point out your starting implementations.

Author response:
**A1:** To reproduce our experimental results, we have uploaded the source code in the Supplementary Material. Additionally, we will make the source code publically available on GitHub upon publication. By the way, our source code is not based on an existing framework like PyCIL.

Reviewer comment (Experiments):
The scale of dataset (CIFAR-100, TinyImageNet) is small. Whether the proposed method works with bigger datasets is an important issue that needs to be addressed.

Author response:
**A1:** Thanks for your suggestion! We have evaluated the proposed two approaches on ImageNet-100. The results are reported in the following table, where "*" indicates non-exemplar methods, "-last" means the average accuracy of the last task. IL2A is not included in the comparison because we do not have enough GPU memory to train their model on ImageNet-100. Our methods (YoooP and YoooP+) still significantly outperform the baselines in terms of average accuracy on the large ImageNet-100 dataset.

Table 1: Evaluation on ImageNet-100.
|  ImageNet-100  | Half-10 Avg | Half-10 last |
|  ----  | ----  | ---- |
| * YoooP | 75.97\% | 68.96\% |
| * YoooP+ | 77.64\% | 70.54\% |
| * PASS | 64.3\% | 53.54\% |
| * FeTril | 72.55\% | 64.06\% |
| * SSRE | 60.23\% | 51.20\% |
| * LWF | 47.35\% | 30.53\% |
| iCaRL-CNN | 48.46\% | 36.29\% |
| iCaRL-NME | 59.04\% | 48.69\% |
| BiC | 50.69\% | 31.06\% |
| WA | 61.04\% | 46.24\% |
|  |  |   |

Reviewer comment (Experiments):
The paper lacks the result with 20 phases which is a quite common configuration.

Author response:
**A2:** We have evaluated the proposed methods on TinyImageNet under zero-base 20 phases and half-base 20 phases settings. The results are reported in the following table, where "*" indicates non-exemplar methods, "-last" means the average accuracy of the last task. We can observe that our methods (YoooP and YoooP+) still significantly outperform the baselines in terms of average accuracy on the 20 phases settings.

Table 2: Evaluation on TinyImageNet.
|  TinyImageNet  | Zero-20 Avg | Zero-20 last | Half-20 Avg | Half-20 last |
|  ----  | ----  | ---- | ----  | ---- |
| * YoooP | 46.00\% | 30.97\% | 48.19\% | 33.31\% |
| * YoooP+ | **48.36%** | **31.26\%** | **51.27\%** | **35.28\%** |
| * PASS | 33.76\% | 19.36\% | 33.24\% | 25.21\% |
| * IL2A | 25.39\% | 17.73\% | 39.9\% | 29.43\% |
| * FeTril | 31.84\% | 18.38\% | 48.96\% | 41.44\% |
| * SSRE | 25.78\% | 15.25\% | 46.2\% | 38.84\% |
| * LWF | 22.93\% | 9.57\% | 13.39\% | 5.62\%
| iCaRL-CNN | 30.33\% | 11.31\% | 28.9\% | 20.54\% |
| iCaRL-NME | 36.07\% | 18.35\% | 33.68\% | 27.3\% |
| BiC | 34.29\% | 13.14\% | 40.81\% | 19.91\% |
| WA | 35.87\% | 13.17\% | 27.93\% | 11.91\% |
|    |  |    |  |  |

Reviewer comment (Evaluation):
The effect of an optimized prototype based on the attentional mean-shift method seems limited in many situations.

Author response:
**A3:** Our prototype optimization method is versatile and can be applied not only to prototype-based methods but also to various scenarios in class incremental learning, as demonstrated in [1][2][3][4].

**References**

[1] Incremental Classifier and Representation Learning. (CVPR 2017)

[2] Large Scale Incremental Learning. (CVPR 2019)

[3] BEEF: Bi-Compatible Class-Incremental Learning via Energy-Based Expansion and Fusion. (ICLR 2023)

[4] Feature Boosting and Compression for Class-incremental Learning. (ECCV 2022)

Reviewer comment (Experiments):
For verifying the strength of proposed augmentation, why don’t you apply the new strategy to PASS instead of adding Gaussian noise to prototype? Would you show the performance of it?

Author response:
**A4:** We have applied our new prototype augmentation (PA) strategy to the PASS, and then evaluated its performance on CIFAR-100 under the zero-base 10 phases setting. The results are reported in the following table. We can observe that our prototype augmentation method can significantly improve the performance of PASS, even without prototype optimization.

Table 3: Comparison of adding Gaussian noise and our strategy on PASS.
| CIFAR-100 | zero-10 Avg | zero-10 Last |
| -------- | -------- | -------- |
| PASS (Adding noise)  | 51.94\% | 35.81\% |
| PASS with PA | 55.98\% | 41.03\% |

Reviewer comment (Evaluation):
Could you show the classification confusion matrix result of your approach?

Author response:
**A5:** Thanks for your suggestion! We randomly choose 10 classes to show a part of the classification confusion matrix of YoooP+, and the entire confusion matrix of 100 classes is presented in Appendix E of the revised version.

Table 4: Confusion Matrix (randomly sampled 10x10 submatrix) for CIFAR-100 with zero-base 10 phases setting
| Confusion Matrix | Road | Lobster | Cattle | Couch | Shrew | Aquarium fish | Castle | Palm tree | Snail | Bowl |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| **Road** | 79 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **Lobster** | 0 | 76 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **Cattle** | 1 | 0 | 58 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **Couch** | 0 | 0 | 0 | 48 | 0 | 0 | 0 | 0 | 0 | 0 |
| **Shrew** | 0 | 0 | 6 | 0 | 54 | 0 | 0 | 0 | 0 | 0 |
| **Aquarium fish** | 0 | 0 | 0 | 0 | 0 | 54 | 0 | 0 | 0 | 0 |
| **Castle** | 0 | 0 | 0 | 0 | 0 | 0 | 24 | 0 | 0 | 0 |
| **Palm tree** | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 61 | 0 | 0 |
| **Snail** | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 41 | 0 |
| **Bowl** | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 44 |

Reviewer comment (Writing):
In section 3, when define the probabilities over all classes C_(1:t ), isn’t softmax(wG(F(x;θ))) more accurate notation than softmax(wF(x;θ))?

Author response:
**A6:** Thanks for your suggestion! We have already updated the notation with $softmax(G(F(x; \theta); w))$ in Section 3 highlighted in blue.

Reviewer comment (Writing):
The function c(,) in equation 2, equation 4 seems to be different. Is the c(,) in equation 2 cosine similarity function and one in equation 4 classifier (=G(F(p_k;θ)))?

Author response:
**A7:** Thanks for your correction! After fixing Eq.4 in our revised version, $c(\cdot,\cdot)$ in both Eq. 2 and Eq.4 denotes the cosine similarity function. In Eq. 4, it is used to compute the logits for all the classes $j\in C_{1:t}$ as the cosine similarities between each vector in the weight matrix of the final-layer classifier and (1) a sample's representation $z_i$ (the first term in Eq. 4); or (2) each class's prototype $p_k$ (the second term in Eq. 4).

$L_{t,C}(\theta,w)\triangleq\frac{1}{|C_t|}\frac{1}{n_t}\sum_{k\in C_t}\sum_{i\in [n_t]:y_i=k}\ell([c(z_i, w_j)]\_{j\in C_{1:t}, k)} + \frac{1}{|C_{1:{t-1}}|}\sum_{k\in C_{1:{t-1}}}\ell([c(p_k, w_j)]\_{j\in C_{1:t}}, k)$

Reviewer comment (Reproducibility):
How can we define the initial class-wise prototype p_k? Is it the same with the prototype used in PASS? (k-th class mean prototype)?

Author response:
**A8:** We randomly choose one sample from each class-$k$ and use its embedding as the initial prototype $p_k$. Our prototype update method is an extension of the classical mean-shift algorithm [1] that replaces the pre-defined kernel similarity with a learnable attention score. As stated in [1], random initialization of $p_k$ can guarantee the independence of the solution to the initialization problem.

**References**

[1] Mean shift, mode seeking, and clustering, TPAMI 1995.