From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper is proposing to utilize unlabelled data in class-incremental learning setting. The proposed method partially follows online learning with partial information over MDP formalism. Although the authors discuss this setting in detail, resulting application of EXP3 is still somewhat unnatural. Four experts reviewed the paper independently and 3 out of 4 thinks the paper has major issues which needs to be fixed before publication:
- Its empirical design should be improved significantly. Currently, the baselines are rather simplistic and not utilizing  unlabelled data. Although they are not applied to class incremental learning setting, many methods can utilize unlabelled data like contrastive approaches. Evaluation such simple baselines is necessary to justify the complexity of the proposed method.
- The setting needs to be justified. First of all, the unlabelled data stream and its relationship to the labelled stream should be clarified further. What are the assumptions needed?, are they realistic? is there any real application which would benefit from this method?
- Although the setting is significantly different than classical class-incremental learning, there is a very minor performance improvement. This is extremely surprising as using significant amount of additional information does not simply improve results. This should be explained and studied.
justification_for_why_not_higher_score: The paper is simply not ready to be published. There are major issues which need to be fixed before publication: 1) justification of the proposed new problem, 2) better empirical study with closer baselines and fair comparisons as well as results from simple ML methods, 3) explanation for lack of performance.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Any further questions? 
comment: As the end of the discussion is approaching, we are curious whether our feedback has addressed your concerns.

If you still have some remaining concerns, we are very happy to engage in a discussion to address them.

If you do not have more concerns, we would really appreciate it if you have more comments on how to improve our paper. 

Thanks in advance! Looking forward to hearing from your soon.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Any further questions? 
comment: As the end of the discussion is approaching, we are curious whether our feedback has addressed your concerns.

If you still have some remaining concerns, we are very happy to engage in a discussion to address them.

If you do not have more concerns, we would really appreciate it if you have more comments on how to improve our paper. 

Thanks in advance! Looking forward to hearing from your soon.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Any further questions?
comment: As the end of the discussion is approaching, we are curious whether our feedback has addressed your concerns.

If you still have some remaining concerns, we are very happy to engage in a discussion to address them.

If you do not have more concerns, we would really appreciate it if you have more comments on how to improve our paper. 

Thanks in advance! Looking forward to hearing from your soon.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Further feedback
comment: Thanks a lot for the further questions from the reviewer. 

&nbsp;

**For Q1**: We fully agree that it is important to show at what correlation magnitude forgetting is mitigated. The table below provides more ablation results on CIFAR-100, $N=5$. From the table, we can observe that a higher correlation magnitude achieves better results. We can also observe that even if all overlapping classes are removed, our method can still perform better than the baseline. This is because our approach can find unlabeled images that contain sufficiently related objects to old classes. These objects can activate the related neurons for old classes. Please kindly refer to Figure 3(a) to see the visualization results.

|        Setting         | Average Acc. (%)  | Last-phase Acc. (%) | 
| ---------------------- | ------------ | ------------ | 
|Baseline         | `57.12` | `47.49` |
| Non-overlapping        | `61.52` | `51.70` |
| 10% overlapping        | `61.71` | `51.93` |
| 30% overlapping        | `61.89` | `52.03` |
| 50% overlapping        | `62.01` | `52.31` |
| 70% overlapping        | `62.13` | `52.57` |
| 100% overlapping        | `62.15` | `52.62` |

&nbsp;

**For Q2:** In the table below, we provide the results of our method in two different CL setups, using the state-of-the-art CIL method FOSTER (Wang et al., 2022) as the baseline. 

&nbsp;

**Average accuracy (%), CIFAR-100, $N=5$**
|        Method         | Training from 50 classes, then 10 classes per phase | Training from scratch, then 20 classes per phase  | 
| ---------------------- | ------------ | ------------ | 
|FOSTER  (Wang et al., 2022)      | `70.6` | `72.5` |
| FOSTER  + ours       | `72.0` | `73.7` |

&nbsp;

**Average accuracy (%), CIFAR-100, $N=25$**
|        Method         | Training from 50 classes, then 2 classes per phase | Training from scratch, then 4 classes per phase  | 
| ---------------------- | ------------ | ------------ | 
|FOSTER  (Wang et al., 2022)      | `63.8` | `70.7` |
| FOSTER  + ours       | `67.0` | `72.7` |

We will include the tables and corresponding discussions in the revision. Please feel free to add more comments if you have any further concerns.

From: ICLR.cc/2023/Conference/Paper1364/Reviewer_5n8q
title: The concerns remain
comment: 1. non-overlapping is not sufficient for validating low correlation. Adding more data to prevent forgetting is against the purpose of incremental learning. Thus, it is important to show at what condition  (e.g. correlation magnitude between data removed from old classes and data added from other sources) forgetting is mitigated. More important, the relationship between the forgetting and the correlation.  Therefore, non-overlapping is far from serving this purpose.

2. I checked Table A3. Three methods are used as baseline (LwF, iCarl, LUCIR). None of them is closed to the state-of-the-art in this class incremental learning setup. Please use the top performers as the baseline.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Further discussions?
comment: Dear reviewer,

We are curious whether our feedback has addressed your concerns. Feel free to add new comments in the next few days if you have any further questions. We are happy to have further discussions!

Thanks,

Authors

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Further discussions? 
comment: Dear reviewer,

We are curious whether our feedback has addressed your concerns. Feel free to add new comments in the next few days if you have any further questions. We are happy to have further discussions!

Thanks,

Authors

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Further discussions? 
comment: Dear reviewer,

We are curious whether our feedback has addressed your concerns. Feel free to add new comments in the next few days if you have any further questions. We are happy to have further discussions!

Thanks,

Authors

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Further discussions?
comment: Dear reviewer,

We are curious whether our feedback has addressed your concerns. Feel free to add new comments in the next few days if you have any further questions. We are happy to have further discussions!

Thanks,

Authors

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Our Feedback to Reviewer DLTo
comment: We thank the reviewer for finding our paper "well-written" and our method "novel and interesting". Below is our feedback on the questions.

&nbsp;

**R4-Q1: more training details?**

Thanks for the suggestion. We fully agree that showing the training details in the main paper is important. We have moved the training details to Section 4 *implementation details* in the revised paper from Appendix.  

&nbsp;

**R4-Q2: comparing more methods in Table 2?**

Thanks for the suggestion. We have added the results of (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017) to Table 2 in the revised paper.

&nbsp;

**R4-Q3: more visualization results?**

Thanks for the suggestion. We will provide more visualization results for different classes (like Figure 3(a)) in our final version.

&nbsp;

**R4-Q4: more recent related papers?**

We have added this related work (ELI) to Table 2 in the revised paper. We can observe that our method consistently outperforms it on two datasets (CIFAR-100 and ImageNet-100), e.g. improving the average accuracy of the $N=5$ setting by $3.19$ and $8.49$ percentage points on CIFAR-100 and ImageNet-100, respectively.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Our Feedback to Reviewer 5n8q
comment: We thank the reviewer for finding our paper "well-written" and our method "achieving clear improvements". Below is our feedback on your questions. 

&nbsp;

**R3-Q1: the assumption of accessing free image stream (like Google Images) is NOT carefully validated?**

Thanks for the question. Actually, we provided ablation results to validate this point in the second block (Rows 3-6) of Table 3 (main paper).  

Rows 3-6 are respectively as follows.
Row 3 *Overlapping* means including samples from the overlapping classes between CIFAR-100 and ImageNet. Row 4  *Non-overlapping* means using only the samples of non-overlapping classes between CIFAR-100 and ImageNet (more realistic than *Overlapping*). Row 5 *new class data* means using only the current-phase new class data (i.e., without using any free data stream) as candidates to select placebos. Row 6 *Old data* means the original old class data are all accessible when computing KD loss (i.e., upper bound of KD effect). Please note that in Row 3 and Row 4, two classes are considered *overlapping* if their classes or super-classes overlap. For example, *n02640242 - sturgeon* in ImageNet-1k is regarded as an overlapping class of the *fish* in CIFAR-100, because they overlap at the level of super-class (i.e., *fish*). 

When comparing Row 4 with Row 2 (Table 3), we can find that our method (Row 2) is robust to the change of data streams: even if all overlapping classes are removed, our method can still achieve the same-level performance. Comparing Row 5 with Row 2, we can get a clear sense that using additional unlabeled data is definitely helpful.
Comparing Row 6 with Row 2, we see that our method achieves comparable results to the upper bound.

&nbsp;

**R3-Q2: another CL setup?**

We have included these results in Appendix H and Table A3. In Table A3 (appendix), we supplemented the results of using the equal-size split on CIFAR 100. We can observe that using our PlaceboCIL significantly improves the performance compared to the baselines, e.g., boosting the average accuracy of LwF by $3.83$ percentage points. Using the online learning algorithm can further boost performance. The observations are similar to those using the original setting, i.e., training the $0$-th phase model on half of all classes. Due to the time limitation, we will provide the results on other datasets in the final version.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Our Feedback to Reviewer rXB7
comment: We thank the reviewer for finding our idea "interesting". Below is our feedback on the questions.

&nbsp;

**R2-Q1: rehearsal-free?**

Yes. It is possible to use placebos to improve CIL performances in the *rehearsal-free* setting. Some variants of our method can achieve this. 

In the table below, we provide the results of the *rehearsal-free* variants of our method: 1) PlacboCIL (original) w/o exemplars: we don't use any old-class exemplars to compute loss, but we store one exemplar per class for the computation of evaluation function (Eq. 4); 2) PlacboCIL (higher confident) w/o exemplars: we don't use any old-class exemplars, and select the unlabeled data with higher confidence as placebos. 

From Table R2, we can observe that using placebos is effective in the *rehearsal-free* settings. Our original PlaceboCIL improves the average accuracy of the baseline by $6.30$ percentage points. The *higher confident* version performs worse but still gets clear margins: it improves the average accuracy of the baseline by $3.40$ percentage points.

We have added the above results and discussions in Appendix J and Table A5.

**Table R2: Ablation results (%) for the *rehearsal-free* setting on CIFAR-100, N =5. Baseline: LUCIR+AANets.**
|   No.   | Setting | Average acc.  | Last-phase acc. | 
| ---------------------- | ------------ | ------------ | ------------ | 
| 1          | Baseline | 66.72 | 57.77 | 
| 2          | PlaceboCIL (original)  | 67.16 | 59.14 | 
| 3          | PlaceboCIL (higher confidence)   | 66.97 | 58.12 | 
| 4         | Baseline *w/o* exemplars | 50.01 | 39.12 | 
| 5         | PlaceboCIL (original) *w/o* exemplars | 56.31 | 44.02 | 
| 6         | PlaceboCIL (higher confidence) *w/o* exemplars  | 53.41 | 41.93 | 

&nbsp;

**R2-Q2: what if the exemplars are totally removed from (4)?**

In Table R2, Row 5 shows the results when the exemplars are totally removed from Eq. 4. We can observe that the performance is much lower compared to using exemplars (Rows 1-3). However, comparing Row 5 to Row 4, it is clear that using our PlaceboCIL still achieves significant improvements over baseline without exemplars, i.e., 6.3 percentage point improvements of the average accuracy. 

&nbsp;

**R2-Q3: are the results strictly fair?**

For this question, please kindly refer to Section 3.2 *allocating mini-batch-based memory for placebos*. Specifically, in order to not exceed the memory budget, we randomly remove the same number (i.e., |$\mathcal{U} + \mathcal{P}$|) of samples from the training data of new classes. Removing a few training samples from new data does not significantly affect the CIL performance, as validated in [B]. All experiments of our PlaceboCIL in the main paper use this *strict budget* setting. Therefore, the results in Tables 1 and 2 are strictly fair regarding the memory budget. 

[B] "RMM: Reinforced memory management for class-incremental learning." Advances in Neural Information Processing Systems 34 (2021): 3478-3490.

&nbsp;

**R2-Q4: the ablation study in Table 3 is very unclear.**

We hope the following rephrasing of the ablation settings solves this question.
- In the first block, Row 1 is to show the results of two baselines, and Row 2 is for our method. All the following ablation settings (Rows 3-10) are based on Row 2.
- The second block shows the variants of our method (i.e., based on Row 2)? of using 4 kinds of free data streams: Row 3 *Overlapping* means including samples from the overlapping classes between CIFAR-100 and ImageNet. Row 4 *Non-overlapping* means using only the samples of non-overlapping classes between CIFAR-100 and ImageNet (more realistic than *Overlapping*). Row 5 *New data* means using only the current-phase new class data (i.e., without using any free data stream) as candidates to select placebos. Row 6 *Old data* means the original old class data are all accessible when computing KD loss (i.e., upper bound of KD effect). Please note that in Row 3 and Row 4, two classes are considered *overlapping* if their classes or super-classes overlap. For example, *n02640242 - sturgeon* in ImageNet-1k is regarded as an overlapping class of the *fish* in CIFAR-100 because they overlap at the level of super-class (i.e., *fish*).
- The third block is to test different policy learning methods (based on Row 2). Row 7 is for using fixed evaluation functions ($\beta_i$=$\gamma_i$=$1$). Row 8 uses the offline RL (the REINFORCE algorithm) to train the selection policy. 
- The fourth block is to test different placebo selection strategies. Row 9 uses unlabeled data with higher confidence. Row 10 uses them randomly.

&nbsp;

**R2-Q5: the layout of Figure 1 needs improvement, especially (b).**

Thanks for the suggestion. We have updated Figure 1 to make it look better. If you have any further suggestions, we're happy to update it accordingly.

From: ICLR.cc/2023/Conference/Paper1364/Authors
title: Our Feedback to Reviewer 4WyN
comment: We thank the reviewer for finding our paper "well organized" and "contributes novelty". Below is our feedback on the questions.

&nbsp;

**R1-Q1: the results are not so significant?**

Thanks for the good question. We address the reviewer's concerns from the following aspects:
- We have to argue that the *higher confidence* setting (in Tables 3 and A1) is actually based on part of our method. There are two main contributions of our method: online placebo selection and mini-batch-based memory reusing strategy. The *higher confidence* setting uses the *mini-batch-based memory reusing strategy*. The reason is that without this *reusing strategy* it is not intuitive how to efficiently select unlabeled data without extra memory.
- We agree that a confident selection metric can provide relatively good improvements on some baselines (e.g., LUCIR-AANets). However, the confident selection metric can not work as well as our method evaluation function on all baselines. For example, in Table A1 (appendix), we can see the average accuracies of our PlaceboCIL are $2.08$ and $1.82$ percentage points higher than the average accuracies of the confident selection metric on LwF and LUCIR, respectively. This is because our online learning method updates the policy to generate different evaluation functions to adapt to different baselines and environments. 
- We have compared our method with existing CIL methods of using additional data (i.e., GD+ext (Lee et al., 2019) and MUC-LwF (Liu et al., 2020a). In the second block of Table 2 (main paper), we can see that our method consistently outperforms them. For example, our method (FOSTER w/ PlaceboCIL) beats GD+ext and MUC-LwF by $8.8$ and $12.94$ percentage points, respectively.
- We would like to highlight that the improvement from the proposed online learning becomes more significant if the number of exemplars is smaller. Our online learning method improves the last phase accuracy by $1.02$ percentage points on LUCIR-AANets ($N=5$) if we use $20$ exemplars for each class. If we reduce the number of exemplars to $5$ per class, the performance improvement becomes $3.13$ percentage points. 

**Table R1: Average accuracy (%) on CIFAR-100, $N=5$. The results are copied from Table 2 (main paper) and Table A1 (appendix)**
|          Setting          | LwF | iCaRL | LUCIR | LUCIR-AANets |
| ---------------------- | ------------ | ------------ | ------------ | ------------ |
| Higher confidence         | 56.98 | 60.43 | 63.60 | 66.97  |
| PlaceboCIL (ours)         | 59.06 |61.01 | 65.42 | 67.16 |
| Difference       |2.08 | 0.58 | 1.82 | 0.19 |

&nbsp;

**R1-Q2: reporting the mean and standard deviation?**

Thanks for the suggestion. We reported the mean and standard deviation in the third block of Table 2 (POD-AANets w/ and w/o ours; FOSTER w/ and w/o ours). After revision, we added the standard deviations to the results of GD+ext and MUC-LwF. Since many comparing methods didn't report the standard deviations in their original papers, we need time to re-run their experiments. Hence, we will report the means and standard deviations for them in the final version. 

&nbsp;

**R1-Q3: ImageNet-1K for modified ResNet-32?**

We resized the images from ImageNet-1k to $32\times32$ to fit the modified ResNet-32. Please kindly refer to more implementation details in the code provided in the supplementary materials.

From: ICLR.cc/2023/Conference/Paper1364/Reviewer_4WyN
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The setting of this paper is to use wild data (free data) as an additional resource to do class-incremental learning. This paper proposes an online placebo (free images) selection policy to select good images from the free data stream. The selected images are used for knowledge distillation. Thus the proposed method can be combined with existing CIL methods to improve their performances. To train the evaluation policy, this paper introduces an online reinforcement learning algorithm, which reduces training computation costs compared with offline reinforcement learning. Experiments show that the proposed method can enhance the performances of many existing methods.
strength_and_weaknesses: Strengths:
(1) This paper proposes an evaluation function to select good images from the free data stream. The proposed methods can provide a plug-in component for existing CIL methods.
(2) The experiments validate the effectiveness of the proposed framework.
(3) The paper is well organized, clearly motivated, and contributes novelty.

Weaknesses:
(1) The results are not so significant. There are some existing works using wild/free images as additional data for incremental learning. Compared with previous works, this work is a universal plug-in component to enhance many existing CIL works. However, this paper uses the methods without additional data as baselines, which obviously have improvements. In fact, one intuitive way of using free images is to find a confidence score to select good images or generate pseudo labels, which should be the true baseline. From Table.3 and A1, we can find that "Higher confidence" seems a baseline that does not use the proposed method but also can select good images for knowledge distillation. The gaps between the proposed method and "Higher confidence" is even close to random errors for LUCIR-AANet and iCaRL. From Table.3, using the proposed methods gets only 0.58% improvement in average accuracy for iCaRL, and the results for LUCIR-AANet is only 0.37%, which means "Higher confidence" is the basic baseline that should be compared with. This indicates that the overall good performance is largely due to the additional data, even a simple confident selection metric can provide relatively good improvements. 
(2) It would be helpful to report the mean and standard deviation to take out any concerns related to randomness.
(3) How to use ImageNet-1K as additional dataset for modified ResNet-32? Because the image size of CIFAR-100 is 32x32. 

clarity,_quality,_novelty_and_reproducibility: Generally speaking, this paper is clear and easy to follow.
summary_of_the_review: There are concerns about the results. The baseline methods compared are without additional data, which is somehow unfair. Moreover, the proposed offline RL method takes extra costs but only brings marginal improvement. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1364/Reviewer_rXB7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper presents a method to leverage free unlabeled image as "placebos" to calculate knowledge distillation loss for class incremental learning. A corresponding online learning algorithm and a memory reusing strategy are proposed to avoid extra memory usage. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the method.
strength_and_weaknesses: The main strength of the paper is the idea of levering free/public available data to help class incremental learning.

However, there are multiple weaknesses that influence the overall quality of the paper:
- I actually have a fundamental question of the problem setting. Since the idea is to use the placebos of old classes, is it possible to totally get rid of old classes, i.e., rehearsal-free? 
- In (4), the old class exemplars still play an important role in both the rehearsal loss (first term) and the distillation loss (second term). What if the exemplars are totally removed from (4)?
- I doubt that the results in Table 1 and 2 are strictly fair. Since saving the free data batch requires additional memory, shouldn't the competing methods get the same amount of memory to save buffered examples? For example, if $|\mathcal{U}|  + |\mathcal{P}| = 1200$, the buffer size for other methods should be $1200$ + "buffer size used in the results". 
- The ablation study in Table 3 is very unclear.
- The layout of Figure 1 needs improvement, especially (b).
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the paper is good.
The novelty and originality of the work is good, but the empirical evaluation is questionable.
The code is available so it should be reproducible.
summary_of_the_review: The paper presents an interesting idea of levering free/public available data to help class incremental learning. However, the empirical evaluation of the method itself needs careful check and explanation. Therefore, I recommend weak reject initially.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1364/Reviewer_5n8q
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes using placebos (chosen from a free image stream such as Google Images) in class incremental learning. It formulates the policy training process as an online Markov Decision Process, and achieve improvement over prior works.
strength_and_weaknesses: Strength:
1. The paper is written well.
2. Clear improvement is achieved over baseline.

Weakness:
1. The assumption of accessing free image stream (like Google Images) is NOT carefully validated. If the free image stream includes images of old classes or highly correlated to old classes, this essentially increases the size of memory which is not fair.  A clear distinction in terms of statistics between memory and placebo should be provided. (Please do NOT just say they are from different data sources.)

2. Another CL setup is missing: the 0th phrase does not have half of classes, but just the number of incremental class (e.g. 5 or 10 classes).
clarity,_quality,_novelty_and_reproducibility: The major concern is if accessing free image stream (like Google Images) is valid, which significantly affect the quality of this paper. However this is not provided.
summary_of_the_review: Reject is rated, given the crucial assumption of accessing free image stream is not carefully validated. Another concern is missing an important setup (see weakness).
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1364/Reviewer_DLTo
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper aims to address the catastrophic forgetting problem in class-incremental learning (CIL). As most of the old samples are not accessible in CIL, the authors propose to use unlabeled external data (i.e., placebos) to compute the knowledge distillation loss to consolidate the old knowledge. To achieve online placebo selections, the authors propose to formulate the CIL task as an online MDP and design a novel online learning algorithm for learning an online policy to control the placebo selection. Further, they propose a mini-batch-based memory-reusing strategy to avoid extra memory usage. The authors conduct extensive experiments on three CIL datasets, CIFAR-100, ImageNet-100, and ImageNet-1k, and show their method achieves state-of-the-art performance. 
strength_and_weaknesses: Strengths

1. The proposed online MDP formulation for CIL is novel and interesting. I think it makes sense to formulate the CIL as an online MDP and learn a policy to control the learning process, e.g., placebo selection. I think this formulation can also be used to learn policies to adjust other components in CIL. 

2. The idea of using extra unlabeled data is interesting and reasonable. Because unlabeled data can be obtained easily from the Internet without much effort. I think this is a promising direction to improve performance when training with imbalanced data, e.g., CIL. 

3. This paper is well-written and easy to follow. The authors clearly explain their motivation in the introduction. Besides, the authors also provide extensive figures and visualizations to make their explanations clearer. For example, Figure 1(b) clearly visualizes the problem when using the new data to compute the KD loss. Figure 1(c) demonstrates the effectiveness of the proposed placebo selection method. 

4. Extensive experiment results are provided to show the effectiveness of the proposed method. For example, in Table 1, the authors combine their method with five different CIL baselines and achieve consistent improvements. It is impressive to see that their method can work with many different methods and KD losses. In Table 2, the authors show their method achieves state-the-art performance on three datasets. Furthermore, the authors also provide detailed ablation results and visualizations. In summary, I think the experiments are very impressive. 

5. The authors provide open-source code and detailed configures. These materials are helpful for the following researchers. 

Weaknesses

1. It would be better to move more training details (e.g., Appendix E training configures) from the appendices to the main papers. Because I think these details are very important for reproducing the results. 

2. The authors should also compare some famous CIL methods (e.g., LUCIR (Hou et al., 2019) and iCaRL (Rebuffi et al., 2017)) in Table 2 because they are very important baselines for class-incremental learning. 

3. The authors can provide more visualization results for different classes (like Figure 3(a)) in the appendix. These visualization results can help the following researchers to understand how the placebo images work in the class-incremental learning models. 

4. Some recent related papers published in CVPR 2022 should also be discussed and compared. For example, [1] also uses the same benchmark protocol and should be compared in Table 2. 

[1] Joseph, K. J., et al. "Energy-based Latent Aligner for Incremental Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

clarity,_quality,_novelty_and_reproducibility: Overall, this paper is a high-quality work. 

Clarity: This paper is well-written and easy to follow. The authors use a lot of figures and visualizations to help the audience to understand the motivation and the proposed method.

Novelty: This work is novel as it proposes an online MDP formulation for CIL, a novel online learning algorithm to train a policy for the online MDP, and a mini-batch-based memory reusing strategy to avoid extra memory usage. 

Reproducibility: The authors provide open-source code in the supplementary and training details in the appendix. 

summary_of_the_review: Overall, I think this is a high-quality paper with a novel method and extensive experiment results. The proposed direction, selecting extra data to help to improve CIL, is promising in my view. Further, this paper achieves state-of-the-art performance and provides open-source code. So, I recommend acceptance. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper