Paper: Overcoming bias towards base sessions in few-shot class-incremental learning (FSCIL) (ICLR-2024)

Reviewer comment (Novelty):
The overall method seems less novel: utilizing the frozen pre-trained backbone to improve the forgetting issue is not new (e.g., [1]). The runtime increases for inference, while [1] does not as the knowledge merging happens in the parameter space; aggregating the prediction from multiple classifiers is an ensemble operation.

Author response:
A: Although aggregating predictions falls under the category of ensemble operations, we claim that our method differs significantly from conventional ones. Unlike typical ensemble methods that do not specify particular types of classification modules, our pipeline employs two distinct types: the base-favored and incremental-favored modules (Sec 3.4). It is crucial to note that simply combining a large number of classification modules does not necessarily lead to improvement. This is because many current methods excel only in the base, and combining such methods does not address the persisting issue of performance bias. As we have proposed in Sec 5.1, incorporating a classification module favoring the incremental session is essential for achieving overall balanced performance.

Reviewer comment (Experiments):
What is the rationale behind choosing ALICE as the baseline model over other options? Is this choice based on heuristics or specific reasons?

Author response:
A: While SAVC and BiDistFSCIL are the latest approaches, the high IMB values in Table 3 indicate that their performance exhibits a pronounced bias toward the base session. This results in an inflated value of the biased conventional metric, misleadingly suggesting proficiency as FSCIL methods. Conversely, ALICE achieved the most balanced performance according to our proposed metrics. Therefore, we decided to use ALICE as our baseline.

Reviewer comment (Reproducibility):
In the CUB200 benchmark, if the settings remain consistent with those in other papers, why do the previous methods listed in Table 3 exhibit significantly lower performance, even on the base classes (a_0), compared to their reported results in their original papers?

Author response:
A: The discrepancy arises from our evaluation using the newly proposed metrics. While previous approaches assess the performance for all classes encountered up to the current session after training at each session, we assess session-level adaptability by exclusively evaluating the classes relevant to a specific session only after training on the last session (Sec 4.2). Hence, our reported values differ from those provided in the original papers.

Reviewer comment (Experiments):
The comparison with prior methods may not be entirely fair, as all of them exclusively use ResNet-18 or ResNet-20, which have significantly fewer parameters compared to ResNet-50 or ViT. Is the observed performance boost primarily attributed to the larger number of parameters?

Author response:
A: We acknowledge that the improvement in performance compared to existing methods can be attributed to the larger backbone. We would appreciate it if the focus could shift from such a perspective to ‘exploring how to effectively utilize backbones with a larger number of parameters (especially in the size constraint of ResNet50) in FSCIL.’ Tables 1 and 2 illustrate that when using a larger backbone, simply employing LN, GAT, or NCM individually results in a biased performance—either exhibiting high accuracy only in the base session (LN and GAT) or relatively higher incremental performance with a notable decline in base performance (NCM). We emphasize that integrating the two specific types of classification modules we proposed can effectively address this issue by achieving an optimal balance in performance compared to the naive use of a large backbone.

Reviewer comment (Experiments):
An essential baseline is missing: Would fine-tuning the pre-trained model on the base classes and using it as a new "frozen pre-trained model" lead to an overall performance improvement? In the current method pipeline, it assumes that the pre-trained model possesses knowledge highly correlated with each dataset. However, this assumption may be vulnerable.

Author response:
A: We assumed that fine-tuning the backbone with base classes induces an inevitable bias towards the base classes, a notion supported by the biased performance of current methods observed in our experiments. Such biased features may not only be unhelpful but could potentially be detrimental in adapting to the incremental sessions. Consequently, we posited the frozen pretrained backbone would be better rather than fine-tuned ones even if the pretrained knowledge is not highly correlated with the incremental sessions. While there may be instances where fine-tuning with base classes can be highly beneficial for adaptation to incremental sessions if there is a significant similarity between the distributions of base and incremental sessions, we believe such instances are generally rare. Therefore, we did not fine-tune the pretrained backbone with the base classes. However, as you suggested, using a fine-tuned backbone as a baseline could provide a more in-depth understanding of the impact of fine-tuning with base classes. We will carefully consider this advice.

Reviewer comment (Experiments):
The two proposed benchmarks, Flower102 and DCC, may not be entirely suitable for FSCIL. The fundamental concept of FSCIL involves a large-scale dataset in the base session, which is reasonable for offline data collection and training. However, Flower102 and DCC have relatively small scales, with only 20 and 80 images per class for the base sessions, totaling 1200 and 5360 images, respectively. This can pose challenges for algorithm development, particularly as models grow larger. Additionally, the limited number of test images can result in significant evaluation variation.

Author response:
A: From the perspective of the number of images per base class, (as we have described in sec 4.1,) the proposed datasets have more images—Flowers102 has 40, DCC has 80 images—than CUB200 where there are around 30 training images per base class. Furthermore, they can serve as valuable tools for assessing the robustness against the target distribution shifts. However, we agree that more test images would have been beneficial.

Reviewer comment (Experiments):
In Table 8, are the experiments conducted excluding the baseline model as in Fig. 2?

Author response:
A: Table 8 indicates the results evaluated with our proposed metrics where both baseline and pretrained backbones are used. The setup is the same as that of Figure 4.

Reviewer comment (Experiments):
Should “Flowers102” be replaced by “Flowers100” as only the first 100 classes are used?

Author response:
A: As you mentioned, since only the first 100 classes of Flowers102 are utilized, renaming it to Flowers100 seems appropriate to distinguish it from the original Flowers102.

Reviewer comment (Writing):
The paper lacks a clear motivation of the proposed methods and details of important parts of the model.

Author response:
A: Using only LN or NCM individually does not yield optimal outcomes. LN excels in the base session but falters in the incremental sessions, while NCM performs well in the incremental sessions but poorly in the base. Given their contrasting tendencies, we hypothesized that the two classification modules can potentially complement each other. Indeed, we observed that aggregating their output distributions significantly improved overall performance (Section 5.1).

- The detailed process of the aggregation
The aggregation process entails summing the output distributions from LN and NCM. However, we noticed LN's output distribution exhibits excessive sharpness compared to NCM, leading to LN’s dominance when simply summed.
To address this, we smooth LN’s distribution with an optimal temperature [ref.1] before adding it to NCM's. Through multiple experiments, we find the optimal temperature, observing that LN's smoothened distribution slightly dominates when evaluating test data for the base session while NCM's has a slightly stronger influence when evaluation for the incremental sessions. Consequently, this aggregation enhances overall performance balance.

Reviewer comment (Evaluation):
The empirical results are not compelling enough to validate the effectiveness of the proposed pipeline and evaluation metrics.

Author response:
A: We appreciate your feedback on our oversight.

Reviewer comment (Theory):
One important contribution of the paper is proposing the pipeline to mitigate the performance issue; however, there is no detailed analysis of the reason behind the phenomenon. Why is the proposed pipeline able to tackle this issue?

Author response:
A: Using only LN or NCM individually does not yield optimal outcomes. LN excels in the base session but falters in the incremental sessions, while NCM performs well in the incremental sessions but poorly in the base. Given their contrasting tendencies, we hypothesized that the two classification modules can potentially complement each other. Indeed, we observed that aggregating their output distributions significantly improved overall performance (Section 5.1).

- The detailed process of the aggregation
The aggregation process entails summing the output distributions from LN and NCM. However, we noticed LN's output distribution exhibits excessive sharpness compared to NCM, leading to LN’s dominance when simply summed.
To address this, we smooth LN’s distribution with an optimal temperature [ref.1] before adding it to NCM's. Through multiple experiments, we find the optimal temperature, observing that LN's smoothened distribution slightly dominates when evaluating test data for the base session while NCM's has a slightly stronger influence when evaluation for the incremental sessions. Consequently, this aggregation enhances overall performance balance.

Reviewer comment (Experiments):
In the incremental sessions, whether there have two results for one sample from the NCM classifier and Lightweight Network, and how to deal with this situation.

Author response:
A: We summarize the detailed process of prediction aggregation below.
The aggregation process primarily entails summing the two output distributions from LN and NCM. However, we noticed LN's output distribution exhibits excessive sharpness compared to NCM, leading to LN’s dominance when simply summed, making the prediction identical to using only LN's output distribution.
To address the issue, we smooth the LN’s output (softmax) distribution with an appropriate temperature [ref.1] before adding it to that of NCM. Through multiple experiments, we determine the optimal temperature value for this adjustment. With this modification, we observed that when evaluating test data for the base session, LN's smoothened output distribution has a slightly stronger influence than that of NCM. Conversely, when evaluating test data for the incremental sessions, the influence of NCM's output distribution is slightly stronger than that of LN. Consequently, this aggregation process allows for achieving a more balanced overall performance.
[ref.1] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733–3742, 2018.

Reviewer comment (Evaluation):
In the experiments, I'm concerned about the accuracy of each session and the accuracy of each session compared to other methods.

Author response:
A: We summarize the details about our proposed ‘accuracy of each session’ (Section 4.2) below.
As we have described in Sec 4.2, unlike previous methods, we have conducted evaluations only after completing training on the last session. While previous approaches assess the performance for all classes encountered up to the current session after training at each session, we assess session-level adaptability by exclusively evaluating the classes relevant to a specific session only after training on the last session.

If we have misunderstood your question, please feel free to clarify, and we appreciate your understanding.

Reviewer comment (Experiments):
Whether the paper makes the comparison without using a pre-trained model.

Author response:
A: We conducted experiments in our pipeline without utilizing a pretrained backbone (the baseline is still ALICE). The results of the experiments are as follows.

| CUB200 |a_0 | Inc_avg | IMB | A_s |
|----------|----------|----------|----------|----------|
| w/o pretrained | 73.60 | 50.68 | 22.92 | 52.76 |
| Ours | 79.54 | 63.77 | 15.77 | 65.20 |

| Flowers102 |a_0 | Inc_avg | IMB | A_s |
|----------|----------|----------|----------|----------|
| w/o pretrained | 94.92 | 84.84 | 10.08 | 85.96 |
| Ours | 96.42 | 94.11 | 2.31 | 94.37 |

| DCC |a_0 | Inc_avg | IMB | A_s |
|----------|----------|----------|----------|----------|
| w/o pretrained | 76.94 | 44.10 | 32.84 | 49.57 |
| Ours | 80.97 | 55.00 | 25.97 | 59.33 |

The results show that using only the baseline backbone exhibits biased performance towards the base session. This reaffirms the significant assistance that the pretrained backbone provides in adapting to incremental sessions.

[ref.1] Zhirong Wu et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR 2018.