Paper: PromptCCD: Learning Gaussian Mixture Prompt Pool for Continual Category Discovery (ICLR-2024)

Reviewer comment (Theory):
In the proposed method, the top-k "means" are treated as prompts. However, how should one interpret this? I am unclear about how adding the mean of the GMM can enhance the guidance provided to the backbone. It's worth noting that the GMM is detached from the backbone. Consequently, learning in the GMM module is somewhat disconnected from the optimization process. This is akin to integrating additional information, such as features, into the system.

Author response:
The motivation of our proposed model is to design a prompt module that can guide or instruct a self-supervised vision foundation model for better CCD. Moreover, not only can it guide the model, but we also believe there is a need to design a prompt module that suits the open-world nature of CCD with unlabelled data that contains unseen categories over time. Thus, intuitively we choose the Gaussian mixture model (GMM) as the candidate for our prompt design. By using the GMM mean component, it acts as a class prototype (to accommodate the absence of label information) which guides/enforces the model about the criteria / to look at (information/features that the model should care about) when making the decision to discover a category. We further show in **Appendix I**, that our GMM coupled with self-supervised vision foundation models actually produce prompts that are discriminative within different classes. Moreover, we conduct an experiment in Table 14, where instead of taking the “top-k” mean components/prompts, we vary the prompts “random-k” to observe the effect of difference prompt relevancy towards overall CCD performance. We can see that varying the prompt hurts the model’s performance, especially for the "NEW" ACC i.e., novel categories.

Reviewer comment (Novelty):
The authors claim that 'these prompts facilitate adaptation to emerging data, thus preventing catastrophic forgetting.' However, in subsequent sections, the issue of forgetting is addressed by drawing samples from previous stages and reusing them in the second stage. This approach is reminiscent of a replay mechanism.

Author response:
Indeed, our method can be viewed as a replay method, unlike existing methods, it does not need to store any samples, but the parameter-efficient GMM can be used to generate an infinite number of replay samples on the fly. Also,  we use our replay differently than the usual replay-based method in the continual learning literature. Here the replay data is generated randomly by our GMM in each stage for each component and will be used to fit the next GMM which eventually builds/transfers previous GMM components which eventually leads to the formation of component means a.k.a prompts into the current stage GMM. Thus, by prompting these “previous components” from the previous replay samples, our model can tackle catastrophic forgetting.

Reviewer comment (Evaluation):
Concerning the estimation of the novel class number, Table 3 provides interesting insights. Apart from the fine-grained CUB200, the estimation remains relatively consistent, irrespective of the class number across different stages. Even though the estimation does not align perfectly with the ground truth, the final performance might actually surpass scenarios where the class number is known, as illustrated by the comparison between Tiny200 in Tables 2 and 3. Has the study provided any rationale or explanation for this observation?

Author response:
Thank you very much for this comment. We followed this suggestion to investigate the cause of this phenomenon. While investigating the phenomenon between the TinyImageNet results obtained from both **Table 2 and Table 3**, we discovered a technical flaw in Table 3’s experiments Specifically, an incorrect training configuration was mistakenly loaded for training when using the GCD’s category number estimator. We have loaded the correct training configuration to retrain all models and updated the results in Table 3 accordingly. Based on the comparison of our model with other models in the unknown K settings in Table 3, we can see that our model outperforms all other models on all datasets under all metrics, except for the “OLD” accuracy of the Imagenet100 dataset. By comparing our proposed method in both Tables 2 and 3, we can observe that our proposed model has a slight decrease in accuracy for the “OLD” class when applying the estimated class number, while the “NEW” accuracy decreases more. This could be due to the prior introduced by the SS-Kmeans, which favours the “OLD” categories more. In addition, we conducted another experiment in **Appendix D, Table 7**, where we seamlessly adopted the GMM-based dynamic category number estimator into our CCD framework. We would like to emphasize that Table 7 aligns better with our proposed method and is therefore more preferred. As a result, we will swap Table 3 and Table 7 in our final manuscript.

Reviewer comment (Experiments):
Why is the final block of the backbone updated? Prompt learning aims to use prompts to 'instruct' the foundational model to access rich information without modifying it, as seen in methods like L2P and VPT[1]. It's worth noting that updating the backbone is geared towards acquiring new 'knowledge'. Is the overall performance improvement primarily attributed to the model being tailored to a specific dataset, or is it due to the 'instruction' provided by prompt learning?

Author response:
Our model $H_{\theta}: \{\phi, f_{\theta}\}$ i.e., the self-supervised vision foundation model (in this case DINO) consists of $\phi$, an MLP projection head, and $f_{\theta}=\{f_e, f_b\}$ a transformer-based feature backbone that includes an input embedding layer $f_e$ and self-attention blocks $f_b$. During training, we only optimize $\phi$ and the final block of $f_b$. After training, we only require $f_{\theta}$ to extract the token classification i.e., $z\texttt{[CLS]}$ feature is used for clustering. If we fully freeze the backbone $f_b$, then unlike the L2P supervised continual learning implementation which has a final learnable fully connected layer for classification, our model will eventually not learn anything and act as a frozen DINO. Therefore, we further optimize the final block of the backbone to prevent this issue. For reference, we provide comparison results between frozen DINO, frozen DINO with learned L2P prompt pool and our proposed model in **Appendix H**. We can see that Our fine-tuned model outperforms the frozen models in all ACC metrics and for all datasets. Moreover, our fine-tuned model also appears to generalize better to data distributions (C100 & CUB200 datasets) that the DINO foundation model has not encountered before, which further justifies the design choice of our method for CCD.

Reviewer comment (Writing):
The description in the GMP module is somewhat confusing: If I've interpreted it correctly, both the letter 'C' in Section 2.2 and the letter 'K' in Section 3.2 represent the number of classes. If that's the case, consistency in notation would be appreciated. Are the top-k prompts selected from among these C/K Gaussian components?

Author response:
1. We thank the reviewer for their careful review,  the number of components in GMM is similar to the number of categories i.e, $C$ and the top-K prompts are selected among these $C$ Gaussian components.  
2. To optimize GMM, we make use of classification token $z\texttt{[CLS]}$ extracted in the “gradient detach” mode of our own backbone $f_{\theta}$ as GMM samples.  
3. Based on the ablation study the optimal number of samples is 100 samples per component.  
4. For clarity we further provide a pseudo code of PromptCCD w/GMP model training in **Appendix A**.

Reviewer comment (Reproducibility):
The method for obtaining GMM samples (as represented by the dots in Fig. 4) for optimizing the GMM is unclear.

Author response:
2. To optimize GMM, we make use of classification token $z\texttt{[CLS]}$ extracted in the “gradient detach” mode of our own backbone $f_{\theta}$ as GMM samples.

Reviewer comment (Experiments):
How many samples advance to subsequent stages?

Author response:
3. Based on the ablation study the optimal number of samples is 100 samples per component.

Reviewer comment (Reproducibility):
Which query function is used? I assume the cls token of the backbone output is used as in L2P?

Author response:
As explained in **R-si3G-Q6 Clarifications of our GMP module sub-section (2)** we make use of our own backbone model $f_{\theta}$ as the query function to extract the classification token $z\texttt{[CLS]}$.

Reviewer comment (Novelty):
The novelty of the paper is mainly on the proposed Gaussian Mixture Prompt Pool, which is rather limited. The authors argue that "the fixed-size prompt module restricts the model's ability to select prompts to a maximum of $M$", but the proposed Gaussian Mixture Prompt Pool seems also a fixed-size prompt. In addition, what is the advantage of GMM compared to K-means, if we also store mean and covariance for replaying?

Author response:
Our Gaussian mixture prompt pool (GMP) differs from L2P and DualPrompt prompt pool in three ways. First, GMP parameters are not fixed as they depend on the number of C components (as shown in **Appendix H**), which is crucial in CCD tasks where the number of classes in the unlabelled data can grow or reduce over time. Second, we argue that GMM is advantageous over K-means when working with high/complex dimensional data. By using GMM, we allow a sample to have multiple memberships towards different cluster components, i.e., non-convex clusters. K-means assumes that the probability a sample belongs to a cluster is one, which in real-world data might not be true. Third, we show in **Appendix I** that GMM outperforms K-means for all clustering metrics, which means better prompt quality. Therefore, it is natural to choose GMM as a pool of prompts. For further discussion, please refer to **General response - Clarification on L2P & DualPrompt vs Our GMP**.

Reviewer comment (Experiments):
The conclusion in ablation studies is incoherent. When the prompt selection is set to 5, and GMM sampling is set to 100, the "Spherical" outperforms "Diagonal". But the optimal number of prompts seems to be 10. The conclusion of ablation studies should be adjusted.

Author response:
Our model’s GMP ablation studies consider three factors: covariance type for GMM, number of prompts, and number of GMM sampling for replay. **Table 4** shows that the optimal GMM covariance types for CIFAR100 and CUB200 datasets are “Spherical” and “Full,” respectively, with five prompts and 100 GMM samples. Although there is some ambiguity on the covariance type, we set the default configurations to be “Diagonal” for the GMM covariance type, five prompts, and 100 GMM samples, which appears to be a good trade-off. We further perform an additional ablation study in **Appendix K** where we keep all covariance of GMMs to be spherical. The results show that 5 prompts and 100 samples lead to the best “ALL” accuracy

Reviewer comment (Presentation):
The method is described with an excessive level of detail that, while potentially beneficial, also burdens the reader with information overload. A more concise presentation, perhaps through structured pseudocode similar to that provided for the evaluation, could clarify the methodology more effectively.

Author response:
We thank reviewer 1nUF for their suggestions. We have revised the manuscript with updated  **Figures 2 & 3** with added captions and provide the pseudo-code to explain the overall process in **Appendix A**.

Reviewer comment (Presentation):
The depiction of the method across Figures 2-4 is spread over multiple figures with captions that do not sufficiently convey the information, thereby diluting the explanatory power of the visuals.

Author response:
We thank reviewer 1nUF for their suggestions. We have revised the manuscript with updated  **Figures 2 & 3** with added captions and provide the pseudo-code to explain the overall process in **Appendix A**.

Reviewer comment (Theory):
Given that a Gaussian Mixture Model (GMM) is employed, does the assumption of similarity between learning stages in conventional continual learning scenarios, which may vary greatly, potentially limit the model's capabilities?

Author response:
As shown in **Appendix I, Table 13**, by learning GMM as a pool of prompt, and finetuned the final layer of the backbone, we observe the performance on C100 and CUB200 datasets to be interesting, we see that our proposed model guided by GMM’s mean components generalized better (compared to frozen DINO backbone) to datasets that the DINO foundation model has not encountered before which further justifies the design choice of our method. This argument is further justified in **Appendix H**, where we show that GMM’s prompts represent class prototypes. Thus, our method does not limit the model’s capability when the task identity for each stage is different.

Reviewer comment (Experiments):
Considering that a portion of the data is initially labeled, is it possible to leverage this labeled subset to learn a preliminary GMM that could subsequently be integrated into the broader GMM for the entire dataset?

Author response:
We used Gaussian mixture models (GMMs) even during the initial learning with labelled data. The GMM learned the features from the labelled data and transferred this knowledge in the form of GMM random samples and fit them to the next GMM.

Reviewer comment (Experiments):
This method employs DINO as the pre-training model. Although a similar pre-training approach was also adopted in [1], the problems addressed are different. The Continuous Learning task has annotation information, so the utilization of DINO’s self-supervised model is justified. However, in the CCD problem, there is no annotation information, and most incremental tasks are self-supervised. For models that are unsupervised, and the training data may have overlaps, using DINO is not equitable.

Author response:
We would like to clarify that for L2P, they did not use a self-supervised pretrained model. Instead, they adopt a fully supervised pretrained model on very large-scale labelled data i.e., ImageNet-21K. In contrast, we use the fully self-supervised pretrained model DINO, which has been widely adopted in the static GCD task, by leveraging its strong feature generalization capability. DINO is a self-supervised (SSL) pretrained model (no labels are used during training). We want to point out that the SSL model has been widely adopted and justified in both NCD [**A**] and all GCD literature so far. It is a valid and reasonable choice to use such models because no supervision, such as class labels, was used to train the feature backbone.

Reviewer comment (Experiments):
The performance enhancement of this method stems more from DINO, and the paper lacks a comparison with methods without DINO pre-training models. The table in the paper does not specify the model's parameters used by different methods.

Author response:
We provide the table comparison of each model’s learnable parameters in **Appendix H**. Here we show that our model’s GMM parameters are flexible as we only need to define the number of components to scale it. Overall, compared to our baseline models (L2P and DualPrompt), Our proposed model’s learnable parameters are only 0.33% higher with category number C set to 100, which is still parameter efficient.

Reviewer comment (Experiments):
In Appendix C of the paper, the authors propose a method for dynamically adapting to the number of unknown categories, denoted as PromptCCD+. The method employed by PromptCCD in the main text to cope with unknown categories is to fix the number of categories to a predefined value.

Author response:
Yes, in the main table experiments i.e., **Table 2**, we fix the number of categories to a predefined value for all models and datasets for fair comparison.

Reviewer comment (Evaluation):
It would be beneficial to provide a direct comparison between the number of dynamically adapted unknown categories and the number of actual categories. This would offer a clearer insight into the effectiveness of the proposed method.

Author response:
We provide the tables for comparison between the predicted number of categories and the ground truth categories in **Table 3** (GCD [**B**] category number estimator) and **Appendix D** (Ours, by seamlessly adopting the GMM based dynamic category number estimator from GPC [**C**]). Overall, in this realistic scenario, our model still consistently outperforms other methods by a large margin across different datasets.