Paper: Adaptive Knowledge Transfer for Generalized Category Discovery (ICLR-2024)

Reviewer comment (Writing):
I believe the idea of knowledge utilization is good, but the proposed terms seem a bit of "big" to me. Technically speaking, knowledge generation, alignment and distillation are respectively training on labeled data, filtering out the important feature dimensions and applying contrastive loss. If there is no obvious significance, I would suggest using more focused terms that better convey the precise purpose.

Author response:
Thank you for your clarification. We aim to convey a clear overview of explicit knowledge transfer within GCD problems. While we instantiate knowledge generation, alignment, and distillation using a known-class pre-trained model, AL and CSM, and nMI, our framework allows for flexibility and alternative methods can be employed for these components. For instance, as outlined in the paper, the contrastive loss we utilize can be replaced by other knowledge distillation methods. The primary objective of this writing is to elucidate the explicit knowledge transfer framework for GCD problems, highlight the core challenges associated with knowledge transfer in GCD problems, and offer insights for future research.

Reviewer comment (Experiments):
As far as I am concerned, the most important component is regarding the "knowledge distillation" (as shown in Tab. 4), where an InfoNCE-like loss is used to do the trick. Within this loss, the MixUp-based negative sample generation seems an important ingredient, yet with no proper ablation studies. Given the potential similarity between the proposed generation procedure and the ones used in [a,b], the authors should provide more theoretical and empirical insights on the difference and significance.

Author response:
We respectfully disagree with you on three points:
1. Knowledge distillation is not the most important component of our method. It holds equal importance with other components, AL and CSM. The ablation study in Tab.4 demonstrates that each component significantly contributes to the model's improvement. As discussed in the paper, not all knowledge in known class data is beneficial for novel class learning. This indicates that nMI does not always enhance the model's performance, as supported by the evidence in Tab.12 in Appendix G. This underscores the importance of AL and CSM.
2. We would like to clarify that we do not claim novelty in negative sample generation within the paper, which is widely used across various contrastive learning domains. All the other negative sample generation strategies that avoid class collision issues like [2, 3] can also be applied to our method. 
3. Negative sample generation is not an important component in our method. As shown in the table below, the model's performance without the "negative sample generation" technique remains satisfiable. The reason we opted to use "negative sample generation" is to address the class collision issues of novel classes, as evidenced in the presented results.

| Method | CUB | | | StanfordCars | | | Aircraft | | |
|-------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|
|                         | All                      | Seen                              | Novel                        | All  | Seen | Novel | All  | Seen | Novel |
| Ours w/o NSG                 | 67.2                     | 73.2                              | 64.2                         | 58.9 | 80.2 | 48.6  | 54.1 | 60.3 | 51.0  |
| Ours                     | 67.1                     | 73.7                              | 63.8                         | 59.2 | 79.1 | 49.6  | 55.9 | 60.7 | 53.6  |

Reviewer comment (Presentation):
In Fig. 6, does the visualizations mean that there are more channels transferred in generic datasets like ImageNet, yet less channels transferred in fine-grained datasets?

Author response:
We would like to clarify that Fig. 6 illustrates the average outcome of the Channel Selection Matrix generated for each sample in the corresponding class. It can well reflect whether CSM can produce a general selection method for a specific class. However, it does not represent the number of open channels. Additionally, it's worth noting that the values represented by the color depth of each subfigure in Fig. 6 differ, with the maximum value of 0.002 on ImageNet being much smaller than observed on several other datasets. Here, we present the average number of open channels on novel classes in the table below, indicating that fewer channels are transferred in generic datasets, while more channels are transferred in fine-grained datasets.

|                                    | CUB                  | StanfordCars         | Aircraft             | CIFAR 100-80         | ImageNet100         |
|:-----------------------------------:|----------------------|----------------------|----------------------|----------------------|----------------------|
| #open channels | 316                    | 284                    | 254                    | 224               | 245 |

Reviewer comment (Novelty):
The proposed techniques are well-known in the field and I have the feeling that they are not specifically designed for the GCD problem.

Author response:
We would like to clarify the novelty of our method and its close relationship with the GCD problem in three points: 
1. Our framework represents a novel approach to addressing GCD challenges, specifically designed for explicit knowledge transfer in GCD tasks. To our best knowledge, no similar framework has been proposed previously for GCD tasks. As discussed in the introduction, knowledge generation, knowledge alignment, and knowledge distillation in our framework aim to generate known class knowledge representation, facilitate more targeted knowledge transfer, and perform knowledge transfer, respectively. Each component is specifically designed for the task of explicit knowledge transfer in GCD.
2. The techniques proposed in our work are intricately aligned with this innovative framework. Our proposed AL and CSM are unique solutions crafted specifically to tackle knowledge transfer challenges in the GCD problem. AL aligns the known-class representation space to the joint representation space, and CSM selects meaningful channels, enabling more targeted knowledge transfer. To the best of our knowledge, no such techniques have been proposed before.
3. We utilize the contrastive loss to maximize the mutual information. The contrastive loss we introduce is not a traditional one; rather, it uniquely combines AL and CSM, tailoring it to the intricacies of the GCD problem. This novel contrastive loss enables effective knowledge transfer between known and novel classes, unlike traditional knowledge transfer methods.

Reviewer comment (Evaluation):
The experimental results on some datasets are much better than the others but worse on CUB, It would be good to discuss the possible reasons why this is happening.

Author response:
We would like to point out that the goal of GCD is to learn a model, that can better classify known classes and cluster novel classes, concurrently. Since most existing methods can adapt the model's learning focus by adjusting the weight of $\mathcal{L} _ {u}$ in Equ (1), it is not adequate to evaluate the performance of the model based only on a single metric in the GCD problem. A case in point is the CUB dataset, where the DCCL method exhibits superior performance on novel classes compared to all other methods, yet its performance on known classes is inferior. This observation suggests a potential bias in the DCCL method towards novel class data. Unfortunately, comprehensive analysis is difficult as DCCL has not released its code. When considering all metrics comprehensively, our model is also better than other models on CUB.

Reviewer comment (Novelty):
The contrastive loss used in the paper is quite common and it would be good to further discuss the novelty of the method.

Author response:
We would like to clarify the novelty of our method and its close relationship with the GCD problem in three points: 
1. Our framework represents a novel approach to addressing GCD challenges, specifically designed for explicit knowledge transfer in GCD tasks. To our best knowledge, no similar framework has been proposed previously for GCD tasks. As discussed in the introduction, knowledge generation, knowledge alignment, and knowledge distillation in our framework aim to generate known class knowledge representation, facilitate more targeted knowledge transfer, and perform knowledge transfer, respectively. Each component is specifically designed for the task of explicit knowledge transfer in GCD.
2. The techniques proposed in our work are intricately aligned with this innovative framework. Our proposed AL and CSM are unique solutions crafted specifically to tackle knowledge transfer challenges in the GCD problem. AL aligns the known-class representation space to the joint representation space, and CSM selects meaningful channels, enabling more targeted knowledge transfer. To the best of our knowledge, no such techniques have been proposed before.
3. We utilize the contrastive loss to maximize the mutual information. The contrastive loss we introduce is not a traditional one; rather, it uniquely combines AL and CSM, tailoring it to the intricacies of the GCD problem. This novel contrastive loss enables effective knowledge transfer between known and novel classes, unlike traditional knowledge transfer methods.

Reviewer comment (Evaluation):
In Table 4, the inclusion of AL appears to result in a performance decline on the Scars benchmark, yet the authors provide no explanation for this, which makes me question the generalizability of the method.

Author response:
Firstly, concerning AL, its objective is to transform the representation space of known classes into a joint representation space that is more advantageous for novel classes. However, because the transformation introduces novel classes, it may have a negative impact on known classes, depending on the relationship between known and novel classes, which is dataset-specific. We further conduct an ablation study on the coarse-grained dataset in Appendix G, where we also observe a performance drop in known classes and an increase in novel classes when applying AL. The potential negative impact of AL underscores the importance of the design of the CSM. It is noteworthy that the features obtained after both AL and CSM can be considered beneficial for novel class learning. This observation is further supported by the results of the ablation study. Secondly, we would like to clarify that our model integrates not only AL but also nMI and CSM. The utilization of all three components collectively results in notable improvements across all known and novel classes in all datasets. This underscores the method's robust generalizability.

Reviewer comment (Presentation):
In Table 5, the distinction between AL and ours seems to be inadequately clarified, rendering this part somewhat hard to follow.

Author response:
Apologies for any confusion. In Table 5, we assess the model with a frozen known-class pre-trained encoder, adapter layer, and cluster head (denoted as 'AL') to evaluate the quality of the Adapter Layer. The adapter layer and cluster head are learned by $\mathcal{L} _ {base}$. When the number of layers is set to 0, it means we only learn a cluster head. "Ours" denotes our final model with all components. Importantly, when the number of layers is set to 0, "Ours" directly applies nMI without AL and CSM modules.

Reviewer comment (Evaluation):
I wonder why there is a observed decline in results on the Scars dataset following the incorporation of the Adapter Layer (AL). Is this reduction in performance linked to specific characteristics of the dataset itself, or are there elements of the method that may not be as effective when applied to Scars?

Author response:
Firstly, concerning AL, its objective is to transform the representation space of known classes into a joint representation space that is more advantageous for novel classes. However, because the transformation introduces novel classes, it may have a negative impact on known classes, depending on the relationship between known and novel classes, which is dataset-specific. We further conduct an ablation study on the coarse-grained dataset in Appendix G, where we also observe a performance drop in known classes and an increase in novel classes when applying AL. The potential negative impact of AL underscores the importance of the design of the CSM. It is noteworthy that the features obtained after both AL and CSM can be considered beneficial for novel class learning. This observation is further supported by the results of the ablation study. Secondly, we would like to clarify that our model integrates not only AL but also nMI and CSM. The utilization of all three components collectively results in notable improvements across all known and novel classes in all datasets. This underscores the method's robust generalizability.

Reviewer comment (Presentation):
Some experiments and tables in this paper lack explanation and details. For example, the "BL" and "Clu" settings in Table 1 require further clarification.

Author response:
“BL” is the baseline model. In detail, the baseline model has the same architecture as the final model and only uses $\mathcal{L} _ {base}$ in Equ (1) to learn known and novel classes. The loss can influence both the encoder and the classifier. Meanwhile,  “Clu.” denotes that we directly cluster the features generated by the pre-trained known-class model. The model of “Clu.” is also the model with the same architecture as the final model and uses the same $\mathcal{L} _ {base}$ to learn known and novel classes. But now, the encoder is frozen, meaning the loss can only influence the classifier. In Appendix H, we also discuss the details of clustering.

Reviewer comment (Experiments):
The specification of the baseline model used in the ablation study needs to be provided.

Author response:
This baseline model is the same model “BL” that we discussed above.

Reviewer comment (Reproducibility):
Implementation details of crNCD are missing.

Author response:
As crNCD is originally designed for NCD problems, we have adapted it to the GCD setting using a widely employed architecture utilized in other GCD methods. Additionally, to mitigate the influence of different self-labeling designs, we employ the same $\mathcal{L} _ {base}$ (Equ.1) as in our method, instead of using the original $\mathcal{L} _ {opt}$ from crNCD. The original $\mathcal{L} _ {opt}$ in crNCD did not perform well in the GCD setting, as indicated in the table below. It is important to note that we have made no modifications to the core loss $\mathcal{L} _ {rKT}$.

| Method         | CUB | | | StanfordCars | | | Aircraft | | |
|---------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|
|                                 | All                      | Seen                              | Novel                        | All  | Seen | Novel | All  | Seen | Novel |
| $\mathcal{L} _ {opt}$    + crNCD | 56.9                     | 57.8                              | 56.4                         | 46.6 | 62.3 | 39.1  | 45.2 | 59.0 | 43.4  |
| $\mathcal{L} _ {base}$ + crNCD        | 64.1                     | 75.2                              | 58.6                         | 54.8 | 76.5 | 44.3  | 53.1 | 57.0 | 51.3  |

Reviewer comment (Novelty):
The key idea of knowledge distillation under the NCD setting has been explored in the literature, for example [1, 2], while discussion and analysis are missing.

Author response:
We would like to clarify the distinctions between our method and those proposed in [1,2]. We note that a detailed discussion of [2] has been provided in the related work section. 

- Here we summarize the three main differences between our methods and [2] for clarity. 
  1. Our paper proposes a novel explicit knowledge transfer framework that introduces a fresh perspective for advancing knowledge transfer and [2] is only one of the concrete implementations of our proposed framework. Compared with the limited form of knowledge transfer in [2], our framework is more flexible and effective.
  2. [2] performs knowledge distillation on the output space, while we conduct it in the feature space which contains more information. [2] does not employ knowledge selection to filter out harmful knowledge or knowledge alignment to transfer knowledge more efficiently; instead, they use a simple weight function to amplify knowledge distillation when class relations are strong. 
  3. [2] is specifically designed for the NCD problem and it achieves poor results in GCD (table above). 
- Regarding [1], the primary objective of their loss function $\mathcal{L} _ {sKLD}$ is to enable the model to capture both global and local information within an image. In contrast, our method is specifically designed to facilitate the transfer of known class knowledge to novel class learning. Notably, [1] employs a two-branch learning framework, with one branch focusing on local part-level information and the other on overall characteristics. They transfer knowledge between these two branches. They do not require knowledge alignment because they perform knowledge distillation in the same label space. In contrast, our approach employs an upper branch dedicated to extracting valuable known class knowledge and proposes AL and CSM to do effective knowledge transfer in different representation spaces. These fundamental differences in the purpose and methods distinguish our work from [1].

Reviewer comment (Theory):
The paper lacks an explanation of how the naive Mutual Information loss is derived in the format of InfoNCE from the objective of mutual information.

Author response:
Thank you for your suggestion. In response, we provide the derivation of infoNCE from mutual information and the details can be found in Appendix A.

Reviewer comment (Experiments):
Herbarium is also a commonly used dataset to evaluate the performance of GCD. However, this is missing in the experiments.

Author response:
We supplement the results on the Herbarium19 dataset, which is a long-tailed dataset. The results show we achieve comparable results on the Herbarium19 with competitive SimGCD, while PromptCAL, DCCL, and crNCD do not report results on the Herbarium19 dataset.

| Method | Herbarium | | |
|-------------------------|-------------------------------|-------------------------|-------------------------------|
|                         | All                           | Seen | Novel |
| K-means                 | 12.9                          | 12.9 | 12.8  |
| Rankstats               | 27.9                          | 55.8 | 12.8  |
| UNO                     | 28.3                          | 53.7 | 14.7  |
| ORCA                    | 20.9                          | 30.9 | 15.5  |
| GCD                     | 35.4                          | 51   | 27    |
| OpenCons                | 39.3                          | 58.9 | 28.6  |
| PromptCAL               | -                             | -    | -     |
| DCCL                    | -                             | -    | -     |
| SimGCD                  | 43.3                          | 57.9 | 35.3  |
| crNCD                   | -                             | -    | -     |
| Ours                    | 43.0                          | 56.2 | 35.9  |

Reviewer comment (Experiments):
The paper mentions training the model for 20 epochs on labeled data in the first stage, but it lacks an explanation for choosing this specific number. It would be beneficial to provide a rationale for selecting 20 epochs and elaborate on how this choice impacts the results.

Author response:
We employ the "early stopping" technique and have observed that the model's loss and accuracy on the validation set stabilize after 20 epochs. Here the model is the known-class pre-trained model.

Reviewer comment (Experiments):
In the ablation study concerning the adapter layer, each layer is composed of a linear layer and a ReLU layer. It would be helpful to include an experiment using only one ReLU layer to assess the impact of this specific component.

Author response:
Thank you for your suggestion. We add it in the revised version of the paper. As shown in table below, if we use only a ReLU layer in the adapter layer, the model will experience a significant drop in all datasets. The decline can be attributed to the inherent limitation of a single ReLU layer, which lacks the capacity required for meaningful feature transformation, ultimately resulting in information loss. This observation emphasizes the critical role of transforming the original representation space into a joint representation space, highlighting the importance of the adapter layer in extracting meaningful information.

| Method | CUB | | | StanfordCars | | | Aircraft | | |
|-------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|
|                         | All                      | Seen                              | Novel                        | All  | Seen | Novel | All  | Seen | Novel |
| ReLU                    | 65.9                     | 73.3                              | 62.1                         | 56.4 | 76.0 | 47.0  | 54.7 | 62.6 | 50.7  |
| ReLU + Linear           | 67.1                     | 73.7                              | 63.8                         | 59.2 | 79.1 | 49.56 | 55.9 | 60.7 | 53.6  |

Reviewer comment (Experiments):
Table 4 exclusively utilizes fine-grained datasets. To provide a more comprehensive evaluation of the proposed method, it would be also important to analyze each component on a coarse-grained dataset.

Author response:
We appreciate your suggestion and have appended the ablation on the coarse-grained dataset to Appendix G. The observation in the table below regarding the coarse-grained dataset reveals a significant drop in novel class performance after using nMI. We attribute this to two main reasons:

- The representation space initialized with known classes has not encountered novel classes during training, potentially introducing bias in the final knowledge space towards known classes. This motivated the design of AL in our approach.

- The knowledge in known class data is not universally useful, and some information may even be harmful to novel class learning. This phenomenon is more pronounced in coarse-grained datasets. Consequently, we introduce CSM to address this issue by filtering out harmful knowledge. We provide a detailed analysis of this phenomenon in the subsequent section. 

The table results highlight the limitations of directly using nMI, particularly in scenarios with a large semantic gap between classes. The subsequent improvement seen after employing AL and CSM underscores their efficacy in enabling more targeted knowledge transfer for novel class learning.

| nMI | AL | CSM | CIFAR100-80 | | |
|----------------------|---------------------|----------------------|-------------------------|-------------------------|-------------------------|
|                      |                     |                      | All                     | Known | Novel |
|                      |                     |                      | 79.2                    | 78.7  | 80.0  |
|    &#10004;       |                     |                      | 79.9                    | 84.6  | 70.5  |
| &#10004;          | &#10004;         |                      | 82.2                    | 84.1  | 78.3  |
| &#10004;          | &#10004;         | &#10004;          | 82.8                    | 84.0  | 80.3  |

Reviewer comment (Experiments):
The paper mentions the usage of techniques such as dynamic conception generation [4] and cluster size regularization [5]. Analyzing the effects of these techniques would further reveal contributions to the overall performance of the proposed method.

Author response:
Firstly, we have to clarify that we do not utilize the "dynamic conception generation"[10] technique. Secondly, the cluster size regularization technique is widely employed in unsupervised learning[5, 6] and GCD[7, 8, 9], and it is not our contribution. It ensures meaningful clustering by preventing the model from grouping all data into a single cluster. Therefore, without cluster size regularization, our model and most other GCD models will decrease greatly. Here, we also present the model's performance without cluster size regularization (CSR) in the table below. This table confirms the importance of cluster size regularization as discussed earlier. Nevertheless, our method can still contribute to performance improvement even without cluster size regularization.

| Method | CUB | | | StanfordCars | | | Aircraft | | |
|-------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|
|                         | All                      | Seen                              | Novel                        | All  | Seen | Novel | All  | Seen | Novel |
| Baseline w/o CSR    | 39.6  | 47.6  | 35.6  | 33.2  | 59.3  | 20.6  | 35.6  | 37.5  | 34.7 |
| Ours w/o CSR  | 41.4  | 45.9  | 39.2  | 37.6  | 55.6  | 28.8  | 37.2  | 37.7  | 36.9 |
| Baseline            | 61.7  | 68.0  | 58.5  | 49.6  | 56.3  | 46.2  | 51.8  | 71.9  | 42.0 |
| Ours          | 66.8  | 75.6  | 62.5  | 55.6  | 60.5  | 53.1  | 57.6  | 75.9  | 48.8 |

Reviewer comment (Experiments):
A potential weakness of the paper could be the use of ReLU activations, which are known for their "dead neuron" issue, potentially leading to some neurons becoming inactive due to poor initialization. This characteristic of ReLU could result in the unintentional filtering out of certain channels that might otherwise be useful for learning a more robust embedding space.

Author response:
We opt for the ReLU activation function due to its simplicity and efficiency. Following your insights, we have conducted an analysis to count the number of dead neurons in the model, and the results are presented in the table below. Fortunately, our final model utilizing ReLU does not exhibit this issue. We believe this is potentially due to our truncated norm initialization. In the subsequent discussion, we will delve into a comparative analysis of the model's performances when employing ReLU and GeLU.

|                                    | CUB                  | StanfordCars         | Aircraft             | CIFAR 100-80         |
|:-----------------------------------:|----------------------|----------------------|----------------------|----------------------|
| #dead neurons | 0                    | 0                    | 0                    | 0               |

Reviewer comment (Experiments):
How would the model's performance be impacted if activation functions other than ReLU, such as GeLU, were utilized?

Author response:
(1) We appreciate your attention to the activation function choice. As depicted in the table below, adopting GeLU resulted in marginal changes in model performance. These experimental findings indicate that both GeLU and ReLU deliver comparable performances, providing evidence that there are few or no dead neurons in our model. Combining this with the empirical results mentioned above, we maintain that ReLU remains a straightforward and effective choice under a good initialization.

| Method | CUB | | | StanfordCars | | | Aircraft | | |
|-------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|--------------------------|-----------------------------------|------------------------------|
|                         | All                      | Seen                              | Novel                        | All  | Seen | Novel | All  | Seen | Novel |
| GeLU                    | 66.3                     | 73.8                              | 62.5                         | 58.3 | 77.6 | 49.0  | 56.2 | 61.0 | 53.8  |
| ReLU                    | 67.1                     | 73.7                              | 63.8                         | 59.2 | 79.1 | 49.6  | 55.9 | 60.7 | 53.6  |

Reviewer comment (Presentation):
Regarding Figure 2, is the 'Cls' in the lower branch designed exclusively for labeled samples? Additionally, in the upper branch, what mechanism allows the model to generate categories for the unknown classes after the classification step?

Author response:
(2) In Figure 2, the 'Cls' in the lower branch and the upper branch both can classify labeled and unlabeled data. The 'Cls' in the upper branch is learned based on $\mathcal{L} _ {base} $  in Equ (1). The loss term $\mathcal{L} _ {u} $ in $\mathcal{L} _ {base} $ is the self-labeling loss. Specifically, it utilizes the predictions of 'Cls' in one view of the image to generate pseudo-labels for the other view of the same image. This mechanism enables the model to assign categories to unlabeled data. As a result, the 'Cls' component can classify both known and novel classes. For a more in-depth explanation of $\mathcal{L} _ {u}$, we invite you to refer to Appendix D. Your interest is greatly appreciated.