Paper: Meta-Collaboration in Distillation: Pooled Learning from Multiple Students (ICLR-2024)

Reviewer comment (Experiments):
While the idea of meta-learning for multi-teacher KD has potential, the claims require more empirical and analytical support. Broader experimentation, justification of design choices, and engagement with relevant studies would help validate the work's novelty and significance.

Author response:
Thank you for your insightful comments and for highlighting this clarity concern. We want to address a potential misunderstanding. This paper does not involve multi-teacher distillation; rather, it focuses on a single pre-trained teacher model, with all other models being student models that are concurrently undergoing training.

A significant contribution of this paper is the enhancement of knowledge distillation efficacy through the implementation of instance-wise weights, particularly in scenarios where a substantial learning capacity gap exists between the teacher model and the student model. In the conventional knowledge distillation framework, the final loss is a combination of cross-entropy loss and KL-divergence-based KD loss. Our objective is to learn weights for each of these components in an instance-wise manner. This approach allows us to concentrate on hard labels for specific data points and soft labels from the teacher for another set of points, thereby achieving an optimal trade-off between learning from the teacher model and the hard labels.

Additionally, beyond learning instance-wise and loss component-wise weights, this work aims to capitalize on the learning patterns of student cohorts. This is realized through the utilization of C-Net.

Reviewer comment (Experiments):
The evaluation on small datasets is insufficient. Following KD work norms, it should test on large-scale datasets like ImageNet and report downstream transfer learning results. The choice of teacher-student pairs could be better motivated by discussing alternatives like lightweight networks.

Author response:
We utilized teacher-student pairs from other papers addressing significant capacity differences between teacher and student models. While we were unable to complete the additional ImageNet experiments within the review period, we expect our method would enjoy significant gains over tranditional KD even on large scale datasets.

Reviewer comment (Novelty):
To properly situate this work in the rapidly advancing KD literature, it needs to discuss closely related recent papers [1,2,3,4,5,6] like those pointed out. These address self-supervised KD, representation matching, offline-online transfer, architecture search for distillation, and automated KD - all highly pertinent topics the paper does not engage with. A more comprehensive literature review would strengthen the paper.

Author response:
Thanks for pointing out the additional literature. However, again there seems to be some misunderstanding. The primary objective of this paper is to enhance the effectiveness of knowledge distillation through the utilization of instance-wise weights. In scenarios characterized by a substantial learning capacity disparity between the teacher model and the student model, knowledge distillation efficacy is frequently suboptimal and, at times, inferior to learning with hard labels. The papers referenced seem to improve distillation by introducing additional information or modifying the student model itself. Nevertheless, we will conduct a thorough comparison and contrast of our work with these referenced papers.

Reviewer comment (Novelty):
Critical missing related work on bidirectional knowledge distillation with multiple models. [A] Deep mutual learning. CVPR'18 [B] Dual learning for machine translation. NeurIPS'16 [C] Bidirectional Distillation for Top-K Recommender System, WWW'21 and on consensus learning with heterogeneous models. [D] Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering, WWW'22. These existing works, especially [A] and [C] should be compared theoretically and empirically in the manuscript.

Author response:
We present comparison with the additional baselines requested. We found “[C] Bidirectional Distillation for Top-K Recommender System, WWW'21 and on consensus learning with heterogeneous models.” to be specific to recommender systems and therefore compare against [1], another Bidirectional knowledge distillation method with is similar to our setup. 

[1] Li, Lujun, and Zhe Jin. "Shadow knowledge distillation: Bridging offline and online knowledge transfer." Advances in Neural Information Processing Systems 35 (2022): 635-649.


| Teacher                 | Student            | CE   | KD   | DML[A]  | SKD[1]  | Ours |
|--------------------------|---------------------|---------|------|-------------|------------|--------|
|| Resnet10_xxs | 31.85| 33.45| 33.41     | 31.59    | 36.19|  
| Resnet10_l ( 72.2) | Resnet10_xs  | 42.75| 44.87| 42.16      | 38.53 | 47.57 |
|                                | Resnet10_s   | 52.48  | 55.38| 51.13| 48.11| 58.36|           
|| Resnet10_m   | 64.28          | 66.93| 62.06| 56.97| 69.42|     
|--------------------------|---------------------|---------|------|-------------|------------|--------|      
|| Resnet10_xxs | 31.85          | 33.95| 32.04| 31.03| 35.97|           
| Resnet34  (79.47)   | Resnet10_xs | 42.75| 44.87| 42.47 | 39.56 | 47.53|
|| Resnet10_s   | 52.48          | 55.56| 51.22| 47.19| 58.1 |           
| |Resnet10_m   | 64.28          | 67.27| 63.12| 56.8 | 69.21|

Reviewer comment (Theory):
In Eq.7, in my opinion, the optimization of $\phi$ should be the outer loop, since Eq.7 is the optimization for C-Net.

Author response:
```
In Eq.7, in my opinion, the optimization of \phi should be the outer loop, since Eq.7 is the optimization for C-Net.
```
Due to the bi-level optimization formulation, the C-Net parameters undergo updates based on the most recent model parameters. Initially, we adjust the model (student) parameters using loss gradients derived from the weighting parameters acquired from C-Net. Consequently, the updated parameters depend on the weights obtained from C-Net. Subsequently, employing these updated parameter values, we calculate loss gradients for C-Net.

Reviewer comment (Novelty):
The novelty is limited in my view. This work follows the widely used online distillation framework, except that the proposed C-NET part is trained with meta-learning. There are no clear improvements to the framework.

Author response:
A primary contribution of this paper lies in enhancing the effectiveness of knowledge distillation by incorporating instance-wise weights in situations characterized by a significant disparity in learning capacity between the teacher and student models. In the typical framework of any knowledge distillation, the final loss is a combination of cross-entropy loss and KL-divergence-based KD loss. We aim to learn weights for each of these components on an instance-wise basis. This approach allows us to emphasize hard labels for specific data points and soft labels from the teacher for another set of points, facilitating an optimal trade-off between learning from the teacher model and the hard labels. This trade-off is crucial as diverse students exhibit varying learning capacities, and when a student model is considerably smaller than the teacher model, some learning from hard labels proves beneficial.

Beyond learning instance-wise and loss component-wise weights, our work seeks to leverage insights from student cohorts, inspired by online distillation. However, unlike traditional online distillation, there is no explicit matching imposed among the student models through a loss function. Instead, this information is implicitly shared among the student models through the utilization of a common C-Net.

Reviewer comment (Theory):
Can you provide further theoretical analysis or insights into how the meta-collaboration process influences the learning patterns of the student models?

Author response:
The effect on learning patterns of different student models while training with weights from C-Net like neural network is well studied in  [1]. The same setting is applicable here as well.
 
[1]Shu, Jun, et al. "Meta-weight-net: Learning an explicit mapping for sample weighting." Advances in neural information processing systems 32 (2019).

Reviewer comment (Novelty):
The novelty looks incremental. It looks to me that the paper combines offline KD and online KD by training multiple students simultaneously while learning the loss weights with the meta-learning strategy (bi-level optimization).

Author response:
Thank you for your insightful comments and for bringing attention to the clarity issue. We wish to address a potential misunderstanding. A key contribution of this paper is the enhancement of knowledge distillation efficacy through the incorporation of instance-wise weights. In scenarios where there exists a significant learning capacity gap between the teacher model and the student model, knowledge distillation efficacy is often suboptimal, and in some cases, inferior to learning with hard labels.

To alleviate this issue, our proposal involves the utilization of instance-wise and loss component-wise weights. Typically, in any knowledge distillation framework, the final loss is a combination of cross-entropy loss and KL-divergence based knowledge distillation (KD) loss. Our objective is to learn weights for each of these components on an instance-wise basis. This approach enables us to focus on hard labels for specific sets of data points and soft labels from the teacher for another set of points. This allows us to achieve an optimal trade-off between learning from the teacher model and the hard labels, a crucial consideration given the diverse learning capacities of different students. When a student model is significantly smaller than the teacher model, some learning from hard labels proves to be beneficial.  In addition to learning instance-wise and loss component-wise weights in this work we also want to benefit from learning patterns of the student cohorts. This is achieved via using C-Net.

Reviewer comment (Experiments):
It misses some ablation studies to show how the learned weights by C-NET are better than other weight strategies, e.g., manually set.

Author response:
Thank you for your insightful comments and for bringing attention to the clarity issue. We wish to address a potential misunderstanding. A key contribution of this paper is the enhancement of knowledge distillation efficacy through the incorporation of instance-wise weights. In scenarios where there exists a significant learning capacity gap between the teacher model and the student model, knowledge distillation efficacy is often suboptimal, and in some cases, inferior to learning with hard labels.

To alleviate this issue, our proposal involves the utilization of instance-wise and loss component-wise weights. Typically, in any knowledge distillation framework, the final loss is a combination of cross-entropy loss and KL-divergence based knowledge distillation (KD) loss. Our objective is to learn weights for each of these components on an instance-wise basis. This approach enables us to focus on hard labels for specific sets of data points and soft labels from the teacher for another set of points. This allows us to achieve an optimal trade-off between learning from the teacher model and the hard labels, a crucial consideration given the diverse learning capacities of different students. When a student model is significantly smaller than the teacher model, some learning from hard labels proves to be beneficial.  In addition to learning instance-wise and loss component-wise weights in this work we also want to benefit from learning patterns of the student cohorts. This is achieved via using C-Net.

Reviewer comment (Theory):
Why use PooledStudent logits as the supervision? Theoretical or empirical explanation is supposed to be provided.

Author response:
In the context of handling a knowledge distillation scenario characterized by a substantial disparity between the student and teacher models, it has been observed that incorporating outputs from intermediate models—those with learning capacities between the teacher and student models—enhances the knowledge distillation process. Consequently, we introduce the PooledStudent logits in conjunction with the teacher logits to amplify the knowledge transfer from the teacher model to the student model.  Here again, we control the participation of the PooledStudent logits in the training process using instance-wise loss-component-wise loss.  Further in Appendix B.1 we study the effect of the addition of PooledStudent logits.