Paper: A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation (ICLR-2024)

Reviewer comment (Novelty):
Given the previous work of Zhou et al. (2021), the presented method is not very novel, since the basic idea of teaching teachers is the same.

Author response:
**W1. The presented method is not very novel, Zhou et al. 2021 (MetaDistil) also proposes teaching the teacher:** Although our framework was motivated by Zhou et al. 2021 [1] (MetaDistil), there are several major drawbacks of MetaDistil, which our method, MPDistil, circumvents: 

* In the meta-loop of MetaDistil, the teacher model is trained to minimize the gap between the teacher and the student. Therefore, the teacher is always optimized for a low-performing student with a suboptimal objective. On the contrary, MPDistil formulates a competitive learning framework wherein both teacher and student models are encouraged to outperform each other.

* MetaDistil and most of the MetaKD techniques distil knowledge for different tasks in isolation, failing to leverage the commonalities across tasks. This phenomenon is counter-intuitive in real-world settings, where a learner studies other associated subjects (*e.g.,* Mathematics and Physics) for better learning. MPDistil encourages the student model to formulate suitable learning curricula using a suitable reward system, leading to more generalized and robust student performances.

The empirical results highlight the effectiveness of each of these components within MPDistil and demonstrate significant performance improvement over the contemporary KD and MetaKD methods.

Reviewer comment (Experiments):
The performance gain could be mostly attributed to a better teaching model. It would be better to conduct some cross-verifications by using the meta-teacher obtained in the proposed method as the teachers for other models.

Author response:
**W2. Gain attributed to a better teaching model. Conduct cross-verification by using meta teacher obtained from the method as a teacher for other baselines:** We distil the BERT student model using PKD (Sun et al., 2019 [2]) from the meta-teacher obtained from our method. On average, the student model distilled from the meta-teacher obtains $0.2$% higher accuracy than the student model distilled from the BERT-12L teacher model. Moreover, the statistical study reported in Section 5 also confirms the positive correlation between the meta-teacher and the student's performances. However, as shown in Tables 1-4, curriculum learning can aid an additional performance boost of $2.2$%. Therefore, competitive teacher training and student curriculum learning are equally responsible for robust student performance.

Reviewer comment (Experiments):
Larger language models are encouraged to be used in the experiments to better show the value of distillation.

Author response:
**W3. Distillation results using LLMs.** As suggested, we have empirically validated our method, MPDistil, with OPT-1.3B (Zhang, Roller and Goyal et al., 2022 [3]), a decoder-only LLM. We report the accuracy and $\Delta \text{Margin}$ on SuperGLUE dev tasks with OPT-1.3B model in Tables 12 and 13 of the updated paper, respectively. The average student-teacher margin remains at $-0.58$ with MPDistil, whereas the PKD baseline achieves an average margin of $-1.9$. Moreover, the student model distilled with MPDistil outperforms the teacher model on 2 out of 6 SuperGLUE tasks, indicating its effectiveness. Table 1b and Table 3 highlight the effectiveness of MPDistil with the DeBERTa-v2-xx-large (He et al., 2020 [4]) model, the largest encoder-only language model.

Reviewer comment (Experiments):
Have you tried using the meta-teacher as the teach in the baselines?

Author response:
**Q1. Have you tried using meta-teacher as the teacher in baselines?** We distil the BERT student model using PKD (Sun et al., 2019 [2]) from the meta-teacher obtained from our method. On average, the student model distilled from the meta-teacher obtains $0.2$% higher accuracy than the student model distilled from the BERT-12L teacher model. However, the performances of the PKD-distilled student still fall short of the student model distilled with MPDistil.

Reviewer comment (Experiments):
If my understanding is correct, it seems to me that the main benefits come from the “student curriculum learning" process that exploits data points from other tasks that are in general not available to the other baselines the authors compare against and the teacher model itself. In that sense, it is not so surprising that the student model outperforms the teacher and that the improvements margins are high enough — this should probably be made a bit more explicit by the authors. (Indeed, it is known that the student-model can certainly outperform the teacher-model in scenarios it has access to more examples, see e.g., [1, 2, 3]).

Author response:
**W1. Major benefits arise by exploiting the data points from other tasks in student curriculum learning:** We would like to clarify that the superior performance of the student model is attributed not only to the curriculum learning framework but also to the competitive joint optimization strategy. We can observe the performances demonstrated by the ablation model without curriculum learning as empirical evidence. Interestingly, the ablation model reports positive gain on four tasks, where the distilled student model outperforms the teacher model. Tables 2 and 3 highlight the competitiveness of the ablation model without curriculum learning on SuperGLUE tasks. Section 5 also highlights the results of the statistical studies performed on the meta-teacher and the student model. The study finds a strong positive correlation between the meta-teacher improvement and the student performance. Therefore, we can conclude that only the curriculum learning framework may not be sufficient for teaching a robust student unless the teacher is robust and generalized.

Reviewer comment (Presentation):
My major concern is that the paper is overall hard to follow. Specifically, in the introduction section, the authors list several challenges of MetaKD, while these challenges seem scattered and the major challenges are missed.

Author response:
**W1. Drawbacks in existing meta-KD techniques:** We are sorry for the confusion. Let us elaborate on the major drawbacks of the existing meta-knowledge distillation techniques -

* The major drawback of the existing MetaKD approaches is the optimization objective. In the meta-loop, the teacher model is usually trained to minimize the gap between the teacher and the student. Therefore, the teacher is always optimized for a low-performing student with a sub-optimal objective.

* Majority of the MetaKD techniques distil knowledge for different tasks in isolation, failing to leverage the commonalities across tasks. This phenomenon is counter-intuitive in real-world settings, where a learner studies other associated subjects (*e.g.,* Mathematics and Physics) for better learning.

Reviewer comment (Theory):
The proposed method consists of several steps and the authors make several adjustments in each step. However, these adjustments seem quite straightforward; I wonder the purpose of these designs and why they work.

Author response:
**W2. Purpose of various steps in our method:** Steps 3 and 4 of our proposed framework tackle the drawbacks, as mentioned in response to W1. In step 3 of our method, we formulate collaborative and competitive loss functions that encourage the meta-teacher model to perform better than the student. In step 4, we create a reward system that encourages the student to adopt suitable optimal curricula (sequences of tasks), helping to obtain better results than the teacher. Therefore, through this competitive framework, both teacher and student learn better.

Reviewer comment (Experiments):
Concerning the detailed design of the proposed method, the meta-teacher takes hidden state representation from both the teacher and student models to generate the final output. This could be an indirect learning from ground truth labels across the student models. Besides the efficiency advantages, I wonder whether it is more effective to directly use the parameter-efficient fine-tuning.

Author response:
**W3. Effectiveness of parameter-efficient fine-tuning:** As discussed in Section 3 of the paper, the primary objective of our designed meta-teacher is to act as a discriminator responsible for generating final class output irrespective of the representation learner. Under the competitive objective formulation, the meta-teacher aims to generate more correct labels for teacher-generated representations. Under the collaborative loss formulation, however, the meta-teacher jointly optimizes both teacher and student. Being a discriminator model, the meta-teacher does not learn the latent representations from original inputs but only consumes the representations obtained from other representation learning models. Therefore, we argue that parameter-efficient fine-tuning of the original teacher model can not discriminate the teacher and student models fairly and may be ineffective in a competitive setup.

Reviewer comment (Experiments):
The curriculum learning has been widely studied in the literature of KD and I wonder whether it is necessary to include it in the meta-learning-based KD. The authors should provide detailed explanation on this with more ablation studies.

Author response:
**W4. Necessity of curriculum learning in MPDistil:** Tables 1-4 contain the ablation model without curriculum learning. As elaborated in the main paper, MPDistil with curriculum learning outperforms the ablation model in 11 out of 13 tasks, with a significant margin of $2.2$%. The empirical evidence highlights the necessity of curriculum learning in meta-knowledge distillation.

Reviewer comment (Presentation):
Please clearly organize the challenges and motivation of this paper.

Author response:
**Q1. Challenges and motivation of the paper:** The major challenges with the existing meta-knowledge distillation techniques can be summarized as follows -

* The major drawback of the existing MetaKD approaches is the optimization objective. In the meta-loop, the teacher model is usually trained to minimize the gap between the teacher and the student. Therefore, the teacher is always optimized for a low-performing student with a suboptimal objective.

* Majority of the MetaKD techniques distil knowledge for different tasks in isolation, failing to leverage the commonalities across tasks. This phenomenon is counter-intuitive in real-world settings, where a learner studies different associated subjects (*e.g.,* Mathematics and Physics) for better learning.

Reviewer comment (Writing):
Concerning the detailed design of each part, what are they purposed for?

Author response:
**Q2. Purpose of step-3 and step-4 in the proposed methodology:** Steps 3 and 4 of our proposed framework tackle most of the drawbacks of the existing meta-knowledge distillation techniques. In step 3, we formulate collaborative and competitive loss functions that encourage the meta-teacher model to perform better than the student. In step 4, we create a reward system that encourages the student to adopt suitable optimal curricula (sequences of tasks), helping to obtain better results than the teacher. Therefore, through this competitive framework, both teacher and student learn better. To the best of our knowledge, the idea of training the teacher "better" has not been explored yet. In a real-world scenario, the teacher model might not always try to match the student's level for better knowledge transfer but can also try improving itself and the student model together and can even try improving by maximizing the margin from the student.

Reviewer comment (Evaluation):
The framework is very heavy since it consists of bi-level optimization and reinforcement learning loss.

Author response:
**W1. The framework is very heavy:** We would like to clarify that our proposed method is one of the most optimized generalized meta-knowledge distillation methods. Traditional meta-KD techniques (e.g., MetaDistil proposed by Zhou et al., 2021 [1] ) follow a bi-level optimization to fine-tune the teacher model in a meta loop. Section 3 of our paper highlights that our formulated meta teacher has only $\sim 0.001$% parameters than the original teacher model. This makes our framework adaptable, even for large language models. Secondly, in MetaDistil, the inner-learner model is trained on the training split, whereas, in our framework, we fine-tune the student curriculum model on a much smaller (only $10$% of the original training data) quiz dataset. Therefore, in a comparable setting, the total number of gradient updates is almost $90$% less than that of MetaDistil.

Reviewer comment (Reproducibility):
The framework consists of four steps. Some important training details are missing. It is hard to reproduce.

Author response:
**W2. Important training details are missing:** We have furnished all the necessary details in Appendix B.2 to reproduce the results reported in the paper. We request the reviewer to elaborate on specific details which are missing.

Reviewer comment (Evaluation):
The curriculum model parameters are learned via a reinforcement learning loss, which may lead to unstable training. The performance variance of the proposed method is not reported.

Author response:
**Q2. Unstable curriculum learning:** As per the suggestion, we have added the standard deviation of performance metrics obtained in different episodes of curriculum learning in Appendix Tables 9-11 of the updated paper. We observe a meagre variance across different episodes for all the tasks for both BERT and DeBERTa models. For instance, On the WiC task, MPDistil shows an average standard deviation of $1.1$% with the BERT model.