From: ICLR.cc/2023/Conference/Paper757/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper757/Reviewer_TfTn
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper explores the capacity mismatch problem between the student model and the teacher model in knowledge distillation from the view of soft labels (or logit outputs). It approximately quantifies the capacity gap as a function of the variances of student and teacher logits. It subsequently addresses the gap by proposing a method (PESF-KD) that smoothens the output of the teacher model in a parameter-efficient online fashion, i.e., adaptively and efficiently updates the teacher output by using the student output. The paper empirically demonstrates the efficacy of the PESF-KD method over other label smoothening techniques. The authors also propose an online KD method (FT-KD) but that is quite similar to existing methods.
strength_and_weaknesses: The primary strengths of the paper in my view:

1. Simple and efficient method proposed to tackle the capacity mismatch problem in knowledge distillation.
2. Principled extension of existing theory in the domain to quantify the knowledge distillation capacity gap as an approximate function of the variances of student and teacher model outputs.
3. Decent amount of empirical results and analysis.

The primary weaknesses of the paper in my view:

1. The novelty of the capacity gap analysis as well as the proposed PESF-KD method seem to be incremental. Both the method and the analysis seem to be relatively simple extensions of existing work.
2. In the experimental results section, a detailed analysis of how different methods fare vis-a-vis compute cost and accuracy (trade-off) would add more value to the paper. The existing results don't give any insights into the training trajectory of the various methods (how accuracy improves as training progresses or resource consumption increases). 
3. The proposed methods (FT-KD and PESF-KD) are simple and efficient but are not really motivated by the analysis that the authors provide. The empirical justification does seem reasonable but motivating the methods from the analysis would be more coherent and insightful.
clarity,_quality,_novelty_and_reproducibility: The paper has been well-written and is easy to follow. The novelty of the analysis and proposed methods is incremental in my opinion. The experiments do seem reproducible with the code provided (to be clear, I did not reproduce the experiments). 
summary_of_the_review: I believe the paper tries to tackle an important problem in the knowledge distillation domain. The proposed analysis and methods are valuable but relatively simple extensions of existing work. I believe the paper would be more valuable if a more detailed analysis (instead of the approximate analysis based on existing work) is presented and new methods are proposed based on the analysis.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper757/Reviewer_7EDN
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors discussed the issues of previous distillation methods and present a new method that exploits the tuning of soft labels of the teacher network.

Hence, this paper tackle an interesting and relevant problem. There efforts to validate hypothesis by presenting empirical results.

There are some issues that should be addressed before the publication of this paper (see following sections).
strength_and_weaknesses: The authors hypothesize that previous methods to transfer knowledge using the soft labels from teachers to students may be problematic due to the mismatched model capacities.

Previous literature approaches this problem through (i) temperature parameters, (ii) label smoothing, and (iii) teacher-student joint training.

These methods *rarely* explain the effect of the smoothed soft labels to enhance the KD performance.

Contributions:

- Discovery, analysis, and validation of the effect of the smoothed soft label.
- A new, *less time-consuming*, and *adaptative* transfer of teacher's knowledge method.

They argue that KD assumes the student has fewer parameters. This is not necessarily true; there is a line of works based on self-distillation, for which the teacher and student networks have the same number of parameters.

> I believe that the authors should adequately state the different techniques of KD. And after that, explicitly state their research niche, i.e., KD, for which the teacher has a more significant number of parameters than the student.

*The capacity mismatch problem*:  When independently trained from scratch:
- The larger model: output sharper values and obtain better accuracy,
- The smaller model: output smoother values and obtain poorer accuracy.

The authors should define $G_{gap}$ before mentioning it in parag. 3, Sec. 2.1. I assume it is the capacity gap. It was only described in Sec. 4.1, formulae and meaning.

The authors should provide a better intuition about their *label adjustment*. For instance, they are differentiating true labels, soft (hard?) labels, and a better explanation of the generalization errors. Shortly, the authors need to clearly and convincingly state the decrease in student performance due to what.

In the *Training objectives* sub-section of Sec. 3.2, the authors argue that their method is better than Zhang et al., 2018 in terms of the following dimensions:
* Parameter efficiency
* Student-friendly KD

Questions related to the previous sub-section:
* How did you measure the dimensions? Empirically? Theoretically? The authors should mention it asap in the text.
* Do you argue that you introduced the adapter module? However, as far as I understood, they were introduced by He et al., 2018. Did you present it in the context of KD?
* Eq. 2 is in Sec. 3.1, so the sentence *Except for the above modifications in Sec. 3.1 the rest of the student's training objectives take the same form as Equation 2.* It does not make sense to me.

Shortly, Sec. 3. should be rewritten entirely.

Sec. 4 focuses on understanding and dealing with the sharpness (or fidelity) gap between teacher and student networks.

Since the authors of Stanton et al., 2021 argue, among other conclusions, that:
. Good student accuracy does not imply good distillation fidelity.
. If the student matched the teacher perfectly, the student could not outperform the teacher. Or in other words, the student would be unable to achieve better generalization than the teacher.

Hence, what is the point in narrowing this gap (equivalently, improving fidelity between teacher and student)? This discussion is central to motivating the main contributions of this paper. I suggest the authors start Sec. 4. with this discussion.

OBS: I am considering:
. *Fidelity*: the ability of a student to match a teacher's predictions.
. *Generalization*: the performance of a student in predicting unseen.

Suggestions:

- Include the references to the improved effectiveness of label smoothing in the Parag. 1, Sec. 2.1, for NLP and CV.
- Correct latex " " to \`\` \'\' throughout the text.
- The font size in the legend of Fig. 1 is too small.
clarity,_quality,_novelty_and_reproducibility: There are issues related these criteria (see above comments).

Missing code to reproducing the results.
summary_of_the_review: The paper tackles a challenging problem, however, there are important and relevant concerns about the narrowed application of the method. Additionally, some parts of the text are difficult to understand, which unable a proper evaluation of the article's merit.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper757/Reviewer_s5wY
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors provide empirical and some theoretical analysis on the capacity mismatch problem, where they propose two online adaptive soft label tuners for knowledge distillation: Fine Tuned KD (FT-KD) and Parameter-Efficient and Student-Friendly distillation (PESF-KD).

FT-KD is an online distillation method where all the weights of the teacher network are trainable, while PESF-KD is a parameter-efficient online distillation technique where a plastic residual connection module (adapter) is appended to a teacher network for training, while the original parameters of the teacher network are fixed. PESF-KD and FT-KD share the same loss function, a weighted linear combination of the vanilla KD Loss and the cross entropy.
strength_and_weaknesses: Strengths
----------------

- The parameter efficiency of PESF-KD via the adapter module is an elegant solution for adaptive temperature estimation for knowledge distillation. 

- Moreover, the sharpness gap metric is effective in capturing the smoothness gap between teacher and student networks. It is a simple and interpretable metric that can be employed for analysis in the capacity mismatch problem.

- The derivation of the sharpness gap with respect to its mean and variance provides more insight on the gap metric, particularly at initialization which I find interesting.

Weaknesses
----------------

- Misspelled “Hinton assumption” in Section 4.2. 

- Misspelled “vanilla” in the penultimate paragraph in Section 1 (Intro).

- Section 3.1 suggests that label adjustment is a contribution of this work, which to my knowledge is not accurate. Equation 2 in Section 3.1 is a conventional KD loss function (Guo et al, 2020: Equation 2), hence the concept of label adjustment cannot be fairly attributed to this work. In my view, the sole and key contribution of FT-KD is that the weights of the teacher network are implicitly being updated when optimizing the student network on its loss function for adaptive temperature optimization. 

- Figure 3 doesn’t demonstrate a clear trend when manually adjusting the temperature and doesn’t seem to be consistent with the constraints of Equation 1 and 7 as described in Section 4.2. It is unclear to me how both high and low temperatures for vanilla KD both result in a large sharpness gap. Intuitively, for vanilla KD, an increased temperature should result in a smoother teacher and consequently a small sharpness gap. However, I could imagine that an extremely large temperature could impede training all together. In short, it appears that the correlation between the temperature and the sharpness gap is not well communicated in the manuscript. For Figure 3, could you possibly calculate the correlation coefficient of the temperature parameter and the sharpness gap to measure the trend.

- The experimental results in Tables 1, 2, 3, 4 and 5 are not conducted over multiple runs (no mean and variance).
clarity,_quality,_novelty_and_reproducibility: Clarity
---------
The contributions of the manuscript are clearly stated and demonstrated in the results section. I think some more work could be done in Section 4.2, as the authors could label the axis in Figure 2 and similarly rework the corresponding justification for the results in Figure 3.

Quality
---------
The quality of the manuscript is good. Great effort has been made in providing different perspectives in studying the sharpness gap between teacher and student networks. However, a fair number of the empirical results have not been run across multiple seeds, which is necessary metric for robustness. This, unfortunately, makes it difficult to trust the results.

Novelty
----------
I find the PESF-KD method to be novel and elegant, largely due to how parameter efficient the method is. I am, however, unsettled on the novelty FT-KD, as it can be interpreted as a baseline onllne distillation method, this is exacerbated by the novelty claim for label adjustment. 

Reproducibility
------------------
There was no code released for this work yet, hence cannot claim that this is reproducible.



summary_of_the_review: In summary, I am largely satisfied with the PESF-KD method and I find it to be a novel contribution to the knowledge distillation literature. My concern is largely around the sharpness gap claims in Figure 3 which I found to be inconsistent with the analysis in Equation 7 and also the novelty claim for label adjustment in Section 3.1.

I am marginally rejecting this paper, but will be happy to reassess my recommendation if you can address these questions/suggestions:

- Could you confirm if the paper claims that label adjustment is one of the contribution of this work? If yes, could you justify why? If not, could you make it more clear in the manuscript.
- How do both high and low temperatures for vanilla KD both result in a large sharpness gap as described in Section 4.2?
- Could you run your experiments across multiple runs. (Table 1, 2, 3, 4 & 5)


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper757/Reviewer_6bch
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a online distillation framework called PESF-KD (parameter-efficient and student-friendly KD) that combines multiple design choices. It is online through teacher adapters and trains both teacher and student with hard labels too. Analysis on smoothness gaps are presented and experiments on multiple vision and NLP datasets show it can improve performance.

strength_and_weaknesses: Strengths:

1.. The paper's experiments are extensive, ranging from vision to NLP with multiple datasets. 

2.. The analysis on the correlation between sharpness/smoothness gap and distillation accuracy is interesting. 

Weaknesses:

1.. The method description quality is not that good, and the motivation of each design choice is very unclear. It seems many components can be found from prior works, e.g., DML and He et al. 2022 but there is no clear explanation of why we use them at this variant of the method.

-- "Therefore, to reduce the impact of this part of the error, we modified the teacher’s soft labels called “label adjustment”, i.e., the maximum probability labels are guaranteed to be the labeled true labels." It is not clear what label adjustment is and how the max prob are guaranteed to be the true labels from the text and the equations. 

-- "Unlike DML (Zhang et al., 2018), this approach does not have an explicit KL loss term in the student-to-teacher direction." There is indeed an L_KL loss term in the equation above that is jointly optimized by the student and the teacher in my understanding, so I'm not sure how this point is different. If this means that only the student optimizes KL loss in FT-KD, I'm not sure why the teacher cannot optimize it too. The justification of the design choices are not motivated in method description. The same applies to PESF-KD: why do we need the adapters instead of optimizing the whole teacher, and why now the KL loss is bidirectional? How and why do these affect the smoothness gap? Should they be bundled together instead of being studied separately?

-- "The difference is that the provider of the soft labels is the student network." I'm not sure why the "provider" of soft labels is now the student network. Students always give soft labels. Does it mean that the teacher now learn from the student too? If so why not use this design in FT-KD.


2.. The paper has many grammar errors, e.g., "The teacher and the student are jointly trained to make the teacher’s knowledge more friendly to the students called online distillation", "Our approach achieves better parameter-efficient and student-friendly KD…" 

3.. Figure 2's axes are not labeled.

4.. The CIFAR-100 improvement seems to be within random variance. 

5.. Many other baseline methods' quoted results are lower than the vanilla KD but they were considered to be better than vanilla KD, at least in the original paper - can the paper explain why?

clarity,_quality,_novelty_and_reproducibility: Clarity is a major issue. Novelty seems to be mainly on the analysis part but not the method part. Code is in the supplement.
summary_of_the_review: The paper's motivation in design choices are unclear, and presentation quality is a major issue. Therefore I recommend rejection. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper757/Reviewer_dRUb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper improves logits based knowledge distillation (KD) from the perspective of softening/smoothing the predicted labels from the teacher model. The authors show that the mismatch between large teacher’s and small student’s predictive distributions is due to the sharpness of the predicted labels from the teacher model. They also design some metrics to study this sharpness phenomenon. To mitigate it, a slightly modified mutual learning method called FT-KD, and a claimed more efficient & student-friendly KD method called PESF-KD, are presented. The authors validate the effectiveness of their methods on both CV tasks (image classification with CIFAR100 and ImageNet) and NLP tasks (SST-2 for sentiment classification, RTE for natural language inference, and QQP for paraphrase similarity matching).
strength_and_weaknesses: Strengths.

+ How to well transfer knowledge from a large teacher model to a much smaller student model is a critical topic in KD research.

+ The analysis of the capacity mismatch between the teacher and student model via smoothing the sharpness of  the soft labels from the teacher model is interesting.

+ Performance evaluation is conducted on both CV and NLP tasks.

+ The proposed methods show performance improvements, although not large or impressive.

+ Sample code is provided.

Weaknesses.

- The method.

Although I like the experimental findings in Section 4. I have a lot of concerns: 1) experimental settings for Figure 1 to Figure 9 are totally missing, which makes them hard to be convincing; 2) how about the observations when using different teacher-student pairs (with either similar or different network architectures? 3) the metrics (Eq. 4 and its reduced version Eq. 7) are not explicitly used in the designs of FT-KD and PESF-KD. This breaks the flow of the method section. Why not to add a related loss in the optimization?  

The proposed FT-KD and PESF-KD are heavily based on existing works deep mutual learning (DML) [1] and parameter-efficient transfer learning methods [2-3], leading to limited technical novelty. What's more, they actually need huge training costs instead of more efficient claimed by the authors, in my understanding (see below comments for details).
 
To me, claiming FT-KD as a new contribution (the authors call it an adaptive knowledge transfer learning method) is misleading to some degree. As FT-KD merely removes the online knowledge transfer path from the student to the teacher, in DML. Besides, in FT-KD, is the teacher model pre-trained beforehand or trained from scratch jointly the student model?  I think, it should be pre-trained beforehand as the authors state that it will be fine-tuned during the training of the student model, is it true?

Note that PESF-KD (may also include FT-KD) uses the pre-trained teacher model. That is, PESF-KD is not a real online KD method. It not only needs the pre-trained stage, but also the online joint training stage. As a result, the claimed 'more efficient' advantage is totally misleading.
 
[1] Deep Mutual Learning, CVPR 2018.

[2] TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning, NeurIPS 2020.

[3] Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.

- The experiments.

The improvement of the proposed methods is mostly marginal, compared to DML, or other competitive methods.

Note that PESF-KD (may also include FT-KD) uses the pre-trained teacher model, and thus the experimental comparisons of training cost in Table 2, Table 3, etc. are wrong. That is, PESF-KD is not a real online KD method. It not only needs the pre-trained stage, but also the online joint training stage. As a result, the claimed 'more efficient' advantage is totally misleading.

In performance comparison, the authors only consider relatively old KD methods, many recently proposed high-performance KD methods are ignored. Some representative methods are SSKD, SRRL, SemCKD, ReviewKD and SimKD. 


- Others.

The writing can be improved significantly: 1) there are so many bolded descriptions; 2)  the organization of figures is bad; 3) there are many typos and grammar errors.


**----Update----**

I keep my original score as my concerns are not well addressed or ignored in the rebuttal.
clarity,_quality,_novelty_and_reproducibility: The idea is basically easy to understand, but the presentation is not good. This paper suffers from low novelty. Sample code is provided.

Please refer to my comments in 'Strength And Weaknesses' for details.
summary_of_the_review: This paper is below the acceptance bar of ICLR.

Please refer to my comments in 'Strength And Weaknesses' for details.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold