Paper: Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information (ICLR-2024)

Reviewer comment (Theory):
The role of the teacher as a "provider of estimates for the unknown Bayes conditional probability distribution" is a theory for why distillation works that applies well mainly in the context of multi-class classification, and especially in the case where the input is images. There are other explanations for why knowledge distillation works, such as curriculum learning and regularization mechanisms (see e.g. [1]).

Author response:
**Response**: Yes, we agree that the experiments we have implemented so far are for image multi-classification problem. Having said this, our method, i.e., the MCMI estimator, is applicable to any DNN which outputs a probability distribution in scenarios where knowledge distillation (KD) is applicable, no matter how large or small the dimension of the probability distribution is. Note that the dimension of the probability distribution is 2 in binary classification and greater 2 in multi-classification. We have tested the MCMI estimator in the cases of 10 class classification (CIFAR 10), 100 class classification (CIFAR 100), and 1000 class classification (ImageNet). From our results (Table1 vs Table 2, and Table 9 vs Figures 14 to 19), it follows that in general, the accuracy performance gain offered by the MCMI estimator over the MLL estimator is even more when the number of classes is smaller. This can be explained by a larger percentage increase in the value of CMI when the number of classes is smaller. For example, the percentage increase in the value of CMI offered by the MCMI estimator in Table 1 is several magnitude larger than that in Table 2. Since CMI is well defined in the case of binary classification, there is no reason to believe that the MCMI estimator won’t work for binary classification problems to which KD is applicable. Of course, for those problems to which KD is applicable, how much gain the MCMI estimator would offer in comparison with the MLL estimator, especially for NLP datasets,  needs to be found out in future work.

Reviewer comment (Experiments):
The author should either make the above limitation more explicit in the text by explicitly restricting the scope of their claims to multi-classification and images, or provide evidence that their technique gives substantial improvements on binary classification tasks in NLP datasets (and even in vision datasets).

Author response:
**Response**: Yes, we agree that the experiments we have implemented so far are for image multi-classification problem. Having said this, our method, i.e., the MCMI estimator, is applicable to any DNN which outputs a probability distribution in scenarios where knowledge distillation (KD) is applicable, no matter how large or small the dimension of the probability distribution is. Note that the dimension of the probability distribution is 2 in binary classification and greater 2 in multi-classification. We have tested the MCMI estimator in the cases of 10 class classification (CIFAR 10), 100 class classification (CIFAR 100), and 1000 class classification (ImageNet). From our results (Table1 vs Table 2, and Table 9 vs Figures 14 to 19), it follows that in general, the accuracy performance gain offered by the MCMI estimator over the MLL estimator is even more when the number of classes is smaller. This can be explained by a larger percentage increase in the value of CMI when the number of classes is smaller. For example, the percentage increase in the value of CMI offered by the MCMI estimator in Table 1 is several magnitude larger than that in Table 2. Since CMI is well defined in the case of binary classification, there is no reason to believe that the MCMI estimator won’t work for binary classification problems to which KD is applicable. Of course, for those problems to which KD is applicable, how much gain the MCMI estimator would offer in comparison with the MLL estimator, especially for NLP datasets,  needs to be found out in future work.  

We really thank the reviewer for reading our responses and raising our score. 

We have done some further experiments to address your concern about binary classification task as explained in the sequel. 
We acknowledge the fact raised by the reviewer in that “distillation works depending on the setting”; hence, the answer to the question that whether our method works for binary classification task might be elusive. As such, we have conducted some experiments on binary classification task to show the effectiveness of the MCMI teacher compared to the MLL teacher. Please kindly refer to **Appendix A.13** for the details. Briefly speaking, the MCMI teacher is also effective in binary classification task as well. 

We hope these extra experiments resolves all the concerns you have. Again, thank you for all your valuable comments.

Reviewer comment (Experiments):
One of the main reasons why knowledge distillation is such a popular technique is because the teacher can generate pseudo-labels for new, unlabeled examples, increasing the size of the student's dataset (known as semi-supervised distillation or distillation with unlabeled examples, see e.g. [2, 3]). In order to apply the current approach, it seems one requires the ground-truth labels and thus has to give up a significant part of the power of knowledge distillation as a technique.

Author response:
**Response:** No, that is not the case. The requirements for our approach are nothing more and nothing less than those for KD since we do not change the KD framework. The knowledge in terms of training samples and their labels required to train the teacher using MCMI is identical to that required to train the teacher using MLL. To further demonstrate this, We have implemented semi-supervised distillation, and demonstrated once again that the MCMI teacher outperforms the MLL teacher in the semi-supervised distillation as well. Please see appendix A.9 for details.

Reviewer comment (Experiments):
It is unclear whether the proposed method and theory work well or apply in NLP datasets and binary classification contexts.

Author response:
**Response**: Please refer to our response to Comment 1.  

**Response**: Yes, we agree that the experiments we have implemented so far are for image multi-classification problem. Having said this, our method, i.e., the MCMI estimator, is applicable to any DNN which outputs a probability distribution in scenarios where knowledge distillation (KD) is applicable, no matter how large or small the dimension of the probability distribution is. Note that the dimension of the probability distribution is 2 in binary classification and greater 2 in multi-classification. We have tested the MCMI estimator in the cases of 10 class classification (CIFAR 10), 100 class classification (CIFAR 100), and 1000 class classification (ImageNet). From our results (Table1 vs Table 2, and Table 9 vs Figures 14 to 19), it follows that in general, the accuracy performance gain offered by the MCMI estimator over the MLL estimator is even more when the number of classes is smaller. This can be explained by a larger percentage increase in the value of CMI when the number of classes is smaller. For example, the percentage increase in the value of CMI offered by the MCMI estimator in Table 1 is several magnitude larger than that in Table 2. Since CMI is well defined in the case of binary classification, there is no reason to believe that the MCMI estimator won’t work for binary classification problems to which KD is applicable. Of course, for those problems to which KD is applicable, how much gain the MCMI estimator would offer in comparison with the MLL estimator, especially for NLP datasets,  needs to be found out in future work.

Reviewer comment (Experiments):
It is unclear whether there is a way to apply this technique in the context of semi-supervised distillation.

Author response:
**Response**: Yes, the same as in the case of KD. Please refer to our response to Comment 2.  

**Response:** No, that is not the case. The requirements for our approach are nothing more and nothing less than those for KD since we do not change the KD framework. The knowledge in terms of training samples and their labels required to train the teacher using MCMI is identical to that required to train the teacher using MLL. To further demonstrate this, We have implemented semi-supervised distillation, and demonstrated once again that the MCMI teacher outperforms the MLL teacher in the semi-supervised distillation as well. Please see appendix A.9 for details.

Reviewer comment (Writing):
As far as I can tell, the value $T$ is not defined, is this for the softmax?

Author response:
Yes, $T$ is for softmax. Please note that we did not make any changes over KD framework; we only replace the conventional MLL teacher by the MCMI teacher. Therefore, all the notations employed in this paper, including $T$, are adopted from the knowledge distillation (KD) framework.

[1]Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in neural information processing systems 32 (2019).

Reviewer comment (Reproducibility):
What is the value of $T$?

Author response:
As discussed in the response to the second comment above, $T$ is indeed the temperature of the softmax function in KD. Additionally, the value of $T$ is the same as that used in the benchmark methods in their original paper. Hence, $T=1$ for all the KD methods tested over ImageNet dataset, and $T=4$ for the majority of the KD methods tested over CIFAR-100 methods. 

As also emphasized in the main body of the paper, one of the prominent advantage of the proposed method is “Plug-and-play” nature of MCMI teacher in that we do not change any hyper-parameters in the underlying knowledge transfer methods, all of which are the same as in the corresponding benchmark methods.

Reviewer comment (Experiments):
If so, is it possible that the same effect can be achieved by simply running this method with temperature scaling? I.e. drop the CMI term?

Author response:
Thank you for your insightful comment. The temperature scaling (which is referred to as post-hoc regularization) has different effect compared to CMI regularization. 

In fact, the gain obtained via maximizing CMI is orthogonal to that obtained via tuning the $T$ values. More precisely, although by scaling $T$ for teacher, while fixing the $T$ for student, one can get some gain, this enhancement is independent of the gains achieved through maximizing the CMI value. To clarify, we have included **Appendix A.7**, where we conducted experiments with different $T$ values for the teacher and student, observing that the MCMI teacher consistently provides gains for any such combinations.

Reviewer comment (Experiments):
With regards to 6.2. my understanding is that this is using the negative scores during training, so is this really zero-shot classification? Why do you expect this?

Author response:
No, we are indeed aligned with the zero-shot setting for knowledge distillation as discussed in Section 3 of Hinton’s paper [1], where the authors specifically performed zero-shot distillation for the MNIST dataset. It's important to note that, unlike our method, the conventional knowledge distillation used in [1] does not work for other challenging datasets such as CIFAR-100 and CIFAR-10.

We further note that our method is different from negative scoring method used in zero-shot learning which needs predefined semantic vectors for both seen and unseen classes and assigns positive scores to related semantics and negative scores to unrelated semantics during the evaluation stage [2]. We argue that our method is more generalized, as during the training process, teacher don’t know which classes will be omitted when distill knowledge to the student model.

Lastly, we assume that you are asking why our teacher can lead to a student with better zero-shot performance. This is due to the fact that the information regarding the labels for the missing classes is still contained within $P_X ^*$ of the samples presented in the dataset. To elucidate, consider the following example in which the objective is to train a DNN to classify three classes: dog, cat, and car. Assume that the training samples for dog is entirely missing from the training dataset. Suppose that for a particular training sample from the cat class, $P_X ^*=[0.30,0.68,0.02]$, with the probability values corresponding to the dog, cat, and car classes, respectively. The value of 0.30 suggests that this sample exhibits some features resembling the dog class, such as a similarity in the shape of legs. Such information provided from the BCPD vectors for the samples of existing classes can collectively help the DNN to classify the samples for the missing class. We note that the one-hot vector for this sample is $[0,1,0]$, and therefore, such information does not exist when one-hot vectors are used as unbiased estimate of $P_X ^*$.

[1] Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff ”Distilling the Knowledge in a Neural Network” stat. 2015

[2] Huynh, Dat, and Ehsan Elhamifar. "Fine-grained generalized zero-shot learning via dense attribute-based attention." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

Reviewer comment (Presentation):
In Figure 3, why is the heat map on the terrier not on the body of the animal? Bottom, third from left.

Author response:
As discussed in our response to comment 1, the CMI is a class-based (cluster-based) concept, and cannot be evaluated by looking at a specific image. In other words, to evaluate the CMI value for a DNN, all the probability vectors (or the corresponding feature maps) within one class should be considered **as a whole**.

If the CMI value for a given class is high, then the output probability vectors and their corresponding feature maps contain more information. These information show up in two formats: (i) the prototype, and (i) the contextual information which can be the background or the unique characters of the object in the images (noise of the cluster). 

Once looking at the Eigen-CAM for MLL teacher, for all the images the MLL teacher only captures the dog’s head (prototype); however, for the MCMI teacher this focus is more dispersed ranging from dog’s head, doge’s legs, and the background of the image (contextual information). As such, the MCMI teacher contains more information when we consider **all the images from one class**. 

Nevertheless, depicting only four images is not a good representation of the images in one class. To this end, in appendix A9 we sample and depict more images from four specific classes of the ImageNet training set to better illustrate this fact.

Reviewer comment (Theory):
While it is intuitively clear that using the conditional mutual information as the regularizer term can capture the contextual information of \(X\) (Image) and provide additional information to a student model, the direct connection between conditional mutual information and the optimal Bayes classifier is less explained. I believe explaining this connection is important because this approach is motivated from the importance of optimal classifier \(P^{*}_{X}\).

Author response:
We truly appreciate this theoretic question in general. However, at this point, the formulation of this theoretic question is not clear at all for a couple of reasons: (1) the underlying joint distribution of (X,Y) and hence P_X^* are unknown, and any amenable assumption would go against the current philosophy of deep learning---if the joint distribution is known, there is no need for learning; and (2) CMI $I(X;\hat{Y}|Y)$ highly depends on the underlying deep neural network (DNN) which varies from one setting to another and for which deep understanding still lacks. Without certain assumptions on the underlying DNN, there might be no theoretical relationship between  $I(X;\hat{Y}|Y)$ and $P_X^*$. For example, when the underlying DNN is poorly designed---all output probability vectors huddle together,  $I(X;\hat{Y}|Y)$ could be always near 0 no matter what the joint distribution of $(X,Y)$ is. This setting is very different from the work [1] where the law of total probability for variance can be nicely applied in general. To overcome this difficulty, in this paper we instead choose the following approaches to justify and evaluate the proposed MCMI estimator:

1. Since the purpose of estimating $P_X^*$ is for the benefit of the student in KD, a good litmus test for an estimator is to see how the estimator can help the student improve its accuracy. In this regard, our proposed MCMI estimator is thoroughly compared to the MLL estimator and shown  superior to the MLL estimator across the board for all tested KD frameworks. 

2. A synthetic Gaussian dataset is created and used to evaluate how close our MCMI estimate is to P_X^* . Please see appendix A.6 for details. Again, the results therein demonstrate the superiority of  the MCMI estimator to the MLL estimator in this synthetic setting as well.

Reviewer comment (Theory):
Could you elaborately explain why minimizing \(I(X , \hat{Y} | Y)\) can make the teacher model \(f\) to be more similar to the optimal bayes classifier?

Author response:
We guess you mean maximizing $I(X;\hat{Y}|Y)$. We won’t refer to the unknown Bayes class probability $P_X^*$ as the optimal Bayes classifier since the context for defining the optimality is not clear. In our view, the unknown Bayes class probability $P_X^*$ is defined by nature, or by human beings as a whole; it should be the average of all predictions made by a very large group of people. Intuitively, $P_X^*$ is influenced by different contextual information of the object in $X$ with respect to the corresponding image cluster, which would cause different people to make different predictions, and hence should stay a little away from the probability vector corresponding to the pure prototype (i.e., the object) which is the label one-hot vector in conventional deep learning, and the “centroid” vector $Q^Y$ defined in Eq. (11) in our case. It is the difference between $P_X^*$ and the prototype probability vector that reflects the variety of contextual information of the object. Maximizing $I(X;\hat{Y}|Y)$ to some degree would enable the teacher model f to extract more contextual information of the object into its output probability vector, thereby making it to be closer to $P_X^*$. This is indeed further confirmed by our results on synthetic Gaussian dataset in appendix A.6.