Paper: Cosine Similarity Knowledge Distillation for Individual Class Information Transfer (ICLR-2024)

Reviewer comment (Experiments):
The proposed model depends on the batch size. This could be a limitation when the batch size needs to be adjusted for reasons like memory constraints.

Author response:
As illustrated in Figure 4 of the main paper, our method demonstrates superior performance across various batch sizes, with the exception of very large batches (e.g., 512), when compared to vanilla KD. This observation implies that our approach, which achieves high performance even in small batch sizes, holds a distinct advantage over vanilla KD methods in real-world applications where memory constraints pose challenges.

Reviewer comment (Experiments):
Can this learning paradigm be used in an Out-of-distribution task or experiment?

Author response:
We are following your advice and conducting experiments on the out-of-distribution task. We are pleased to share our findings from the following experiment, in which the in-domain data is CIFAR-100, and the out-domain data is CIFAR-10. 

|Teacher-Student|Res34-Res18|WRN40_2-WRN16-1|WRN40_2-WRN40-1|WRN40_2-WRN16-2|
|:---:|:---:|:---:|:---:|:---:|
|KD|72.67|46.08|59.48|62.07|
|Ours|**73.01**|**49.91**|**61.51**|**64.19**|
|Difference|**+0.34**|**+3.83**|**+2.04**|**+2.13**|

We conducted three runs for each architecture and reported the average. While this approach is tailored for standard KD tasks, we demonstrate its effectiveness even in OOD tasks, surpassing vanilla KDs. We plan to incorporate this into our future research. Your comments are greatly appreciated.

Reviewer comment (Writing):
In Eq. (2) and (3), what does ":" mean? What's the difference between "[:,j]" in (2) and "[i,:]" in (3)? Does "[:,j]" mean concatenating all vectors from 1 to j (if so, please refer to C3)? It is stated that p_s, p_t ∈ R^{B×C} which is a matrix. How to compute the cosine similarity for two matrices based on Eq. (1)?

Author response:
**Answer about Q1 and Q2**

First of all, I apologize for any confusion regarding the notation. I explain this first and then share the amendments at the end. We adopted the meaning of '$:$' from Python array slicing. I am truly sorry for not being able to indicate this. The notation $i$ refers to the $i$-th element in the batch size, and the notation $j$ refers to the $j$-th class in the total number of classes.

As a result, $p_{[i, :]}$ represents the distribution of prediction about single image sample, taking the form of vectors in $\mathbb{R}^{C}$. In contrast, $p_{[:, j]}$ denotes the distribution of prediction across samples for a single class, expressed as vectors in $\mathbb{R}^{B}$.

Therefore, these expression allows us to easily compute the cosine similarity between the two vectors, not two matrix.

Several concerns have been raised about the methodology, prompting the following correction. Your feedback on these adjustments would be greatly appreciated.

The modification of Equations transforms it into the notation of ‘$tr$’ because the meaning of $[:, j]$ means a transpose function of $[i, :]$. (i.e., $p_{[i, :]}^{s, t}=p_{i}^{s,t} \in \mathbb{R}^{C}, p_{[:, j]}^{s, t}= p_{tr, j}^{s,t}\in \mathbb{R}^{B}$)

The $p^s$ denotes the student's prediction, and the $p^t$ denotes the teacher’s prediction. $B$ refers to the size of the batch and $C$ refers to the number of classes.

Previous formation eq (2):

#### $L_{CSKD} ( p_{[:, j]}^s, p_{[:, j]}^t, T ) =$ $1-\cos (\theta) =  1- $$p_{[:, j]}^s\cdot p_{[:, j]}^t\over\vert p_{[:, j]}^s\vert\vert p_{[:, j]}^t \vert$

After transformation eq (2):

#### $L_{CSKD} (p_{tr, j}^{s}, p_{tr, j}^{t}, T)  =$ $1-\cos (\theta) = 1- $$p_{tr, j}^{s}\cdot p_{tr, j}^{t}\over\vert p_{tr, j}^s\vert\vert p_{tr, j}^t \vert$


Previous formation eq (3):

### $p_{[i,:]}^{s,t}  =$ $e^{z_{i}^{s,t}/T}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T}}}$


After transformation eq (3):

### $p_{i}^{s,t}  =$ $e^{z_{i}^{s,t}/T}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T}}}$



Previous formation eq (4):

### $cs_{i}=$$p_{[i, :]}^s\cdot p_{[i, :]}^t\over\vert p_{[i, :]}^s\vert\vert p_{[i, :]}^t \vert$

After transformation eq (4):

### $cs_{i}=$$p_{i}^s\cdot p_{i}^t\over\vert p_{i}^s\vert\vert p_{i}^t \vert$

Previous formation eq (7) and eq (8):

#### $L_{CSWT} ( p_{[:, j]}^s, p_{[:, j]}^t, T_{i} ) =$ $1-\cos (\theta) = 1- $$p_{[:, j]}^s\cdot p_{[:, j]}^t\over\vert p_{[:, j]}^s\vert\vert p_{[:, j]}^t \vert$

### $p_{[i,:]}^{s,t}  =$ $e^{z_{i}^{s,t}/T_{i}}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T_{i}}}}$

After transformation eq (7) and eq (8):

#### $L_{CSWT} ( p_{tr, j}^s, p_{tr, j}^t, T_{i} ) =$ $1-\cos (\theta) = 1- $$p_{tr, j}^s\cdot p_{tr, j}^t\over\vert p_{tr, j}^s\vert\vert p_{tr, j}^t \vert$

### $p_{i}^{s,t}  =$ $e^{z_{i}^{s,t}/T_{i}}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T_{i}}}}$

Previous formation eq (9):

#### $L_{Total} ( p^s, p^t, \theta_{s}, \theta_{t}, T, T_{i} ) =$ $L_{CE} ( p^s ; \theta_{s}) + \alpha($$1\over{C}$$\sum_{j}^{C}L_{CSKD}(p_{[:,j]}^s, p_{[:,j]}^t, T))+$$1\over{C}$$\sum_{j}^{C}L_{CSWT}(p_{[:,j]}^s, p_{[:,j]}^t, T_i)$

After transformation eq (9):

#### $L_{Total} ( p^s, p^t, \theta_{s}, \theta_{t}, T, T_{i} ) =$ $L_{CE} ( p^s ; \theta_{s}) + \alpha($$1\over{C}$$\sum_{j}^{C}L_{CSKD}(p_{tr, j}^s, p_{tr, j}^t, T))+$$1\over{C}$$\sum_{j}^{C}L_{CSWT}(p_{tr, j}^s, p_{tr, j}^t, T_i)$

We provide an explanation for the notation $p_i^{s,t}$ because, when computing cosine similarity between $p^{s,t}_{tr, j}$ , the initial step involves obtaining $ p_i^{s,t} $ and subsequently transposing to derive 

$p^{s,t}_{tr, j}$

For instance, for $p_i^{s,t}$, we pass the logits of the $i$-th image through the softmax function, and then we obtain the probability value for that image. Extending this to a batch size(let's say, with 64 images), we acquire the probability distribution for each of the 64 images ($p^{s,t}$), presented in the form of a $B \times C$ matrix. By transposing this matrix $(C \times B)$ and isolating the distribution for $j$-th class, we obtain 

$p^{s,t}_{tr, j}$

—our desired output. This $p^{s,t}_{tr, j}$ can then be utilized for calculating cosine similarity.


I hope the above modifications aid your understanding.

Reviewer comment (Writing):
By the explanation below Eq. (3), it seems that the loss only focuses on one class j. From Eq. (9) the answer appears to be yes. It is suggested to make the definition clearer or reorganize the presentation order. It is better to explain all notations formally without assuming that audiences already understand what is presented.

Author response:
**Answer about Q1 and Q2**  

First of all, I apologize for any confusion regarding the notation. I explain this first and then share the amendments at the end. We adopted the meaning of '$:$' from Python array slicing. I am truly sorry for not being able to indicate this. The notation $i$ refers to the $i$-th element in the batch size, and the notation $j$ refers to the $j$-th class in the total number of classes.

As a result, $p_{[i, :]}$ represents the distribution of prediction about single image sample, taking the form of vectors in $\mathbb{R}^{C}$. In contrast, $p_{[:, j]}$ denotes the distribution of prediction across samples for a single class, expressed as vectors in $\mathbb{R}^{B}$.

Therefore, these expression allows us to easily compute the cosine similarity between the two vectors, not two matrix.

Several concerns have been raised about the methodology, prompting the following correction. Your feedback on these adjustments would be greatly appreciated.

The modification of Equations transforms it into the notation of ‘$tr$’ because the meaning of $[:, j]$ means a transpose function of $[i, :]$. (i.e., $p_{[i, :]}^{s, t}=p_{i}^{s,t} \in \mathbb{R}^{C}, p_{[:, j]}^{s, t}= p_{tr, j}^{s,t}\in \mathbb{R}^{B}$)

The $p^s$ denotes the student's prediction, and the $p^t$ denotes the teacher’s prediction. $B$ refers to the size of the batch and $C$ refers to the number of classes.

Previous formation eq (2):

#### $L_{CSKD} ( p_{[:, j]}^s, p_{[:, j]}^t, T ) =$ $1-\cos (\theta) =  1- $$p_{[:, j]}^s\cdot p_{[:, j]}^t\over\vert p_{[:, j]}^s\vert\vert p_{[:, j]}^t \vert$

After transformation eq (2):

#### $L_{CSKD} (p_{tr, j}^{s}, p_{tr, j}^{t}, T)  =$ $1-\cos (\theta) = 1- $$p_{tr, j}^{s}\cdot p_{tr, j}^{t}\over\vert p_{tr, j}^s\vert\vert p_{tr, j}^t \vert$


Previous formation eq (3):

### $p_{[i,:]}^{s,t}  =$ $e^{z_{i}^{s,t}/T}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T}}}$


After transformation eq (3):

### $p_{i}^{s,t}  =$ $e^{z_{i}^{s,t}/T}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T}}}$



Previous formation eq (4):

### $cs_{i}=$$p_{[i, :]}^s\cdot p_{[i, :]}^t\over\vert p_{[i, :]}^s\vert\vert p_{[i, :]}^t \vert$

After transformation eq (4):

### $cs_{i}=$$p_{i}^s\cdot p_{i}^t\over\vert p_{i}^s\vert\vert p_{i}^t \vert$

Previous formation eq (7) and eq (8):

#### $L_{CSWT} ( p_{[:, j]}^s, p_{[:, j]}^t, T_{i} ) =$ $1-\cos (\theta) = 1- $$p_{[:, j]}^s\cdot p_{[:, j]}^t\over\vert p_{[:, j]}^s\vert\vert p_{[:, j]}^t \vert$

### $p_{[i,:]}^{s,t}  =$ $e^{z_{i}^{s,t}/T_{i}}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T_{i}}}}$

After transformation eq (7) and eq (8):

#### $L_{CSWT} ( p_{tr, j}^s, p_{tr, j}^t, T_{i} ) =$ $1-\cos (\theta) = 1- $$p_{tr, j}^s\cdot p_{tr, j}^t\over\vert p_{tr, j}^s\vert\vert p_{tr, j}^t \vert$

### $p_{i}^{s,t}  =$ $e^{z_{i}^{s,t}/T_{i}}\over{\sum_{k=1}^{C}{e^{z_{k}^{s,t}/T_{i}}}}$

Previous formation eq (9):

#### $L_{Total} ( p^s, p^t, \theta_{s}, \theta_{t}, T, T_{i} ) =$ $L_{CE} ( p^s ; \theta_{s}) + \alpha($$1\over{C}$$\sum_{j}^{C}L_{CSKD}(p_{[:,j]}^s, p_{[:,j]}^t, T))+$$1\over{C}$$\sum_{j}^{C}L_{CSWT}(p_{[:,j]}^s, p_{[:,j]}^t, T_i)$

After transformation eq (9):

#### $L_{Total} ( p^s, p^t, \theta_{s}, \theta_{t}, T, T_{i} ) =$ $L_{CE} ( p^s ; \theta_{s}) + \alpha($$1\over{C}$$\sum_{j}^{C}L_{CSKD}(p_{tr, j}^s, p_{tr, j}^t, T))+$$1\over{C}$$\sum_{j}^{C}L_{CSWT}(p_{tr, j}^s, p_{tr, j}^t, T_i)$

We provide an explanation for the notation $p_i^{s,t}$ because, when computing cosine similarity between $p^{s,t}_{tr, j}$ , the initial step involves obtaining $ p_i^{s,t} $ and subsequently transposing to derive 

$p^{s,t}_{tr, j}$

For instance, for $p_i^{s,t}$, we pass the logits of the $i$-th image through the softmax function, and then we obtain the probability value for that image. Extending this to a batch size(let's say, with 64 images), we acquire the probability distribution for each of the 64 images ($p^{s,t}$), presented in the form of a $B \times C$ matrix. By transposing this matrix $(C \times B)$ and isolating the distribution for $j$-th class, we obtain 

$p^{s,t}_{tr, j}$

—our desired output. This $p^{s,t}_{tr, j}$ can then be utilized for calculating cosine similarity.


I hope the above modifications aid your understanding.

Reviewer comment (Experiments):
Though cosine similarity is scale-invariant, the significance of a cosine similarity value depends on vector length (i.e., the dimension of the vector). For a 200-dimensional vector, 0.4 may be a relatively large value, but for a 5-dimensional vector, 0.7 does not mean two vectors are similar. For classification, the number of classes and batch size are two factors that affect the vector length. That means, even for the same task, for different batch sizes, the proposed method would get different student models.

Author response:
**About Q3) with W1~W3**
Thank you for your thoughtful comments regarding the potential impact of vector length on cosine similarity values.

Also, thank you very much for mentioning that the number of classes and batch size, which affect the vector length, are important factors in classification tasks.

As a result, the proposed method does not perform concatenating all vectors, and we agree that student models may exhibit variations in performance for the same task due to differences in batch sizes. This is expressed experimentally in Figure 4, (C) in our paper.


We argue that the advantage of the proposed methodology is that the student can learn various predictions.

This is because the existing KL-divergence aims to accurately match the student's prediction distribution and the teacher's prediction distribution. This requires ‘exact equality’ of logit values between students and teachers for a zero loss value. However, these strict matching criteria place overly strict constraints on students, limiting their ability to acquire flexible dark knowledge from teacher distributions, ultimately leading to poor student performance.

To overcome these constraints, we introduce a loss function based on cosine similarity, which boasts scale-invariant properties. This distinguishes the approach by providing flexibility in the student logit without the need to exactly replicate the teacher distribution. As shown in Figure 2 of this paper, using cosine similarity increases the flexibility of dark knowledge acquisition by student distribution. Verification through entropy analysis experiments (see Figure 3 of this paper) highlights that our method exhibits higher entropy compared to existing KD methods. (It is worth noting that higher entropy corresponds to a greater amount of dark knowledge.)

Reviewer comment (Experiments):
The authors must take a long time to conduct extensive experiments in following sections to show the advantage of the proposed method. However, concerns exist in the results due to unclear definitions and intuitions (please refer to C1 to C3).

Author response:
**About Q4)**

We appreciate the reviewer's consideration and feedback. The authors acknowledge that conducting extensive experiments in the upcoming section to demonstrate the strengths of the proposed approach may take some time. We are currently working on acquiring further comparative analyses, including visualizations such as "tSNE" and "logits correlation", which we will incorporate.

We recognize the concern regarding unclear definitions and reliance on intuition, and we are committed to addressing these issues. The forthcoming revisions will aim to provide a more precise and well-defined analysis to alleviate any concerns about the results. Thank you for highlighting these points, and we will work diligently to enhance the clarity and rigor of our work.