Paper: Knowledge Distillation via Flow Matching (ICLR-2024)

Reviewer comment (Experiments):
My major concern is the generalization, given that distillation is a well-defined topic, but this method seems like doesn't work well in heterogeneous architecture settings.

Author response:
Thank you for your valuable suggestions.

In our experiments, the heterogeneous teacher-student pairs, ResNet50-MobileNetV1 on ImageNet-1k and VGG13-MobileNetV2 on CIFAR-100, did not achieve SOTA accuracy. This outcome can be attributed to suboptimal configurations of the loss function and meta-encoder for FM-KD, rather than suggesting that FM-KD is ineffective in heterogeneous architecture settings.

We experiment with the optimal settings, and the additional experimental results are presented in Table I and II. We can observe that FM-KD beats all previous methods and achieves SOTA performance.

Table I (ImageNet-1k, loss function is set as DIST, K=8)

| T-S pair | Meta-encoder | R50-MBV1 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=8) | 2-MLP | 74.22 |
| Top-1 Acc. (K=8) | 3-MLP | 74.28 |

Table II (CIFAR-100, meta-encoder is set as Swin-Transformer, loss function is set as DKD)

| T-S pair | VGG13-VGG8 | VGG13-MBV2 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=1) | 75.21 | 69.68 |
| Top-1 Acc. (K=2) | 74.86 | 69.52 |
| Top-1 Acc. (K=4) | 75.42 | 69.94 |
| Top-1 Acc. (K=8) | 74.46 | 69.94 |

Reviewer comment (Experiments):
This method seems to not work well especially when there is a big difference in size between teacher models and student models.

Author response:
Thank you for your valuable suggestions.

In our experiments, the heterogeneous teacher-student pairs, ResNet50-MobileNetV1 on ImageNet-1k and VGG13-MobileNetV2 on CIFAR-100, did not achieve SOTA accuracy. This outcome can be attributed to suboptimal configurations of the loss function and meta-encoder for FM-KD, rather than suggesting that FM-KD is ineffective in heterogeneous architecture settings.

We experiment with the optimal settings, and the additional experimental results are presented in Table I and II. We can observe that FM-KD beats all previous methods and achieves SOTA performance.

Table I (ImageNet-1k, loss function is set as DIST, K=8)

| T-S pair | Meta-encoder | R50-MBV1 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=8) | 2-MLP | 74.22 |
| Top-1 Acc. (K=8) | 3-MLP | 74.28 |

Table II (CIFAR-100, meta-encoder is set as Swin-Transformer, loss function is set as DKD)

| T-S pair | VGG13-VGG8 | VGG13-MBV2 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=1) | 75.21 | 69.68 |
| Top-1 Acc. (K=2) | 74.86 | 69.52 |
| Top-1 Acc. (K=4) | 75.42 | 69.94 |
| Top-1 Acc. (K=8) | 74.46 | 69.94 |

Reviewer comment (Experiments):
Can you explain or verify why this method can't work well when the gap is big between teacher and student?

Author response:
Thank you for your interesting question. FM-KD is effective in your mentioned scenarios. Specifically, as shown in Tables I and II, FM-KD is effective in heterogeneous architecture settings with a big gap between the teacher and the student. In scenarios where a bigger gap exists between the teacher and the student, we conduct experiments using TIMM's ResNet50 as the teacher and ResNet18 as the student. The experimental results in Table III (refer to Appendix G) indicate that FM-KD remains effective.

Table III (ImageNet-1k, epoch is set as 350, teacher is set as ResNet 50 from TIMM, student is set as ResNet18)

| Model |  ResNet50 | ResNet18 (DIST) | ResNet18 (FM-KD, K=8) |
| ---- | ---- | ---- | ---- |
| Type | Teacher | Student | Student |
| Top-1 Acc. | 80.12 | 72.89 | 73.71 |

Reviewer comment (Novelty):
This paper is similar to DiffKD where it is a combination of generative model and knowledge distillation, except that it replaces the diffusion model with the rectified flow, which has limited innovation.

Author response:
**A1.** Thank you for your valuable feedback. Our approach does not merely substitute the diffusion model with Rectified Flow or directly apply Rectified Flow for knowledge transfer. Specifically, DiffKD employs the original training loss of the diffusion model (i.e., $L_{diff}$) and combines it with two additional auxiliary losses (i.e., $L_{ae}$ and $L_{diffkd}$) as detailed in its original paper for training. By contrast, we utilize our proposed training loss $L_{FM-KD}$ (solely one), guaranteed by Theorem 3.1 in the original/revised paper, which differs from each of $L_{diff}$, $L_{ae}$, and $L_{diffkd}$. Moreover, from Eq. 2 (i.e. original Rectified flow loss function) to Eq. 3 (i.e. our proposed serial loss function) in our original/revised paper we implement some critical improvements. Without these improvements, Rectified flow (w.r.t. Eq. 2) in the WRN-40-2-WRN-16-2 pair on CIFAR-100 would have encountered the problem of vanishing gradients, resulting in a final test accuracy 3.42%.

It might be helpful to provide a clear outline of our contributions to ensure a better understanding.

1. We introduce Rectified flow instead of diffusion processes since Rectified flow is able to establish a bridge between two empirical distributions (teacher feature/logit distribution and student feature/logit distribution) and is well suited for knowledge transfer.

2. The serial training paradigm in Eq. 3 is essential for effective and stable training. If trained directly in the form of the original Rectified flow, VP SDE and VE SDE, the student accuracy with WRN-40-2-WRN-16-2 pair on CIFAR-100 $\leq$ 5%. However, as demonstrated in Appendix M, the accuracy $\geq$ 72% if the FM-KD framework is used for training.

3. FM-KD is theoretically guaranteed and has been demonstrated to stable training with various meta-encoder and loss functions by a wide range of ablation experiments. By contrast, DiffKD is an empirical design with no theoretical guarantees. DiffKD prevents vanishing gradient through $L_{diffkd}$ outlined in its paper, but this form leads to unstable training.

4. We prove and explain the effectiveness of this training paradigm as it can approximate an implicit ensemble method in Proposition 3.2.

5. We design OFM-KD, which can be used in the Online KD scenario. Our proposed OFM-KD outperforms prior Online KD algorithms in most teacher-student pairs.

6. FM-KD is more effective than DiffKD on CIFAR-100 and ImageNet-1k, as shown in Table 1 & 2 and Appendix K & L of the revised paper. The experimental results in the original main paper where FM-KD did not reach SOTA performance were merely due to not utilizing the optimal settings.

7. FM-KD is more efficient than DiffKD, as demonstrated in Appendix J.

8. We design FM-KD$^\Theta$ in Sec. 3.5 of the revised paper in this discussion phase, which incurs no additional inference computational cost.

9. We unify VP ODE [1*,2*,3*], VE ODE [1*,2*,3*], and Rectified flow into the FM-KD framework in Appendix M in this discussion phase.

Reviewer comment (Evaluation):
The FM-KD method proposed in this paper modifies the structure of the student network and increases the cost during model inference, which is a shortcoming of a model compression algorithm. The computational cost during inference is not given in the paper, and if refer to Fig. 5, the increase in cost is obvious and there is some unfairness in comparison with other methods. Are there still significant advantages of the FM-KD approach over baseline models that increase the number of parameters to make time-consuming approximations?

Author response:
**A2.** Thank you for your valuable feedback. We address your concerns in the following three Parts.

**Part 1:** FM-KD leads to significant performance improvements, and the added computational overhead in inference is acceptable. Additionally, FM-KD has the ability to adjust the Number of Function Evaluations (NFE) during inference to balance effectiveness and efficiency.

**Part 2:** As demonstrated in Tables I and II, in comparison to DiffKD, a method that also elevates computational cost in inference, FM-KD is evidently more cost-effective regarding performance and GPU latency.

Table I (ImageNet-1k)

|Method | FM-KD | DiffKD |
| ---- | ---- | ---- |
| Meta-encoder | 2-MLP | 2-BottleNeck+Conv+BN+MLP |
| Type | logit | both feature and logit |
| R34-R18 | 73.17 | 72.49 |
| R50-MBV1 | 74.22 | 73.78 |

More details of Table I can be found in Appendix K of the revised paper.

Table II (ImageNet-1k)

| Method | DiffKD | FM-KD | DiffKD | FM-KD |
| ---- | ---- | ---- | ---- | ---- |
| Mode | training | training | inference | inference |
| R50-MBV1 | 0.2431 | 0.1281 | 0.0517 | 0.0151 |
| R34-R18 | 0.2376 | 0.1139 | 0.0390 | 0.0150 |

Where x-MLP refers to x-layer MLP. The measurements in Table II were obtained under FM-KD using 2-MLP as meta-encoder. More details can be found in Appendix J of the revised paper.

**Part 3:** We appreciate the opportunity to enhance our work and have designed FM-KD$^{\Theta}$ without extra computational costs in inference. For a comprehensive explanation, kindly refer to our detailed response regarding your Q4.

**A3.** We have presented GPU latency measurements during inference in Table III. We can substantiate that the additional computational cost of FM-KD in inference is much lower compared with DiffKD.

Table III (ImageNet-1k, hardware setting is 2*RTX 3080ti)

| Method | DiffKD | FM-KD | FM-KD $^{\Theta}$| DiffKD | FM-KD | FM-KD $^{\Theta}$ | classical KD methods |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Mode | training | training | training | inference | inference | inference | inference |
| R50-MBV1 | 0.2431 | 0.1281 | 0.1281 | 0.0517 | 0.0151 | 0.0108 | 0.0108 |
| R34-R18 | 0.2376 | 0.1139 | 0.1140 | 0.0390 | 0.0150 | 0.0108 | 0.0108 |

**A4.** Thank you for your valuable comment.

To ensure a fair comparison with other knowledge distillation algorithms such as DKD, DIST, and SPKD, we have further designed FM-KD$^{\Theta}$, which refines the process by distilling $Z_0$ obtained from FM-KD into an existing classification head (i.e. the original student's classification head) $\mathcal{T}_\mathrm{vanilla}(\cdot)$, thereby ensuring that no extra inference cost is incurred. More details can be found in Sec. 3.5 of the revised paper.

Table IV (CIFAR-100)

| Teacher | ResNet56 | WRN-40-2 | WRN-40-2 | ResNet32$\times$4 | VGG13 | VGG13 | WRN-40-2 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Student | ResNet20 | WRN-16-2 | WRN-40-1 | ResNet8$\times$4 | VGG8 | MobileNetV2 | ShuffleNetV1 |
| Top-1 Acc. | 72.20 | 75.98 | 74.99 | 76.52 | 74.82 | 69.90 | 77.19 |

Table V (ImageNet-1k)

| T-S pair | R34-R18 | R50-MBV1 |
| ---- | ---- | ---- |
| Top-1 Acc. | 72.14 | 73.29 |

The experimental results of FM-KD$^{\Theta}$ are presented in Tables IV and V, where the implementation details on ImageNet-1k and CIFAR-100 can be found in Appendix I. By comparing the accuracy of prior knowledge distillation algorithms like DKD, DIST, and SPKD in Tables 1 and 2 of the revised paper, we can conclude that FM-KD$^{\Theta}$ achieve SOTA performance across a wide range of teacher-student pairs.

**A5.** Thank you for your interesting question.

Yes, DiffKD is an algorithm that increases the computational cost during inference. As indicated in Tables I and II, by utilizing a lightweight meta-encoder 2-MLP, FM-KD exceeds DiffKD, which uses 2-BottleNeck+Conv+BN+MLP as meta-encoder, by margins of 0.68% and 0.44% in the ResNet34-ResNet18 and ResNet50-MobileNetV2 pairs with a lower GPU latency (less than 1/3 DiffKD's in the ResNet50-MobileNetV2 pair), respectively.

Reviewer comment (Evaluation):
The experimental performance is not outstanding enough. There is no significant advantage over other knowledge distillation methods in Table 3 of the experiments for object detection, and there are also several tests in the image classification experiments that do not perform as well as other existing methods.

Author response:
**A6.** Thank you for your feedback. We apologize for our inability to resolve this weakness in object detection. This result stems from the fact that in image classification, flow matching with the teacher at the logit level generally yields performance closer to that of the teacher. However, in object detection, flow matching with the teacher at the FPN (Feature Pyramid Network) level does not directly translate into improved performance in the final metric, mAP. We have already incorporated this point into our limitations (please see Sec. 7).

**A7.** Thank you for your valuable comment. The experimental results of FM-KD in Tables 1 and 2 of the original paper are not obtained from optimal settings. It is demonstrated from Fig. 5 of the revised paper (Table 6 of the original paper) that FM-KD performs best on ImageNet-1k when using MLP as its architecture. Moreover, FM-KD can achieve better performance by using DKD as a loss function on CIFAR-100.

We are pleased to experiment with the optimal settings, and the additional experimental results are presented in Tables VI and VII. It can be observed that FM-KD outperforms all prior methods and achieves SOTA performance.

Table VI (ImageNet-1k, loss function is set as DIST, K=8)

| T-S pair | Meta-encoder | R50-MBV1 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=8) | 2-MLP | 74.22 |
| Top-1 Acc. (K=8) | 3-MLP | 74.28 |

Table VII (CIFAR-100, meta-encoder is set as Swin-Transformer, loss function is set as DKD)

| T-S pair | VGG13-VGG8 | VGG13-MBV2 |
| ---- | ---- | ---- |
| Top-1 Acc. (K=1) | 75.21 | 69.68 |
| Top-1 Acc. (K=2) | 74.86 | 69.52 |
| Top-1 Acc. (K=4) | 75.42 | 69.94 |
| Top-1 Acc. (K=8) | 74.46 | 69.94 |

Reviewer comment (Novelty):
The proposed method FM-KD heavily relies on two existing works, DiffKD (arxiv 2023) and Rectified Flow (ICLR 2023). Specifically, Rectified Flow is directly used to replace vanilla diffusion in DiffKD, making the diffusion of student representation (either feature or logits) have straight paths between any two steps. This makes FM-KD rather incremental. The authors claim FM-KD as a totally new knowledge transfer framework applicable to different types of teacher-student architectures, different metric-based distillation methods, and different representations, but this is mostly misleading, as DiffKD has already achieved this goal in the context of the authors' viewpoint.

Author response:
Thank you for your valuable feedback. FM-KD draws some inspiration from Rectified flow, yet it is distinct from DiffKD. The primary motivation behind developing FM-KD was to tackle potential limitations we identified in the modeling approach used in DiffKD. Unlike DiffKD, which incorporates three extra empirical losses, FM-KD employs just a single additional loss from theoretical derivation and performs better than DiffKD in both effectiveness and efficiency (please see Tables 1 and 2 of the revised paper).

**The main differences between FM-KD and DiffKD consist of three Parts:**

**Part 1: The loss in FM-KD is different from the form of each loss in DiffKD.** The losses in DiffKD are redundant and stacked. DiffKD employs the original training loss of the diffusion model (i.e., $L_{diff}$) and combines it with two additional auxiliary losses (i.e., $L_{ae}$ and $L_{diffkd}$) as detailed in its original paper for training. By contrast, we utilize our proposed training loss $L_{FM-KD}$ (solely one), guaranteed by Theorem 3.1 in the original/revised paper, which differs from each of $L_{diff}$, $L_{ae}$, and $L_{diffkd}$. To the best of our knowledge, this serial training paradigm has not been previously introduced in existing work.

**Part 2: FM-KD uses serial training paradigm while DiffKD uses original diffusion model training paradigm.** We believe the importance of our proposed serial training paradigm lies in its ability to effectively and stably propagate gradients back to the earlier layers of the student. By contrast, DiffKD might face challenges in functioning solely on the original diffusion model training paradigm, as it may struggle with preventing gradient vanishing. Though DiffKD aligns the sampling result $Z_0$ with the teacher feature/logit $X_T$ (i.e., $L_{diffkd}$ in DiffKD's original paper), ensuring backward gradient propagation to the student's earlier layers, DiffKD is highly sensitive to large Number of Function Evaluation (NFE) and the meta-encoder architecture in our experiments. We are pleased to note that FM-KD seems to avoid this particular issue.

**Part 3: DiffKD=$X^S$->Gaussian noise->$X^T$ and FM-KD=$X^S$->$X^T$.** DiffKD does not immediately effectuate the transition from $X^S$ to $X^T$. Instead, it first transforms $X^S$ into a Gaussian noise and subsequently translates this noise to $X^T$. This dual-stage transformation process might potentially be overly complex. By contrast, FM-KD offers a more straightforward approach by directly forming a bridge between the distributions of student and teacher features/logits.

We would like to note that the performance of DiffKD, which may be attributed to its heuristically design, could be perceived as not fully meeting the expected standards. For example, with the ResNet50-MobileNetV1 pair on ImageNet-1k, DiffKD obtains Top-1 Val Acc. 73.78% by increasing the GPU latency from 0.0108 to 0.0517 during inference, while FM-KD obtains Top-1 Val Acc. 74.22% by increasing the GPU latency from 0.0108 to 0.0151 during inference in Appendix J & K. Thus, FM-KD is more effective than DiffKD.

Reviewer comment (Novelty):
The underlying reason for applying a diffusion process to augment feature representation matching in knowledge distillation is not clear enough. There is a lack of clear and reasonable new technical insight.

Author response:
It may be helpful to clarify that our method does not employ a diffusion process, typically associated with SDE forward form. Instead, our approach is grounded in Rectified flow, which belongs to ODE forward form. The motivation for applying Rectified flow to augment feature representation matching (i.e. knowledge transfer) is as follows

1. Rectified flow can establish a bridge between two empirical distributions. This characteristic makes the Rectified flow particularly suitable for knowledge distillation in transferring knowledge from the teacher to the student, as both teacher and student features/logits are sampled from empirical distributions.

2. It is difficult to accomplish the knowledge transfer between two complex empirical distributions by one step. Thus, adopting the multi-step sampling from Rectified flow can realize the progressive transformation (i.e. probability path) from the student feature/logit distribution to the teacher feature/logit distribution, which leads to effective knowledge transfer.

3. As proven in Proposition 3.2 of our original/revised paper, FM-KD is implemented through Rectified flow and can be approximated as an implicit ensemble method, which is why FM-KD can achieve more precise knowledge transfer. Unfortunately, other noise schedules, such as VP ODE [*1,*3,*4] and VE ODE [*1,*3,*4], cannot achieve this goal.

4. FM-KD has an property that can be used to realize the trade-off between performance and efficiency by changing NFE in inference.

Reviewer comment (Experiments):
Rectified Flow assumes straight paths between any two steps during diffusion. Its original purpose is for faster diffusion, not for more accurate performance. It is unclear whether applying this design to feature representation matching in knowledge distillation is reasonable, especially given that from Table 2 on ImageNet-1K, FM-KD performs on par or worse than DiffKD.

Author response:
Thank you for your insightful question, our response consists of three points.

1. In our theoretical derivation, Rectified flow can be approximated an implicit ensemble method for effective knowledge transfer, other noise schedules, such as VP SDE [1*,3*,4*] and VE SDE [1*,3*,4*] cannot. This contrasts with other noise schedules like VP SDE and VE SDE, which may not be as effective due to the varying influence of truncation errors in non-straight noise schedules on the final result $Z_0$ from the Euler method.

2. Rectified flow can accelerate sampling (equivalent to minimizing the curvature) [2*] making it possible to be effective with very small NFE scenarios. Consequently, the fewer the NFEs, the smaller the training and inference computational burden, so we apply Rectified flow for feature representation matching. 

3. Rectified flow is not the only implementation of FM-KD. The assumption that the trajectory between two empirical distributions is straight is merely one modeling approach in optimal transport theory. It's viable, but not the only one. We unify VP ODE (w.r.t. VP SDE), VE ODE (w.r.t. VE SDE) and Rectified flow in FM-KD and present this in Appendix M of the revised paper. Interestingly, we discover that Rectified flow performed best, perhaps because Rectified flow can be approximated an implicit ensemble method.

Reviewer comment (Evaluation):
The authors did not discuss the limitations of the proposed method.

Author response:
Sorry that our oversight caused you concern, we have added limitation to Sec. 7 of the revised paper.

Reviewer comment (Experiments):
Since the proposed method is closely related to DiffKD, with the key difference being diffusion with Rectified Flow (straight paths) vs. vanilla diffusion (non-straight paths), DiffKD and DiffKD+RectifiedFlow should always be used as baselines instead of other methods.

Author response:
Thanks for this suggestion, and here we provide more experiments and discussions.

**For diffusion with vanilla diffusion (non-straight paths)**, we clarify that it is not feasible to compare FM-KD with vanilla diffusion processes since vanilla diffusion processes simply do not work in knowledge transfer. Specifically, in our exploratory studies, we try most of the ODE form of popular diffusion processes (non-straight paths), including VP ODE (converted from VP SDE and DDPM) and VE ODE (converted from VE SDE and NCSN). We also try vanilla Rectified flow (straight path). However, if these methods use the vanilla training paradigm:

Score function matching in VP ODE and VE ODE:

$ argmin_{v_\theta} E_{(Z_1,Z_0,t)} ||s_{v_\theta}(Z_t,t) - \nabla_{Z_t} \log p(Z_t|Z_0)||_2^2.  $

Velocity matching in Rectified flow:

$ argmin_{v_\theta} E_{(Z_1,Z_0,t)}||g_{v_\theta}(Z_t,t) - \frac{\partial Z_t}{\partial t}||_2^2. $

In our experiments, their final test accuracy will not exceed 5% with WRN-40-2-WRN-16-2 pair on CIFAR-100 (suffer from vanishing gradient). After we introduce the training paradigm in Eq. 3, as guaranteed by Theorem 3.1, Rectified flow can finally work effectively. Thus, the central focus of our paper is the training paradigm presented in Eq. 3. More importantly, we unify VP ODE, VE ODE, and Rectified flow within FM-KD through Theorem 3.1 in Appendix M, and we experimental demonstrate the effective modeling of both VP ODE and VE ODE. However, Rectified flow outperforms VP ODE and VE ODE in terms of performance and stability.

**For DiffKD**, we have presented additional comparative experimental results in Table III and substantiated that FM-KD is more effective than DiffKD. Furthermore, we conduct DiffKD to use the meta-encoder architecture (i.e. 2-MLP) from FM-KD, and FM-KD to use the meta-encoder architecture (2-Bottleneck+Conv+BN+MLP) from DiffKD, with the results presented in Table IV. The final results indicate that "DiffKD uses FM-KD's meta-encoder" underwent training collapse at epoch 1, whereas FM-KD (i.e. 74.26%) significantly surpassed DiffKD (i.e. 73.78%) in test accuracy when using the same meta-encoder architecture as DiffKD.

Table IV (with ResNet50-MobileNetV1 pair on ImageNet-1k)

| Method |DiffKD uses FM-KD's meta-encoder | FM-KD uses DiffKD's meta-encoder | DiffKD uses DiffKD's meta-encoder | FM-KD uses FM-KD's meta-encoder |
| ---- | ---- | ---- | ---- | ---- |
Top-1 Acc. | NAN | 74.26 | 73.78 | 74.22 |

**For DiffKD+RectifiedFlow**, we implemented subtle adjustments to the parameters $\alpha_t$ and $\sigma_t$ in $p(Z_t|Z_0) = \mathcal{N}(Z_t | \alpha_tZ_0,\sigma_t^2I)$, setting them to $1-t$ and $t$, respectively. These modifications were accompanied by a learning rate warm-up strategy specifically for DiffKD+Rectified flow applied to CIFAR-100. The results of these experiments are presented in Table V. From these results, it appears that FM-KD tends to be more effective in comparison to DiffKD+Rectified flow.

Table V (CIFAR-100)

| Teacher | ResNet56 | WRN-40-2 | WRN-40-2 |
| ---- | ---- | ---- | ---- |
| Student | ResNet20 | WRN-40-1 | WRN-16-2 |
| DiffKD+Rectified flow | 72.00 | 74.28 | 76.07 |
| DiffKD | 71.92 | 74.09 | 76.13 |
| FM-KD (K=4) | 75.12 | 76.24 | 77.69 |

Reviewer comment (Evaluation):
From the results in Table 2 on ImageNet-1K, FM-KD performs on par or worse than DiffKD, raising a critical question about the necessity of replacing vanilla diffusion (e.g., DDIM) with Rectified Flow. Additionally, the assumption of straight paths for any two steps during diffusion brings into question its suitability for feature representation matching in knowledge distillation.

Author response:
We address your concerns in the following two points.

1. The experimental results of FM-KD (i.e. Table 2 of the original paper) on ImageNet-1k are not obtained from optimal settings. We conduct additional experiments and the additional experimental results in Table III demonstrate that FM-KD substantially outperforms DiffKD.

2. DDIM cannot be used to replace Rectified flow since DDIM is merely an ODE solver and is not related to whether 'the path is straight or not'. We think you are referring to different ODE processes with non-straight paths, such as VP ODE (converted from VP SDE and DDPM) and VE ODE (converted from VE SDE and NCSN). In Appendix M of our revised paper, we unify VP ODE, VE ODE, and Rectified flow into the FM-KD framework for efficient training. The experimental results in Appendix M demonstrate that training with Rectified flow is more effective and stable than using VP ODE and VE ODE in the WRN-40-2-WRN-16-2 pair on CIFAR-100.

Reviewer comment (Evaluation):
There is no discussion on the extra cost to inference with the trained student model. The architecture of the trained student model is no longer the same as the baseline, which violates the basic purpose of knowledge distillation—model compression.

Author response:
We address this review in two Parts:

**Part 1:** We have presented GPU latency measurements during inference in Table VI. We can substantiate that the additional computational cost of FM-KD in inference is much lower compared with DiffKD.

Table VI (ImageNet-1k, hardware setting is 2*RTX 3080ti)

| Method | DiffKD | FM-KD | FM-KD $^{\Theta}$| DiffKD | FM-KD | FM-KD $^{\Theta}$ | classical KD methods |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Mode | training | training | training | inference | inference | inference | inference |
| R50-MBV1 | 0.2431 | 0.1281 | 0.1281 | 0.0517 | 0.0151 | 0.0108 | 0.0108 |
| R34-R18 | 0.2376 | 0.1139 | 0.1140 | 0.0390 | 0.0150 | 0.0108 | 0.0108 |


**Part 2:** To ensure a fair comparison with other knowledge distillation algorithms such as DKD, DIST, and SPKD, we have further designed FM-KD$^{\Theta}$, which refines the process by distilling $Z_0$ in FM-KD into an existing classification head (i.e. the original student's classification head) $\mathcal{T}_\mathrm{vanilla}(\cdot)$, thereby ensuring that no extra inference cost is incurred. More details can be found in Sec. 3.5 of the revised paper.

Table VII (CIFAR-100)

| Teacher | ResNet56 | WRN-40-2 | WRN-40-2 | ResNet32$\times$4 | VGG13 | VGG13 | WRN-40-2 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Student | ResNet20 | WRN-16-2 | WRN-40-1 | ResNet8$\times$4 | VGG8 | MobileNetV2 | ShuffleNetV1 |
| Top-1 Acc. | 72.20 | 75.98 | 74.99 | 76.52 | 74.82 | 69.90 | 77.19 |

Table VIII (ImageNet-1k)

| T-S pair | R34-R18 | R50-MBV1 |
| ---- | ---- | ---- |
| Top-1 Acc. | 72.14 | 73.29 |

The experimental results of FM-KD$^{\Theta}$ are presented in Tables VII and VIII, where the implementation details on ImageNet-1k and CIFAR-100 can be found in Appendix I of our revised paper. By comparing FM-KD$^{\Theta}$ with prior knowledge distillation algorithms like DKD, DIST, and SPKD in Tables 1 and 2 of the revised paper, we can conclude that FM-KD$^{\Theta}$ achieve SOTA performance across a wide range of teacher-student pairs.

Reviewer comment (Presentation):
There is no discussion of related works in the main paper. Although the authors included this in the Appendix, this placement is not proper to the best of the reviewer's understanding.

Author response:
Thank you for pointing out this presenting problem in our paper. We have moved Related work to Sec. 5 in the main paper.