Paper: Investigating the Impact of Data Distribution Shifts on Cross-Modal Knowledge Distillation (ICLR-2024)

Reviewer comment (Experiments):
The experiments show that the proposed enhanced KD method demonstrates performance improvement compared to the Baseline. However, can it achieve state-of-the-art performance compared to existing cross-modal KD methods (e.g., [a,b]) tailored to specific multimodal tasks?

Author response:
Thank you for your review and valuable feedback. We appreciate your concerns and would like to further clarify the intent and contributions of our research.

Firstly, we want to emphasize that the primary goal of our study is to explore the impact of data distribution shifts on cross-modal knowledge distillation, Specifically, the introduction of perceptual divergence distillation loss aims to eliminate inconsistent predictions in the solution space through masking, thereby validating the content of our proposed hypothesis. We observed a substantial increase in our PSSM distillation loss with larger differences in input data distribution, indicating that the inconsistency in the solution space caused by data distribution differences can lead to the failure of cross-modal knowledge distillation. We have expanded the discussion on this aspect in therevised version , and specific details can be found in *Appendix B.2* in the updated paper.

Secondly, we acknowledge that the practical application contributions of our method are relatively limited, and our focus leans more towards theoretical research. We will consider future directions for our work to comprehensively address practical applications and strive to provide more practical solutions for real-world scenarios.

In summary, our work is oriented towards validating specific hypotheses and making theoretical contributions. Once again, thank you for your suggestions and the time you dedicated to the review. We will make every effort to improve and meet your expectations.

Reviewer comment (Evaluation):
More qualitative cases are suggested to be provided to better illustrate the effects the proposed enhanced cross-modal KD method has.

Author response:
Thank you for your valuable feedback. To better illustrate the effectiveness of our proposed enhanced cross-modal knowledge distillation method, we have made the following additions:

On the sklearn dataset, we conducted additional analysis to demonstrate the effectiveness of our approach. Specifically, through experiments, we can elucidate that the efficacy of the PSSM method lies in its capability to induce consistency in the solution space of the "Target" section, while preserving the distinctiveness of the solution space for the "Non-target" section. For specific details, please refer to Appendix B.2 in the updated paper.
We trust that these supplemental experiments provide a more comprehensive understanding of the effectiveness of our approach. Your insightful suggestions have significantly contributed to the refinement of our work.

Reviewer comment (Presentation):
The figures are not clear enough, especially when zoomed in.

Author response:
Thank you for your valuable feedback. We have revised all the figures by using the vector graphics to make it clear when zoomed in. We apologize for any inconvenience caused by the initial lack of clarity in the pictures. We look forward to your improved experience with the updated image quality.

Reviewer comment (Novelty):
Although this article provides a SOLUTION SPACE DIVERGENCE HYPOTHERSIS, the proposed PERCEPTUAL SOLUTION SPACE MASK (PSSM) is simple and trivial and plays a similar role to other common knowledge distillation methods.

Author response:
**Response to Weakness 1:**
I extend my sincere appreciation for your invaluable feedback.

Concerning our proposed Perceptual Solution Space Mask (PSSM), it is imperative to emphasize its principal objective, which involves the introduction of a perceptual divergence distillation loss to rectify inconsistent predictions within the solution space through the application of masking. It is essential to clarify that our objective is not primarily focused on presenting a highly innovative knowledge distillation method; instead, it is directed towards substantiating our hypothesis concerning solution space divergence.

As elucidated in the revised paper (Appendix B.2 in the updated paper.), we observe a significant increase in the PSSM distillation loss when confronted with more substantial disparities in input data distribution. Conversely, the degree of enhancement is marginal or even diminishing when faced with minor discrepancies in data distribution. This observation signifies that the incongruity in the solution space, stemming from differences in data distribution, is a pivotal factor contributing to the inefficacy of crossmodal knowledge distillation.

Further elaboration on our research contributions and motivations is provided in Section 1 of the paper, underscoring the central theme of our investigation encapsulated in the paper's title: "Investigating the Impact of Data Distribution Shifts on Crossmodal Knowledge Distillation."

In conclusion, I wish to reiterate my gratitude for your constructive feedback. Your insightful suggestions have significantly enriched the overall quality of the paper.

Reviewer comment (Experiments):
There are few experiments in this paper. Please compare it with more classic single-modal and cross-modal knowledge distillation methods.

Author response:
**Response to Weakness 2:**
I appreciate your valuable feedback.

We have addressed and incorporated your suggestions, including conducting comparative experiments with five classic single-modal knowledge distillation methods.
Results demonstrate that, as data distribution differences increase, none of the single-modal methods exhibit effective performance, affirming the universal impact of data shift on cross-modal knowledge distillation.

Specific comparative results are presented in **Tables 8, 9, and 10** for your reference, offering a comprehensive view of our method's performance amidst data distribution differences and emphasizing the persistent challenge in traditional single-modal methods. We have added these experiments on **Page 17-18** in the revised version.

Thank you once again for your insightful feedback, contributing significantly to the paper's improvement.

Reviewer comment (Experiments):
Please compare it with more classical unimodal and crossmodal knowledge distillation methods as part of your experiments.

Author response:
**Response to Weakness 2:**
I appreciate your valuable feedback.

We have addressed and incorporated your suggestions, including conducting comparative experiments with five classic single-modal knowledge distillation methods.
Results demonstrate that, as data distribution differences increase, none of the single-modal methods exhibit effective performance, affirming the universal impact of data shift on cross-modal knowledge distillation.

Specific comparative results are presented in **Tables 8, 9, and 10** for your reference, offering a comprehensive view of our method's performance amidst data distribution differences and emphasizing the persistent challenge in traditional single-modal methods. We have added these experiments on **Page 17-18** in the revised version.

Thank you once again for your insightful feedback, contributing significantly to the paper's improvement.

Reviewer comment (Theory):
It is relatively difficult to connect data distribution shifts with KL divergence between two different probability distributions. The connection between data distribution shifts (in this paper, modality differences) and the "solution space" seems a little bit absurd, and further clarification is needed.

Author response:
We sincerely apologize for any confusion caused by inaccuracies in our expression. Typically, neural network parameters represent the solution space. However, in our paper, we establish a connection between changes in data distribution (specifically, modal differences) and the "solution space" based on the provided assumptions 1 and 2. Specifically, under the conditions of assumptions 1 and 2, the only distinction between cross-modal knowledge distillation and single-modal knowledge distillation is the difference in data distribution. Thus, we can replace the solution space, originally represented by network parameters, with the output probability distribution after inputting the data into the network. In summary, this connection is often invalid without the prerequisites of the first two assumptions.

Finally, we have revised and refined the expressions related to this section in the paper, making it more rigorous. For detailed information, please **refer to the updatedversion, specifically page 4**. We appreciate once again the constructive feedback you provided, which played a crucial role in enhancing the quality of the paper.

Reviewer comment (Theory):
Please check Eqn. (11), it might be wrong after the = symbol.

Author response:
Thank you for your meticulous review.We have revised it in the revised version.

Reviewer comment (Experiments):
For A.3, what are the results if input noisy MNIST to the student and use MNIST-M for the teacher?

Author response:
The reason we did not use "input noisy MNIST to student and use MNIST-M for teacher" is that, with the introduction of noise, the teacher's accuracy would sharply decline, while the student network's accuracy would remain high. In this scenario, the student would not effectively acquire knowledge from the teacher. To address your inquiry, we conducted additional experiments by employing noisy MNIST as input for the student network and MNIST-M as input for the teacher network. The results are shown in the table below:

| Noise Level | Teacher Accuracy | Student w/o KD | Student KD DKD | Student KD DKD with PSSM |
|:------------:|:----------------:|:---------------:|:---------------:|:------------------------:|
| 0            | 74.87           | 92.98           | 88.46 (-4.52 ± 0.18) | **92.15** (-0.83 ± 0.38)      |
| 1            | 37.77           | 92.98           | 70.28 (-22.70 ± 7.20) | **90.02** (-2.96 ± 2.00)  |
| 2            | 19.77           | 92.98            | 70.87 (-22.14 ± 7.31) | **79.28** (-13.70 ± 0.80) |
| 3            | 15.11            | 92.98            |67.52 (-25.96 ± 5.46) | **77.67** (-15.31 ± 4.03) |
| 4            | 12.42            | 92.98            |71.36 (-21.62 ± 0.80) | **74.86** (-18.12 ± 4.29) |
| 5            | 11.36            | 92.98            |64.36 (-28.62 ± 2.90) | **72.86** (-20.12 ± 3.49) |

Reviewer comment (Writing):
It would be beneficial to include a comprehensive symbol list in the Appendix. This addition will enhance the clarity of the notation used throughout the paper.

Author response:
**Response to Weakness 1:**
Thank you for providing constructive and meaningful suggestions. We will include a comprehensive symbol list in the appendix to enhance the clarity of symbol usage throughout the paper. This addition will offer readers a more comprehensive reference, ensuring they can accurately understand the symbols and their meanings used in the paper. The specific details can be found in the updated PDF paper in the *appendix A.*

| Symbol                  | Definition                                       |
|-------------------------|--------------------------------------------------|
| ${x}^{a}$               | Data from modality 'A'                           |
| ${x}^{b}$               | Data from modality 'B'                           |
| ${\mathcal{F}_{t}}$     | Teacher function (trained on ${x}^{a}$)          |
| ${\mathcal{F}_{s}}$     | Student function (trained on ${x}^{a}$ or ${x}^{b}$)  |
| $T$                     | Temperature parameter                            |
| ${y}_{i}$               | The true hard labels                             |
| ${s}_{i}$               | Soft predictions                                |
| ${\varepsilon}$         | Approximation error                              |
| $O(\cdot)$              | Estimation error                                 |
| $\sigma$                | The softmax operation                            |
| ${{\left| \cdot \right|}_{C}}$ | Some function class capacity measure   |
| $n$                     | The number of data points                        |

Reviewer comment (Presentation):
The format of the paper requires further attention. Specifically, the use of `\cite` and `\citet` appears confusing. A consistent and clear citation style should be maintained throughout the manuscript.

Author response:
**Response to Weakness 3:**
Thank you very much for your meticulous review. When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \verb|\citet{}|. Otherwise, the citation should be in parenthesis using \verb|\citep{}|. We have thoroughly checked and corrected all citations in the paper to ensure consistency. The revised results can be found in the updated PDF version of the paper. Once again, we appreciate your valuable suggestions, which have significantly enhanced the standardization of the paper.