Original: Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation
Revised: Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation

Original: Abstract
Revised: Abstract

Original: Distillation in neural networks using only the samples randomly drawn from a
Revised: Distillation in neural networks using only the samples randomly drawn from a

Original: Gaussian distribution is possibly the most straightforward solution one can think of for the complex problem of knowledge transfer from one network (teacher) to the other (student).
Revised: Gaussian distribution is possibly the most straightforward solution one can think of for the complex problem of knowledge transfer from one network (teacher) to the other (student).

Original: If successfully done, it can eliminate the requirement of teacher’s training data for knowledge distillation and avoid often arising privacy concerns in sensitive applications such as healthcare.
Revised: If successfully done, it can eliminate the requirement of teacher’s training data for knowledge distillation and avoid often arising privacy concerns in sensitive applications such as healthcare.

Original: There have been some recent attempts at Gaussian noise-based data-free knowledge distillation, however, none of them offer a consistent or reliable solution.
Revised: There have been some recent attempts at Gaussian noise-based data-free knowledge distillation, however, none of them offer a consistent or reliable solution.

Original: We identify the shift in the distribution of hidden layer activation as the key limiting factor, which occurs when Gaussian noise is fed to the teacher network instead of the accustomed training data.
Revised: We identify the shift in the distribution of hidden layer activation as the key limiting factor, which occurs when Gaussian noise is fed to the teacher network instead of the accustomed training data.

Original: We propose a simple solution to mitigate this shift and show that for vision tasks, such as classification, it is possible to achieve a performance close to the teacher by just using the samples randomly drawn from a Gaussian distribution.
Revised: We propose a simple solution to mitigate this shift and show that for vision tasks, such as classification, it is possible to achieve a performance close to the teacher by just using the samples randomly drawn from a Gaussian distribution.

Original: We validate our approach on CIFAR10, CIFAR100, SVHN, and Food101 datasets.
Revised: We validate our approach on CIFAR10, CIFAR100, SVHN, and Food101 datasets.

Original: We further show that in situations of sparsely available original data for distillation, the proposed
Revised: We further show that in situations of sparsely available original data for distillation, the proposed

Original: Gaussian noise-based knowledge distillation method can outperform the distillation using the available data with a large margin.
Revised: Gaussian noise-based knowledge distillation method can outperform the distillation using the available data with a large margin.

Original: Our work lays the foundation for further research in the direction of noise-engineered knowledge distillation using random samples.
Revised: Our work lays the foundation for further research in the direction of noise-engineered knowledge distillation using random samples.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks are an excellent choice for various real-world computer vision tasks.
Revised: Deep neural networks are an excellent choice for various real-world computer vision tasks.

Original: However, their high computational and space complexity becomes a bottleneck when it comes to deployment on resource-constrained devices.
Revised: However, their high computational and space complexity becomes a bottleneck when it comes to deployment on resource-constrained devices.

Original: To address this issue, the idea of Knowledge Distillation (KD), proposed by [1], has recently gained significant attention.
Revised: To address this issue, the idea of Knowledge Distillation (KD), proposed by [1], has recently gained significant attention.

Original: It enables the transfer of knowledge from a large neural network to a comparatively smaller one.
Revised: It enables the transfer of knowledge from a large neural network to a comparatively smaller one.

Original: The lightweight small neural network, often referred to as student, gains the information learned by the over-parameterized large model, referred to as teacher, and delivers a similar or sometimes even better performance [2].
Revised: The lightweight small neural network, often referred to as student, gains the information learned by the over-parameterized large model, referred to as teacher, and delivers a similar or sometimes even better performance [2].

Original: Traditionally, in KD, original data is passed through the accurate teacher model to collect soft labels, and subsequently, these soft labels are used to supervise the student [3].
Revised: Traditionally, in KD, original data is passed through the accurate teacher model to collect soft labels, and subsequently, these soft labels are used to supervise the student [3].

Original: However, in certain applications such as medical, it often becomes impossible to release the original data due to security concerns, sensitive information, or storage restrictions.
Revised: However, in certain applications such as medical, it often becomes impossible to release the original data due to security concerns, sensitive information, or storage restrictions.

Original: In contrast, a trained model (teacher) usually does not hold a specific record or specific piece of information thus, no certainty or validity of the information extracted through a neural network can be guaranteed, thereby making them safer for public release.
Revised: In contrast, a trained model (teacher) usually does not hold a specific record or specific piece of information thus, no certainty or validity of the information extracted through a neural network can be guaranteed, thereby making them safer for public release.

Original: However, these models certainly contain information in the parameterized form for performing the downstream tasks, such as classification or object detection.
Revised: However, these models certainly contain information in the parameterized form for performing the downstream tasks, such as classification or object detection.

Original: [Figure]
Revised: [Figure]

Original: Due to these reasons, recently there has been an interest in data-free distillation - that is, distilling knowledge from one neural network to the other without using any data.
Revised: Due to these reasons, recently there has been an interest in data-free distillation - that is, distilling knowledge from one neural network to the other without using any data.

Original: Majority of methods in this direction aim to create synthetic images either by optimizing the randomly generated samples [4, 5, 6, 7, 8] or by training a generative network [9, 10], which often is computationally expensive.
Revised: Majority of methods in this direction aim to create synthetic images either by optimizing the randomly generated samples [4, 5, 6, 7, 8] or by training a generative network [9, 10], which often is computationally expensive.

Original: This motivates the research community to ask the question Can distillation be done by using the samples randomly drawn from a known distribution such as standard normal distribution?
Revised: This motivates the research community to ask the question Can distillation be done by using the samples randomly drawn from a known distribution such as standard normal distribution?

Original: To best of our knowledge, we for the first time provide a satisfactory answer to the question and show that yes, it is possible to do data-free KD by only using images that are randomly sampled from a Gaussian distribution.
Revised: To best of our knowledge, we for the first time provide a satisfactory answer to the question and show that yes, it is possible to do data-free KD by only using images that are randomly sampled from a Gaussian distribution.

Original: Our experiments show that a considerably better performance is possible using only the
Revised: Our experiments show that a considerably better performance is possible using only the

Original: Gaussian noise as the input, in contrast to what prior works ([9], [10]) have shown for this particular setting.
Revised: Gaussian noise as the input, in contrast to what prior works ([9], [10]) have shown for this particular setting.

Original: We identify that the key limitation, which the prior works failed to identify is the covariate shift in the distribution of hidden layer activations of the teacher network caused by feeding Gaussian noise instead of the expected original data.
Revised: We identify that the key limitation, which the prior works failed to identify is the covariate shift in the distribution of hidden layer activations of the teacher network caused by feeding Gaussian noise instead of the expected original data.

Original: As shown in Figure 1a, the activation distribution when input is Gaussian noise (blue) is shifted, thus no longer aligns with the activation distribution when input is original data (red).
Revised: As shown in Figure 1a, the activation distribution when input is Gaussian noise (blue) is shifted, thus no longer aligns with the activation distribution when input is original data (red).

Original: We propose a simple yet surprisingly effective solution to reduce this shift by using the current statistics instead of using statistics of the original data (running statistics) in the teacher’s batch normalization (BatchNorm) layers and thus distill the knowledge from teacher into student network using just the Gaussian noise.
Revised: We propose a simple yet surprisingly effective solution to reduce this shift by using the current statistics instead of using statistics of the original data (running statistics) in the teacher’s batch normalization (BatchNorm) layers and thus distill the knowledge from teacher into student network using just the Gaussian noise.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • We explain how covariate shift in the teacher network interferes with Gaussian noise-based KD and propose a simple solution to eliminate it, thus enable knowledge transfer using Gaussian noise.
Revised: • We explain how covariate shift in the teacher network interferes with Gaussian noise-based KD and propose a simple solution to eliminate it, thus enable knowledge transfer using Gaussian noise.

Original: • We show how this phenomena of covariate shift occurs in the student network as well and mitigate it in a similar way.
Revised: • We show how this phenomena of covariate shift occurs in the student network as well and mitigate it in a similar way.

Original: • We obtain student performance close to the teacher without using any original or synthetic data.
Revised: • We obtain student performance close to the teacher without using any original or synthetic data.

Original: We further show that in the presence of limited data, the proposed approach outperforms the traditional distillation method with a large margin.
Revised: We further show that in the presence of limited data, the proposed approach outperforms the traditional distillation method with a large margin.

Original: • Our work counters the commonly accepted notion of requiring realistic training data for data-free KD.
Revised: • Our work counters the commonly accepted notion of requiring realistic training data for data-free KD.

Original: Thus, we lay the foundation for noise-based data-free KD.
Revised: Thus, we lay the foundation for noise-based data-free KD.

Original:  Related work
Revised:  Related work

Original: Knowledge Distillation: KD is the transfer of knowledge from one neural network to the other.
Revised: Knowledge Distillation: KD is the transfer of knowledge from one neural network to the other.

Original: First successful implementation of knowledge transfer was shown by [11] for the purpose of model compression.
Revised: First successful implementation of knowledge transfer was shown by [11] for the purpose of model compression.

Original: Later, the term knowledge distillation was popularized by [1], where the authors explored so called "dark knowledge" of a teacher model which is transferred to a student model with the help of soft labels.
Revised: Later, the term knowledge distillation was popularized by [1], where the authors explored so called "dark knowledge" of a teacher model which is transferred to a student model with the help of soft labels.

Original: Since then, the domain of KD has witnessed many interested ideas related to improved distillation strategies such as teacher assistant [12] and self-distillation [13].
Revised: Since then, the domain of KD has witnessed many interested ideas related to improved distillation strategies such as teacher assistant [12] and self-distillation [13].

Original: KD approaches for performing various downstream tasks such as object detection [14], semantic segmentation [15], and also the applications in different modalities, including speech [16] and graph [17] have been explored.
Revised: KD approaches for performing various downstream tasks such as object detection [14], semantic segmentation [15], and also the applications in different modalities, including speech [16] and graph [17] have been explored.

Original: A detailed survey on KD is recently reported by [2].
Revised: A detailed survey on KD is recently reported by [2].

Original: Data-free Knowledge Distillation: The problem of data-free KD, i.e., the constraint that the original dataset, which was used to train and supervise the teacher, is unavailable for distillation makes the tasks naturally challenging.
Revised: Data-free Knowledge Distillation: The problem of data-free KD, i.e., the constraint that the original dataset, which was used to train and supervise the teacher, is unavailable for distillation makes the tasks naturally challenging.

Original: This often is encountered in privacy-sensitive settings, where the original dataset cannot be shared.
Revised: This often is encountered in privacy-sensitive settings, where the original dataset cannot be shared.

Original: There are various prior works with a similar context in mind.
Revised: There are various prior works with a similar context in mind.

Original: Most of these methods first construct a set of synthetic images that resembles original data, then distill the knowledge of the teacher network into the student network using that synthetic data [4, 5, 6, 8, 9, 10].
Revised: Most of these methods first construct a set of synthetic images that resembles original data, then distill the knowledge of the teacher network into the student network using that synthetic data [4, 5, 6, 8, 9, 10].

Original: Some approaches synthesize images one-by-one or batch-by-batch, whereas some train a generator model to gain access to limitless synthetic data.
Revised: Some approaches synthesize images one-by-one or batch-by-batch, whereas some train a generator model to gain access to limitless synthetic data.

Original: For example, [4] stored activation records of the teacher model when original data is feed-forwarded through it, and then later used it to create synthetic images by optimizing random noise using gradient descent.
Revised: For example, [4] stored activation records of the teacher model when original data is feed-forwarded through it, and then later used it to create synthetic images by optimizing random noise using gradient descent.

Original: [5] introduced DeepInversion intending to create realistic images, which makes use of feature map and BatchNorm statistics to optimize random images into high-fidelity images.
Revised: [5] introduced DeepInversion intending to create realistic images, which makes use of feature map and BatchNorm statistics to optimize random images into high-fidelity images.

Original: [18] takes a different approach to simultaneously generate images, as well as train the student.
Revised: [18] takes a different approach to simultaneously generate images, as well as train the student.

Original: They introduced Zero-shot knowledge transfer, which utilized adversarial learning between the generator model and the student model.
Revised: They introduced Zero-shot knowledge transfer, which utilized adversarial learning between the generator model and the student model.

Original: Their intuition was to search for images on which the student poorly matches the teacher and then train the student with those.
Revised: Their intuition was to search for images on which the student poorly matches the teacher and then train the student with those.

Original: Gaussian Noise-based Knowledge Distillation: KD using Gaussian noise has been presented as a baseline numerous times in prior works, which resulted in a student accuracy slightly better than random chance.
Revised: Gaussian Noise-based Knowledge Distillation: KD using Gaussian noise has been presented as a baseline numerous times in prior works, which resulted in a student accuracy slightly better than random chance.

Original: Although a decent accuracy was reported on MNIST (88% [9], 75% [19]), probably due to smaller networks and less complex dataset, on more challenging datasets like CIFAR10, a poor performance (14% [5], 15% [9], 11% [10], 10% [19]) was observed.
Revised: Although a decent accuracy was reported on MNIST (88% [9], 75% [19]), probably due to smaller networks and less complex dataset, on more challenging datasets like CIFAR10, a poor performance (14% [5], 15% [9], 11% [10], 10% [19]) was observed.

Original: Apart from these, [6] presented an interesting, although a highly constrained scenario where the student had an identical architecture to the teacher and student’s weights were initialized as quantized values of teacher’s weights.
Revised: Apart from these, [6] presented an interesting, although a highly constrained scenario where the student had an identical architecture to the teacher and student’s weights were initialized as quantized values of teacher’s weights.

Original: Further when the BatchNorm layer parameters from teacher are copied to student and kept frozen, Gaussian noise based distillation resulted in good performance.
Revised: Further when the BatchNorm layer parameters from teacher are copied to student and kept frozen, Gaussian noise based distillation resulted in good performance.

Original: Moreover, instead of training the student network end-to-end as in traditional KD, [6] partitioned the student and teacher networks across BatchNorm layers resulting in multiple teacher-student pairs of blocks and distilled each pair of blocks separately.
Revised: Moreover, instead of training the student network end-to-end as in traditional KD, [6] partitioned the student and teacher networks across BatchNorm layers resulting in multiple teacher-student pairs of blocks and distilled each pair of blocks separately.

Original: Although this approach works, it cannot be used for a pair of architectures not having similar block structures and is unnecessarily complex.
Revised: Although this approach works, it cannot be used for a pair of architectures not having similar block structures and is unnecessarily complex.

Original: In contrast to all above, our aim is to design a simple approach for Gaussian noise-based data free distillation.
Revised: In contrast to all above, our aim is to design a simple approach for Gaussian noise-based data free distillation.

Original: This would not only relax the constraints of approaches such as [6] but also save the efforts required in creating synthetic dataset.
Revised: This would not only relax the constraints of approaches such as [6] but also save the efforts required in creating synthetic dataset.

Original:  Methodology 3.1 Preliminaries
Revised:  Methodology 3.1 Preliminaries

Original: Knowledge Distillation: KD is achieved by training the student network using the output of the teacher network as soft labels instead of using the ground truth labels [3].
Revised: Knowledge Distillation: KD is achieved by training the student network using the output of the teacher network as soft labels instead of using the ground truth labels [3].

Original: This results in matching of the output probability distributions of both the networks for a given input data.
Revised: This results in matching of the output probability distributions of both the networks for a given input data.

Original: Formally, for a k-class classification, given a dataset D that consists of data samples x , a teacher network T trained on D , and a student network S with randomly initialized weights, the following cost function is optimized:
Revised: Formally, for a k-class classification, given a dataset D that consists of data samples x , a teacher network T trained on D , and a student network S with randomly initialized weights, the following cost function is optimized:

Original: [Equation]
Revised: [Equation]

Original: where L XE ( ., . ) is the standard cross-entropy term.
Revised: where L XE ( ., . ) is the standard cross-entropy term.

Original: Batch Normalization: BatchNorm was introduced by [20] that made the training of deep neural networks more stable and allowed to use larger learning rates, hence faster convergence.
Revised: Batch Normalization: BatchNorm was introduced by [20] that made the training of deep neural networks more stable and allowed to use larger learning rates, hence faster convergence.

Original: Further, it makes neural networks more robust to initialization schemes and changes in the learning rate.
Revised: Further, it makes neural networks more robust to initialization schemes and changes in the learning rate.

Original: BatchNorm first normalizes the output of the hidden layer (input to BatchNorm layer) using the first (mean) and the second (variance) statistical moments of the current batch.
Revised: BatchNorm first normalizes the output of the hidden layer (input to BatchNorm layer) using the first (mean) and the second (variance) statistical moments of the current batch.

Original: Then, these normalized outputs are shifted and scaled according to the two trainable parameters γ and β , respectively.
Revised: Then, these normalized outputs are shifted and scaled according to the two trainable parameters γ and β , respectively.

Original: These parameters are tuned as any other neural network parameters, and they help in choosing the optimum distribution as the input for the next layer, thus regaining expressiveness lost in the normalization step.
Revised: These parameters are tuned as any other neural network parameters, and they help in choosing the optimum distribution as the input for the next layer, thus regaining expressiveness lost in the normalization step.

Original: Formally, let h be the activations of a neuron in the previous layer (input to the BatchNorm layer), m be the batch-size, and ϵ be a small constant used for numerical stability.
Revised: Formally, let h be the activations of a neuron in the previous layer (input to the BatchNorm layer), m be the batch-size, and ϵ be a small constant used for numerical stability.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where µ B = 1 m (cid:80) i h i and σ 2 B = 1 m (cid:80)
Revised: where µ B = 1 m (cid:80) i h i and σ 2 B = 1 m (cid:80)

Original: i
Revised: i

Original: ( h i − µ B ) 2 .
Revised: ( h i − µ B ) 2 .

Original: The h i and (cid:98) h
Revised: The h i and (cid:98) h

Original: i show normalized and subsequently shifted values of h i .
Revised: i show normalized and subsequently shifted values of h i .

Original: Above mentioned activation normalization using mini-batch or current statistics results in improved training.
Revised: Above mentioned activation normalization using mini-batch or current statistics results in improved training.

Original: During evaluation or inference, the batch statistics µ B and σ B are replaced with running or population statistics, µ r = E B [ µ B ] and σ r = mm − 1 E B [ σ 2 B ] , where E B [ . ] denotes expectation over training mini-batches.
Revised: During evaluation or inference, the batch statistics µ B and σ B are replaced with running or population statistics, µ r = E B [ µ B ] and σ r = mm − 1 E B [ σ 2 B ] , where E B [ . ] denotes expectation over training mini-batches.

Original: Accordingly, the normalization steps are modified as follows:
Revised: Accordingly, the normalization steps are modified as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ r = αµ r
Revised: where µ r = αµ r

Original: + (1 − α ) µ B and σ r = ασ r + (1 − α ) σ B .
Revised: + (1 − α ) µ B and σ r = ασ r + (1 − α ) σ B .

Original: Here α is a smoothing factor, which is there to prioritize statistics of the current batch.
Revised: Here α is a smoothing factor, which is there to prioritize statistics of the current batch.

Original: 3.2 Distillation using Gaussian Noise
Revised: 3.2 Distillation using Gaussian Noise

Original: We consider the most straightforward approach for KD in the absence of original data, just using random images sampled from a Gaussian distribution as an input to teacher and student network.
Revised: We consider the most straightforward approach for KD in the absence of original data, just using random images sampled from a Gaussian distribution as an input to teacher and student network.

Original: This precondition to use Gaussian noise for KD might seem to make the task of distillation almost impossible since the input Gaussian distribution is notably different from the original data distribution, and does not contain any structural properties or representation as the original data.
Revised: This precondition to use Gaussian noise for KD might seem to make the task of distillation almost impossible since the input Gaussian distribution is notably different from the original data distribution, and does not contain any structural properties or representation as the original data.

Original: We hypothesize that, for successful KD, across two input data: first using which the teacher is trained, and second using which student is trained, having similar output distributions throughout the teacher network’s hidden layers is more important than having the same structural or representation properties at the input; thus, leading to the extraction of meaningful information from the teacher network.
Revised: We hypothesize that, for successful KD, across two input data: first using which the teacher is trained, and second using which student is trained, having similar output distributions throughout the teacher network’s hidden layers is more important than having the same structural or representation properties at the input; thus, leading to the extraction of meaningful information from the teacher network.

Original: To understand this better let us consider a trivial case where samples drawn from a standard Gaussian distribution ( N (0 , 1) ) are passed through a BatchNorm layer optimized for some real data D .
Revised: To understand this better let us consider a trivial case where samples drawn from a standard Gaussian distribution ( N (0 , 1) ) are passed through a BatchNorm layer optimized for some real data D .

Original: As during distillation only student network is trained, BatchNorm performs normalization of input according to (4).
Revised: As during distillation only student network is trained, BatchNorm performs normalization of input according to (4).

Original: Consider the following statistics of the normalized signal.
Revised: Consider the following statistics of the normalized signal.

Original: [Equation]
Revised: [Equation]

Original: Equation (5) and (6) clearly show the deviation or covariate shift in the output of BatchNorm when
Revised: Equation (5) and (6) clearly show the deviation or covariate shift in the output of BatchNorm when

Original: Gaussian noise is passed instead of the original data.
Revised: Gaussian noise is passed instead of the original data.

Original: This suggest that output of BatchNorm should be shifted by
Revised: This suggest that output of BatchNorm should be shifted by

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: ϵ and scaled by σ 2 r + ϵ before feeding it to the subsequent layers.
Revised: ϵ and scaled by σ 2 r + ϵ before feeding it to the subsequent layers.

Original: However, this is not trivial for intermediate hidden layers of the teacher network as the activation statistics are unknown.
Revised: However, this is not trivial for intermediate hidden layers of the teacher network as the activation statistics are unknown.

Original: Alternatively, as explained below, we may allow BatchNorm to simply use current statistics instead of running statistics as the corrections required are nothing but the factors of running statistics.
Revised: Alternatively, as explained below, we may allow BatchNorm to simply use current statistics instead of running statistics as the corrections required are nothing but the factors of running statistics.

Original: 3.2.1 Mitigating the Shift
Revised: 3.2.1 Mitigating the Shift

Original: The solution to mitigate the observed shift and successfully perform distillation using Gaussian noise is to effectively make use of the BatchNorm layers.
Revised: The solution to mitigate the observed shift and successfully perform distillation using Gaussian noise is to effectively make use of the BatchNorm layers.

Original: Our goal is to align the activation distribution of hidden layers throughout the teacher network when Gaussian noise is used as an input to that when original data is used as an input.
Revised: Our goal is to align the activation distribution of hidden layers throughout the teacher network when Gaussian noise is used as an input to that when original data is used as an input.

Original: We consider the case of a BatchNorm layer placed after a hidden layer H within a teacher network.
Revised: We consider the case of a BatchNorm layer placed after a hidden layer H within a teacher network.

Original: Let B represent a batch of images sampled from original dataset D , G be a batch of random samples sampled from a Gaussian distribution, h be the vector of activations for some neuron n belonging to the hidden layer H when inputs are fed to the teacher with a batch size m , and P ( h ) be the activation distribution of that
Revised: Let B represent a batch of images sampled from original dataset D , G be a batch of random samples sampled from a Gaussian distribution, h be the vector of activations for some neuron n belonging to the hidden layer H when inputs are fed to the teacher with a batch size m , and P ( h ) be the activation distribution of that

Original: n th neuron over the batch.
Revised: n th neuron over the batch.

Original: Due to the covariate shift, we observe that: P ( h | G )
Revised: Due to the covariate shift, we observe that: P ( h | G )

Original: ̸
Revised: ̸

Original: = P ( h | B ) .
Revised: = P ( h | B ) .

Original: Generally, during KD, the BatchNorm layers in the teacher network are setup for inference, hence use running statistics ( µ r , σ r ), i.e., Equation (4) to calculate the normalized activations h .
Revised: Generally, during KD, the BatchNorm layers in the teacher network are setup for inference, hence use running statistics ( µ r , σ r ), i.e., Equation (4) to calculate the normalized activations h .

Original: This leads to:
Revised: This leads to:

Original: [Equation]
Revised: [Equation]

Original: Thus ,
Revised: Thus ,

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:98) h is the scaled and shifted activation vector, which is the output of the BatchNorm layer for the n th neuron, as defined in Equation (4).
Revised: Here, (cid:98) h is the scaled and shifted activation vector, which is the output of the BatchNorm layer for the n th neuron, as defined in Equation (4).

Original: This drifted (cid:98) h propagates to the next hidden layer and eventually to the output layer without any meaningful information being passed from the hidden layer H .
Revised: This drifted (cid:98) h propagates to the next hidden layer and eventually to the output layer without any meaningful information being passed from the hidden layer H .

Original: Therefore, for KD using Gaussian noise, we propose to use current batch statistics ( µ B , σ B ) for the normalization step, i.e., Equation (2) in all BatchNorm layers of the teacher network.
Revised: Therefore, for KD using Gaussian noise, we propose to use current batch statistics ( µ B , σ B ) for the normalization step, i.e., Equation (2) in all BatchNorm layers of the teacher network.

Original: The intuition is to align the two drifted distributions at the normalization step.
Revised: The intuition is to align the two drifted distributions at the normalization step.

Original: This makes sense only because γ and β have the inherent information about the distribution that the next layer expects and are designed to shift a standard normal distribution.
Revised: This makes sense only because γ and β have the inherent information about the distribution that the next layer expects and are designed to shift a standard normal distribution.

Original: Therefore, even though P ( h | G ) ̸
Revised: Therefore, even though P ( h | G ) ̸

Original: = P
Revised: = P

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: B ) holds true:
Revised: B ) holds true:

Original: [Equation]
Revised: [Equation]

Original: Hence , P ( (cid:98)
Revised: Hence , P ( (cid:98)

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: This reduces the covariate shift due to a different input distribution from the expected one at every stage (BatchNorm layer) in the teacher network.
Revised: This reduces the covariate shift due to a different input distribution from the expected one at every stage (BatchNorm layer) in the teacher network.

Original: It obtains P ( (cid:98) h ) similar to that of using original data irrespective of P ( h ) , and hence leads to the transfer of meaningful information.
Revised: It obtains P ( (cid:98) h ) similar to that of using original data irrespective of P ( h ) , and hence leads to the transfer of meaningful information.

Original: Our approach is visually explained in Figure 2, where we show an expanded view of a BatchNorm layer for three neurons.
Revised: Our approach is visually explained in Figure 2, where we show an expanded view of a BatchNorm layer for three neurons.

Original: It shows how our proposed method effectively uses the normalization step to align the two activation distributions: (i) when Gaussian noise is fed as inputs (ii) when original data is fed as inputs.
Revised: It shows how our proposed method effectively uses the normalization step to align the two activation distributions: (i) when Gaussian noise is fed as inputs (ii) when original data is fed as inputs.

Original: Note that the covariate shift discussed throughout this paper is not the same as the internal covariate shift hypothesized by [20], even though ours might be happening in the hidden layers of the neural network as well.
Revised: Note that the covariate shift discussed throughout this paper is not the same as the internal covariate shift hypothesized by [20], even though ours might be happening in the hidden layers of the neural network as well.

Original: The internal covariate shift occurs due to constantly changing network parameters in the preceding layer, thus affecting the input distribution for the following layer.
Revised: The internal covariate shift occurs due to constantly changing network parameters in the preceding layer, thus affecting the input distribution for the following layer.

Original: On the other hand, we consider the shift due to change in the input that is being fed to the network, which in turn affects the input distribution of hidden layers as well.
Revised: On the other hand, we consider the shift due to change in the input that is being fed to the network, which in turn affects the input distribution of hidden layers as well.

Original: 3.3 Inference
Revised: 3.3 Inference

Original: Similar to how a teacher network trained using original data distribution cannot directly handle the
Revised: Similar to how a teacher network trained using original data distribution cannot directly handle the

Original: Gaussian noise, the student network trained using Gaussian noise will also not be able to provide correct output during test time, as the running statistics of BatchNorm layers in the student expect
Revised: Gaussian noise, the student network trained using Gaussian noise will also not be able to provide correct output during test time, as the running statistics of BatchNorm layers in the student expect

Original: Gaussian noise activation patterns.
Revised: Gaussian noise activation patterns.

Original: The inference is, therefore, done using current statistics.
Revised: The inference is, therefore, done using current statistics.

Original: Moreover, each mini-batch needs to be independent and identically distributed, and large enough to approximate the test data distribution.
Revised: Moreover, each mini-batch needs to be independent and identically distributed, and large enough to approximate the test data distribution.

Original: Usually, it should not matter how the input is arranged during the inference, therefore, a simple workaround to this problem is to first adjust the running statistics of the student network catering to the test data by feed-forwarding some amount of shuffled test data before evaluating the student network.
Revised: Usually, it should not matter how the input is arranged during the inference, therefore, a simple workaround to this problem is to first adjust the running statistics of the student network catering to the test data by feed-forwarding some amount of shuffled test data before evaluating the student network.

Original: Note that no back-propagation or optimization steps are required as only the running statistics need to be adjusted.
Revised: Note that no back-propagation or optimization steps are required as only the running statistics need to be adjusted.

Original: We summarize the training and evaluation phases of our approach in Algorithm 1 and Algorithm 2 respectively.
Revised: We summarize the training and evaluation phases of our approach in Algorithm 1 and Algorithm 2 respectively.

Original: Algorithm
Revised: Algorithm

Original: Requires: pretrained teacher T ( . )
Revised: Requires: pretrained teacher T ( . )

Original: Initialize: student S ( . ; θ ) with parameters θ for B in 1 , 2 , ..., B 1 do
Revised: Initialize: student S ( . ; θ ) with parameters θ for B in 1 , 2 , ..., B 1 do

Original: G ∼ N (0 , 1) y T ← T ( G | µ B , σ B ) y S ← S ( G | θ, µ B , σ B ) θ ← θ − η ∂L KD ∂θ end for
Revised: G ∼ N (0 , 1) y T ← T ( G | µ B , σ B ) y S ← S ( G | θ, µ B , σ B ) θ ← θ − η ∂L KD ∂θ end for

Original: Algorithm
Revised: Algorithm

Original: Requires: pretrained student S ( . ; θ ) for B in 1 , 2 , ..., B 2 do
Revised: Requires: pretrained student S ( . ; θ ) for B in 1 , 2 , ..., B 2 do

Original: X ∼ D y S ← S
Revised: X ∼ D y S ← S

Original: ( X | θ, µ B , σ B ) y label ← argmax ( y S ) end for
Revised: ( X | θ, µ B , σ B ) y label ← argmax ( y S ) end for

Original:  Experiments
Revised:  Experiments

Original: In this section, we discuss the experiments we performed on various datasets in combination with different standard neural network architectures to demonstrate the working of our proposed method.
Revised: In this section, we discuss the experiments we performed on various datasets in combination with different standard neural network architectures to demonstrate the working of our proposed method.

Original: We use CIFAR10 [21] as a proof of concept to establish the validity of our method, and for certain ablations.
Revised: We use CIFAR10 [21] as a proof of concept to establish the validity of our method, and for certain ablations.

Original: We further test our proposed approach on SVHN [22], CIFAR100, and Food101 [23].
Revised: We further test our proposed approach on SVHN [22], CIFAR100, and Food101 [23].

Original: Note that, in all the experiments, for Gaussian noise, we generate the image pixels from Gaussian distribution having mean, µ = 0 and variance, σ 2 = 1 .
Revised: Note that, in all the experiments, for Gaussian noise, we generate the image pixels from Gaussian distribution having mean, µ = 0 and variance, σ 2 = 1 .

Original: 4.1 State of BatchNorm
Revised: 4.1 State of BatchNorm

Original: In this experiment, we discuss the effect of using the BatchNorm layer in two different states, i.e., compare between using running statistics and current statistics.
Revised: In this experiment, we discuss the effect of using the BatchNorm layer in two different states, i.e., compare between using running statistics and current statistics.

Original: Note that we do this change only in the case of the teacher network.
Revised: Note that we do this change only in the case of the teacher network.

Original: In contrast, the student network always uses current statistics, even at the time of evaluation on the validation set.
Revised: In contrast, the student network always uses current statistics, even at the time of evaluation on the validation set.

Original: We do this experiment on CIFAR10 by considering two independent parameters: (i) the state of BatchNorm layers in the teacher network, and (ii) the input that is being fed to both the networks.
Revised: We do this experiment on CIFAR10 by considering two independent parameters: (i) the state of BatchNorm layers in the teacher network, and (ii) the input that is being fed to both the networks.

Original: We distill a ResNet-34 teacher network pretrained on CIFAR10 into student networks of varying architectures, which are: ResNet-34, ResNet-18, and MobileNetV2.
Revised: We distill a ResNet-34 teacher network pretrained on CIFAR10 into student networks of varying architectures, which are: ResNet-34, ResNet-18, and MobileNetV2.

Original: In case of original data, we use standard data augmentations like random crop after padding, random horizontal flip, and normalization.
Revised: In case of original data, we use standard data augmentations like random crop after padding, random horizontal flip, and normalization.

Original: Whereas, while using Gaussian noise, we independently sample pixels for creating a 32 × 32 × 3 image from a standard Gaussian distribution.
Revised: Whereas, while using Gaussian noise, we independently sample pixels for creating a 32 × 32 × 3 image from a standard Gaussian distribution.

Original: In both cases, the batch size is 256, and an Adam optimizer with a learning rate of 10 − 3 for tuning the parameters of the student network is used 3 .
Revised: In both cases, the batch size is 256, and an Adam optimizer with a learning rate of 10 − 3 for tuning the parameters of the student network is used 3 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: As expected, the conventional setting of using the original data as input and the teacher with the running statistics computed from the original data is the ideal case, which results in the best performance.
Revised: As expected, the conventional setting of using the original data as input and the teacher with the running statistics computed from the original data is the ideal case, which results in the best performance.

Original: Using current statistics does not hinder the learning of the student network either, that is, in the case of original data.
Revised: Using current statistics does not hinder the learning of the student network either, that is, in the case of original data.

Original: However, while using Gaussian noise with running statistics, the activation distribution gets mismatched.
Revised: However, while using Gaussian noise with running statistics, the activation distribution gets mismatched.

Original: On the other hand, using current statistics results in the mitigation of this mismatch.
Revised: On the other hand, using current statistics results in the mitigation of this mismatch.

Original: Hence, in the former, the student network is unable to learn at all, whereas in the latter, it achieves an accuracy close to the ideal case, considering that we rely on the random noise 4 .
Revised: Hence, in the former, the student network is unable to learn at all, whereas in the latter, it achieves an accuracy close to the ideal case, considering that we rely on the random noise 4 .

Original: The distribution shifts observed in this experiment are shown in Figure 1b.
Revised: The distribution shifts observed in this experiment are shown in Figure 1b.

Original: We use activations of ‘avgpool’ layer in the teacher network (ResNet-34) to obtain these plots.
Revised: We use activations of ‘avgpool’ layer in the teacher network (ResNet-34) to obtain these plots.

Original: The figure shows the distribution of average activations across all neurons with respect to different settings discussed above.
Revised: The figure shows the distribution of average activations across all neurons with respect to different settings discussed above.

Original: We observe that while using Gaussian noise, current statistics result in a distribution similar to that of using original data.
Revised: We observe that while using Gaussian noise, current statistics result in a distribution similar to that of using original data.

Original: Also, in the case of Gaussian noise with BatchNorm layers using the running statistics, most neurons have close to zero average activations, implying that they are rarely fired.
Revised: Also, in the case of Gaussian noise with BatchNorm layers using the running statistics, most neurons have close to zero average activations, implying that they are rarely fired.

Original: Their information is, therefore, never used in deciding the outcome, hence that particular information is never learned by the student network.
Revised: Their information is, therefore, never used in deciding the outcome, hence that particular information is never learned by the student network.

Original: These plots are consistent with the results in Table 1.
Revised: These plots are consistent with the results in Table 1.

Original: 4.2 Results on Other Datasets
Revised: 4.2 Results on Other Datasets

Original: We perform experiments similar to that of CIFAR10 on SVHN, CIFAR100 and Food101 to show the applicability of our method on various datasets and neural network architectures.
Revised: We perform experiments similar to that of CIFAR10 on SVHN, CIFAR100 and Food101 to show the applicability of our method on various datasets and neural network architectures.

Original: Since we did not find a considerable variance in the case of CIFAR10, here we report the results on single runs.
Revised: Since we did not find a considerable variance in the case of CIFAR10, here we report the results on single runs.

Original: The teacher-student pairs are mentioned in Table 2.
Revised: The teacher-student pairs are mentioned in Table 2.

Original: We use standard data augmentation and a similar setup of hyperparameters, i.e., batches of 256 samples, and optimization by Adam optimizer with a learning rate of 10 − 3 .
Revised: We use standard data augmentation and a similar setup of hyperparameters, i.e., batches of 256 samples, and optimization by Adam optimizer with a learning rate of 10 − 3 .

Original: Table 2 shows the obtained results.
Revised: Table 2 shows the obtained results.

Original: We see similar trends as before, except that the Gaussian noise is not at random chance even with running statistics in the case of SVHN.
Revised: We see similar trends as before, except that the Gaussian noise is not at random chance even with running statistics in the case of SVHN.

Original: We believe that this is due to relatively lower covariate shift of activation distribution compared to other more sophisticate datasets.
Revised: We believe that this is due to relatively lower covariate shift of activation distribution compared to other more sophisticate datasets.

Original: For CIFAR100 and Food101 datsets we try out teacher-student architectures which are different from the previous experiments to introduce more variability.
Revised: For CIFAR100 and Food101 datsets we try out teacher-student architectures which are different from the previous experiments to introduce more variability.

Original: Table 2 shows that a large amount of gap between real data-based distillation and data-free distillation is reduced by Gaussian noise and as we see in the next section, the remaining gap can be filled by other mechanisms.
Revised: Table 2 shows that a large amount of gap between real data-based distillation and data-free distillation is reduced by Gaussian noise and as we see in the next section, the remaining gap can be filled by other mechanisms.

Original: 4.3 Limited Data
Revised: 4.3 Limited Data

Original: In this experiment, we try to find out the amount of additional data needed for extra efforts after
Revised: In this experiment, we try to find out the amount of additional data needed for extra efforts after

Original: Gaussian noise-based KD.
Revised: Gaussian noise-based KD.

Original: It would be interesting to use the synthetic data for this, however, we rely on samples from the training dataset as proxy due to a large variation in the nature of synthetic data generation approaches.
Revised: It would be interesting to use the synthetic data for this, however, we rely on samples from the training dataset as proxy due to a large variation in the nature of synthetic data generation approaches.

Original: We first pretrain our student model using Gaussian distillation and then finetune it using original data.
Revised: We first pretrain our student model using Gaussian distillation and then finetune it using original data.

Original: First, we show how the performance varies with respect to the amount of data used for finetuning, then we briefly show the same for different configurations.
Revised: First, we show how the performance varies with respect to the amount of data used for finetuning, then we briefly show the same for different configurations.

Original: We compare our case of pretraining using Gaussian noise against traditional KD using the same fractional amount of data (baseline).
Revised: We compare our case of pretraining using Gaussian noise against traditional KD using the same fractional amount of data (baseline).

Original: For finetuning, a subset of training data is sampled randomly and a reduced learning rate of 10 −is used.
Revised: For finetuning, a subset of training data is sampled randomly and a reduced learning rate of 10 −is used.

Original: Figure 3 (Left) compares the accuracy of the ResNet-18 (student) on CIFAR10 when it is finetuned using the given fraction of training data to its counterpart that is trained from scratch.
Revised: Figure 3 (Left) compares the accuracy of the ResNet-18 (student) on CIFAR10 when it is finetuned using the given fraction of training data to its counterpart that is trained from scratch.

Original: Further, Table 3 shows the student accuracy for other architectures and datasets with 1% of training data.
Revised: Further, Table 3 shows the student accuracy for other architectures and datasets with 1% of training data.

Original: We observe a considerable improvement in the student network after finetuning, specifically for
Revised: We observe a considerable improvement in the student network after finetuning, specifically for

Original: MobileNetV2 on CIFAR10 (about 4%).
Revised: MobileNetV2 on CIFAR10 (about 4%).

Original: In addition, pretraining using Gaussian noise outperforms the baseline by a large margin.
Revised: In addition, pretraining using Gaussian noise outperforms the baseline by a large margin.

Original: [Table]
Revised: [Table]

Original: 4.4 BatchNorm State in Student
Revised: 4.4 BatchNorm State in Student

Original: Here, we explore how the batch size or inference, in general, affects the student network trained using our Gaussian noise-based approach.
Revised: Here, we explore how the batch size or inference, in general, affects the student network trained using our Gaussian noise-based approach.

Original: We use the ResNet-18 student network distilled using our approach from ResNet-34 teacher trained on CIFAR10.
Revised: We use the ResNet-18 student network distilled using our approach from ResNet-34 teacher trained on CIFAR10.

Original: We use a sufficient amount of evaluation data to adjust the running statistics of the student’s BatchNorm layers, which is 20 batches of sizeeach 5 .
Revised: We use a sufficient amount of evaluation data to adjust the running statistics of the student’s BatchNorm layers, which is 20 batches of sizeeach 5 .

Original: After the student is trained using Gaussian noise, its running statistics are not adjusted for test data distribution.
Revised: After the student is trained using Gaussian noise, its running statistics are not adjusted for test data distribution.

Original: Therefore, the accuracy using running statistics is almost at random chance (blue), shown in Figure 3 (Middle).
Revised: Therefore, the accuracy using running statistics is almost at random chance (blue), shown in Figure 3 (Middle).

Original: When we use the current statistics and vary the batch size, we see that the accuracy increases with an increase in batch size due to better estimates of µ B and σ B (black).
Revised: When we use the current statistics and vary the batch size, we see that the accuracy increases with an increase in batch size due to better estimates of µ B and σ B (black).

Original: Further, we adjust the running statistics using some amount of original data, while keeping the rest of the parameters fixed, considering the case in which the student model needs to infer on a single sample or a batch of non-i.i.d.
Revised: Further, we adjust the running statistics using some amount of original data, while keeping the rest of the parameters fixed, considering the case in which the student model needs to infer on a single sample or a batch of non-i.i.d.

Original: samples.
Revised: samples.

Original: We observe that the adjusted running statistics yield a performance similar to that of large batch sizes (red).
Revised: We observe that the adjusted running statistics yield a performance similar to that of large batch sizes (red).

Original:  Discussion 5.1 Limitations
Revised:  Discussion 5.1 Limitations

Original: Absence of BatchNorm: If there are no BatchNorm layers present in the teacher-student architecture, then the proposed method will not work.
Revised: Absence of BatchNorm: If there are no BatchNorm layers present in the teacher-student architecture, then the proposed method will not work.

Original: In that case, one would need to learn the input data distribution to obtain activation distribution similar to that of original data.
Revised: In that case, one would need to learn the input data distribution to obtain activation distribution similar to that of original data.

Original: Training: As opposed to traditional KD, the teacher in the proposed method is required to use the current statistics while distilling the knowledge to student.
Revised: Training: As opposed to traditional KD, the teacher in the proposed method is required to use the current statistics while distilling the knowledge to student.

Original: In order to correctly calculate these statistics and output correct soft labels, a large enough batch size is required during training period.
Revised: In order to correctly calculate these statistics and output correct soft labels, a large enough batch size is required during training period.

Original: Results of study regarding batch size are included in Appendix A.4.
Revised: Results of study regarding batch size are included in Appendix A.4.

Original: Inference: As explained in Section 3.3, the student model might be limited by how it can be used during inference.
Revised: Inference: As explained in Section 3.3, the student model might be limited by how it can be used during inference.

Original: The student network trained using our proposed approach should either be used with a sufficiently large batch of test samples or its running statistics should be adjusted beforehand using a small amount of test data.
Revised: The student network trained using our proposed approach should either be used with a sufficiently large batch of test samples or its running statistics should be adjusted beforehand using a small amount of test data.

Original: Without doing so, the student model trained using Gaussian noise does not get adapted to real data distribution, as shown in Figure 3 (Right).
Revised: Without doing so, the student model trained using Gaussian noise does not get adapted to real data distribution, as shown in Figure 3 (Right).

Original: The plot shows average
Revised: The plot shows average

Original: 5.2 Rethinking Knowledge Transfer
Revised: 5.2 Rethinking Knowledge Transfer

Original: Recently, [24] explored different types of structured noise (which proves to be far better than Gaussian noise) as a way to learn representations using a specific contrastive loss and questioned the necessity of huge data-driven vision systems.
Revised: Recently, [24] explored different types of structured noise (which proves to be far better than Gaussian noise) as a way to learn representations using a specific contrastive loss and questioned the necessity of huge data-driven vision systems.

Original: [18] also observed that low-level features are sufficient for performing KD.
Revised: [18] also observed that low-level features are sufficient for performing KD.

Original: This paper shows that using just the Gaussian noise (not even low-level features) we can successfully transfer knowledge from one neural network to the other to a large extent.
Revised: This paper shows that using just the Gaussian noise (not even low-level features) we can successfully transfer knowledge from one neural network to the other to a large extent.

Original: We have not yet explored the possibility of using structured noise, which could prove to be more effective 6 .
Revised: We have not yet explored the possibility of using structured noise, which could prove to be more effective 6 .

Original: This challenges the generally accepted notion of requiring synthetic data with representations similar to original data and provides motivation to explore methods similar to ours for tasks such as domain adaptation, transfer learning, etc.
Revised: This challenges the generally accepted notion of requiring synthetic data with representations similar to original data and provides motivation to explore methods similar to ours for tasks such as domain adaptation, transfer learning, etc.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we explore KD solely based on Gaussian noise.
Revised: In this paper, we explore KD solely based on Gaussian noise.

Original: We do this by mitigating the covariate shift in the hidden layers of the teacher network, which is caused due to the change in the input data distribution.
Revised: We do this by mitigating the covariate shift in the hidden layers of the teacher network, which is caused due to the change in the input data distribution.

Original: Further, we show how similar phenomena can occur in the student network, and how we can use a similar approach there as well.
Revised: Further, we show how similar phenomena can occur in the student network, and how we can use a similar approach there as well.

Original: The experiments show that student network trained using our approach can achieve a greater performance, in contrast to what prior works have shown for Gaussian noise setting, and more so in the presence of limited data.
Revised: The experiments show that student network trained using our approach can achieve a greater performance, in contrast to what prior works have shown for Gaussian noise setting, and more so in the presence of limited data.

Original: Future work could be to try
Revised: Future work could be to try

Original: Gaussian noise-based distillation with lower input dimensions such that it feed-forwards through both teacher and student network since the Gaussian noise does not contain structural information anyway.
Revised: Gaussian noise-based distillation with lower input dimensions such that it feed-forwards through both teacher and student network since the Gaussian noise does not contain structural information anyway.

Original: Further future work could involve comparing how our approach fares to using unrelated data for KD, trying our approach on structured noise, and extending our findings to other domains such as domain adaptation.A Appendix A.1 Toy problem
Revised: Further future work could involve comparing how our approach fares to using unrelated data for KD, trying our approach on structured noise, and extending our findings to other domains such as domain adaptation.

Original: In order to further understand how running statistics and current statistics with Gaussian noise as input compare against real data, we consider the following toy problem of binary classification of 2D points distributed as concentric circles (real data).

Original: We take a simple Multilayer perceptron (MLP) with 2 hidden layers and 2 BatchNorm layers.

Original: This is done to restrict the input and embedding space (output from penultimate layer) to 2 dimensions each which can be directly visualized, rather than looking at their distributions.

Original: Figure 4 shows the real data on which the MLP is trained, the input

Original: Gaussian noise samples, and the embeddings in each of the three cases, respectively from left to right.

Original: We see that, although the distribution of Gaussian noise is considerably different from real data, the embeddings are close to the embeddings of real data while using current statistics as compared to using running statistics.

Original: A.2 Handling different number of BatchNorm layers

Original: Here we consider CIFAR10 dataset on which teacher is trained and the teacher-student pair of

Original: ResNet34-ResNet18.

Original: While performing distillation using Gaussian noise, we randomly choose a given percentage of BatchNorm layers ( P ) from the teacher network and restrict those layers to use running statistics, whereas the remaining BatchNorm layers use current statistics.

Original: We vary the percentage of such layers and report the result in Table 4.

Original: We observe that as the percentage of number of BatchNorm layers using running statistics increases, the student accuracy decreases.

Original: Note that, the number of BatchNorm layers in the teacher network remains same.

Original: [Table]

Original: A.3 Adjusting the student

Original: From Figure 5, we observe that the student activations for original data with running statistics (which are adapted to Gaussian noise) follows a very different distribution compared to other two, and does not output meaningful information.

Original: Neurons in that case either explode or rarely activate.

Original: As described in Section 4.4 of the paper, the remedy is to use current statistics while inference or adjust

Original: [Figure]

Original: [Figure]

Original: the running statistics to make them adapt to real data distribution using a small subset of evaluation data.

Original: Figure 6a further shows the number of batches of evaluation data used to adjust the running statistics vs the student’s accuracy when those adjusted running statistics are used.

Original: Note that the batch size is kept constant at 16.

Original: We found that, similar to how the batch size needs to be just large enough while using current statistics, here as well, the number of batches or data in general needs to be sufficient enough to get a good performance out of the student model.

Original: A.4 Significance of Batch size during distillation

Original: As stated in Section 5.1 of the paper, as opposed to the traditional knowledge distillation, the teacher in our approach needs to rely on the current statistics of the input.

Original: In order to get the correct estimates in the BatchNorm layer, the batch size during distillation should be large enough.

Original: A comparison is shown in Figure 6b for the case of CIFAR10, where teacher is ResNet-34 and student is ResNet-18.

Original: Here, the distillation is performed with the given batch size for both teacher and student models, and the same batch size is used while inference to calculate current statistics.

Original: A.5 Experiments with Dead Leaves

Original: We conduct additional experiments to understand the potential of the proposed approach.

Original: In particular we notice the missing spatial consistency among pixels in the samples randomly drawn from a

Original: Gaussian distribution.

Original: We, therefore, consider random samples obtained using the Dead leaves

Original: [Table]

Original: Shapes

Original: [Table]

Original: model.

Original: Figure 7 shows some of the Dead leave samples.

Original: We repeat the distillation experiments on CIFAR10 dataset with Dead leaves samples.

Original: We use the same hyperparameter values as the Guassian noise distillation experiments.

Original: In addition to ResNet-34, ResNet-

Original: [Table]

Original: and

Original: MobileNetV2, we consider WRN-28-10 and WRN-16-8 as student networks.

Original: Table 5 shows the obtained results.

Original: We observe a considerable improvement in the performance of the proposed approach with Dead leave samples.

Original: In particular, for MobileNetV2, test accuracy improves more than 4%.

Original: Furthermore, for WRN-28-10, the proposed noise-based KD (90.75%) is very close to the real data-based KD (92.41%).

Original: Moreover, we consider batch size varying from 32 to 512 and distill ResNet-34, pretrained on CIFAR10, into ResNet-18 using the proposed approach with Dead leaves samples.

Original: We observe test accuracy of the student distilled with batch size of 32, 64, 128, 256, and 512 as 88.09%, 88.21%, 89.35%, 89.4%, and 89.19%, respectively.

Original: This shows that the further improvements can be obtained by adjusting the nature of random samples.

Original: For further evaluation, we consider DenseNet169-DenseNet121 as teacher-student combination for

Original: CIFAR10 dataset

Original: [Table]

Original: and perform distillation using the proposed approach with Dead leaves samples.

Original: Table 6 shows the obtained results where we observe similar improvements as with the previous experiments.

Original: [Table]

Original: A.6 Activation distributions

Original: This section contains the distributions of activations of the teacher networks with respect to the model architectures and datasets that they were trained on.

Original: A.6.1 CIFAR

Original: [Figure]

Original: In Figure 8, we observe a similar trend in the activation distribution of all layers.

Original: The red curve denotes the activation distribution of original data with running statistics, which is the ideal case.

Original: However, the trivial use of Gaussian noise with running statistics (blue) results in a significantly different activation distribution.

Original: Our proposed approach makes sure that between Gaussian noise with running statistics and Gaussian noise with current statistics (black), the activation distribution of the latter is comparatively similar to the ideal case, thus reducing the shift.

Original: We observe a similar trend throughout all the following plots for SVHN (Figure 9), Food101(Figure 10), and CIFAR (Figure 11).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A.7 Experiment settings

Original: In all of the experiments, we have used standard datasets and model architectures.

Original: For Food101, we use PyTorch built-in architectures.

Original: For SVHN and CIFAR10, we use a custom implementation 8 of

Original: ResNet architectures, which is tailored towards low-resolution images by making the initial kernel shape smaller.

Original: For CIFAR100, we use an open-source implementation 9 of Wide-ResNets.

Original: In all cases we have used Adam optimizer with a standard learning rate of 0.001, except for the training of CIFAR100 teacher, where we use SGD with learning rate of 0.1, momentum of 0.9, and weight decay of 0.0005.

Original: In all experiments, learning rate scheduler strategy is a standard one to reduce learning rate when a plateau hits.

Original: We train teacher and student models in each experiment for 100 epochs and 200 epochs respectively, and wherever not specified we use a batch size of 256.

Original: All of the datasets used have predefined train-test splits with labels available for both, hence the same are used.

Original: Note that the train split is used only in case of training the teacher, and test split is used as evaluation data.

Original: State of BatchNorm.

Original: Handling BatchNorm layers to calculate either running statistics or current statistics is straightforward that can be done by putting the model in one of the two PyTorch modes - train and eval .

Original: Traditionally, during distillation using real data, we put the teacher in eval mode where BatchNorm layers use running statistics.

Original: For our approach to distill using Gaussian noise, we need the teacher to use current statistics, which is done by putting the teacher in train mode.

Original: Note that, even though the teacher is in train mode, it only allows calculation of current statistics for the

Original: BatchNorm layers.

Original: No update happens to the teacher’s weights as there is no optimizer bound to the teacher’s weights.

Original: All of the experiments are performed on a system with 16-core Intel x86 CPU with 128 GB RAM and NVIDIA RTX 3090 GPU.