Original: Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation
Revised: Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Distillation in neural networks using only the samples randomly drawn from a
Revised: Distillation in neural networks using only the samples randomly drawn from a

Original: Gaussian distribution is possibly the most straightforward solution one can think offor the complex problem of knowledge transfer from one network (teacher) to theother (student).
Revised: Gaussian distribution is possibly the most straightforward solution one can think offor the complex problem of knowledge transfer from one network (teacher) to theother (student).

Original: If successfully done, it can eliminate the requirement of teacher’straining data for knowledge distillation and avoid often arising privacy concerns insensitive applications such as healthcare.
Revised: If successfully done, it can eliminate the requirement of teacher’straining data for knowledge distillation and avoid often arising privacy concerns insensitive applications such as healthcare.

Original: There have been some recent attemptsat Gaussian noise-based data-free knowledge distillation, however, none of themoffer a consistent or reliable solution.
Revised: There have been some recent attemptsat Gaussian noise-based data-free knowledge distillation, however, none of themoffer a consistent or reliable solution.

Original: We identify the shift in the distribution ofhidden layer activation as the key limiting factor, which occurs when Gaussiannoise is fed to the teacher network instead of the accustomed training data.
Revised: We identify the shift in the distribution ofhidden layer activation as the key limiting factor, which occurs when Gaussiannoise is fed to the teacher network instead of the accustomed training data.

Original: Wepropose a simple solution to mitigate this shift and show that for vision tasks, suchas classification, it is possible to achieve a performance close to the teacher by justusing the samples randomly drawn from a Gaussian distribution.
Revised: Wepropose a simple solution to mitigate this shift and show that for vision tasks, suchas classification, it is possible to achieve a performance close to the teacher by justusing the samples randomly drawn from a Gaussian distribution.

Original: We validate ourapproach on CIFAR10, CIFAR100, SVHN, and Food101 datasets.
Revised: We validate ourapproach on CIFAR10, CIFAR100, SVHN, and Food101 datasets.

Original: We further showthat in situations of sparsely available original data for distillation, the proposed
Revised: We further showthat in situations of sparsely available original data for distillation, the proposed

Original: Gaussian noise-based knowledge distillation method can outperform the distillationusing the available data with a large margin.
Revised: Gaussian noise-based knowledge distillation method can outperform the distillationusing the available data with a large margin.

Original: Our work lays the foundation forfurther research in the direction of noise-engineered knowledge distillation usingrandom samples.
Revised: Our work lays the foundation forfurther research in the direction of noise-engineered knowledge distillation usingrandom samples.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks are an excellent choice for various real-world computer vision tasks.
Revised: Deep neural networks are an excellent choice for various real-world computer vision tasks.

Original: However,their high computational and space complexity becomes a bottleneck when it comes to deploymenton resource constrained devices.
Revised: However,their high computational and space complexity becomes a bottleneck when it comes to deploymenton resource-constrained devices.

Original: To address this issue, the idea of Knowledge Distillation (KD),proposed by [1], has recently gained significant attention.
Revised: To address this issue, the idea of Knowledge Distillation (KD),proposed by [1], has recently gained significant attention.

Original: It enables the transfer of knowledge from alarge neural network to a comparatively smaller one.
Revised: It enables the transfer of knowledge from alarge neural network to a comparatively smaller one.

Original: The lightweight small neural network, oftenreferred as student, gains the information learned by the over-parameterized large model, referred asteacher and delivers a similar or sometimes even better performance [2].
Revised: The lightweight small neural network, oftenreferred as student, gains the information learned by the over-parameterized large model, referred asteacher and delivers a similar or sometimes even better performance [2].

Original: Traditionally, in KD original data is passed through the accurate teacher model to collect soft labels and subsequently these soft labels are used to supervise the student [3].
Revised: Traditionally, in KD, original data is passed through the accurate teacher model to collect softlabels, and subsequently, these soft labels are used to supervise the student [3].

Original: However, in certainapplications such as medical, it often becomes impossible to release the original data due to security concerns, sensitive information, or storage restrictions.
Revised: However, in certainapplications such as medical, it often becomes impossible to release the original data due to securityconcerns, sensitive information, or storage restrictions.

Original: In contrast a trained model (teacher) usually does not hold a specific record or specific piece of information thus no certainty or validity of theinformation extracted through a neural network can be guaranteed thereby making them safer for public release.
Revised: In contrast, a trained model (teacher) usuallydoes not hold a specific record or specific piece of information thus, no certainty or validity of theinformation extracted through a neural network can be guaranteed, thereby making them safer forpublic release.

Original: However these models certainly contain information in the parameterized form for performing the downstream tasks, such as classification or object detection
Revised: However, these models certainly contain information in the parameterized form forperforming the downstream tasks, such as classification or object detection.

Original: [Figure]
Revised: [Figure]

Original: Due to these reasons, recently there has been an interest in data-free distillation - that is, distillingknowledge from one neural network to the other without using any data.
Revised: Due to these reasons, recently there has been an interest in data-free distillation - that is, distillingknowledge from one neural network to the other without using any data.

Original: Majority of methods in thisdirection aim to create synthetic images either by optimizing the randomly generated samples [4, 5, 6]or by training a generative network [7, 8], which often is computationally expensive.
Revised: Majority of methods in thisdirection aim to create synthetic images either by optimizing the randomly generated samples [4, 5,6, 7, 8] or by training a generative network [9, 10], which often is computationally expensive.

Original: This motivatesthe research community to ask the question Can distillation be done by using the samples randomlydrawn from a known distribution such as standard normal distribution?
Revised: Thismotivates the research community to ask the question Can distillation be done by using the samplesrandomly drawn from a known distribution such as standard normal distribution?

Original: To best of our knowledge,we for the first time provide a satisfactory answer to the question and show that yes, it is possibleto do data-free KD by only using images that are randomly sampled from a Gaussian distribution.
Revised: To best of ourknowledge, we for the first time provide a satisfactory answer to the question and show that yes, itis possible to do data-free KD by only using images that are randomly sampled from a Gaussiandistribution.

Original: Our experiments show that a considerably better performance is possible using only the Gaussiannoise as the input, in contrast to what prior works ([7], [8]) have shown for this particular setting.
Revised: Our experiments show that a considerably better performance is possible using only the Gaussian noise as the input, in contrast to what prior works ([9], [10]) have shown for this particularsetting.

Original: We identify that the key limitation, which the prior works failed to identify is the covariate shift inthe distribution of hidden layer activations of the teacher network caused by feeding Gaussian noiseinstead of the expected original data.
Revised: We identify that the key limitation, which the prior works failed to identify is the covariateshift in the distribution of hidden layer activations of the teacher network caused by feeding Gaussiannoise instead of the expected original data.

Original: As shown in Figure 1a, the activation distribution when input is Gaussian noise (blue) is shifted, thus no longer aligns with the activation distribution when input isoriginal data (red).
Revised: As shown in Figure 1a, the activation distribution wheninput is Gaussian noise (blue) is shifted, thus no longer aligns with the activation distribution wheninput is original data (red).

Original: We propose a simple yet surprisingly effective solution to reduce this shift byusing current batch statistics instead of using statistics of the original data (running statistics) in theteacher’s batch normalization (BatchNorm) layers and thus distill the knowledge from teacher intostudent network using just the Gaussian noise.
Revised: We propose a simple yet surprisingly effective solution to reduce thisshift by using current batch statistics instead of using statistics of the original data (running statistics)in the teacher’s batch normalization (BatchNorm) layers and thus distill the knowledge from teacherinto student network using just the Gaussian noise.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • We explain how covariate shift in the teacher network interferes with Gaussian noise-based KD and propose a simple solution to eliminate it, thus enable knowledge transfer using
Revised: • We explain how covariate shift in the teacher network interferes with Gaussian noise-basedKD and propose a simple solution to eliminate it, thus enable knowledge transfer using

Original: Gaussian noise.
Revised: Gaussian noise.

Original: • We show how this phenomena of covariate shift occurs in the student network as well and
Revised: • We show how this phenomena of covariate shift occurs in the student network as well and

Original: mitigate it in a similar way.
Revised: mitigate it in a similar way.

Original: • We obtain student performance close to the teacher without using any original or synthetic
Revised: • We obtain student performance close to the teacher without using any original or synthetic

Original: data.
Revised: data.

Original: We further show that in the presence of limited data, the proposed approach
Revised: We further show that in the presence of limited data, the proposed approach

Original: outperforms the traditional distillation method with a large margin.
Revised: outperforms the traditional distillation method with a large margin.

Original: Our
Revised: Our

Original: work counters the commonly accepted notion of requiring realistic training data for
Revised: work counters the commonly accepted notion of requiring realistic training data for

Original: data-free KD.
Revised: data-free KD.

Original: Thus, we lay
Revised: Thus, we lay

Original: the foundation for noise-based data-free KD
Revised: the foundation for noise-based data-free KD

Original:  Related work
Revised:  Related work

Original: Knowledge Distillation: KD is the transfer of knowledge from one neural network to the other.
Revised: Knowledge Distillation: KD is the transfer of knowledge from one neural network to the other.

Original: First successful implementation of knowledge transfer was shown by [9] for the purpose of modelcompression.
Revised: First successful implementation of knowledge transfer was shown by [11] for the purpose of modelcompression.

Original: Later, the term knowledge distillation was popularized by [1], where the authorsexplored so called "dark knowledge" of a teacher model which is transferred to a student modelwith the help of soft labels.
Revised: Later, the term knowledge distillation was popularized by [1], where the authorsexplored so called "dark knowledge" of a teacher model which is transferred to a student modelwith the help of soft labels.

Original: Since then, the domain of KD has witnessed many interested ideasrelated to improved distillation strategies such as teacher assistant [10] and self-distillation [11].
Revised: Since then, the domain of KD has witnessed many interested ideasrelated to improved distillation strategies such as teacher assistant [12] and self-distillation [13].

Original: KD approaches for performing various downstream tasks such as object detection [12], semanticsegmentation [13], and also the applications in different modalities, including speech [14] and graph[15] have been explored.
Revised: KD approaches for performing various downstream tasks such as object detection [14], semanticsegmentation [15], and also the applications in different modalities, including speech [16] and graph[17] have been explored.

Original: A detailed survey on KD is recently reported by [2].
Revised: A detailed survey on KD is recently reported by [2].

Original: Data-free Knowledge Distillation: The problem of data-free KD, i.e., the constraint that the originaldataset, which was used to train and supervise the teacher, is unavailable for distillation makes thetasks naturally challenging.
Revised: Data-free Knowledge Distillation: The problem of data-free KD, i.e., the constraint that the originaldataset, which was used to train and supervise the teacher, is unavailable for distillation makes thetasks naturally challenging.

Original: This often is encountered in privacy-sensitive settings, where the originaldataset cannot be shared.
Revised: This often is encountered in privacy-sensitive settings, where the originaldataset cannot be shared.

Original: There are various prior works with a similar context in mind.
Revised: There are various prior works with a similar context in mind.

Original: Most ofthese methods first construct a set of synthetic images that resembles original data, then distill theknowledge of the teacher network into the student network using that synthetic data [4, 5, 6, 7, 8].
Revised: Most ofthese methods first construct a set of synthetic images that resembles original data, then distill theknowledge of the teacher network into the student network using that synthetic data [4, 5, 6, 8, 9, 10].

Original: Some approaches synthesize images one-by-one or batch-by-batch, whereas some train a generatormodel to gain access to limitless synthetic data.
Revised: Some approaches synthesize images one-by-one or batch-by-batch, whereas some train a generatormodel to gain access to limitless synthetic data.

Original: For example, [4] stored activation records of theteacher model when original data is feed-forwarded through it, and then later used it to create syntheticimages by optimizing random noise using gradient descent.
Revised: For example, [4] stored activation records of theteacher model when original data is feed-forwarded through it, and then later used it to create syntheticimages by optimizing random noise using gradient descent.

Original: [5] introduced DeepInversion intendingto create realistic images, which makes use of feature map and BatchNorm statistics to optimizerandom images into high-fidelity images.
Revised: [5] introduced DeepInversion intendingto create realistic images, which makes use of feature map and BatchNorm statistics to optimizerandom images into high-fidelity images.

Original: [16] takes a different approach to simultaneously generateimages, as well as train the student.
Revised: [18] takes a different approach to simultaneously generateimages, as well as train the student.

Original: They introduced Zero-shot knowledge transfer, which utilizedadversarial learning between the generator model and the student model.
Revised: They introduced Zero-shot knowledge transfer, which utilizedadversarial learning between the generator model and the student model.

Original: Their intuition was to searchfor images on which the student poorly matches the teacher and then train the student with those.
Revised: Their intuition was to searchfor images on which the student poorly matches the teacher and then train the student with those.

Original: Gaussian Noise-based Knowledge Distillation: KD using Gaussian noise has been presented as abaseline numerous times in prior works, which resulted in a student accuracyjust slightly better thanrandom chance.
Revised: Gaussian Noise-based Knowledge Distillation: KD using Gaussian noise has been presented asa baseline numerous times in prior works, which resulted in a student accuracy slightly better thanrandom chance.

Original: Although a decent accuracy was reported on MNIST (88% [7], 75% [17]), probablydue to smaller networks and less complex dataset, on more challenging datasets like CIFAR-10, apoor performance of (14% [5], 15% [7], 11% [8], 10% [17]) was observed.
Revised: Although a decent accuracy was reported on MNIST (88% [9], 75% [19]), probablydue to smaller networks and less complex dataset, on more challenging datasets like CIFAR10, apoor performance of (14% [5], 15% [9], 11% [10], 10% [19]) was observed.

Original: Different from these, [6]presented an interesting, although highly constrained, scenario where the student had an identicalarchitecture to the teacher and student’s weights were initialized as quantized values of teacher’sweights.
Revised: Different from these,[6] presented an interesting, although highly constrained, scenario where the student had an identicalarchitecture to the teacher and student’s weights were initialized as quantized values of teacher’sweights.

Original: Further when the BatchNorm layer parameters from teacher are copied to student and keptfrozen, Gaussian noise based distillation resulted in good performance.
Revised: Further when the BatchNorm layer parameters from teacher are copied to student and keptfrozen, Gaussian noise based distillation resulted in good performance.

Original: Moreover, instead of trainingthe student network end-to-end as in traditional KD, [6] partitioned the student and teacher networksacross BatchNorm layers resulting in multiple teacher-student pairs of blocks and distilled each pairof blocks separately.
Revised: Moreover, instead of trainingthe student network end-to-end as in traditional KD, [6] partitioned the student and teacher networksacross BatchNorm layers resulting in multiple teacher-student pairs of blocks and distilled each pairof blocks separately.

Original: Although this approach works, it cannot be used for a pair of architectures nothaving similar block structures and is unnecessarily complex.
Revised: Although this approach works, it cannot be used for a pair of architectures nothaving similar block structures and is unnecessarily complex.

Original: In contrast to all above, our aim is to design a simple approach for Gaussian noise-based data freedistillation.
Revised: In contrast to all above, our aim is to design a simple approach for Gaussian noise-based data freedistillation.

Original: This would not only relax the constraints of approaches such as [6] but also save theefforts required in creating synthetic dataset.
Revised: This would not only relax the constraints of approaches such as [6] but also save theefforts required in creating synthetic dataset.

Original:  Methodology
Revised:  Methodology

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Knowledge Distillation: KD is achieved by training the student network using the output of theteacher network as soft labels instead of using the ground truth labels [3].
Revised: Knowledge Distillation: KD is achieved by training the student network using the output of theteacher network as soft labels instead of using the ground truth labels [3].

Original: This results in matching ofthe output probability distributions of both the networks for a given input data.
Revised: This results in matching ofthe output probability distributions of both the networks for a given input data.

Original: Formally, for a k-classclassification, given a dataset D that consists of data samples x , a teacher network T trained on D ,and a student network S with randomly initialized weights, the following cost function is optimized:
Revised: Formally, for a k-classclassification, given a dataset D that consists of data samples x , a teacher network T trained on D ,and a student network S with randomly initialized weights, the following cost function is optimized:

Original: [Equation]
Revised: [Equation]

Original: where L XE ( ., . ) is the standard cross-entropy term.
Revised: where L XE ( ., . ) is the standard cross-entropy term.

Original: Batch Normalization: BatchNorm was introduced by [18] that made the training of deep neuralnetworks more stable and allowed to use larger learning rates, hence faster convergence.
Revised: Batch Normalization: BatchNorm was introduced by [20] that made the training of deep neuralnetworks more stable and allowed to use larger learning rates, hence faster convergence.

Original: Further,it makes neural networks more robust to initialization schemes and changes in the learning rate.
Revised: Further,it makes neural networks more robust to initialization schemes and changes in the learning rate.

Original: BatchNorm first normalizes the output of the hidden layer (input to BatchNorm layer) using the first(mean) and the second (variance) statistical moments of the current batch.
Revised: BatchNorm first normalizes the output of the hidden layer (input to BatchNorm layer) using the first(mean) and the second (variance) statistical moments of the current batch.

Original: Then, these normalizedactivations are shifted and scaled according to the two trainable parameters γ and β , respectively.
Revised: Then, these normalizedoutputs are shifted and scaled according to the two trainable parameters γ and β , respectively.

Original: These parameters are tuned as any other neural network parameters, and they help in choosingthe optimum distribution as the input for the next layer, thus regaining expressiveness lost in thenormalization step.
Revised: Theseparameters are tuned as any other neural network parameters, and they help in choosing the optimumdistribution as the input for the next layer, thus regaining expressiveness lost in the normalizationstep.

Original: Formally, let h be the activations of a neuron in the previous layer (input to the BatchNorm layer), mbe the batch-size, and ϵ be a small constant used for numerical stability.
Revised: Formally, let h be the activations of a neuron in the previous layer (input to the BatchNorm layer), mbe the batch-size, and ϵ be a small constant used for numerical stability.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where µ B = 1 m (cid:80) i h i and σ 2 B = 1 m (cid:80)
Revised: where µ B = 1 m (cid:80) i h i and σ 2 B = 1 m (cid:80)

Original: i
Revised: i

Original: ( h i − µ B ) 2 .
Revised: ( h i − µ B ) 2 .

Original: The h i and (cid:98) h
Revised: The h i and (cid:98) h

Original: i show normalized and subsequentlyshifted values of h i .
Revised: i show normalized and subsequentlyshifted values of h i .

Original: Above mentioned activation normalization using min-batch or current statisticsresults in improved training.
Revised: Above mentioned activation normalization using mini-batch or current statisticsresults in improved training.

Original: During evaluation or inference, the current statistics µ B and σ B arereplaced with running or population statistics, µ r = E B [ µ B ] and σ r = mm − 1 E B [ σ 2 B ] , where E B [ . ]denotes expectation over training mini-batches.
Revised: During evaluation or inference, the current statistics µ B and σ B arereplaced with running or population statistics, µ r = E B [ µ B ] and σ r = mm − 1 E B [ σ 2 B ] , where E B [ . ]denotes expectation over training mini-batches.

Original: Accordingly, the normalization steps are modified asfollows:
Revised: Accordingly, the normalization steps are modified asfollows:

Original: [Equation]
Revised: [Equation]

Original: where µ r = αµ r
Revised: where µ r = αµ r

Original: +(1 − α ) µ B and σ r = ασ r +(1 − α ) σ B .
Revised: + (1 − α ) µ B and σ r = ασ r + (1 − α ) σ B .

Original: Here α is a smoothing factor to prioritizestatistics of the current batch.
Revised: Here α is a smoothing factor, which isto prioritize statistics of the current batch.

Original: 3.2 Distillation using Gaussian Noise
Revised: 3.2 Distillation using Gaussian Noise

Original: We consider the most straightforward approach for KD in the absence of original data, just usingrandom images sampled from a Gaussian distribution as an input to teacher and student network.
Revised: We consider the most straightforward approach for KD in the absence of original data, just usingrandom images sampled from a Gaussian distribution as an input to teacher and student network.

Original: This precondition to use Gaussian noise for KD might seem to make the task of distillation almostimpossible since the input Gaussian distribution is notably different from the original data distribution,and does not contain any structural properties or representation as the original data.
Revised: This precondition to use Gaussian noise for KD might seem to make the task of distillation almostimpossible since the input Gaussian distribution is notably different from the original data distribution,and does not contain any structural properties or representation as the original data.

Original: We hypothesizethat, for successful KD, across two input data: first using which the teacher is trained, and secondusing which student is trained, having similar output distributions throughout the teacher network’shidden layers is more important than having the same structural or representation properties at theinput. Thus, leading to the extraction of meaningful information from the teacher network.
Revised: We hypothesizethat, for successful KD, across two input data: first using which the teacher is trained, and secondusing which student is trained, having similar output distributions throughout the teacher network’shidden layers is more important than having the same structural or representation properties atthe input; thus, leading to the extraction of meaningful information from the teacher network.

Original: Tounderstand this better let us consider a trivial case where samples drawn from a standard Gaussiandistribution ( N (0 , 1) ) are passed through a BatchNorm layer optimized for some real data D .
Revised: Tounderstand this better let us consider a trivial case where samples drawn from a standard Gaussiandistribution ( N (0 , 1) ) are passed through a BatchNorm layer optimized for some real data D .

Original: Asduring distillation only student network is trained, BatchNorm performs normalization of inputaccording to (4).
Revised: Asduring distillation only student network is trained, BatchNorm performs normalization of inputaccording to (4).

Original: Consider the following statistics of the normalized signal.
Revised: Consider the following statistics of the normalized signal.

Original: [Equation]
Revised: [Equation]

Original: Equation (5) and (6) clearly show the deviation or covariate shift in the output of BatchNorm when
Revised: Equation (5) and (6) clearly show the deviation or covariate shift in the output of BatchNorm when

Original: Gaussian noise is passed instead of the original data.
Revised: Gaussian noise is passed instead of the original data.

Original: This suggest that output of BatchNorm shouldbe shifted by

Original: [Equation] √ [Equation] r + ϵ and scaled by σ 2 r + ϵ before feeding it to the subsequent layers.
Revised: This suggest that output of BatchNorm shouldbe shifted by µ r √ σ 2 r + ϵ and scaled by σ 2 r + ϵ before feeding it to the subsequent layers.

Original: However,this is notso trivial for intermediate hidden layers of the teacher network as the activation statisticsare unknown.
Revised: However,this is not trivial for intermediate hidden layers of the teacher network as the activation statisticsare unknown.

Original: Alternatively, as explained below, we may allow BatchNorm to simply use current
Revised: Alternatively, as explained below, we may allow BatchNorm to simply use currentstatistics of running statistics the corrections required but the factors of runningstatistics.

Original: 3.2.1 Mitigating the Shift
Revised: 3.2.1 Mitigating the Shift

Original: The solution to mitigate the observed shift and successfully perform distillation using Gaussian noiseis to make use of the BatchNorm layers effectively.
Revised: The solution to mitigate the observed shift and successfully perform distillation using Gaussian noiseis to make use of the BatchNorm layers effectively.

Original: Our goal is to align the activation distributionof hidden layers throughout the teacher network when Gaussian noise is used as an input withwhen original data is used as an input.
Revised: Our goal is to align the activation distributionof hidden layers throughout the teacher network when Gaussian noise is used as an input to thatwhen original data is used as an input.

Original: We consider the case of a BatchNorm layer placed after ahidden layer H within a teacher network.
Revised: We consider the case of a BatchNorm layer placed after ahidden layer H within a teacher network.

Original: Let B represent a batch of images sampled from originaldataset D , G be a batch of random samples sampled from a Gaussian distribution, h be the vector ofactivations for some neuron n belonging to the hidden layer H when inputs are fed to the teacher witha batch-size m , and P ( h ) be the activation distribution of that
Revised: Let B represent a batch of images sampled from originaldataset D , G be a batch of random samples sampled from a Gaussian distribution, h be the vector ofactivations for some neuron n belonging to the hidden layer H when inputs are fed to the teacher witha batch-size m , and P ( h ) be the activation distribution of that

Original: n th neuron over the batch.
Revised: n th neuron over the batch.

Original: Due to thecovariate shift, we observe that: P ( h | G )
Revised: Due to thecovariate shift, we observe that: P ( h | G )

Original: ̸
Revised: ̸

Original: = P ( h | B ) .
Revised: = P ( h | B ) .

Original: Generally, during KD, the BatchNorm layers inthe teacher network are setup for inference, hence uses running statistics ( µ r , σ r ), i.e., Equation (4)to calculate the normalized activations h .
Revised: Generally, during KD, the BatchNorm layersin the teacher network are setup for inference, hence use running statistics ( µ r , σ r ), i.e., Equation (4)to calculate the normalized activations h .

Original: This leads to:
Revised: This leads to:

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:98) h is the scaled and shifted activation vector, which is the output of the BatchNorm layer forthe n th neuron, as defined in Equation (4).
Revised: Here, (cid:98) h is the scaled and shifted activation vector, which is the output of the BatchNorm layer forthe n th neuron, as defined in Equation (4).

Original: This drifted (cid:98) h propagates to the next hidden layer andeventually to the output layer without any meaningful information being passed from the hiddenlayer H .
Revised: This drifted (cid:98) h propagates to the next hidden layer andeventually to the output layer without any meaningful information being passed from the hiddenlayer H .

Original: Therefore, for KD using Gaussian noise, we propose to use current batch statistics ( µ B ,σ B ) for the normalization step, i.e., Equation (2) in all BatchNorm layers of the teacher network.
Revised: Therefore, for KD using Gaussian noise, we propose to use current batch statistics ( µ B ,σ B ) for the normalization step, i.e., Equation (2) in all BatchNorm layers of the teacher network.

Original: The intuition is to align the two drifted distributions at the normalization step.
Revised: The intuition is to align the two drifted distributions at the normalization step.

Original: This makes senseonly because γ and β have the inherent information about the distribution that the next layer expectsand are designed to shift a standard normal distribution.
Revised: This makes senseonly because γ and β have the inherent information about the distribution that the next layer expectsand are designed to shift a standard normal distribution.

Original: Therefore, even though P ( h | G ) ̸
Revised: Therefore, even though P ( h | G ) ̸

Original: = P ( h | B )holds true:
Revised: = P ( h | B )holds true:

Original: [Equation]
Revised: [Equation]

Original: Hence , P ( (cid:98) h | G, µ B , σ B , γ , β ) = P ( (cid:98) h | B, µ B , σ B , γ , β )
Revised: Hence , P ( (cid:98) h | G, µ B , σ B , γ , β ) = P ( (cid:98) h | B, µ B , σ B , γ , β )

Original: Thus, this approach reduces the covariate shift due to a different input distribution from the expectedone at every stage (BatchNorm layer) in the teacher network.
Revised: This reduces the covariate shift due to a different input distribution from the expected one at everystage (BatchNorm layer) in the teacher network.

Original: It obtains P ( (cid:98) h ) similar to that of usingoriginal data irrespective of P ( h ) , and hence leads to the transfer of meaningful information.
Revised: It obtains P ( (cid:98) h ) similar to that of using originaldata irrespective of P ( h ) , and hence leads to the transfer of meaningful information.

Original: Ourapproach is visually explained in Figure 2, where we show an expanded view of a BatchNorm layerfor three neurons.
Revised: Our approachis visually explained in Figure 2, where we show an expanded view of a BatchNorm layer for threeneurons.

Original: It shows how our proposed method effectively uses the normalization step to alignthe two activation distributions: (i) when Gaussian noise is fed as inputs (ii) when original data isfed as inputs.
Revised: It shows how our proposed method effectively uses the normalization step to align the twoactivation distributions: (i) when Gaussian noise is fed as inputs (ii) when original data is fed asinputs.

Original: Note that the covariate shift discussed throughout this paper is not the same as theinternal covariate shift hypothesized by [18], even though ours might be happening in the hidden layers of the neural network as well.
Revised: Note that the covariate shift discussed throughout this paper is not the same as the internalcovariate shift hypothesized by [20], even though ours might be happening in the hidden layers of the neural network as well.

Original: The internal covariate shift occurs due to constantly changingnetwork parameters in the preceding layer, thus affecting the input distribution for the following layer.
Revised: The internal covariate shift occurs due to constantly changing networkparameters in the preceding layer, thus affecting the input distribution for the following layer.

Original: Whereas, we consider the shift due to change in the input that is being fed to the network, which inturn affects the input distribution of hidden layers as well.
Revised: On theother hand, we consider the shift due to change in the input that is being fed to the network, which inturn affects the input distribution of hidden layers as well.

Original: 3.3 Inference
Revised: 3.3 Inference

Original: Similar to how a teacher network trained using original data distribution cannot directly handle the
Revised: Similar to how a teacher network trained using original data distribution cannot directly handle the

Original: Gaussian noise, the student network trained using Gaussian noise will also not be able to providecorrect output during test time, as the running statistics of BatchNorm layers in the student expect
Revised: Gaussian noise, the student network trained using Gaussian noise will also not be able to providecorrect output during test time, as the running statistics of BatchNorm layers in the student expect

Original: Gaussian noise activation patterns.
Revised: Gaussian noise activation patterns.

Original: Therefore, the inference is done using current statistics.
Revised: The inference is, therefore, done using current statistics.

Original: Moreover,each mini-batch needs to be independent and identically distributed, and large enough to approximatethe test data distribution.
Revised: Moreover,each mini-batch needs to be independent and identically distributed, and large enough to approximatethe test data distribution.

Original: Usually, it should not matter how the input is arranged during the inference,therefore, a simple workaround to this problem is to first adjust the running statistics of the studentnetwork catering to the test data by feed-forwarding some amount of shuffled test data beforeevaluating the student network.
Revised: Usually, it should not matter how the input is arranged during the inference,therefore, a simple workaround to this problem is to first adjust the running statistics of the studentnetwork catering to the test data by feed-forwarding some amount of shuffled test data beforeevaluating the student network.

Original: Note that no back-propagation or optimization steps are required asonly the running statistics need to be adjusted.
Revised: Note that no back-propagation or optimization steps are required asonly the running statistics need to be adjusted.

Original: We summarize the training and evaluation parts of ourapproach in Algorithm 1 and Algorithm 2 respectively.
Revised: We summarize the training and evaluation phases ofour approach in Algorithm 1 and Algorithm 2 respectively.

Original: Requires: pretrained teacher T ( . )
Revised: Requires: pretrained teacher T ( . )

Original: Initialize: student S ( . ; θ ) with parameters θ for B in 1 , 2 , ...,
Revised: Initialize: student S ( . ; θ ) with parameters θ for B in 1 , 2 , ...,

Original: B do
Revised: B do

Original: G ∼ N (0 , 1) y T ← T ( G | µ B , σ B ) y S ← S ( G | θ, µ B , σ B ) θ ← θ − η ∂L KD ∂θ end for
Revised: G ∼ N (0 , 1) y T ← T ( G | µ B , σ B ) y S ← S ( G | θ, µ B , σ B ) θ ← θ − η ∂L KD ∂θ end for

Original: Algorithm
Revised: Algorithm

Original: Requires: pretrained student S ( . ; θ ) for batch in 1 , 2 , ..., N do
Revised: Requires: pretrained student S ( . ; θ ) for batch in 1 , 2 , ..., N do

Original: X ∼ D y S ← S
Revised: X ∼ D y S ← S

Original: ( X | θ, µ B , σ B ) y label ← argmax ( y S ) end for
Revised: ( X | θ, µ B , σ B ) y label ← argmax ( y S ) end for

Original:  Experiments
Revised:  Experiments

Original: In this section, we discuss the experiments we performed on various datasets in combination withdifferent standard neural network architectures to demonstrate the working of our proposed method.
Revised: In this section, we discuss the experiments we performed on various datasets in combination withdifferent standard neural network architectures to demonstrate the working of our proposed method.

Original: We use CIFAR-10 [19] as a proof of concept to establish the validity of our method, and for certainablations.
Revised: We use CIFAR10 [21] as a proof of concept to establish the validity of our method, and for certainablations.

Original: We further test our proposed approach on SVHN [20], CIFAR-100, and Food101[21].
Revised: We further test our proposed approach on SVHN [22], CIFAR100, and Food101[23].

Original: Note that, in all the experiments, for Gaussian noise, we generate the image pixels from Gaussiandistribution having mean, µ = 0 and variance, σ 2 = 1 .
Revised: Note that, in all the experiments, for Gaussian noise, we generate the image pixels from Gaussiandistribution having mean, µ = 0 and variance, σ 2 = 1 .

Original: 4.1 State of BatchNorm
Revised: 4.1 State of BatchNorm

Original: In this experiment, we discuss the effect of using the BatchNorm layer in two different states, i.e.,compare between using running statistics and current batch statistics.
Revised: In this experiment, we discuss the effect of using the BatchNorm layer in two different states, i.e.,compare between using running statistics and current batch statistics.

Original: Note that we do this changeonly in the case of the teacher network.
Revised: Note that we do this changeonly in the case of the teacher network.

Original: In contrast, the student network always uses current batchstatistics, even at the time of evaluation on the validation set.
Revised: In contrast, the student network always uses current batchstatistics, even at the time of evaluation on the validation set.

Original: We do this experiment on CIFAR-10 byconsidering two independent parameters: (i) the state of BatchNorm layers in the teacher network,and (ii) the input that is being fed to both the networks.
Revised: We do this experiment on CIFAR10 byconsidering two independent parameters: (i) the state of BatchNorm layers in the teacher network,and (ii) the input that is being fed to both the networks.

Original: We distill a ResNet-34 teacher network pretrained on CIFAR-10 into student networks of varyingarchitectures, which are: ResNet-34, ResNet-18, and MobileNetV2.
Revised: We distill a ResNet-34 teacher network pretrained on CIFAR10 into student networks of varyingarchitectures, which are: ResNet-34, ResNet-18, and MobileNetV2.

Original: Inthe case of original data,we use standard data augmentations like random crop after padding, random horizontal flip, andnormalization.
Revised: In case of original data, we usestandard data augmentations like random crop after padding, random horizontal flip, and normalization.

Original: Whereas, while using Gaussian noise, we independently sample pixels for creatinga 32 × 32  image from a standard Gaussian distribution.
Revised: Whereas, while using Gaussian noise, we independently sample pixels for creating a 32 ×32 image from a standard Gaussian distribution.

Original: In both cases, the batch size is 256, andan Adam optimizer with a learning rate of 10 − 3 for tuning the parameters of the student network isused 1 .
Revised: In both cases, the batch size is 256, and an Adamoptimizer with a learning rate of 10 − 3 for tuning the parameters of the student network is used 1 .

Original: The results in form of mean ± standard deviation from three runs are shown in Table 1.
Revised: The results in form of mean ± standard deviation from three runs are shown in Table 1.

Original: As expected,the conventional setting of using the original data as input and the teacher using the running statistics
Revised: As expected,the conventional setting of using the original data as input and the teacher using the running statistics

Original: [Table]
Revised: [Table]

Original: computed from the original data is the ideal case, which results in the best performance.
Revised: computed from the original data is the ideal case, which results in the best performance.

Original: Usingcurrent batch statistics does not hinder the learning of the student network either, that is, in the case oforiginal data.
Revised: Usingcurrent batch statistics does not hinder the learning of the student network either, that is, in the case oforiginal data.

Original: However, while using Gaussian noise with running statistics, the activation distributiongets mismatched (Figure 1b).
Revised: However, while using Gaussian noise with running statistics, the activation distributiongets mismatched (Figure 1b).

Original: Whereas using current batch statistics results in the mitigation of thismismatch.
Revised: Whereas using current batch statistics results in the mitigation of thismismatch.

Original: Hence, in the former case, the student network cannot learn at all, whereas in the latter, itcan achieve an accuracy close to the ideal case, considering that we rely on the random noise.
Revised: Hence, in the former, the student network is unable to learn at all, whereas in the latter, itachieves an accuracy close to the ideal case, considering that we rely on the random noise.

Original: The distribution shifts, in this case, are shown in the Figure 1b.
Revised: The distribution shifts observed in this experiment are shown in the Figure 1b.

Original: We use activations of the ‘avgpool’layer in the teacher network (ResNet-34) to obtain these plots.
Revised: We use activationsof ‘avgpool’ layer in the teacher network (ResNet-34) to obtain these plots.

Original: Figure 1b shows the distribution ofaverage activations across all neurons with respect to different settings discussed above.
Revised: Figure 1b shows thedistribution of average activations across all neurons with respect to different settings discussed above.

Original: We observethat while using Gaussian noise,using current batch statistics results in a distribution similar tothat of using original data.
Revised: We observe that while using Gaussian noise, current batch statistics result in a distribution similarto that of using original data.

Original: Also, in the case of Gaussian noise with BatchNorm layers using therunning statistics, most neurons have close to zero average activations, meaning they rarely get fired.
Revised: Also, in the case of Gaussian noise with BatchNorm layers using therunning statistics, most neurons have close to zero average activations, implying that they rarely fire.

Original: Therefore, their information is never used in deciding the outcome, hence that particular informationwill not ever be learned by the student network.
Revised: Their information is, therefore, never used in deciding the outcome, hence that particular informationis never learned by the student network.

Original: These plots go along with the results in Table 1.
Revised: These plots are consistent with the results in Table 1.

Original: 4.2 Results on Other Datasets
Revised: 4.2 Results on Other Datasets

Original: We perform experiments similar to that of CIFAR10 on SVHN, CIFAR100 and Food101 to show theapplicability of our method on various datasets and neural network architectures.
Revised: We perform experiments similar to that of CIFAR10 on SVHN, CIFAR100 and Food101 to show theapplicability of our method on various datasets and neural network architectures.

Original: Since we did notfind a considerable variance in the case of CIFAR-10, here we report the results on single runs.
Revised: Since we did notfind a considerable variance in the case of CIFAR10, here we report the results on single runs.

Original: The teacher-student pairs are mentioned in the Table 2.
Revised: The teacher-student pairs are mentioned in the Table 2.

Original: We use standard data augmentation and asimilar setup of hyperparameters, i.e., batches of 256 samples, and optimization by Adam optimizerwith a learning rate of 10 − 3 .
Revised: We use standard data augmentation and asimilar setup of hyperparameters, i.e., batches of 256 samples, and optimization by Adam optimizerwith a learning rate of 10 − 3 .

Original: The results are shown in Table 2.
Revised: Table 2 shows the obtained results.

Original: We see similar results as before, except that the Gaussian noise isnot at random chance even with running statistics in the case of SVHN.
Revised: We see similar trends as before, except that the Gaussian noise isnot at random chance even with running statistics in the case of SVHN.

Original: We believe that this is due torelatively lower covariate shift of activation distribution compared to other more sophisticate datasets.
Revised: We believe that this is due torelatively lower covariate shift of activation distribution compared to other more sophisticate datasets.

Original: For CIFAR100 and Food101 datsets we try out teacher-student architectures which are different fromthe previous experiments to introduce more variability.
Revised: For CIFAR100 and Food101 datsets we try out teacher-student architectures which are different fromthe previous experiments to introduce more variability.

Original: Table 2 shows that a large amount of gapbetween real data-based distillation and data-free distillation is reduced by Gaussian noise and as wesee in the next section that the remaining gap can be filled by other mechanisms.
Revised: Table 2 shows that a large amount of gapbetween real data-based distillation and data-free distillation is reduced by Gaussian noise and as wesee in the next section, the remaining gap can be filled by other mechanisms.

Original: 4.3 Limited Data
Revised: 4.3 Limited Data

Original: In this experiment, we try to find out the amount of additional data needed for extra efforts after
Revised: In this experiment, we try to find out the amount of additional data needed for extra efforts after

Original: Gaussian noise-based KD.
Revised: Gaussian noise-based KD.

Original: It would be interesting to use the synthetic data for this, however, we relyon samples from the training dataset asa proxy due to a large variation in the nature of syntheticdata generation approaches.
Revised: It would be interesting to use the synthetic data for this, however, werely on samples from the training dataset as proxy due to a large variation in the nature of syntheticdata generation approaches.

Original: We first pretrain our student model using Gaussian distillation and thenfinetune it using original data.
Revised: We first pretrain our student model using Gaussian distillation and thenfinetune it using original data.

Original: First, we show how the performance varies with respect to the amountof data used for finetuning, then we brieflyshow the same for different configurations
Revised: First, we show how the performance varies with respect to the amountof data used for finetuning, then we briefly

Original: We compare our case of pretraining using Gaussian noise against traditional KD using the same fractional amount of data (baseline)
Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: For finetuning

Original: [Table]
Revised: [Table]

Original: For finetuning, a subset of training data is sampled randomly and a reduced learning rate of 10 − 4is used.
Revised: a subset of training data is sampled randomly and a reduced learning rate of 10 − 4is used.

Original: Figure 3(Left) compares the accuracy of the ResNet18 (student) on CIFAR10 when it isfinetuned using the given fraction of training data to its counterpart that is trained from scratch.
Revised: Figure 3(Left) compares the accuracy of the ResNet18 (student) on CIFAR10 when it isfinetuned using the given fraction of training data to its counterpart that is trained from scratch.

Original: Further, Table 3 shows the student accuracy in case of other architectures and datasets for 1% oftraining data.
Revised: Further [Table] Table 3 shows the student accuracy for other architectures and datasets with 1% of trainingdata.

Original: We observe a considerable improvement in the student network after finetuning, specifically in thecase of MobileNetV2 on CIFAR-10 of about 4%.
Revised: We observe a considerable improvement in the student network after finetuning, specifically for MobileNetV2 on CIFAR10 (about 4%).

Original: In addition, pretraining using Gaussian noiseoutperforms the baseline by a large margin.
Revised: In addition, pretraining using Gaussian noise outperformsthe baseline by a large margin.

Original: 4.4 BatchNorm State in Student
Revised: 4.4 BatchNorm State in Student

Original: Here, we explore how the batch size or inference, in general, affects the student network trainedusing our Gaussian noise-based approach.
Revised: Here, we explore how the batch size or inference, in general, affects the student network trainedusing our Gaussian noise-based approach.

Original: We use the ResNet-18 student network distilled using ourapproach from ResNet-34 teacher trained on CIFAR-10.
Revised: We use the ResNet-18 student network distilled using ourapproach from ResNet-34 teacher trained on CIFAR10.

Original: We use a minimal amount  of evaluation datato adjust the running statistics of the student’s BatchNorm layers, which is 20 batches of size 16.
Revised: We use a sufficient amount (study on numberof batches for adjusting estimates is included in supplementary material) of evaluation data to adjustthe running statistics of the student’s BatchNorm layers, which is 20 batches of size 16.

Original: After the student is trained using Gaussian noise, its running statistics are not adjusted for test datadistribution.
Revised: After the student is trained using Gaussian noise, its running statistics are not adjusted for test datadistribution.

Original: Therefore, the accuracy using running statistics is almost at random chance (blue),shown in Figure 3(Middle).
Revised: Therefore, the accuracy using running statistics is almost at random chance (blue),shown in Figure 3(Middle).

Original: Then we use the current statistics and vary the batch size, where wesee that the accuracy increases with an increase in batch size due to better estimates of µ B and σ B(black).
Revised: When we use the current statistics and vary the batch size, we see thatthe accuracy increases with an increase in batch size due to better estimates of µ B and σ B (black).

Original: Further, we adjust the running statistics using some amount of original data, while keepingthe rest of the parameters fixed, considering the case in which the student model needs to infer on asingle sample or a batch of non i.i.d.
Revised: Further, we adjust the running statistics using some amount of original data, while keeping the restof the parameters fixed, considering the case in which the student model needs to infer on a singlesample or a batch of non-i.i.d.

Original: samples.
Revised: samples.

Original: We observe that the adjusted running statistics yieldperformance similar to that of large batch sizes (red).
Revised: We observe that the adjusted running statistics yield aperformance similar to that of large batch sizes (red).

Original:  Discussion5.1 Limitations
Revised:  Discussion

Revised: 5.1 Limitations

Original: Absence of BatchNorm: If there are no BatchNorm layers present in the teacher-student architecture,then the proposed method will not work.
Revised: Absence of BatchNorm: If there are no BatchNorm layers present in the teacher-student architecture,then the proposed method will not work.

Original: In that case, one would need to learn the input datadistribution to obtain activation distribution similar to that of original data.
Revised: In that case, one would need to learn the input datadistribution to obtain activation distribution similar to that of original data.

Original: Training: As opposed to traditional KD, the teacher in the proposed method is required to use thecurrent batch statistics while distilling the knowledge to student.
Revised: Training: As opposed to traditional KD, the teacher in the proposed method is required to use thecurrent batch statistics while distilling the knowledge to student.

Original: In order to correctly calculate thesestatistics and output correct soft labels labels, a large enough batch size is required during trainingperiod 2 .
Revised: In order to correctly calculate thesestatistics and output correct soft labels labels, a large enough batch size is required during trainingperiod 2 .

Original: Inference: As explained in Section 3.3, the student model might be limited by how it can be usedduring inference.
Revised: Inference: As explained in Section 3.3, the student model might be limited by how it can be usedduring inference.

Original: The student network trained using our proposed approach should either be usedwith a sufficiently large batch of test samples or its running statistics should be adjusted beforehandusing a small amount of test data.
Revised: The student network trained using our proposed approach should either be usedwith a sufficiently large batch of test samples or its running statistics should be adjusted beforehandusing a small amount of test data.

Original: Without doing so, the student model trained using Gaussian noisedoes not get adapted to real data distribution, as shown in Figure 3(Right).
Revised: Without doing so, the student model trained using Gaussian noisedoes not get adapted to real data distribution, as shown in Figure 3(Right).

Original: The plot shows averageactivation values for neurons in the ‘avgpool’ layer of a student network trained using our approachfor CIFAR-10.
Revised: The plot shows averageactivation values for neurons in the ‘avgpool’ layer of a student network trained using our approach for CIFAR10.

Original: The distribution of points for Gaussian noise (blue) and original data with currentstatistics (orange) is similar.
Revised: The distribution of points for Gaussian noise (blue) and original data with currentstatistics (orange) is similar.

Original: Whereas the points for original data with running statistics (red) eitherlie close to zero or explode to a very high value (clipped to 1.0).
Revised: Whereas the points for original data with running statistics (red) eitherlie close to zero or explode to a very high value (clipped to 1.0).

Original: 5.2 Rethinking knowledge transfer
Revised: 5.2 Rethinking knowledge transfer

Original: Recently, [22] explored different types of structured noise (which proves to be far better than Gaussiannoise) as a way to learn representations using a specific contrastive loss and questioned the necessityof huge data-driven vision systems.
Revised: Recently, [24] explored different types of structured noise (which proves to be far better than Gaussiannoise) as a way to learn representations using a specific contrastive loss and questioned the necessityof huge data-driven vision systems.

Original: [16] also observed that low-level features are sufficient forperforming KD.
Revised: [18] also observed that low-level features are sufficient forperforming KD.

Original: This paper, shows that using just the Gaussian noise (not even low-level features) wecan successfully transfer knowledge from one neural network to the other to a large extent.
Revised: This paper, shows that using just the Gaussian noise (not even low-level features) wecan successfully transfer knowledge from one neural network to the other to a large extent.

Original: We havenot yet explored the possibility of using structured noise, which could prove to be more effective.
Revised: We havenot yet explored the possibility of using structured noise, which could prove to be more effective.

Original: This challenges the generally accepted notion of requiring synthetic data with representations similarto original data and provides motivation to explore methods similar to ours for tasks such as domainadaptation, transfer learning, etc.
Revised: This challenges the generally accepted notion of requiring synthetic data with representations similarto original data and provides motivation to explore methods similar to ours for tasks such as domainadaptation, transfer learning, etc.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we explore KD solely based on Gaussian noise.
Revised: In this paper, we explore KD solely based on Gaussian noise.

Original: We do this by mitigating the covariateshift in the hidden layers of the teacher network, which is caused due to the change in the input datadistribution.
Revised: We do this by mitigating the covariateshift in the hidden layers of the teacher network, which is caused due to the change in the input datadistribution.

Original: Further, we show how similar phenomena can occur in the student network, and howwe can use a similar approach there as well.
Revised: Further, we show how similar phenomena can occur in the student network, and howwe can use a similar approach there as well.

Original: The experiments show that student network trainedusing our approach can achieve a greater performance, in contrast to what prior works have shownfor Gaussian noise setting, and more so in the presence of limited data.
Revised: The experiments show that student network trainedusing our approach can achieve a greater performance, in contrast to what prior works have shownfor Gaussian noise setting, and more so in the presence of limited data.

Original: Future work could be to try
Revised: Future work could be to try

Original: Gaussian noise-based distillation with lower input dimensions such that it feed-forwards throughboth teacher and student network since the Gaussian noise does not contain structural informationanyway.
Revised: Gaussian noise-based distillation with lower input dimensions such that it feed-forwards throughboth teacher and student network since the Gaussian noise does not contain structural informationanyway.

Original: Further future work could involve comparing how our approach fares to using unrelated datafor KD, trying our approach on structured noise, and extending our findings to other domains such asdomain adaptation.
Revised: Further future work could involve comparing how our approach fares to using unrelated datafor KD, trying our approach on structured noise, and extending our findings to other domains such asdomain adaptation.