Original: Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Revised: Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation

Original: Abstract
Revised: Abstract

Original: Data-free Knowledge Distillation (DFKD) has attracted attention recently thanksto its appealing capability of transferring knowledge from a teacher network to astudent network without using training data.
Revised: Data-free Knowledge Distillation (DFKD) has attracted attention recently thanksto its appealing capability of transferring knowledge from a teacher network to astudent network without using training data.

Original: The main idea is to use a generatorto synthesize data for training the student.
Revised: The main idea is to use a generator tosynthesize data for training the student.

Original: As the generator gets updated, thedistribution of synthetic data will change.
Revised: As the generator gets updated, the distribution of synthetic data will change.

Original: Such distribution shift could be large ifthe generator and the student are trained adversarially, causing the student to forgetthe knowledge it acquired at  previous steps.
Revised: Such distribution shift could be large if thegenerator and the student are trained adversarially, causing the student to forget theknowledge it acquired at the previous steps.

Original: To alleviate this problem, we proposea simple yet effective method called Momentum Adversarial Distillation (MAD)which maintains an exponential moving average (EMA) copy of the generatorand uses synthetic samples from both the generator and the EMA generator totrain the student.
Revised: To alleviate this problem, we proposea simple yet effective method called Momentum Adversarial Distillation (MAD)which maintains an exponential moving average (EMA) copy of the generator anduses synthetic samples from both the generator and the EMA generator to trainthe student.

Original: Since the EMA generator can be considered as an ensembleof the generator’s old versions and often undergoes a smaller change in updatescompared to the generator, training on its synthetic samples can help the studentrecall the past knowledge and prevent the student from adapting too quickly to  newupdates of the generator.
Revised: Since the EMA generator can be considered as an ensemble of thegenerator’s old versions and often undergoes a smaller change in updates comparedto the generator, training on its synthetic samples can help the student recall thepast knowledge and prevent the student from adapting too quickly to the newupdates of the generator.

Original: Our experiments on six benchmark datasets includingbig datasets like ImageNet and Places365 demonstrate the superior performance of
Revised: Our experiments on six benchmark datasets includingbig datasets like ImageNet and Places365 demonstrate the superior performance of

Original: MAD over competing methods for handling the large distribution shift problem.
Revised: MAD over competing methods for handling the large distribution shift problem.

Original: Our method also compares favorably to existing DFKD methods and even achievesstate-of-the-art results in some cases.
Revised: Our method also compares favorably to existing DFKD methods and even achievesstate-of-the-art results in some cases.

Original:  Introduction
Revised:  Introduction

Original: With the development of deep learning, more pretrained deep neural networks have been releasedto the public [5, 10, 18, 38, 43].
Revised: With the development of deep learning, more pretrained deep neural networks have been releasedto the public [5, 10, 17, 36, 41].

Original: However, their superior performances often come with big sizes,causing difﬁculties in deployment of these pretrained networks on resource-constrained devices.
Revised: However, their superior performances often come with big sizes,causing difﬁculties in deployment of these pretrained networks on resource-constrained devices.

Original: Thisleads to the demand for transferring  knowledge from a cumbersome pretrained source network (called“teacher”) to a compact target network (called “student”) with a minimal loss of performance.
Revised: Thisleads to the demand for transferring the learned knowledge from a cumbersome pretrained sourcenetwork (called “teacher”) to a compact target network (called “student”) with a minimal loss ofperformance.

Original: Thistask is regarded as Knowledge Distillation (KD)[19].
Revised: This task is regarded as Knowledge Distillation (KD)

Original: The original idea of KD is to make use of class probabilities predicted by the teacher which encapsulate the hidden correlation among classes as training signals for the student.
Revised: The original idea of KD is to make use of class probabilities predicted by the teacher which encapsulatethe hidden correlation among classes as training signals for the student.

Original: Note that such “dark” knowledge [19] is generally not available if the student is trained directly on raw data with one-hotlabels.
Revised: Note that such “dark”knowledge [18] is generally not available if the student is trained directly on raw data with one-hotlabels.

Original: Later, various KD methods have been proposed to improve the quality of knowledge transfer.
Revised: Later, various KD methods have been proposed to improve the quality of knowledge transfer.

Original: For example, AT [47] matches the spatial attention maps at intermediate layers of the student and the teacher.
Revised: For example, AT [45] matches the spatial attention maps at intermediate layers of the student and theteacher.

Original: SPD [42] encourages the similarity between the student’s and teacher’s feature correlationmatrices.
Revised: SPD [40] encourages the similarity between the student’s and teacher’s feature correlationmatrices.

Original: PKT [36] transfers the conditional probability between two samples computed via kerneldensity estimation on the feature space.
Revised: PKT [34] transfers the conditional probability between two samples computed via kerneldensity estimation on the feature space.

Original: RKD [35] exploits relational knowledge for distillation
Revised: RKD [33] exploits relational knowledge for distillation

Original: VID
Revised: VID

Original: [1] maximizes a variational lower bound of the mutual information (MI) between the student’s andteacher’s representations.
Revised: [1] maximizes a variational lower bound of the mutual information (MI) between the student’s andteacher’s representations.

Original: CRD [41] uses contrastive learning as a proxy for maximizing MI.
Revised: CRD [39] uses contrastive learning as a proxy for maximizing MI.

Original: The common drawback of these methods is the reliance on samples from the teacher training set.
Revised: The main drawback of these methods is the reliance on samples from the teacher training set.

Original: However, in practice, accessing the original training data is usually infeasible due to many reasonssuch as data privacy (e.g., healthcare data containing personal information) or data regarded asintellectual property of the vendors.
Revised: However,in practice, accessing the original training data is usually infeasible due to many reasons such asdata privacy (e.g., healthcare data containing personal information) or data regarded as intellectualproperty of the vendors.

Original: Addressing this critical issue, Data-Free Knowledge Distillation(DFKD) methods have been introduced [7, 13, 27, 29, 31, 34, 44–46].
Revised: Addressing this critical issue, Data-Free Knowledge Distillation (DFKD)methods have been introduced [7, 12, 26, 28, 30, 32, 42–44].

Original: A common DFKD approach isto use a generator network to synthesize training data and jointly train the generator and the studentin an adversarial manner [13, 31, 46].
Revised: A common DFKD approach is to use agenerator network to synthesize training data and jointly train the generator and the student in anadversarial manner [12, 30, 44].

Original: Under this adversarial learning scheme, the student attempts tomake predictions as close as possible to the teacher’s on synthetic data generated by the generator,while the generator tries to create samples that maximize the mismatch between the student’s and theteacher’s predictions.
Revised: Under this adversarial learning scheme, the student attempts to makepredictions as close as possible to the teacher’s on synthetic data generated by the generator, while thegenerator tries to create samples that maximize the mismatch between the student’s and the teacher’spredictions.

Original: This adversarial game enables a rapid exploration of synthetic distributionsuseful for knowledge transfer between the teacher and the student.
Revised: This adversarial game enables a rapid exploration of synthetic distributions useful forknowledge transfer between the teacher and the student.

Original: At the same time, it could alsolead to large shifts in the synthetic distributions, causing the student to forget useful knowledgeacquired at the previous steps and suffer from performance drops [3].
Revised: At the same time, it could also lead to largeshifts in the synthetic distributions, causing the student to forget useful knowledge acquired at theprevious steps and suffer from performance drops [3].

Original: In this paper, we propose a simple yet effective method called Momentum Adversarial Distillation(MAD) to mitigate the large distribution shift problem in adversarial DFKD.
Revised: In this paper, we propose a simple yet effective method called Momentum Adversarial Distillation(MAD) to mitigate the large distribution shift problem in adversarial DFKD.

Original: MAD maintains anexponential moving average (EMA) copy of the generator which is responsible for storing informationabout past updates of the generator.
Revised: MAD maintains anexponential moving average (EMA) copy of the generator which is responsible for storing informationabout past updates of the generator.

Original: By using synthetic samples from the EMA generator as additionaltraining data for the student besides those from the generator, MAD can ensure that the student canrecall the old knowledge, hence, is less prone to forgetting.
Revised: By using synthetic samples from the EMA generator as additionaltraining data for the student besides those from the generator, MAD can ensure that the student canrecall the old knowledge, hence, is less prone to forgetting.

Original: Moreover, to reduce the negative effectcaused by spurious solutions of an unconditional generator when learning on large datasets suchas ImageNet, we propose to use a class-conditional generator that takes the sum of a noise vectorand a class embedding vector as input, and train this generator with a new objective that suppressesthe presence of spurious solutions.
Revised: Moreover, to reduce the negative effectcaused by spurious solutions of an unconditional generator when learning on large datasets suchas ImageNet, we propose to use a class-conditional generator that takes the sum of a noise vectorand a class embedding vector as input, and train this generator with a new objective that suppressesthe presence of spurious solutions.

Original: This technique requires only a small change in the generator’sarchitecture but enables MAD (and possibly other adversarial DFKD methods) to learn surprisinglywell on large datasets.
Revised: This technique requires only a small change in the generator’sarchitecture but enables MAD (and possibly other adversarial DFKD methods) to learn surprisinglywell on large datasets.

Original: Through extensive experiments on three small and three large image datasets,we demonstrate that our proposed method is far better than related baselines [3, 31] in dealing withthe large distribution shift problem.
Revised: Through extensive experiments on three small and three large image datasets,we demonstrate that our proposed method is far better than related methods [3, 30] in dealing withthe large distribution shift problem.

Original: In some cases, MAD even outperforms current state-of-the-artmethods [8, 14].
Revised: In some cases, MAD even outperforms current state-of-the-artmethods [8, 13].

Original:  Adversarial Data-Free Knowledge Distillation
Revised:  Adversarial Data-Free Knowledge Distillation

Original: Let T be a teacher network pretrained on some dataset D train and S be a fresh student network.
Revised: Let T be a teacher network pretrained on some dataset D train and S be a fresh student network.

Original: Let T ( · ) and S ( · ) denote outputs of the teacher and student networks before the softmax activation ,respectively.
Revised: Let T ( · ) and S ( · ) denote outputs of the teacher and student networks before the softmax activation ,respectively.

Original: In Data-Free Knowledge Distillation (DFKD), we want to transfer knowledge from
Revised: In Data-Free Knowledge Distillation (DFKD), we want to transfer knowledge from

Original: T to S so that S performs as well as or even better than T on the original test set D test but with aconstraint that no training data for S is available.
Revised: T to S so that S performs as well as or even better than T on the original test set D test but with aconstraint that no training data for S is available.

Original: An intuitive way to deal with such constraint islearning an additional generator network G that can generate synthetic data for training S from a noisedistribution p ( z ) usually chosen to be the standard Gaussian distribution N (0 , I) .
Revised: An intuitive way to deal with such constraint islearning an additional generator network G that can generate synthetic data for training S from a noisedistribution p ( z ) usually chosen to be the standard Gaussian distribution N (0 , I) .

Original: Adversarial Belief
Revised: Adversarial Belief

Original: Matching (ABM) [31] proposed an adversarial learning framework between S and G via optimizingthe following min-max objective:
Revised: Matching (ABM) [30] proposed an adversarial learning framework between S and G via optimizingthe following min-max objective:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (2) where L KD ( x ) denotes the knowledge distillation (KD) loss, i.e., the discrepancy between S ( x ) and
Revised: (2) where L KD ( x ) denotes the knowledge distillation (KD) loss, i.e., the discrepancy between S ( x ) and

Original: T ( x ) .
Revised: T ( x ) .

Original: In ABM, L KD ( x ) is the Kullback-Leibler (KL) divergence between class probabilities of T and
Revised: In ABM, L KD ( x ) is the Kullback-Leibler (KL) divergence between class probabilities of T and

Original: S computed on x :
Revised: S computed on x :

Original: [Equation]
Revised: [Equation]

Original: where Tp ( x ) = softmax ( T ( x )) and Sp ( x ) = softmax ( S ( x )) denote the class probabilities of T and Scomputed on x , respectively; and C is the total number of classes.
Revised: where Tp ( x ) = softmax ( T ( x )) and Sp ( x ) = softmax ( S ( x )) denote the class probabilities of T and Scomputed on x , respectively; and C is the total number of classes.

Original: The core idea behind the optimization in Eq.
Revised: The core idea behind the optimization in Eq.

Original: 1 is to encourage G to generate samples on which theoutputs of S are very different from those of T (or the KD loss is large).
Revised: 1 is to encourage G to generate samples on which theoutputs of S are very different from those of T (or the KD loss is large).

Original: Typically, the generated
Revised: Typically, the generated

Original: + + samples have not been observed by S during training (otherwise, the KD loss will be small) andthe learning task of S is to match T on these novel samples.
Revised: + + samples have not been observed by S during training (otherwise, the KD loss will be small) andthe learning task of S is to match T on these novel samples.

Original: It is expected that in the ideal case, viacontinuous adversarial exploration of G , S can be exposed to a diverse enough set of synthetic sampleswhich allows S to match T with an arbitrarily small prediction error on D test .
Revised: It is expected that in the ideal case, viacontinuous adversarial exploration of G , S can be exposed to a diverse enough set of synthetic sampleswhich allows S to match T with an arbitrarily small prediction error on D test .

Original: In practice, we usually implement Eq.
Revised: In practice, we usually implement Eq.

Original: 1 by alternately optimizing S and G in n S and n G steps,respectively.
Revised: 1 by alternately optimizing S and G in n S and n G steps,respectively.

Original: Thus, to avoid confusion later, we refer to a n S -step update of S as an update stage of Sand similarly, a n G -step update of G as an update stage of G .
Revised: Thus, to avoid confusion later, we refer to a n S -step update of S as an update stage of Sand similarly, a n G -step update of G as an update stage of G .

Original:  Momentum Adversarial Distillation3.1 Handling large distribution shifts with an additional EMA generator
Revised:  Momentum Adversarial Distillation3.1 Handling large distribution shifts with an additional EMA generator

Original: Although theoretically sound, the optimization in Eq.
Revised: Although theoretically sound, the optimization in Eq.

Original: 1 has a practical problem: If the update of Gvaries too much, the distribution of synthetic samples generated by G will change signiﬁcantly overtwo consecutive update steps of S , which in turns causes S to catastrophically forget what it haslearned at the previous stages [3] to adapt to the new update of G .
Revised: 1 has a practical problem: If the update of Gvaries too much, the distribution of synthetic samples generated by G will change signiﬁcantly overtwo consecutive update steps of S , which in turns causes S to catastrophically forget what it haslearned at the previous stages [3] to adapt to the new update of G .

Original: In order to address this problem, wepropose to maintain an exponential moving average (EMA) of the generator G , denoted by ˜ G , duringlearning and use synthetic samples from both G and ˜ G to train S as follows:
Revised: In order to address this problem, wepropose to maintain an exponential moving average (EMA) of the generator G , denoted by ˜ G , duringlearning and use synthetic samples from both G and ˜ G to train S as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ 0 , λ 1 ≥ 0 are coefﬁcients.
Revised: where λ 0 , λ 1 ≥ 0 are coefﬁcients.

Original: The parameters θ t ˜ G of the momentum generator ˜ G at the updatestage t of S (after n G steps update of G ) are computed as:
Revised: The parameters θ t ˜ G of the momentum generator ˜ G at the updatestage t of S (after n G steps update of G ) are computed as:

Original: [Equation]
Revised: [Equation]

Original: where α ( 0 < α < 1 ) is the momentum.
Revised: where α ( 0 < α < 1 ) is the momentum.

Original: If α is close to 1, ˜ G will change very slightly compared to G .
Revised: If α is close to 1, ˜ G will change very slightly compared to G .

Original: Therefore, by using synthetic samples from ˜ G as additional training data for S besides those from G ,we can alleviate the large exploratory distribution shift caused by the large update of G and achieve astable update of S .
Revised: Therefore, by using synthetic samples from ˜ G as additional training data for S besides those from G ,we can alleviate the large exploratory distribution shift caused by the large update of G and achieve astable update of S .

Original: We name our proposed method Momentum Adversarial Distillation (MAD).
Revised: We name our proposed method Momentum Adversarial Distillation (MAD).

Original: See
Revised: See

Original: Fig.
Revised: Fig.

Original: 1 for an illustration.
Revised: 1 for an illustration.

Original: 3.2 Enabling MAD to learn well on large datasets
Revised: 3.2 Enabling MAD to learn well on large datasets

Original: During our experiments, we observed that MAD, with the unconditional generator G described in
Revised: During our experiments, we observed that MAD, with the unconditional generator G described in

Original: Section 2, is not able to learn well on large datasets such as ImageNet.
Revised: Section 2, is not able to learn well on large datasets such as ImageNet.

Original: Our hypothesis is that incase of large datasets, the objective of G in Eq.
Revised: Our hypothesis is that incase of large datasets, the objective of G in Eq.

Original: 2 induces a very large number spurious solutions,which hampers the learning of G .
Revised: 2 induces a very large number spurious solutions,which hampers the learning of G .

Original: To see this, ﬁrst let us recall the objective of G which is maximizingthe KL divergence between Sp ( x ) and Tp ( x ) over some synthetic sample x = G ( z ) .
Revised: To see this, ﬁrst let us recall the objective of G which is maximizingthe KL divergence between Sp ( x ) and Tp ( x ) over some synthetic sample x = G ( z ) .

Original: If we, forexample, assume that there are 3 classes in total and Tp ( x ) is ﬁxed at [1 , 0 , 0] , then we could train
Revised: If we, forexample, assume that there are 3 classes in total and Tp ( x ) is ﬁxed at [1 , 0 , 0] , then we could train

Original: G to generate x so that Sp ( x ) is either [0 , 1 , 0] or [0 , 0 , 1] .
Revised: G to generate x so that Sp ( x ) is either [0 , 1 , 0] or [0 , 0 , 1] .

Original: In this toy example, we see 2 spurioussolutions for 3 classes.
Revised: In this toy example, we see 2 spurioussolutions for 3 classes.

Original: If a dataset has 1000 classes like ImageNet, there will be at least 999 spurioussolutions.
Revised: If a dataset has 1000 classes like ImageNet, there will be at least 999 spurioussolutions.

Original: Besides class numbers, larger input sizes also increase the space and the number of(spurious) solutions.
Revised: Besides class numbers, larger input sizes also increase the space and the number of(spurious) solutions.

Original: Unfortunately, the more spurious solutions, the more likely G could jump fromone spurious solution to another in successive update steps, causing instability in training G .
Revised: Unfortunately, the more spurious solutions, the more likely G could jump fromone spurious solution to another in successive update steps, causing instability in training G .

Original: To overcome this problem, we propose to condition G on a class label y as G ( z + e y ) where e yis a trainable embedding of y .
Revised: To overcome this problem, we propose to condition G on a class label y as G ( z + e y ) where e yis a trainable embedding of y .

Original: We found that using the sum of z and e y as input to G instead ofconcatenation allows our method to learn much better, possibly becausethe noise in updating e y isabsorbed into the stochasticity of z via summation rather than concatenation.
Revised: We found that using the sum of z and e y as input to G instead ofconcatenation allows our method to learn much better, possibly because summation makes z ande y inseparable, thus, forces G to use both vectors during training while concatenation does not.

Original: We illustrate this ideain Fig. 1, and provide an empirical justiﬁcation in Appdx.
Revised: Weillustrate this idea in Fig. 1, and provide an empirical justiﬁcation in Appdx.

Original: B.3.
Revised: B.3.

Original: Denoted by E the list of trainableembedding vectors for all classes ( E = ( e 1 , ..., e C ) ), we train both G and E together by minimizingthe following loss: 
Revised: Denoted by E thelist of trainable embedding vectors for all classes ( E = ( e 1 , ..., e C ) ), we train both G and E togetherby minimizing the following loss: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Cat ( C ) is the uniform categorical distribution over C classes; λ 2 , λ 3 , λ 4 ≥ 0 are hyperparameters; and L NLL ( x, y ) and L NormReg ( e y ) are the negative log-likelihood and the norm regularizationlosses respectively, deﬁned as follows:
Revised: where Cat ( C ) is the uniform categorical distribution over C classes; λ 2 , λ 3 , λ 4 ≥ 0 are hyperparameters; and L NLL ( x, y ) and L NormReg ( e y ) are the negative log-likelihood and the norm regularizationlosses respectively, deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where d e denotes the dimensionality of e y ; and γ ≥ 1 is a scaling hyperparameter.
Revised: where d e denotes the dimensionality of e y ; and γ ≥ 1 is a scaling hyperparameter.

Original: L NormReg ( e y )servers as a constraint that restricts the norm of e y to be smaller than γ × √ d e .
Revised: L NormReg ( e y )servers as a constraint that restricts the norm of e y to be smaller than γ × √ d e .

Original: An explanationof the formula γ × √ d e is provided in Appdx.
Revised: An explanationof the formula γ × √ d e is provided in Appdx.

Original: C. Intuitively, via minimizing L NLL ( x, y ) , G couldmaintain its focus on predicting y throughout its entire update stage rather than jumping betweendifferent spurious solutions.
Revised: C. Intuitively, via minimizing L NLL ( x, y ) , G couldmaintain its focus on predicting y throughout its entire update stage rather than jumping betweendifferent spurious solutions.

Original: Besides, since y is sampled uniformly, the synthetic data will be evenlydistributed among classes.
Revised: Besides, since y is sampled uniformly, the synthetic data will be evenlydistributed among classes.

Original: In case the teacher T has BatchNorm [20] layers, we can make use of the running mean ¯ µ (cid:96) andrunning variance ¯ ω (cid:96) of each BatchNorm layer (cid:96) of T to guide the data synthesis of G by adding the
Revised: In case the teacher T has BatchNorm [19] layers, we can make use of the running mean ¯ µ (cid:96) andrunning variance ¯ ω (cid:96) of each BatchNorm layer (cid:96) of T to guide the data synthesis of G by adding the

Original: BatchNorm moment matching (BNmm) loss [45] below to L G , E (weighted by λ 5 ≥ 0 ):
Revised: BatchNorm moment matching (BNmm) loss [43] below to L G , E (weighted by λ 5 ≥ 0 ):

Original: [Equation]
Revised: [Equation]

Original: (9) where µ (cid:96) , ω (cid:96) are the empirical mean and variance of the features at the BatchNorm layer (cid:96) w.r.t.
Revised: where µ (cid:96) , ω (cid:96) are the empirical mean and variance of the features at the BatchNorm layer (cid:96) w.r.t.

Original: thebatch of synthetic samples from G .
Revised: thebatch of synthetic samples from G .

Original: During our experiment, we observed that using L BNmm improvesknowledge distillation of the student S .
Revised: During our experiment, we observed that using L BNmm improvesknowledge distillation of the student S .

Original:  Related Work
Revised:  Related Work

Original: In Data-Free Knowledge Distillation (DFKD), it is critical to synthesize data that are useful fortransferring knowledge from T to S .
Revised: In Data-Free Knowledge Distillation (DFKD), it is critical to synthesize data that are useful fortransferring knowledge from T to S .

Original: Existing DFKD methods differ mainly in their objectivesto guide the data synthesis.
Revised: Existing DFKD methods differ mainly in their objectives toguide the data synthesis.

Original: These methods generally fall into either the adversarial camp or thenon-adversarial camp.
Revised: These methods generally fall into either the adversarial camp or othercamp.

Original: Non-adversarial DFKD methods [7, 27, 29, 34, 44, 46] make use of certainheuristics to search for synthetic data that resembles the original training data D T , train .
Revised: Non-adversarial DFKD methods [7, 26, 28, 32, 42, 44] make use of certain heuristics tosearch for synthetic data that resembles the original training data D T , train .

Original: For example, ZSKD [34] and DAFL [7] consider prediction probabilities of T and the conﬁdence of T as heuristics.
Revised: For example, ZSKD[32] and DAFL [7] consider prediction probabilities of T and the conﬁdence of T as heuristics.

Original: In DAFL, synthetic data is generated via a generator G rather than being optimized directly like in
Revised: In DAFL, synthetic data is generated via a generator G rather than being optimized directly like in

Original: ZSKD.
Revised: ZSKD.

Original: KegNet [46] extends DAFL by making G conditioned on class labels.
Revised: KegNet [44] extends DAFL by making G conditioned on class labels.

Original: Adversarial DFKDmethods [8, 14, 17, 37, 45, 50] leverage adversarial learning to explore the data space more efﬁciently.
Revised: Adversarial DFKDmethods [8, 13, 16, 35, 43, 48] leverage adversarial learning to explore the data space more efﬁciently.

Original: Most methods of this type are derived from the ABM [31] discussed in Section 2 with additionalobjectives to improve the quality and/or diversity of synthetic data.
Revised: Most methods of this type are derived from the ABM [30] discussed in Section 2 with additionalobjectives to improve the quality and/or diversity of synthetic data.

Original: For example, RDSKD [17] usesa diversity-seeking loss [30]; CMI [14] uses an inverse contrastive loss to improve diversity; and
Revised: For example, RDSKD [16] usesa diversity-seeking loss [29]; CMI [13] uses an inverse contrastive loss to improve diversity; and

Original: DeepInversion [45] uses Batch Norm moment matching (BNmm) and DeepDream’s inception losses(total variation, L2)
Revised: DeepInversion [43] uses Batch Norm moment matching (BNmm) and DeepDream’s inception losses(total variation, L2)

Original: [32] to generate visually interpretable images.
Revised: [31] to generate visually interpretable images.

Original: DeepInversion [45] is one of a few DFKD methods [12, 29, 45] that have been shown to learn wellon the full-size ImageNet dataset.
Revised: DeepInversion [43] is one of a few DFKD methods [11, 28, 43] that have been shown to learn wellon the full-size ImageNet dataset.

Original: Its success has inspired subsequent works on large-scale continuallearning [39] and data-free object detection [6].
Revised: Its success has inspired subsequent works on large-scale continuallearning [37] and data-free object detection [6].

Original: However, training DeepInversion on ImageNet isvery time-consuming because this method optimizes synthetic images directly and each completeoptimization requires a very large number of iterations (about 20,000).
Revised: However, training DeepInversion on ImageNet isvery time-consuming because this method optimizes synthetic images directly and each completeoptimization requires a very large number of iterations (about 20,000).

Original: LS-GDFD [29] learns togenerate synthetic data instead but requires one generator for each class (that is, 1,000 generatorsfor 1,000 classes) to avoid the “mode collapse” problem and achieve good performance.
Revised: LS-GDFD [28] learns togenerate synthetic data instead but requires one generator for each class (that is, 1,000 generatorsfor 1,000 classes) to avoid the “mode collapse” problem and achieve good performance.

Original: FastDFKD[12] leverages meta-learning to speed up the training process signiﬁcantly.
Revised: FastDFKD[11] leverages meta-learning to speed up the training process signiﬁcantly.

Original: On the contrary, byapplying our proposed technique in Section 3.2, our MAD only needs 1 generator (compared to1,000 of LS-GDFD) and 6,000 training iterations (compared to 20,000 × of DeepInversion) to achievereasonably good performance on ImageNet (Section 5.2.2).
Revised: On the contrary, byapplying our proposed technique in Section 3.2, our MAD only needs 1 generator (compared to1,000 of LS-GDFD) and 6,000 training iterations (compared to 20,000 × of DeepInversion) to achievereasonably good performance on ImageNet (Section 5.2.2).

Original: In addition, our technique is orthogonalto and can combine with the meta-learning idea in [12] for further improvement.
Revised: In addition, our technique is orthogonalto and can combine with the meta-learning idea in [11] for further improvement.

Original: Besides the DFKD methods discussed above, there is a line of works that consider a less extremescenario in which unlabeled transfer sets are given for training.

Original: These transfer sets can be verydifferent from the original dataset in terms of distribution and semantics and can be freely collectedfrom open resources.

Original: Nayak et al.

Original: [33] analyzed various kinds of transfer sets ranging from randomnoise images to natural images and found that the target-class balance property of the transfer set andthe similarity between the transfer set and the original training set play important roles in improvingknowledge distillation results.

Original: Fang et al.

Original: [11] argued that although the transfer images can besemantically different from the original traning images, they still share common local visual patterns.

Original: Therefore, the authors proposed an interesting method called MosaicKD which combines differentlocal patches extracted from the transfer images to craft synthetic mosaic images that capture thesemantics of the original data while enjoys realistic local structures.

Original: Adversarial trainingmethods like  GANs has been shown to suffer from the catastrophic forgettingproblem of the discriminator [26, 40].
Revised: Adversarial training like those in GANs has been shown to suffer from the catastrophic forgettingproblem in the discriminator [25, 38].

Original: These works consider GANs training as a continual learningproblem and leverage existing methods to mitigate the issue.
Revised: These works consider GANs training as a continual learningproblem and leverage existing methods to mitigate the issue.

Original: For example, Liang et al.
Revised: For example, Liang et al.

Original: [26] use either
Revised: [25] use either

Original: EWC [23] or SI [49] to enforce similarity between the discriminator’s weights at the current stepand at the last checkpoint.
Revised: EWC [22] or SI [47] to enforce similarity between the discriminator’s weights at the current stepand at the last checkpoint.

Original: Hoang et al.
Revised: Hoang et al.

Original: [40] suggest some other tricks such as using a momentumoptimizer (e.g., Adam [21]) or applying gradient penalty [16] to the discriminator.
Revised: [38] suggest some other tricks such as using a momentumoptimizer (e.g., Adam [20]) or applying gradient penalty [15] to the discriminator.

Original: In this paper, weconsider the catastrophic forgetting problem of the student as a consequence of the large distributionshift problem caused by the generator.
Revised: In this paper, weconsider the catastrophic forgetting problem of the student as a consequence of the large distributionshift problem caused by the generator.

Original: Therefore, we focus on regularizing the generator rather thanthe student.
Revised: Therefore, we focus on regularizing the generator rather thanthe student.

Original: Besides, from this perspective, the catastrophic forgetting problem in GANs is indeedless severe than that in adversarial DFKD.
Revised: Besides, from this perspective, the catastrophic forgetting problem in GANs is indeedless severe than that in adversarial DFKD.

Original: It is because in GANs, the discriminator is trained onreal data with a ﬁxed distribution, which can somewhat reduce the effect of the distribution shift offake (synthetic) data while in adversarial DFKD, real data is not available.
Revised: It is because in GANs, the discriminator is trained onreal data with a ﬁxed distribution, which can somewhat reduce the effect of the distribution shift offake (synthetic) data while in adversarial DFKD, real data is not available.

Original: Recent works that alsoaddress the student forgetting problem in DFKD like ours include DFKD-Mem [3] and PRE-DFKD[2].
Revised: Recent works that alsoaddress the student forgetting problem in DFKD like ours include DFKD-Mem [3] and PRE-DFKD[2].

Original: DFKD-Mem [3] stores the past synthetic samples in a memory bank and uses these samples asadditional training data for the student.
Revised: DFKD-Mem [3] stores the past synthetic samples in a memory bank and uses these samples asadditional training data for the student.

Original: Our method, on the other hand, uses an EMA generator togenerate old synthetic samples on-the-ﬂy, which is more memory efﬁcient and adapts better to thestudent update.
Revised: Our method, on the other hand, uses an EMA generator togenerate old synthetic samples on-the-ﬂy, which is more memory efﬁcient and adapts better to thestudent update.

Original: PRE-DFKD [2] models past synthetic data via a VAE [22] and treats the decoderof this VAE as a replay generator.
Revised: PRE-DFKD [2] models past synthetic data via a VAE [21] and treats the decoderof this VAE as a replay generator.

Original: However, training a VAE on a continuous stream of syntheticsamples could be unstable and could lead to another catastrophic forgetting problem on its own.
Revised: However, training a VAE on a continuous stream of syntheticsamples could be unstable and could lead to another catastrophic forgetting problem on its own.

Original: Inaddition, VAE is not effective for generating images with large size (e.g., ImageNet images) due tothe “posterior collapse” problem [28].
Revised: Inaddition, VAE is not effective for generating images with large size (e.g., ImageNet images) due tothe “posterior collapse” problem [27].

Original:  Experiments
Revised:  Experiments

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: Datasets We consider the image classiﬁcation task and evaluate our proposed method on 3 smallimage datasets (CIFAR10 [24], CIFAR100 [24], TinyImageNet [25]), and 3 large image datasets(ImageNet [9], Places365 [51], Food101 [4]).
Revised: Datasets We consider the image classiﬁcation task and evaluate our proposed method on 3 smallimage datasets (CIFAR10 [23], CIFAR100 [23], TinyImageNet [24]), and 3 large image datasets(ImageNet [9], Places365 [49], Food101 [4]).

Original: Details are provided in Appdx.
Revised: Details are provided in Appdx.

Original: A.1.
Revised: A.1.

Original: [Table] Network architectures For the small datasets, we follow [8, 14] and consider ResNet34/ResNet18[18] and WRN40-2/WRN16-2 [48] for the teacher/student.
Revised: Network architectures For the small datasets, we follow [8, 13] and consider ResNet34/ResNet18[17] and WRN40-2/WRN16-2 [46] for the teacher/student.

Original: We observed that knowledge distillationwith the WRN architectures is more challenging than with the ResNet architectures, possibly because
Revised: We observed that knowledge distillationwith the WRN architectures is more challenging than with the ResNet architectures, possibly because

Original: WRN40-2/WRN16-2 have much fewer parameters than ResNet34/ResNet18.
Revised: WRN40-2/WRN16-2 have much fewer parameters than ResNet34/ResNet18.

Original: For the large datasets,we use the AlexNet architecture for both the teacher and student for fast training.
Revised: For the large datasets,we use the AlexNet architecture for both the teacher and student for fast training.

Original: Since AlexNet doesnot have any BatchNorm layer, we exclude the BNmm loss (Eq. 9) from the total loss of the generatorin our experiments on the large datasets.
Revised: Since AlexNet doesnot have any BatchNorm layer, we exclude the BNmm loss (Eq. 9) from the total loss of the generatorin our experiments on the large datasets.

Original: Architectures of the generator w.r.t.
Revised: Architectures of the generator w.r.t.

Original: different datasets aregiven in Appdx.
Revised: different datasets aregiven in Appdx.

Original: A.4.
Revised: A.4.

Revised: [Table]

Original: Training settings of teacher
Revised: Training settings of teacher

Original: For ImageNet, we make use of pretrained networks provided by
Revised: For ImageNet, we make use of pretrained networks provided by

Original: PyTorch.
Revised: PyTorch.

Original: For other datasets, we train the teacher from scratch.
Revised: For other datasets, we train the teacher from scratch.

Original: Detailed training settings of theteacher are given in Appdx.
Revised: Detailed training settings of theteacher are given in Appdx.

Original: A.2.
Revised: A.2.

Original: Training settings of MAD If not otherwise speciﬁed, we set the momentum α in Eq.
Revised: Training settings of MAD If not otherwise speciﬁed, we set the momentum α in Eq.

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 95 andthe length of the noise vector to 256.
Revised: 95 andthe length of the noise vector to 256.

Original: We train the student S using SGD and Adam for the small andlarge datasets, respectively.
Revised: We train the student S using SGD and Adam for the small andlarge datasets, respectively.

Original: We train the generator G using Adam for both the small and large datasets.
Revised: We train the generator G using Adam for both the small and large datasets.

Original: To reduce the difﬁculty of training G for the large datasets, we pretrain G for some steps before themain KD training by setting the coefﬁcient of L KD to 0 and only optimizing the remaining losses in
Revised: To reduce the difﬁculty of training G for the large datasets, we pretrain G for some steps before themain KD training by setting the coefﬁcient of L KD to 0 and only optimizing the remaining losses in

Original: Eq. 6.
Revised: Eq. 6.

Original: The generator G is class-conditional (as described in Section 3.2) for the large datasets, andunconditional for the small datasets.
Revised: The generator G is class-conditional (as described in Section 3.2) for the large datasets, andunconditional for the small datasets.

Original: We empirically found that this leads to better results.
Revised: We empirically found that this leads to better results.

Original: For furtherdetails about the training settings of MAD, please refer to Appdx.
Revised: For furtherdetails about the training settings of MAD, please refer to Appdx.

Original: A.3.
Revised: A.3.

Original: 5.2 Results
Revised: 5.2 Results

Original: 5.2.1 Comparison with existing DFKD methods
Revised: 5.2.1 Comparison with existing DFKD methods

Original: In Table 1, we compare MAD with existing DFKD methods [7, 8,  13, 14, 34, 45, 50] on the smalldatasets.
Revised: In Table 1, we compare MAD with existing DFKD methods [7, 8, 12, 13, 32, 43, 48] on the smalldatasets.

Original: The results of baselines are taken from [14, 50].
Revised: The results of baselines are taken from [13, 48].

Original: We checked the results of the teachertrained by us and found that they are quite similar to the results of the teacher reported in [14] and in
Revised: We checked the results of the teachertrained by us and found that they are quite similar to the results of the teacher reported in [13] and in

Original: Table 1 (details in Appdx.
Revised: Table 1 (details in Appdx.

Original: B.1).
Revised: B.1).

Original: This means our results of MAD are comparable with those of thebaselines.
Revised: This means our results of MAD are comparable with those of thebaselines.

Original: We see that MAD outperforms all the DFKD methods on CIFAR10 and CIFAR100 in casethe teacher and student are ResNets [18] ( ♥ ).
Revised: We see that MAD outperforms all the DFKD methods on CIFAR10 and CIFAR100 in casethe teacher and student are ResNets [17] ( ♥ ).

Original: In case the teacher and student are WideResNets [48]( ♦ ), our method still achieves signiﬁcantly better results than most of the baselines such as DAFL[7] and ZSKT [34] and only performs worse than CMI [14].
Revised: In case the teacher and student are WideResNets [46]( ♦ ), our method still achieves signiﬁcantly better results than most of the baselines such as DAFL[7] and ZSKT [32] and only performs worse than CMI [13].

Original: We hypothesize it is mainly because ofthe differences in model design and training settings of CMI and MAD.
Revised: We hypothesize it is mainly because ofthe differences in model design and training settings of CMI and MAD.

Original: For example, CMI has anadditional contrastive-loss-based objective that encourages the generator’s diversity while MAD doesnot.
Revised: For example, CMI has anadditional contrastive-loss-based objective that encourages the generator’s diversity while MAD doesnot.

Original: CMI also performs more updates of the student and generator per training step than our method 1 .
Revised: CMI also performs more updates of the student and generator per training step than our method 1 .

Original: However, our ultimate goal for this section is not to show that MAD can achieve state-of-the-artresults in all cases (which often requires intensive hyper-parameter tuning) but to verify the soundnessof our implementation of MAD.
Revised: However, our ultimate goal for this section is not to show that MAD can achieve state-of-the-artresults in all cases (which often requires intensive hyper-parameter tuning) but to verify the soundnessof our implementation of MAD.

Original: In order to see clearly the advantage of MAD, we need to compare
Revised: In order to see clearly the advantage of MAD, we need to compare

Original: MAD with its related baselines under the same settings .
Revised: MAD with its related baselines under the same settings .

Original: This will be presented in Section 5.2.2.
Revised: This will be presented in Section 5.2.2.

Revised: 5.2.2 Comparison with related baselines

Revised: We consider two related baselines of MAD which are ABM [30] and DFKD-Mem [3].

Revised: ABM learnsthe student S with only synthetic samples from G .

Revised: DFKD-Mem, on the other hand, stores pastsynthetic samples in a memory bank, and uses samples from this memory bank (dubbed “memorysamples”) as well as those generated by G as training data for S .

Revised: We trained ABM and DFKD-Memusing exactly the same settings for training MAD.

Original:  Dataset
Revised: For DFKD-Mem, we set the memory size to 8,192. For other memory sizes, the results remain relatively similar as shown in Appdx. B.2. Dataset

Original: CIFAR CIFAR
Revised: CIFAR CIFAR

Original: TinyIN
Revised: TinyIN

Original: ImageNet
Revised: ImageNet

Original: Places
Revised: Places

Original: Food
Revised: Food

Original: Arch.
Revised: Arch.

Original: ♦ ♦ ♥ ♣ ♣ ♣
Revised: ♦ ♦ ♥ ♣ ♣ ♣

Original: Teacher 94.
Revised: Teacher 94.

Original: ABM 92.
Revised: ABM 92.

Original: DFKD-Mem 92.
Revised: DFKD-Mem 92.

Original: MAD 92.
Revised: MAD 92.

Original: [Figure]
Revised: [Figure]

Original: 5.2.2 Comparison with related baselines

Original: We consider two related baselines of MAD which are ABM [31] and DFKD-Mem [3].

Original: ABM learnsthe student S with only synthetic samples from G .

Original: DFKD-Mem, on the other hand, stores pastsynthetic samples in a memory bank, and uses samples from this memory bank (dubbed “memorysamples”) as well as those generated by G as training data for S .

Original: We trained ABM and DFKD-Memusing exactly the same settings for training MAD.

Original: For DFKD-Mem, we set the memory size to 8,192.

Original: For other memory sizes, the results remain relatively similar as shown in Appdx.

Original: B.2. From Table 2 and Fig.
Revised: From Table 2 and Fig.

Original: 2, it is clear that MAD signiﬁcantly outperforms both ABM and DFKD-Memon all datasets.
Revised: 2, it is clear that MAD signiﬁcantly outperforms both ABM and DFKD-Memon all datasets.

Original: In addition, the performance gaps between our method and the two baselines tend tobe larger for larger datasets.
Revised: In addition, the performance gaps between our method and the two baselines tend tobe larger for larger datasets.

Original: For example, MAD achieves about 1.5/2.8%, 2.5/3.6%, and 4.2/2.2%higher accuracy than ABM/DFKD-Mem on CIFAR100, TinyImageNet, and ImageNet, respectively.
Revised: For example, MAD achieves about 1.5/2.8%, 2.5/3.6%, and 4.2/2.2%higher accuracy than ABM/DFKD-Mem on CIFAR100, TinyImageNet, and ImageNet, respectively.

Original: These empirical results suggest the importance of the EMA generator ˜ G in mitigating the largedistribution shift caused by G .
Revised: These empirical results suggest the importance of the EMA generator ˜ G in mitigating the largedistribution shift caused by G .

Original: In this experiment, we found that DFKD-Mem often performs worse than ABM on CIFAR100 and
Revised: In this experiment, we found that DFKD-Mem often performs worse than ABM on CIFAR100 and

Original: TinyImageNet.
Revised: TinyImageNet.

Original: We found the decay of the student learning rate is the main reason for this.
Revised: We found the decay of the student learning rate was the main reason for this.

Original: Asshown in Fig.
Revised: Asshown in Fig.

Original: 3a, the distillation loss on memory samples surges when the (student) learning rate isdecayed and cannot recover if the new learning rate is too small ( lr S = 1e-4), which is in contrastto the distillation loss on synthetic samples from G or ˜ G (Figs. 3b,3c).
Revised: 3a, the distillation loss on memory samples surges when the (student) learning rate isdecayed and cannot recover if the new learning rate is too small ( lr S = 1e-4), which is in contrastto the distillation loss on synthetic samples from G or ˜ G (Figs. 3b,3c).

Original: This implies a potential issueof storing old samples in a memory bank instead of using the EMA generator as memory samplescould be completely out-of-date if the student suddenly change its state (e.g., via learning rate decay).
Revised: This implies a potential issueof storing old samples in a memory bank instead of using the EMA generator as memory samplescould be completely out-of-date if the student suddenly change its state (e.g., via learning rate decay).

Original: However, even when the learning rate does not change (e.g., from step 0 to step 100 on CIFAR100 oron ImageNet/Places365), DFKD-Mem still performs worse than our method. 
Revised: However, even when the learning rate does not change (e.g., from step 0 to step 100 on CIFAR100 oron ImageNet/Places365), DFKD-Mem still performs worse than our method. [Figure] [Figure] [Figure]

Original: 5.2.3 Comparing the changes in update of ˜ G and G
Revised: 5.2.3 Comparing the changes in update of ˜ G and G

Original: In order to see whether ˜ G actually has smaller changes in update than G or not, we perform thefollowing experiment: Let G t and ˜ G t be the versions of the generator and the EMA generatorrespectively at step t , and S t − τ be the version of the student at step t − τ ( 0 < τ < t ).
Revised: In order to see whether ˜ G actually has smaller changes in update than G or not, we perform thefollowing experiment: Let G t and ˜ G t be the versions of the generator and the EMA generatorrespectively at step t , and S t − τ be the version of the student at step t − τ ( 0 < τ < t ).

Original: We thenmeasure two different average Jensen-Shannon (JS) divergences between the prediction probabilitiesof S t − τ and T on two separate sets of synthetic samples from G t and ˜ G t .
Revised: We thenmeasure two different average Jensen-Shannon (JS) divergences between the prediction probabilitiesof S t − τ and T on two separate sets of synthetic samples from G t and ˜ G t .

Original: We hypothesize that if G hasa smaller change in the distribution of synthetic samples than G , S will memorize the samples from˜ G more and will match T better on those samples, which leads to a smaller average JS divergence.
Revised: We hypothesize that if G hasa smaller change in the distribution of synthetic samples than G , S will memorize the samples from˜ G more and will match T better on those samples, which leads to a smaller average JS divergence.

Original: This hypothesis is clearly reﬂected on results in Fig.
Revised: This hypothesis is clearly reﬂected on results in Fig.

Original: 4, which veriﬁes the reasonability of using ˜ G inalleviating the large distribution shift problem caused by G .[Figure] [Figure] v
Revised: 4, which veriﬁes the reasonability of using ˜ G inalleviating the large distribution shift problem caused by G .

Original: [Figure] r g [Figure] 5.2.4 Visualization of synthetic samples
Revised: 5.2.4 Visualization of synthetic samples

Original: In Fig.
Revised: In Fig.

Original: 5, we visualize the synthetic data generated by G and ˜ G .
Revised: 5, we visualize the synthetic data generated by G and ˜ G .

Original: Although the generated images are notvisually realistic, they are visually diverse, suggesting no mode collapse has occurred during training
Revised: Although the generated images are notvisually realistic, they are visually diverse, suggesting no mode collapse has occurred during training

Original: MAD.
Revised: MAD.

Original: Besides, samples generated by ˜ G are different from those generated by G which indicates thatthe EMA generator could act as a complement for the generator in our model.
Revised: Besides, samples generated by ˜ G are different from those generated by G which indicates thatthe EMA generator could act as a complement for the generator in our model.

Original: 5.3 Sensitivity Analysis
Revised: 5.3 Sensitivity Analysis

Original: Below we investigate some choices of hyperparameters that could affect the performance of MAD.
Revised: Below we investigate some choices of hyperparameters that could affect the performance of MAD.

Original: Unless stated otherwise, the dataset we use is CIFAR100.[Figure] [Figure] [Figure]
Revised: Unless stated otherwise, the dataset we use is CIFAR100.

Original: λ 0 1.

Original: λ

Original: [Figure]

Original: [Figure] [Figure] [Figure] Different values of the coefﬁcients in L S
Revised: Different values of the coefﬁcients in L S

Original: We can control the relative importance of samples from˜ G over those from G by changing the values of the two coefﬁcients λ 0 and λ 1 in L S (Eq. 4).
Revised: We can control the relative importance of samples from˜ G over those from G by changing the values of the two coefﬁcients λ 0 and λ 1 in L S (Eq. 4).

Original: As shownin Fig.
Revised: As shownin Fig.

Original: 6 (right), the best result is obtained when λ 0 = λ 1 = 1 which is our default setting for MAD.
Revised: 6 (right), the best result is obtained when λ 0 = λ 1 = 1 which is our default setting for MAD.

Original: Decreasing either λ 0 or λ 1 will lead to worse performance.
Revised: Decreasing either λ 0 or λ 1 will lead to worse performance.

Original: In the worst case when λ 0 = 0 andλ 1 = 1 , the model learns for some epochs and then suddenly stops learning.
Revised: In the worst case when λ 0 = 0 and [Figure] λ 1 = 1 , the model learns for some epochs and then suddenly stops learning.

Original: The main reason is that˜ G often updates much slower than G and S
Revised: The main reason is that˜ G often updates much slower than G and S

Original: so we need adversarial samples from G to keep S learning.
Revised: so we need adversarial samples from G to keep S learning.

Original: Otherwise, S will overﬁt the samples from ˜ G and learn nothing.
Revised: Otherwise, S will overﬁt the samples from ˜ G and learn nothing.

Original: Different values of the momentum α
Revised: Different values of the momentum α

Original: In Fig.
Revised: In Fig.

Original: 6 (middle)
Revised: 6 (middle)

Original: , we show the classiﬁcation accuracyof the student of MAD with α in {0.2, 0.4, 0.6, 0.8, 0.95, 0.99, 0.999, 1.0}.
Revised: , we show the classiﬁcation accuracyof the student of MAD with α in {0.2, 0.4, 0.6, 0.8, 0.95, 0.99, 0.999, 1.0}.

Original: We observe that theperformance of our method increases when α becomes larger as ˜ G is more different from G .
Revised: We observe that theperformance of our method increases when α becomes larger as ˜ G is more different from G .

Original: However,if α is too large (e.g., 0.999), the performance drops since the update of G is very small, resultingin almost no change in the distribution of synthetic samples.
Revised: However,if α is too large (e.g., 0.999), the performance drops since the update of G is very small, resultingin almost no change in the distribution of synthetic samples.

Original: In case α = 1 . 0 , MAD only achievesabout 50% prediction accuracy on test data.
Revised: In case α = 1 . 0 , MAD only achievesabout 50% prediction accuracy on test data.

Original: Different update steps of S From Fig.
Revised: Different update steps of S From Fig.

Original: 6 (left), we see that increasing the number of update stepsfor S usually leads to better performances of MAD since S learns to match T better.
Revised: 6 (left), we see that increasing the number of update stepsfor S usually leads to better performances of MAD since S learns to match T better.

Original: In exchange, thetraining time will increase.
Revised: In exchange, thetraining time will increase.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented Momentum Adversarial Distillation (MAD), a simple yet effective method to dealwith the large distribution shift problem in adversarial Data-Free Knowledge Distillation (DFKD).
Revised: We have presented Momentum Adversarial Distillation (MAD), a simple yet effective method to dealwith the large distribution shift problem in adversarial Data-Free Knowledge Distillation (DFKD).

Original: MAD maintains an exponential moving average (EMA) copy of the generator which, by design,encapsulates information about past updates of the generator and is updated at a slower pace than thegenerator.
Revised: MAD maintains an exponential moving average (EMA) copy of the generator which, by design,encapsulates information about past updates of the generator and is updated at a slower pace than thegenerator.

Original: By training the student on samples from both the generator and the EMA generator, MADcan prevent the student from adapting too much to the generator at the current step and forgetting oldknowledge learned from the generator at the previous steps.
Revised: By training the student on samples from both the generator and the EMA generator, MADcan prevent the student from adapting too much to the generator at the current step and forgetting oldknowledge learned from the generator at the previous steps.

Original: We have also described a new type ofconditional generator along with a new loss for training it which enable our model to learn well onlarge datasets.
Revised: We have also described a new type ofconditional generator along with a new loss for training it which enable our model to learn well onlarge datasets.

Original: Our experiments on various datasets demonstrated the superior performance of ourmethod over competing baselines that either use only a generator or use a memory bank in place ofan EMA generator.
Revised: Our experiments on various datasets demonstrated the superior performance of ourmethod over competing baselines that either use only a generator or use a memory bank in place ofan EMA generator.

Original: We note that our idea of using an EMA generator to mitigate large distribution shifts is general andcan be generalized to other machine learning problems besides DFKD.
Revised: We note that our idea of using an EMA generator to mitigate large distribution shifts is general andcan be generalized to other machine learning problems besides DFKD.

Original: For example, the techniquecan be adapted for general continual learning (in which large distribution shifts can happen graduallyor suddenly with no clear task boundaries), and source-data-free domain adaptation.
Revised: For example, the techniquecan be adapted for general continual learning (in which large distribution shifts can happen graduallyor suddenly with no clear task boundaries), and source-data-free domain adaptation.

Original: Our method isalso well suited for DFKD with other data types such as video, text, or graph.
Revised: Our method isalso well suited for DFKD with other data types such as video, text, or graph.

Original: Limitations In our current implementation of MAD, we perform an additional forward pass through˜ G for every training step of S .
Revised: Limitations In our current implementation of MAD, we perform an additional forward pass through˜ G for every training step of S .

Original: This increases the total training time of by about 40% compared to
Revised: This increases the total training time of by about 40% compared to

Original: ABM.
Revised: ABM.

Original: However, this technical problem can be somewhat addressed by ﬁrst storing synthetic samplesfrom G and ˜ G in a buffer before each training stage of S and then training S with samples from thebuffer only.
Revised: However, this technical problem can be somewhat addressed by ﬁrst storing synthetic samplesfrom G and ˜ G in a buffer before each training stage of S and then training S with samples from thebuffer only.

Original: This will be left for future work.
Revised: This will be left for future work.

Original: Negative Social Impacts
Revised: Negative Social Impacts

Original: The DFKD problem may have negative social impacts related to dataprivacy as generated data could somehow reveal the original training data.
Revised: The DFKD problem may have negative social impacts related to dataprivacy as generated data could somehow reveal the original training data.

Original: However, as shown in
Revised: However, as shown in

Original: Fig.
Revised: Fig.

Original: 5, our proposed method does not attempt to improve the visual interpretability of synthetic databut addresses the large distribution shift problem.
Revised: 5, our proposed method does not attempt to improve the visual interpretability of synthetic databut addresses the large distribution shift problem.

Original: This target seems to be harmless to the society.
Revised: This target seems to be harmless to the society.

Original:  Checklist
Revised:  Checklist

Original: For all authors...
Revised: For all authors...

Original: • Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: • Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: • Did you describe the limitations of your work?
Revised: • Did you describe the limitations of your work?

Original: [Yes] See Section • Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] See Section 6• Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSection 6• Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] SeeSection 6• Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: • Did you state the full set of assumptions of all theoretical results?
Revised: • Did you state the full set of assumptions of all theoretical results?

Original: [N/A] • Did you include complete proofs of all theoretical results?
Revised: [N/A]• Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments... • Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments...• Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Experiment and Appendix • Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] See Experimentand Appendix• Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix • Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Appendix• Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Because running multiple times is very time-consuming
Revised: [No] Because running multiple times is very time-consuming

Original: and unnecessary as we observed small standard deviations on most datasets (e.g., onImageNet, the standard error of 3 different runs of MAD is about 0.3% while the gapsin performance between MAD and baselines are about 2-4%)
Revised: and unnecessary as we observed small standard deviations on most datasets (e.g., onImageNet, the standard error of 3 different runs of MAD is about 0.3% while the gaps in performance between MAD and baselines are about 2-4%)

Original: • Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: • Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] They are provided in Appdx.
Revised: [Yes] They are provided in Appdx.

Original: A.34.
Revised: A.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...• If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... • If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: • Did you mention the license of the assets?
Revised: • Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: • Did you include any new assets either in the supplemental material or as a URL?
Revised: • Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: • Did you discuss whether
Revised: • Did you discuss whether

Original: and how consent was obtained from people whose data you’re
Revised: and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: • Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: • Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: • Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: • Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: • Did you describe any potential participant risks, with links to Institutional Review
Revised: • Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: • Did you include the estimated hourly wage paid to participants and the total amount
Revised: • Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Experimental SetupA.1 Datasets

Original: In Table 3, we provide information about the image size, the number of classes, and the number oftraining/test samples of the datasets used in our experiment.

Original: [Table]

Original: A.2 Training Settings of Teacher

Original: We provide training settings of the teacher w.r.t.

Original: different datasets in Table 4.

Original: [Table]

Original: A.3 Training Settings of MAD

Original: In Tables 5,6,7, we provide the training settings of MAD used in this paper.

Original: Despite multiple attempts,we could not ﬁnd a global conﬁguration that works well for all datasets and architectures.

Original: In practice, we do not optimize the student and the generator via the plain losses in Eq.

Original: 4 and Eq.

Original: 6,respectively but with some additional regularizations on the output logits of T , S and G .

Original: This preventsour losses from being NaN when the logits grow too big.

Original: Speciﬁcally, we deﬁne L S as follows:

Original: [Equation]

Original: [Equation]

Original: where max( | S ( · ) | − δ, 0) ensures that the output logit of S is between [ − δ, δ ] ; ζ 0 ≥ 0 is a coefﬁcient.

Original: And we deﬁne L G , E as follows:

Original: [Equation]

Original: where G lg denotes the generator that produces logits instead of normalized images; σ ( · ) denotesthe sigmoid function; max( | u | − ν, 0) ensures that the output logit u of G lg is between [ − ν, ν ] ;max( | T ( x ) | − δ, 0) ensures that the output logit of T w.r.t.

Original: the synthetic sample x is between [ − δ, δ ] ;ζ 1 , ζ 2 ≥ 0 are coefﬁcients.

Original: We train MAD on multiple NVIDIA A100-SXM2-32GB and A100-SXM4-40GB GPUs.

Original: Due to theuse of different teacher/student architectures, the use of GPUs with different numbers and types, andthe share of computational resources, it is hard to compute exactly the training time of our method butroughly it took about 1-2 days, 3-4 days, and 4-6 days to train MAD on CIFAR10/100, TinyImageNet,and ImageNet/Places365/Food101, respectively.

Original: Dataset

Original: Arch.

Original: [Table]

Original: [Table]

Original: The teacher/student architecture settings are ResNet34/ResNet18 ( ♥ ), WRN40-2/WRN16-2 (

Original: , and

Original: [Table]

Original: [Table]

Original: A.4 Generator Architectures

Original: In Table

Original: [Table]

Original: we show different architectures of the generator w.r

Original: [Table]

Original: t. different image sizes.

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: B Additional Experimental Results

Original: B.1 Results of Teacher

Original: reports the results of our teacher on all the benchmark datasets.

Original: On the small datasets, ourteacher achieves very similar performance compared to the one in [14].

Original: [Table]

Original: In [14]

Original: [Table]

Original: B.2 Results of DFKD-Mem with Different Memory Sizes

Original: In Fig.

Original: 7, we show the classiﬁcation results of DFKD-Mem with different memory sizes on CIFAR100and ImageNet.

Original: On CIFAR100, DFKD-Mem achieves the best result at memory size = 2048 but stillunderperforms ABM and MAD.

Original: On ImageNet, the performance of DFKD-Mem is proportional tothe memory size and is highest at memory size = 8192.

Original: This result, however, is still worse than thatof MAD.

Original: Figs.

Original: 3a,3b display the average distillation loss (avg L KD ) curves of DFKD-Mem w.r.t.different memory sizes.

Original: We see that increasing the memory size increases the avg L KD on memorysamples but does not affect the avg L KD on samples from G .

Original: It is because the avg L KD on memorysamples is very small compared to the counterpart on samples from G .

Original: [Figure]

Original: [Figure]

Original: B.3 Empirical Analysis of Different Types of Generators

Original: In Fig.

Original: [Figure]

Original: we show the results of MAD on ImageNet with three different types of generators whichare unconditional (“uncond”), conditional-via-concatenation (“cat”), conditional-via-summation(“sum”).

Original: MAD with the “uncond” generator eventually collapses during training but not with the“cat” or the “sum” generators (Fig. 9a).

Original: This is because the “uncond” generator has learned to jumpbetween different spurious solutions as visualized in Fig.

Original: Among all types of generators, the“sum” generator enables stable training of our model and gives the best accuracy and crossentropyon D test (Figs. 9a,b).

Original: The “cat” generator only yields good results at λ 3 = 0 .

Original: 3 ( λ 3 is the coefﬁcientof L NLL in Eq. 6).

Original: The reason is that if λ 3 is too small (e.g., 0.1), L NLL will be high (Fig. 9g) andspurious solutions of G cannot be suppressed.

Original: G will jump between these solutions, leading to highvariance when maximizing L KD (Fig. 9f).

Original: By contrast, if λ 3 is too big (e.g., 3.0, 10.0), G will beoptimized towards predicting y correctly (small L NLL as shown in Fig.

Original: 9g) rather than generating goodadversarial samples for knowledge transfer from T to S (small L KD as shown in Fig. 9f).

Original: This causes

Original: S to achieve tiny L KD (Fig. 9e) and match T very well (Fig. 9d) on samples from G but generalizespoorly to unseen sample from D test (Fig. 9a).

Original: However, for any value of λ 3 , MAD with the “cat”generator performs worse than the counterpart with the “sum” generator, and even worse than thecounterpart with the “uncond” generator during early epochs of training (Fig. 9a).

Original: To explain thisphenomenon, we ﬁrst provide the formulas of the ﬁrst layers of the three kinds of generators belowas these generators are only different in the ﬁrst layer:

Original: [Equation]

Original: where W , U , b are trainable weights and bias.

Original: We hypothesize that due to the stochasticity of zsampled from a ﬁxed distribution, W tends to be robust to changes.

Original: And since the “sum” generatoruses W to transform e y , its output will not be affected much by the update of e y .

Original: In other words, the

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: noise in updating e y is absorbed into the stochasticity of z via summation.

Original: The “cat” generator, onthe other hand, uses a different weight matrix U to transform e y .

Original: Since the update of U only dependson the current version of e y and vice versa, and e y can change arbitrarily, updating both U and e ysimultaneously in case of the “cat” generator can lead to unstable and nonoptimal 2 training.

Original: The“uncond” generator does not encounter any problem with e y like the “cat” generator so it can enable

Original: MAD to learn faster than the “cat” generator.

Original: C Derivation of L NormReg in Section 3.2

Original: Recall that in our design of the class-conditional generator, we use K trainable class embeddingvectors e 1 , ..., e K to represent K classes in the training data.

Original: These embedding vectors can be regardedas the centers of K Gaussian distributions (or clusters) N ( e k , I) ( k = 1 , ..., K ) corresponding to

Original: K classes and are optimized together with the generator G via Eq.

Original: To prevent these embeddingvectors from changing arbitrarily, we need to constraint their norms to be smaller than a threshold byminimizing the loss L NormReg in Eq. 8.

Original: An important question is “What is a reasonable upper boundfor the norm of each embedding vector e k ?”.

Original: Let ξ denote the upper bound for the norm of e k .

Original: By constraining (cid:107) e k (cid:107) 2 to be smaller than ξ , weensure that e k is inside a hyperball of radius ξ .

Original: Intuitively, we should choose ξ so that the K Gaussianclusters won’t overlap each other.

Original: Note that in high dimensional space, we can generally treat each

Original: Gaussian cluster N ( e k , I) as a hypersphere of radius √ d e centered at e k ( d e = dim( e k ) ).

Original: One simpleway to allow these K hyperspheres not to overlap each other when their centers are inside a hyperballof radius ξ is to make sure that the total volume of K hyperballs of radius √ d e is smaller than thevolume of the hyperball of radius ξ .

Original: Mathematically, it means:

Original: [Equation]

Original: where V d ( r ) denotes the volume of a d -ball of radius r .

Original: When