Original: Private GANs, Revisited
Revised: Private GANs, Revisited

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We show that with improved training, the standard approach for differentiallyprivate GANs – updating the discriminator with noisy gradients – achieves orcompetes with state-of-the-art results for private image synthesis.
Revised: We show that with improved training, the standard approach for differentially private GANs – updating the discriminator with noisy gradients – achieves or competes with state-of-the-art results for private image synthesis.

Original: Existing instantiations of this approach neglect to consider how adding noise only to discriminatorupdates disrupts the careful balance between generator and discriminator necessaryfor successful GAN training.
Revised: Existing instantiations of this approach neglect to consider how adding noise only to discriminator updates disrupts the careful balance between generator and discriminator necessary for successful GAN training.

Original: We show that a simple fix restores parity: taking morediscriminator steps between generator steps.
Revised: We show that a simple fix restores parity: taking more discriminator steps between generator steps.

Original: Finally, with the goal of restoringparity between generator and discriminator, we experiment with further modifications to improve discriminator training and see further improvements in generationquality.
Revised: Furthermore, with the goal of restoring parity, we experiment with further modifications to improve discriminator training and see further improvements in generation quality.

Original: For MNIST at  ε = 10 , our private GANs improve the record FID from48.4 to 13.0, as well as downstream classifier accuracy from 83.2% to 95.0%.
Revised: For MNIST at a privacy budget of ε = 10 , our private GANs improve the record FID from 48.4 to 13.0, as well as downstream classifier accuracy from 83.2% to 95.0%.

Original:  Introduction
Revised:  Introduction

Original: A recent line of work has studied leveraging deep generative models to produce DP syntheticdata.
Revised: A recent line of work has studied leveraging deep generative models to produce differentially private synthetic data.

Original: Initial efforts focused on privatizing generative adversarial networks (GANs) [16] by usingdifferentially private stochastic gradient descent (DPSGD)
Revised: Initial efforts focused on privatizing generative adversarial networks (GANs) [14] by using differentially private stochastic gradient descent (DPSGD)

Original: [1] to update the GAN discriminator –an approach referred to as DPGAN [35, 6, 29].
Revised: [1] to update the GAN discriminator – an approach referred to as DPGAN [33, 6, 27].

Original: Follow-up work has studied alternative approachesto privatizing GANs [20, 22, 8, 33], as well as other generative modelling frameworks, such asmaximum mean discrepancy [17, 31], Sinkhorn divergences [7], and energy-based models [9].
Revised: Follow-up work has studied alternative approaches to privatizing GANs [18, 20, 8, 31], as well as other generative modelling frameworks, such as maximum mean discrepancy [15, 29], Sinkhorn divergences [7], and energy-based models [9].

Original: Towards generating high-dimensional, complex data, these studies have primarily focused on imagesynthesis, which has served as the testbed of choice for research in (non-private) generative modelling.
Revised: Towards generating high-dimensional, complex data, these studies have primarily focused on image synthesis, which has served as the testbed of choice for research in (non-private) generative modelling.

Original: For the task of labelled image synthesis, the literature has corroborated that departing from thebaseline DPGAN approach, either in the privatization scheme, or modelling framework altogether,leads to significant improvements in generation quality.
Revised: For the task of labelled image synthesis, the literature has corroborated that departing from the baseline DPGAN approach, either in the privatization scheme, or modelling framework altogether, leads to significant improvements in generation quality.

Original: Proposed explanations attribute these resultsto inherent limitations of the DPGAN framework, suggesting that: (1) privatizing discriminatortraining is sufficient for privacy, but may be overkill when only the generator needs to be released[22]; or (2) adversarial objectives may be unsuited for training under privacy [7].
Revised: Proposed explanations attribute these results to inherent limitations of the DPGAN framework, suggesting that: (1) privatizing discriminator training is sufficient for privacy, but may be overkill when only the generator needs to be released [20]; or (2) adversarial objectives may be unsuited for training under privacy [7].

Original: We demonstrate that the reported poor results of DPGANs should not be attributed to inherentlimitations of the framework, but rather, training issues.
Revised: We demonstrate that the reported poor results of DPGANs should not be attributed to inherent limitations of the framework, but rather, training issues.

Original: More precisely: we propose that theasymmetric noise addition in DPGANs (adding noise to discriminator updates only) weakens thediscriminator relative to the generator, disrupting the careful balance necessary for successful GANtraining.
Revised: More precisely: we propose that the asymmetric noise addition in DPGANs (adding noise to discriminator updates only) weakens the discriminator relative to the generator, disrupting the careful balance necessary for successful GAN training.

Original: To account for this, we propose to take more discriminator steps between generator updates.
Revised: To account for this, we propose to take more discriminator steps between generator updates.

Original: With this change, DPGANs improve significantly (see Figure 1), achieving or competing withstate-of-the-art results in private image synthesis.
Revised: With this change, DPGANs improve significantly (see Figure 1), achieving or competing with state-of-the-art results in private image synthesis.

Original: Furthermore, we demonstrate that this perspective on private GAN training (“restoring parity to adiscriminator weakened by DP noise”) is an effective heuristic for improving private GANs.
Revised: Furthermore, we demonstrate that this perspective on private GAN training (“restoring parity to a discriminator weakened by DP noise”) is an effective heuristic for improving private GANs.

Original: We makeother modifications to discriminator training – large batch sizes and discriminator step schedules –and see futher improvements.
Revised: We make other modifications to discriminator training – large batch sizes and discriminator step schedules – and see futher improvements.

Original: 10 Privacy budget
Revised: 10 Privacy budget

Original: F I D
Revised: F I D

Original: sc
Revised: sc

Original: o r e
Revised: o r e

Original: non-private (3.2) n D = n D =10 n D = n D = (a) MNIST FID over a training run n D = n D = n D = (b) Corresponding images at (10 , 10 − 5 )
Revised: non-private (3.2) n D = n D =10 n D = n D = (a) MNIST FID over a training run n D = n D = n D = (b) Corresponding images at (10 , 10 − 5 )

Original: -DP
Revised: -DP

Original:  Preliminaries
Revised:  Preliminaries

Original: Our goal is to train a generative model on sensitive data that is safe to release, i.e., it does not leak thesecrets of individuals in the training dataset.
Revised: Our goal is to train a generative model on sensitive data that is safe to release, i.e., it does not leak the secrets of individuals in the training dataset.

Original: We do this by ensuring the training algorithm A – whichtakes as input the sensitive dataset D ∈ L and returns the parameters of a trained (generative) modelθ ∈ Θ – satisfies differential privacy.
Revised: We do this by ensuring the training algorithm A – which takes as input the sensitive dataset D ∈ U and returns the parameters of a trained (generative) model θ ∈ Θ – satisfies differential privacy.

Original: Definition 1 (Differential Privacy [15]) .
Revised: Definition 1 (Differential Privacy [13]) .

Original: A randomized algorithm A : L → Θ is ( ε, δ ) - differentiallyprivate if for every pair of neighbouring datasets D, D ′ ∈ L , we have
Revised: A randomized algorithm A : U → Θ is ( ε, δ ) - differentially private if for every pair of neighbouring datasets D, D ′ ∈ U , we have

Original: [Equation]
Revised: [Equation]

Original: for all S ⊆ 
Revised: for all S ⊆ Θ .

Original: [Equation] We adopt the add/remove definition of DP, and say two datasets D and D ′ are neighbouring if theydiffer in at most one entry, that is, D = D ′ ∪ { x } or D ′ =
Revised: We adopt the add/remove definition of DP, and say two datasets D and D ′ are neighbouring if they differ in at most one entry, that is, D = D ′ ∪ { x } or D ′ =

Original: D ∪ { x } .
Revised: D ∪ { x } .

Original: We highlight one convenient property of DP, known as closure under post-processing , which saysthat interacting with a privatized model (e.g., using it to compute gradients on non-sensitive data,drawing samples from it) does not lead to any further privacy violation.
Revised: We highlight one convenient property of DP, known as closure under post-processing , which says that interacting with a privatized model (e.g., using it to compute gradients on non-sensitive data, drawing samples from it) does not lead to any further privacy violation.

Original: DPSGD.
Revised: DPSGD.

Original: A gradient-based training algorithm can be privatized by employing differentially privatestochastic gradient descent (DPSGD) [27, 5, 1] as a drop-in replacement for SGD.
Revised: A gradient-based training algorithm can be privatized by employing differentially private stochastic gradient descent (DPSGD) [25, 5, 1] as a drop-in replacement for SGD.

Original: DPSGD involvesclipping per-example gradients and adding Gaussian noise to their sum, which effectively boundsand masks the contribution of any individual point to the final model parameters.
Revised: DPSGD involves clipping per-example gradients and adding Gaussian noise to their sum, which effectively bounds and masks the contribution of any individual point to the final model parameters.

Original: Privacy analysis of
Revised: Privacy analysis of

Original: DPSGD follows from a number of classic tools in the DP toolbox: the Gaussian mechanism, privacyamplification by subsampling, and composition [14, 13, 4, 34].
Revised: DPSGD follows from a number of classic tools in the DP toolbox: the Gaussian mechanism, privacy amplification by subsampling, and composition [12, 1, 4, 32].

Original: We employ the DPSGD analysis of[23] implemented in  Opacus [36].
Revised: We employ the DPSGD analysis of [21] implemented in the Opacus library [34].

Original: DPGANs.
Revised: DPGANs.

Original: Algorithm 1 details the training algorithm for DPGANs, which is effectively an instantiation of DPSGD.
Revised: Algorithm 1 details the training algorithm for DPGANs, which is effectively an instantiation of DPSGD.

Original: Note that only gradients for the discriminator D must be privatized (via clippingand noise), and not those for the generator G due to post-processing.
Revised: Note that only gradients for the discriminator D must be privatized (via clipping and noise), and not those for the generator G due to post-processing.

Original:  Frequent discriminator steps improves private GANs
Revised:  Frequent discriminator steps improves private GANs

Original: In this section, we discuss our main finding: the number of discriminator steps taken between eachgenerator step ( n D from Algorithm 1) plays a significant role in the success of private GAN training.
Revised: In this section, we discuss our main finding: the number of discriminator steps taken between each generator step ( n D from Algorithm 1) plays a significant role in the success of private GAN training.

Original: For a fixed setting of DPSGD hyperparameters, there is an optimal range of values for n D thatmaximizes generation quality, in terms of both visual quality and utility for downstream classifiertraining.
Revised: For a fixed setting of DPSGD hyperparameters, there is an optimal range of values for n D that maximizes generation quality, in terms of both visual quality and utility for downstream classifier training.

Original: This value is often quite large ( n D ≈ 100 in some cases).
Revised: This value is often quite large ( n D ≈ 100 in some cases).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: This setting reports its best FID and downstream accuracy at t = 50 K iterations ( ε ≈ 2 . 85 ).

Original: As training progresses, we observe mode collapse alongside deterioration in evaluation metrics.

Original: 3.1 Experimental details
Revised: 3.1 Experimental details

Original: Setup.
Revised: Setup.

Original: We focus on labelled generation of MNIST and FashionMNIST.
Revised: We focus on labelled generation of MNIST and FashionMNIST.

Original: To build a strong baseline,we begin from an open source PyTorch [25] implementation 1 of DCGAN [26] that performs wellnon-privately, and copy their training recipe.
Revised: To build a strong baseline, we begin from an open source PyTorch [23] implementation 2 of DCGAN [24] that performs well non-privately, and copy their training recipe.

Original: We adapt their architecture to our purposes: removing
Revised: We adapt their architecture to our purposes: removing

Original: BatchNorm layers (which not compatible with DPSGD) and adding label embedding layers to enablelabelled generation.
Revised: BatchNorm layers (which not compatible with DPSGD) and adding label embedding layers to enable labelled generation.

Original: Training this configuration non-privately yields labelled generation that achieves
Revised: Training this configuration non-privately yields labelled generation that achieves

Original: FID scores of 3.2 on MNIST and 15.9 on FashionMNIST.
Revised: FID scores of 3.2 on MNIST and 15.9 on FashionMNIST.

Original: Finally, we note that these models are notsmall: D and G have 1 . 72 M and 2 .
Revised: Finally, we note that these models are not small: D and G have 1 . 72 M and 2 .

Original: 27 M trainable parameters respectively.
Revised: 27 M trainable parameters respectively.

Original: Privacy implementation.
Revised: Privacy implementation.

Original: To privatize training, we use Opacus [36] which implements per-examplegradient computation and the RDP accounting of [23].
Revised: To privatize training, we use Opacus [34] which implements per-example gradient computation and the RDP accounting of [21].

Original: For our baseline setting, we use the following
Revised: For our baseline setting, we use the following

Original: DPSGD hyperparameters: we keep the non-private (expected) batch size B = 128 , and use a noisescale σ = 1 and clipping norm C = 1 .
Revised: DPSGD hyperparameters: we keep the non-private (expected) batch size B = 128 , and use a noise scale σ = 1 and clipping norm C = 1 .

Original: Under these settings, we have the budget for T = 450 , 000discriminator steps when targeting (10 , 10 − 5 ) -DP.
Revised: Under these settings, we have the budget for T = 450 ,discriminator steps when targeting (10 , 10 − 5 ) -DP.

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate our generative models by examining the visual quality and utility fordownstream tasks of generated images.
Revised: We evaluate our generative models by examining the visual quality and utility for downstream tasks of generated images.

Original: Following prior work, we measure visual quality by computingthe Fréchet Inception Distance (FID) [18] between 60 , 000 generated images and 10 , 000 real imagesfrom the test set.
Revised: Following prior work, we measure visual quality by computing the Fréchet Inception Distance (FID) [16] between 60 , 000 generated images and 10 , 000 real images from the test set.

Original: To measure downstream task utility, we again follow prior work, and train a CNNclassifier on 60 , 000 generated image-label pairs and report its accuracy on the real test set.
Revised: To measure downstream task utility, we again follow prior work, and train a CNN classifier on 60 , 000 generated image-label pairs and report its accuracy on the real test set.

Original: 3.2 More frequent discriminator steps improves generation quality
Revised: 3.2 More frequent discriminator steps improves generation quality

Original: We plot in Figures 1 and 2 the evolution of FID and downstream accuracy during DPGAN training forboth MNIST and FashionMNIST, under varying discriminator update frequencies n D .
Revised: We plot in Figures 1a and 2 the evolution of FID and downstream accuracy during DPGAN training for both MNIST and FashionMNIST, under varying discriminator update frequencies n D .

Original: The effect ofWe this parameter has outsized impact on the final results.
Revised: The effect of this parameter has outsized impact on the final results.

Original: For MNIST, n D = 50 yields the best results;on FashionMNIST, the best FID is obtained at n D = 200 and the best accuracy at n D = 100 .
Revised: For MNIST, n D = 50 yields the best results; on FashionMNIST, the best FID is obtained at n D = 200 and the best accuracy at n D = 100 .

Original: We emphasize that varying the frequency of discriminator steps, relative to generator steps, does notaffect the privacy cost  of Algorithm 1.
Revised: We emphasize that increasing n D , the frequency of discriminator steps relative to generator steps, does not affect the privacy cost ε of Algorithm 1.

Original: For any setting of n D , we perform the same number of noisygradient queries on real data.
Revised: For any setting of n D , we perform the same number of discriminator steps (noisy gradient queries on real data) – what changes is the total number of generator steps taken over the course of training, which is reduced by a factor of 1 /n D .

Original: 3.3 Private GANs are on a path to mode collapse
Revised: 3.3 Private GANs are on a path to mode collapse

Original: In Figures 1 and 2a, we observe that at low discriminator update frequencies ( n D = 10) , the best
Revised: In Figures 1a and 2a, we observe that at low discriminator update frequencies ( n D = 10) , the best

Original: FID and accuracy scores occur early in training, well before the privacy budget we are targeting isexhausted.
Revised: FID and accuracy scores occur early in training, well before the privacy budget we are targeting is exhausted.

Original: 3 At 50 , 000 discriminator steps ( ε ≈ 2 . 85 ), n D = 10 has better FID (30.6) and accuracy(83.3%) than the other settings.
Revised: At 50 , 000 discriminator steps ( ε ≈ 2 . 85 ), n D = 10 has better FID (30.6) and accuracy (83.3%) than the other settings.

Original: However, these results degrade by the end of training.
Revised: However, these results degrade by the end of training.

Original: In Figure 3,we plot the evolution of generated images in this setting over the course of training, and observequalitative evidence of mode collapse, co-occurring with deterioration in evaluation metrics.
Revised: In Figure 3, we plot the evolution of generated images in this setting over the course of training, and observe qualitative evidence of mode collapse, co-occurring with deterioration in evaluation metrics.

Original: These results suggest that fixing other DPSGD hyperparameters, there is an optimal setting forthe discriminator step frequency n D that strikes a balance between: (1) being too low, causing themodel performance peak early in training and undergo mode collapse; resulting in all subsequenttraining to consume additional privacy budget without improving the model ; and (2) being too high,preventing the generator from taking enough steps to converge before the privacy budget is exhausted(an example of which is pictured in Figure 2a).
Revised: These results suggest that fixing other DPSGD hyperparameters, there is an optimal setting for the discriminator step frequency n D that strikes a balance between: (1) being too low, causing the model performance peak early in training and undergo mode collapse; resulting in all subsequent training to consume additional privacy budget without improving the model ; and (2) being too high, preventing the generator from taking enough steps to converge before the privacy budget is exhausted (an example of which is pictured in Figure 2a).

Original: Striking this balance results in the most effectiveutilization of privacy budget towards improving the generator.
Revised: Striking this balance results in the most effective utilization of privacy budget towards improving the generator.

Original:  Why does taking more steps help?
Revised:  Why does taking more steps help?

Original: Figure 4 compares the accuracy of the GAN discriminator (on held-out real and fake examples)immediately before each generator step, between non-private training and private training for different settings of n D .
Revised: Figure 4 compares the accuracy of the GAN discriminator (on held-out real and fake examples) immediately before each generator step, between non-private training and private training for different settings of n D .

Original: Non-privately, discriminator accuracy stays around 60% throughout training.
Revised: Non-privately, discriminator accuracy stays around 60% throughout training.

Original: Introducing DP ( n D = 1) leads to a qualitative difference: DP causes discriminator accuracy to dropto 50% immediately, and never recovers.
Revised: Introducing DP ( n D = 1) leads to a qualitative difference: DP causes discriminator accuracy to drop to 50% immediately, and never recovers.

Original: For other settings of n D , we make three observations: (1)
Revised: For other settings of n D , we make three observations: (1)

Original: Larger n D corresponds to higher accuracy; (2) The generator improves during the periods in whichthe discriminator stays above 50% accuracy; and (3) accuracy decreases throughout training, and thedegradation of results co-occurs with accuracy of the discriminator dropping below 50%.
Revised: Larger n D corresponds to higher accuracy; (2) The generator improves during the periods in which the discriminator stays above 50% accuracy; and (3) accuracy decreases throughout training, and the degradation of results co-occurs with accuracy of the discriminator dropping below 50%.

Original: Based on these observations, we propose the following explanation on why more steps helps:
Revised: Based on these observations, we propose the following explanation on why more steps helps:

Original: • Generator improvement occurs when our discriminator is capable of distinguishing between real and fake data.
Revised: • Generator improvement occurs when our discriminator is capable of distinguishing between real and fake data.

Original: • The asymmetric noise addition introduced by DP to the discriminator makes the task difficult.
Revised: • The asymmetric noise addition introduced by DP to the discriminator makes the task difficult.

Original: • Allowing the discriminator to train longer on a fixed generator improves its accuracy, recovering the non-private case where the generator and discriminator are balanced.
Revised: Allowing the discriminator to train longer on a fixed generator improves its accuracy, recovering the non-private case where the generator and discriminator are balanced.

Original:  Better generators via better discriminators
Revised:  Better generators via better discriminators

Original: Our proposed explanation in Section 4 provides a suggestion for improving GAN training: effectivelyuse our privacy budget to maximize the number of generator steps taken when the discriminator hasa high accuracy.
Revised: Our proposed explanation in Section 4 provides a suggestion for improving GAN training: effectively use our privacy budget to maximize the number of generator steps taken when the discriminator has a high accuracy.

Original: We experiment with modifications to the private GAN training recipe to improvediscriminator accuracy, which translate to improved generation.
Revised: We experiment with modifications to the private GAN training recipe to improve discriminator accuracy, which translate to improved generation.

Original: 5.1 Larger batch sizes
Revised: 5.1 Larger batch sizes

Original: Several recent works have demonstrated that for classification tasks, large batch sizes improve theperformance of DPSGD, after tuning for the optimal noise scale σ accordingly [30, 2, 11].
Revised: Several recent works have demonstrated that for classification tasks, large batch sizes improve the performance of DPSGD, after tuning for the optimal noise scale σ accordingly [28, 2, 11].

Original: GANtraining is typically conducted with small batch sizes (for example, DCGAN uses B = 128 , which we adopt; StyleGAN uses B = 32 ).
Revised: GAN training is typically conducted with small batch sizes (for example, DCGAN uses B = 128 , which we adopt; StyleGAN uses B = 32 ).

Original: Therefore it is interesting to see if large batch sizes indeedimprove private GAN training.
Revised: Therefore it is interesting to see if large batch sizes indeed improve private GAN training.

Original: We scale up batch sizes, considering B ∈ { 64 , 128 , 512 , 2048 } , andsearch for the optimal noise scale σ and n D .
Revised: We scale up batch sizes, considering B ∈ { 64 , 128 , 512 , 2048 } , and search for the optimal noise scale σ and n D .

Original: We target two settings, the high privacy ( ε = 1 ) and lowprivacy ( ε = 10 ) regimes.
Revised: We target two settings, the high privacy ( ε = 1 ) and low privacy ( ε = 10 ) regimes.

Original: We report the best results from our hyperparameter search in in Table 1.
Revised: We report the best results from our hyperparameter search in in Table 1.

Original: We find that larger batchsizes leads to improvements: for ε = 10 , the best MNIST and FashionMNIST results are achieved at
Revised: We find that larger batch sizes leads to improvements: for ε = 10 , the best MNIST and FashionMNIST results are achieved at

Original: B = 2048 .
Revised: B = 2048 .

Original: For ε = 1 , the best results are achieved at B = 512 .
Revised: For ε = 1 , the best results are achieved at B = 512 .

Original: 5.2 Discriminator step scheduling
Revised: 5.2 Discriminator step scheduling

Original: Our observations from Section 3 and 4 motivate us to consider a discriminator step frequency schedule.
Revised: Our observations from Section 3 and 4 motivate us to consider a discriminator step frequency schedule.

Original: As pictured in Figure 4, discriminator accuracy drops during training as the quality of generatedimages improve, which co-occurs with degradation of results.
Revised: As pictured in Figure 4, discriminator accuracy drops during training as the quality of generated images improve, which co-occurs with degradation of results.

Original: In this scenario, we want to take moresteps to improve the discriminator.
Revised: In this scenario, we want to take more steps to improve the discriminator.

Original: However, using a large discriminator update frequency at thebeginning of training is wasteful – as evidenced by the fact that low n D achieves the best FID andaccuracy early in training.
Revised: However, using a large discriminator update frequency at the beginning of training is wasteful – as evidenced by the fact that low n D achieves the best FID and accuracy early in training.

Original: Hence we propose to start at a low discriminator update frequency, andramp up when our discriminator is performing poorly.
Revised: Hence we propose to start at a low discriminator update frequency, and ramp up when our discriminator is performing poorly.

Original: The accuracy on real data must be released with DP.
Revised: The accuracy on real data must be released with DP.

Original: While this is feasible, it introduces the additionalproblem of having to find the right split of privacy budget for the best performance.
Revised: While this is feasible, it introduces the additional problem of having to find the right split of privacy budget for the best performance.

Original: We observe thatdiscriminator accuracy is related to discriminator accuracy on fake samples only (which are free toevaluate on, by post-processing).
Revised: We observe that discriminator accuracy is related to discriminator accuracy on fake samples only (which are free to evaluate on, by post-processing).

Original: Hence we use it as a proxy to assess discriminator performance.
Revised: Hence we use it as a proxy to assess discriminator performance.

Original: The step schedule is parameterized by two terms, β and d .
Revised: The step schedule is parameterized by two terms, β and d .

Original: β is the decay parameter used to computethe exponential moving average (EMA) of discriminator accuracy on fake batches before eachgenerator update.
Revised: β is the decay parameter used to compute the exponential moving average (EMA) of discriminator accuracy on fake batches before each generator update.

Original: We use β = 0 .
Revised: We use β = 0 .

Original: 99 in all settings.
Revised: 99 in all settings.

Original: d is the accuracy threshold that upon falling below,we increase the update frequency.
Revised: d is the accuracy threshold that upon falling below, we increase the update frequency.

Original: We try d = 0 .
Revised: We try d = 0 .

Original: 6 and d = 0 .
Revised: 6 and d = 0 .

Original: 7 , finding that 0 .
Revised: 7 , finding that 0 .

Original: 7 works better forlarge batches.
Revised: 7 works better for large batches.

Original: Additionally, we promise a grace period of 2 / (1 − β ) = 200 generator steps beforemoving on to the next update frequency.
Revised: Additionally, we promise a grace period of 2 / (1 − β ) = 200 generator steps before moving on to the next update frequency.

Original: This formula is motivated by the fact that β -EMA’s value isprimarily determined by its last 2 / (1 − β ) observations.
Revised: This formula is motivated by the fact that β -EMA’s value is primarily determined by its last 2 / (1 − β ) observations.

Original: The additional benefit of the step schedule is that it means we do not have to search for the optimalupdate frequency.
Revised: The additional benefit of the step schedule is that it means we do not have to search for the optimal update frequency.

Original: Although the step schedule introduces the extra hyperparameter of the threshold d ,we found that these two settings ( d = 0 . 6 and 0 . 7 ) were sufficient to improve over results of a muchmore extensive hyperparameter search.
Revised: Although the step schedule introduces the extra hyperparameter of the threshold d , we found that these two settings ( d = 0 . 6 and 0 . 7 ) were sufficient to improve over results of a much more extensive hyperparameter search.

Original: 5.3 Comparison with state of the art
Revised: 5.3 Comparison with state of the art

Original: Table 1 summarizes our best experimental settings.
Revised: Table 1 summarizes our best experimental settings.

Original: In the low privacy/high ε regime, most of ourresults are dramatically better than prior work 4 – for example, decreasing FID from 48.4 to 13.0 andincreasing accuracy from 83.2% to 95.0% on MNIST.
Revised: In the low privacy/high ε regime, most of our results are dramatically better than prior work 4 – for example, decreasing FID from 48.4 to 13.0 and increasing accuracy from 83.2% to 95.0% on MNIST.

Original: In the high privacy/low ε regime, improvementsare not quite as extreme, but can still be significant (FID for MNIST), and only compare negatively tostate-of-the-art for accuracy on Fashion MNIST.
Revised: In the high privacy/low ε regime, improvements are not quite as extreme, but can still be significant (FID for MNIST), and only compare negatively to state-of-the-art for accuracy on Fashion MNIST.

Original: Broadly speaking, we see an improvement fromtaking larger batch sizes, and then another with an adaptive step schedule.
Revised: Broadly speaking, we see an improvement from taking larger batch sizes, and then another with an adaptive step schedule.

Original: We provide some examplegenerated images in Figures 5 and 6 for ε = 10 , and Figures 7 and 8 for ε = 1 .
Revised: We provide some example generated images in Figures 5 and 6 for ε = 10 , and Figures 7 and 8 for ε = 1 .

Original:  Discussion and related work
Revised:  Discussion and Related Work

Original: DP generative modelling.
Revised: DP generative modelling.

Original: Differentially private GANs (DPGANs) were introduced by Xie et al.[35].
Revised: Differentially private GANs (DPGANs) were introduced by Xie et al.

Original: Subsequent works proposed alternative privatizations schemes for GANs settings [29, 20, 22,8, 33]; they did not consider a significantly larger number of discriminator steps per generator step. 
Revised: Subsequent works proposed alternative privatizations schemes for GANs settings [27, 18, 20, 8, 31]; they did not consider a significantly larger number of discriminator steps per generator step. [Table]

Original: Other private generative models include VAEs [10], maximum mean discrepancy [17], Sinkhorndivergences [7], energy-based models [9], and normalizing flows [32].
Revised: Other private generative models include VAEs [10], maximum mean discrepancy [15], Sinkhorn divergences [7], energy-based models [9], and normalizing flows [30].

Original: We show that a well-tuned
Revised: We show that a well-tuned

Original: DPGAN competes with or outperforms these approaches.
Revised: DPGAN competes with or outperforms these approaches.

Original: [Table]

Original: Custom approaches versus a well-tuned DPSGD.
Revised: Custom approaches versus a well-tuned DPSGD.

Original: An ongoing debate pertains to the best techniques and architectures for private ML.
Revised: An ongoing debate pertains to the best techniques and architectures for private ML.

Original: Roughly speaking, there are two schools of thought.
Revised: Roughly speaking, there are two schools of thought.

Original: Oneinvestigates novel architectures for privacy, which may be outperformed by more traditional approaches in the non-private setting.
Revised: One investigates novel architectures for privacy, which may be outperformed by more traditional approaches in the non-private setting.

Original: Some examples include [7, 17, 31], a variety of generative modelsspecifically designed to be compatible with differential privacy.
Revised: Some examples include [7, 15, 29], a variety of generative models specifically designed to be compatible with differential privacy.

Original: The other focuses on searching withinthe space of tried-and-tested methods that are understood to work well non-privately.
Revised: The other focuses on searching within the space of tried-and-tested methods that are understood to work well non-privately.

Original: Some examplesinclude the works of [12, 21, 37], who demonstrate that, similar to the non-private setting, large-scale
Revised: Some examples include the works of [11, 19, 35], who demonstrate that, similar to the non-private setting, large-scale

Original: CNN and Transformer architectures can achieve state-of-the-art results for image classification and
Revised: CNN and Transformer architectures can achieve state-of-the-art results for image classification and

Original: NLP tasks.
Revised: NLP tasks.

Original: The primary modifications to the pipeline are along the lines of changing the batch size,modifying normalization layers, etc., most of which would be explored in a proper hyperparametersearch in the non-private setting.
Revised: The primary modifications to the pipeline are along the lines of changing the batch size, modifying normalization layers, etc., most of which would be explored in a proper hyperparameter search in the non-private setting.

Original: Our work fits into the latter line: we show that novel generativemodels introduced for privacy can be outperformed by GANs trained with well-tuned DPSGD.
Revised: Our work fits into the latter line: we show that novel generative models introduced for privacy can be outperformed by GANs trained with well-tuned DPSGD.

Original: Tabular data.
Revised: Tabular data.

Original: Our investigation focused on image datasets, namely MNIST and Fashion MNIST,while many important applications of private data generation involve tabular data .
Revised: Our investigation focused on image datasets, namely MNIST and Fashion MNIST, while many important applications of private data generation involve tabular data .

Original: While [28] findthat private GAN-based approaches fail to preserve even basic statistics in these settings, we believethat our techniques may yield similar improvements.
Revised: While [26] find that private GAN-based approaches fail to preserve even basic statistics in these settings, we believe that our techniques may yield similar improvements.

Original:  Conclusion.
Revised:  Conclusion

Original: Our most important contribution is to show that private GANs have been underratedby the research community, and in fact can achieve state-of-the-art results by properly exploring thesearch space.
Revised: Our most important contribution is to show that private GANs have been underrated by the research community, and in fact can achieve state-of-the-art results by properly exploring the search space.

Original: We hope and anticipate this will inspire the community to revisit private GANs, andquickly improve upon our results
Revised: We hope and anticipate this will inspire the community to revisit private GANs, and quickly improve upon our results.

Revised: Acknowledgments and Disclosure of Funding

Revised: Work performed in part while AB was interning at Huawei.

Revised: AB and GK are supported by an NSERC

Revised: Discovery Grant, a University of Waterloo Startup Grant, an unrestricted gift from Google, and an unrestricted gift from Apple.

Revised: AB is supported by a Vector Scholarship in Artificial Intelligence.

Original: B Generated samples
Revised: B Generated samples

Original: We provide a few non-cherrypicked samples from MNIST and FashionMNIST at ε = 1 and ε = 10 .
Revised: We provide a few non-cherrypicked samples from MNIST and FashionMNIST at ε = 1 and ε = 10 .