Scalable and Efﬁcient Training of Large Convolutional Neural Networks with Differential Privacy
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Large convolutional neural networks (CNN) can be difﬁcult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping.
We proposean efﬁcient and scalable implementation of this clipping on convolutional layers,termed as the mixed ghost clipping, that signiﬁcantly eases the private training interms of both time and space complexities, without affecting the accuracy.
The improvement in efﬁciency is rigorously studied through the ﬁrst complexity analysisfor the mixed ghost clipping and existing DP training algorithms.
Extensive experiments on vision classiﬁcation tasks, with large ResNet, VGG,and Vision Transformers, demonstrate that DP training with mixed ghost clippingadds 1 ⇠ 10% memory overhead and < 2 ⇥ slowdown to the standard non-privatetraining.
Speciﬁcally, when training VGG19 on CIFAR10, the mixed ghost clippingis 3 ⇥ faster than state-of-the-art Opacus library with 18 ⇥ larger maximum batchsize.
To emphasize the signiﬁcance of efﬁcient DP training on convolutional layers,we achieve 96.7% accuracy on CIFAR10 and 83.0% on CIFAR100 at ✏ = 1 using
BEiT, while the previous best results are 94.8% and 67.4%, respectively.
Weopen-source a privacy engine that implements DP training of CNN with a few linesof code.
Introduction
Deep convolutional neural networks (CNN) [16, 30] are the backbone in vision-related tasks, includingimage classiﬁcation [28], object detection [37], image generation [19], video recognition [39] andaudio classiﬁcation [22].
A closer look at the dominating success of deep CNNs reveals its basis ontwo factors.
The ﬁrst factor is the strong capacity of the convolutional neural networks, usually characterized bythe enormous model size.
Recent state-of-the-art progresses usually result from very large models,with millions to billions of trainable parameters.
For example, ImageNet [10] accuracy grows whenlarger VGGs [40] (increasing 11 to 19 layers = )
69% to 74% accuracy) or ResNets [20] (increasing18 to 152 layers = )
70% to 78% accuracy) are used [42].
Consequently, the eager to use larger modelsfor better accuracy naturally draws people’s attention to the scalability and efﬁciency of training.
The second factor is the availability of big data, which oftentimes contain private and sensitiveinformation.
The usage of such data demands rigorous protection against potential privacy attacks.
In fact, one standard approach to guarantee the protection is by differentially private (DP) [13, 14]training of the models.
Since [1], CNNs have achieved promising results under strong DP guarantee:
CIFAR10 achieves 92.4% accuracy in [45] and ImageNet achieves 81.1% accuracy in [9].
Unifying the two driving factors of CNNs leads to the DP training of large CNNs.
However, thefollowing challenges are hindering our application of large and private CNNs in practice
Challenge 1: Time and space efﬁciency in DP training.
DP training can be extremely inefﬁcient inmemory and speed.
For example, a straightforward implementation in Tensorﬂow Privacy libraryshows that DP training can be 1000 ⇥ slower than the non-DP training, even on a small RNN [5];other standard DP libraries, Opacus [49] and JAX [29, 41], which trade off memory for speed, couldnot ﬁt a single datapoint into GPU on GPT2-large [32]; addtionally, 3 ⇠ 9 ⇥ slowdown of DP traininghas been reported in [29, 9, 41] using JAX.
The computational bottleneck comes from the per-sample gradient clipping at each iteration (see(2.1)), a necessary step in DP deep learning.
I.e., denoting the loss as
P i L i , we need to clip theper-sample gradient { @ L i @
W } i individually.
This computational issue is even more severe when weapply a large batch size, which is necessary to achieve high accuracy of DP neural networks.
In[32, 29, 33], it is shown that the optimal batch size for DP training is signiﬁcantly larger than forregular training.
For instance, DP ResNet18 achieves best performance on ImageNet when batchsize is 64*1024 [29]; and DP ResNet152 and ViT-Large use a batch size 2 20 in [33].
As a result, anefﬁcient implementation of per-sample gradient clipping is much-needed to fully leverage the beneﬁtof large batch training.
Challenge 2: Do large DP vision models necessarily harm accuracy?
An upsetting observationin DP vision models is that, over certain relatively small model size, larger DP CNNs seem tounderperform smaller ones.
This is observed in models that are either pre-trained or trained fromscratch [25, 1].
As an example of the pre-trained cases, previously state-of-the-art CIFAR10 isobtained from a small DP linear model [45], and the ﬁne-tuned DP ResNet50 underperforms DP
ResNet18 on ImageNet [29].
On the contrary, the empirical evidence in DP language modelsshows that larger models can consistently achieve better accuracy [32].
Interestingly, we empiricallydemonstrate that the trend in [32] can possibly hold in vision models as well.
1.1 Contributions
In this work, we propose new algorithms to efﬁciently train large-scale CNNs with DP optimizers.
To be speciﬁc, our contributions are as follows.
We propose a novel implementation, termed as the mixed ghost clipping , of the per-sample gradient clipping for 1D ⇠ 3D convolutional layers.
The mixed ghost clipping is the ﬁrst
method that can implement per-sample gradient clipping without per-sample gradients ofthe convolutional layers.
It works with any DP optimizer and any clipping function almost
as memory efﬁciently as in standard training, thus signiﬁcantly outperforming existing
implementation like Opacus [49].
In some tasks, mixed ghost clipping also claims supremacy on speed using a ﬁxed batch size.
The speed can be further boosted (say 1 . 7 ⇥ faster than the fastest alternative DP algorithms
and only 2 ⇥ slower than the non-private training) when the memory saved by our method isused to ﬁt the largest possible batch size.3.
We provide the ﬁrst complexity analysis of mixed ghost clipping in comparison to othertraining algorithms.
This analysis clearly indicates the necessity of our layerwise decision principle, without which the existing methods suffer from high memory burden.4.
Leveraging our algorithms, we can efﬁciently train large DP models, such as VGG, ResNet,Wide-ResNet, DenseNet and Vision Transformer (ViT).
Using DP ViTs at ImageNet scale,
we are the ﬁrst to train convolutional ViTs under DP and achieve dominating SOTA on CIFAR10/100 datasets, thus bringing new insights that larger vision models can consistently
achieve better accuracy under DP.
1.2 Previous arts
The straightforward yet highly inefﬁcient way of per-sample gradient clipping is to use batch size 1and compute gradients with respect to each individual loss.
Recently, more advanced methods havesigniﬁcantly boosted the efﬁciency by avoiding such a naive approach.
The most widely appliedmethod is implemented in the Opacus library [49], which is fast but memory costly as per-samplegradients g i = @
L i @
W are instantiated to compute the weighted gradient
P
i C i · g i in (2.1).
A moreefﬁcient method, FastGradClip [31], is to use a second back-propagation with weighted loss
P
i C i ·L i 87 to indirectly derive the weighted gradient.
In all above-mentioned methods and [38], the per-sample gradients are instantiated, whereas this canbe much inefﬁcient and not necessary according to the ‘ghost clipping’ technique, as to be detailed in
Section 3.
In other words, ghost clipping proves that the claim ‘DP optimizers require access to theper-sample gradients’ is wrong.
Note that ghost clipping is ﬁrstly proposed by [18] for linear layers,and then extended by [32] to sequential data and embedding layers for language models.
However,the ghost clipping has not been extended to convolutional layers, due to the complication of theconvolution operation and the high dimension of data (text data is mostly 2D, yet image data are 3Dand videos are 4D).
In fact, we will show that even ghost clipping alone is not satisfactory for CNNs,and therefore propose the mixed ghost clipping, that narrows the efﬁciency gap between DP trainingand the regular training.
Preliminaries2.1 Differential privacy
Differential privacy (DP) has become the standard approach to provide privacy guarantee for modernmachine learning models.
The privacy level is characterized through a pair of privacy quantities( ✏ ,  ) , where smaller ( ✏ ,  ) means stronger protection.
Deﬁnition 2.1 ([14]) .
A randomized algorithm M is ( " ,  ) -DP
if for any neighboring datasets S, S 0that differ by one arbitrary sample, and for any event E , it holds that
[Equation]
In deep learning where the number of parameters are large, the Gaussian mechanism [14, Theorem
A.1] is generally applied to achieve DP at each training iteration, i.e. we use regular optimizers onthe following privatized gradient:
[Equation]
where C is any function whose output is upper bounded by R/
k g i k
and R is known as the clippingnorm.
To name a few examples of C , we have the Abadi’s clipping min( R/ k
g i k , 1) in [1] and theglobal clipping I ( k g i k < Z ) · R/Z for any constant Z in [6].
Here  is the noise multiplier thataffects the privacy loss ( ✏ ,  ) , but R only affects the convergence, not the privacy.
In words, DP training switches from updating with
P i g i to updating with the private gradient e g :
SGD with private gradient is known as DP-SGD; Adam with private gradient is known as DP-Adam.
Algorithmically speaking, the Gaussian mechanism can be decomposed into two parts: the per-samplegradient clipping and the Gaussian noise addition.
From the viewpoint of computational complexity,the per-sample gradient clipping is the bottleneck, while the noise addition costs negligible overhead.
In this work, our focus is the implementation of per-sample gradient clipping (2.1).
We emphasizethat our implementation is only on the algorithmic level, not affecting the mathematics and thusnot the performance of DP optimizers.
That is, our mixed ghost clipping provides exactly the sameaccuracy results as Opacus, FastGradClip, etc.
2.2 Per-sample gradient for free during standard back-propagation
In DP training, the per-sample gradient is a key quantity which can be derived for free from thestandard back-propagation.
We brieﬂy introduce the back-propagation on linear layers, following theanalysis from [18, 32], so as to prepare our new clipping implementation for convolutional layers.
Note that the convolutional layers can be viewed as equivalent to the linear layers in Section 2.3.
Let the input of a hidden layer be a 2 R B ⇥ ··· ⇥ d (a.k.a.
post-activation).
Here a can be in highdimension: for sequential data such as text, a 2 R B ⇥ T ⇥ d where T is the sequence length; for imagedata, a 2 R B ⇥ H ⇥ W ⇥ d where ( H, W ) is the dimension of image and d is number of channels; for3D objects or video data, a 2 R B ⇥ H ⇥ W ⇥ D ⇥ d where D is the depth or time length, respectively.
Denote the weight of a linear layer as W 2 R d ⇥ p , its bias as b 2 R p and its output (a.k.a. preactivation) as s 2 R B ⇥ ··· ⇥ p , where B is the batch size and p is the output dimension.
In the l -th layer of a neural network with L layers in total, we denote its weight, bias, input and outputas W ( l ) , b ( l ) , a ( l ) , s ( l ) respectively, and the activation function as  .
Consider
[Equation]
Clearly the i -th sample’s hidden feature a ( l ) ,i at layer l is freely extractable during the forward pass.
Let L =
P ni =
1 L i be the total loss and L i be the per-sample loss with respect to the i -th sample.
During a standard back-propagation, the following partial product is maintained,
[Equation]
so as to compute the standard gradient @
L @ W ( l ) =
P i @
L i @
W ( l ) in (2.4).
Here  is the Hadamard productand · is the matrix product.
Therefore, @
L @ s ( l ) ,i is also available for free from (2.3) and extractable by
Pytorch hooks, which allows us compute the per-sample gradient by
[Equation]
> 1 .
2.3 Equivalence between convolutional and linear layer
In a convolutional layer 1 , the forward pass is
[Equation]
in which F is the folding operation and U is the unfolding operation.
To be clear, we consider a2D convolution that a ( l,i ) 2 R H in ⇥ W in ⇥ d ( l ) is the input of hidden feature, ( H in , W in ) is the inputdimension, d ( l ) is the number of input channels.
Then U unfolds the hidden feature from dimension( H in , W in , d ( l ) ) to ( H out W out , d ( l ) k H k W ) , where k H , k W are the kernel sizes and ( H out , W out ) is theoutput dimension.
After the matrix multiplication with W ( l ) 2 R d ( l ) k H k W ⇥ p ( l ) , the intermediateoutput s ( l ) ,i is folded by F from dimension ( H out W out , p ( l ) ) to ( H out , W out , p ( l ) ) .
To present concisely, we ignore the layer index l and write the per-sample gradient of weight for theconvolutional layer, in analogy to the linear layer in (2.4), [Equation]
[Equation]
Here F  1 is the inverse operation of F and simply ﬂattens all dimensions except the last one: from( H out , W out , p ( l ) ) to ( H out W out , p ( l ) ) .
From (2.6), we derive the per-sample gradient norm for theconvolutional layers from the same formula as in [32, Appendix F], [Equation]  [Equation]
[Equation]
Ghost clipping for Convolutional Layers
Leveraging our derivation in (2.7), we propose the ghost clipping to compute the clipped gradientwithout ever generating the per-sample gradient @
L i @
W .
The entire procedure is comprised of the ghostnorm computation and the second back-propagation, as demonstrated in Figure 1.
3.1 Ghost norm: computing gradient norm without the gradient
The per-sample gradient norm is required to compute the per-sample C i in (2.1).
While it is naturalto instantiate the per-sample gradients and then compute their norms [38, 31, 49, 9, 33], this isnot always optimal nor necessary.
Instead, we can leverage (2.7), the ghost norm, to compute theper-sample gradient norm and avoid the possibly expensive per-sample gradient.
Put differently, when
T =
H out W out is small, the multiplication U ( a i ) U ( a i ) > plus F ⇣ @
L @
s i ⌘
F ⇣ @
L @
s i ⌘ > is cheap, butthe multiplication F ⇣ @
L @
s i ⌘ > U ( a i ) is expensive.
We demonstrate the ghost clipping’s supremacyover complexity empirically in Table 4 and theoretically in Table 2.
[Figure]
3.2 Second back-propagation: weighted loss leads to weight gradient
We conduct a second back-propagation with the weighted loss
P
i C i
L i to derive the weighted 167 gradient
P i C
i g i in (2.1), which costs extra time.
In contrast, Opacus [49] and JAX [9, 41] generateand store the per-sample gradient g i for all i 2 [ B ] .
Thus the weighted gradient is directly computablefrom g
i as the memory is traded off for faster computation.
However, in some cases like Table 7 on
ImageNet and Table 9 on CIFAR100, we can use larger batch size to compensate the slowdown ofthe second back-propagation.
Mixed Ghost Clipping: To be a ghost or not, that is the question
Algorithm
Parameters: number of layers L , gradient clipping norm R .
for l = 1 , 2 , . . .
, L do if 2 T 2( l ) < p ( l ) d ( l ) then
W ( l ) .
ghost_norm =
True .
Forward pass
Compute a ( l +1) ,i =  ( F ( U ( a ( l ) ,i ) W ( l ) + b ( l ) ))
Compute per-sample losses L i .
for l = L, L  1 , . . .
, 1 do
Compute @
L @ s ( l ) ,i = @
L @ s ( l +1) ,i W ( l +1)
0 ( s ( l ) ,i ) .
Mixed ghost norm in ﬁrst back-propagation if W ( l ) .
ghost_norm
=
True then k
[Equation]
⇣
[Equation]
[Equation] [Equation]
⌘ else @
L i @
W ( l ) =
F
[Equation] [Equation]
[Equation]
[Equation] [Equation]
[Equation]
[Equation]
!
k @
L i @
W ( [Equation] k 2 Fro
Compute per-sample gradient norm k @
L i @
W k 2 Fro =
P l k @
L i @
W ( l ) k 2 Fro
Compute weighted loss L weighted =
P i C ( k @ L i @
W k Fro ; R ) · L i
Second back-propagation with L weighted to generate
P i C i @
L i @
W
While the ghost norm offers the direct computation gradient norm at the cost of an indirect computation of the weighted gradient, we will show that ghost clipping alone may not be sufﬁcientfor efﬁcient DP training, as we demonstrate in Table 4 and Figure 3.
In Table 2, we give the ﬁrstﬁne-grained analysis of the space and time complexity for DP training algorithms.
Our analysis givesthe precise condition when the per-sample gradient instantiation (adopted in Opacus [49]) is more orless efﬁcient than our ghost norm method.
To take the advantage of both methods, we propose themixed ghost clipping method in Algorithm 1, which applies the ghost clipping or non-ghost clippingby a layerwise decision.
We highlight that the key reason supporting the success of mixed ghost clipping method is itslayerwise adaptivity to the dimension parameters, ( p ( l ) , d ( l ) , T ( l ) , k H , k W ) , which vary largely acrossdifferent layers (see Figure 2).
The variance results from the fact that images are non-sequentialdata, and that the convolution and pooling can change the size ( T = H out W out ) of hidden featuresdrastically.
In the next two sections, we will analyze rigorously the time and memory complexities of the regulartraining and the DP training, using ghost or non-ghost clippings.
Remark 4.1 .
In Algorithm 1, we present the mixed ghost clipping that prioritizes the space complexityby (4.1).
We also derive and implement a speed-priority version by comparing the time complexityof ghost norm and gradient instantiation in Table 1.
However, the efﬁciency difference is empiricallyinsigniﬁcant.
4.1 Complexity analysis
We now break each clipping method into operation modules and analyze their complexities.
A similarbut coarse analysis from [32] only claims, on sequential layers, O ( BT 2 ) space complexity with ghostclipping and O ( Bpd ) without ghost clipping.
The time complexity and/or convolutional layers arenot analyzed until this work.
Complexity Back-propagation
Ghost norm
Grad instantiation Weighted grad
Time
[Equation]
Space
[Equation]
Here B is the batch size, D = dk H k W where d is the number of input channels, k is the kernelsizes, p is the number of output channels, and T =
H out W out .
We leave the detailed complexitycomputation in Appendix D. Leveraging Table 1, we give the complexities of different clippingalgorithms in Table 2.
Complexity
Time
Space
Opacus [Equation]
[Equation]
FastGradClip [31]
[Equation]
Ghost clipping (ours)
[Equation]
Mixed ghost clipping (ours)
SEE CAPTION
[Equation]
4.2 Layerwise decision in mixed clipping
From the space complexity in Table 2, we derive the layerwise decision that selects the more memoryefﬁcient of FastGradClip (gradient instantiation) and ghost clipping (ghost norm):
Choose ghost norm over per-sample gradient instantiation if 2 T 2 < pD = pdk H k W .
[Figure]
c o n v128 112 c o n v 256c o n vc o n v   c o n vc o n v   c o n vc o n v 8 f c  f c1 f c
Ghost
[Figure]
-ghost
[Figure]
[Table]
Therefore, our mixed ghost clipping is a mixup of FastGradClip and the ghost clipping (c.f. Figure 1).
We note that the decision by the mixed ghost clipping (4.1) depends on different dimensions: theghost clipping depends on the size of hidden features (height H and width W ) which in turn dependson kernel size, stride, dilation and padding (see Appendix C for introduction of convolution), whileonly the non-ghost clipping depends on the number of channels.
In ResNet and VGG, the hiddenfeature size decreases as layer depth increases, due to the shrinkage from the convolution and poolingoperation; on the opposite, the number of channels increases in deeper layers.
Remark 4.2 (Ghost clipping favors bottom layers) .
As a consequence of decreasing hidden feature sizeand increasing number of channels, there exists a depth threshold beyond which the ghost clipping ispreferred in bottom layers, where the save in complexity is substantial.
In Figure 2 and Table 3, asthe layer of VGG 11 goes deeper, the hidden feature size shrinks from 224 ! 112 ! · · · !
14 and 216 the number of channels increases from
[Equation]
Performance
We compare our ghost clipping and mixed ghost clipping methods to state-of-the-art clipping algorithms, namely Opacus [49] and FastGradClip [31], which are implemented in Pytorch.
We are awareof but will not compare to implementations of these two algorithms in JAX [4], e.g. [29, 41, 9], so asto only focus on the algorithms rather than the operation framework.
All experiments run on one
Tesla V100 GPU (16GB RAM).
We highlight that switching from the regular training to DP training only needs a few lines of codeusing our privacy engine (see Appendix F).
For CNNs, we use models from https://github.com/kuangliu/pytorch-cifar on CIFAR10 [27] and models from Torchvision [42] on ImageNet [10].
For ViTs, regardless of datasets, we use models from PyTorch Image Models [46].
5.1 Time and memory efﬁciency (ﬁxed batch size)
We ﬁrst measure the time and space complexities when the physical batch size is ﬁxed.
Here wedeﬁne the physical batch size (or the virtual batch size) as the number of samples actually fed into thememory, which is different from the logical batch size.
For example, if we train with batch size 1000but can only feed 40 samples to GPU at one time, we back-propagate 25 times before updating theweights for 1 time.
This technique is known as the gradient accumulation and is widely applied inlarge batch training, which particularly beneﬁts the accuracy of DP training [32, 29, 9, 33].
From Table 4 and the extended Table 6, we see a clear advantage of mixed ghost clipping: ourclipping only incurs  1% memory overhead than the regular training, and is the fastest DP trainingalgorithm.
In contrast on ResNet18, Opacus uses 5 ⇥ memory, and FastGradClip uses 2 ⇥ memory.
Even the ghost clipping uses 1 . 2 ⇥ memory of regular training, while being slower than both Opacusand FastGradClip.
Similar phenomenon is observed on ImageNet in Table 7.
[Table]
5.2 Maximum batch size and throughput
Importantly, the speed efﬁciency in Table 4 can be further boosted, if we use up the saved memory toincrease the batch size.
To stress test the maximum physical batch size and the throughput of eachclipping method, we train ResNet [20], VGG [40], MobileNet[23], ResNeXt [47],AlexNet[26],WideResNet[51], DenseNet[24] and ViTs on CIFAR10 and ImageNet, as summarized partially in Figure 3and in Table 7, respectively.
For example, on VGG19 and CIFAR10, the mixed ghost clipping has amaximum batch size 18 ⇥ bigger (thus 3 ⇥ faster) than Opacus, 3 ⇥ bigger (thus 1 . 7 ⇥ faster) than 246 FastGradClip, and⇥ bigger (thus 1 . 3 ⇥ faster) than the ghost clipping.
Similarly, on Wide-ResNet50and ImageNet, the mixed ghost clipping has a maximum batch size 5 ⇥ bigger than Opacus, and< 0 .
3% more memory costly than the non-private training.
5.3 Vision transformers with convolution on ImageNet scale
In addition to training large-scale CNNs such as ResNet152, we apply our mixed ghost clipping totrain ViTs, which substantially outperform existing SOTA on CIFAR10 and CIFAR100.
Notice thatthe ViTs are pretrained on ImageNet scale, by which we resize CIFAR images (from 32 ⇥ 32 pixelsto 224 ⇥ 224 pixels).
It is worth mentioning that ViT [12] is originally proposed as a substitute of CNN.
Hence it and manyvariants do not contain convolutional layers.
Here we speciﬁcally consider the convolutional ViTs,including ScalableViT[48], XCiT[2], Visformer[8], CrossVit[7], NesT[53], CaiT[44], DeiT[43],
BEiT[3], PiT[21], and ConViT[15].
Performance of these ViTs on CIFAR10 and CIFAR100 are listed in Appendix E for a single-epoch DP training and several ViTs already beat previous SOTA, even though we do not apply additionaltechniques as in [9, 33] (e.g. learning rate schedule or random data augmentation).
[Table]
By training multiple epochs with best performing ViTs in Table 8 and Table 9, we achieve new
SOTA under DP in Table 5, with substantial improvement especially for strong privacy guarantee (e.g.✏ < 2 ).
Our DP training is at most 2 ⇥ slower and 10% more memory expensive than the non-privatetraining, even on BEiT large, thus signiﬁcantly improving the 9 ⇥ slowdown reported in [9].
Discussion
We have shown that DP training can be efﬁcient for large CNNs and ViTs with convolutional layers.
For example, in comparison to non-private training, we reduce the training time to < 2 ⇥ and thememory overhead to < 10% for all vision models examined (up to 303.4 million parameters),including BEiT that achieves SOTA accuracy on CIFAR100 ( +15 .
6% absolutely at ✏ = 1 ).
We haveobserved that for many tasks and large CNNs and ViTs, the memory overhead of DP training can beas low as less than 1%.
We emphasize that our DP training only improves the efﬁciency, not affecting the accuracy, andtherefore is generally applicable, e.g. with SOTA data augmentations in [9].
With efﬁcient trainingalgorithms, we look forward to applying DP CNNs to generation tasks [19] , seq-to-seq learning [17],text classiﬁcation [52], reinforcement learning [34], and multi-modal learning.
Further reducing timecomplexity and prioritizing speed in DP training is another future direction.
In particular, our layerwise decision principle in (4.1) highlights the advantages of ghost clippingwhen T =
HW is small.
This advocates the use of large kernel sizes in DP learning, as they shrinkthe hidden feature aggressively, and have been shown to be highly accurate on non-private tasks[20, 11, 36].