Differentially Private Bias-Term only Fine-tuning of Foundation Models
Abstract
We study the problem of differentially private (DP) fine-tuning of large pre-trained models — a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.
Existing work has demonstrated that high accuracy is possible under strong privacy constraint, yet requires significant computational overhead or modifications to the network architecture.
We propose differentially private bias-term fine-tuning (DP-BiTFiT), which matches the state-of-the-art accuracy for DP algorithms and the efficiency of the standard BiTFiT. DP-BiTFiT is model agnostic (not modifying the network architecture), parameter efficient (only training about 0 .
1% of the parameters), and computation efficient (almost removing the overhead caused by DP, in both the time and space complexity).
On a wide range of tasks, DP-BiTFiT is 2 ∼ 30 × faster and uses 2 ∼ 8 × less memory than DP full fine-tuning, even faster than the standard full fine-tuning.
This amazing efficiency enables us to conduct DP fine-tuning on language and vision tasks with long-sequence texts and high-resolution images, which were computationally difficult using existing methods.
Introduction
Fine-tuning from large pre-trained neural networks is one of the most critical technique in deep learning, yielding strong performance in a variety of domains Pan & Yang (2009); Kenton &
Toutanova (2019); Goyal et al.
Among different methods, full fine-tuning is the most prevalent one, which trains all the model parameters on the downstream tasks and achieves high accuracy within a small number of training epochs.
However, full fine-tuning on large models can be burdensome in terms of the computation and the deployment, since a full copy of fine-tuned model parameters is needed for each task.
To alleviate this issue the parameter efficient fine-tuning only trains a substantially small portion of the model parameters, in contrast to the full fine-tuning.
At high level, the parameter efficient fine-tuning methods can be divided into two categories.
⟨ 1 ⟩ Model-aware methods, meaning a relatively small number of parameters are introduced into the neural network architecture and only the new parameters are optimized.
Examples include LoRA Hu et al.
(2021), Adapter Houlsby et al. (2019), and Compacter Mahabadi et al.
⟨ 2 ⟩ Model-agnostic methods, meaning that only a subset of existing parameters are trainable.
Examples include training only the output linear layer (linear probing, Kornblith et al.
only the layer normalization layer Houlsby et al.
and bias-term fine-tuning (BiTFiT; Zaken et al.
We illustrate the differences in Equation (1):
W 0 , b 0 are the pre-trained weights and biases, ‘ ˆ ’ indicates trainable parameters, and θ is the additional parameters.
[Equation]
Empirically, these parameter efficient fine-tuning methods have achieved high accuracy that is comparable to the full fine-tuning in the standard non-private setting.
For instance, linear probing of ResNet He et al.
and Vision Transformer (ViT, Dosovitskiy et al.
(2020)) achieves 80% accuracy on the ImageNet dataset Sun et al.
; Kornblith et al.
; LoRA and BiTFiT of
RoBERTa Liu et al.
and BERT Kenton & Toutanova (2019) achieve about 94% on SST2, 87% on MNLI, and on average 85% across the General Language Understanding Evaluation (GLUE) datasets He et al.
; Hu et al.
In addition, parameter efficient methods are faster than full fine-tuning and save the communication cost significantly in the distributed learning.
Parallel to these developments, the success of deep learning models relies on the availability of large datasets, which may contain sensitive information to be protected rigorously.
This privacy issue is well-known for neural networks can be vulnerable to privacy attacks: membership information can be leaked from the purchase records via Google and Amazon online services Shokri et al.
(2017); sensitive texts can be reconstructed by specifically designed prefix on GPT2 Carlini et al.
(2021) and so can images in CIFAR10 and MNIST Haim et al.
To protect against such privacy risks, the standard technique is differential privacy (DP, formally stated in Definition 2.1), which randomizes the standard optimizers by updating with the private gradient in Equation (2).
A recent line of work has extensively studied the DP fine-tuning in both computer vision and language tasks, often achieving less than 3% accuracy drop across different settings via full fine-tuning De et al.
; Li et al.
(2021); Bu et al.
(2022b,a), linear probing Mehta et al.
(2022), LoRA, Adapter, or Compacter Yu et al.
(2021a).
In fact, fine-tuning or pre-training from large dataset is considered necessary in the DP deep learning literature.
As a matter of fact, full fine-tuning DP-GPT2 only achieves 24.2 BLEU score ( ϵ = 8 ) on E2E dataset if randomly initialized Li et al.
(2021), in starking contrast to 63.2 BLEU if pre-trained; similarly, state-of-the-art (SOTA) DP accuracy on ImageNet is 48% ( ϵ = 10 ) without pre-training Kurakin et al.
but 86.7% accuracy if pre-trained De et al.
Specifically, parameter efficient DP fine-tuning has empirically demonstrated strong accuracy (see our Table 3) with 3 ∼ 4 × memory saving and 2 ∼ 3 × speedup compared to DP full fine-tuning by Opacus (c.f. Figure 3 and (Yu et al., 2021a, Table 3)).
Although previous works have shed light on various DP fine-tuning methods, we are the first to study DP-BiTFiT specifically and to show two distinctive advantages of it.
[Figure]
Firstly, DP-BiTFiT is model-agnostic and remains its parameter efficiency around 0.1% across models by Table 1.
While linear probing is also model-agnostic, the parameter efficiency can be as high as 8% in ResNet50.
Other methods like LoRA, Adapter and Compacter are architecture-dependent and possibly parameter inefficient, making them difficult to directly apply on arbitrary neural networks:
LoRA and Adapter may need to train more than 12% on BART-large Lewis et al.
(2020) to achieve high accuracy by (He et al., 2021, Figure 1& 4).
Secondly, DP-BiTFiT is computationally efficient, almost as much as the standard BiTFiT and significantly more efficient than DP full fine-tuning, particularly with large models and high-dimensional input data.
For examples of DP full fine-tuning, Li et al.
(2021) have reported 2 ∼ 4 × slowdown on large language models for four advanced private codebases and up to 5 × memory overhead, compared to the standard fine-tuning; even on small networks, 11 codebases across Tensorflow, JAX, and Pytorch have demonstrated 0 .
2 ∼ 5 × slowdown and 3 ∼ 100 × reduction in maximum batch size in Subramani et al.
See more discussion in Section 3.3.
1.1 Contributions
Algorithmically, we propose the Differentially Private Bias-Term Fine-Tuning (DP-BiTFiT) in Algorithm 1 that is highly accurate under DP constraint, on par with SOTA in Section 4.
Specially, we propose a two-phase training to close the accuracy gap between DP-BiTFiT and DP full-finetuning.
DP-BiTFiT is model-agnostic and only optimizes 0.1% of the model parameters on BERT,
RoBERTa, GPT2, ViT, ResNet, and so on (see Table 1).
Thus DP-BiTFiT is one of the most parameter efficient fine-tuning methods among DP LoRA, Adapter, linear probing, etc.
We open-source a computationally efficient implementation of DP-BiTFiT, whose time and space complexity is almost the same as the standard non-DP BiTFiT, while being faster than non-DP full fine-tuning and other DP fine-tuning (see Figure 1).
This advantage is analyzed in Table 2, and demonstrated via the substantial speedup and memory-saving in Figureand Figure 4.
DP-BiTFiT is the unique DP algorithm that has a computation overhead independent of the feature dimension T 1 .
This is due to the activation-free forward pass that only happens in the no-weight training 2 .
Therefore, DP-BiTFiT enjoys a special advantage to work efficiently on long-sequence texts and high-resolution images (see Figure 3).
layers
, noise scale
1: for iteration t = 1 , · · · , T do 2:
Subsample a batch B t ⊆ { 1 , . . . , n } from training set with probability p 3: for layer l ∈ L, L − 1 , · · · , 1 do 4:
Get output gradient ∂ L ∂ s l 5:
Compute per-example gradient and its norm: ∂ L i ∂
b l = ∂ L ∂ s l,i ⊤ 1 = ⇒ ∥
∂ L i ∂ b l ∥ 2 F 6:
Aggregate gradient norms across all layers:
∥ ∂ L i ∂ b ∥ 2 F = (cid:80) l ∥
∂ L i ∂ b l ∥ 2 F 7:
Compute clipping factor: C i = C (
∥ ∂
L i ∂ b ∥ F ; R ) 8:
Compute sum of clipped gradients G =
(cid:80) i C
i ∂ L i ∂ b 9:
Add Gaussian noise G = G + σR · N (0 , I ) 10:
Descend on bias terms with the gradient G by SGD/Adam/...
Preliminaries 2.1 Deep learning with differential privacy
We recall the classic ( ϵ, δ ) -DP, under which we learn deep neural networks.
Definition 2.1 (Dwork et al.
A randomized algorithm M is ( ε, δ ) -differentially private if, for any two neighboring datasets S, S ′ that differ by one datapoint and for any event E , we have P [Equation]
E ] [Equation] ε P [ M ( S ′ )
∈ E ] + δ.
In deep learning, DP can be achieved through applying an off-the-shelf optimizer (SGD or Adam) with a privately released stochastic gradient in place of the regular (cid:80) i g i .
The private stochastic gradient is computed by first getting a minibatch I via Poisson sampling, then compute
Private gradient (cid:88)
[Equation]
[Equation]
where C is any function 3 R + → R subject to C ( x ) ≤ R/x , g i is the i -th per-sample gradient, R is the clipping threshold, and σ is the noise multiplier.
The private gradient is guaranteed to be DP through the sampled-Gaussian mechanism and the associated tight privacy accounting to compose over the iterations (see, e.g., Abadi et al., 2016;
Wang et al., 2019; Mironov et al., 2019; Koskela et al., 2020; Bu et al., 2020; Gopi et al., 2021, and the references therein.).
2.2 Backward propagation
We briefly introduce the back-propagation, which reveals a simple yet important difference between the gradients of weights and those of biases.
We consider a linear layer, indexed as the l -th layer, with weight W l ∈ R d × p and bias as b l ∈ R p .
We leave the derivation of other layers such as normalization and convolution in Appendix A.
We denote the mini-batched input of this layer as a l ∈ R B × T × d and the immediate output as s l ∈ R B × T × p , where B is the batch size and T is the feature dimension 4 : a l +1 = ϕ
( s l ) , s l = a l W l + b l .
Here ϕ is any non-parametric inter-layer operation, e.g. the non-linear activation (like ReLU), pooling, padding, and so on.
We write L = (cid:80) ni
=1 L i as the total loss and L i as the per-sample loss of the i -th sample.
During a standard back-propagation of L layers, the chain rule keeps track of the output gradient at each layer in a just-in-time fashion:
[Equation]
This output gradient ∂ L ∂ s l is used to compute per-sample gradient of weights and biases, ∂ L i
[Equation]
a l,i , ∂ L i ∂ b l ⊤ =
(cid:88) j ∂ L i ∂ s l,j ⊤ ∂ s l,j ∂ b l = ∂ L ∂ s l,i ⊤ 1 .
Notably, the weight gradient needs the activation tensor a l to compute an expensive O ( BTpd ) tensor multiplication.
Memory-wise, { a l } l across all layers is very costly to store (taking more than 95% memory across VGG, ResNet, DenseNet, RoBERTa, etc.
by (Jain et al., 2020, Figure 3)).
In sharp contrast, the computation of bias gradient does not need a l , and the multiplication with 1 in
Equation (4) is actually a cheap O ( BTp ) summation on ∂ L ∂ s l : B × T × p → B × p .
Differentially private Bias-Term Fine-Tuning
We propose DP-BiTFiT, to privately train only the bias terms in a neural network by combining
Equation (4) and Equation (2).
We use shaded lines to represent the additional DP operations in
Algorithm 1, and add DP-related variables and operations in red in the computation graph by Figure 2.
We provide the implementation details of DP-BiTFiT in Appendix B.
3.1 Parameter efficiency
DP-BiTFiT enjoys exactly the same parameter efficiency as the standard BiTFiT, training merely about 0.1% of the total parameters in large models.
We demonstrate that DP-BiTFiT is one of the most parameter-efficient fine-tuning through a list of models in Table 1, extended in Table 9.
[Table]
An advantage of this parameter efficiency is reflected in the computation efficiency, given that most parameters do not require gradients to be computed: we show in Table 2 and Section 3.3 that DPBiTFiT is much more efficient than full fine-tuning (DP and even non-DP).
Additionally, the parameter efficiency also translates to the communication efficiency in the distributed learning.
For example, the 64-bit communication cost of DP full fine-tuning is 64 MD where M is the number of worker and
D is the total number of parameters, which can be improved to 0 .
064 MD by DP-BiTFiT.
3.2 Complexity of weight and bias training
We present in Table 2 the complexity of DP training on weights and biases, for one layer mapping B × T l × d l to B × T l × p l .
To elaborate on Footnote 4, for text data, T l is the sequence length, d l is input dimension, and p l is output dimension; for image data and specially in a convolution layer, T l is height times width, d l is the input channels times kernel sizes, p l is the output channels (c.f.
(Bu et al., 2022a, Section 2.3)).
Notice that the total complexity of training a network is summed across all layers, e.g. the time complexity of standard full training is 6 B (cid:80) l T l p l d l , DP full fine-tuning is over 8 B (cid:80) l
T l p l d l , and DP-BiTFiT is about 4 B (cid:80)
l T l p l d l .
Therefore, our complexity analysis indicates that DP-BiTFiT is 6 / 4 = 1 . 5 × faster than non-private full fine-tuning and over 8 / 4 = 2 × faster than DP full fine-tuning.
[Table]
Here, the DP weight training (full fine-tuning) uses three efficient implementations that are equivalent mathematically but have different complexity: Opacus Yousefpour et al.
(2021), GhostClip Goodfellow (2015)
; Li et al.
(2021), and MixGhostClip Bu et al.
(2022a).
The first two implementations are illustrated in Figure 2, of which MixGhostClip is a hybridization that reduces to GhostClip when
T is small.
These implementations have been thoroughly analyzed in (Bu et al., 2022a, Appendix
C), and we take the complexity result from (Bu et al., 2022a, Table 1).
For the complexity of bias training in Table 2, it suffices to analyze Line 5 of Algorithm 1.
We refer the interested readers to
Table 5 for details, where we also apply the complexity analysis of weight training on other methods beyond full fine-tuning, including DP LoRA and DP Adapter.
3.3 Scalability of DP algorithms
From the complexity analysis in Table 2, we observe that DP training on weights can be memory costly, especially when the models are large and the data is high-dimensional.
In sharp contrast,
DP-BiTFiT is amazingly scalable since its computational overhead is negligible and independent of
T (though the total complexity, mainly due to forward and output gradient, is still linear in T ).
Efficiency of DP training v.s. feature dimension To empirically evaluate the computation efficiency of DP fine-tuning methods, we measure the time and GPU memory for a fixed batch size.
We depict the high-dimensional data issue in Figure 3, in which the memory saving and speedup by DP-BiTFiT is substantial.
We expect to observe greater efficiency advantage of DP-BiTFiT on higher dimensional data, e.g. in document-level language tasks with T ≈ 20000 by Beltagy et al. (2020), and in high-resolution image tasks, such as 1024 × 1024 CelebA-HQ Karras et al.
and
Flickr-Faces-HQ Karras et al.
(2019) where T can be of order 10 5 in the convolution layers.
[Figure]
Efficiency of DP training model size To stress-test the computation efficiency of DP-BiTFiT with large models, we apply the maximum batch size with respect to each fine-tuning method, instead of using a fixed one across different methods.
Therefore, DP-BiTFiT can further leverage its memory efficiency to achieve the best throughput.
Here we consider a setting of high-dimensional but small ResNet
) and the other setting of low-dimensional large
[Figure]
Experiments
We now test the accuracy of DP-BiTFiT on natural language and computer vision tasks, with the settings in Appendix D. For DP full fine-tuning algorithms, we use GhostClip Li et al.
(2021) on texts, and MixedGhostClip Bu et al.
(2022a) on images, which achieve SOTA efficiency and accuracy on these datasets respectively.
We report ϵ using RDP Mironov (2017).
4.1 Language tasks
We experiment on MNLI-m(mismatch)
Williams et al.
(2018), QQP Iyer et al.
, QNLI Rajpurkar et al. and SST2 datasets Socher et al. (2013).
Competitive algorithms include reparameterized gradient perturbation (RGP, Yu et al.
(2021b))
, LoRA, Adapter and Compacter Yu et al.
(2021a).
We use the same setup as Li et al.
(2021) on RoBERTa models, only increasing the learning rate for DP-BiTFiT. Additional results with different clipping functions and under a stronger privacy guarantee ϵ = 3 can be found in Table 10.
In Table 3, DP-BiTFiT is highly parameter efficiency and on-par with other DP fine-tuning in terms of accuracy.
As indicated by Figure 1 and Figure 3, over 2 × speedup and over 3 × memory saving is observed, when switching from DP full fine-tuning to DP-BiTFiT across datasets.
We also provide the evidence that DP-BiTFiT is performant, even outperforming DP full fine-tuning, on the natural language generation.
We refer interested readers to Table 11 where we apply GPT2 to
E2E dataset, under the same setting as Li et al.
(2021); Bu et al.
(2022b).
[Table]
Additional
[Table]
to networks
[Table]
Image classification
We further experiment with DP-BiTFiT on CIFAR10/CIFAR100 ( 32 × 32 pixels, resized to 224 × 224 ) and CelebA ( 218 × 178 pixels, not resized) after pre-training on ImageNet ( 224 × 224 pixels).
Unlike language tasks, DP-BiTFiT can be less satisfactory, e.g. inducing 30% test accuracy gap in CelebA [Smiling] classification in Table 12, though this gap can often be closed by a two-phase training , denoted as X + BiTFiT: we firstly apply DP full fine-tuning for X epochs then DP-BiTFiT for the rest of training.
Hence, X +BiTFiT becomes DP full fine-tuning when X equals total epochs, and reduces to DP-BiTFiT when X = 0 .
Empirically speaking, it suffices to use X ≤ 2 to achieve comparable accuracy to full fine-tuning, while still enjoying the speedup.
The effectiveness of two-phase training is verified in Table 4 and Appendix E.3.
1+BiTFiT outperforms previous SOTA by DP full fine-tuning
Bu et al.
(2022a) that used BEiT-large: CIFAR10 97 .
1% → 98 . 8% ; CIFAR100 86 .
2% → 88 .
7% , under ϵ = 2 .
2+BiTFiT is comparable to previous SOTA, 87 .
58% → 86 .
71% on
CelebA in Table 12, under ϵ = 3 / 8 .
CIFAR10 DP-BiTFiT 1+BiTFiT 2+BiTFiT DP full ϵ =11.
ϵ =
[Table]
Discussion
In this work, we study DP-BiTFiT to privately train only the bias terms of neural networks.
The highlight of DP-BiTFiT is the accuracy, the parameter efficiency and the computation efficiency, which is realized by not forward caching the activation tensors, and not back-propagating the gradient of weights.
This consequently allows DP-BiTFiT to be as fast and memory-saving as its non-private counterpart, thus particularly suitable for large models and high-dimension data.
For future directions, DP-BiTFiT can be readily combined with prefix-based tuning and weights -based fine-tuning, e.g. DP Adapter+BiTFiT and DP LoRA+BiTFiT, via f ( x ; W 0 , ˆ b , ˆ θ ) using the notation in Equation (1).
Specifically, such combination can decide whether weight and/or bias should be fine-tuned in a layer-wise manner .
For instance, we can optimize only the embedding layer (which has no bias terms) and all bias terms in other layers.
We expect this interpolating approach between full fine-tuning and BiTFiT, in parallel to our two-phase training, to circumvent the limitation that
DP-BiTFiT is sometimes sub-optimal on small models or difficult tasks.
A Detailed analysis of back-propagation
We rigorously analyze the neural network represented in Section 2.2: for sample index i ∈ [ B ] ,
[Equation]
(cid:124)(cid:123)(cid:122)(cid:125)
[Equation]
[Equation]
l (cid:124)(cid:123)(cid:122)(cid:125)
R 1 × p [Equation]
[Equation]
Then the per-sample weight gradient is given by the chain rule as
[Equation]
in which the second equality holds when there is no parameter sharing (so that each per-sample loss only depends on i -th input and output).
The last equality holds for the same reason.
Similarly, we have the per-sample bias gradient as
[Equation]
We additionally demonstrate that bias gradient is independent of the input a l , on the convolution (1d/2d/3d) and the normalization layers.
For the convolution, s l is the inversely folded output and a l is the unfolded input, then the forward pass is the same as that of linear layer in Equation (5).
Notice that T is the product of hidden feature dimension (c.f. Bu et al.
(2022a)), which depends on the padding, kernel sizes, strides, etc.
For the batch, layer, group, and instance normalization, the forward pass is
[Equation]
[Equation]
which can be analyzed similarly to that of Equation (5).
B Implementation of DP-BiTFiT
In this section we describe the implementation of DP-BiTFiT, which only uses Pytorch backward hook but not the forward hook, and thus is different from existing packages such as FastGradClip
Lee & Kifer (2020), Opacus Yousefpour et al.
(2021), Private Transformers Li et al.
, Private
CNN Bu et al.
(2022a).
Notice that in these packages, the forward hook is used to store the activation tensor a l for all layers, which incurs huge memory burden.
The Pytorch backward hook is a function, to be registered on a torch Module (or a layer in the neural network), that will be executed in the backward propagation.
The backward hook automatically extracts the input gradient ∂ L ∂ a l and the output gradient ∂ L ∂ s l of the layer.
In DP-BiTFiT, we call register_backward_hook to register a backward hook for Line 5 of
Algorithm 1.
An example for a linear layer: R B × T × d
→ R B × T × p looks like
[Table]
Here the attribute norm_sample stores the per-sample gradient norm (cid:13)(cid:13)(cid:13)
∂ L i ∂ b l (cid:13)(cid:13)(cid:13) F , and the attribute grad_sample stores the R B × p per-sample gradient of bias.
Then the implementation of DP-BiTFiT for one iteration looks like
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
[Figure]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where biases is the collection of all bias terms in all layers.
C Complexity analysis
We provide more details on analyzing the time and space complexity.
The analysis for full fine-tuning has been presented in (Bu et al., 2022a, Appendix C) and is adapted here for the parameter efficient fine-tuning: for example, Adapter Houlsby et al.
(2019) uses two matrices W down ∈ R p × r , W up ∈ R r × p that constitute x ←− x + GeLU ( x · W down ) W up
Hence the complexity, in comparison to full-finetuning, changes by replacing d → 2 r .
LoRA Hu et al.
(2021) also uses two matrices W down ∈ R d × r , W up ∈ R
r × p that constitute x ←− x · W + x · W down W up
Hence the complexity, in comparison to full-finetuning, changes by replacing pd → r ( p + d ) .
[Table]
4 BTpd 2 BTpd +2 BTpd +2 BT (
pr + dr ) +
BTpr
BTp +3 Bp
[Table]
For per-sample bias gradient clipping, we need ∂ L i ∂ b l ⊤ =
∂ L ∂ s l,i ⊤ 1 in Equation (4), which consists of the per-sample gradient instantiation (i.e. summation along the feature dimension, from R Tp
→ R p , ∂ L ∂ s l,i → ∂ L i ∂ b l ), and computing the per-sample gradient norm (i.e. taking the square at each index and summing all indices ).
Here each operation in italic takes Bp time complexity, meaning the total time complexity is 3 Bp , but the space complexity is Bp if operated in-place.
D Experiment details D.1 Language tasks
Throughout this work, the text datasets are processed and loaded from Huggingface Lhoest et al.
We follow the same setup as Li et al.
(2021); Bu et al.
(2022b), such as δ = 0 .
5 × sample size.
The full fine-tuning is implemented by Private Transformers codebase, version 0.2.0 (i.e. GhostClip algorithm Li et al.
For text classification, we experiment on four datasets: MNLI(m) , the matched splits from MultiGenre Natural Language Inference Corpus; QQP , the Quora Question Pairs2 dataset; QNLI
The
Stanford Question Answering dataset; SST2 The Stanford Sentiment Treebank dataset.
To give a fair comparison, we use the same optimizer as in Li et al.
(2021), i.e. DP-Adam with
Abadi’s clipping.
[Table]
For E2E generation task, we experiment GPT2 models using the same optimizer as in Bu et al. (2022b), using DP-AdamW with automatic clipping.
[Table]
D.2 Image tasks
We give the experiments settings for image classification.
For CIFAR10 and CIFAR100, we use the same setting as Bu et al.
(2022a), e.g. 5 epochs for CrossViT/ViT and 3 epochs for BEiT-large.
For
CelebA, we use the same setting as Bu et al.
(2022b), e.g. 10 epochs.
We use DP-Adam with Abadi’s clipping.
We do not apply tricks such as random data augmentation, weight standardization Qiao et al.
(2019), or parameter averaging Polyak & Juditsky (1992).
Our experiments are heavily based on Private CNN (i.e. MixGhostClip algorithm Bu et al.
(2022a))
and
TIMM codebases.
[Table]
E
E.1 Parameter efficiency of DP-BiTFiT
[Table]
E.2 More results on DP-BiTFiT and language tasks
[Table]
AUTO DP Abadi DP AUTO standard DP Abadi DP AUTO DP Abadi DP AUTO ϵ = ∞
ϵ =
ϵ = ϵ =
ϵ = 3 ϵ = ∞
ϵ =
ϵ = ϵ =
ϵ =
Accuracy SST 94.
Accuracy QNLI 91.
Accuracy QQP 87.
Accuracy MNLI-m 85.
RoBERTa-large standard DP Abadi DP AUTO DP Abadi DP AUTO standard DP Abadi DP AUTO DP Abadi DP AUTO ϵ = ∞
ϵ = ϵ =
ϵ =
ϵ = 3 ϵ = ∞
ϵ =
ϵ = ϵ =
ϵ =
Accuracy SST 96.
Accuracy QNLI 93.
Accuracy QQP 87.
Accuracy MNLI-m 90.
Same setting as Appendix D
[Table]
E.3 More results on two-phase training
[Table]