P UBLIC D ATA -A SSISTED M IRROR D ESCENT FOR P RIVATE M ODEL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we revisit the problem of using public data to improve the privacy/utility trade-offs for differentially private (DP) model training.
Here, public data refers to auxiliary data sets that have no privacy concerns.
We consider public training data sets that are from the same distribution as the private training data.
For convex losses, we show that a variant of Mirror Descent provides population risk guarantees which are independent of the dimension of the model ( p ).
Specifically, we apply Mirror Descent with the loss generated by the public data as the mirror map , and using DP gradients of the loss generated by the private (sensitive) data.
To obtain dimension independence, we require G 2 Q ≤ p public data samples, where G Q is the Gaussian width of the smallest convex set Q such that the public loss functions are 1-strongly convex with respect to (cid:107) · (cid:107) Q .
We further show that our algorithm has a natural “noise stability” property: If in a bounded region around the current iterate, the public loss satisﬁes α v -strong convexity in a direction v , then using noisy gradients instead of the exact gradients shifts our next iterate in the direction v by an amount proportional to 1 /α v (in contrast with DP stochastic gradient descent (DP-SGD), where the shift is isotropic).
Analogous results in prior works had to explicitly learn the geometry using the public data in the form of preconditioner matrices.
Our method is also applicable to non-convex losses, as it does not rely on convexity assumptions to ensure DP guarantees.
We demonstrate the empirical efﬁcacy of our algorithm by showing privacy/utility trade-offs on linear regression, deep learning benchmark datasets (WikiText-2, CIFAR-10, and EMNIST).
We show that our algorithm not only signiﬁcantly improves over traditional DP-SGD, which does not have access to public data
but also improves over DP-SGD on models that have been pretrained with the public data to begin with
I NTRODUCTION
Differentially Private Stochastic Gradient Descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016), and its variants (Kairouz et al., 2021b) have become the de facto standard algorithms for training machine learning models with differential privacy (DP) (Dwork et al., 2006).
While DP-SGD is known to be optimal in terms of obtaining both optimal excess empirical risk (Bassily et al., 2014), and excess population risk (Bassily et al., 2020b) for convex losses, the obtained error guarantees suffer from an explicit polynomial dependence on the model dimension ( p ).
This polynomial dependence signiﬁcantly impacts the privacy/utility trade-off when p ≥ n priv , where n priv is the number of private training samples.
Thus, even empirically, when DP-SGD is used to train large deep learning models, there is a signiﬁcant drop in accuracy compared to the nonprivate counterpart (Papernot et al., 2020).
In this paper, we revisit the problem of effectively using public data (i.e., data drawn from the same distribution as the private training data set, but without privacy concerns) to improve the privacy/utility trade-offs for DP model training.
Speciﬁcally, we design a central DP variant of mirror descent (Nemirovsky & Yudin, 1983) that uses the loss function generated by the public data as the mirror map and DP gradients on the private/sensitive data as the linear term, ensuring population risk guarantees for convex losses with no explicit dependence on dimensions as long as n pub ≥ p , where n pub is the number of records in the public data set.
We show both theoretically and empirically that our DP variant of mirror descent, assisted with public data, can improve the privacy-utility trade-offs by effectively reducing the variance in the noise added to the gradients in DP model training.
Our empirical results are on standard benchmark data sets like CIFAR-10, EMNIST, and WikiText-2.
Learning Geometry with Mirror Maps:
Common to most DP model training algorithms, including DP-SGD, DP-FTRL (Kairouz et al., 2021b), and our algorithm, is a DP estimator of the gradient of the loss ∇ θ L ( θ t ; D priv ) =
(cid:80) d ∈ D priv ∇ θ (cid:96) ( θ t ; d ) generated by the private data set D priv at a given model state θ t ∈ R p .
This DP estimator essentially adds isotropic Gaussian noise N (0 , σ 2 I p ) to ∇ θ L ( θ t ; D priv ) , where σ depends on the privacy parameters ( ε, δ ) and the maximum allowable value of (cid:107)∇ θ (cid:96) ( θ t ; d ) (cid:107) 2 (a.k.a.
the clipping norm (Abadi et al., 2016)).
1 It is well known that for most learning tasks, the set of gradients vectors in L ( θ t ; D priv ) are seldom isotropic (Gur-Ari et al., 2018; Agarwal et al., 2019).
Hence, it is natural to wonder if the Gaussian noise in the DP estimator can be made to respect the geometry of the gradients.
Prior works (Zhou et al., 2020; Asi et al., 2021; Kairouz et al., 2021a) have used public data ( D pub ) to explicitly learn this geometry, mostly in the form of preconditioner matrices (Duchi et al., 2011) to be multiplied to the estimated noisy gradients.
In this paper, we take an implicit approach towards respecting this geometry, by using the loss L ( θ ; D pub ) generated by the public data as the mirror map in classical mirror descent.
As a ﬁrst order approximation (formalized in Section 4), one can view it as doing DP-SGD on L ( θ ; D priv ) while using L ( θ ; D pub ) as a regularizer.
This approach has the following advantages: (i) The information of the geometry is “free”, i.e., one does not need to learn the preconditioner explicitly from the public data, (ii) Unlike prior works (Zhou et al., 2020; Kairouz et al., 2021a), one does not need to assume that the gradients of L ( θ ; D priv ) lie in a ﬁxed rank subspace, (iii)
The achieved excess population risk guarantees have better dependence on n pub =
| D pub | compared to prior results (Asi et al., 2021), and (iv) Because the geometry is implicitly deﬁned, the implementation does not need to maintain an additional data structure for the preconditioner, and hence is much easier to implement.
Empirically, under our best-effort comparison, our baseline algorithm improves over the state of the art (Asi et al., 2021).
We note that differentially private mirror descent has been considered before by Talwar et al.
(2014) and Wang et al. Their results are not directly comparable to ours because (i) they do not have access to in-distribution public data (ii) as shown in Bassily et al. (2014), without public data, it is impossible to achieve the dimension independent bounds we achieve (iii) in our experiments we solve unconstrained optimization problems, but those works choose the mirror map based on the constraint set rather than the data set.
We note that the utility bounds we prove in this paper also apply to a public data-assisted variant of the accelerated mirror descent algorithm considered in Wang et al.
In-distribution vs. Out-of-distribution Public Data: Prior works have considered both settings where the public data set D pub comes from the same distribution as the private data D priv (a.k.a. indistribution ) (Bassily et al., 2018a;
Zhou et al., 2020; Kairouz et al., 2021a; Asi et al., 2021), and where the distributions are different (a.k.a. out-of-distribution ) (Abadi et al., 2016; Papernot et al., 2016; 2018; Liu et al., 2021).
In principle, our algorithm can be used in out-of-distribution settings , but our results in this paper are for the in-distribution case.
In the in-distribution setting, it is typical that there are fewer public data samples available than private data samples – i.e., n pub (cid:28) n priv – as it is harder to obtain public data sets than ones with privacy constraints attached.
In-distribution public data could come from either altruistic opt-in users (Merriman, 2014; Avent et al., 2017) or from users who are incentivized to provide such data (e.g., mechanical turks).
Out-of-distribution public data may be easier to obtain but can have various degrees of freedom; e.g., the domains of private and public data may not be identical, the representation of some classes may vary, the distributions can be mean shifted, etc.
It is usually hard to quantify these degrees of freedom to the extent that we can provide precise guarantees.
Hence, we leave this aspect for future exploration, and work with the idealized assumption that the public data comes from the same distribution as the private data, or, at least, that the differences between these two distributions are not material.
Design of Algorithms without Relying on Convexity for Privacy: While most of our theoretical utility guarantees are speciﬁc to convex functions, our algorithm can be used seamlessly in nonconvex settings.
The main reason is that its DP guarantee does not rely on convexity.
Prior work that provided similar dimension-independent excess population risk guarantees under the the same conditions as ours (i.e., n pub ≥ p ) (Feldman et al., 2018) heavily relied on convexity to ensure DP and hence, cannot be used in non-convex settings.
Choice of Benchmark for Empirical Comparison: Mirror descent (Nemirovsky & Yudin, 1983; Hazan, 2019) as a ﬁrst step optimizes the mirror map function.
In our setting, this corresponds to pre-training on the public loss function L ( θ ; D pub ) before running the DP optimization procedure on L ( θ ; D priv ) .
Since pre-training on public data is intuitive and easy, we always compare to DP-SGD (and its variants) that have been pre-trained to convergence with the public loss.
We show that our algorithm outperforms even pre-trained DP-SGD.
To the best of our knowledge, ours is the only empirical work that compares to this strong (but fair) benchmark.
Other Uses of Public Data in DP Learning: The use of in-distribution public data has been extensively explored both theoretically and empirically.
On the theoretical side, it has been shown (Alon et al., 2019; Bassily et al., 2020a) that a combination of private and public data samples can yield asymptotically better worst-case PAC learning guarantees than either on their own.
Another line of work (Papernot et al., 2016; 2018; Bassily et al., 2018b; Dwork & Feldman, 2018; Nandi & Bassily, 2020) considers public data that is unlabelled, but otherwise comes from the same distribution as the private data; the primary goal is to use the private data to generate labels for the public data, which can then be used arbitrarily.
So far only two papers have considered out-of-distribution data.
Bassily et al.
(2020c) assume that whether a data record is public or private depends on its label; e.g., the public data may contain many negative examples, but few positive examples.
They show that halfspaces can be learned in this model.
Liu et al. (2021) consider synthetic data generation and provide guarantees that depend on the R ´ enyi divergences between the public and private distributions. Abadi et al.
and Tramer & Boneh (2020) provided techniques to effectively use out-of-distribution public data for pre-training for DP-SGD.
However, they did not consider techniques to improve a pre-trained model using private and public data, which is the focus of our work.
1. P ROBLEM F ORMULATION
Consider the classic differentially private stochastic convex optimization (DP-SCO) (Chaudhuri et al., 2011; Bassily et al., 2014; 2019; 2020b) setting.
Let τ be a distribution over a ﬁxed domain D .
Given a data set D ∈ D ∗ drawn i.i.d.
from τ , and a loss function (cid:96) priv : R p × D
→ R , the objective is to approximately solve arg min θ ∈C
E d ∼ τ [ (cid:96) priv ( θ ; d )] , while preserving DP.
Here, C ⊆ R p is the constraint set.
Usually one solves the SCO problem via empirical risk minimization (ERM), i.e., θ priv ∈ arg min θ ∈C
L ( θ ; D ) , where L ( θ ; D ) =
| D | (cid:80) d ∈ D (cid:96) priv ( θ ; d ) , and then uses θ priv as a proxy.
Up to a dependence on dimensionality p , in the DP setting, a direct translation from ERM to the SCO setting provides the optimal rates (Bassily et al., 2014; 2019; 2020b).
Since we will strive for dimension-independent convergence rates, we thus safely ignore the issue of non-optimality due to dimensionality, and treat the ERM and SCO problems as almost equivalent.
In this paper, we consider the DP-SCO setting with heterogeneous data , where there are two data sets D priv (with n priv samples) and D pub (with n pub samples) drawn i.i.d.
from the same distribution .
The private data set D priv consists of records from individuals who require privacy protection, whereas the public data set D pub does not require any privacy protection.
Since obtaining such data can be expensive, for our empirical evaluation, we assume n pub is a small fraction of n priv (e.g., < 5%).
We will also use a separate public loss function (cid:96) pub .
For our theoretical analysis, we will assume (cid:96) priv is convex and L -Lipschitz, whereas (cid:96) pub is strongly convex (both with respect to the (cid:96) 2 -norm).
In practice, one will likely choose (cid:96) priv = (cid:96) pub , but clip the gradients of (cid:96) priv for privacy.
In general, (cid:96) pub can be arbitrary, but for the analysis, we will assume that (cid:96) pub and (cid:96) priv roughly share a minimizer and that ∇ (cid:96) pub ( θ ; d ) has bounded variance (see Assumption 3.1 for a formal statement).
We refer the reader to Appendix A for a reference for the notation used throughout the paper.
1. O UR C ONTRIBUTIONS
Dimension Independent Excess Population Risk for Constrained ERMs: The standard private SCO excess empirical loss bound for a constraint set C is: O
[Equation]
, where L is the Lipschitz constant of the loss function and n is the number of samples in the dataset.
If we pre-train on public data and are able to ﬁnd a solution within distance O (cid:16) 1 √ p (cid:17) of the optimal solution, we can set C to be the ball of radius O (cid:16) 1 √ p (cid:17) centered on the pre-trained model and obtain a similar dimension-independent error rate.
There are a couple of issues with this approach: (i) It bakes in the utility assumption about the constraint set into the algorithm.
This is unsatisfactory because if we are unlucky (due to stochasticity or choice of hyperparameters) and the optimal solution does not lie within this ball, using this choice of C may restrict us to a range of poor solutions that we cannot escape even if the number of private samples is very large, and (ii) It forces one to operate with projected gradient descent which may signiﬁcantly impact accuracy in settings where the utility assumptions may not hold, e.g., in deep learning.
We improve upon this by showing that given n pub public samples in addition to n priv private samples, using our PDA-DPMD ( Public Data-Assisted Differentially Private Mirror Descent ) algorithm, a variant of mirror descent that implicitly reshapes the noisy gradients according to the public loss function, we can achieve excess empirical loss O
[Equation]
The bound is proven in Theorem 3.2.
Here, V is a measure of the variance of the gradients of (cid:96) pub , and G Q is the Gaussian width of the smallest set Q such that the public loss functions are 1-strongly convex with respect to (cid:107) · (cid:107) Q .
In particular, G Q is at most √ p if the public loss functions are 1-strongly convex, but can be constant if e.g. the public loss functions have a much larger strong convexity parameter in all but a constant number of basis directions.
Note that if we have n pub =
G 2 Q public samples, then the 1 / √ n pub and G Q terms cancel out, i.e. this error bound has no explicit dependence on dimension.
Using standard techniques, we can turn this into a dimension-independent excess population loss bound (see Theorem 3.5), again assuming n pub ≥ G 2 Q .
In addition, to the best of our knowledge, ours is the ﬁrst work on augmenting private training with public data to show a theoretical improvement over DP-SGD (here the dependence 1 √ n pub ) due to pre-training on public data.
In particular, we show pre-training improves the standard DP-SGD bounds even under a totally isotropic geometry.
Local Noise-stability of Mirror Descent: We show that in addition to achieving better excess population loss bounds, PDA-DPMD has the following “local noise-stability” property: If in a bounded region around the current iterate, the public loss satisﬁes α v -strong convexity in a direction v , then using noisy gradients instead of the exact gradients shifts our next iterate in the direction v by an amount proportional to 1 /α v (see Theorem 3.6).
That is, PDA-DPMD effectively rescales the effect of adding noise in any direction to be inversely proportional to the curvature in that direction.
Note that this is in spite of the fact that for privacy, the noise we add to the gradients is usually isotropic.
Furthermore, this rescaling is done purely as a function of the public loss function used by PDA-DPMD.
In other words, a practitioner implementing PDA-DPMD simply needs to choose an appropriate loss function, and PDA-DPMD will “automatically” rescale the effects of noise to match the loss function’s curvature.
Empirical Evaluation: On both synthetic and real-world benchmark data sets, we show that PDADPMD outperforms DP-SGD, even when they are pre-trained on the public data set.
We provide two sets of experiments.
First, a linear regression on a synthetic data set which closely matches the utility assumptions in the theoretical analysis.
Second, we provide results on standard deep learning benchmark data sets (WikiText-2, CIFAR-10, and EMNIST).
In Section 3, we consider using DP-SGD and PDA-DPMD to solve a least squares linear regression problem on a synthetic data set generated via the process
b i ∼ N ( (cid:104) a
i , θ ∗ (cid:105) , σ 2 ) , where θ ∗ ∈
R p is the true model.
The feature vectors a i ’s are drawn i.i.d.
from some ﬁxed distribution.
We ﬁx the number of private data samples, and set the number of public samples to be a ﬁxed constant times the dimension ( p ).
We observe that as expected, public data allows us to substantially improve the error in two ways: (i) Pre-training : DP-SGD initialized from a model pre-trained on public data has nearly-constant mean-squared error, whereas DP-SGD from a random initialization has meansquared error scaling with the dimension, and (ii) Adapting to geometry using public loss : While DP-SGD initialized from a pre-trained model already achieves near-constant loss, we also observe that PDA-DPMD outperforms DP-SGD due to its error’s dependence on the Gaussian width G Q rather than the dimension.
We note that the observed improvement is “automatic” once we choose the mean-squared error to be the loss function.
We also conduct experiments in Section 4 on two real world tasks: next word prediction on WikiText-2, and image classiﬁcation on CIFAR-10 and EMNIST (ByMerge split).
We consider 4% of the original training data as public and pretrain on it.
On all datasets, we can observe that an approximate version of PDA-DPMD outperforms DP-SGD in terms of test loss.
On CIFAR-10, the improvement is more than 5%; on EMNIST, 7%; on WikiText-2, log perplexity is improved by more than 0.3%, which is a notable improvement for perplexity.
For the deep learning experiments, running PDA-DPMD was computationally expensive.
We derive a ﬁrst-order approximation that can be viewed as regular DP-SGD on a convex combination of private and the public losses, i.e., α ∇L ( θ ; D priv )+(1 − α )
∇L ( θ ; D pub ) , α ∈ [0 , 1] , where ∇L ( θ ; D priv ) is privatized by clipping and adding noise.
This approximation makes the running time of our algorithm comparable to DP-SGD when run on the data set D priv ∪ D pub .
B ACKGROUND
Differential Privacy: Differential Privacy (DP) – originally deﬁned by Dwork et al.
(2006) – is a formal method for quantifying the privacy leakage from the output of a data analysis procedure.
A randomized algorithm M : D ∗ → Y is ( ε, δ ) -DP
if, for all neighbouring dataset pairs D, D (cid:48) ∈ D ∗ and all measurable sets of outputs S ⊆
Y , we have
[Equation]
We deﬁne two datasets to be neighbouring if they differ only by the addition or removal of one person’s record .
We ensure differential privacy by adding Gaussian noise to functions of bounded sensitivity.
In particular, if (cid:96) is L -Lipschitz in its ﬁrst parameter, then (cid:107)∇ θ (cid:96) ( θ ; d ) (cid:107) 2 ≤ L for all θ and d ∈ D .
Thus adding noise drawn from N (0 , σ 2 · I p ) to the sum (cid:80) i ∇ θ (cid:96) ( θ, d i ) over people’s records satisﬁes DP, where σ scales with L and the desired privacy parameters.
The composition and postprocessing properties of differential privacy ensure that, as long as each step in our iterative algorithm satisﬁes differential privacy, then so does the overall system.
We refer the reader to (Dwork & Roth, 2014) for further details of the standard privacy analysis of algorithms like ours.
Due to space constraints, we defer background on Mirror Maps and Gaussian Width to Appendix B.
P UBLIC D ATA A SSISTED D IFFERENTIALLY P RIVATE M IRROR D ECENT
In this section, we present our algorithm, PDA-DPMD, dimension-independent excess empirical and population loss bounds for PDA-DPMD (Theorems 3.2 and 3.5), and a “noise stability” property (Theorem 3.6).
We defer all the proofs in this section to Appendix C.
Algorithmic Description: Our main algorithm, Public Data-Assisted Differentially Private Mirror Descent (PDA-DPMD), is given as Algorithm 1.
PDA-DPMD is a variant of mirror descent using noisy gradients, but we also pre-train on public data and use the public loss as our mirror map Ψ .
: Public/private datasets D pub , D priv sizes n pub , n priv , private/public loss functions (cid:96) priv , (cid:96) pub privacy parameters ( ε, δ ) , number of iterations T , learning rate η : { 0 , 1 , . .
, T − 1 } → R + , constraint set: C , clipping norm L : an upper bound on max θ ∈C (cid:107)∇ (cid:96) priv ( θ ) (cid:107)1: Ψ( θ ) :
= 1 n pub (cid:80) d ∈ D pub (cid:96) pub ( θ ; d ) , θ 0 ← arg min θ ∈C Ψ( θ ) , σ 2 ← 8 L 2 T log(1 /δ )
( εn priv )2: for do 3: g t
1 n priv (cid:80) d ∈ D priv ∇ (cid:96) priv ( θ ; d ) 4: θ t +1 ← arg min θ ∈C [ η t (cid:104) g
t b t , θ (cid:105) + B Ψ
( θ, θ t )] , where b t ∼ N (0 , σ 2 · I p ) 5: end for 6: return = 1 T (cid:80) Tt
=1 θ t
Note that Line 4 of PDA-DPMD is equivalent to the following: Choose θ t +1 / 2 to be the point such that ∇ Ψ( θ t +1 / 2 ) = ∇ Ψ( θ t ) − η ( g t + b t ) , and then use the Bregman projection θ t +1 = arg min θ ∈C B Ψ ( θ, θ t +1 / 2 ) .
Intuitively, PDA-DPMD is similar to DP-SGD, with the main difference being we apply the gradient steps to ∇ Ψ( θ ) rather than to θ itself.
Note that PDA-DPMD reshapes the gradient and noise automatically given (cid:96) pub and D pub .
In contrast, e.g., private AdaGrad implementations (Kairouz et al., 2020; Asi et al., 2021) assume knowledge of the geometry of the loss function has already been learned prior to running their algorithms.
Also, for an appropriate choice of Ψ , one can recover an algorithm that projects the private gradients to a low-dimensional subspace as in the algorithms of Zhou et al.
(2020) and Kairouz et al.
Dimension Independent Empirical Risk Bounds for Convex Losses: We bound the excess empirical loss on the public loss function (cid:96) pub , compared to the private population minimizer θ ∗ (rather than the empirical minimizer).
This is because the empirical minimizer θ emp of the private loss function could be far away from θ ∗ , and in turn ∇ Ψ( θ emp ) could be much larger in expectation than ∇ Ψ( θ ∗ ) .
Our PDA-DPMD excess empirical loss bound will be in terms of ∇ Ψ( θ ) , where θ is the point we are comparing to, so it is preferable to use θ = θ ∗ for this reason.
We will use the following “bounded variance” assumption on the distribution of the datasets and the public loss function:
Assumption 3.1.
For some minimizer θ ∗ ∈ arg min θ ∈C E d ∼ τ [ (cid:96) priv ( θ ; d )] we have that θ ∗ is also the minimizer of E d ∼ τ [ (cid:96) pub ( θ ; d )] in C and E d ∼ τ (cid:104) (cid:107)∇ (cid:96) pub ( θ ∗ ; d ) − E d ∼ τ [ ∇ (cid:96) pub ( θ ∗ ; d )] (cid:107) 2 2 (cid:105) ≤ V 2 .
In particular, this implies
[Equation]
We note that while Assumption 3.1 is written as generally as possible and thus captures scenarios where (cid:96) pub and (cid:96) priv could potentially be very different loss functions, the reader can think of them as differing only slightly.
Indeed, Assumption 3.1 captures several scenarios we might see in practice, such as (i) (cid:96) pub =
(cid:96) priv (which can occur if (cid:107)C(cid:107) 2 is small), (ii) (cid:96) priv is the clipped version of (cid:96) pub (see e.g., Song et al. (2021) for a discussion on the effects of clipping on the loss function), and (iii) (cid:96) pub is (cid:96) priv but with a regularizer added.
Our empirical loss bound now follows by using Assumption 3. to control the Bregman divergence between the initial iterate and the population minimizer:
Theorem 3.2.
Suppose the private loss function L := [Equation] is L -Lipschitz and convex.
Suppose (cid:96) pub is α -strongly convex, and let Q be the minimal convex body containing the origin such that each (cid:96) pub ( θ ; d ) is 1 -strongly convex with respect to the Minkowski norm (cid:107) · (cid:107) Q (deﬁned as (cid:107) x (cid:107) Q = min { c ∈ R ≥
| x ∈ cQ } ).
Then PDA-DPMD is ( ε, δ ) -differentially private with respect to the private database D priv and choosing η t = η for all t we have:
[Equation]
The above bound is scale-invariant, so to simplify the presentation of this section, we assume, without loss of generality, that α = 1 (this also implies Q is contained within the unit (cid:96) 2 -ball, i.e. (cid:107) Q (cid:107) 2 ≤ 1 ).
By rescaling Ψ and η appropriately, we do not affect the behavior of PDA-DPMD, but get that Ψ is 1-strongly convex.
By chaining the following lemma with Theorem 3.2, we get an excess empirical loss bound with respect to the sample minimizer rather than the population minimizer as desired.
Lemma 3.3.
Let τ be a distribution over D , (cid:96) :
C × D → R be a function such that (cid:96) ( θ ; d ) is L -Lipschitz and convex in θ for any ﬁxed d ∈ supp ( τ ) .
Let θ ∗ be the minimizer of E d ∼ τ
[ (cid:96) ( θ ; d )] .
Then, we have
[Equation]
The lemma follows by using convexity and a bound on the expected (cid:96) 2 -norm difference between the gradient of the empirical and population losses.
Excess Population Risk of PDA-DPMD:
We now translate our excess empirical loss bound to a excess population loss.
We use Lemma F.5 of Bassily et al.
(2014), restated in Lemma 3.4 for convenience, which provides a black-box translation from empirical loss to population loss:
Lemma 3.4.
For any ( ε, δ ) -DP algorithm for minimizing 1 n priv (cid:80) d ∈ D priv (cid:96) ( θ ; d ) over C , the expected excess population loss exceeds the expected excess empirical loss by O ( L (cid:107)C(cid:107) 2 ε + (cid:107)C(cid:107) 22 δ ) .
Given this lemma, it is straightforward to derive excess population loss bounds:
Theorem 3.5.
For η =
[Equation]
, and setting ε
[Equation]
, the expected population loss of PDA-DPMD is
[Equation]
Theorem 3.5 follows immediately from Theorem 3.2, Lemma 3.3, and Lemma 3.4.
Note that if n pub ≥ G 2 Q , which is at most p , then the above bound has no explicit dependence on dimension.
For comparison, if we were to only train on public data, the standard non-private excess population loss bound has dependence O (1 / √ n pub ) (and no dependence on dimension).
So in the regime where n pub ≈ G 2 Q and n priv (cid:29) n pub , our bound is asymptotically much better than the baseline of training only on public examples.
In Appendix D we show what values Q and G Q take on for some standard stochastic convex optimization problems to help the reader understand the bound in Theorem 3.5.
Local Stability Properties of PDA-DPMD:
If the public loss function has a Hessian everywhere that has the same eigenvectors regardless of location (but perhaps different eigenvalues), we can bound the impact of adding noise to the gradient on each update in mirror descent:
Theorem 3.6.
Suppose for the public loss function Ψ , its Hessian is deﬁned everywhere, and for a ﬁxed orthonormal basis { v i } i , the Hessian at every point can be written as (cid:80) i
w i ( θ ) v i v (cid:62) i for scalar functions w i : R p
→ R + such that for all i, θ , we have w i ( θ ) ≥ α .
Fix an iteration t as well as private gradient g t in PDA-DPMD.
Let θ ∗ be the value of θ t +1 after performing the mirror descent update with b t = 0 at iteration t , and let { (cid:101) w i } i , c ≥ 0 be such that for each i the smallest value of w i ( θ ) in the ellipsoid E :=
( (cid:80) i 1 (cid:101) w i v i v (cid:62) i ) B R (where B R is the (cid:96) 2 ball of radius
R :=
η (1 + c ) √ pσ ), centered at θ ∗ , is at least (cid:101) w i .
Then for any (unit) direction v = (cid:80) i a
i
v
i ,
[Equation]
where ˆ θ is the value of the next iterate θ t +1 if noise is added.
At a high level, Theorem 3.6 follows by observing that ˆ θ − θ ∗ can be expressed as the inverse Hessian of Ψ evaluated at some point (cid:101) θ times the noise vector b t .
If the magnitude of b t is small enough, (cid:101) θ stays within E and we can lower bound the inverse Hessian by 1 (cid:102)
w i v i v (cid:62) i .
In the low-probability event b t ’s magnitude is large enough, we can instead lower bound the inverse Hessian by 1 α I p .
Note that the condition w i ( θ ) ≥ α can be enforced by adding an (cid:96) 2 -regularizer to the public loss function (since mirror descent only cares about differences in the gradients of the public loss function, the private training phase of PDA-DPMD behaves the same regardless of where this regularizer is centered).
In contrast for DP-SGD, E (cid:104) |(cid:104) ˆ θ − θ ∗ , v (cid:105)| (cid:105) =
(cid:112) 2 /π · ησ for any direction v .
To corroborate our theoretical results with empirical validation, we consider the linear regression problem with mean squared error loss: 1 n priv (cid:107) X θ − y (cid:107) 22 .
We vary the dimensionality of the problem p from 500 to 6000.
For each p , we generate 10,000 private samples and 1 .
5 p public samples.
The optimal θ ∗ is sampled from N (0 , I p ) .
To introduce a non-isotropic geometry, we sample the feature vector x i such thatof the ﬁrst p/ 5 features and 80 of the last 4 p/ 5 features, chosen uniformly at random, are set to 0 .
05 , and the rest of the features are set to 0.
In this way, the expected (cid:96) 2 -norm of each feature vector (and in turn each gradient) is dimension-independent, and thus the effects of clipping should not vary with p .
The predicted variable y i is sampled from N ( θ ∗ · x i , 0 . 01)
so that the population mean squared error loss is always 0.01, i.e. independent of dimension.
Since the norms of the gradients and number of private samples are dimension-independent, our error bound is proportional to G Q / √ p , i.e. we do not expect it to vary much with dimension.
We set ε = 1 , δ = 10 − 5 .
We consider three algorithms: (i) standard DP-SGD with a “cold start”, i.e. using a random initialization, (ii) standard DP-SGD with a “warm start” on the model pre-trained with public data, and (iii) PDA [Figure] [Figure]
Figure 1a shows the empirical loss of cold- and warm-start DP-SGD.
Since the dimension-dependent loss of DP-SGD is the excess empirical loss, we report empirical rather than population loss.
As expected, pre-training with public data allows DP-SGD to achieve nearly dimension-independent error.
Figure 1b compares warm-start DP-SGD and PDA-DPMD.
The loss of PDA-DPMD is never worse than that of warm-start DP-SGD, and can be substantially lower for smaller dimensions.
We observed that the ratio of the maximum and minimum eigenvalues of the Hessian X (cid:62) X decreases as p increases, which means that the Hessian has poorly-concentrated eigenvalues at small p but gets closer to the identity matrix as p increases.
Since PDA-DPMD recovers warm start DP-SGD when the Hessian is the identity, we can expect that PDA-DPMD obtains less of an advantage over DP-SGD as the Hessian gets closer to the identity.
We provide additional details in Appendix E.
First-order Approximation to Mirror Descent: In practice, the Mirror Descent (MD) step in
Line 4 of Algorithm 1 can be computationally expensive.
For settings where (i) the problem is unconstrained , i.e., C = R p and (ii) the public loss function Ψ( θ ) may not be strongly convex with respect to the (cid:96) 2 -norm, we can instead use the following more efﬁcient approximation:
[Equation]
where η t is the learning rate, and α t ∈ [0 , 1] balances the weight of private and public gradient.
The derivation of this formula is in Appendix F.1.
Notice that α t = 1 corresponds to DP-SGD on private data only.
In our experiment, we decrease α t with a cosine schedule, i.e. α t = cos( πt/ (2 K )) where K is a hyperparameter that controls how fast α t decays.
In practice, instead of computing g t and ∇ Ψ( θ t ) using all the private and public data, we can estimate them with stochastic gradients.
Now, we demonstrate the efﬁcacy of our technique (Algorithm 1) with the update step in (1) on two real world tasks across three benchmark data sets: next word prediction on WikiText-2 (Merity et al., 2017), and image classiﬁcation on CIFAR-10 (Krizhevsky, 2009) and EMNIST (ByMerge split) (Cohen et al., 2017).
For each dataset, we randomly assign 4% of the original training data as public, and the rest as private.
We do not consider a larger amount of in-distribution public data as that could make the problem trivial.
We ﬁrst pre-train on the public data, then use Algorithmwith update rule (1).
We compare our algorithm with two baselines: “cold-start” DP-SGD, which uses the private data only, and “warm-start” DP-SGD, which pre-trains the model with public data and then ﬁne-tunes with the private data.
We demonstrate an increased beneﬁt from the public data over and above just pre-training, which to our knowledge, has not been achieved in prior work.
For WikiText-2, we use an LSTM model from Asi et al.
For CIFAR-10 and EMNIST, we use network architectures considered in prior works (Papernot et al., 2020; Kairouz et al., 2021b).
See Appendix F.2 for more details.
Empirical Evaluation on WikiText-2: Our setup mainly follows Asi et al.
As a preprocessing step, we take the top 7,999 most frequent words and convert the rest into a special token representing unknown word.
The data set is then split into 48,764 length-35 sequences, and we consider sequence-level privacy here.
After pre-training, we ﬁne-tune the model with batch size 250 for 20 epochs.
We search for optimal K in { 100 , 200 , 500 } .
For two different privacy levels, ε = 15 .and 1 .
71 at δ =
10 − 5 (corresponding to σ = 0 .
5 and 1 . 08 , respectively), Figure 2 shows the test [Table] loss for PDA-DPMD with K = 100 and 50 respectively, and for the two baselines.
From Table 1, we see that PDA-DPMD obtains the smallest test loss for both the privacy levels.
Also, comparing the two DP-SGD baselines, we can see that using public data for pre-training provide trained models with higher utility.
Though our work’s focus is on in-distribution public data, we additionally compare with SoTA (Asi et al., 2021) which uses {+WikiText-
+}
(Merity et al., 2017) as the public data.
In that setting, our warm start DP-SGD baseline is better than the proposed SoTA in (Asi et al., 2021) by 1.1% for ε = 1 . 0 and 6.6% for ε = 3 .
in terms of test perplexity.
See Appendix F for more details.
Empirical Evaluation on CIFAR-10: CIFAR-10 consists of 50,000 training images and 10, test images from 10 classes.
After pre-training, we ﬁne-tune the model with batch size 500 forepochs.
We search for optimal K in { 200 , 500 , 1000 , 2000 , 5000 } .
In Figure 3, for two different privacy levels, ε = 3 . 51 and 0 .
19 at δ = 10 − 5 (corresponding to σ = 1 . 51 and 20 . 0 , respectively), we report the test loss and accuracy for K = 2000 , and for the two baselines.
From Table 1, we see that PDA-DPMD provides the best accuracy (even if by a small margin over the warm started DP-SGD baseline).
Moreover, PDA-DPMD also results in signiﬁcantly lower test loss compared to both the baselines for both privacy levels, which conﬁrms with our theoretical analysis for the population loss.
[Figure]
[Figure]
Empirical Evaluation on EMNIST: EMNIST (ByMerge split) consists of 697,932 training images and 116,323 test images from 47 classes.
After pre-training, we ﬁne-tune with batch size 500 for 50 epochs.
We search for optimal K in { 200 , 500 , 1000 , 2000 , 5000 } .
In Figure 4 and Table 1, for σ = 0 .
41 and 1 . 89 , corresponding to privacy ε = 25 . 80 and 0 .
48 at δ = 10 − 6 , we report the test loss and accuracy for K = 500 , and for the two baselines.
We see a similar trend as with CIFAR-10.
[Figure]
E THICS AND R EPRODUCIBILITY
Ethics: This work focuses on improving the privacy/utility trade-offs for model training with sensitivity data by using publicly available data.
We envision that this work will make the adoption of (differentially) private model training more ubiquitous, and hence improve on the current privacy landscape in the context of machine learning.
Our experimental results are on standard benchmark data publicly available data sets, and do not have any ethical concerns.
Reproducibility: Our experiments are either on synthetic data sets, or on standard publicly available data sets.
We have provided full details (including hyperparameter choices) for the experimental setup to reproduce the results in the paper.
We will make the code for our experiments public as soon as possible.
Additionally, we have detailed proofs for all our theoretical results in the appendix
A N OTATION R EFERENCE
We recall here the notation used throughout the paper: α
A strong convexity parameter b
Noise added to gradients for privacy
B Ψ
The Bregman divergence induced by Ψ
C
The constraint set d
A single sample
D pub , D priv
The public and private data sets respectively
D
The universe of samples ε, δ
The privacy parameters η
The learning rate/step size g
A (batch) gradient
G Q
The Gaussian width of Q (cid:96) pub , (cid:96) priv
The (per-example) public and private loss functions respectively
L
The Lipschitz constant of (cid:96) priv
L
The empirical (private) loss on dataset D , i.e. L ( θ ; D ) =
| D | (cid:80) d ∈ D (cid:96) ( θ ; d ) n pub , n priv
The number of public and private samples respectively p
The dimensionality of the optimization problem Ψ
The empirical public loss, i.e. Ψ( θ ) = 1 | D pub | (cid:80) d ∈ D pub (cid:96) ( θ ; d ) ; also the mirror map
Q
A set such that Ψ is 1-strongly convex with respect to (cid:107) · (cid:107) Q
T
The number of iterations in the optimization algorithm θ
A solution to the optimization problem
V
A bound on the variance of the public gradients
B M ISSING B ACKGROUND
Mirror Maps: A mirror map is a differentiable function Ψ : R p
→ R that is strictly convex.
Since Ψ is strictly convex and differentiable, ∇ Ψ : R p
→ R p provides a bijection from R p to itself.
One can view θ as lying in a primal space and ∇ Ψ( θ ) as lying in a dual space.
In turn, we could now consider optimizing over the value ∇ Ψ( θ ) in the dual space instead of θ primal space.
Mirror descent does exactly that, performing gradient descent in the dual space by computing the gradient g t = ∇ (cid:96) ( θ t ) (where θ t lies in the primal space), taking a step in the opposite direction in the dual space, and then using the inverse of the mirror map to determine θ t +1 .
Mirror descent is essentially motivated as minimizing a (linearized) loss plus a Bregman divergence (induced by Ψ ) as the regularizer (Nemirovsky & Yudin, 1983).
More formally, similar to proximal gradient descent, mirror descent is equivalent to taking the gradient g t and performing the update θ t +1 = arg min θ ∈C [ η (cid:104) g t , θ (cid:105) + B Ψ
θ, θ t )]
where B Ψ ( θ 1 , θ 2 ) =
Ψ( θ 1 ) − Ψ
θ 2 ) − (cid:104)∇ Ψ( θ 2 ) , θ 1 − θ 2 (cid:105) is the Bregman divergence generated by Ψ .
Note that, if Ψ( θ ) = (cid:107) θ (cid:107) 22 , then the Bregman divergence is simply B Ψ ( θ 1 , θ 2 )
= (cid:107) θ 1 − θ 2 (cid:107) 22 and mirror descent is equivalent to the usual gradient descent.
Gaussian Width:
Given a bounded set Q ⊂ R d , the Gaussian width of Q , G Q , is a measure of how isotropic the set is.
G Q is deﬁned as E g ∼ N (0 , I p ) max x ∈ Q (cid:104) g, x (cid:105) .
Although the Gaussian width is well-deﬁned for any bounded set, for this paper it sufﬁces to consider deﬁning the Gaussian width of convex sets containing the origin such that max x ∈ Q (cid:107) x (cid:107) 2 = 1 .
If Q is just the unit (cid:96) 2 -ball, the “most isotropic” set satisfying this condition, then we have G Q = √ p ; in particular, since every set Q satisfying max x ∈ Q (cid:107) x (cid:107) 2 = 1 is contained in the (cid:96) 2 -ball, this is the maximum Gaussian width of any such set.
On the other hand, if Q is just the line from the origin to a single unit vector, we have G Q = Θ(1) .
More generally, for any ellipsoid centered at the origin whose axes have radii 0 ≤ r i ≤ 1 , 1 ≤ i ≤ p , we have that the Gaussian width of this ellipsoid is (cid:112)(cid:80) pi =1 r 2 i .
As other examples, the Gaussian width of the (cid:96) 1 -ball of radius 1 is roughly log p , and the Gaussian width of the (cid:96) ∞ ball of radius 1 / √ p is roughly √ p .
C M ISSING P ROOFS FROM S ECTION
Proof of Theorem 3.2.
The privacy guarantee follows from the moments accountant analysis of
Abadi et al.
For the utility guarantee, following the analysis of Theorem 3.2 of Talwar et al. [Equation] we have:
[Equation]
Let θ ∗ in particular be the minimizer satisfying Assumption
[Equation]
By α -strong convexity
[Equation]
we have:
[Equation]
Plugging this into Eq.
(2) and noting that any Ψ we sample is 1-strongly convex with respect to (cid:107) · (cid:107) Q , we get:
[Equation]
We will show that without loss of generality
[Equation]
we can assume ∇ Ψ( θ 0 )
=
and E d ∼ τ [ ∇ (cid:96) pub ( θ ∗ ; d )]
= 0 .
Once we have this assumption, Assumption 3.1 completes the proof.
The assumption follows since by convexity of C we have
[Equation]
Then for any choice of Ψ and C where either θ 0 or θ ∗ is on the boundary of C , suppose we extend C inﬁnitesmally along the line { θ {+
+} + c ( θ ∗ − θ ) | c ∈ R } (i.e., take a point on this line inﬁnitesmally outside of C and update C to be the convex hull of itself and this point).
Then by (3) we have that θ ∗ , θ 0 , deﬁned as the minimizers in C , move apart from each other along this line and in turn by strong convexity the quantity (cid:107)∇ Ψ( θ ∗ ) − ∇ Ψ
( θ 0 ) (cid:107) 22 cannot decrease.
This implies that for any ﬁxed (cid:96) pub and τ , the quantity E (cid:104) (cid:107)∇ Ψ
( θ ∗ ) − ∇ Ψ( θ 0 ) (cid:107) 22 (cid:105) is maximized for a choice of C such that ∇ Ψ
( θ 0 ) = 0 and E d ∼ τ [ ∇ (cid:96) pub ( θ ∗ ; d )]
= 0 .
Proof of Lemma 3.3.
By convexity, for all θ ∈ C we have L ( θ ; D ) ≥ L ( θ ∗ ; D ) + (cid:104)∇L ( θ ∗ ; D ) , θ − θ ∗ (cid:105) .
Note that by optimality of θ ∗ and convexity, for all θ ∈ C we have (cid:104) E d ∼ τ [ ∇ (cid:96) ( θ ∗ ; d )] , θ − θ ∗ (cid:105) ≥ 0 .
In turn, by the Cauchy-Schwarz inequality we can conclude that L ( θ ∗ ; D ) − min θ ∈C L ( θ ; D ) is always upper bounded by (cid:107)C(cid:107) 2 · (cid:107)∇L ( θ ∗ ; D ) − E d ∼ τ [ ∇ (cid:96) ( θ ∗ ; d )] (cid:107) 2 .
By L -Lipschitzness of each (cid:96) ( θ ; d ) we have:
[Equation]
Which completes the proof.
Proof of Theorem 3.6.
Let b t be the noise added for privacy.
Without noise, mirror descent would set θ ∗ to be such that:
[Equation]
Similarly, given the noisy gradient g t + b t , mirror descent would set ˆ θ to be such that:
[Equation]
We then have:
[Equation]
Since we assume the Hessian of Ψ is deﬁned everywhere, we have that ∇ Ψ(ˆ θ ) − ∇ Ψ( θ ∗ )
= ∇ 2 Ψ( (cid:101) θ )(ˆ θ − θ ∗ ) for some (cid:101) θ on the line between ˆ θ and θ ∗ .
In turn, we have:
[Equation]
The norm x of b t sampled from N (0 , σ 2 I p ) has the chi distribution, i.e. pdf proportional to ( x/σ )
p − 1 e − ( x/σ ) 2 / 2 .
In particular, this gives the following standard tail bound:
[Equation]
So with probability at least 1 − e − c 2 p/ 2 , we have the event E that (cid:107) b t (cid:107) 2 is at most (1 + c ) √ pσ .
Conditioned on this event, we have that ˆ θ , and thus (cid:101) θ , is in E , i.e. the lower bounds (cid:101) w i apply to ∇ 2 Ψ( (cid:101) θ ) .
Since conditioning on E only decreases the expectation of |(cid:104) ˆ θ − θ ∗ , v (cid:105)| , we have:
[Equation]
=
[Equation]
When E does not happen, we have w i ( θ ) ≥ α everywhere.
So we have:
[Equation]
To determine E [ |(cid:104) b t , v (cid:105)||¬E ] , note that the distribution of (cid:104) b t , v (cid:105) conditioned on ¬E is equivalent to the distribution of the ﬁrst coordinate of b t conditioned on ¬E .
We can sample b t by ﬁrst sampling its norm (cid:107) b t (cid:107) 2 conditioned on ¬E , and then sampling a point on the sphere with radius (cid:107) b t (cid:107) 2 (no conditioning is required here).
The expected absolute value of any coordinate ( b t ) i given (cid:107) b t (cid:107) 2 can be bounded as:
[Equation]
The inequality is Jensen’s inequality, and the equality uses the fact that the coordinates b i on the sphere are identically distributed, and so we have:
[Equation]
We now just need to bound the expectation of (cid:107) b t (cid:107) 2 , given that it is at least R .
Since the distribution of (cid:107) b t (cid:107) 2 /σ has pdf proportional to x p − 1 e − x 2 / 2 , this expectation is σ times:
[Equation]
Where Γ is the gamma function and P is the regularized gamma function.
Analytically, we can verify that Γ(( p +1) / 2) Γ( p/ 2) ≤ (cid:112) p/ 2 for all p ≥ 1 , and (1 − P (( p +1) / 2 , (1+ c ) 2 p/ 2))
(1 − P ( p/ 2 , (1+ c ) 2 p/ 2)) ≤ 3(1 + c ) 2 for all p ≥ 1 .
So we get:
[Equation]
Putting it all together, we get:
[Equation]
Now applying the law of total expectation gives the theorem statement.
D E XAMPLES OF S TOCHASTIC C ONVEX O PTIMIZATION P ROBLEMS
In this section, we consider two canonical stochastic convex optimization problems and the associated quantity G Q , to help understand the bound in Theorem 3.5.
In both cases we specify the public loss; as previously mentioned, a natural choice is to take the private loss to be the clipped version of the public loss to ensure Lipschitzness holds.
Mean Estimation: Suppose we assume we know the true covariance matrix Σ in the mean estimation problem for data drawn from a multivariate Gaussian.
Then mean estimation is equivalent to minimizing loss function 12 n (cid:80) i ( x i − θ ) (cid:62) Σ − 1 ( x i − θ ) , where x i is the i th sample point, and θ is our estimated mean.
The Hessian is Σ − 1 .
By rescaling, wlog assume the maximum eigenvalue of Σ is 1.
Now, the smallest set Q such that the public loss function is 1-strongly convex with respect to the Q -norm is the ellipsoid centered at the origin whose axes are in the directions of the eigenvectors of Σ , and whose length in the direction of the eigenvector v i is the corresponding eigenvalue
λ i .
The
Gaussian width of Q is thus (cid:112)(cid:80) pi
=
λ 2 i .
In particular, if e.g. Σ only has a few large eigenvalues (i.e., the data only has variance in a few directions), this is much smaller than p .
Linear Regression: We choose the loss to be the mean squared error.
Now, the public loss function is proportional to (cid:107) X θ − y (cid:107) 22 , where X is the (public) feature matrix, y are the dependent variables, and θ is our model.
In particular, this has Hessian X (cid:62) X .
By normalizing, we can assume wlog the smallest eigenvalue of this matrix is 1, and so similarly to mean estimation the quantity G Q is (cid:112)(cid:80) p i =1 1 /λ
i , where λ i are the eigenvalues of X
(cid:62) X (i.e., the singular values of X ).
Note that unlike in the previous example (where we assumed we knew Σ a priori and thus the Hessian was ﬁxed), the Hessian here depends on the public data.
E A DDITIONAL D ETAILS FOR THE E XPERIMENT ON L INEAR R EGRESSION
Note that the exact optimum on the public data can be computed exactly as θ ∗ pub =
( X (cid:62) X ) − 1 X (cid:62) y .
The mirror descent step can also be solved exactly by applying the inverse of the Hessian X (cid:62) X to the gradient, since the Hessian is the same everywhere.
For numerical stability, we add a small constant times the identity matrix to the Hessian before computing its inverse.
We also normalize the Hessian of the loss function so its inverse (which is applied to the gradient before taking a step in PDA-DPMD) has maximum eigenvalue of one.
This ensures that if the Hessian were a multiple of the identity matrix, DP-SGD and PDA-DPMD would behave exactly the same for the same hyperparameter choice.
We perform a grid search over the learning rate, clipping norm, and number of epochs used and report the best empirical loss.
We perform 20 trials for each algorithm and dimension.
F A DDITIONAL D ETAILS ON R EAL - WORLD E XPERIMENTS F. F IRST - ORDER A PPROXIMATION TO M IRROR D ESCENT
In practice, the Mirror Descent (MD) step in Line 4 of Algorithm 1 is computationally the most challenging.
In the following, we provide an approximation of this step in the setting where i) the problem is unconstrained , i.e., C = R p and ii).
the public loss Ψ( θ ) may not be strongly convex with respect to the (cid:96) 2 -norm.
This approximation makes Algorithm 1 efﬁcient in practice.
Consider the following equivalence of Line 4 of Algorithm 1, with Ψ( θ ) , the public loss on D pub , replaced by (cid:98) Ψ( θ ) =
Ψ( θ ) + 12 (cid:107) θ (cid:107) 22 .
This follows from (Hazan, 2019, Lemma 5.5).
[Equation]
where (5) follows from the ﬁrst-order approximation Ψ( θ )
≈ Ψ( θ i )
+ (cid:104)∇ Ψ( θ i ) , θ − θ i (cid:105) .
In the experiments, we replace g i + b i + 1 t · η i ∇ Ψ( θ i ) with α i
( g i + b i ) + (1 − α i ) ∇ Ψ
( θ i ) , where α i ∈ (0 , 1] .
This reparamertization helps with more effective hyperparameter tuning while training deep learning models.
We therefore have the update rule (1).
F. S ETUP
Network architectures:
Table 2 shows model architectures for CIFAR-10, EMNIST, & WikiText2.
[Table]
Hyperparameter Tuning: We keep the clipping norm to be 1.
One small difference with the standard DP-SGD update rule is that we enforce an additional clipping step for the privatized gradient for the image classiﬁcation task, where the clipping norm is the same as the clipping norm of for individual gradient.
The reason for this additional step is that the norm of the averaged clipping gradients should still be upper bounded by the clipping norm.
The only hyperparameters that need to be tuned are the learning rate and K that controls the decaying of α t .
For the learning rate, we consider a grid of { 1 , 2 , 5 }× 10 i for different
i s such that the optimal learning rate does not appear on the boundary.
We search for the optimal K in { 100 , 200 , 500 } for WikiText-2 and { 200 , 500 , 1000 , 2000 , 5000 } for the image classiﬁcation tasks.
F. F ULL P LOTS FOR S ECTION
In Figure
[Figure]
we plot the complete version of Figure
[Figure]
and in Figure
[Figure]
we show the complete version of Figure 3.
[Figure]
[Figure]
F. W IKI T EXT -2 WITH W IKI T EXT -103 AS P UBLIC D ATA
We compare with the SoTA (Asi et al., 2021) which uses WikiText-103 as public data.
Speciﬁcally, we consider their “LargeAux” setting under ε = 1 . 0 and 3 . 0 .
Since the implementation for Asi et al.
(2021) is not public as of writing this work, we make our best effort to match their experiment setup.
We note that the data preprocessing and the number of iterations used (thus the noise multiplier for achieving the same ε ) might differ.
We preprocess WikiText-103 as follows.
After processing WikiText-2 as described in Section 4, we convert all words that does not appear in the processed WikiText-2 as the unknown token.
Then, we split the sentences into length-35 sequences, and remove all sequences that overlap with WikiText-2.
Finally, we randomly sample 48,764 sequences, in order to match the “LargeAux” setting where the public dataset is of the same size as the private training dataset.
Figure 7 shows the results.
In our setting, cold start DP-SGD reaches similar log perplexity as those in (Asi et al., 2021), while the warm-start DP-SGD is already better than Asi et al. (2021) (LargeAux).
The ﬁnal test log perplexities are summarized below, with the results in (Asi et al., 2021) converted from perplexity to log perplexity.
Algorithm ε = 3 .ε
[Figure]
Asi et al.
(2021) DP-SGD (cold) 5.
Asi et al.
(2021) (LargeAux) 5.
Our DP-SGD (cold) 5.
Our DP-SGD (warm) 5.
[Figure]