Differentially Private Generalized Linear Models Revisited
Anonymous Author(s)
AfﬁliationAddress
Abstract
We study the problem of ( ǫ, δ ) -differentially
private learning of linear predictors with convex losses.
We provide results for two subclasses of loss functions.
The ﬁrst case is when the loss is smooth and non-negative but not necessarily Lipschitz (such as the squared loss).
For this case, we establish an upperbound on the excess population risk of
[Equation]
,where n is the number of samples, d is the dimension of the problem, andw ∗ is the minimizer of the population risk.
Apart from the dependence onk w ∗ k , our bound is essentially tight in all parameters.
In particular, we showa lower bound of
[Equation]
We also revisit the previously studied case of Lipschitz losses [SSTT20].
For this case, we close thegap in the existing work and show that the optimal rate is (up to log factors)
[Equation]
, where rank is the rank of the design matrix.
This improves over existing work in the high privacy regime.
Finally, ouralgorithms involve a private model selection approach that we develop to enableattaining the stated rates without a-priori knowledge of k w ∗ k .
Introduction
Ensuring privacy of users’ data in machine learning algorithms is an important desideratum formultiple domains, such as social and medical sciences.
Differential privacy (DP) has become the goldstandard of privacy-preserving data analysis as it offers formal and quantitative privacy guaranteesand enjoys many attractive properties from an algorithmic design perspective [DR + 14].
However,for various basic machine learning tasks, the known risk guarantees are potentially sub-optimal andpessimistic.
In this work, we make progress towards resolving one of the most basic machine learning problemsunder differential privacy: learning linear predictors with convex losses.
1.1 Related Work
Differentially private machine learning has been thoroughly studied for over a decade.
In the Lipschitzconvex setting, tight rates are known for both the empirical and population risk [BST14, BFTGT19].
Speciﬁcally, it was shown that in the constrained setting, dependence on the dimension in the formof Ω (cid:0) √ dnǫ (cid:1) is unavoidable even for generalized linear models (GLM)
(see Section 2 for a formaldeﬁnition).
By contrast, in the unconstrained setting, it has been shown that dimension independentrates are possible for GLMs [JT14].
In this setting, assuming prior knowledge of k w ∗ k , the bestknown rate is O
[Equation]
[SSTT20], where rank is the expected rank of the design matrix.
[Figure]
However, without prior knowledge of k w ∗ k , these methods exhibit quadratic dependence on k w ∗ k .
Furthermore, these results crucially rely on the assumption that the loss is Lipschitz to bound thesensitivity.
Although gradient clipping has been proposed to remedy this problem [SSTT20, CWH20],it is known that the solution obtained by clipping may not coincide with the one of the original model.
Without Lipschitzness, work on differentially private GLMs has largely been limited to linearregression [Wan18, CWZ21].
Here, dimension independent rates have only been obtained undercertain sparsity assumptions.
More generally, smooth non-negative losses have been studied in the non-private setting by [SST10],where it was shown such functions can obtain risk guarantees with linear dependence on the minimizernorm (as in the Lipschitz case).
This work also established a lower bound of Ω (cid:0) 1 √ n (cid:1) on the excesspopulation risk for this class of loss functions.
[Sha15] additionally establishes a lower bound of
[Equation]
on the excess population risk by way of linear regression 1 .
1.2 Our Contributions
Smooth nonegative GLMs.
Our primary contribution is a new and nearly optimal rate for theproblem of differentially private learning of smooth GLMs.
In this setting, we focus on characterizingthe excess risk in terms of n, d, ǫ and k w ∗ k .
Speciﬁcally, we show that it is possible to achieve a rateof
[Equation]
o(cid:17) on the excess population risk.
Our new rates exhibit aninteresting low/high dimensional transition at d ≈ ( k w ∗ k nǫ ) 2 / 3 .
First, in the low dimensional regime, 50 we develop a novel analysis of noisy gradient decent (GD) inspired by techniques from [SST10].
Inparticular, we show that Noisy GD gives an improved rate for non-negative smooth functions (notnecessarily GLMs).
This is based on an average stability analysis of Noisy GD.
As we elaborate insection 3.1, a straightforward application of uniform stability leads to sub-optimal bounds and hence anew analysis is required.
We note in passing that this upper bound works for (unconstrained) DP-SCOwith smooth (non-Lipschitz) losses, which is of independent interest.
For the high dimensional regime,we perform random projections of the data (speciﬁcally, the Johnson-Lindenstrauss transform) fordimensionality reduction, roughly reducing the problem to its low dimensional counterpart.
We alsodevelop a lower bound for the excess risk under DP of
[Equation]
We note thatnon-privately a lower bound of
[Equation]
is known on the excess populationrisk [SST10, Sha15].
We note that these private and non-private lower bounds imply that our boundis optimal up to factors of k w ∗ k
(see Table 1).
Lipschitz GLMs.
For the Lipschitz case, we close a subtle but important gap in existing rates.
Inthis setting, it has been shown that one can characterize the excess risk in terms of the expected rank of the design matrix, rank, instead of d [SSTT20].
In this setting, the best known rate wase
O
[Equation]
We show an improved rate of
[Equation]
Thisimproves in the high privacy regime where ǫ ≤ rank n .
In fact, the upper bound O
[Equation]
for thisrate can be obtained with only minor adjustments to the regularization method of [JT14].
Oursecond contribution in this setting is extending the lower bound of [SSTT20] to hold for all valuesof k w ∗ k > 0 and rank ∈ [ n ] .
This is in contrast to the original lower bound which only holds forproblem instances where k w ∗ k 2 = rank and rank ∈ [ nǫ ] .
Model selection.
As part of our methods, we develop a differentially private model selectionapproach which eliminates the need for a-priori knowledge of k w ∗ k .
Although such methods are 73 well established in the non-private case, (see e.g. [SSBD14]), in the private case no such methodshave been established.
Our method, as in the non-private case, performs a grid search over estimatesof k w ∗ k and picks the best model based on the loss.
However, in the private setting we must account 76 for the fact the the loss evaluation must be privatized.
This is non-trivial in the non-Lipschitz smoothcase as the loss at a point w may grow quadratically with k w k .
Non-private settings.
As by-products, we give the following new results for the non-private setting.
For details on the parameters used below we refer to Table 1.1.
We show that gradient descent, when run on convex non-negative e
H smooth functions (notnecessarily GLMs), it achieves the optimal rate of O (cid:0)
[Equation]
k w ∗ kkYk √ n (cid:1) (see Corollary 1).
This is donevia an average-stability analysis of gradient descent.
This result is interesting as it also shows GDonly needs n iterations, which is known not to work for non-smooth SCO [BFGT20, ACKL21,
AKL21].2.
In Section D, we give a procedure to boost the conﬁdence of algorithms for risk minimization withconvex non-negative e
H smooth functions (not necessarily GLMs).
The standard boosting analysisbased on Hoeffding’s inequality does not give a bound with a linear dependence on the parameters( k w ∗ k , kXk , kYk ) , and hence a tighter analysis is required.
1.3 Techniques
Upper bounds.
We give two algorithms for both the smooth and Lispschitz cases.
The ﬁrst methodis simple and has two main steps.
First, optimize the regularized empirical risk over the constraintset { w : k w k ≤ B } for some B ≥ k w ∗ k .
Then output a perturbation of the regularized minimizer 93 with Gaussian noise (which is not requried to be in the constraint set).
This method is akin to that of[JT14] with the modiﬁcation that the regularized minimizer is constrained to a ball.
We elaborate onthis key difference shortly.
The second method is based on dimensionality reduction.
We use smaller dimensional data-obliviousembeddings of the feature vectors.
A linear JL transform sufﬁces to give embeddings with therequired properties.
We then run a constrained DP-SCO method (Noisy SGD) in the embedded space,and use the transpose of the JL transform to get a d dimensional model.
In this method, the embeddingdimension required is roughly the threshold on dimension at which the rates switch from dimensiondependent to independent bounds.
We also remark that [NêUZ20] applied a similar technique toprovide dimension independent classiﬁcation error guarantees for privately learning support vectormachines under hard margin conditions.
We note that a crucial part in all of these methods is the use of constrained optimization as a subroutine,where the constraint set is a ball of radius k w ∗ k .
This is in stark contrast to the Lipschitz case where 106 existing methods such as those presented by [JT14, SSTT20] rely on the fact that projection is notrequired.
In the smooth case however, constrained optimization helps ensure that the norm of thegradient is roughly bounded by the diameter of the constraint set.
We note that in the high dimensionalregime, the property that the ﬁnal output of the algorithm can have large norm is still crucial to thesuccess of our algorithms.
Lower bounds.
For our lower bounds in the smooth case we rely on the connection betweenstability and privacy.
Speciﬁcally, we will utilize a lemma from [CH12] which bounds the accuracyof one-dimensional differentially private algorithms.
We then combine this with packing arguments to obtain stronger lower bounds for high dimensional problems.
For the Lipschitz case, we adapt themethod of [SSTT20].
Preliminaries
In the following we detail several concepts needed for the presentation of this paper.
Risk minimization.
Let X ⊆
R
d be the domain of features, and Y ⊆
R be the domain of responses.
119 A linear predictor is any w ∈ R d .
Let ℓ : R d × ( X × Y )
→ R be a loss function.
Given some 120 unknown distribution
D over ( X ×Y ) , we deﬁne the population loss L ( w ; D ) =
E ( x,y ) ∼D [ ℓ ( w ; ( x, y ))] .
Given some dataset S ∈ ( X × Y ) n drawn i.i.d.
from D , the objective is to obtain b w ∈ R dwhich minimizes the excess population risk, L ( b w ; D ) − min
w ∈ R d
{ L ( w ; D ) } .
Given a population riskminimization problem, we will denote w ∗ to be a minimum norm solution to this problem.
Wedeﬁne the empirical risk as
[Equation]
ℓ ( w ; ( x, y )) .
We deﬁne the following quantitiesfor notational convenience: ε risk ( w ) =
L ( w ; D ) − L ( w ∗ ; D ) , ε erm ( w ) =
b
L ( w ; S ) − b
L ( w ∗ ; S ) , andε gen ( w ) =
L ( w ; D ) − b
L ( w ; S ) .
We deﬁne B B to be the Euclidean ball of radius B on R d .
Generalized linear models.
We will more speciﬁcally be interested in the problem of learninggeneralized linear models, where there exists some function φ : R ×Y
→ R such that the loss functioncan be written as ℓ ( w ; ( x, y ))
=
φ ( h w, x i , y ) .
We deﬁne parameter bounds kXk =
max x ∈X k x kand kYk 2 = max y ∈Y | φ (0 , y ) | .
Note that for many common smooth non-negative loss functions, the 131 latter condition is the moral equivalent of assuming labels bounded by kYk .
For ease of notation, we 132 write φ ( h w, x i , y ) as φ y ( h w, x i ) and denote function φ y :
z 7→ φ ( z, y ) .
We say that the loss function 133 is G -Lipschitz GLM
if all y ∈ Y , φ y : R → R is G -Lipschitz.
We similarly deﬁne H -smooth GLM.
Differential privacy [DKM + 06].
We restrict our investigation to the class of algorithms whichminimize the excess population risk under the constraint of differential privacy.
A randomizedalgorithm A is said to be ( ǫ, δ ) differentially private (i.e., ( ǫ, δ ) -DP) if for any pair of datasets S and
S ′ differing in one point and any event E in the range of A it holds that
[Equation]
For our lower bounds, we will make use of the following Lemma from [CH12].
Lemma 1.
Let Z be a data domain and let S and S ′ be two datasets each in Z n that differ in atmost ∆ entries, and let A : Z n → R be any ( ǫ, δ ) -DP algorithm.
For all τ ∈ R , if ∆ ≤ log(1 / 2 γ ) ǫand δ ≤ 116 (1 − e − ǫ ) , then E [ |A ( S ) − τ | + |A ( S ′ )
− τ ′ | ] ≥ 14 | τ − τ ′ | .
Finally, we introduce the Johnson-Lindenstrauss (JL) transform to perform random projections.
Deﬁnition 1 ( ( α, β ) -JL property) .
A distribution over matrices R k × d satisﬁes ( α, β ) -JL property iffor any u, v ∈ R d , P [ |h Φ u, Φ v i − h u, v i| > α k u k k v k ] ≤ β.
It is well known that several such “data-oblivious” (i.e. independent of u , v ) distributions exist withk
=
O (cid:16) log(1 /β ) α(cid:17)
[Nel11].
We note that the JL property is typically described as approximation ofnorms (or distances), but it is easy to deduce the above dot product preservation property from it; forcompleteness we give this as Lemma 12.
Finally, we use Φ D to denote the push-forward measure ofthe distribution D under the map Φ : ( x, y ) 7→ (Φ x, y ) .
Similarly, given a data set S = { ( x i , y i ) } i , 150 we deﬁne Φ S :
= { (Φ x i , y i ) i }
Smooth Non-negative GLMs
For smooth non-negative GLMs, we present new upper and lower bounds on the excess risk.
Forour upper bounds, we here assume that the algorithm is given access to some upper bound on k w ∗ k , 154 that we denote by B .
We later show in Section 5 how to obtain such a rate without prior knowledgeof k w ∗ k .
We also emphasize that the privacy of these algorithms holds regardless of whether or not
B k w
3.1 Upper Bounds
Before presenting our algorithms, we highlight some key ideas underlying all our methods.
A crucialproperty of non-negative smooth loss functions which allows one to bound sensitivity, and thusensure privacy, is the self-bounding property (e.g. Sec.
12.1.3 in [SSBD14]), which states that for ane
H -smooth non-negative function f and u ∈ dom(f) , k∇ f ( u ) k ≤ q e
Hf ( u ) .
This property implies that the gradient grows at most linearly with k w k .
More precisely, we have the 163 following.
Lemma 2.
Let ℓ be an H -smooth non-negative GLM.
Then for any w ∈ B B and ( x, y ) ∈ ( X × Y )we have k∇ ℓ ( w, ( x, y ))
k ≤ 2 kYk √ H kXk + 2 HB kXk 2 .
In order to leverage this property, all our algorithms in this setting utilize constrained optimizationas a subroutine, where the constraint set is B B and the Lipschitz constant is G = 2 kYk √ H kXk + 168 2 HB kXk 2 .
This, in conjunction with the self-bounding property ensures reasonable bounds onsensitivity.
In turn, this allows us to ensure privacy without excessive levels of noise.
Finally, we note that our upper bounds for smooth GLMs distinguish low and high dimensionalregimes, transitioning at d = min (cid:16) (cid:16)
B √ H kXk kYk (cid:17) 23 ,(cid:17) ( nǫ ) 23 .
3.1. Low dimensional regime
We start with the low dimensional setting where we use techniques developed for constrained DPSCO for Lipschitz losses (not necessarily GLMs).
Existing private algorithms for DP-SCO (e.g.,[BFTGT19, FKT20]) lead to excess risk bounds that scale with HB 2 √ n .
On the other hand, the optimalnon-private rate [SST10] scales with √ HB √ n , which may indicate that the private rate implied by theknown methods is sub-optimal.
We show that this gap can be closed by a novel analysis of private
GD.
A standard proof of excess risk of (noisy) gradient descent for smooth convex functions is based onuniform stability [HRS15, BFTGT19].
However, this still leads to sub-optimal rates.
Hence we turnto an average stability based analysis of GD, yielding the following result.
Theorem 1.
Let ℓ be a non-negative, convex, e
H -smooth loss function, bounded at zero by kYk 2 .
Let
B, n > 0 ,
[Equation]
[Equation] [Equation]
HB .
Then, for any ǫ, δ > 0 , Algorithm 1 invoked with
[Equation]
is ( ǫ, δ ) -differentiallyprivate.
Further, given a dataset S of n ≥ n 0 i.i.d samples from an unknown distribution D , the 186 excess risk of output of Algorithm 1 is bounded as,
[Equation]
We note that since H -smooth GLMs satisfy the Theorem condition with parameter
[Equation]
H ≤ H kXk 2 , we 188 obtain results for GLMs as a direct corollary.
Non-private risk bound: As a corollary (see Corollary 1 in Appendix A.3.1), with no privacyconstraint, the above result (setting ǫ → ∞ and δ = 1 ) shows that gradient descent achieves the 191 optimal excess risk bound, previously shown to be achievable by regularized ERM and one-pass SGD[SST10].
The lower bound, n 0 , simply means that the trivial solution “zero" has larger excess risk.
Proof sketch of Theorem 1.
The privacy proof simply follows from [BST14] since the loss functionis G -Lipschitz in the constraint set.
For utility, we ﬁrst introduce two concepts used in the proof.
Let S be a dataset of n i.i.d samples { ( x i , y i ) } ni =1 , S ( i ) be the dataset where the i -th data point isreplaced by an i.i.d.
point ( x ′ , y ′ ) .
Let A be an algorithm which takes a dataset as input and outputs
A ( S ) .
The average argument stability of A , denoted as ε av − stab
( A ) is deﬁned as
[Equation]
The average regret of gradient descent (Algorithm 1) with iterates { w t } Tt =1 is
[Equation]
The key arguments are as follows: we ﬁrst bound the generalization error, or on-average stability, interms of average argument stability and excess empirical risk (Lemma 4).
We then bound averageargument stability in terms of average regret (Lemma 5).
Finally, in Lemma 6, we provide bounds onexcess empirical risk and average regret of noisy gradient descent.
Substituting these in the excessrisk decomposition gives the claimed bound.
The full proof with the above lemmas is deferred to
Appendix A.3.1.
Input: Dataset S , loss function ℓ , constraint set W , steps T , learning rate η , noise scale σ1: w 0 =2: for t = 1 to T − 1 do 3: ξ ∼ N (0 , σ 2 I ) 4: w t +1 =
Π W (cid:16) w t − η (cid:16)b
L ( w t ; S ) + ξ (cid:17)(cid:17) where Π W is the projection on to W 5: end for
Output: b w = 1 T
P
Tt =1 w j 3.1.2 High dimensional regime
In the high dimensional setting, we present two techniques.
JL method.
In the JL method, Algorithm 2, we use a data-oblivious JL map Φ to embed all featurevectors in dataset S to k <
d dimensions.
Let dataset e
S = { (Φ x i , y i ) } ni =1 .
We then run projected
Noisy GD method (Algorithm 1) on the loss with dataset e
S and the diameter of the constraint set as2 B .
Finally, we map the returned output e w back to d dimensions using Φ ⊤ to get b w =
Φ ⊤ e w .
Wenote that no projection is performed on b w and thus the output may have large norm due to re-scalinginduced by Φ ⊤ .
Theorem 2.
Let k
= O
[Equation] [Equation]
min
[Equation]
and n 0 = HB
[Equation]
kXkkYk.
Algorithm 2 satisﬁes ( ǫ, δ ) -differential privacy.
Given a dataset S of n ≥ n 0 i.i.d samples, of the output b w is bounded by
[Equation]
Proof sketch of Theorem 2.
From the JL property, with k = O (log ( n/δ ) /α 2 ) , w.h.p.
norms ofall feature vectors and w ∗ , as well as inner products between are preserved upto an α tolerance (see
Deﬁnition 1).
The preservation of norms of feature vectors implies the gradient norms are preserved,and thus privacy guarantee of sub-routine, Algorithm 1, sufﬁces to establish DP.
For the utility proof,from our analysis of Noisy GD (Theorem 1) and using the JL property, the excess risk of b w under D 221 w.r.t.
the risk of Φ w ∗ under Φ D is bounded as,
[Equation]
From smoothness and JL property, the, “JL error” is:
[Equation]
The above is optimized for the value of k prescribed in Theorem 2, substituting which gives theclaimed bound.
Algorithm
Input: Dataset S =
{ ( x 1 , y 1 ) , ..., ( x n , y n ) } , loss function ℓ , k , B, η, T , σ1:
Sample JL matrix Φ ∈ R k × d 2: e
S = { (Φ x 1 , y 1 ) , (Φ x 2 , y 2 ) , . . .
, (Φ x n , y n ) } 3: e w = NoisyGD
( e
S, ℓ, B 2 B , T, η, σ 2 )
Output: b w =
Φ ⊤ e w
Input: Dataset S =
{ ( x 1 , y 1 ) , ..., ( x n , y n ) } , loss function ℓ , λ, B, σ1: e w = arg min w ∈B B n
L ( w ; S ) + λ 2 k w ko 2: ξ ∼ N (0 , σ 2 I d ) Output: b w = e w + ξ
Constrained regularized ERM + output perturbation.
Our second technique is constrainedregularized ERM with output perturbation (Algorithm 3).
A similar technique for the Lipschitz casewas seen in [JT14], however we note that the addition of the constraint set B B is crucial in boundingthe sensitivity in the smooth case.
Theorem 3.
Let n 0 = HB 2 kXkkYk.
Then Algorithm 3 run with σ 2 =
O (cid:18) ( kYk
[Equation]
H 2 B 2 kXk 4 ) log(1 /δ )
λ 2 n 2 ǫ(cid:19) and λ = (cid:18) ( kYk + HB kXk 2 )
√ H kXk Bnǫ (cid:19) 2 /(log
(1 /δ )) 1 / 3 satisﬁes ( ǫ, δ ) -differential privacy.
Given adataset S of n ≥ n 0 i.i.d samples, the excess risk of its output b w is bounded as
E [ ε risk ( b w
[Equation] [Equation]
We note that we can use the same technique in the low dimensional setting too, yielding a rateof √ HB kXkkYk + kYk 2 √ n + GB √ d nǫ .
However, in contrast to Theorem 1 and 2, these results have anadditional kYk 2 √ n term (in both regimes).
Thus, in the regime when kYk ≤ √ HB kXk , the two upperbounds are of the same order.
Proof sketch of Theorem 3.
The privacy proof follows the Gaussian mechanism guarantee togetherwith the fact that the ℓ 2 -sensitivity of constrained regularised ERM is O (cid:0)
Gλn (cid:1)
[BE02].
For utility, weuse the Rademacher complexity based result of [SST10] to bound the generalization error of e w .
Theother term, error from noise, E [ L ( b
[Equation]
−
[Equation]
( Hσ 2 kXk 2 ) from smoothness of GLM.
240 Combining these two and optimizing for λ gives the claimed result.
3.2 Lower Bounds
The proof technique for our lower bound relies on the connection between differential privacy andsensitivity shown in [CH12].
We remark that the underlying mechanisms behind the proof is similarin nature to the method seen in [SU17].
The problem instance we use is the squared loss function, andthus note that these results hold additionally for the more speciﬁc case of linear regression.
Finally,we present the following theorem for empirical risk, but a reduction found in [BFTGT19] implies thefollowing result holds for population risk as well (up to log factors).
The proof is in Appendix A.7.
Theorem 4.
Let ǫ ∈
[1 /n, 1] , δ ≤ 116 (1 − e − ǫ ) .
For any ( ǫ, δ ) -DP algorithm, A , there exists a 249 dataset S with empirical minimizer of norm B such that the excess empirical risk of
A is lowerbounded by Ω
[Equation]
We note this implies our upper bound is optimal whenever B kXk ≤ kYk , which is a commonly 252 studied regime in linear regression [SST10, KL15].
Lipschitz GLMs
In the Lipschitz case, we close a subtle gap in existing rates.
We recall that in this setting a moreprecise characterization in terms of the expected rank of the design matrix is possible (as opposed tousing d ).
The best known upper bound is
[Equation]
(cid:17) assuming knowledge of k w ∗ k .
This boundwas shown to be optimal when ǫ ≥ rank /n and k w ∗ k = √ rank [SSTT20].
We ﬁrst show that in the high privacy regime where ǫ ≤ rank /n , an improved rate is possible.
Speciﬁcally, we show that in this regime constrained regularized ERM with output perturbation andachieves the optimal rate.
In fact, we note that the method of [JT14] (i.e. unconstrained regularized
ERM with output perturbation), can obtain this rate when ǫ =
O (1) if the regularization parameteris set differently.
We present the constrained version in order to leverage Rademacher complexityarguments and provide a slightly cleaner bound that holds for all ǫ > 0 .
Theorem 5.
Algorithm 3 run with parameters σ 2 = 4 G 2 kXk 2 log(1 /δ )
λ 2 n 2 ǫ
[Equation]
G
[Equation] [Equation]
√
[Equation]
satisﬁes ( ǫ, δ ) -DP.
Given a dataset of n i.i.d.
samples from D , its output has excess risk
[Equation]
We also state and prove a similar bound using the JL technique in Appendix B.
Next, we generalize the lower bound of [SSTT20] to show this new bound is optimal for all settingsof B , rank, and ǫ .
We now show that a modiﬁcation of the lower bound present in [SSTT20] showsour upper bound is tight.
We note that their lower bound only held for problem instances whererank =
k w ∗ k 2 and ǫ ≤ rank /n .
By contrast, the upper bound O ( √ rank k w ∗ k nǫ ) holds for any values ofrank and k w ∗ k .
Theorem 6.
Let G, kYk , kXk , B > 0 , ǫ ≤ 1 and δ ≤ ǫ .
For any ( ǫ, δ ) -DP algorithm A , there 273 exists a G -Lipschitz GLM loss bounded at zero by kYk and a distribution D with k w ∗ k ≤ B such 274 that the output of
A on S ∼ D n satisﬁes
[Equation]
All proofs for this section are deferred to Appendix B.
Adapting to k w ∗ k
Our method for privately adapting to k w ∗ k is given in Algorithm 4.
We start by giving a high level 278 overview and deﬁning some necessary preliminaries.
The algorithm works in the following manner.
First we deﬁne a number of “guesses” K for k w ∗ k , B 1 , ..., B K where B j = 2 j : ∀ j ∈ [ K ] .
Then 280 given black box access to a DP optimization algorithm,
A , Algorithm 4 generates K candidatevectors w 1 , ..., w K using A , training set S 1 ∈
( X × Y ) n/ 2 , and the guesses B 1 , ..., B K .
We assume
A satisﬁes the following accuracy assumption for some conﬁdence parameter β > 0 .
Assumption 1.
There exists a function ERR :
R + 7→ R + such that for any B ∈ R + , whenever B ≥ k w ∗ k , w.p. at least 1 − β 4 K under the randomness of S 1 ∼ D n 2 and A it holds that 285 ε risk ( A ( S 1 , B ); D
[Equation]
After generating the candidate vectors, the goal is to pick guess with the smallest excess populationrisk in a differentially private manner using a validation set S 2 .
The following assumption on Aallows us both to ensure the privacy of the model selection algorithm and verify that
[Equation]
289 provides a tight estimate of L ( w j ;
[Equation]
Assumption 2.
There exist a function ∆ : R + 7→ R + such that for any dataset S 2 ∈ ( X × Y ) n/ 2 291 and B >
[Equation]
Speciﬁcally, our strategy will be to use the Generalized Exponential Mechanism, GenExpMech , of[RS15] in conjunction with a penalized score function.
Roughly, this score function penalty ensuresthe looser guarantees on the population loss estimate when B is large do not interfere with the lossestimates at smaller values of B .
We provide the relevant details for GenExpMech in Appendix C.1.
We now state our result.
Theorem 7.
Let ℓ : R d × ( X × Y ) be a smooth non-negative loss function such that ℓ (0 , ( x, y ))
≤kYk 2 for any x,
y ∈
( X × Y ) .
Let ǫ, δ, β ∈ [0 , 1] .
Let K > 0 satisfy ERR (2 K ) ≥ kYk 2 .
Let A bean ( ǫ 2 K , δ 2 K ) -DP algorithm satisfying Assumption 2.
Then Algorithm 4 is ( ǫ, δ ) -DP.
Further, if Asatisﬁes Assumption 1 and S 1 ∼ D n/ 2 then Algorithm 4 outputs ¯ w s.t.
with probability at least 1 − β ,ε risk (
[Equation]
We note that we develop a generic conﬁdence boosting approach to obtain high probability guaranteesfrom our previously described algorithms in Section 5, and thus obtaining algorithms which satisfy 1is straightforward.
We provide more details on how our algorithms satisfy Assumption 2 in Appendix
C.4.
The following Theorem details the guarantees implied by this method for output perturbationwith boosting (see Theorems 12,14).
Full details are in Appendix C.3.
Theorem 8.
Let K, ǫ, δ, β >
and A be the algorithm formed by running 3 with boosting and privacy parameters ǫ ′ = ǫK , δ ′ =
δK .
Then there exists a setting of K such that
K =
Θ
[Equation]
and Algorithm 4 run with A and K is ( ǫ, δ ) -DPand when given S ∼ D n , satisﬁes the following w.p. at least 1 − β (letting B ∗ = 2 max {k w ∗ k , 1 } )
[Equation]
nǫ o
[Equation]
Conﬁdence Boosting: We give an algorithm to boost the conﬁdence of unconstrained, smooth
DP-SCO (with possibly non-Lipschitz losses).
We split the dataset S into m + 1 chunks and run an( ǫ, δ ) -DP algorithm over the m chunks to get m models, and then use Report Noisy Max mechanismto select a model with approximately the least empirical risk.
We show that this achieves the optimalrate of
[Equation]
whereas the previous high probability result of [SST10] had an additional
[Equation]
(cid:16)
[Equation]
term, which was also limited to only GLMs.
The key idea is that non-negativity, convexity,smoothness and loss bounded at zero, all together enable strong bounds on the variance of the loss,and consequently give stronger concentration bounds.
The details are deferred to Appendix D.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is largely theoretical.
Moreover, there are no foreseeable negative societal impacts forour proposed algorithms.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Missing Proofs from Section 3.1 (Smooth GLMs)A.1 Utility Lemmas
Fact 1.
[SSBD14] For a e
H -smooth non-negative function f , and for any u ∈ dom( f ) , we havek∇
f ( u ) k ≤ q e
Hf ( u ) .