O N D YNAMIC N OISE I NFLUENCE IN D IFFERENTIAL P RIVATE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Protecting privacy in learning while maintaining the model performance has become increasingly critical in many applications that involve sensitive data.
Private
Gradient Descent (PGD) is a commonly used private learning framework, which adds noise according to the Differential Privacy protocol.
Recent studies show that dynamic privacy schedules of decreasing noise magnitudes can improve loss at the ﬁnal iteration, and yet theoretical understandings of the effectiveness of such schedules and their connections to optimization algorithms remain limited.
In this paper, we provide comprehensive analysis of noise inﬂuence in dynamic privacy schedules to answer these critical questions.
We ﬁrst present a dynamic noise schedule minimizing the utility upper bound of PGD, and show how the noise inﬂuence from each optimization step collectively impacts utility of the ﬁnal model.
Our study also reveals how impacts from dynamic noise inﬂuence change when momentum is used.
We empirically show the connection exists for general non-convex losses, and the inﬂuence is greatly impacted by the loss curvature.
I NTRODUCTION
In the era of big data, privacy protection in machine learning systems is becoming a crucial topic as increasing personal data involved in training models (Dwork et al., 2020) and the presence of malicious attackers (Shokri et al., 2017; Fredrikson et al., 2015).
In response to the growing demand, differential-private (DP) machine learning (Dwork et al., 2006) provides a computational framework for privacy protection and has been widely studied in various settings, including both convex and non-convex optimization (Wang et al., 2017; 2019; Jain et al., 2019).
One widely used procedure for privacy-preserving learning is the (Differentially) Private Gradient
Descent (PGD) (Bassily et al., 2014; Abadi et al., 2016).
A typical gradient descent procedure updates its model by the gradients of losses evaluated on the training data.
When the data is sensitive, the gradients should be privatized to prevent excess privacy leakage.
The PGD privatizes a gradient by adding controlled noise.
As such, the models from PGD is expected to have a lower utility as compared to those from unprotected algorithms.
In the cases where strict privacy control is exercised, or equivalently, a tight privacy budget , accumulating effects from highly-noised gradients may lead to unacceptable model performance.
It is thus critical to design effective privatization procedures for
PGD to maintain a great balance between utility and privacy.
Recent years witnessed a promising privatization direction that studies how to dynamically adjust the privacy-protecting noise during the learning process, i.e., dynamic privacy schedules , to boost utility under a speciﬁc privacy budget.
One example is (Lee & Kifer, 2018), which reduced the noise magnitude when the loss does not decrease, due to the observation that the gradients become very small when approaching convergence, and a static noise scale will overwhelm these gradients.
Another example is (Yu et al., 2019), which periodically decreased the magnitude following a predeﬁned strategy, e.g., exponential decaying or step decaying.
Both approaches conﬁrmed the empirically advantages of decreasing noise magnitudes.
Intuitively, the dynamic mechanism may coordinate with certain properties of the learning task, e.g., training data and loss surface.
Yet there is no theoretical analysis available and two important questions remain unanswered: 1) What is the form of utility-preferred noise schedules?
When and to what extent such schedules improve utility?
To answer these questions, in this paper we develop a principled approach to construct dynamic schedules and quantify their utility bounds in different learning algorithms.
Our contributions
[Table]
are summarized as follows.
For the class of loss functions satisfying the Polyak-Lojasiewicz condition (Polyak, 1963), we show that a dynamic schedule improving the utility upper bound is shaped by the inﬂuence of per-iteration noise on the ﬁnal loss.
As the inﬂuence is tightly connected to the loss curvature, the advantage of using dynamic schedule depends on the loss function consequently.
2) Beyond gradient descent, our results show the gradient methods with momentum implicitly introduce a dynamic schedule and result in an improved utility bound.
We empirically validate our results on convex and non-convex (no need to satisfy the PL condition) loss functions.
Our results suggest that the preferred dynamic schedule admits the exponentially decaying form, and works better when learning with high-curvature loss functions.
Moreover, dynamic schedules give more utility under stricter privacy conditions (e.g., smaller sample size and less privacy budget).
R ELATED W ORK
Differentially Private Learning .
Differential privacy (DP) characterizes the chance of an algorithm output (e.g., a learned model) to leak private information in its training data when the output distribution is known.
Since outputs of many learning algorithms have undetermined distributions, the probability of their privacy leakages is hard to measure.
A common approach to tackle this issue is to inject randomness with known probability distribution to privatize the learning procedures.
Examples include output perturbation (Chaudhuri et al., 2011), objective perturbation (Chaudhuri et al., 2011) and gradient perturbation (Abadi et al., 2016; Bassily et al., 2014; Wu et al., 2017).
Among these approaches, the Private Gradient Descent (PGD) has attracted extensive attention in recent years because it can be ﬂexibly integrated with variants of gradient-based iteration methods, e.g., stochastic gradient descent, momentum methods (Qian, 1999), and Adam (Kingma & Ba, 2014), for both convex and non-convex problems.
Dynamic Policies for Privacy Protection .
Wang et al.
(2017) studied the empirical risk minimization using dynamic variation reduction of perturbed gradients.
They showed that the utility upper bound can be achieved by gradient methods under uniform noise parameters.
Instead of enhancing the gradients, Yu et al.
; Lee & Kifer (2018) showed the beneﬁts of using a dynamic schedule of privacy parameters or equivalently noise scales.
In addition, Pichapati et al.
Thakkar et al.
(2019) showed that the adaptive clipping norm, which controls sensitivity of batch gradients, similarly improves the convergence.
Utility Upper Bounds .
A utility upper bound is a critical metric for privacy schedules that characterizes the maximum utility that a schedule can deliver in theory.
Wang et al.
(2017) is the ﬁrst to prove the utility bound under the PL condition.
In this paper, we improve the upper bound by a more accurate estimation of the dynamic inﬂuence of step noise.
Remarkably, by introducing a dynamic schedule, we further boost the sample-efﬁciency of the upper bound.
With a similar intuition,
Feldman et al.
(2020) proposed to gradually increase the batch size, which reduces the dependence on sample size accordingly.
Recently, Zhou et al. proved the utility bound by using the momentum of gradients (Polyak, 1964; Kingma & Ba, 2014).
Table 1 summarizes the upper bounds of methods studied in this paper (in the last block of rows) and results from state-of-the-art algorithms based on private gradients.
Our work shows that considering the dynamic inﬂuence can lead to a tighter bound.
P RIVATE G RADIENT D ESCENT
Notations .
We consider a learning task by empirical risk minimization (ERM) f ( θ ) =
1 N (cid:80) Nn =1 f ( θ ; x n ) on a private dataset { x n } Nn =1 and θ ∈ R D .
The gradient methods are deﬁned as θ t +
=
θ t − η t ∇ t , where ∇ t = ∇ f ( θ t ) = 1 N (cid:80) n ∇ f ( θ t ; x n ) denotes the non-private gradient at iteration t , η t is the step learning rate.
∇ ( n ) t = ∇ f
( θ t ; x n ) denotes the gradient on a sample x n .
I c denotes the indicator function that returns 1 if the condition c holds, otherwise 0 .
Assumptions .
In this paper, we assume f ( θ ) is continuous and differentiable.
Many commonly used loss functions satisfy this assumption, e.g., the logistic function.
(2) For a learning task, only ﬁnite amount of privacy cost is allowed where the maximum cost is called privacy budget and denoted as R .
(3) Generally, we assume that loss functions f
( θ ; x ) (sample-wise loss) are G -Lipschitz continuous and f ( θ ) (the empirical loss) is M -smooth.
Deﬁnition 3.1 ( G -Lipschitz continuity) .
A function f ( · ) is G -Lipschitz continuous
if, for G >and all x, y in the domain of f ( · ) , f ( · ) satisﬁes (cid:107) f
( y )
− f ( x ) (cid:107) ≤ G (cid:107) y − x (cid:107) 2 . .
Deﬁnition 3.2 ( m -strongly convexity) .
A function f ( · ) is m -strongly convex if f ( y ) ≥ f
( x ) + ∇ f ( x ) T ( y − x ) + m 2 (cid:107) y − x (cid:107) 2 , for some m > 0 and all x, y in the domain of f ( · ) .
Deﬁnition 3.3 ( M -smoothness) .
A function is M -smooth w.r.t.
l 2 norm if f ( y ) ≤ f ( x ) + ∇ f ( x ) T ( y − x ) + M 2 (cid:107) y − x (cid:107) 2 , for some constant M > 0 and all x, y in the domain of f ( · ) .
For a private algorithm M ( d ) which maps a dataset d to some output, the privacy cost is measured by the bound of the output difference on the adjacent datasets.
The adjacent datasets are deﬁned to be datasets that only differ in one sample.
In this paper, we use the zero-Concentrated Differential
Privacy (zCDP, see Deﬁnition 3.4) as the privacy measurement, because it provides the simplicity and possibility of adaptively composing privacy costs at each iteration.
Various privacy metrics are discussed or reviewed in (Mironov, 2017;
Bun et al., 2018; Abadi et al., 2016; Desfontaines & Pejó, 2019).
An example is Moment Accoutant (Abadi et al., 2016), which has a tighter privacy bound but is more complicated in computing privacy cost due to the minimization problem involved.
We note that alternative metrics can be adapted to our study without major impacts to the analysis.
Deﬁnition 3.4 ( ρ -zCDP (Bun & Steinke, 2016)) .
Let ρ > 0 .
A randomized algorithm M : D n → R satisﬁes ρ -zCDP if, for all adjacent datasets d, d (cid:48)
∈ D n , D α ( M ( d ) (cid:107)M ( d (cid:48) )) ≤ ρα, ∀ α ∈ (1 , ∞ ) where D α ( ·(cid:107)· ) denotes the Rényi divergence (Rényi, 1961) of order α .
zCDP provides a linear composition of privacy costs of sub-route algorithms.
When the input vector is privatized by injecting Gaussian noise of N (0 , σ 2 t I ) for the t -th iteration, the composed privacy cost is proportional to (cid:80) t ρ t where the step cost is ρ t = 1 σ 2 t .
For simplicity, we absorb the constant coefﬁcient into the (residual) privacy budget R .
The formal theorems for the privacy cost computation of composition and Gaussian noising is included in Lemmas B.1 and B.2.
Generally, we deﬁne the Private Gradient Descent (PGD) method as iterations for t = 1 . . .
T :
[Equation]
where φ t = g t is the gradient privatized from ∇ t as shown in Algorithm 1, G/N is the bound of sensitivity of the gathered gradient excluding one sample gradient, and ν t ∼ N (0 , I ) is a vector element-wisely subject to Gaussian distribution.
We use σ t to denote the noise scale at step t and use σ to collectively represents the schedule ( σ 1 , . . .
, σ T ) if not confusing.
When the Lipschitz constant is unknown, we can control the upper bound by scaling the gradient if it is over some constant.
The scaling operation is often called clipping in literatures since it clips the gradient norm at a threshold.
[-+
-]
After the gradient is noised, we apply a modiﬁcation, φ ( · ) , to enhance its utility.
In this paper, we consider two types of φ ( · ) :
[Equation]
We now show that the PGD using Algorithm 1 guarantees a privacy cost less than R : Input Raw 6: return
7: else 8:
Terminate
Theorem 3.1.
Suppose f ( θ ; x ) is G -Lipschitz continuous and the PGD algorithm with privatized gradients deﬁned by Algorithm 1, stops at step T .
The PGD algorithm outputs θ T and satisﬁes ρ -zCDP where ρ ≤ 12 R .
Note that Theorem 3.1 allows σ t to be different throughout iterations.
Next we present a principled approach for deriving dynamic schedules optimized for the ﬁnal loss f ( θ T ) .
D YNAMIC P OLICIES BY M INIMIZING U TILITY U PPER B OUNDS
To characterize the utility of the PGD, we adopt the Expected Excess Risk (EER), which notion is widely used for analyzing the convergence of random algorithms, e.g., (Bassily et al., 2014; Wang et al., 2017).
Due to the presence of the noise and the limitation of learning iterations, optimization using private gradients is expected to reach a point with a higher loss (i.e., excess risk) as compared to the optimal solution without private protection.
Deﬁne θ ∗ =
arg min θ f ( θ ) , after Algorithm 1 is iterated for T times in total, the EER gives the expected utility degradation:
[Equation]
Due to the variety of loss function and complexity of recursive iterations, an exact EER with noise is intractable for most functions.
Instead, we study the worst case scenario, i.e., the upper bound of the EER, and our goal is to minimize the upper bound.
For consistency, we call the upper bound of
EER divided by the initial error as ERUB.
Since the analytical form of EER is either intractable or complicated due to the recursive iterations of noise, studying the ERUB is a convenient and tractable alternative.
The upper bound often has convenient functional forms which are (1) sufﬁciently simple, such that we can directly minimize it, and (2) closely related to the landscape of the objective depending on both the training dataset and the loss function.
As a consequence, it is also used in previous PGD literature (Pichapati et al., 2019; Wang et al., 2017) for choosing proper parameters.
Moreover, we let ERUB min be the achievable optimal upper bound by a speciﬁc choice of parameters, e.g., the σ and T .
In this paper, we consider the class of loss functions satisfying the Polyak-Lojasiewicz (PL) condition which bounds losses by corresponding gradient norms.
It is more general than the m -strongly convexity.
If f is differentiable and M -smooth, then m -strongly convexity implies the PL condition.
Deﬁnition 4.1 (Polyak-Lojasiewicz condition (Polyak, 1963)) .
For f ( θ ) , there exists µ > 0 and for every θ , (cid:107)∇ f ( θ ) (cid:107) 2 ≥ 2 µ ( f ( θ ) − f ( θ ∗ )) .
The PL condition helps us to reveal how the inﬂuence of step noise propagates to the ﬁnal excess error, i.e., EER.
Though the assumption was also used previously in Wang et al.
; Zhou et al.
(2020), neither did they discuss the propagated inﬂuence of noise.
In the following sections, we will show how the inﬂuence can tighten the upper bound in gradient descent and its momentum variant.
4. G RADIENT D ESCENT M ETHODS
For the brevity of our discussion [Equation] we ﬁrst deﬁne the following constants:
[Equation]
which satisfy κ ≥ 1 and γ ∈ [0 , 1)
Note that κ is the condition number of f ( · ) if f ( · ) is strongly convex.
κ tends to be large if the function is sensitive to small differences in inputs, and 1 /α tends to be large if more samples are provided and with a less strict privacy budget.
The convergence of PGD under the PL condition has been studied for private (Wang et al., 2017) and non-private (Karimi et al., 2016; Nesterov & Polyak, 2006; Reddi et al., 2016)
ERM.
Below we extend the bound in (Wang et al., 2017) by considering dynamic inﬂuence of noise and relax σ t to be dynamic:
Theorem 4.1.
Let α , κ and γ be deﬁned in Eq.
(2) , and η t = 1 M .
Suppose f ( θ ; x i ) is G -Lipschitz and f ( θ ) is M -smooth satisfying the Polyak-Lojasiewicz condition.
For PGD, the following holds:
[Equation]
In Eq.
(3), the step noise magnitude σ 2 t has an exponential inﬂuence , q t , on the EER.
The dynamic characteristic of the inﬂuence is the key to prove a tighter bound.
Plus, on the presence of the dynamic inﬂuence, it is natural to choose a dynamic σ 2 t .
When relaxing q t to a static 1 , a static σ 2 t was studied by Wang et al..
They proved a bound which is nearly optimal except a ln 2 N factor.
To get the optimal bound, in the following sections, we look for the σ and T that minimize the upper bound.
U NIFORM S CHEDULE
The uniform setting of σ t has been previously studied in Wang et al.
Here, we show that the bound can be further tighten by considering the dynamic inﬂuence of iterations and a proper T .
Theorem 4.2.
Suppose conditions in Theorem 4.1 are satisﬁed.
When σ 2 t = T/R , let α , γ and κ be deﬁned in Eq.
(2) and let T be:
[Equation]
Meanwhile, if κ ≥ 11 − c > 1 , 1 /α > 1 /α 0 for some constant c ∈ (0 , 1) and α 0 > 0 , the corresponding bound is:
[Equation]
Sketch of proof.
The key of proof is to ﬁnd a proper T to minimize
[Equation]
where we use σ t = (cid:112) T/R .
Vanishing its gradient is to solve γ T ln γ + ακ (1 − γ T ) − ακTγ T ln γ = 0 , which however is intractable.
In (Wang et al., 2017), T is chosen to be O (ln(1 /α )) and ERUB is relaxed as γ T + ακT 2 .
The approximation results in a less tight bound as O ( α (1 + κ ln 2 (1 /α ))) which explodes as κ → ∞ .
We observe that for a super sharp loss function, i.e., a large κ , any minor perturbation may result in tremendously ﬂuctuating loss values.
In this case, not-stepping-forward will be a good choice.
Thus, we choose T =
[Equation]
which converges to 0 as κ → + ∞ .
The full proof is deferred to the appendix.
4.1. D YNAMIC S CHEDULE
A dynamic schedule can improve the upper bound delivered by the uniform schedule.
First, we observe that the excess risk in Eq.
(3) is upper bounded by two terms: the ﬁrst term characterizes the error due to the ﬁnite iterations of gradient descents; the second term, a weighted sum, comes from error propagated from noise at each iteration.
Now we show for any { q t | q t > 0 , t = 1 , . .
, T } (not limited to the q t deﬁned in Eq.
(3)), there is a unique σ t minimizing the weighted sum:
Lemma 4.1 (Dynamic schedule) .
Suppose σ t satisfy (cid:80)
Tt =1 σ − 2 =
R .
Given a positive sequence { q t } , the following equation holds:
[Equation]
Remarkably, the difference between the minimum and T (cid:80) Tt =
1 q t (uniform σ t ) monotonically increases by variance of √ q t w.r.t.
t .
We see that the dynamics in σ t come from the non-uniform nature of the weight q t .
Since q t presents the impact of the σ t on the ﬁnal error, we denote it as inﬂuence .
Given the dynamic schedule in
Eq.
(6), it is of our interest to which extent the ERUB can be improved.
First, we present Theorem 4. to show the optimal T and ERUB.
Theorem 4.3.
Suppose conditions in Theorem 4.1 are satisﬁed.
Let α , κ and γ be deﬁned in Eq.
When η t = 1 M , σ t (based on Eqs.
(3) and (6) )
and the T minimizing ERUB are, i.e.,
[Equation]
Meanwhile, when κ ≥ 1 and 1 /α ≥ 1 /α 0 for some positive constant α 0 , the minimal bound is:
[Equation]
4.1. D ISCUSSION
In Theorems 4.2 and 4.3, we present the tightest bounds for functions satisfying the PL condition, to our best knowledge.
We further analyze the advantages of our bounds from two aspects: sample efﬁciency and robustness to sharp losses.
[Equation]
l o ss descent repeated 100 times on two differently-curved loss functions. Solid lines are optimization trajectories and dashed horizontal lines are the averaged ﬁnal losses.
Sample efﬁciency .
Since dataset cannot be inﬁnitely large, it is critical to know how accurate the model can be trained privately with a limited number of samples.
Formally, it is of interest to study when κ is ﬁxed and N is large enough such that α (cid:29) 1 .
Then we have the upper bound in Eq.
(5) as
ERUB
[Equation]
min [Equation] where we ignore κ and other logarithmic constants with ˜ O as done in Wang et al.
As a result, we get a bound very similar to (Wang et al., 2017), except that R is replaced by R MA =
(cid:15) 2 / ln(1 /δ )
using Moment Accountant.
In comparison, based on
Lemma B.3, R = 2 ρ = 2 (cid:15) + 4 ln(1 / [Equation]
) + 4 (cid:112) ln( [Equation]
)( (cid:15) + ln [Equation] 1 /δ )
if θ T satisﬁes ρ -zCDP.
Because ln(1 /δ ) > 1 , it is easy to see
R zCDP > R MA when (cid:15) ≤ 2 ln(1 /δ ) .
As compared to the one reported in (Wang et al., 2017), our bound saved a factor of ln N and thus is require less sample to achieve the same accuracy.
Remarkably, the saving is due to the maintaining of the inﬂuence terms as shown in the proof of Theorem 4.2.
Using the dynamic schedule, we have ERUB dynamic min ≤ O
( α ) =
O (cid:16)
[Equation]
2 MN 2 R (cid:17) , which saved another ln N factor in comparison to the one using the uniform schedule Eq.
As shown in Table 1, such advantage maintains when comparing with other baselines.
Robustness .
Besides sample efﬁciency, we are also interested in robustness of the convergence under the presence of privacy noise.
Because of the privacy noise, the convergence of private gradient descent will be unable to reach an ideal spot.
Speciﬁcally, when the samples are noisy or have noisy labels, the loss curvature may be sharp.
The sharpness also implies lower smoothness, i.e., a small M or has a very small PL parameter.
Thus, gradients may change tremendously at some steps especially in the presence of privacy noise.
As illustrated in the left ﬁgure, the highly-curved loss function (the green curve) results in mean higher ﬁnal loss (the red dashed line) than the ﬂatten curve (purple and blue lines).
Such changes have more critical impact when only a less number of iterations can be executed due to the privacy constraint.
Assume α is some constant while κ (cid:29) 1 /α , we immediately get:
[Equation]
Both are robust, but the dynamic schedule has a smaller factor since 1 /α could be a large number.
In addition, the factor implies that when more samples are used, the uniform schedule is less robust.
4. G RADIENT D ESCENT M ETHODS WITH M OMENTUM
Section 4.1 shows that the step noise has an exponentially increasing inﬂuence on the ﬁnal loss, and therefore a decreasing noise magnitude improves the utility upper bound by a ln N factor.
On the other hand, it also suggests that noise in the beginning iterations has very minor inﬂuence on the ﬁnal iterations, because the accumulated noise overwhelmed the propagated information.
In addition, the proper magnitude of decreasing noise can be hard to ﬁnd.
The momentum method was originally proposed to stabilize the gradient estimation in Stochastic
Gradient Descent (SGD) (Polyak, 1964).
In this section, we show that momentum can be used to reduce the variance of privacy noise for full-batch gradient descent, and improve the utility upper bound.
The momentum method or the exponentially moving average was also used in previous private learning literature.
Pichapati et al.
used the momentum as an estimation of gradient mean, without discussions of convergence improvements.
Zhou et al.
gave a bound for the Adam with DP.
However, the derivation is based on gradient norm, which results in a looser bound as shown in Table 1.
The momentum method stabilizes gradients by moving average history coordinate values and thus greatly reduces the variance.
The φ ( m t , g t ) can be rewritten as:
[Equation]
where β ∈ [0 , 1] .
Note v t +1 is a biased estimation of the gradient expectation while m t +1 is unbiased.
Theorem 4.4 (Convergence under PL condition) .
Suppose f ( θ ; x i ) is G -Lipschitz, and f ( θ ) is Msmooth and satisﬁes the Polyak-Lojasiewicz condition.
Assume β (cid:54) = γ and β ∈ (0 , 1) .
Let η t = η 0 2 M and η 0 ≤ 8 (cid:16)(cid:112) 1 + 64 βγ
( γ − β ) − 2 (1 − β ) − 3 + 1 (cid:17) − 1 .
Then the following holds:
[Equation]
where
[Equation]
The upper bound includes three parts that inﬂuence the bound differently: (1) Convergence.
The convergence term is mainly determined by η 0 and κ .
η 0 should be in (0 , κ ) such that the upper bound can converge.
A large η 0 will be preferred to speed up convergence if it does not make the rest two terms worse.
(2) Noise Variance.
The second term compressed in U 3 is the effect of noise.
One difference introduced by the momentum is the factor (1 − β ) / (1 − β t ) which decreases by t .
(1 − β ) / (1 − β t ) decreases faster than γ t at the beginning and converges to a non-zero constant 1 − β .
Therefore, the inﬂuence factor in U 3 , γ T − t , will be reduced to γ T which maintains constant.
Furthermore, when t > ˆ T , the moving average of σ 2 t reweighted the γ T − t which also makes the quickly changing inﬂuence term smoother.
(3) Momentum Effect.
The momentum effect term can improve the upper bound when η 0 is small.
For example, when β = 0 .
9 and γ = 0 .
99 , then η 0 ≤ 0 .
98 /M which is a rational value.
Following the analysis, when M is large which means the gradient norms will signiﬁcantly ﬂuctuate, the momentum term may take the lead.
Adjusting the noise scale in this case may be less useful for improving utility.
Theorem 4.5 (Uniform schedule) .
Suppose the assumptions in Theorem 4.4 are satisﬁed.
Let σ 2 t = T/R , and let:
[Equation]
If some positive constant c , 1 /α > 1 /α 0 and α 0 > 0 , the following inequality holds:
[Equation]
Theorem 4.6 (Dynamic schedule) .
Suppose the assumptions in Theorem 4.4 are satisﬁed.
Let α (cid:48) =
[Equation]
α γ (1 − γβ 2 ) , β < γ and ˆ T = max t s.t.
[Equation]
− 1 ≥ 1 − β
[Equation]
β t .
Use the following schedule and iteration:
[Equation]
, where q t = c 1 γ T + t I T ≤ ˆ T + γ ˆ T − 1 c 2 γ T − t I T>
ˆ T for some positive constants c 1 and c 2 .
The following inequality holds:
[Equation]
Notably, the dynamic schedule should not be decreasing when t ≤ ˆ T , which is counterintuitive.
Discussion.
Theoretically, the dynamic schedule is more inﬂuential in vanilla gradient descent methods than the momentum variant.
The result is mainly attributed to the averaging operation.
The moving averaging, (1 − β ) (cid:80) ti =1 β t −
i g i / (1 − β t ) , increase the inﬂuence of the under-presented initial steps and decrease the one of the over-sensitive last steps.
4. P RIVATE S TOCHASTIC G RADIENT D ESCENT (NEW SECTION ON REBUTTAL)
Though PGD provides a guarantee for utility and privacy, it is not practical for large-scale problems.
Therefore, studying the convergence of Private Stochastic Gradient Descent (PSGD) is necessary.
The Algorithm 1 can be easily extended to PSGD by letting n be the batch size.
Accordingly, we replace N in the deﬁnition of α by n because the term is from the sensitivity of batch data (see Eq.
For clarity, we assume T is the number of iterations rather than epochs, ˜ ∇ t for mean stochastic gradient.
According to (Yu et al., 2019), when privacy is measured by zCDP, there are two ways to account for the privacy cost of PSGD depending on the batch-sampling method: sub-sampling with or without replacement.
In this paper, we focus on the random subsampling with replacement (RS for short) since it is preferred for deep learning in literatures (Abadi et al., 2016; Feldman et al., 2020).
When a batch of data are randomly sampled, the privacy cost of one iteration is cp 2 /σ t where c is some constant, p = n/N is the sample rate, and 1 /σ 2 t is the full-batch privacy cost.
Details of the sub-sampling theorems are referred to the Theorem 3 of (Yu et al., 2019) and their empirical setting.
Threfore, we can replace the privacy constraint (cid:80) t p 2 /σ 2 t
=
R by (cid:80) t 1 /σ 2 t
=
R (cid:48)
where
R (cid:48)
= R/p 2 = N 2 n 2 R .
Remarkably, we omit the constant c because it will not affect the results regarding uniform or dynamic schedules.
Notice N 2 R in the α is replaced by n 2 R (cid:48)
=
N 2 R .
Thus, the form of α is not changed.
Now we study the utility bound of the PSGD.
Because the batch of gradients are sub-sampled from the training set, additional randomness will be injected.
To quantify the randomness, we deﬁne a random vector ξ t with E [ ξ t ] = 0 and E (cid:107) ξ t (cid:107) 2 ≤ D such that ˜ ∇ t ≤ ∇ t + σ g ξ t /n for some positive constant σ g .
With this trick, we can easily extend the PGD bounds to PSGD bounds by following theories.
Theorem 4.7 (Utility bounds of PSGD) .
Let α , κ and γ be deﬁned in Eq.
(2) , and η t = 1 M .
Suppose f ( θ ; x i ) is G -Lipschitz and f ( θ ) is M -smooth satisfying the Polyak-Lojasiewicz condition.
For PSGD, when batch size satisﬁes n = max { N √ R, 1 } , the following holds:
[Equation]
where α g =
[Equation]
Theorem 4.8 (PSGD with momentum) . Let α g = D [Equation]
µN 2 R ( f
[Equation] − f ( θ ∗ )) . Suppose assumptions in Theorem 4.4 holds.
When batch size satisﬁes n = max { N √ R, 1 } , the U 3 ( σ, T ) has to be replaced by
[Equation]
when PSGD is used.
As shown above, the utility bound of PSGD differs from the PGD merely by α g σ 2 g . Note α g =
O ( DN 2 R ) which ﬁts the order of dynamic-schedule bounds.
In addition, α and other variables are not changed.
Hence, the conclusions w.r.t.
the dynamic/uniform schedules maintain the same.
E XPERIMENTS
We empirically validate the properties of privacy schedules and their connections to learning algorithms.
In this section, we brieﬂy review the schedule behavior on quadratic losses under varying data sensitivity.
Details of experimental setups and empirical results are available in Appendix D.
We ﬁrst show the estimated inﬂuence of step noise q t (by retraining the private learning algorithms, ref.
Appendix D) in Fig.
2 Left.
We see the trends of inﬂuence are approximately in an exponential form of t .
This obvervation motivates the use of exponential decay schedule in practice.
We then show the trends on the variance of inﬂuence (dashed lines with the right axis) and relative ﬁnal losses (solid lines with the left axis) in the Middle Pane, where uni denotes the uniform schedule baseline, exp is an exponential schedule, dyn denotes the dynamic schedule minimizing the ERUB.
The inﬂuences increases steeply when the data scale is large and therefore have a large variance.
Meanwhile, dynamic schedules show improvements of the ﬁnal loss when the variance is large.
It reveals the connection between the inﬂuence and the dynamic advantage (refer to Lemma 4.1).
We lastly evaluate the impacts from momentum in the Right Pane, using a Deep Neural Network (DNN) with 2 layers and 100 hidden units.
Because of time costs of training deep networks, we do not estimate the inﬂuence by retraining and then compute schedules.
Instead, we grid-search for the schedule hyper-parameters to ﬁnd the best one.
We see that inﬂuence modeled by an exponential function ( expinﬂ ) has comparable performance of the inﬂuence modeled by linear combination of two reverse exponential functions ( momexpinﬂ ).
The latter only shows advantage in the setting that data scale is 25 and the number of iteration is only 100 , which is expected by our analysis Theorems 4. and 4.6.
The inherent reason is that the dynamic schedule is more effective when T is larger.
[Figure]
C ONCLUSION
When a privacy budget is provided for a certain learning task, one has to carefully schedule the privacy usage through the learning process.
Uniformly scheduling the budget has been widely used in literature whereas increasing evidence suggests that dynamically schedules could empirically outperform the uniform one.
This paper provided a principled analysis on the problem of optimal budget allocation and connected the advantages of dynamic schedules to both the loss structure and the learning behavior.
We further validated our results through empirical studies
A C OMPARISON OF ALGORITHMS
[Table]
We present Table 2 as an sumpplementary to the Table 1.
Asymptotic upper bounds are achieved when sample size N approaches inﬁnity.
Both R and R (cid:15),δ with R (cid:15),δ <
R are the privacy budgets of corresponding algorithms.
Speciﬁcally, R (cid:15),δ = (cid:15) 2 / ln(1 /δ ) <
R when the private algorithm is ( (cid:15), δ ) -DP with (cid:15) ≤ 2 ln(1 /δ ) .
PGD+Adv .
Adv denotes the Advanced Composition method (Bassily et al., 2014).
The method assumes that loss function is 1 -strongly convex which implies the PL condition and optimized variable is in a convex set of diameter 1 w.r.t.
l 2 norm.
PGD+MA .
MA denotes the Moment Accoutant (Abadi et al., 2016) which improve the composed privacy bound versus the Advanced Composition.
The improvement on privacy bound lead to a enhanced utility bound, as a result.
PGD+Adv+BBImp .
The dynamic method assumes that the loss is 1 -strongly convex and data comes in stream with n ≤ N samples at each round.
Their utility upper bound is achieved at some probability p with any positive c .
Adam+MA .
The authors prove a convergence bound for the gradient norms which is extended to loss bound by using PL condition.
They also presents the results for AdaGrad and GD which are basically of the same upper bound.
Out theorems improve their bound by using the recursive derivation based on the PL condition, while their bound is a simple application of the condition on the gradient norm bound.
GD, Non-Private .
This method does not inject noise into gradients but limit the number of iterations.
With the bound, we can see that our utility bound are optimal with dynamic schedule.
GD+zCDP .
We discussed the static and dynamic schedule for the gradient descent method where the dynamic noise inﬂuence is the key to tighten the bound.
Momemtum+zCDP .
Different from the GD+zCDP, momentum methods will have two phase of utility upper bound.
When T is small than some positive constant ˆ T , the bound is as tight as the non-private one.
Afterwards, the momentum has a bound degraded as the GD bound. [Table]
A. C OMAPRISON OF GENERALIZATION BOUNDS
Exceot the emirical risk bounds in Table [Table] it is also of our interest to study the true risk bound
[Table]
or generalization error bound.
By leveraging the generic learning-theory tools
[Table]
we extend our results to the True Excess Risk (TER) for (strongly) convex functions as follows.
Different from the EER
[Table]
we deﬁne TER for a model θ as follows,
[Equation]
where the second expectation is over the randomness of generating θ .Z
Assume a dataset d consist of n samples drawn i.i.d.
from the distribution X .
In the recent literatures, there are two tricks which could extend the empirical bounds to the true excess risk.
One is proposed by Shalev-Shwartz et al.
(2009) where the true excess risk of PGD can be bounded in high probability.
For example,
Bassily et al.
(2014) achieve a ln 2 NN bound with N 2 iterations.
Instead of relying on the probabilistic bound, Bassily et al.
(2019) use the uniform stability to give a tighter bound.
Later, Feldman et al.
(2020) improve the efﬁency of gradient computation to achieve the similar bound.
Both methods will introduce an additive term to the empirical bounds.
In this paper, we adopt both methods to see how our result can be extended.
(1) True Risk by Unﬁrom Stability .
Following Bassily et al.
(2019), we use the uniform stability (deﬁned in Deﬁnition A.1) to extend the empirical bounds.
We restate the related deﬁnition and theorems as follows.
Deﬁnition A.1 (Uniform stability) .
Let s > 0 .
A randomized algorithm M : X → Θ is s -uniformly stable w.r.t.
the loss function f if for any neighor datasets d and d (cid:48) , we have
[Equation]
where the expectation is over the internal randomness of M .
Theorem A.1 (See, e.g., (Shalev-Shwartz & Ben-David, 2014)) .
Suppose M : X → Θ is s -uniformly stable algorithm w.r.t.
the loss function f .
Let D be any distribution from over X and let d ∼ D N .
The following holds true.
[Equation]
where the second expectation is over the internal randomness of M .
f ( M ( d ); D ) and f ( M ( d ); d ) represent the true loss and the empirical loss, respecitvely.
Theorem A.2 (Uniform stability of PGD from (Bassily et al., 2019)) .
Suppose η < 2 /M for M smooth, G -Lipschitz f ( θ ) .
Then PGD is s -uniformly stable with s =
G 2 Tη/N .
Combining Theorems A.1 and A.2, we can get
[Equation]
Plug in the empirical ERUB’s and we can get the O bounds for PGD with/without dynamic schedules in Table 3.
(2) True Risk in High Probability .
Alternatively, we consider the high-probability true risk bound.
Based on Section 5.4 from (Shalev-Shwartz et al., 2009) (restated in Theorem A.3), we can bridge the EER to the TER.
Theorem A.3.
Let f ( · ; x ) be G -Lipschitz, µ -strong convex loss function for any x ∈ X .
With probability at least 1 − p over the randomness of sampling the data set d , the following inequality is satisﬁed.
[Equation]
(16) where θ ∗ = arg min θ f ( θ ) .
To apply the Eq.
(16), we need to extend ERUB, the expectation bound, to high-probability bound.
Following (Bassily et al., 2014)
(Section D), we repeate the PGD with privacy budget R/k for k times.
Note, the output of all repetitions is still of R budget.
When k = 1 , let the ERUB of the algorithm be denoted as F ( R ) .
Then the ERUB of one execution of the k repetitions is F ( R/k ) where privacy is accounted by zCDP.
When k = log 2 (1 /p ) for p ∈ [0 , 1] , by Markov’s inequality, there exists one repetition whose ERUB is F ( R/ log 2 (1 /p )) with probability at least 1 − 1 / 2 k = 1 − p .
Combined with Eq.
(16), we use the bounds of uniform schedule and dynamic schedules in Section 4.1.3 to obtain
[Equation]
where we still ignore the κ and other constants.
Similary, we can extend the momentum methods.
We summarize the results and compare them to the prior works in Table 3 where we include an additional method: Snowball Stochastic Gradient Descent (SSGD+MA) where the batch size is gradually increased.
The method assumes that loss functions are strongly convex within a convex body of diameter 1 w.r.t.
l 2 norm.
Discussion .
Though the limitation of the true risk bounds is on the extension from empirical bounds, our method has remarkably improved efﬁency in T from N or N 2 to ln( N ) .
That implies less iterations are required for converging to the same generalization error.
B P RELIMINARIES B. P RIVACY
Lemma B.1 (Composition & Post-processing) .
Let two mechanisms be M : D n → Y and M (cid:48) :
D n × Y
→ Z .
Suppose M satisﬁes ( ρ 1 , a ) -zCDP and M (cid:48) ( · , y ) satisﬁes ( ρ 2 , a ) -zCDP for ∀ y ∈ Y .
Then, mechanism M (cid:48)(cid:48) : D n → Z (deﬁned by M (cid:48)(cid:48) ( x ) =
M (cid:48) ( x, M ( x )) ) satisﬁes ( ρ 1 + ρ 2 ) -zCDP.
Deﬁnition B.1 (Sensitivity) .
The sensitivity of a gradient query ∇ t to the dataset { x i } Ni =1 is
[Equation]
(19) where ∇ ( n ) t denotes the gradient of the n -th sample.
Lemma B.2 (Gaussian mechanism (Bun & Steinke, 2016)) .
Let f : D n → Z have sensitivity ∆ .
Deﬁne a randomized algorithm M : D n → Z by M ( x ) ← f ( x ) + N (0 , ∆ 2 σ 2 I ) .
Then M satisﬁes 12 σ 2 -zCDP.
Lemma B.3 ((Bun & Steinke, 2016)) .
If M is a mechanism satisfying ρ -zCDP, then M is ( ρ + 2 (cid:112) ρ ln(1 /δ ) , δ ) -DP for any δ > 0 .
By solving ρ + 2 (cid:112) ρ ln(1 /δ )
= (cid:15) , we can get ρ = (cid:15) + 2 ln(1 /δ )
+ 2 (cid:112) ln(1 /δ )
( (cid:15) + ln(1 /δ ) .
B. A UXILIARY LEMMAS
Lemma B.4.
If max n (cid:107) x n (cid:107) 2 = 1 and 1 N (cid:80) n x n = 0 , then the gradient sensitivity of the squared loss will be
[Equation]
where Θ M is the set of all possible parameters θ t generated by the learning algorithm M .
Proof.
According to the deﬁnition of sensitivity in Eq.
(19), we have
[Equation]
where we use i denotes the index of sample in the dataset.
Here, we assume it is constant 1.
We may get
[Equation]
where f ( θ ; x i )
=
12 ( x (cid:62) i θ − 1) 2 .
Thus,
[Equation]
Since (cid:107) x n (cid:107) 2 ≤ 1 and 1 N (cid:80) Nn =1 x n = 0 , f ( [Equation] ) = 1 2 N N (cid:88) n
[Equation]
n
[Equation]
Lemma B.5.
Assume assumptions in Theorem 4.4 are satisﬁed.
Given variables deﬁned in Theorem 4.4, the following inequality holds true:
[Equation]
Proof.
We ﬁrst handle the inner summation.
By smoothness, the inequality (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤
M (cid:107) x − y (cid:107) holds true.
Thus,
[Equation]
(cid:107) [Equation] (cid:107)
[Equation]
(cid:19) where the last inequality is by Cauchy-Schwartz inequality.
Because 1 b t = {+
+}
[Equation]
≤ {+
+}
[Equation]
and η t = η 0 2 M ,
[Equation]
where I ( · ) is the indicating function which output 1 if the condition holds true, otherwise 0 .
Denote the left-hand-side of the conclusion as LHS.
We plug Eq.
(20) into LHS to get
LHS ≤
T (cid:88)
[Equation]
where we relax the upper bound by 1 b t = [Equation]
Using Lemma B.6 can directly lead to the conclusion:
[Equation]
Lemma B
[Equation] Given variables deﬁned in Theorem 4.4, the following inequality holds true:
[Equation]
Proof.
We ﬁrst derive the summation:
[Equation]
[Equation] Now, we substitute U 1 ( t, i ) into LHS and replace t − i by j , i.e., t =
j + i , to get
LHS =
T (cid:88) t
= γ T − t t − 1 (cid:88) i =
(cid:107) v
i +1 (cid:107) 1 − β (cid:18) ( t − i )
β t − i + β t − i +1 − β 1 − β (cid:19) =
T − 1 (cid:88) i
= (cid:107) v i +1 (cid:107)
T (cid:88) t
= i + γ T − t 1 − β (cid:18) ( t − i )
β t − i + β t − i +1 − β 1 − β (cid:19) =
T − 1 (cid:88) i
= (cid:107) v
i +1 (cid:107)
T − i (cid:88)
j = γ T − ( j + i ) 1 − β (cid:18) jβ j + β j +1 − β 1 − β (cid:19) =
T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107)
T − i (cid:88)
j = γ − j 1 − β (cid:18) jβ j + β j +1 − β 1 − β (cid:19) ≤ 1 − β
T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107)
T − i (cid:88)
j = (cid:32) j (cid:18) β γ (cid:19)
j
+ β 1 − β (cid:18) β γ (cid:19) j (cid:33)
Let a = β/γ , we show
T − i (cid:88)
j = ja [Equation] =
T − i (cid:88)
j = j (cid:88) o
= a j =
T − i (cid:88) o =
T − i (cid:88)
j
= o a j =
T − i (cid:88) o
= ( a o − a T − i + 1 − a ) = a − a T − i + (1 − a )− ( T − i ) a T − i + 1 − a ≤ a (1 − a ) 2 .
Thus,
LHS ≤ 1 − β
T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107)  a (1 − a ) 2 + β 1 − β
T − i (cid:88)
j = a j   ≤ 1 − β
T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107) 2 (cid:18) a (1 − a ) 2 + β 1 − β a 1 − a (cid:19) ≤ a (1 − a ) 2 (1 − β ) T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107)
Because γ < 1 , β < a = β/γ and a (1 − a ) 2 + β 1 − β a 1 − a ≤ 2 a (1 − a ) 2 .
Therefore,
LHS ≤ 2 a (1 − a ) 2 (1 − β ) T − 1 (cid:88) i = γ T − i (cid:107) v i +1 (cid:107) 2 = 2 βγ ( γ − β ) 2 (1 − β ) T − 1 (cid:88) i
= γ T − i (cid:107) v
i
[Equation]
Lemma B.7.
Suppose γ ∈ (0 , 1) and β ∈ (0 , 1) .
Deﬁne
[Equation]
If t ≤ ˆ T , 1 − β 1 − β t ≤ γ t − 1 for t = 1 , . . .
, T .
If t > ˆ T , 1 − β 1 − β t < γ ˆ T − 1 .
Proof.
Deﬁne h
( t ) =
γ t − 1 (1 − β t ) whose derivatives are
[Equation]
Simple calculation shows 1 − β t (1 + log γ β ) (cid:12)(cid:12) t
=
= − log γ β < 0 and lim t →
+ ∞ 1 − β t (1 + log γ β )
= 1 .
When t = − log β (1 + log γ β ) denoted as t 0 , 1 − β t (1 + log γ β ) = 0 .
Because 1 − β t (1 + log γ β ) is monotonically increasing by t and γ t − 1 ln γ is negative, h (cid:48)
( t ) ≥if t ≤ t 0 .
Otherwise, h (cid:48) ( t ) < 0 .
Therefore, h ( t ) is a concave function.
Because h (1) = 1 − β and h ( ˆ T ) = γ ˆ T − 1 (1 − β ˆ T ) ≥ 1 − β > 0 , h ( t ) ≥ 1 − β for t = 1 , . . .
, ˆ T .
Thus, for all t ∈ [1 , ˆ T ] , we have 1 − β 1 − β t ≤ γ t − 1 .
For t > ˆ T , because 1 − β 1 − β t monotonically increases by t , we have 1 − β 1 − β t < 1 − β 1 − β ˆ T ≤ γ ˆ T − 1 .
C P ROOFS
Proof of Theorem 3.1.
Because all sample gradient are G -Lipschitz continuous, the sensitivity of the averaged gradient is upper bounded by G/N .
Based on Lemma B.2, the privacy cost of g t is 12 σ 2 t 1 .
Here, we make the output of each iteration a tuple of ( θ t +1 , v t =1 ) .
For the 1 st iteration, because θdoes not embrace private information by random initialization, the mapping,
[Equation]
is ˆ ρ 1 -zCDP where ˆ ρ 1 = 12 σ 2 t .
Suppose the output of the t -th iteration, ( θ t , v t ) , is ˆ ρ t -zCDP.
At each iteration, we have the following mapping ( θ t , v t )
→ ( θ t +1 , v t +1 ) deﬁned as
[Equation]
Thus, the output tuple ( θ t +1 , v t +1 ) is (ˆ ρ t + 12 σ 2 t )
-zCDP by Lemma B.1.
Thus, the recursion implies that ( θ T +1 , v T +1 ) has privacy cost as
[Equation]
Let ρ =
ˆ ρ T +1 .
Then we can get the conclusion.
C. G RADIENT D ESCENTS
Proof of Theorem 4.1.
With the deﬁnition of smoothness in Deﬁnition 3.3 and Eq.
(1), we have f ( θ t
[Equation]
1 ) − [Equation] ( θ [Equation]
) ≤ − η t ∇ (cid:62)
[Equation]
( ∇
[Equation]
+ Gσ
[Equation]
/N )
{++ 
 [Equation] 2 t (cid:107)∇ t + Gσ t ν t /N (cid:107)= − η [Equation]+}
[Equation]
2 Mη t ) (cid:107)∇ t (cid:107) 2 − (1 − [Equation] [Equation] where the last inequality is due to the Polyak-Lojasiewicz condition.
Taking expectation on both sides, we can obtain
[Equation]
which can be reformulated by substacting f ( θ ∗ ) on both sides and re-arranged as
[Equation]
Recursively using the inequality, we can get
[Equation]
Let η t ≡ 1 /M .
Then the above inequality can be simpliﬁed as
[Equation]
Proof of Theorem 4.2.
The minimizer of the upper bound of Eq.
(3) can be written as
[Equation]
where we substitute σ 2 = T/R in the second line.
To ﬁnd the convex minimization problem, we need to vanishing its gradient which involves an equation like Tγ T =
c for some real constant c .
However, the solution is W k ( c ) for some integer k where W is Lambert W function which does not have a simple analytical form.
Instead, because γ T > 0 , we can minimize a surrogate upper bound as following
[Equation]
where we use the surrogate upper bound in the second line and utilize γ = 1 − 1 κ .
However, the minimizer of the surrogate objective is not optimal for the original objective.
When κ is large, the term, − ακγ T T , cannot be neglected as we expect.
On the other hands, T suffers from explosion if κ → ∞ and meanwhile 1 /γ
→ + 1 .
The tendency is counterintuitive since a small T should be taken for sharp losses.
To ﬁx the issue, we change the form of T ∗ as
[Equation]
which gradually converges to 0 as κ → ∞ .
Now we substitute Eq.
(23) into the original objective function, Eq.
(21), to get [Equation] (cid:18)
[Equation]
γ )
α
[Equation]
Notice that
[Equation]
because κ ≥ 11 − c > 1 for some constant c ∈ (0 , 1) .
In addition,
[Equation]
Now, we can get the upper bound of Eq.
(24) as
[Equation]
for some constants c 1 , c 2 and large enough 1 α .
Also, we can get the lower bound
[Equation]
where we use the condition c ∈ (0 , 1) .
Thus, ERUB [Equation]
=
[Equation]
Proof of Lemma
[Equation] By (cid:80) Tt =1 σ − 2 = R and Cauchy-Schwarz inequality, we can derive the achievable lower bound as
[Equation]
where the inequality becomes equality if and only if s/σ 2 t = q t σ 2 t , i.e., σ t = ( s/q t ) 1 / 4 , for some positive constant s .
The equality (cid:80) Tt =1 σ − 2 t =
R immediately suggests √ s = 1 R (cid:80)
Tt =1 √ q t .
Thus, we get the σ t .
Notice
[Equation]
where the variance is w.r.t.
t .
Proof of Theorem 4.3.
The upper bound of Eq.
(3) can be written as
[Equation]
where we make use of Lemma 4.1.
Then, the minimizer of the ERUB is
[Equation] [Equation]
We can substitute Eq.
(26) into ERUB dyn to get
[Equation]
Notice that (cid:0) 1 − √ γ (cid:1) − 2 = κ 2 + κ 2 − κ + 2 κ (cid:112) κ ( κ − 1)
= κ (2 κ − 1 + 2 (cid:112) κ ( κ − 1))
and it is bounded by
[Equation]
Therefore, κ ≤ (cid:0) 1 − √ γ (cid:1) − 1 ≤ 2 κ , with which we can derive
[Equation]
Thus, ERUB dyn min =
Θ (cid:16) κ 2 α κ
α +1 (cid:17) .
C. G RADIENT D ESCENTS WITH M OMENTUM
Proof of Theorem 4.4.
Without loss of generality, we absorb the Cσ t /N into the variance of ν t such that ν t ∼ N (0 , Cσ 2 t N I ) and g t ← ∇ t + ν t .
Deﬁne b t = 1 − β t .
By smoothness and Eq.
(1), we have
[Equation]
where only the U 1 ( t ) is non-negative.
Speciﬁcally
[Equation]
U 1 ( t ) describes the difference between current gradient and the average.
We can expand v t +1 to get an upper bound:
[Equation]
= (1 − β ) 2 (cid:13)(cid:13)(cid:13)(cid:88) t i =1 β t − i ( ∇ t − g i ) (cid:13)(cid:13)(cid:13)= (1 − β ) 2 (cid:13)(cid:13)(cid:13)(cid:88) t i =1 β t − i ( ∇ t − ∇ i ) + (cid:88) t i =1 β t − i ( ∇ i − g i ) (cid:13)(cid:13)(cid:13)≤ 2(1 − β ) 2 (cid:20)(cid:13) (cid:13)(cid:13)(cid:88) t i =1 β t − i ( ∇ t − ∇ i ) (cid:13) (cid:13)(cid:13) 2 + (cid:13) (cid:13)(cid:13)(cid:88) t i =1 β t − i ( ∇ i − g i ) (cid:13) (cid:13)(cid:13) 2 (cid:21) ≤ [Equation]
[Equation] β )  [Equation] (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
variance) +( [Equation] ) (cid:13)(cid:13)(cid:13)(cid:88) t i
=1 β t − i ν
i (cid:13)(cid:13)(cid:13)
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) noise variance 
 where we use (cid:107) x + y (cid:107) 2 ≤ ( (cid:107) x
(cid:107) + (cid:107) y (cid:107) ) 2 ≤ 2( (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 ) .
The last inequality can be proved by
Cauchy-Schwartz inequality for each coordinate.
We plug the U 1 ( t ) into Eq.
and use the PL condition to get
[Equation]
Rearranging terms and taking expectation to show
[Equation]
where γ = 1 − η 0 /κ
= 1 − 2 µη t .
The recursive inequality implies
E [ f ( θ T +1 )] − f ( θ ∗ ) ≤ γ T ( f ( θ 1 ) − f ( θ ∗ )) +
[Equation]
( f ( θ 1 ) − f ( θ ∗ )) + T (cid:88) t = γ T − t 2(1 − β ) η t b t
{+E [ U 2 ( t )] − T (cid:88) t = γ T − t η t b 2 t (1 − 
 Mη t ) E (cid:107) v t +1 (cid:107)(cid:124) (cid:123)(cid:122) (cid:125)+}
U 4 ( t ) .
where we utilize α =
DC 2 2 MN 2 R 1 f ( θ 1 ) − f
( θ ∗ ) and η t =
η 0 2 M .
By Lemma B.5, we have
T (cid:88)
[Equation]
=
[Equation]
γ
[Equation]
t b
[Equation]
U
[Equation]
t ) ≤ [Equation] 2 M ( [Equation]
( γ
[Equation]
[Equation]
T − 1 (cid:88) i
= γ T − i (cid:107) v i +1 (cid:107) 2 .
Thus, by 1 b t ≥ 1 ,
U 4 ( t ) ≤ η 30 βγ 2 M (1 − β ) 3 ( γ − β ) T − [Equation] (cid:88) i = γ
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
M (
[Equation]
η
[Equation] T (cid:88) t = [Equation] γ T
[Equation]
t E (cid:107) v t +
[Equation]
(cid:107)= − η 0 2 M ζ T (cid:88) t = γ T − t E (cid:107) v t +1 (cid:107) where
[Equation]
When a small enough η 0 , e.g., Speciﬁcally,
[Equation]
We can have ζ ≥ 0 .
By the deﬁnition of U 3 ( T, σ ) , we can get
[Equation]
Proof of Theorem 4.5.
Since σ t is static, by deﬁnition of U 3 in Theorem 4.4,
[Equation]
Because 1 − β 1 − β t 1+ β t 1+ β ≤ 1 , the U 3 will be smaller than the corresponding summation in GD with uniform schedule.
By Lemma B.7, when T > ˆ T , we can rewrite U 3 as
[Equation]
where we use σ 2 = T/R in the last line.
Without assuming T > ˆ T , we can generally write the upper bound as
[Equation]
By Theorem 4.4, because ζ ≥ 0 , we have
[Equation]
(cid:33)
[Equation]
where α (cid:48)
=
2 η 0 α .
First, we consider T ≤ ˆ T .
Use T =
[Equation]
to get
[Equation]
[-O (cid:18) κ κ + η 0 /α (cid:18) 
 + 4 κ γ (cid:19)(cid:19) = O (cid:18) κκ + η 0 /α (cid:19)-] where we used ln(1 /γ ) ≥ η 0 /κ and ln(1 + x ) ≤ √ x for any x > 0 .
Second, when T > ˆ T ,
[Equation]
Make use of T =
(cid:108)
[Equation]
to show
[Equation]
Proof of Theorem 4.6.
By Lemma B.7, we can rewrite U 3 as U
[Equation]
≤
[Equation]
T (cid:88)
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation] + [Equation] We derive V 1 and V 2 separately.
For V 1 , we can obtain the upper bound by
[Equation]
For V 2 , we can derive
[Equation]
We ﬁrst consider the ﬁrst term
[Equation]
Similarly
[Equation]
we have
[Equation]
Thus,
[Equation]
Substitute V 1 and V 2 into U 3 to get
[Equation]
where
[Equation]
=
[Equation] where c 1 = [Equation] and c 2 = γ 2ˆ T γ − β 2 .
When T > ˆ T , by Lemma 4.1, the lower bound of R (cid:80) Tt =1 q t σ 2 t is (cid:18)(cid:88) T
[Equation]
 which is achieved when
[Equation]
q
[Equation]
By Theorem 4.4, because ζ ≥ 0 , we have
[Equation]
And the minimum of the upper bound is
[Equation]
where α (cid:48)
=
2 η 0 c 2 α .
Let T = [Equation]
Then,
[Equation]
where c 3 is some constant
[Equation]
When T ≤ ˆ T ,
[Equation]
with which we obtain
[Equation]
where we let q t = γ T + t .
By Lemma
[Equation]
[Equation]
Thus
[Equation]
Let T =
(cid:108) [Equation]
Then,
[Equation]
In summary,
[Equation]
C. S TOCHASTIC G RADIENT D ESCENTS
Proof of Theorem 4.7.
Let ˜ ∇ t be the stochastic gradient of the step t .
By the smoothness, we have
[Equation]
Note that E ( σ g ξ t /n + Gσ t ν t /n )
=
and E ( σ g ξ t /n + Gσ t ν t /n )
2 =
σ 2 g + ( Gσ t /n ) 2 .
Without loss of generality, we can write σ g ξ t + Gσ t ν t as ˜ σ t ζ t where ˜ σ t (cid:44) (cid:113) σ 2 g + ( Gσ t ) 2 and ζ t is a random vector with E ζ t = 0 and E (cid:107) ζ t (cid:107) 2 ≤ D .
Therefore,
[Equation]
Then following the same proof of Theorem 4.1, we can get
[Equation]
where R
[Equation]
Proof of Theorem 4.8.
Without loss of generality, we can write σ g ξ t + Gσ t ν t as ˜ σ t ζ t where ˜ σ t (cid:44) (cid:113) σ 2 g + ( Gσ t ) 2 and ζ t is a random vector with E ζ t = 0 and E (cid:107) ζ t (cid:107) 2 ≤ D .
Therefore, we replace ν t by ζ t and σ 2 t by ˜ σ 2 t /G
= σ 2 g /G 2 + σ 2 t .
Now, we only need to update U 3 ( σ, T ) as
[Equation]
where we deﬁne
[Equation]
We can upper bound U g 3 by
[Equation]
Combine with the factors of U 3 in the PGD bounds:
[Equation]
D E XPERIMENTS
Dataset .
(1) Synthetic data .
We generate a 100 -dimensional dataset including linearly separable data points using sckit-learn package.
The data points are distributed in two points in the hyper-cubic with Euclidean distance of 10 .
In total, 1000 samples are generated for training with the logistic loss.
(2) Real data .
We create a subset of the MNIST dataset (Lecun et al., 1998) including 1000 handwritten images of digit 3 and 5 (MNIST35).
Compared to the original dataset ( 70 ,samples), the small set will be more vulnerable to attack and the private learning will require larger noise (see the 1 /N factor in Eq.
Following the preprocessing in (Abadi et al., 2016), we project the vectorized images into a 60 -dimensional subspace extracted by PCA.
Setup .
The samples are ﬁrst normalized so that (cid:80) Nn =
1 x n =
and the standard deviation is 1 .
Then the sample norms are scaled such that max n (cid:107) x n (cid:107) =
10 (i.e., data scales ).
We ﬁx the learning rate to 0 .
1 based on the corresponding experiments of non-private training (same setting without noise).
The total privacy budget is 0 .
1963 -zCDP (equal to (4 , 10 − 8 ) -DP) which implies R = 0 .
To control the sensitivity of the gradients, we clip gradients by a clipping norm ﬁxed at 4 .
Formally, we scale down the sample gradients to length 4 if its norm is larger than 4 .
Because the schedule highly depends on the iteration number T , we grid search the best T for compared methods.
Therefore, we ignore the privacy cost of such tuning in our experiments which protocol is also used in previous work (Abadi et al., 2016; Wu et al., 2017).
All the experiments are repeated 100 times and metrics are averaged afterwards.
Estimate of inﬂuences by retraining .
In the left pane of Fig.
3, we estimate the inﬂuence of σ t by retraining multiple times.
When keeping σ i for i (cid:54)
=
t ﬁxed (as the ﬁne-tuend uniform schedule σ ), [Figure] the value of σ t is varied from 20 to 200 .
Then we ﬁt a quadratic function of σ t where the coefﬁcient is treated as estimation of q t .
Repeating the estimation for all t in range 1 to 100 could provide us the trend of inﬂuence in the middle pane.
Results .
In the left pane of Fig.
3, the squared ﬁnal loss (i.e., f ( θ T +1 ) ) is approximately a quadratic function of the σ t with small relative variance (i.e., the variance divided by the mean value, 0 .
016 for t = 10 , 60 , 90 , respectively).
When σ t increases, the frequency of clipping increases as well which leads to more variance in the ﬁnal losses.
We use the least square method to ﬁt the relation shown as the solid lines.
The ﬁnal logistic loss is more sensitive to the noise because of the additional uncertainty from the changing Hessian.
We still ﬁt a quadratic function on σ t .
It turns out the relative variances of the quadratic coefﬁcient (approximately the inﬂuence) is small which are 0 .
027 for t = 10 , 60 , 90 , respectively.
The middle pane shows the relationship between the estimated q t and learning steps.
The q t of squared loss is computed by analytical solution using the Hessian eigen values.
The q t of logistic loss is computed by retraining based on uniform schedule.
Both loss functions show an increased inﬂuence as learning continues, which indicates that the dynamic schedule should be decreasing accordingly.
The squared loss has a rather steep trend while the logistic has a relative ﬂatten one.
The reason is that the logistic loss has a larger variance in the gradient norm and therefore clipping happens more frequently (approximately more than 80% gradients are clipped).
As a result, the variance of inﬂuence will be relative small for logistic losses.
Moreover, we vary the data scale (scale all samples uniformly such that all sample norms are less than a speciﬁc value), changing the variance of inﬂuence, as seen in the ﬁgures.
The last pane compares uniform, dynamic and exponential decay schedules Yu et al.
(2019) using ﬁnal losses relative to the uniform schedule.
We set the exponential decay schedule to approximate the dynamic schedule by ﬁtting ˆ σ t = σ 0 exp( − kt ) using the least squares method.
The ﬁnal losses are picked by grid searching the best T ∈ [1 , 100] .
We see that the advantage of the dynamic schedule over the uniform one increases when data scales less than 15 .
But we also notice some loss gaps decrease, suggesting that the data scale is not the inherent reason for the dynamic advantage.
According to Lemma 4.1, the advantage should be proportional to T 2 Var( q t ) which is shown to be decreasing when the data scale is larger than 15 .
When the data scale continues to increase, gradient clipping will change the curvature of the loss function.
Therefore, a increasing Var( Tq t ) is witnessed.
Meanwhile, the loss gap decreases.
In the last row of Fig.
3, the DNN is experimented with the same setting.
Though the inﬂuence increases by t , the variance is small and less dependent on the data scale in comparison to shallow models.
Though the dynamic schedule estimated by retraining inﬂuences does not performs stably, the exp method still decreasingly depends on the variance of inﬂuence as expectation.