Gradient Descent: Robustness to Adversarial Corruption
Abstract
Optimization using gradient descent (GD) is a ubiquitous practice in various machine learning problems including training large neural networks.
Noise-free GD and stochastic GD–corrupted by random noise–have been extensively studied in the literature, but less attention has been paid to an adversarial setting, that is subject to adversarial corruptions in the gradient values.
In this work, we analyze the performance of GD under a proposed general adversarial framework.
For the class of functions satisfying the Polyak-Łojasiewicz condition, we derive finite time bounds on a minimax optimization error.
Based on this bound, we provide a guideline on the choice of learning rate sequence with theoretical guarantees on the robustness of GD against adversarial corruption.
1. Introduction
Optimization using gradient descent (GD) is central to training machine learning models.
In this work, we consider the dynamics of GD under adversarial corruptions, which to our knowledge is a novel formulation.
Optimization using corrupted gradients has been studied in many forms [e g., 2, 7, 15], as discussed in section 1.2 depending on the practical problem of interest.
Contrary to these studies, our framework is not limited to a specific real-world application, but rather attempts to understand the robustness of gradient descent under the worst possible corruption.
In this framework, at each GD iteration n , an adversary corrupts the values of the gradients, up to a budget constraint.
We refer to this framework as adversarial GD (Adv-GD).
The Adv-GD framework yields a mini-max performance measure: the optimization error minimized by GD and maximized by the adversary; that can also be interpreted as a game between the optimization algorithm and the adversary.
Compared to stochastic GD with random corruptions, adversarial corruptions are more challenging to analyze for at least two reasons.
In stochastic GD, it is typically assumed that the noisy gradient is an unbiased estimator of the true gradient.
Adversarial corruptions, however, may introduce bias to the GD dynamics which are more challenging to keep track of.
In addition, in stochastic GD, typically a constant upper bound on the variance of the noisy gradients is as- sumed.
In Adv-GD framework, however, the adversary is allowed to spend an arbitrary amount of its budget, which can be relatively large, to corrupt a gradient observation.
In this work, we focus on the objective functions satisfying the PL condition, which is a very general and practically relevant class of functions, including many machine learning models, e.g., overparameterized neural networks [e.g., see, 5].
Our contributions can be summarized as follows:
• We introduce a novel framework for the analysis of GD under adversarial corruptions, and prove finite time minimax error bounds for the class of functions satisfying the PL condition.
(see, Section 2) •
We generalize the measure of corruption from l 2 norm —the case of variance— to arbitrary l q norms, with q ≥ 1 , which may be of broader interest.
(see, Section 2) •
We provide a guideline on the choice of learning rate sequence with theoretical guarantees on the robustness of GD algorithm against adversarial corruption.
(see, Section 3)
1.1. Motivation
We provide a general framework for evaluating the robustness of GD to adversarial corruption.
In typical analytical results for GD, the corruption is modelled as a stochastic noise (usually with bounded variance), and the convergence is given in expectation.
Alternatively, we provide a worstcase error under a corruption budget assumption.
This interpretation of convergence can be generally applied to all the use cases of GD as an upper bound on the optimization error when the nature of corruption is unknown.
In many modern machine learning applications, such as in IoT and federated learning, the gradients of the model are communicated to a server [3, 11].
The gradients may be arbitrarily corrupted due to communication issues.
The communication may also be intercepted maliciously with the aim of adversarially corrupting the performance.
In these cases the assumption of a stochastic corruption is not suitable.
Our results provide bounds on the convergence error of GD based on the amount of corruptions.
In simple words, we characterize the amount of corruption in gradient values (in terms of l q norm) that is tolerable for GD optimization.
1.2. Related Work
A different approach to adversarial optimization has been considered under the well-established online learning literature.
In this approach, a sequence of typically convex objective functions are selected by an adversary.
The performance is measured as the total loss compared to the cumulative value of the objective functions evaluated at a single point.
Several gradient based algorithms [e.g., online GD , 8, 9, 19], and non gradient based algorithms [e.g., follow the (perturbed) leader , 1, 10] are introduced in this setting.
See [13, 16] for surveys.
In contrast, in Adv-GD framework, the adversary directly corrupts the gradient values.
In addition, the adversary can allocate the corruptions arbitrarily (see the minimax optimization error defined in (3)).
With any growing budget, this violates the assumptions in the online learning setting.
Due to their inherently different assumptions, our results cannot be directly compared with the ones given in the online learning setting.
Several existing work considered GD under adversarial corruption of gradients.
The work of [2] considered Byzantine stochastic GD in a distributed stochastic optimization setting, where a fraction of machines which compute stochastic gradients in every iteration are Byzantine, and may behave adversarially.
In this setting, the main focus of the problem is on detecting Byzantine machines in the presence of available true gradient values.
Similarly, [7] and [15] considered an adversary corrupting a fraction of sample gradients.
Their work focused on detecting the corrupted samples as outliers.
This line of work is different from ours in that, in our setting, the adversary is allowed to arbitrarily allocate the corruptions through the optimization sequence, leading to a minimax (worst case) performance measure.
2. Gradient Descent with Adversarial Corruptions
In this section, we present the Adv-GD framework, as well as other relevant notations and definitions.
Consider the problem of minimizing a differentiable objective function f : X → R , where the domain X is a compact subset of R d .
The goal is to get as close as possible to a global minimum x ∗ ∈ arg min x ∈X f ( x ) .
Recall the standard iterative updating rule of (stochastic) GD:
[Equation]
where k
= 1 , 2 , . .
. is a discrete iteration index, the iterations start at an arbitrary (possibly random) initial point x 0 , and ( η k ∈ R ) ∞
k =1 is the sequence of learning rates.
In vanilla GD, we have simply ˜ g
( x k − 1 )
=
g ( x k − 1 ) , using the notation g ( · ) = ∇ f ( · ) for the gradient.
In stochastic GD, ˜ g
( x k − 1 ) is typically assumed to be an unbiased estimator of g ( x k − 1 ) with bounded variance [e.g., see, 6, and references therein].
That is, in stochastic GD, we have randomly corrupted gradient estimates ˜
g
( x k − 1 ) =
g ( x k − 1 ) + ξ k , where ξ k is a zero mean random observation noise at iteration k , with bounded variance, independent and identically distributed over iterations.
In the Adv-GD framework, we consider adversarially corrupted gradient values.
Specifically, at each iteration k , the optimizer asks for the value of gradient g ( x k − 1 ) .
The gradient values, however, may be adversarially corrupted, so that the optimizer receives ˜ g ( x k − 1 ) ∈ R d .
We are interested in the dynamics of GD update rule (1), under this setting.
Budget Constraint:
With no constraints, the setting is not interesting, as the adversarial corruptions can set the sequence ( x k ) ∞ k =1 to any values in R d .
It is therefore natural to set a budget constraint on adversarial corruptions.
In particular, consider a distance measure D ( ., . )
: R d × R d →
R ≥ 0 .
Let c ( k ) :
=
D ( g ( x k − 1 ) , ˜ g ( x k − 1 )) quantify the amount of corruption in the gradient value, allocated by the adversary, at iteration k .
Then, the total corruption budget b : N → R > 0 imposed on the adversary is a positive and non-decreasing sequence limiting the amount of corruptions:
[Equation]
where ∥ · ∥ l p denotes the l p norm.
We use the notation ∥ · ∥ exceptionally in the case of l 2 norm.
Given a budget, let us define the set of admissible corruptions as follows.
Definition 2.
For a given budget b and time n , the set of admissible corruptions C nb,p is defined as
[Equation]
Performance Measure:
In the Adv-GD framework, the finite time performance of the optimizer is measured in terms of a minimax optimization error; minimized by the optimizer and maximized by the adversary.
Specifically, at each iteration n , the minimax optimization error is defined as
[Equation]
where ξ k = ˜ g
( x k − 1 ) − g ( x k − 1 ) .
Recall that x n is the point after n -th iteration in the GD update rule, x ∗ ∈ arg min x ∈X f ( x ) is a global minimum, and c ( k )
=
D ( g ( x k − 1 ) , ˜ g ( x k − 1 )) is the amount of corruption at iteration k .
The sup is taken over all admissible corruption sequences.
Game Theoretic Interpretation
The Adv-GD framework can be interpreted as a zero-sum game between the optimizer and the adversary.
Before starting the GD iterations, the optimizer selects the learning rate sequence ( η k ) nk =1 .
With the knowledge of the learning rate sequence, the adversary selects the gradient corruption sequence ( ξ k ) n k =1 subject to the admissibility constraint: ( c ( k ))
n k =1 ∈
C nb,p .
At the end of n iterations, the optimizer incurs a loss of f ( x n ) − f ( x ∗ ) , while the adversary incurs a loss of f ( x ∗ ) − f ( x n ) .
The goal of each player is to minimize their own loss.
Corruption Measure
We measure the corruption at each iteration, using l q norm of the distance between the true and corrupted gradient values.
In particular, in our analysis, we consider
[Equation]
In the case of stochastic GD, a typical assumption is the boundedness of the variance of the noisy gradients at each iteration; E [ ∥ ξ k ∥ 2 ] is assumed bounded.
This can be interpreted as measuring the random corruptions in terms of l 2 norm.
In this sense, we generalize the typical measure of corruption from l 2 norm to l q norm (in addition to considering adversarial corruptions, in contrast to random ones).
Considering also the freedom in choosing p for the norm of corruption sequence in eq.
2, we investigate a quite general setting of gradient corruption in gradient descent.
We present our results for the class of functions satisfying the PL condition, with additional assumptions:
Definition 2.2 (PL Condition)
We say that a function satisfies the PL condition with parameter α ∈ R > 0 , if ∥ g ( x ) ∥ 2 ≥ α ( f ( x ) − f ( x ∗ )) , ∀ x ∈ X .
Recall the notation g ( · ) =
∇ f ( · ) .
We use the notation F α PL to denote the class of functions satisfying the PL condition with parameter α .
Definition 2.3 (Strong Convexity)
We say that a function is strongly convex with parameter µ , if f ( x ′ ) ≥ f
( x ) + ⟨ g ( x ) , x ′ − x ⟩ + µ 2 ∥ x ′ − x ∥ 2 , ∀ x ∈ X .
Definition 2.4 (Smoothness)
We say that a function is smooth with parameter β , if f ( x ′ ) ≤ f ( x )+ ⟨ g ( x ) , x ′ − x ⟩ + β 2 ∥ x ′ − x ∥ 2 , ∀ x ∈ X .
We use the notation F β S to denote the class of smooth functions with parameter β .
3. Minimax Error Bounds under the Adv-GD Framework
In this section, we present our main result on the finite time minimax (worst-case) optimization error bound under the Adv-GD framework.
We prove error bounds which depend on the error at the initialization and the corruption budget.
[Table]
Domain r n Minimax optimization error a ∨ b max { a, b } [Table] Objective function b ( n )
Budget up to time n included a ∧ b min { a, b }
[Table]
Iteration indexed points in domain c ( k )
Budget spent at time k a + max( a, 0)
[Table]
PL parameter ˜ g ( · )
Corrupted gradient g ( · )
Gradient
[Table]
Strong convexity
[Table]
ξ k
Corruption at time k η k
Learning rate
[Table]
Theorem 3.1 (Adv-GD: Finite Time Optimization Error) Consider the Adv-GD framework formalized in Section 2.
Assume f ∈ F α PL ∩ F β S , and ∥ g ( x ) ∥ ≤ γ for all x ∈ X .
Let η k = (cid:16) 2 α (cid:16) 1 − ( kk +1 ) ω (cid:17) ∧ 1 β (cid:17) for a fixed ω ≥ 1 .
Define n 0 = min (cid:110) n ∈ N : 2 α (cid:16) 1 − (cid:16) nn +1 (cid:17) ω (cid:17) ≤ 1 β (cid:111) .
Define d q = d ( 12 − 1 q ) + .
We have the following bound on the minimax optimization error defined in (3). r
[Equation]
, where ( A n ∈ R > 0 ) ∞ n =1 , ( B p,n ∈ R > 0 ) ∞ n =1 and ( C p,n ∈ R > 0 ) ∞ n =1 are sequences satisfying: ♢
For n < n 0 : A n = (1 − α 2 β ) n , B p,n = 0 , C p,n ≤ d 2 q 2 β when 1 ≤ p ≤ 2 , and C p,n ≤ d 2 q α when p > 2 .
♢ For n ≥ n 0 : A n ∈ O ( n − ω ) , B p,n ∈ O ( d q n − 1 p ) , and
[Equation]
when 1 ≤ p ≤ 2 and ω ≥ 1 ,
[Equation]
when p > 2 and 1 ≤ ω < 1 + 2 /p,
[Equation]
log
[Equation]
2 /p
[Equation]
, when p > 2 and ω = 1 + 2 /p,
[Equation]
when p > 2 and ω > 1 + 2 /p.
Proofs and further details are provided in the Appendix A. Theorem 3.1 provides very general conditions for the robustness of GD under adversarial corruptions, covering various measures for the corruptions in terms of l p norm of cumulative effect of corruptions over iterations, and l q norm of corruption at each iteration.
In Corollary 3.2, we specialize this theorem for the case of p = 1 , that corresponds to simply adding the corruptions over iterations (cid:13)(cid:13)
[ c ( k )]
Nk =1 (cid:13)(cid:13)
l 1 = (cid:80) nk =1 c ( k ) .
Corollary 3.2 (Adv-GD: Finite Time Optimization Error with p = 1 ) Under the setting of Theorem 3.1, set ω = 2 which amounts to η n = (cid:16) {+
+} α (cid:16) 1 − ( nn +1 ) 2 (cid:17) ∧ 1 β (cid:17) ∼ 1 n .
We then have the following.
For the case of p = 1 :
[Equation]
Recall D ( g ( x k − 1 ) , ˜ g ( x k − 1 ))
=
∥ ξ k ∥ l q .
The case of q = 2 is similar to the typical stochastic GD setting, where, the random corruptions are subject to a bound on their expected l 2 norm.
In Adv-GD framework, d 2 = 1 , by definition.
As q grows larger, d q grows closer to √ d .
Remark 3.3 The particular choice of the learning rate comes from the tradeoff between corruption free optimization and removing the corruption.
A larger learning rate (up to a constant value smaller than 1) is preferred for corruption free GD.
That however cannot efficiently control the corruption.
A very small learning rate can efficiently remove the effect of corruption.
It however results in a very slow convergence for the corruption free part of the optimization.
Efficiently trading off these two directions leads to our choice of learning rate.
The details of this trade off can be found in our analysis in the Appendix A.
Remark 3.4 Theorem 3.1 readily holds true under strong convexity (replacing the PL condition).
This follows from the fact that strong convexity with parameter µ implies PL condition with parameter α = 2 µ .
Numerical Evaluations
We provide numerical experiments for the Adv-GD framework in Appendix B.
The experimental results corroborate our theoretical findings.
4. Conclusion
We introduced the novel framework of Adv-GD, in which gradients are corrupted by an adversary, limited by a given budget.
Thus, our results are applicable for the worst-case scenario of gradient corruptions that might happen in practice.
We provided finite time upper bounds on the worst-case optimization error, for the family of smooth PL functions.
We also show that the robustness of GD algorithm can be achieved via a proper choice of learning rate sequence based on Theorem 3.1 or
Collorary 3.2.
It remains unknown whether the error bounds are optimal.
Future works may also look into obtaining lower bounds for the error, to compare with our upper bounds, and investigate the question of what is the optimal adversarial corruption.
We believe that the novel framework proposed in this study will inspire further theoretical work to address all the open questions.
Appendix A. Proofs and Details of Theoretical Analysis
In this section, we provide a proof for Theorem 3.1.
As a consequence of smoothness assumption ( f ∈ F β S ), we have
[Equation]
The second line follows from the GD update rule (1); the third line follows from the definition of ˜ g ; the fourth line is a rearrangement of the terms.
For simplicity of notation, let us use y k
= f
( x k ) − f ( x ∗ ) and c k = ∥
ξ k ∥ l q .
Also, recall the notation d q = d ( 12 − 1 q ) + .
We can rearrange the last line as follows
[Equation]
The second line follows from ∥ · ∥ ≤ d q ∥ · ∥ l q .
Note that we have ∥ · ∥ ≤ ∥ · ∥ l q for q ≤ 2 , and ∥ · ∥ ≤ d 12 − 1 q ∥ · ∥
l q for q > 2 .
The third line follows from the PL condition ( f ∈ F α PL ).
Note that η k (1 − βη k / 2) ≥ 0 , by definition of η k in the statement of the theorem.
Further simplifying the notation and defining
[Equation]
we can write
[Equation]
We use this recursive relation to bound y n at iteration n based on the following lemma.
Lemma A.1 Let ( y i ∈ R ≥ 0 ) ∞
i =0 be a sequence satisfying
[Equation]
where α i , γ i , β i , c i ∈ R ≥ 0 , for all
i ∈ N .
Define c n = [ c 1 , ..., c n ] .
Then, for p ≥ 1 ,
[Equation]
[Equation]
where
[Equation]
and, λ k,n = (cid:81) ni
= k +1 α
i ; p 1 and p 2 are chosen in a way that 1 /p
+ 1 /p 1 = 1 , and 2 /p
+ 1 /p 2 = 1 .
The proof of Lemma A.1 is given in Section A.1.
The bounds given in Theorem 3.1 then are derived by characterizing the coefficients A n , B p,n , C p,n for the choice of learning rate sequence η k in the statement of the theorem.
Further details are provided in Section A.2.
Optimality of the Bound on y n in Lemma A.1:
We further prove that A n , B p,n and C p,n given in Lemma A.1 are optimal in the following sense.
For all ( ˜ A, ˜ B, ˜ C ) ∈ R 3 such that ˜
A < A n , ˜ B < B p,n , or ˜ C < C p,n , the bound on y n given in Lemma A.1 does not hold true, with ˜ A, ˜ B, and ˜ C replacing the corresponding coefficients.
In other words, the parameters given in (6) in Lemma A. yield the tightest coefficients to which (5) holds true for all systems of inequalities of the form of (4).
This observation is formalized in the following proposition.
Proposition A.2 Fix p ≥ 1 and n ∈ N , and let p 1 , p 2 , A n , B p,n , C p,n , ( α i ) ∞
i =1 , ( γ i ) ∞ i =1 , ( β i ) ∞
i = be defined as in Lemma A.1.
For all ( ˜ A, ˜ B, ˜ C ) ∈ R 3 such that ˜
A < A n , ˜ B < B p,n , or ˜ C < C p,n , there exist sequences ( y i ∈ R > 0 ) ∞ i =
and ( c i ∈ R > 0 ) ∞ i =1 satisfying the recursive relation y i ≤ α i
y
i − 1 + γ i
c i
+ β i c
i , ∀ i ∈ N , and violating (5) , with ˜ A, ˜ B, and ˜ C replacing the corresponding coefficients:
[Equation]
where c n = ( c 1 , ..., c n ) similar to Lemma A.1.
Proof is provided in Section A.3.
We emphasize that Proposition A.2 considers the cases where only one of the coefficients ˜ A , ˜ B or ˜ C is smaller than the corresponding one in Lemma A.1, and not all of them together; the latter would be a weaker statement.
In Lemma A.1, we established a recursive relation on the optimization error y k
= f
( x k ) − f ( x ∗ ) , as follows
[Equation]
using the following notations
[Equation]
Furthermore, in Lemma A.1, we established a bound on y n .
In this section, we provide a proof of Lemma A.1, the details of analysis for the learning rate
η k = (cid:16) 2 α (cid:16) 1 − ( kk +1 ) ω (cid:17) ∧ 1 β (cid:17) as a consequence of Lemma A.1, as well as a proof of Proposition A.2.
A.1. Proof of Lemma A.1 (The Bound on y n Based on the Recursive Relation for y k )
It is clear that
[Equation]
where u n = ( λ k,n γ k ) nk =1 , v n = ( λ k,n β k ) nk =1 , and d n = ( c 21 , ..., c 2 n ) .
It follows by H¨older’s inequality that ⟨ u n , c n ⟩ ≤ ∥ u n ∥ p 1 ∥ c n ∥ p , and
[Equation]
This proves the lemma.
A.
The Analysis of Error Bound for the Choice of η k in Theorem 3.
Recall
[Equation]
It follows by (8) that
[Equation]
Hence λ k,n = (cid:81) ni =
k +1 α
i is given by
[Equation]
The coefficients given in (6) are then as follows:
• If 0 ≤ n < n 0 , then A n = (cid:16) 1 − α 2 β (cid:17) n , B p,n = 0 , and
[Equation]
where the last inequality holds since d 2 q 2 β (cid:16) 1 − (cid:16) 1 −
[Equation]
(cid:17) p 2 (cid:17) − 1 /p 2 is monotonically decreasing for p 2 ∈ [1 , ∞ ) and takes value d 2 q /α at p 2 = 1 . • If n ≥ n 0 , then A n ≤ (cid:16) 1 − α 2 β (cid:17) n 0 − [Equation]
n 0 n + [Equation] ω , and B p,n is given by
[Equation]
where the second last inequality holds since n (cid:88)
[Equation]
1) dt (cid:33) ≤ (cid:18) ω n + (cid:19) p 1 (cid:18) 1+ n + 1+ p 1 ( ω − 1) (cid:19) . For 1 ≤ p ≤ 2 , C p,n is given by C p,n ≤ [Equation]
 [Equation] [Equation] , if ω>
For p > 2 , C p,n is given by C p [Equation] p
[Equation] n ≤
[Equation]
ω (cid:33) p+ n (cid:88) k = n(cid:32) 2 βd 2 q α(cid:18) 1 − (cid:18) k k + (cid:19) ω (cid:19) 2 (cid:18) k
+ n + (cid:19) ω (cid:33)
p≤ (cid:32) d 2 q 2 β (cid:18) nn + (cid:19) ω (cid:33) p 1 − (cid:16) 1 − α 2 β (cid:17) p 2 + (cid:32) 2 βd 2 q α(cid:33) p 2 n (cid:88) k = n(cid:32)(cid:18) ω k
+ (cid:19) 2 (cid:18) k + n + (cid:19) ω (cid:33) p≤ 2 β α (cid:32) d 2 q 2 β (cid:18) nn + (cid:19) ω (cid:33) p+ (cid:32) 2 βd 2 q αω( n +1) ω (cid:33) p 2 n (cid:88) k = n( k +1)
p 2 ( ω − 2) . To analyze the second term, let us use the notation
˜ ω
= p 2 ( ω − 2) and note that
[Equation]
, if ˜ ω
= −
[Equation]
n
[Equation]
+
[Equation]
[Equation] , if ˜ ω ∈ ( −∞ , − 1) ∪ ( [Equation]
[Equation]
≤ ( n
[Equation]
, if
[Equation]
Therefore,
[Equation]
To sum up,
[Equation]
if
[Equation]
if
[Equation]
For example, for the case of p = 1 and ω = 2 , we have
[Equation]
Therefore,
[Equation]
A.3. Proof of Proposition A.
As stated in Proposition A.2 the parameters given in (6) in Lemma A.1 yield the tightest coefficients to which (5) holds true for all systems of inequalities of the form of (4).
In this section, we prove
Proposition A.2.
Let u n and v n be as in the proof of Lemma.
A.1.
• Suppose ˜
A <
A n .
One may take y k
= (cid:81) k
[-ki =
-]i =
α i and c k = 0 for each k , so that both (4) and (7) are satisfied.
• Suppose ˜ B < B p,n .
It follows by H¨older’s inequality that for any r > 0 , there exists c n = ( c 1 , ..., c n ) ∈ R n ≥
so that ∥ c n ∥ p = r and ⟨ u n , c n ⟩ = ∥ u n ∥ p 1 r .
Take non-negative sequence ( y i ) ∞
i =
so that (4) holds with inequalities replaced by equalities, then (recall that B p,n = ∥ u n ∥ p 1 and v n ≥ 0 )
[Equation]
where d n = ( c 21 , ..., c 2 n )
Then (7) holds by taking r > 0 and y 0 >
so that
[Equation]
Suppose ˜ C < C p,n .
For any r > 0 , we can construct c n = ( c 1 , ..., c n ) ∈ R n ≥
so that ∥ c n ∥ p = r and ⟨ v n , d n ⟩ = C p,n r 2 (here d n = ( c 21 , ..., c 2 n ) ) as follows
: – If p > 2 , by H¨older’s inequality, there exists d n = ( d 1 , ..., d m ) ∈ R n ≥
so that ∥ d n ∥ p/ 2 = r 2 and ⟨ v n , d n ⟩ = ∥ v n ∥ p 2 r 2 =
C p,n r 2 .
Take c k = √ d k .
– If 1 ≤ p ≤ 2 , take
[Equation]
k [Equation] r , if k = k0 , otherwise where k 0 ∈ argmax 1 ≤ k ≤ n β k (cid:81) ni = k +1 α
i , then ⟨ v n , d n ⟩ = ∥ v n ∥ ∞
r 2 = C p,n r 2 .
Take non-negative sequence ( y i ) ∞ i =
so that (4) holds with inequalities replaced by equalities, then (recall that u n ≥ 0 )
[Equation]
Then (7) holds by taking r > 0 and y 0 >
so that
[Equation]
Appendix B. Numerical Evaluations
In this section, we provide experimental results on the worst-case optimization error of the GD with adversarial corruptions.
B.1. Experimental Setup GD Loop:
The experiments use a GD loop based on the GD update rule given in (1).
In particular, provided an initial point x 0 , objective function f and its gradient g , the number n of iterations and the corruption sequence ( ξ k ) nk =1 , we iteratively obtain ( x k ) nk
=1 based on (1).
Recall x ∗ =
arg min x ∈X f ( x ) .
Let
[Equation]
denote the optimization error corresponding to a particular choice of corruption sequence ( ξ k ) nk =1 .
In contrast, the worst-case optimization error r n defined in (3) corresponds to the maximum value of R n (( ξ k ) nk =1 ) , for the worst case choice of ( ξ k ) nk =1 subject to the budget constraint.
The learning rate sequence ( η k ) n k =1 is selected according to Corollary 3.2.
Adversary:
The adversary chooses the corruption sequence ( ξ k )
nk =1 in a way that maximizes
R n (( ξ k ) nk =1 ) , and satisfies the budget constraint with p = 1 .
We formulate this as a constrained optimization problem, and solve it using the Sequential Least Squares Programming (SLSQP) method from scipy.optimize package in the SciPy Python library.
This module takes R n : R n → R and a random initialization ( ˆ ξ k ) nk
=1 of the corruption sequence as the inputs and returns a numerical approximation of the worst case corruptions ( ˆ ξ ∗ k ) nk =1 .
We then use R n (( ˆ ξ ∗ k ) nk =1 ) as a numerical approximation of the minimax error r n .
Budget:
We choose b ( n ) ∼ sn ζ for various values of scale s ∈ R > 0 and exponent ζ ∈ [0 , 1] parameters.
In particular, rather than a smooth budget b ( n ) =
sn ζ , we create b ( n ) randomly, in a way that it scales with sn ζ .
In order to do this, we choose b ( n ) =
s ( (cid:80) nk =
ε k ) ζ , where ε k are independent uniform random variables in [0 . 5 , 1 . 5] .
More Experimental Details:
In our experiments, the domain X is selected as [ − 2 , 2] interval.
Since X is one dimensional, we use x to denote x , and use ξ k to denote ξ k in the subsequent context.
Without loss of generality we choose x ∗ = 0 .
The parameters α , β , and γ are obtained by α = min x ∈X | f ′ ( x ) | 2 f ( x ) , β = max x ∈X | f ′′ ( x ) | , γ = max x ∈X | f ′ ( x ) | .
For GD Loop, we set x 0 = 2 .
Furthermore we restrict the iterations to stay within X .
For adversary, the parameters of the SLSQP method are given as ftol = 1 e − 12 , eps = 1 e − 14 , maxiter = 10000 .
Our experiments are run in our internal cluster with 10 CPU servers managed by IBM Spectrum LSF.
Each CPU server has 32 CPU cores and 256 G RAM.
The operating system of these servers are Ubuntu 20.04 LTS.
Other implementation details can be found in our source code, which will be released after the review period.
B.2. Assumptions on Objective Functions
A line of research has investigated other conditions which permit establishing analytical results while conforming to practical considerations.
Examples include: essential strong convexity [12], weak strong convexity [14], restricted secant inequality [18], and quadratic growth [4].
An interesting condition which is weaker and more general than all mentioned above is the Polyak-Łojasiewicz (PL) condition, which does not even require convexity.
In this experiment, we consider various objective functions: a quadratic objective function referred to as f Q and a strongly convex one referred to as f SC .
In addition, we consider two other functions satisfying the PL condition referred to as f PL , 1 and f PL , 2 ; non of them convex; the former is locally convex around x ∗ and the latter is not.
The expression of these objective functions and their plots are given in Table 2 and Figure 1, respectively.
[Table]
Quadratic
[Equation] [Table] f SC f ( x ) = x 2 − cos( x ) +
[Table] f PL ,f ( x ) = x 2 + 0 .
5 ∗ sin 2 (2 x )
[Table] f PL ,f ( x ) =
x 2 + 2 .
1 ∗ sin( | x | − 1 ) ∗ | x |
Figure 2 shows the worst-case optimization error of GD for the four objective functions described above.
The corruption budget is set to b ( n ) ∼ √ n .
The optimization errors reported in the figures are average over three Monte Carlo runs with different random initialization (ˆ ξ k ) nk
=1 of the corruption sequence for the adversary (the SLSQP method mentioned above).
The theoretical bounds proven in Corollary 3.2 are also plotted for comparison.
In all cases, the theoretical bounds hold true.
B.4. The Effect of Corruption Budget
Figure 3 (a) and (b) show the numerical results on the worst-case optimization error for various corruption budgets by varying their scale and exponent parameters.
The corresponding theoretical bounds from Corollary 3.2 are also plotted for comparison.
All the experiments are consistent with the theoretical findings.
As expected, optimization error is larger for a greater corruption budget.
Eventually for a linear budget b ( n ) ∼ n the optimization error does not vanish.
We used f PL , 2 as the objective function for this experiment.
B.5. Adversarial Corruptions
In all previous experiments, we used SLSQP method from scipy.optimize package [17] (which is under the liberal BSD license) to find a numerical approximation of the worst case corruption sequence.
Let us refer to this adversary as ˆ A ∗ .
In this experiment, we compare the optimization error using ˆ A ∗ versus other adversaries which exploit the available budget in a greedy way.
In particular, let b used ( n )
=
(cid:80) nk
=1 c ( k ) denote the amount of budget used up to time n included.
Then, the available budget at time k is b ( k ) − b used ( k − 1) .
The adversary A 1 selects ξ A 1 k = ( b ( k ) − b used ( k − 1))
sgn ( − ˜ g ( x k − 1 )) .
That is a greedy way to push the GD iterates as far as possible from the minimum, while respecting the budget.
We also consider a different adversary, referred to as A 2 , which selects ξ A 2 k =
ξ A 1 k ε k , where ε k are independent uniform random variables in [0 . 5 , 1 . 5] .
Note that A 2 may violate the budget constraint.
It however respects the budget in expectation.
Furthermore, we compare these adversaries with the case without adversary, denoted as A 0 .
Figure 3(c) shows a comparison between the performances of these adversaries; ˆ A ∗ leads to a significantly higher worst-case optimization error compared to A 1 , A 2 and A 0 .
Besides, A 0 leads to the lowest worse-case optimization error since it has no adversary.
This experiment is run using the objective function f PL , 2 .
B.6. The Effect of Learning Rate Sequence
In this experiment, we compare the learning rate sequence given in Corollary 3.2, denoted as ( η ∗ k )
nk =1 , with other learning rate sequences: (cid:16)
[Equation]
The first one is based on Corollary 3.2, the others are the similar learning rate sequences but with varying power in their denominators.
Figure 3(d) shows a comparison between these learning rate sequences.
The worst-case optimization errors of η ∗ k and η (1) k decrease steadily, while the errors of η (2) k and η ( . 5) k stop decreasing after a certain number of GD iterations.
This experiment demonstrates that our theorem provides a guideline on learning rate sequences which make GD robust against adversarial corruption.
B.7. Discussion
In these experiments, we compare the theoretical bounds with the results of numerical experiments on a few example objective functions satisfying the PL condition; including quadratic, strongly convex and non-convex ones.
We show how the theoretical and empirical optimization errors are affected by the budget and compare the adversarial corruption with other types of gradient perturbations and other types of learning rate sequences.
However; these experiments indicate that the order of bounds may need to be be improved, at least based on the cases studied here.
Furthermore, we use the SLSQP method implemented in the scipy.optimize.minimize package to simulate the adversarial corruption.
We only have an approximation of the worst-case corruptions in our experiments, and that is likely to be the reason for the gap between the theoretical upper bound and the results on optimization error in the experiments.
The analytical result, however, is a theoretical upper bound and may not realize with all problem instances.