V ARIANCE R EDUCTION IS AN A NTIDOTE TO B YZ W ORKERS : B ETTER R ATES , W EAKER A SSUMPTIONS AND C OMMUNICATION C OMPRESSION AS A C HERRY ON THE T OP
Anonymous authors
Paper under double-blind review
A BSTRACT
Byz-robustness has been gaining a lot of attention due to the growth of the interest in collaborative and federated learning.
However, many fruitful directions, such as the usage of variance reduction for achieving robustness and communication compression for reducing communication costs, remain weakly explored in the ﬁeld.
This work addresses this gap and proposes Byz-VR-MARINA –a new Byztolerant method with variance reduction and compression.
A key message of our paper is that variance reduction is key to ﬁghting Byz workers more effectively.
At the same time, communication compression is a bonus that makes the process more communication efﬁcient.
We derive theoretical convergence guarantees for Byz-VR-MARINA outperforming previous state-of-the-art for general non-convex and Polyak-Łojasiewicz loss functions.
Unlike the concurrent Byz-robust methods with variance reduction and/or compression, our complexity results are tight and do not rely on restrictive assumptions such as boundedness of the gradients or limited compression.
Moreover, we provide the ﬁrst analysis of a Byz-tolerant method supporting non-uniform sampling of stochastic gradients.
Numerical experiments corroborate our theoretical ﬁndings.
I NTRODUCTION
Distributed optimization algorithms play a vital role in the training of the modern machine learning models.
In particular, some tasks require training of deep neural networks having billions of parameters on large datasets (Brown et al., 2020; Kolesnikov et al., 2020).
Such problems may take years of computations to be solved if executed on a single yet powerful machine (Li, 2020).
To circumvent this issue, it is natural to use distributed optimization algorithms allowing to tremendously reduce the training time (Goyal et al., 2017;
You et al., 2020).
In the context of speeding up the training, distributed methods are usually applied in data centers (Mikami et al., 2018).
More recently, similar ideas have been applied to train models using open collaborations (Kijsipongse et al., 2018;
Diskin et al., 2021), where each participant (e.g., a small company/university or an individual) has very limited computing power but can donate it to jointly solve computationally-hard problems.
Moreover, in Federated Learning (FL) applications (McMahan et al., 2017; Koneˇcn´y et al., 2016;
Kairouz et al., 2021), distributed algorithms are natural and the only possible choice since in such problems, the data is privately distributed across multiple devices.
In the optimization problems arising in collaborative and federated learning, there is a high risk that some participants deviate from the prescribed protocol either on purpose or not.
{+In this paper, we call such participants as Byz workers 
+}
For example, such peers can maliciously send incorrect gradients to slow down or even destroy the training.
Indeed, these attacks can break the convergence of na¨ıve methods such as Parallel-SGD (Zinkevich et al., 2010).
Therefore, it is crucial to use secure (a.k.a.
Byz-robust/Byz-tolerant) distributed methods for solving such problems.
However, designing distributed methods with provable Byz-robustness is not an easy task.
The non-triviality of this problem comes from the fact that the stochastic gradients of good/honest/regular workers are naturally different due to their stochasticity and possible data heterogeneity.
At the same time, malicious workers can send the vectors looking like the stochastic gradients of good peers or create small but time-coupled shifts.
Therefore, as it is shown in (Baruch et al., 2019; Xie et al., 2020; Karimireddy et al., 2021), Byz workers can circumvent popular defences based on applying robust aggregation rules (Blanchard et al., 2017; Yin et al., 2018;
Damaskinos et al., 2019; Guerraoui et al., 2018; Pillutla et al., 2022) with Parallel-SGD .
Moreover, in a broad class of problems with heterogeneous data, it is provably impossible to achieve any predeﬁned accuracy of the solution (Karimireddy et al., 2022; El-Mhamdi et al., 2021).
Nevertheless, as it becomes evident from the further discussion, several works have provable Byz tolerance and rigorous theoretical analysis.
In particular, Wu et al.
(2020) propose a natural yet elegant solution to the problem of Byz-robustness based on the usage of variance-reduced methods (Gower et al., 2020) and design the ﬁrst variance-reduced Byz-robust method called Byrd-SAGA , which combines the celebrated SAGA method (Defazio et al., 2014) with geometric median aggregation rule.
As a result, reducing the stochastic noise of estimators used by good workers makes it easier to ﬁlter out Byz workers (especially in the case of homogeneous data).
However, Wu et al.
(2020) derive their results only for the strongly convex objectives, and the obtained convergence guarantees are signiﬁcantly worse than the best-known convergence rates for SAGA , i.e., their results are not tight, even when there are no Byz workers and all peers have homogeneous data.
It is crucial to bypass these limitations since the majority of the modern, practically interesting problems are non-convex.
Furthermore, it is hard to develop the ﬁeld without tight convergence guarantees.
All in all, the above leads to the following question:
Q1:
Is it possible to design variance-reduced methods with provable Byz-robustness and tight theoretical guarantees for general non-convex optimization problems?
In addition to Byz-robustness, one has to take into account that na ¨ ıve distributed algorithms suffer from the so-called communication bottleneck —a situation when communication is much more expensive than local computations on the devices.
This issue is especially evident in the training of models with a vast number of parameters (e.g., millions or trillions) or when the number of workers is large (which is often the case in FL).
One of the most popular approaches to reducing the communication bottleneck is to use communication compression (Seide et al., 2014; Koneˇcn´y et al., 2016; Suresh et al., 2017), i.e., instead of transmitting dense vectors (stochastic gradients/Hessians/higher-order tensors) workers apply some compression/sparsiﬁcation operator to these vectors and send the compressed results to the server.
Distributed learning with compression is a relatively well-developed ﬁeld, e.g., see (Vogels et al., 2019; Gorbunov et al., 2020b; Richt ´ arik et al., 2021; Philippenko & Dieuleveut, 2021) and references therein for the recent advances.
Perhaps surprisingly, there are not many methods with compressed communication in the context of Byz-robust learning.
In particular, we are only aware of the following works (Bernstein et al., 2018; Ghosh et al., 2020; 2021; Zhu & Ling, 2021),.
Bernstein et al.
(2018) propose signSGD to reduce communication cost and and study the majority vote to cope with the Byz workers under some additional assumptions about adversaries.
However, it is known that signSGD is not guaranteed to converge (Karimireddy et al., 2019).
Next, Ghosh et al.
(2020; 2021) apply aggregation based on the selection of the norms of the update vectors.
In this case, Byz workers can successfully hide in the noise applying SOTA attacks (Baruch et al., 2019).
Zhu & Ling (2021) study Byz-robust versions of compressed SGD ( BR-CSGD ) and SAGA ( BR-CSAGA ) and also propose a combination of
DIANA (Mishchenko et al., 2019; Horv´ath et al., 2019b) with BR-CSAGA called BROADCAST .
However, the derived convergence results for these methods have several limitations.
First of all, the analysis is given only for strongly convex problems.
In addition, it relies on restrictive assumptions.
Namely, Zhu & Ling (2021) assume uniform boundedness of the second moment of the stochastic gradient in the analysis of BR-CSGD and BR-CSAGA .
This assumption rarely holds in practice, and it also implies the boundedness of the gradients, which contradicts the strong convexity assumption.
Next, although the bounded second-moment assumption is not used in the analysis of BROADCAST ,
Zhu & Ling (2021) derive the rates of BROADCAST under the assumption that the compression operator is very accurate, which implies that in theory workers apply almost no compression to the communicated messages (see remark (5) under Table 2).
Finally, even if there are no Byz workers and no compression, similar to the guarantees for Byrd-SAGA , the rates obtained for BR-CSGD ,
[Table]
[Table] [Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
BR-CSAGA , and BROADCAST are outperformed with a large margin by the known rates for SGD and SAGA .
All of these limitations lead to the following question:
Q2:
Is it possible to design distributed methods with compression, provable Byz-robustness and tight theoretical guarantees without making strong assumptions?
In this paper, we give conﬁrmatory answers to Q1 and Q2 by proposing and rigorously analyzing a new Byz-tolerant variance-reduced method with compression called Byz-VR-MARINA .
Detailed related work overview is deferred to Appendix A.
Our Contributions.
Before we proceed, we need to specify the targetted problem.
We consider a centralized distributed learning in the possible presence of malicious or so-called Byz peers.
We assume that there are n clients consisting of the two groups: [ n ] = G (cid:116) B , where G denotes the set of good clients and B is the set of bad/malicious/Byz workers.
The goal is to solve the following optimization problem
[Equation]
where G = |G| and functions f i,j ( x ) are assumed to be smooth, but not necessarily convex.
Here each good client has its dataset of the size m , f i,j ( x ) is the loss of the model, parameterized by vector x ∈ R d , on the j -th sample from the dataset on the i -th client.
Following the classical convention (Lyu et al., 2020), we make no assumptions on the malicious workers B , i.e., Byz workers are allowed to be omniscient .
Our main contributions are summarized below.
(cid:5) New method: Byz-VR-MARINA .
We propose a new Byz-robust variance-reduced method with compression called Byz-VR-MARINA (Alg. 1).
In particular, we make VR-MARINA (Gorbunov et al., 2021b), which is a variance-reduced method with compression, applicable to the context of Byztolerant distributed learning via using the recent tool of robust agnostic aggregation of Karimireddy et al.
As Tbl.
1 shows, Byz-VR-MARINA and our analysis of the method leads to several important improvements upon the previously best-known methods.
[Table]
(cid:5) New SOTA results.
Under quite general assumptions listed in Section 2, we prove theoretical convergence results for Byz-VR-MARINA in the cases of smooth non-convex (Thm. 2.1) and
PolyakŁojasiewicz (Thm. 2.2) functions.
As Tbl.
2 shows, our complexity bounds in the non-convex case are always better than previously known ones when the target accuracy ε is small enough.
In the PŁ case, our results improve upon previously known guarantees when the problem has bad conditioning or when ε is small enough.
Moreover, we provide the ﬁrst theoretical convergence guarantees for Byz-tolerant methods with compression in the non-convex case for arbitrary adversaries.
(cid:5) Byz-tolerant variance-reduced method with tight rates.
Our results are tight, i.e., when there are no Byz workers, our rates recover the rates of VR-MARINA , and when additionally no compression is applied, we recover the optimal rates et al.
In contrast, this is not the case for previously known variance-reduced Byz-robust methods such as Byrd-SAGA , BR-CSAGA , and BROADCAST that in the homogeneous data scenario have worse rates than single-machine SAGA .
(cid:5) Support of the compression without strong assumptions.
As we point out in Tbl. 2, the analysis of BR-CSGD and BR-CSAGA relies on the bounded second-moment assumption, which contradicts strong convexity, and the rates for BROADCAST are derived under the assumption that the compression operator almost coincides with the identity operator, meaning that in practice workers essentially do not use any compression.
In contrast, our analysis does not have such substantial limitations.
(cid:5) Enabling non-uniform sampling.
In contrast to the existing works on Byz-robustness, our analysis supports non-uniform sampling of stochastic gradients.
Considering the dependencies on smoothness constants, one can quickly notice our rates’ even more signiﬁcant superiority compared to the previous SOTA results.
Byz-VR-MARINA : B YZ -T OLERANT V ARIANCE R EDUCTION WITH C OMMUNICATION C OMPRESSION
We start by introducing necessary deﬁnitions and assumptions.
Robust aggregation.
One of the main building blocks of our method relies on the notion of ( δ, c ) -
Robust Aggregator introduced in (Karimireddy et al., 2021; 2022).
Deﬁnition 2.1 ( ( δ, c ) -Robust Aggregator) .
Assume that { x 1 , x 2 , . . .
, x n } is such that there exists a subset G ⊆
[ n ] of size |G| =
G ≥ (1 − δ ) n for δ < 0 .
and there exists σ ≥ 0 such that
G ( G − 1) (cid:80) i,l ∈G E [ (cid:107) x
i − x l (cid:107) 2 ] ≤ σ 2 where the expectation is taken w.r.t.
the randomness of { x i } i ∈G .
We say that the quantity (cid:98) x is ( δ, c )
-Robust Aggregator ( ( δ, c ) - RAgg ) and write (cid:98) x
= RAgg ( x 1 , . . .
, x n ) for some c > 0 , if the following inequality holds:
[Equation]
where x = 1 |G| (cid:80) i ∈G x i .
If additionally (cid:98) x is computed without the knowledge of σ 2 , we say that (cid:98) x is ( δ, c ) -Agnostic
Robust Aggregator ( ( δ, c ) - ARAgg ) and write (cid:98) x
= ARAgg ( x 1 , . . .
, x n ) .
In fact, Karimireddy et al.
(2021; 2022) propose slightly different deﬁnition, where they assume that E (cid:107) x i − x l (cid:107) 2 ≤ σ 2 for all ﬁxed good workers i, l ∈ G , which is marginally stronger than what we assume.
Karimireddy et al.
(2021) prove tightness of their deﬁnition, i.e., up to the constant c one cannot improve bound (2), and prove that popular “middle-seekers” such as
Krum (Blanchard et al., 2017), Robust Federated Averaging ( RFA )
(Pillutla et al., 2022), and Coordinate-wise Median ( CM ) (Chen et al., 2017) do not satisfy their deﬁnition.
However, there is a trick called bucketing (Karimireddy et al., 2022) that provably robustiﬁes Krum / RFA / CM .
Nevertheless, the difference between our deﬁnition and the original one from (Karimireddy et al., 2021; 2022) is very subtle and it turns out that Krum / RFA / CM with bucketing ﬁt Deﬁnition 2.1 as well (see Appendix D).
Compression.
We consider unbiased compression operators, i.e., quantizations .
Deﬁnition 2.2 (Unbiased compression (Horv´ath et al., 2019b)) .
Stochastic mapping Q : R d → R d is called unbiased compressor/compression operator if there exists ω ≥ 0 such that for any x ∈ R d
[Equation]
For the given unbiased compressor Q ( x ) , one can deﬁne the expected density as ζ Q = sup x ∈ R d E [ (cid:107)Q ( x ) (cid:107) 0 ] , where (cid:107) y (cid:107) 0 is the number of non-zero components of y ∈ R d .
The above deﬁnition covers many popular compression operators such as RandK sparsiﬁcation (Stich et al., 2018), random dithering (Goodall, 1951; Roberts, 1962), and natural compression (Horv ´ ath et al., 2019a)
(see also the summary of various compression operators in (Beznosikov et al., 2020)).
There exist also other classes of compression operators such as δ -contractive compressors (Stich et al., 2018) and absolute compressors (Tang et al., 2019; Sahu et al., 2021).
However, these types of compressors are out of the scope of this work.
Assumptions.
The ﬁrst assumption is quite standard in the literature on non-convex optimization.
Assumption 2.1.
We assume that function f : R d → R is L -smooth, i.e., for all x, y ∈ R
d we have (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) .
Moreover, we assume that f is uniformly lower bounded by f ∗ ∈ R , i.e., f ∗ = inf x ∈ R d f ( x ) .
Next, we need to restrict the data heterogeneity of regular workers.
Indeed, in arbitrarily heterogeneous scenario, it is impossible to distinguish regular workers and Byz workers.
Therefore, we use a quite standard assumption about the heterogeneity of the local loss functions.
Assumption 2.2 ( ζ 2 -heterogeneity) .
We assume that good clients have ζ 2 -heterogeneous local loss functions for some ζ ≥ 0 , i.e.,
[Equation]
We emphasize here that the homogeneous data case ( ζ = 0 ) is realistic in collaborative learning.
This typically means that the workers have an access to the entire data.
For example, this can be implemented using so-called dataset streaming when the data is received just in time in chunks (Diskin et al., 2021; Kijsipongse et al., 2018)
(this can also be implemented without using the server via special protocols similar to BitTorrent).
The following assumption is a reﬁnement of a standard assumption that f i is L i -smooth for all i ∈ G .
Assumption 2.3 (Global Hessian variance assumption (Szlendak et al., 2021)) .
We assume that there exists L ± ≥
such that for all x, y ∈ R d
[Equation]
If f i is L i -smooth for all i ∈ G , then the above assumption is always valid for some L ± ≥
such that L 2 avg − L 2 ≤ L 2 ± ≤ L 2 avg , where L 2 avg = 1 G (cid:80) i
∈G L 2 i (Szlendak et al., 2021).
Moreover, Szlendak et al.
(2021) show that there exist problems with heterogeneous functions on workers such that (5) holds with L ± = 0 , while L avg > 0 .
We propose a generalization of the above assumption for samplings of stochastic gradients.
Assumption 2.4 (Local Hessian variance assumption) .
We assume that there exists L ± ≥
such that for all x, y ∈ R d
[Equation]
where ∆ i ( x, y )
=
∇ f i ( x ) − ∇ f i ( y ) and (cid:98) ∆ i ( x, y ) is an unbiased mini-batched estimator of ∆ i ( x, y ) with batch size b .
We notice that the above assumption covers a wide range of samplings of mini-batched stochastic gradient differences, e.g., standard uniform sampling or importance sampling.
We provide the examples in Appendix E.1.
We notice that all previous works on Byz-robustness focus on the standard uniform sampling only.
However, uniform sampling can give m times worse constant L 2 ± than importance sampling.
This difference signiﬁcantly affect the complexity bounds.
New Method: Byz-VR-MARINA .
Now we are ready to present our new method—Byz-tolerant
Variance-Reduced MARINA
( Byz-VR-MARINA ).
Our algorithm is based on the recently proposed variance-reduced method with compression ( VR-MARINA ) from (Gorbunov et al., 2021b).
At each iteration of Byz-VR-MARINA , good workers update their parameters x k +
=
x
k − γg k using estimator g k received from the parameter-server (line 7).
Next (line 8), with (typically small) probability p each good worker i ∈ G computes its full gradient, and with (typically large) probability 1 − p this worker computes compressed mini-batched stochastic gradient difference Q ( (cid:98) ∆ i ( x k +1 , x k )) , where (cid:98) ∆ i ( x k +1 , x k ) satisﬁes Assumption 2.4.
After that, the server gathers the results of computations from the workers and applies ( δ, c ) - ARAgg to compute the next estimator g k +1 (line 10).
Let us elaborate on several important parts of the proposed algorithm.
First, we point out that with large probability 1 − p good workers need to send just compressed vectors Q ( (cid:98) ∆ i ( x k +1 , x k )) , i ∈ G .
Indeed, since the server knows when workers compute full gradients and when they compute compressed stochastic gradients, it needs just to add g k to all received vectors to perform robust aggregation from line 10.
Moreover, since the server knows the type of compression operator that good workers apply, it can typically easily ﬁlter out those Byz workers who try to slow down the training via sending dense vectors instead of compressed ones (e.g., if the compression operator is
RandK sparsiﬁcation, then Byz workers cannot send more than K components; otherwise they will be easily detected and can be banned).
Next, the right choice of probability p allows equalizing the communication cost of all steps when good workers send dense gradients and compressed gradient differences.
The same is true for oracle complexity: if p ≤ b / m , then the computational cost of full-batch computations is not bigger than that of stochastic gradients.
k + = k − = (cid:40) ∇ c k = g k + Q (cid:16)(cid:98) ∆
i ( x k + x k ) (cid:17) , otherwise, , minibatched estimator (cid:98) ∆ i ( k + ) of ∇ f i ( x k +1 ) − ∇ f i ( x k ) ; Q ( · ) for i ∈ G are computed independently 9: end = ARAgg ( g k + , . . . , g k +1 n ) 11: end for 12: Return: ˆ x K chosen uniformly at random from { x k } K − 1 k =
Challenges in designing variance-reduced algorithm with tight rates and provable Byzrobustness.
In the introduction, we explain why variance reduction is a natural way to handle
Byz attacks (see the discussion before Q1 ).
At ﬁrst glance, it seems that one can take any variancereduced method and combine it with some robust aggregation rule to get the result.
However, this is not as straightforward as it may appear.
As one can see from Table 2, combination of SAGA with geometric median estimator ( Byrd-SAGA ) gives the rate ˜ O
( m 2 b 2 (1 − 2 δ ) µ 2 ) (smoothness constant and logarithmic factors are omitted) in the smooth strongly convex case — this rate is in fact O ( m 2 b 2 µ 2 ) times worse than the rate of SAGA even when δ = 0 .
Therefore, it becomes clear that the full potential of variance reduction in Byz-robust learning is not revealed via Byrd-SAGA .
The key reason for that is the sensitivity of SAGA (and SAGA -based methods) to the unbiasedness of the stochastic estimator in the analysis.
Since Byrd-SAGA uses the geometric median for the aggregation, which is necessarily biased, it is natural that it has a much worse convergence rate than
SAGA even in the δ = 0 case.
Moreover, one can’t solve such an issue by simply changing one robust estimator for another since all known robust estimators are generally biased.
To circumvent this issue, we consider Geom-SARAH / PAGE -based estimator (Horv´ath & Richt´arik, 2019; Li et al., 2021) and study how it interacts with the robust aggregation.
In particular, we observe that the averaged pair-wise variance for the stochastic gradients of good workers could be upper bounded by a constant multiplied by E (cid:107) x k +1 − x k (cid:107) 2 plus some additional terms appearing due to heterogeneity (see Lemma E.2).
Then, we notice that the robust aggregation only leads to the additional term proportional to E (cid:107) x k +1 − x k (cid:107) 2 (plus additional terms due to heterogeneity).
We show that this term can be directly controlled using another term proportional to − E (cid:107) x k +1 − x k (cid:107) 2 , which appears in the original analysis of PAGE / VR-MARINA .
These facts imply that although the difference between Byz-VR-MARINA and VR-MARINA is only in the choice of the aggregation rule, it is not straightforward beforehand that such a combination should be considered and that it will lead to better rates.
Moreover, as we show next, we obtain vast improvements upon the previously best-known theoretical results for Byz-tolerant learning.
General Non-Convex Functions.
Our main convergence result for general non-convex functions follows.
All proofs are deferred to Appendix E.
Theorem 2.1.
Let Assumptions 2.1, 2.2, 2.3, 2.4 hold.
Assume that 0 < γ ≤ 1 L + √ A , where
A =
[Equation]
Then for all K ≥the point (cid:98) x K choosen uniformly at random from the iterates x 0 , x 1 , . . .
, x K produced by Byz-VRMARINA satisﬁes
[Equation]
where Φ
= f
( x 0 ) − f ∗ + γp (cid:107) g 0 − ∇ f ( x 0 ) (cid:107) 2 and E [ · ] denotes the full expectation.
We highlight here several important properties of the derived result.
First of all, this is the ﬁrst theoretical result for the convergence of Byz-tolerant methods with compression in the non-convex case with arbitrary adversaries.
Next, when ζ > 0 the theorem above does not guarantee that E [ (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 ] can be made arbitrarily small.
However, this is not a drawback of our analysis but rather an inevitable limitation of all algorithms in heterogeneous case.
This is due to Karimireddy et al.
(2022) who proved a lower bound showing that in the presence of Byz workers, all algorithms satisfy E [ [Equation]
] = Ω( δζ 2 ) , i.e., the constant term from (7) is tight up to the factor of 1 / p .
However, when ζ = 0 , Byz-VR-MARINA can achieve any predeﬁned accuracy of the solution, if δ is such that ARAgg is ( δ, c ) -robust
(see Theorem D.1).
Finally, as Table 2 shows 3 , Byz-VR-MARINA achieves E [ [Equation] ] ≤ ε 2 faster than all previously known Byz-tolerant methods for small enough ε .
Moreover, unlike virtually all other results in the non-convex case, Theorem 2.1 does not rely on the uniformly bounded variance assumption, which is known to be very restrictive (Nguyen et al., 2018).
For further discussion we refer to Appendix E.5.
Functions Satisfying Polyak-Łojasiewicz (PŁ) Condition.
We extend our theory to the functions satisfying Polyak-Łojasiewicz condition (Polyak, 1963; Łojasiewicz, 1963).
This assumption generalizes regular strong convexity and holds for several non-convex problems (Karimi et al., 2016).
Moreover, a very similar assumption appears in over-parameterized deep learning (Liu et al., 2022).
Assumption 2.5 (PŁ condition) .
We assume that function f satisﬁes Polyak-Łojasiewicz (PŁ) condition with parameter µ , i.e., for all x ∈ R
d
there exists x ∗ ∈ argmin x ∈ R d f ( x ) such that
[Equation]
Under this and previously introduced assumptions, we derive the following result.
Theorem 2.2.
Let Assumptions 2.1, 2.2, 2.3, 2.4, 2.5 hold.
Assume that 0 < γ ≤ min (cid:110)
[Equation]
, where A = 6(1 − p ) p (cid:16) 4 cδp + 12 G (cid:17) [Equation] +
[Equation]
(cid:17)
+ [Equation] [Equation] p
[Equation]
(cid:16) 4 cδ (1+ ω )
[Equation]
+ ω 2 G (cid:17) L 2 ± .
Then for all
K ≥
the iterates produced by Byz-VR-MARINA satisfy
[Equation]
where Φ
= f
( x 0 ) − f ∗ + 2 γp (cid:107) g 0 − ∇ f ( x 0 ) (cid:107) 2 .
Similarly to the general non-convex case, in the PŁ-setting Byz-VR-MARINA is able to achieve E [ f ( x K ) − f ( x ∗ )]
= O ( cδζ 2 / µ ) accuracy, which matches (up to the factor of 1 / p ) the lower bound from Karimireddy et al.
(2022) derived for µ -strongly convex objectives.
Next, when ζ = 0 , Byz-VRMARINA converges linearly asymptotically to the exact solution.
Moreover, as Table 2 shows, our convergence result in the PŁ-setting outperforms the known rates in more restrictive strongly-convex setting.
In particular, when ε is small enough, Byz-VR-MARINA has better complexity than BTARDSGD .
When the conditioning of the problem is bad (i.e., L / µ (cid:29) 1 ) our rate dominates results of
[Figure]
BR-CSGD , BR-CSAGA , and BROADCAST .
Furthermore, both BR-CSGD and BR-CSAGA rely on the uniformly bounded second moment assumption (contradicting the strong convexity), and the rate of the BROADCAST algorithm is based on the assumption that ω
= O ( µ 2 / L 2 ) implying that
Q ( x ) ≈ x (no compression) even for well-conditioned problems.
N UMERICAL E XPERIMENTS
In this section, we demonstrate the practical performance of the proposed method.
The main goal of our experimental evaluation is to showcase the beneﬁts of employing SOTA variance reduction to remedy the presence of Byz workers.
For the task, we consider the standard logistic regression model with (cid:96) 2 -regularization f i,j ( x ) =
− y i,j log( h ( x, a i,j )) − (1 − y i,j ) log(1 − h ( x, a i,j ))
+ λ (cid:107) x (cid:107) 2 , where y i,j ∈ { 0 , 1 } is the label, a i,j ∈ R d represents the features vector, λ is the regularization parameter and h ( x, a )
= 1 / (1+ e − a (cid:62) x ) .
One can show that this objective is smooth, and for λ > 0 , it is also strongly convex, therefore, it satisﬁes PŁcondition.
We consider a9a LIBSVM dataset (Chang & Lin, 2011) and set λ = 0 .
In the experiments, we focus on an important feature of Byz-VR-MARINA : it guarantees linear convergence for homogeneous datasets across clients even in the presence of Byz workers, as shown in Theorem 2.2.
To demonstrate this experimentally, we consider the setup with four good workers and one Byz worker, each worker can access the entire dataset , and the server uses coordinate-wise median with bucketing as the aggregator (see the details in Appendix D).
We consider ﬁve different attacks:
• No Attack (NA): clean training; • Label Flipping (LF): labels are ﬂipped, i.e., y i,j → 1 − y i,j ; • Bit Flipping (BF): a Byz worker sends an update with ﬂipped sign; • A Little is enough (ALIE) (Baruch et al., 2019): the Byz workers estimate the mean µ G and standard deviation σ G of the good updates, and send µ G − zσ G to the server where z is a small constant controlling the strength of the attack; • Inner Product Manipulation (IPM) (Xie et al., 2020): the attackers send − (cid:15)G (cid:80) i ∈G ∇ f i ( x ) where (cid:15) controls the strength of the attack.
For bucketing, we use s = 2 , i.e., partitioning the updates into the groups of two, as recommended by Karimireddy et al.
We compare our Byz-VR-MARINA with the baselines without compression ( SGD , BR-SGDm (Karimireddy et al., 2021)) and the baselines with random sparsiﬁcation (compressed SGD and DIANA ( BR-DIANA ).
We do not compare against Byrd-SAGA (and BRCSAGA , BROADCAST from Zhu & Ling (2021)), which consumes large memory that scales linearly with the number of local data points and is not well suited for memory-efﬁcient batched gradient computation (e.g., used in PyTorch).
Our implementation is based on PyTorch (Paszke et al., 2019).
Figure 1 showcases that, indeed, we observe linear convergence of our method while no baseline achieves this fast rate.
In the ﬁrst row, we display methods with no compression, and in the second row, each algorithm uses random sparsiﬁcation.
We defer further details and additional experiments with heterogeneous data to Appendix B.
C ONTENTS 1 Introduction
2 Byz-VR-MARINA : Byz-Tolerant Variance Reduction with Communication Compression 3 Numerical Experiments
A Detailed Related Work
B Extra Experiments and Experimental details
B.1 General setup . .
B.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Extra Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3.
Heterogeneous Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3.
Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3.
Extra dataset: w8a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Useful Facts
D Further Details on Robust Aggregation
E Missing Proofs and Details From Section
E.1 Examples of Samplings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Key Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 General Non-Convex Functions . .
E.4 Functions Satisfying Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . .
E.
Further Details on the Obtained Results and the Comparison from Table 2 . . . . .
A D ETAILED R ELATED W ORK
Byz-robustness.
Classical approaches to Byz-tolerant optimization are based on applying special aggregation rules to Parallel-SGD (Blanchard et al., 2017;
Chen et al., 2017; Yin et al., 2018; Damaskinos et al., 2019; Guerraoui et al., 2018; Pillutla et al., 2022).
It turns out that such defences are vulnerable to the special type of attacks (Baruch et al., 2019; Xie et al., 2020).
Moreover,
Karimireddy et al.
(2021) propose a reasonable formalism for describing robust aggregation rules (see Def. 2.1) and show that almost all previously known defences are not robust according to this formalism.
In addition, they propose and analyze new Byz-tolerant methods based on the usage of Polyak’s momentum (Polyak, 1964) ( BR-SDGm ) and momentum variance reduction (Cutkosky & Orabona, 2019) ( BR-MVR ).
This approach is extended to the case of heterogeneous data and aggregators agnostic to the noise level by Karimireddy et al.
and
He et al.
(2022) propose an extension to the decentralized optimization over ﬁxed networks.
Gorbunov et al.
(2021a) propose an alternative approach based on the usage of AllReduce (Patarasuk & Yuan, 2009) with additional veriﬁcations of correctness and show that their algorithm has complexity not worse than ParallelSGD when the target accuracy is small enough.
Wu et al.
(2020) are the ﬁrst who applied variance reduction mechanism to tolerate Byz attacks (see the discussion above Q1 ).
We also refer reader to (Chen et al., 2018; Rajput et al., 2019; Rodr ´ ıguez-Barroso et al., 2020; Xu & Lyu, 2020;
Alistarh et al., 2018; Allen-Zhu et al., 2021; Regatti et al., 2020; Yang & Bajwa, 2019a;b; Gupta et al., 2021; Gupta & Vaidya, 2021; Peng et al., 2021) for other advances in Byz-robustness (see the detailed summaries in (Lyu et al., 2020; Gorbunov et al., 2021a)).
We further progress the ﬁeld by obtaining new theoretical SOTA convergence results in our work.
Compressed communications.
Methods with compression are relatively well studied in the literature.
The ﬁrst theoretical results were derived in (Alistarh et al., 2017; Wen et al., 2017; Stich et al., 2018; Mishchenko et al., 2019).
During the last several years the ﬁeld has been signiﬁcantly developed.
In particular, compressed methods are analyzed in the conjuction with variance reduction (Horv´ath et al., 2019b; Gorbunov et al., 2020b; Danilova & Gorbunov, 2022), acceleration (Li et al., 2020; Li & Richt´arik, 2021;
Qian et al., 2021b), decentralized communications (Koloskova et al., 2019; Kovalev et al., 2021), local steps (Basu et al., 2019; Haddadpour et al., 2021), adaptive compression (Faghri et al., 2020), second-order methods (Islamov et al., 2021;
Safaryan et al., 2021), and min-max optimization (Beznosikov et al., 2021; 2022).
However, to our knowledge, only one work studies communication compression in the context of Byz-robustness (Zhu & Ling, 2021)
(see the discussion above Q2 ).
Our work makes a further step towards closing this signiﬁcant gap in the literature.
Variance reduction is a powerful tool allowing to speed up the convergence of stochastic methods (especially when one needs to achieve a good approximation of the solution).
The ﬁrst variancereduced methods were proposed by Schmidt et al.
; Johnson & Zhang (2013); Defazio et al.
Optimal variance-reduced methods for (strongly) convex problems are proposed in (Lan & Zhou, 2018; Allen-Zhu, 2017; Lan et al., 2019) and for non-convex optimization in (Nguyen et al., 2017; Fang et al., 2018; Li et al., 2021).
Despite the noticeable attention to these kinds of methods (Gower et al., 2020), only a few papers study Byz-robustness in conjunction with variance reduction (Wu et al., 2020; Zhu & Ling, 2021; Karimireddy et al., 2021).
Moreover, as we mentioned before, the results from Wu et al.
; Zhu & Ling (2021) are not better than the known ones for non-parallel variance-reduced methods, and Karimireddy et al.
(2021) rely on the uniformly bounded variance assumption, which is hard to achieve in practice.
In our work, we circumvent these limitations.
Non-uniform sampling.
Originally proposed for randomized coordinate methods (Nesterov, 2012;
Richt ´ arik & Tak ´ a ˇ c, 2016; Qu & Richt ´ arik, 2016), non-uniform sampling is extended in multiple ways to stochastic optimization, e.g., see (Horv ´ ath & Richt ´ arik, 2019; Gower et al., 2019;
Qian et al., 2019; Gorbunov et al., 2020b;a; Qian et al., 2021a).
Typically, non-uniform sampling of stochastic gradients allows better dependence on smoothness constants in the theoretical results.
Inspired by these advances, we propose the ﬁrst Byz-robust optimization method supporting non-uniform sampling of stochastic gradients.
[Figure]
[Figure]
B E XTRA E XPERIMENTS AND E XPERIMENTAL DETAILS B. G ENERAL SETUP
Our running environment has the following setup:
[Table]
[Figure] [Figure]
• 24 CPUs: Intel(R)
Xeon(R)
Gold 6146 CPU @ 3.20GHz , • GPU:
NVIDIA TITAN Xp with CUDA version 11.3, • PyTorch version: 1.11.0.
[Figure] [Figure]
, LF, BF, ALIE, IPM) on w8a dataset, where each worker access full dataset with 4 good and
Byz worker.
In the ﬁrst row, we do not use any compression, in the second row each method uses Rand K sparsiﬁcation with K = 0 .
1 d .
B. E XPERIMENTAL SETUP
For each experiment, we tune the step size using the following set of candidates { 0 . 5 , 0 .
The step size is ﬁxed.
We do not use learning rate warmup or decay.
We use batches of sizefor all methods.
Each experiment is run with three varying random seeds, and we report the mean optimality gap with one standard error.
The optimal value is obtained by running gradient descent ( GD ) on the complete dataset for 1000 epochs.
Our implementation of attacks and robust aggregation schemes is based on the public implementation from (Karimireddy et al., 2022) available at https://github.com/epfml/byzantine-robust-noniid-optimizer .
Our codes are available via an anonymized repository at https://anonymous.4open.science/r/
VR_Byzantine-2766 .
We select the same set of hyperparameters as (Karimireddy et al., 2022), i.e.,
• RFA: the number of steps of smoothed Weisﬁeld algorithm T = 8 ; see Section D for details • ALIE: a small constant that controls the strength of the attack z is chosen according to (Baruch et al., 2019), • IPM: a small constant that controls the strength of the attack (cid:15) = 0 .
B. E XTRA E XPERIMENTS B.3. H ETEROGENEOUS D ATA
In this case, we randomly shufﬂe dataset and we sequentially distribute it among 15 good workers, where each worker has approximately the same amount of data and there is no overlap.
We include ﬁve Byz workers who have access to an entire dataset and the exact updates computed at each client.
For the aggregation, we consider three rules: standard averaging ( AVG ), coordinate-wise median ( CM ) with bucketing , and robust federated averaging ( RFA ) with bucketing (see the details in Appendix D).
Discussion.
In Figure 2, we can see that momentum ( BR-SGDm ) the variance reduction ( Byz-VRMARINA ) techniques consistently outperform the SGD baseline while none of them dominates for all the attacks.
Byz-VR-MARINA is particularly useful in the clean data regime and against the
ALIE and IPM attacks, and the BR-SGDm algorithm provides the best performance for label and bit ﬂipping attacks.
It would be interesting to automatically select the best technique, e.g., momentum or
VR-MARINA , that provides the best defense against any given attack.
We leave this for future work.
B.3. C OMPRESSION
In this section, we consider the same setup as for the previous experiment with a difference that we employ communication compression.
We choose random unbiased sparsiﬁcation for with sparsity level 10% .
We compare our Byz-VR-MARINA algorithm to compressed SGD and DIANA ( BRDIANA ).
Discussion.
In Figure 3, we can see that Byz-VR-MARINA consistently outperforms both baselines except for the bit ﬂipping attack.
However, even in this case, it seems that Byz-VR-MARINA only needs more epochs to provide the better solution while SGD cannot further improve regardless of the number of epochs.
B.3. E XTRA DATASET : W 8 A
In Figures 4–6, we perform the same experiments, but for the different LIBSVM dataset: w8a .
We note that the obtained results are consistent with our observations for the a9a dataset.
C U SEFUL F ACTS
For all a, b ∈ R d and α > 0 , p ∈ (0 , 1] the following relations hold:
[Equation]
Lemma C.1 (Lemma 5 from Richt´arik et al.
Let a, b > 0 .
If 0 ≤ γ ≤ 1 √ a + b , then aγ 2 + bγ ≤ 1 .
The bound is tight up to the factor of 2 since
[Equation]
D F URTHER D ETAILS ON R OBUST A GGREGATION
In Section 2, we consider robust aggregation rules satisfying Deﬁnition 2.1.
As we notice, this deﬁnition slightly differs from the original one introduced by Karimireddy et al.
In particular, we assume
[Equation]
while Karimireddy et al.
(2022) uses E [ (cid:107) x i − x l (cid:107) 2 ] ≤ σ 2 for any ﬁxed i, l ∈ G .
As we show next, this difference is very subtle and condtion (14) also allows to achieve robustness.
We consider robust aggregation via bucketing proposed by Karimireddy et al.
(see Algorithm 2).
This algorithm can robustify some non-robust aggregation rules Aggr .
In particular,
Algorithm 2 Bucketing : Robust Aggregation using bucketing (Karimireddy et al., 2022)
Karimireddy et al.
(2022) show that Algorithm 2 makes Krum (Blanchard et al., 2017), Robust
Federated Averaging ( RFA ) (Pillutla et al., 2022) (also known as geometric median), and
Coordinate-wise Median ( CM )
(Chen et al., 2017) robust, in view of deﬁnition from Karimireddy et al.
Our main goal in this section is to show that Krum ◦ Bucketing , RFA ◦ Bucketing , and CM ◦ Bucketing satisfy Deﬁnition 2.1.
Before we prove this fact, we need to introduce Krum , RFA ,
CM .
Krum.
Let S i ⊆ { x 1 , . . .
, x n } be the subset of n − |B| − 2 closest vectors to x i .
Then, Krumestimator is deﬁned as
[Equation]
Krum requires computing all pair-wise distances of vectors from { x 1 , . . .
, x n } resulting in O ( n 2 ) computation cost for the server.
Therefore, Krum is computationally expensive, when number of workers n is large.
Robust Federated Averaging.
RFA -estimator ﬁnds a geometric median:
[Equation]
The above problem has no closed form solution.
However, one can compute approximate RFA using several steps of smoothed Weiszfeld algorithm having O ( n ) computation cost of each iteration (Weiszfeld, 1937; Pillutla et al., 2022).
Coordinate-wise Median.
CM -estimator computes a median of each component separately.
That is, for t -th coordinate it is deﬁned as
[Equation]
where [ x ] t is t -th coordinate of vector x ∈ R d . CM has O ( n ) computation cost (Chen et al., 2017;
Yin et al., 2018).
Robustness via bucketing.
The following lemma is the key to show robustness of Krum ◦
Bucketing , RFA ◦ Bucketing , and CM ◦ Bucketing in terms of Deﬁnition 2.1.
Lemma D.1 (Modiﬁcation of Lemma 1 from (Karimireddy et al., 2022)) .
Assume that { x 1 , x 2 , . . .
, x n } is such that there exists a subset G ⊆
[ n ] , |G| =
G ≥ (1 − δ ) n and σ ≥such that (14) holds.
Let vectors { y 1 , . .
, y N } , N = n / s 4 be generated by Algorithm 2 and (cid:101) G = { i ∈
[ N ] | B i ⊆ G} , where y i =
| B i | (cid:80) j ∈ B i x j and B i denotes the i -th bucket, i.e.,
B i = {
π (( i − 1) · s + 1) , . .
, π (min { i · s, n } ) } .
Then, | (cid:101) G|
=
(cid:101) G ≥ (1 − δs ) N and for any ﬁxed i, l ∈ (cid:101) G we have
[Equation]
18) where x = 1 G (cid:80) i ∈G x i .
Proof.
The proof is almost identical to the proof of Lemma 1 from (Karimireddy et al., 2022).
Nevertheless, for the sake of mathematical rigor, we provide a complete proof.
Since each Byz peer is contained in no more than 1 bucket and |B| ≤ δn (here B = [ n ] \ G ), we have that number of “bad” buckets is not greater than δn ≤ δsN , i.e., (cid:101) G ≥ (1 − δs ) N .
Next, for any ﬁxed i ∈ (cid:101) G we have E π [ y i |
[Equation]
where E π [ · ] denotes the expectation w.r.t.
the randomness comming from the permutation.
Taking the full expectation, we obtain the ﬁrst part of (18).
To derive the second part we introduce the notation for workers from B i and B l : B i = { k i, 1 , k i, 2 , . .
, k i,s } and B l = { k l, 1 , k l, 2 , . . .
, k l,s } .
Then, for any ﬁxed i, l ∈ (cid:101) G the (ordered) pairs { ( x k i,t , x k l,t ) } st =1 are identically distributed random variables as well as for any ﬁxed t = 1 , . . .
, s vectors x k i,t , x k l,t are identically distributed.
Therefore, we have
[Equation]
Finally, we notice that S
= 0 : for any ﬁxed i 1 , i 2 , i 3 , i 4 the sum contains one term proportional to E [ (cid:104) x i 1 − x i 2 , x i 3 − x i 4 (cid:105) ] and one term proportional to E [ (cid:104) x i 2 − x i 1 , x i 3 − x i 4 (cid:105) ] that cancel out.
This concludes the proof.
Using the above lemma, we get the following result.
Theorem D.1 (Modiﬁcation of Theorem I from (Karimireddy et al., 2022)) .
Let { x 1 , x 2 , . . .
, x n } satisfy the conditions of Lemma D.1 for some δ ≤ δ max .
If Algorithm 2 is run with s = (cid:98) δ max / δ (cid:99) , then
• Krum ◦ Bucketing satisﬁes Deﬁnition 2.1 with c
= O (1) and δ max < 1 / 4 , • RFA ◦ Bucketing satisﬁes Deﬁnition 2.1 with c
= O (1) and δ max < 1 / 2 , • CM ◦ Bucketing satisﬁes Deﬁnition 2.1 with c
= O ( d ) and δ max < 1 / 2 .
Proof.
The proof is identical to the proof of Theorem I from (Karimireddy et al., 2022), since
Karimireddy et al.
(2022) rely only on the general properties of Krum / RFA / CM and (18) to get the result.
E M ISSING P ROOFS AND D ETAILS F ROM S ECTIONE. E XAMPLES OF S AMPLINGS
Below we provide two examples of situations when Assumption 2.4 holds.
In both cases, we assume that f i,j is L i,j -smooth for all i ∈ G , j ∈ [ m ] .
Example E.1 (Uniform sampling with replacement) .
Consider (cid:98) ∆ i ( x, y )
= 1 b (cid:80) j ∈
I
i,k ∆ i,j ( x, y ) , where ∆ i,j ( x, y )
=
∇ f i,j ( x ) − ∇ f i,j ( y ) and I i,k is the set of b i.i.d.
samples from the uniform distribution on [ m ] .
Then, Assumption 2.4 holds with L 2 ± =
L 2 ± , US , where L 2 ± , US ≤ 1 G (cid:80)
i ∈G
L 2 i, ± , US and L 2 i, ± , US is such that
[Equation]
Lemma 2 from Szlendak et al.
(2021) implies that L 2 i, US − L 2 i ≤ L 2 i, ± , US ≤ L 2 i, US , where L 2 i, US is such that 1 m (cid:80) mj
=1 (cid:107) ∆ i,j ( x, y ) (cid:107) 2 ≤ L 2 i, US (cid:107) x − y (cid:107) 2 .
We point out that in the worst case L 2 i, US = 1 m (cid:80) m j =1 L 2 i,j .
Example E.2 (Importance sampling with replacement) .
Consider (cid:98) ∆ ki = 1 b (cid:80) j ∈
I i,k L i L i,j ∆ i,j ( x, y ) , where ∆ i,j ( x, y ) =
∇ f i,j ( x ) − ∇ f i,j ( y ) , L i = 1 m (cid:80) mj
=1 L i,j , and I i,k is the set of b i.i.d.
samples from the distribution D i, IS on [ m ] such that for j ∼ D i, IS we have P { j = t }
=
L i,t mL i .
Then,
Assumption 2.4 holds with L 2 ± =
L 2 ± , IS such that
[Equation]
Lemma 2 from Szlendak et al.
(2021) implies that 1 G (cid:80) i ∈G ( L 2 i − L 2 i ) ≤ L 2 ± , IS ≤ 1 G (cid:80)
i ∈G
L 2 i .
We point out that L 2 i ≤ L 2 i, US and in the worst case mL 2 i =
L 2 i, US .
Therefore, typically L 2 ± , IS <
L 2 ± , US .
Next, we show that Assumption 2.4 holds whenever f i,j is L i,j -smooth for all i ∈ G , j ∈ [ m ] and (cid:98) ∆ i ( x, y ) can be written as
[Equation]
for some random variables ξ i,j such that E [ ξ i,j ] = 1 , i ∈ G , j ∈ [ m ] and max i ∈G ,j ∈ [ m ] E [ ξ 2 i,j ] =
E 2 < ∞ 5 .
Indeed, in this case, we have
[Equation]
meaning that Assumption 2.4 holds with L ± ≤ √ bE max i ∈G ,j ∈ [ m ] L i,j .
However, as we show in
Examples E.1 and E.2 constant L ± can be much smaller than the derived upper bound.
[Equation]
Our theory works for a slightly more general setting than the one we discussed in the main part of the paper.
In particular, instead of Assumption 2.2 we consider a more general assumption on the heterogeneity.
Assumption E.1 ( ( B, ζ 2 ) -heterogeneity) .
We assume that good clients have ( B, ζ 2 ) -heterogeneous local loss functions for some B ≥ 0 , ζ ≥ 0 , i.e.,
[Equation]
When B =
the above assumption recovers Assumption 2.2.
Since we allow B to be positive, it may reduce the value of ζ in some applications (for example, for over-parameterized models).
As we show further, the best possible optimization error that Byz-VR-MARINA can achieve in this case is proportional to ζ 2 .
We refer to Karimireddy et al.
for the study of typical values of parameter
B for some over-parameterized models.
In proofs, we need the following lemma, wich is often used for analyzing SGD -like methods in the non-convex case.
Lemma E.1 (Lemma 2 from Li et al.
Assume that function f is L -smooth and x k +
=
x
k − γg k .
Then
[Equation]
[Equation]
To estimate the “quality” of robust aggregation at iteration k + 1 we derive an upper bound for the averaged pairwise variance of estimators obtained by good peers (see also Deﬁnition 2.1).
Lemma E.2 (Bound on the variance) .
Let Assumptions 2.1, E.1, 2.3, 2.4 hold.
Then for all k ≥
the iterates produced by Byz-VR-MARINA satisfy
[Equation]
where A
[-(cid:16) 8 BpL 2 + 4(1 − p ) (cid:16) ωL 
 + (1 + ω ) L 2 ± + (1+ ω ) L 2 ± b (cid:17)(cid:17) .-][Equation]
Proof.
For the compactness, we introduce new notation: ∆ k = ∇ f ( x k +1 )
− ∇ f ( x k ) .
Let E c k [ · ] denote the expectation w [Equation] r [Equation]
t
[Equation]
c k .
Then, by deﬁnition of g k +1 i for i ∈ G we have
[Equation]
Taking the full expectation and using the tower property E [ E c k [ ·
]] = E [ · ] , we derive
[Equation]
Term E [ T 1 ] can be bounded via Assumption 19:
[Equation]
To estimate E [ T 2 ] we ﬁrst derive an upper bound for E k
[ T 2 ] , where E k [ · ] denotes expectation w.r.t.
the all randomness (compression and stochasticity of the the gradients) coming from the step k +of the algorithm:
E k
[ T 2 ]
[Equation]
∆
[Equation]
where E Q k [ · ] denotes the expectation w.r.t.
the randomness coming from the compression at step k + 1 .
Applying Assumptions 2.4, 2.3, and 2.1 and taking the full expectation, we get
[Equation]
Plugging the upper bounds for E [ T 1 ] and E
[ T 2 ] in (22), we obtain the result.
Using the above lemma, we derive the following technical result, which we rely on in the proofs of the main results.
Lemma E.3 (Bound on the distortion) .
Let Assumptions 2.1, E.1, 2.3, 2.4 hold.
Then for all k ≥the iterates produced by Byz-VR-MARINA satisfy
[Equation]
where A = 48 BL 2 cδ p
[Equation]
Proof.
For convenience, we intoduce the following notation:
[Equation]
[Equation]
Using the introduced notation, we derive
[Equation]
Next, we need to upper-bound the terms from the right-hand side of (25).
Let E c k [ · ] denote the expectation w.r.t.
c k .
Then, in view of (24), we have
[Equation]
Taking expectation E k [ · ] w.r.t.
the all randomness (compression and stochasticity of the the gradients) coming from the step k + 1 of the algorithm and applying the variance decomposition and independence of mini-batch and compression computations on different workers, we get E k
[Equation]
g k +1 − ∇ f [Equation] x k +1 ) [Equation]
[Equation] 1 − [Equation] k 
[Equation]
Let E Q k [ · ] denote the expectation w.r.t.
the randomness coming from the compression at step k + 1 .
The deﬁnition of the unbiased compression operator (Deﬁnition 2.2) implies
[Equation]
Using Assumptions 2.4, 2.
[Equation]
and 2.1 and taking the full expectation
[Equation]
we arrive at
[Equation]
That is, we obtained an upper bound for the ﬁrst term in the right-hand side of (25).
To bound the second term, we use the deﬁnition of ( δ, c ) - ARAgg (Deﬁnition 2.1) and Lemma E.2:
[Equation]
where A
[Equation]
(cid:17)(cid:17) .
Plugging (26) and (27) in [Equation] and using p ≤
[Equation]
we obtain
[Equation]
where
[Equation]
This concludes the proof.
E. G ENERAL N ON -C ONVEX F UNCTIONS
Theorem E.1 (Generalized version of Theorem 2.1) .
Let Assumptions 2.1, E
[Equation]
[-[Equation] where A = 
 BL 2 cδ p + 6(1 − p ) p-]1, 2.3, 2.4 hold. Assume that
[Equation]
[Equation]
[-12 G (cid:17) (cid:16) ωL 
 + (1+ ω ) L [Equation] ± b (cid:17) + 6(1 − p ) p (cid:16) 4-]where A = 48 BL 2 cδ p
[Equation]
+ ω 2 G (cid:17) L 2 ± .
Then for all K ≥
the iterates produced by Byz-VR-MARINA satisfy
[Equation]
(29) where [Equation]
is choosen uniformly at random from x 0 , x 1 , . . .
, x K , and Φ= f [Equation] x [Equation]
− f [Equation] + γp (cid:107) g 0 − ∇ f ( x 0 ) (cid:107) 2 .
The result of Theorem 2.1 is a special case of the statement above with B = 0 , since for B =
we have A = 48 BL
[Equation]
=
[Equation]
, p − 48 Bcδ = p , and the second condition from (28) always holds.
Proof.
For all k ≥ 0 we introduce Φ k = f
( x k ) − f ∗ + γp (cid:107) g k − ∇ f ( x k ) (cid:107) 2 .
Using the results of
Lemmas E.1 and E.3, we derive
[Equation]
where in the last step we use Lemma C.1 and our choice of γ from (28).
Next, in view of (28), we have γ 2 (cid:16) 1 − 48 Bcδp (cid:17) > 0 .
Therefore, summing up the above inequality for k = 0 , 1 , . .
, K and rearranging the terms, we get
[Equation]
It remains to notice, that the lef-hand side equals E [ (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 ] , where (cid:98)
x K is choosen uniformly at random from x 0 , x 1 , . . .
, x K .
On the differences between Byz-VR-MARINA and momentum-based methods.
Karimireddy et al.
(2021) use momentum and momentum-based variance reduction in order to prevent the algorithm from being permutation-invariant, since in the setup considered by Karimireddy et al.
all permuation-invariant algorithms cannot converge to any predeﬁned accuracy even in the homogeneous case.
In our paper, we provide a different perspective on this problem: it turns out that the method can be variance-reduced, Byz-robust, and permutation-invariant at the same time.
Let us ﬁrst reﬁne what we mean by permutation-invariance since our deﬁnition slightly differs from the one used by Karimireddy et al.
That is, consider the homogeneous setup and assume that there are no Byz workers.
We say that the algorithm is permutation-invariant if one can arbitrarily permute the results of stochastic gradients computations (not necessarily one stochastic gradient computation) between workers at any aggregation step without changing the output of the method.
Then, Byz-VR-MARINA is permutation-invariant, since the output in line 10 depends only on g k and the set { ∆ ki } i ∈ [ n ] (note that Q ( x ) =
x , since we assume ω = 0 ), not on their order.
Our results do not contradict the ones from Karimireddy et al.
(2021), since Karimireddy et al.
(2021) assume that the variance of the stochastic gradient is bounded, while we apply variance reduction implying that the variance goes to zero and Byz workers cannot successfuly “hide in the noise” via time-coupled attacks anymore.
Before we move on to the corollaries, we ellaborate on the derived upper bound.
In particular, it is important to estimate E [Φ 0 ] .
By deﬁnition, Φ
= f ( x 0 ) − f ∗ + γ p (cid:107) g 0 − ∇ f ( x 0 ) (cid:107) 2 , i.e., Φ 0 depends on the choice of g 0 .
For example, one can ask good workers to compute h i = ∇ f i ( x 0 ) , i ∈ G and send it to the server.
Then, the server can set g 0 as g
=
ARAgg ( h 1 , . .
h n ) .
This gives us
E [Φ 0 ]
[Equation]
Function f is L -smooth that implies (cid:107)∇ f ( x 0 ) (cid:107) 2 ≤ 2 L (cid:0) f ( x 0 ) − f ∗ (cid:1) .
Using this and δ < p / (48 cB ) and γ ≤ 1 / L , we derive
[Equation]
Plugging this upper bound in (29), we get
[Equation]
Based on this inequality we derive following corollaries.
Corollary E.1 (Homogeneous data, no compression ( ω = 0 )) .
Let the assumptions of Theorem E. hold, Q ( x ) ≡ x for all x ∈ R d (no compression, ω = 0 ), p =
b / m , B = 0 , ζ = 0 , and
[Equation]
Then for all K ≥
we have E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) of the order
[Equation]
(31) where (cid:98) x K is choosen uniformly at random from the iterates x 0 , x 1 , . .
. , x K produced by ByzVR-MARINA and ∆
= f ( x 0 ) − f ∗ .
That is, to guarantee E (cid:2) (cid:107)∇ f ( (cid:98) x K [Equation]
Byz-VR-MARINA requires
[Equation]
communication rounds and
[Equation]
oracle calls per worker.
Corollary E.2 (No compression ( ω = 0 )) .
Let the assumptions of Theorem E.1 hold, Q ( x ) ≡ x for all x ∈ R d (no compression, ω = 0 ), p =
b / m
and
[Equation]
Then for all K ≥
we have E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) of the order
[Equation]
where (cid:98) x K is choosen uniformly at random from the iterates x 0 , x 1 , . .
. , x K produced by Byz-VRMARINA and ∆
= f ( x 0 ) − f ∗ .
That is, to guarantee E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) ≤ ε 2 for ε 2 ≥ 12 cδζ 2 p
[Equation] [Equation]
Byz-VR-MARINA requires
[Equation]
communication rounds and
[Equation]
oracle calls per worker.
Corollary E.3.
Let the assumptions of Theorem E.1 hold, p = min { b / m , 1 / 1+ ω } and γ =
L + √ A, where
[Equation]
Then for all K ≥
we have E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) of the order
[Equation]
[Equation]
where (cid:98) x K is choosen uniformly at random from the iterates x 0 , x 1 , . .
. , x K produced by Byz-VRMARINA and ∆
= f ( x 0 ) − f ∗ .
That is, to guarantee E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) [Equation] ε [Equation]
for ε
[Equation]
Bcδ
Byz-VR-MARINA requires
[Equation]
communication rounds and
[Equation]
oracle calls per worker.
Corollary E.4 (Homogeneous data) .
Let the assumptions of Theorem E.1 hold,
p = min
{ b / m , 1 / 1+ ω } ,
B = 0 , ζ = 0 , and
[Equation]
where
[Equation]
Then for all K ≥
we have E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) of the order
[Equation]
where (cid:98) x K is choosen uniformly at random from the iterates x 0 , x 1 , . .
. , x K produced by ByzVR-MARINA and ∆
= f ( x 0 ) − f ∗ .
That is, to guarantee E (cid:2) (cid:107)∇ f ( (cid:98) x K ) (cid:107) 2 (cid:3) ≤ ε 2 for ε 2 >
Byz-VR-MARINA requires
[Equation]
communication rounds and
[Equation]
oracle calls per worker.
E. F UNCTIONS S ATISFYING P OLYAK -Ł OJASIEWICZ C ONDITION
Theorem E.2 (Generalized version of Theorem 2.2) .
Let Assumptions 2.1, E.1, 2.3, 2.4, 2.5 hold.
Assume that
[Equation]
where A =
BL 2 cδ p +
[Equation]
p
[Equation]
[Equation] p
[Equation]
p (cid:16)
[Equation]
(1+ ω ) p + ω 2 G (cid:17)
L 2 ± .
Then for all K ≥
the iterates produced by Byz-VR-MARINA satisfy
[Equation]
where Φ= f ( x 0 ) − f ( x ∗ ) + 2 γp (cid:107)
[Equation]
[Equation] ∇ f [Equation] x
[Equation] (cid:107) 2 .
The result of Theorem [Equation] 1 is a special case of the statement above with B = 0 , since for B =
we have
[Equation]
± , (cid:16)
[Equation] [Equation] [Equation] 96 Bcδ = p , and the second condition from (43) always holds.
Proof.
For all k ≥ 0 we introduce Φ k = f
( x k ) − f ∗ + 2 γp (cid:107) g k − ∇ f ( x k ) (cid:107) 2 .
Using the results of
Lemmas E.1 and E.3, we derive
[Equation]
p where in the last step we use Lemma C.1 and our choice of γ from (43).
Unrolling the recurrence
[Equation]
we obtain
[Equation]
Taking into account Φ k ≥
[Equation]
f ( x ∗ ) , we get the result.
As in the case of general non-convex smooth functions, we need to estimate Φ 0 to derive complexity results.
Following exactly the same reasoning as in the derivation of (30), we get
[Equation]
Plugging this upper bound in (44), we get
[Equation]
Based on this inequality we derive following corollaries.
Corollary E.5 (Homogeneous data, no compression ( ω = 0 )) .
Let the assumptions of Theorem E. hold, Q ( x ) ≡ x for all x ∈ R d (no compression, ω = 0 ), p =
b / m , B = 0 , ζ = 0 , and
[Equation]
Then for all K ≥
we have E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) of the order
O
[Equation]
where ∆
= f ( x 0 ) − f ( x ∗ ) .
That is, to guarantee E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) ≤ ε for ε >
Byz-VRMARINA requires
O
[Equation]
communication rounds and
[Equation]
oracle calls per worker.
Corollary E.6 (No compression ( ω = 0 )) .
Let the assumptions of Theorem E.2 hold, Q ( x ) ≡ x for all x ∈ R d (no compression, ω = 0 ), p =
b / m and γ =
min
[Equation]

[Equation]
Then for all K ≥
we have E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) of the order
O (cid:32) exp 
[Equation]
[Equation]
where ∆
= f ( x 0 ) − f ( x ∗ ) .
That is, to guarantee E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) ≤ ε for ε ≥ 32 cδζ 2 µ ( p
[Equation]
96 Bcδ )
Byz-VR-MARINA requires
[Equation]
communication rounds and
O
[Equation]
oracle calls per worker.
Corollary E.7.
Let the assumptions of Theorem E.2 hold, p = min { b / m , 1 / 1+ ω } and
[Equation]
  , where
[Equation]
Then for all K ≥
we have E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) of the order
[Equation]
where ∆
= f ( x 0 ) − f ( x ∗ ) .
That is, to guarantee E (cid:2) f
( x K ) −
[Equation]
for ε ≥
[Equation]
Byz-VR-MARINA requires
[Equation]
communication rounds and
O
[Equation]
oracle calls per worker.
Corollary E.8 (Homogeneous data) .
Let the assumptions of Theorem E.2 hold,
p = min { b / m , 1 / 1+ ω } ,
B = 0 , ζ = 0 , and γ = min  
[Equation]
4 µ   , where
A =(cid:32)
[Equation]
max (cid:26)
[Equation]
(1 + ω )
[Equation]
(cid:27)
[Equation]
max (cid:8) mb ,
[Equation]
ω (cid:9)
[Equation]
G (cid:33) (cid:18) ωL
+ (1 + ω )
L 2 ± b (cid:19) .
Then for all K ≥
we have E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) of the order
O (cid:18) exp (cid:18) − min (cid:26) µ
L + √ A, b m, 1 + ω (cid:27)
K (cid:19) ∆(cid:19) , (54) where ∆
= f ( x 0 ) − f ( x ∗ ) .
That is, to guarantee E (cid:2) f
( x K ) − f ( x ∗ ) (cid:3) ≤ ε for ε >
Byz-VRMARINA requires
[Equation]
[Equation]
communication rounds and
O
[Equation] [Equation]
oracle calls per worker.
E. F URTHER D ETAILS ON THE O BTAINED R ESULTS AND THE C OMPARISON FROM T ABLE
In this part, we discuss additional details about the obtained results and on the comparison of methods complexities given in Table 2.
We mostly focus on the results for general smooth non-convex functions.
Similar observations are valid for smooth PŁfunctions as well.
Comparison of the assumptions on the stochastic gradient noise.
Many existing works rely on the uniformly bounded variance assumption (UBV): it is assumed that for all x ∈ R
d the good workers have an access to the unbiased estimators g i ( x ) of ∇ f i ( x ) such that E (cid:107) g
i ( x ) − ∇ f i ( x ) (cid:107) 2 ≤ σ 2 for all i ∈ G and σ ≥ 0 .
This assumption does not hold in many practical situations and even for simple convex ﬁnite-sum problems like sums of quadratic functions with non-identical Hessians.
Moreover, in the situations when this assumption holds, the value of σ 2 can be huge.
However, UBV assumption does not require individual stochastic realizations, i.e., summands f i,j , to be smooth.
In contrast, we use Assumption 2.4 that holds for many situations when UBV assumption does not.
For example, Assumption 2.4 holds whenever all functions f i,j , i ∈ G , j ∈ m are L i,j -smooth (see Appendix E.1).
These facts allow us to cover a large class of problems that does not ﬁt the setup considered in (Karimireddy et al., 2021; 2022; Gorbunov et al., 2021a).
Moreover, since
Assumption 2.4 is more general than smoothness of all f i,j , our analysis covers the setup considered in (Wu et al., 2020; Zhu & Ling, 2021).
However, it is worth mentioning that there exist problems such that UBV assumption holds and Assumption 2.4 does not, e.g., when the gradient noise is additive: (cid:98)
[Equation]
+ ξ i , where E ξ i = 0 and E (cid:107) ξ i (cid:107) 2 = σ 2 .
On the choice of p .
Our analysis is valid for any choice of p ∈ (0 , 1] .
As we explain in footnote 2, the choice of p = min { b / m , 1 / (1+ ω ) } leads to the fair comparison with other results, since this choice implies that the total expected (communication and oracle) cost of steps with full gradients computations/uncompressed communications coincides with the total cost of the rest of iterations.
Indeed, to measure the communication efﬁciency one can use expected density ζ Q (see Deﬁnition 2.2).
Then, the expected number of components that each worker sends to the server at each step is upperbounded by ζ Q (1 − p ) + pd meaning that p = ζ Q / d makes the expected number of components that each worker sends to the server at each step equal to O ( ζ Q ) .
In the case when 1 + ω =
Θ( d / ζ Q ) (which is the case for RandK sparsiﬁcation and (cid:96) 2 -quantization, see (Beznosikov et al., 2020)), one can choose p = 1 / (1+ ω ) .
On the other hand, the expected number of oracle calls per iteration is 2 b (1 − p ) + mp meaning that p
=
b / m makes the expected oracle cost of each iteration equal to O ( b ) like in the case of
SGD .
This means that the best p for oracle complexity and the best p for communication efﬁciency are different in general.
When b / m < 1 / (1+ ω ) , the choice p = min { b / m , 1 / (1+ ω ) } implies that the algorithm could use uncompressed vectors more often without sacriﬁcing the communication cost, and when b / m > 1 / (1+ ω ) , the choice p = min { b/m, 1 / (1 + ω ) } implies that the algorithm could use full gradients more often without sacriﬁcing the oracle cost.
That is, the choice of p ≈ b / m implies better oracle complexity and the choice p ≈ 1 / (1+ ω ) leads to better communication efﬁciency.
Depending on how much these two aspects are important for the particular application, one can choose p in between these two values.
The effect of compression.
For simplicity, consider a homogeneous case ( B = 0 , ζ = 0 ) and let b
= 1 ; similar arguments are valid for the general case.
As Corollary E.4 states, the communication complexity 6 of Byz-VR-MARINA in this case equals
[Equation]
where
[Equation]
The above result shows that the communication complexity becomes worse with the growth of ω .
Larger ω means that the compression Q is more loose, i.e., less information is communicated.
This is a common phenomenon for the methods with communication compression (Horv´ath et al., 2019b;
Gorbunov et al., 2021b).
However, when the compression is not severe, e.g., 1 + ω ≤ m , then the complexity bound becomes
[Equation]
where A =
[Equation]
, which is worse only O ( √ ω ) times than the complexity of Byz-VR-MARINA without compression, while the number of communicated bits/components becomes O ( d / ζ Q ) times smaller.
For example, in the case of RandK sparsiﬁcation we have 1 + ω
=
d / K = d / ζ Q , and 1 + ω ≤ m allows to have quite strong compression, e.g., for m ≥ 1000 , meaning that the dataset has at least 1000 samples, inequality 1 + ω ≤ m implies that workers can send just 0 .
1% of information.
In this case, the communication cost of each iteration becomes ∼ 1000 times cheaper, while the number of communication rounds increases only √ 1 + ω ∼ 30 times.
If the communication is the bottleneck, then the algorithm will converge much faster with compression than without it in this setup.
On the batchsizes.
First, we note that our analysis is valid for any choice of b ≥ 1 .
For simplicity of the further discussion of the batchsizes role in the complexities, consider the homogeneous case ( B = 0 , ζ = 0 ) without compression ( ω = 0 ).
As Corollary E.1 states, the communication complexity of Byz-VR-MARINA in this case equals
[Equation]
Note that the term depending on the ratio of Byz workers δ scales as b − 3 / 2 with the batchsize and the term depending on 1 / G scales as b − 1 .
Table 2 illustrates that previous SOTA results in this case scale as b − 1 or b − 1 / 2 , so, the complexity bound for Byz-VR-MARINA scales with b no worse than the concurrent bounds.
Next, typically, there is no need to take b larger than √ m for SARAH -based variance reduced methods (Horv ´ ath et al., 2022; Li et al., 2021): oracle complexity is always the same (neglecting the differences in the smoothness constants), while the iteration complexity stops improving once b becomes larger than √ m .
However, the complexity bound for Byz-VR-MARINA contains the non-standard term O (cid:0) L ± m √ cδ / √ b 3 ε 2 (cid:1) appearing due to the presence of Byz workers.
For simplicity, we assume that L = Θ( L ± ) (though L ± can be both smaller and larger than L ).
Then, when we increase batchisze b , the communication complexity stops improving once b becomes larger than max { 3 √ cδm 2 , m } .
Interestingly, 3 √ cδm 2 can be larger than the standard value √ m : this is the case when m > 1 / c 2 δ 2 .
In this case, the communication complexity of Byz-VR-MARINA beneﬁts from the slightly larger batchsizes than in the classical case.
This phenomenon has a natural explanation: when we increase the batchsize, the variance of the gradient noise decreases and it becomes even harder for Byz workers to shift the updates of the method signiﬁcantly.