Communication-efficient distributed eigenspace estimation with arbitrary node failures
Abstract
We develop an eigenspace estimation algorithm for distributed environments with arbitrary node failures, where a subset of computing nodes can return structurally valid but otherwise arbitrarily chosen responses.
Notably, this setting encompasses several important scenarios that arise in distributed computing and data-collection environments such as silent/soft errors, outliers or corrupted data at certain nodes, and adversarial responses.
Our estimator builds upon and matches the performance of a recently proposed non-robust estimator up to an additive ˜ O ( σ √ α ) error, where σ 2 is the variance of the existing estimator and α is the fraction of corrupted nodes
Problem overview and background
Modern machine learning has seen the proliferation of heterogeneous distributed environments for training and deploying data science pipelines.
As communication between machines is often the most time-consuming operation in distributed systems, the design of communication-efficient algorithms is of paramount importance for scaling to massive datasets [36].
However, the move to distributed environments also adds several additional layers of complexity in the design of algorithms.
For example, in the distributed setting we would like our algorithms to be robust and providing meaningful answers even in settings where some nodes contain outlier data [4], silently fail during the computation [27, 31], or are compromised and returning malicious results designed to corrupt the central solution.
This work focuses on distributed eigenspace estimation in the context of robustness to node-level corruptions.
Formally, we assume a computing environment with nodes numbered i = 1 , . . .
, m , where every node i observes a local version
A i of an unknown symmetric matrix A ∈ R d × d ; the goal is to approximate the subspace spanned by the r ≪
d principal eigenvectors of A .
Distributed PCA is a standard example in this framework: every machine draws n i.i.d.
samples from an unknown distribution P with covariance matrix A and forms a local empirical covariance matrix A i .
Recently proposed communication-efficient algorithms have every node i transmit V i , the d × r matrix of principal eigenvectors of A i , to a central server, which then aggregates all the local solutions via a carefully-crafted aggregation procedure [10, 18].
We devise and analyze an algorithm that is robust to a wide range of corruptions that can occur to a subset of the computational nodes.
In particular, we assume that some fraction α of the computational nodes can respond with completely arbitrary, but structurally valid, responses (i.e., they return arbitrary matrices
V i with orthonormal columns).
This model encompasses three common forms of node-level corruption that cannot be easily detected by the central machine in isolation:
Silent/soft errors.
While computational errors may be rare on single machines, as distributed workloads span large numbers of nodes the probability that some of them fail becomes significant.
Though catastrophic failures may be detectable, allowing the central server to simply ignore the output of specific nodes, the more nefarious issue is that of so-called silent (or soft) errors [16, 19, 27].
More specifically, a silent error is one where a node returns an erroneous but structurally valid response to the central machine query.
Because the response is structurally valid and the central machine may not have access to the per-node data it is not possible to “validate” the response of each node and, instead, the central estimator must be adapted to be robust to such errors.
Outliers or corrupted data.
In certain settings the data collection may be distributed in addition to the computation.
If some of the nodes are drawing samples from an invalid or corrupted data source they may introduce gross outliers to the set of responses { V i | i ∈ [ m ] } .
Similarly, in the distributed PCA example, while most machines draw a sufficient number of samples, a minority of them may have only a small amount of data available such that the principal eigenspaces of the local empirical covariance matrices are too far from the ground truth, and thus violate standard modelling assumptions in distributed learning.
Again, robustness to such outlier responses must be a feature of the estimator since they cannot be detected by individual nodes (as they do not have information about the global problem).
Adversarial responses.
In some settings, a subset of nodes may be compromised by an adversary who wishes to influence the central solution by crafting and returning malicious V i .
In fact, the adversarial nodes may be collaborating when constructing their responses.
Since the central node does not get to see all the data it cannot validate responses or directly detect adversaries.
Therefore, the estimator itself must be adapted to be robust to collections of responses designed to push the solution in specific directions.
The main contribution of our paper is a communication-efficient algorithm that is robust to node corruptions (as outlined above) for the distributed eigenspace estimation problem.
We note that our corruption model is similar to so-called Byzantine failures [25] in distributed systems.
1.1 Related work
Distributed eigenspace estimation.
The problem of distributed eigenspace estimation has been well-studied in the absence of malicious noise.
One of the challenges in the distributed setting is aggregating local solutions in the presence of symmetry: for example, if v is an eigenvector of A , both ± v are valid solutions to our problem.
Various works deal with such symmetries in different ways; in the algorithms of [5, 18], the central node averages the spectral projectors of the local eigenspaces, and performs an eigendecomposition of the resulting average to approximate the principal eigenspace.
This approach is similar to the algorithms of [3, 11, 26], although the latter works focus on distributed low-rank approximations and do not address the issue of approximating the principal eigenspace directly.
Another standard approach is for the central server to aggregate local solutions after an alignment step designed to remove the orthogonal ambiguity [10, 17, 21] (see also [7] for the nondistributed setting).
Indeed, our work builds on the two-stage algorithm presented and analyzed in [10] for the non-robust setting.
Finally, we briefly mention a recent line of work [12, 21] that adapts the shift-and-invert preconditioning framework [20] to the distributed setting; however, the latter approach leads to algorithms that require multiple rounds of communication.
Robust PCA.
The literature contains a number of different formulations for robust principal component analysis.
The seminal work of Candés et al.
[9] formulated robust PCA as the task of separating an observed matrix
Y ∈ R
d × d into a low-rank and a sparse component – a slightly different problem from that considered in this paper.
Xu et al.
[34] considered the problem of approximating a low-dimensional distribution from a set of n i.i.d.
samples, a constant fraction of which have been individually corrupted by gross outliers.
Follow-up works focused on high-dimensional sparse estimation and applications to sparse robust PCA [2, 15].
However, communication-efficient robust
PCA appears to be overlooked.
A notable exception is the sketch-based algorithm of [30], although the assumptions therein depart significantly from our setting.
Another relevant line of work is that of Byzantine-robust distributed learning (typically focusing on distributed gradient descent); see, e.g., [1, 6, 8, 13, 14, 24, 29, 35] as well as the survey [23].
In these works, an iterative algorithm is distributed across machines that send individual updates to a central server, which combines them using a robust aggregation procedure (e.g., the geometric median [29]).
Such works are more general in scope, but typically lead to estimators that require multiple rounds of communication.
For minimization of strongly convex quadratics, [35] proposes a one-shot algorithm employing a robust aggregation scheme.
However, this special case departs from our setting as their underlying problem is convex and solved under the assumption that solutions are unique.
1.2 Notation
We let S d − 1 denote the unit sphere in d dimensions.
We write ∥ A ∥ F :=
(cid:112) ⟨ A, A ⟩ and ∥
A ∥ 2 := sup x ∈ S d − 1 ∥
Ax ∥ 2 for the Frobenius and spectral norms of a matrix A ∈ R n × d .
We write O n,r for the set of n × r matrices with orthonormal columns and O r ≡ O r,r .
Given U, V ∈ O n,r we write
[Equation]
) for their ℓ 2 → ℓ 2 subspace distance and B dist ( U ; r ) for the scaled unit ball centered at U :
[Equation]
Finally, we use the notation A ≲ B to indicate that A ≤ cB for a dimension-independent constant c > 0 and A ≍ B if A ≲ B and B ≲ A simultaneously.
Robust distributed eigenspace estimation
We now formally introduce the problem setting.
In particular, we assume there exists an unknown symmetric matrix A with spectral decomposition
[Equation]
assuming a nonincreasing ordering on the eigenvalues:
[Equation]
Our goal is to approximate the principal r -dimensional eigenspace V :=
span( V ) of A given m machines, each of which observes a local version A i of A , communicating with a central coordinator.
We assume that m is even for simplicity.
When queried for a response, machine i responds either with an eigenvector matrix spanning the principal eigenspace of the local matrix
A i , or with an arbitrary d × r matrix with orthonormal columns.
The latter case corresponds to so-called compromised machines.
In contrast, prior work [10, 18] assumes that every machine responds truthfully.
Assumption 1 (Corruption model) .
There exists a constant α ∈ (0 , 1 / 2 ) and an index set I bad ⊂ [ m ] with |I bad | /m ≤ α such that the following holds: all nodes i / ∈ I bad observe a symmetric matrix
A i ∈ R d × d .
Moreover, when queried for a response, every node i returns
[Equation]
where the columns of V i ∈ O d,r span the principal r -dimensional eigenspace of A i and Q i ∈
O d,r is an arbitrary d × r matrix with orthonormal columns.
For notational convenience, we also define the set of “good” responses:
[Equation]
Furthermore, we require the principal eigenspace of A to be sufficiently separated from its complement and that the local errors E i :
=
A i − A are not too large.
Assumption 2.
There is a constant δ > 0 such that the following hold:
(Gap)
The matrix A has a nontrivial eigengap:
[Equation]
(Approximation)
For all i ∈ I good , the local observations satisfy:
[Equation]
The problem admits a natural proxy of difficulty in the form of the normalized inverse eigengap κ :
[Equation]
Normalized
[Equation]
eigengap
[Equation]
Our algorithm for the robust distributed eigenspace estimation problem is outlined in Algorithm 1, which is essentially a robust version of the Procrustes fixing algorithm from [10].
The latter (nonrobust) algorithm operates as follows: first, every machine i computes its local eigenvector matrix V i and broadcasts it to the central server.
Because invariant subspaces do not admit unique representations, naively averaging these estimates can fail to reduce the approximation error further.
Instead, the algorithm of [10] first picks one of the local solutions (say V 1 ) as a reference and “aligns” every other solution with it by solving a so-called Procrustes problem :
[Equation]
After the alignment step (8), the solution of which is available in closed form via the SVD [22], the central coordinator computes and returns the empirical average (1 /m ) (cid:80) mi
=1 V i Z i .
2.1 Technical challenges and main result
To robustify the algorithm described above against node failures, we need the following ingredients:
Reference estimation.
In the presence of corruptions one must guard against the possibility of choosing an outlier as a reference solution (which would render the alignment step (8) useless).
The first step of our algorithm robustly determines a reference guaranteed to have nontrivial alignment with the ground truth (Algorithm 2).
Solution aggregation.
With the robust reference at hand, the next step of the algorithm aligns other local solutions with it.
However, since some of the solutions are outliers, we use a robust mean estimation algorithm in the last step of Algorithm 1 to compute the empirical average only over inliers (and possible “benign” outliers) with high probability.
Combining the above ingredients requires working around additional technical challenges.
The first challenge is showing that averaging over inliers that have been aligned with (cid:98) V ref is equivalent (up to small error) to averaging over inliers that have been aligned with the ground truth V .
To do so, we leverage a path-independence result from [33] to view the aligned estimates as the eigenvectors arising from a carefully chosen sequence of perturbations to the unknown matrix
A (Lemma 1).
After aligning local estimates, we approximate the empirical mean over inliers with a spectral filtering algorithm.
Existing analyses of that algorithm focus on vector-valued inputs and lead to error bounds in the Euclidean norm (or other vector norms, see [32]).
Invoking these error bounds in a black-box fashion for matrix-valued inputs (by flattening every eigenvector matrix to a ( dr ) -dimensional vector) has multiple drawbacks.
On one hand, it leads to unnatural scaling as the final result depends on the empirical covariance over flattened inputs.
On the other hand, the resulting error bounds are with respect to the Frobenius norm, while the spectral norm is a standard error measure in eigenspace estimation.
To avoid this, we modify the iterative filtering algorithm so that inputs are not flattened and obtain error bounds with respect to the spectral norm (3).
We analyze each ingredient of Algorithm 1 separately, in Sections 2.2 to 2.4; all proofs appear in the supplementary material.
Our analysis is almost completely deterministic: indeed, the only source of randomness is the filtering algorithm used in the final stage (Algorithm 5).
Algorithm
Input : responses (cid:8)(cid:98) V i (cid:9) i =1 ,...,m , corruption fraction α , failure prob.
p , error parameter ω .
(cid:98) V ref :=
RobustReferenceEstimator (cid:0)(cid:98) V 1 , . .
, (cid:98) V m (cid:1) .
▷ Algorithm 2; Section 2.
(cid:8)(cid:101) V i (cid:9) i =1 ,...,m :=
ProcrustesFixing (cid:0)(cid:8)(cid:98) V 1 , . .
, (cid:98) V m (cid:9) , (cid:98) V ref (cid:1)
▷ Algorithm 3; Section 2.
¯ V :=
AdaptiveFilter ( (cid:8)(cid:101) V 1 , . . .
, (cid:101) V m (cid:9) , 6 , ω, p, α )
▷ Algorithm 5; Section 2.
return ¯ V
Our main theorem on the performance of Algorithm 1 follows below.
The result is first stated for inputs from a generic eigenspace estimation problem.
In Section 3, we apply it in a black-box fashion to specialize its results for distributed PCA.
Theorem 1.
Let Assumptions 1 and 2 hold and suppose that the corruption level α satisfies
[Equation]
For a fixed failure probability p and error parameter ω
[Equation]
Algorithm 1 returns ¯ V ∈ R d × r satisfying:
[Equation]
with probability at least 1 − 2 log ( 6 / ω ) · p .
Moreover
[Equation]
the variance σ 2 satisfies
[Equation]
Remark 1 (Error bound interpretation) .
The error bound (10) decomposes naturally into three terms that can be controlled individually in applications.
The first term, E oracle , corresponds to an oracle estimator that approximates V via the principal eigenspace of 1 / |I good | (cid:80) i ∈I
good A i .
The second term,
E high , represents high-order errors that occur as a result of the alignment step in Algorithm 3 that are dominated by E oracle in “typical” situations.
Finally, the term E robust quantifies the effect of outliers, vanishing as the fraction of corrupted nodes α ↓ 0 .
We comment on the scaling of E robust relative to the error of the non-robust algorithm in the context of distributed PCA in Section 3.
2.2 The robust reference estimator
This section focuses on the analysis of Algorithm 2, which yields the robust reference estimator (cid:98) V ref used to remove the orthogonal ambiguity from local solutions.
We note that the construction of the estimator dates back to the seminal work of Nemirovski and Yudin [28].
Algorithm 2 RobustReferenceEstimator ( Y 1 , . . .
, Y m ) for i
= 1 , . .
, m do ε i :=
min { r ≥ 0 | |B dist ( Y i ; r ) ∩ { Y i } mi =1 | > m 2 } return Y i ⋆ , where i ⋆ := argmin i ∈ [ m ] ε i
Remark 2.
The quantities ε i in Algorithm 2 can be found in time O ( m 2 dr 2 ) by first computing r j :=
dist
( Y i , Y j ) for all j ̸
=
i and setting ε i :=
median ( { r j } j ̸ = i ) .
Note that even though (cid:98) V ref could be chosen among some of the compromised samples, its construction ensures that it inherits the accuracy of the majority of the responses.
Proposition 1 (Robust reference estimator) .
Given a sample { Y 1 , . . .
, Y m } where Y i ∈
O d,r and |{ i ∈ [ m ] | dist( Y i , V ) ≤ ε }| > m / 2 for a fixed ε > 0 , Algorithm 2 outputs
Y i ⋆ satisfying dist( Y i ⋆ , V ) ≤ 3 ε.
2.3 The ProcrustesFixing algorithm
In this section, we formally introduce the Procrustes-fixing procedure and show that it properly aligns all the non-compromised responses given the reference solution described in Section 2.2.
The procedure is described in Algorithm 3; it accepts a set of d × r matrices with orthonormal columns as well as a reference matrix
Y ref of the same shape.
The work [10] provides an error bound for the ProcrustesFixing algorithm under idealized conditions; namely, that the reference solution is equal to the ground truth V .
for i = 1 , . . .
, m do (cid:101) Y i :
=
Y i Z i , where Z i :=
argmin Z ∈
O r ∥
Y i Z − Y ref ∥ F ▷ Procrustes alignment return (cid:8)(cid:101) Y i | i ∈
[ m ] (cid:9)
Theorem 2 (Theorem 2 in [10]) .
Let Assumption 2 hold and let
[Equation]
i , where
[Equation]
Then the following bound holds:
[Equation]
While the setting of Theorem 2 is idealized, when the reference chosen by Algorithm 2 is sufficiently close to V one would expect that the aligned estimates are not far from their ideal version.
The next
Lemma shows that aligning the local solutions with (cid:98) V ref is equivalent to aligning with the ground truth V , up to higher-order errors.
Lemma 1.
Let V i ∈
O d,r span the principal r -dimensional eigenspace of the matrix
A i and let
V ∈ O d,r span the principal r -dimensional invariant subspace of A .
Suppose that there is a
V ref ∈ O d,r satisfying dist( V ref , V ) =
ε < δ r ( A ) / 8 , and define the sets of aligned estimates
[Equation]
Then for any i ∈ I good the following holds:
[Equation]
Putting everything together, we arrive at a deterministic characterization of the error attained by the empirical average over any subset of responses that come from non-compromised nodes and have been aligned with the robust reference estimator.
Note that this characterization does not immediately translate to an algorithm, since the set of compromised nodes is not known a-priori.
Proposition 2 (Error of clean samples) .
Let (cid:98) V ref be the output of Algorithm 2 given inputs (cid:98) V 1 , . . .
, (cid:98) V m .
For any index set S ⊂ I good and i ∈ S , define
[Equation]
Suppose that dist( V, (cid:98) V ref ) =
ε < δ r ( A ) / 8 .
Then the following bound holds:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) .
Proof.
From the triangle inequality, Lemma 1 and Theorem 2 it follows that
[Equation]
(cid:88)
[Equation]
2.4 Analysis of robust mean estimation
We now analyze the last phase of the algorithm, which approximates V using the robust mean of the aligned samples.
The mean estimation procedure used is the randomized iterative filtering method shown in Algorithm 4, which is an appropriate modification of the spectral filtering algorithm from the robust statistics literature for matrix-valued inputs.
Its analysis is available in the Appendix, while
Theorem 3 below summarizes its guarantees.
Algorithm 4 Filter ( S :=
{ X i } i =1 ,...,m , λ ub )
Compute empirical mean and covariance:
[Equation]
Compute leading eigenpair ( λ, v ) of Σ S .
if λ < 18 λ ub then return θ S else
Compute outlier scores τ i := v T
X i − θ S )
( X i − θ S ) T v for i ∈ S .
Sample Z from S following P ( Z
= X i ) =
[Equation]
return Filter ( S \ { Z } , λ ub )
Theorem 3.
Suppose G 0 ⊂ [ m ] and that the corruption level α and failure probability p satisfy α + 6 log(1 /p ) m ≤ and | G 0 | ≥ (1 − α ) m.
Moreover, let Σ G 0 :
= 1 | G 0 | (cid:80) i ∈ G
( X i − θ G 0 )
( X i − θ G 0 ) T , where θ G 0 :
= 1 | G 0 | (cid:80) i ∈ G 0 X i , and fix λ ub ≥ ∥ Σ G 0 ∥ 2 .
Then Algorithm 4 invoked with { X i } mi =1 and λ ub returns an estimate θ λ ub satisfying
P (cid:32)(cid:13)(cid:13)(cid:13) θ λ ub − 1 | G 0 | (cid:88) i ∈ G
X i (cid:13)(cid:13)(cid:13)
[Equation]
Remark 3.
The upper bound α < 1 / 12 appears to be a proof artifact and could likely be improved by optimizing choices of constants in the proof of Theorem 3.
Numerical evidence in Section 3. suggests that the breakdown point of Algorithm 4 is closer to the natural limit of 1 / 2 .
The error in (17) scales with the upper bound λ ub , which may be far from the “optimal” ∥ Σ G
∥ 2 .
We describe an adaptive version of Algorithm 4 that achieves this at a logarithmic additional cost.
Indeed, suppose an upper bound on α is available and the unknown parameter ∥ Σ G 0 ∥ 2 lies in an interval [ λ lb , λ ub ] .
We construct a search grid G as follows:
[Equation]
We are now in good shape to describe our estimator.
To simplify notation, we define the error proxy
[Equation]
Our estimator, θ ˆ λ , is implemented in Alg. 5 and defined as: θ ˆ λ , where ˆ λ :=
min { λ ∈ G | ∥ θ
λ − θ λ ′ ∥
[Equation]
f
[Equation]
p
[Equation]
∀
[Equation]
G
[Equation] [Equation]
Algorithm
[Equation]
Set up search grid: j lo :=
⌊ log 2 λ lb ⌋ , j hi :=
⌈ log 2 λ ub ⌉ .
for j = j hi , . . .
, j lo do θ 2 j = Filter ( S, 2 j )
▷ Algorithmif ∃ k > j such that ∥ θ 2 j − θ 2 k ∥ > f (2 j ; p, α ) + f (2 k ; p, α ) then ▷ f defined in (19) return θ 2 j + return θ 2 j lo
If ∥ Σ G 0 ∥ 2 ∈ [ λ lb , λ ub ] , the estimator attains the optimal error up to a constant while the success probability degrades only logarithmically, as shown by the following Proposition.
Proposition 3.
If G 0 , p , and α satify (16) and ∥ Σ G 0 ∥ 2 ≤ λ ub , the estimator θ ˆ λ from (20) satisfies
[Equation]
with probability at least 1 − 2 log 2 ( λ ub / λ lb )
p.
2.5 Proof sketch of main theorem
We briefly sketch the proof of Theorem 1 here.
We decompose
[Equation]
The error ∆ 1 can be directly controlled by applying Proposition 3 with G 0 ≡ I good combined with the fact that the spectral norm of the matrix Σ I good admits the upper bound in (21); we refer the reader to the supplement for a complete derivation.
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) |I
[Equation]
| (cid:88)
i
[Equation]
good (cid:101)
[Equation]
i
[Equation]
V (cid:13)(cid:13)(cid:13)(cid:13) (21)
Finally, we control the error ∆ 2 by invoking Proposition 2 with S
=
I good , since
[Equation]
Combining the resulting upper bounds yields the error in Theorem 1.
Remark 4.
Both of the terms in (21) are typically small and can be directly controlled in concrete applications such as distributed PCA.
Moreover, while the bound in (21) is not directly computable, it is immediate that ∥ Σ I good ∥ 2 ≲ 1 and thus we may initialize Algorithm 5 with λ ub
= O (1) in the absence of a finer upper bound.
Robust distributed PCA
In this section, we specialize the results of Section 2 to robust distributed PCA for subgaussian distributions.
We first formalize the sampling model for the problem.
Assumption 3 (Subgaussian data) .
Every machine i ∈ I good draws { X ( i ) j } nj =1 iid ∼ P , where P is a zero-mean, subgaussian distribution with covariance matrix A :=
E X ∼P [ XX T ] , and forms
A i := 1 n (cid:80) nj =1 X ( i ) j
( X ( i ) j ) T .
Our main theorem follows directly from Theorem 1 and control of ∥ Σ I good ∥ 2 under Assumption 3.
Theorem 4.
Let Assumptions 1 to 3 hold and suppose that n ≳ κ 2 · ( r ⋆ + log( mn/p )) and α , m and p satisfy (9) .
Then Algorithm 1 initialized with ω = (cid:112) 1 / mn returns a ¯ V satisfying dist( ¯ V , V ) ≲
[Equation]
with probability at least 1 − 2 / n − p .
Here, r ⋆ :=
Tr ( A ) / ∥
A ∥ 2 and ϱ is given by
[Equation]
When κ ≍ 1 , high-order terms in Theorem 4 can be discarded and we arrive at the following:
Corollary 1.
Assume that the conditions of Theorem 4 hold and κ ≍ 1 .
Then:
[Equation]
with probability at least 1 − 2 / n − p .
We briefly compare the error of Algorithm 1 to that of its non-robust counterpart from [10] when κ ≍ 1 .
The latter algorithm returns an estimate ¯ V nonrobust satisfying dist( ¯ V nonrobust , V ) =
˜ O (cid:0)(cid:112) r ⋆ / mn (cid:1) .
Ignoring the √ r factors, which are likely an artifact of our proof, our algorithm also introduces an additive error of the order ˜ O ( (cid:112) α /
1 − α · ( r ⋆ r / mn ) 1 / 4 ) .
Note that for a constant absolute number of corruptions α ∝ 1 / m , this additive factor scales as
[Equation]
If m and n are comparable, this is similar to the error of the non-robust algorithm up to an ( r/r ⋆ ) 1 /factor.
Therefore, the performance of Algorithm 1 degrades gracefully as a function of the corruption level under not too restrictive assumptions on the ratio m/n .
3.1 Numerical study
We provide a brief numerical illustration of the performance of Algorithm 1 on data sampled from an unknown Gaussian distribution D := N (0 , V Λ V T + V ⊥ Λ ⊥ V T ⊥ ) , where [ V
V ⊥ ] ∈ O
d is a random d × d orthogonal matrix and Λ , Λ ⊥ are generated according to the following model:
[Equation]
where η
[Equation]
We simulate an adversary by replacing the first ⌊ αm ⌋ responses by the same V adv ∈
O d,r , chosen to be near-orthogonal to V .
We fix the gap δ = 0 .
25 throughout.
We compare Alg.
1 (labelled Robust in our plots) against two baselines: the algorithm from [10] (labelled Naive ), which corresponds to Alg.
3 using the first response – which is always corrupted in our experiment – as the reference followed by naive averaging; and a version of Alg.
1 without the robust mean estimation step (labelled
Procrustes ).
Our implementation always removes the sample with the largest outlier score in each step of Alg. 4 and uses a simplified error proxy f ( λ ; α ) :
= √ λα instead of (19) in Alg.
Our experiment is illustrated in Figure 1.
Clearly, the baseline methods break down in the presence of corruption, yielding solutions nearly orthogonal to V as α approaches 1 / 2 .
In contrast, the error of Alg.
1 degrades gracefully with α .
We note that our algorithm yields a nontrivial solution even when almost half of the measurements are corrupted ( α = 45% ), in line with intuition suggesting that α ⋆ = 1 / 2 is a natural breakdown point for outlier-robust algorithms.
[Figure]
We report mean distance ± one standard deviation runs subspace
In addition, we study the effect of the number of per-machine samples n and total number of machines m in distributed PCA.
We fix r = 5 , r ⋆ = 2 r , κ = 5 and δ = 0 .
25 and experiment with ( m, n ) = (cid:8) (32 , 2 i ) | i = 5 , . .
, 10 (cid:9) , ( m, n ) = (cid:8) (2 i , 128) | i = 4 , . .
, 9 (cid:9) , α ∈ { 0 . 25 , 0 . 45 } .
The results can be found in Fig.
and are consistent with our theory: for fixed m , a small number of samples per machine n leads to inaccurate local solutions and inaccurate final estimates for all variants, while increasing n leads to a decrease in error for all configurations.
On the other hand, the robust algorithm produces good estimates for all attempted values of m after fixing a sufficiently large sample size n , and increasing m only decreases the error for the robust variant.
Finally, we observe that the error of the non-robust variants plateaus since the corruption fraction α is bounded away from zero.
[Figure]
Discussion
We presented a communication-efficient algorithm for distributed eigenspace estimation that is robust to α = 1 / 12 fraction of nodes returning structurally valid but otherwise potentially adversarial responses.
Numerical evidence suggests its breakdown point is closer to the (optimal) α ∗ = 1 / 2 , which might be achievable by an improved analysis of the filtering procedure in Alg 4.
Our adaptive version of the filtering procedure in Alg.
5 trades off knowledge of (an upper bound on) the corruption level α with the need for a tight bound on ∥ Σ
I good ∥ 2 .
Alternatively, one can design a version of Algorithmthat is adaptive to the corruption level α using a similar construction that evaluates the error proxy f
( λ ; p, α ) for different values of α and fixed λ ≈ ∥
Σ I good ∥ 2 instead.
Acknowledgements
We thank Jayadev Acharya and Damek Davis for their insightful comments and suggestions.
We also thank the anonymous reviewers for invaluable feedback that greatly improved the manuscript.
[-2 U T ⊥ − A 2 ∥ 2 ≤ 2 ∥ Σ 2 ∥ 2 ε + 4 ∥ Σ 2 ∥ 
-]
[-Let j crit be the smallest index for which 2 j ≥ max { λ lb , ∥ Σ G 
-]
[-We recall some notation used therein: given the set of “good” samples G 0 and the initial sample S 
-]