Communication-efficient distributed eigenspace estimation with arbitrary node failures
Abstract
We develop an eigenspace estimation algorithm for distributed environments with arbitrary node failures, where a subset of computing nodes can return structurally valid but otherwise arbitrarily chosen responses.
Notably, this setting encompasses several important scenarios that arise in distributed computing and data-collection environments such as silent/soft errors, outliers or corrupted data at certain nodes, and adversarial responses.
Our estimator builds upon and matches the performance of a recently proposed non-robust estimator up to an additive ˜ O ( σ √ α ) error, where σ 2 is the variance of the existing estimator and α is the fraction of corrupted nodes
Problem overview and background
Modern machine learning has seen the proliferation of heterogeneous distributed environments for training and deploying data science pipelines.
As communication between machines is often the most time-consuming operation in distributed systems, the design of communication-efficient algorithms is of paramount importance for scaling to massive datasets [41].
However, the move to distributed environments also adds several additional layers of complexity in the design of algorithms.
For example, in the distributed setting we would like our algorithms to be robust and providing meaningful answers even in settings where some nodes contain outlier data [4], silently fail during the computation [30, 35], or are compromised and returning malicious results designed to corrupt the central solution.
This work focuses on distributed eigenspace estimation in the context of robustness to node-level corruptions.
Formally, we assume a computing environment with nodes numbered i = 1 , . . .
, m , where every node i observes a local version
A i of an unknown symmetric matrix A ∈ R d × d ; the goal is to approximate the subspace spanned by the r ≪ d principal eigenvectors of A .
Distributed PCA is a standard example in this framework: every machine draws n i.i.d.
samples from an unknown distribution P with covariance matrix A and forms a local empirical covariance matrix A i .
Recently proposed communication-efficient algorithms have every node i transmit V i , the d × r matrix of principal eigenvectors of A i , to a central server, which then aggregates all the local solutions via a carefully-crafted aggregation procedure [10, 20].
We devise and analyze an algorithm that is robust to a wide range of corruptions that can occur to a subset of the computational nodes.
In particular, we assume that some fraction α of the computational nodes can respond with completely arbitrary, but structurally valid, responses (i.e., they return arbitrary matrices
V i with orthonormal columns).
This model encompasses three common forms of node-level corruption that cannot be easily detected by the central machine in isolation:
Silent/soft errors. While computational errors may be rare on single machines, as distributed workloads span large numbers of nodes the probability that some of them fail becomes significant.
Though catastrophic failures may be detectable, allowing the central server to simply ignore the output of specific nodes, the more nefarious issue is that of so-called silent (or soft) errors [18, 21, 30].
More specifically, a silent error is one where a node returns an erroneous but structurally valid response to the central machine query.
Because the response is structurally valid and the central machine may not have access to the per-node data it is not possible to “validate” the response of each node and, instead, the central estimator must be adapted to be robust to such errors.
Outliers or corrupted data. In certain settings the data collection may be distributed in addition to the computation.
If some of the nodes are drawing samples from an invalid or corrupted data source they may introduce gross outliers to the set of responses { V i | i ∈ [ m ] } .
Similarly, in the distributed PCA example, while most machines draw a sufficient number of samples, a minority of them may have only a small amount of data available such that the principal eigenspaces of the local empirical covariance matrices are too far from the ground truth, and thus violate standard modelling assumptions in distributed learning.
Again, robustness to such outlier responses must be a feature of the estimator since they cannot be detected by individual nodes (as they do not have information about the global problem).
Adversarial responses. In some settings, a subset of nodes may be compromised by an adversary who wishes to influence the central solution by crafting and returning malicious V i .
In fact, the adversarial nodes may be collaborating when constructing their responses.
Since the central node does not get to see all the data it cannot validate responses or directly detect adversaries.
Therefore, the estimator itself must be adapted to be robust to collections of responses designed to push the solution in specific directions.
The main contribution of our paper is a communication-efficient algorithm that is robust to node corruptions (as outlined above) for the distributed eigenspace estimation problem.
We note that our corruption model is similar to so-called Byzantine failures [28] in distributed systems.
1.1 Related work
Distributed eigenspace estimation.
The problem of distributed eigenspace estimation has been well-studied in the absence of malicious noise.
One of the challenges in the distributed setting is aggregating local solutions in the presence of symmetry: for example, if v is an eigenvector of A , both ± v are valid solutions to our problem.
Various works deal with such symmetries in different ways; in the algorithms of [5, 20], the central node averages the spectral projectors of the local eigenspaces, and performs an eigendecomposition of the resulting average to approximate the principal eigenspace.
This approach is similar to the algorithms of [3, 11, 29], although the latter works focus on distributed low-rank approximations and do not address the issue of approximating the principal eigenspace directly.
Another standard approach is for the central server to aggregate local solutions after an alignment step designed to remove the orthogonal ambiguity [10, 19, 23] (see also [7] for the nondistributed setting).
Indeed, our work builds on the two-stage algorithm presented and analyzed in [10] for the non-robust setting.
Finally, we briefly mention a recent line of work [12, 23] that adapts the shift-and-invert preconditioning framework [22] to the distributed setting; however, the latter approach leads to algorithms that require multiple rounds of communication.
Robust PCA.
The literature contains a number of different formulations for robust principal component analysis.
The seminal work of Candés et al.
[9] formulated robust PCA as the task of separating an observed matrix
Y ∈ R
d × d into a low-rank and a sparse component – a slightly different problem from that considered in this paper.
Xu et al.
[39] considered the problem of approximating a low-dimensional distribution from a set of n i.i.d.
samples, a constant fraction of which have been individually corrupted by gross outliers.
Follow-up works focused on high-dimensional sparse estimation and applications to sparse robust PCA [2, 16].
However, communication-efficient robust
PCA appears to be overlooked.
A notable exception is the sketch-based algorithm of [34], although the assumptions therein depart significantly from our setting.
Another relevant line of work is that of Byzantine-robust distributed learning (typically focusing on distributed gradient descent); see, e.g., [1, 6, 8, 13, 15, 26, 32, 40] as well as the survey [25].
In these works, an iterative algorithm is distributed across machines that send individual updates to a central server, which combines them using a robust aggregation procedure (e.g., the geometric median [32]).
Such works are more general in scope, but typically lead to estimators that require multiple rounds of communication.
For minimization of strongly convex quadratics, [40] proposes a one-shot algorithm employing a robust aggregation scheme. However, this special case departs from our setting as their underlying problem is convex and solved under the assumption that solutions are unique.
1.2 Notation
We let S d − 1 denote the unit sphere in d dimensions.
We write ∥ A ∥ F :=
(cid:112) ⟨ A, A ⟩ and ∥
A ∥ 2 := sup x ∈ S d − 1 ∥
Ax ∥ 2 for the Frobenius and spectral norms of a matrix A ∈ R n × d .
We write O n,r for the set of n × r matrices with orthonormal columns and O r ≡ O r,r .
Given U, V ∈ O n,r we write
[-= (cid:13)(cid:13) ( I − UU T ) V (cid:13)(cid:13) 
 = (cid:13)(cid:13)-][Equation]
) for their ℓ 2 → ℓ 2 subspace distance and B dist ( U ; r ) for the scaled unit ball centered at U :
[Equation]
Finally, we use the notation A ≲ B to indicate that A ≤ cB for a dimension-independent constant c > 0 and A ≍ B if A ≲ B and B ≲ A simultaneously.
Robust distributed eigenspace estimation
We now formally introduce the problem setting.
In particular, we assume there exists an unknown symmetric matrix A with spectral decomposition
[Equation]
assuming a nonincreasing ordering on the eigenvalues:
[Equation]
Our goal is to approximate the principal r -dimensional eigenspace V :=
span( V ) of A given m machines, each of which observes a local version A i of A , communicating with a central coordinator.
We assume that m is even for simplicity.
When queried for a response, machine i responds either with an eigenvector matrix spanning the principal eigenspace of the local matrix
A i , or with an arbitrary d × r matrix with orthonormal columns.
The latter case corresponds to so-called compromised machines.
In contrast, prior work [10, 20] assumes that every machine responds truthfully.
Assumption 1 (Corruption model) .
There exists a constant α ∈ (0 , 1 / 2 ) and an index set I bad ⊂ [ m ] with |I bad | /m ≤ α such that the following holds: all nodes i / ∈ I bad observe a symmetric matrix
A i ∈ R d × d .
Moreover, when queried for a response, every node i returns
[Equation]
where the columns of V i ∈ O d,r span the principal r -dimensional eigenspace of A i and Q i ∈
O d,r is an arbitrary d × r matrix with orthonormal columns.
For notational convenience, we also define the set of “good” responses:
[Equation]
Furthermore, we require the principal eigenspace of A to be sufficiently separated from its complement and that the local errors E i :
=
A i − A are not too large.
Assumption 2.
There is a constant δ > 0 such that the following hold:
(Gap)
The matrix A has a nontrivial eigengap:
[Equation]
(Approximation)
For all i ∈ I good , the local observations satisfy:
[Equation]
The problem admits a natural proxy of difficulty in the form of the normalized inverse eigengap κ : [Equation] Normalized
[Equation]
eigengap [Equation] Our algorithm for the robust distributed eigenspace estimation problem is outlined in Algorithm 1, which is essentially a robust version of the Procrustes fixing algorithm from [10].
The latter (nonrobust) algorithm operates as follows: first, every machine i computes its local eigenvector matrix V i and broadcasts it to the central server.
Because invariant subspaces do not admit unique representations, naively averaging these estimates can fail to reduce the approximation error further.
Instead, the algorithm of [10] first picks one of the local solutions (say V 1 ) as a reference and “aligns” every other solution with it by solving a so-called Procrustes problem :
[Equation]
After the alignment step (8), the solution of which is available in closed form via the SVD [24], the central coordinator computes and returns the empirical average (1 /m ) (cid:80) mi
=1 V i Z i .
To robustify the algorithm described above against node failures, we need the following ingredients:
Reference estimation.
In the presence of corruptions one must guard against the possibility of choosing an outlier as a reference solution (which would render the alignment step (8) useless).
The first step of our algorithm robustly determines a reference guaranteed to have nontrivial alignment with the ground truth (Algorithm 2).
Solution aggregation.
With the robust reference at hand, the next step of the algorithm aligns other local solutions with it.
However, since some of the solutions are outliers, we use a robust mean estimation algorithm in the last step of Algorithm 1 to compute the empirical average only over inliers (and possible “benign” outliers) with high probability. Combining the above ingredients requires working around additional technical challenges.
The first challenge is showing that averaging over inliers that have been aligned with (cid:98) V ref is equivalent (up to small error) to averaging over inliers that have been aligned with the ground truth V .
To do so, we leverage a path-independence result from [37] to view the aligned estimates as the eigenvectors arising from a carefully chosen sequence of perturbations to the unknown matrix A (Lemma 1). After aligning local estimates, we approximate the empirical mean over inliers with a spectral filtering algorithm. Existing analyses of that algorithm focus on vector-valued inputs and lead to error bounds in the Euclidean norm (or other vector norms, see [36]).
Invoking these error bounds in a black-box fashion for matrix-valued inputs (by flattening every eigenvector matrix to a ( dr ) -dimensional vector) has multiple drawbacks.
On one hand, it leads to unnatural scaling as the final result depends on the empirical covariance over flattened inputs.
On the other hand, the resulting error bounds are with respect to the Frobenius norm, while the spectral norm is a standard error measure in eigenspace estimation.
To avoid this, we modify the iterative filtering algorithm so that inputs are not flattened and obtain error bounds with respect to the spectral norm (3).
We analyze each ingredient of Algorithm 1 separately, in Sections 2.2 to 2.4; all proofs appear in the supplementary material.
Our analysis is almost completely deterministic: indeed, the only source of randomness is the filtering algorithm used in the final stage (Algorithm 5). Algorithm
Input : responses (cid:8)(cid:98) V i (cid:9) i =1 ,...,m , corruption fraction α , failure prob.
p , error parameter ω .
(cid:98) V ref :=
RobustReferenceEstimator (cid:0)(cid:98) V 1 , . .
, (cid:98) V m (cid:1) .
▷ Algorithm 2; Section 2.
(cid:8)(cid:101) V i (cid:9) i =1 ,...,m :=
ProcrustesFixing (cid:0)(cid:8)(cid:98) V 1 , . .
, (cid:98) V m (cid:9) , (cid:98) V ref (cid:1)
▷ Algorithm 3; Section 2.
¯ V :=
AdaptiveFilter ( (cid:8)(cid:101) V 1 , . . .
, (cid:101) V m (cid:9) , 6 , ω, p, α )
▷ Algorithm 5; Section 2.
return ¯ V
Our main theorem on the performance of Algorithm 1 follows below. The result is first stated for inputs from a generic eigenspace estimation problem.
In Section 3, we apply it in a black-box fashion to specialize its results for distributed PCA.
Theorem 1.
Let Assumptions 1 and 2 hold and suppose that the corruption level α satisfies
[Equation]
For a fixed failure probability p and error parameter ω
[Equation]
Algorithm 1 returns ¯ V ∈ R d × r satisfying:
[Equation]
with probability at least 1 − 2 log ( 6 / ω ) · p .
Moreover [Equation] the variance σ 2 satisfies
[Equation]
Remark 1 (Error bound interpretation) . The error bound (10) decomposes naturally into three terms that can be controlled individually in applications. The first term, E oracle , corresponds to an oracle estimator that approximates V via the principal eigenspace of 1 / |I good | (cid:80) i ∈I
good A i .
The second term, E high , represents high-order errors that occur as a result of the alignment step in Algorithm 3 that are dominated by E oracle in “typical” situations.
Finally, the term E robust quantifies the effect of outliers, vanishing as the fraction of corrupted nodes α ↓ 0 .
We comment on the scaling of E robust relative to the error of the non-robust algorithm in the context of distributed PCA in Section 3.
2.2 The robust reference estimator
This section focuses on the analysis of Algorithm 2, which yields the robust reference estimator (cid:98) V ref used to remove the orthogonal ambiguity from local solutions.
We note that the construction of the estimator dates back to the seminal work of Nemirovski and Yudin [31].
Algorithm 2 RobustReferenceEstimator ( Y 1 , . . . , Y m ) for i = 1 , . .
, m do ε i :=
min { r ≥ 0 | |B dist ( Y i ; r ) ∩ { Y i } mi =1 | > m 2 } return Y i ⋆ , where i ⋆ := argmin i ∈ [ m ] ε i
Remark 2.
The quantities ε i in Algorithm 2 can be found in time O ( m 2 dr 2 ) by first computing r j := dist ( Y i , Y j ) for all j ̸
=
i and setting ε i :=
median ( { r j } j ̸ = i ) .
Note that even though (cid:98) V ref could be chosen among some of the compromised samples, its construction ensures that it inherits the accuracy of the majority of the responses.
Proposition 1 (Robust reference estimator) .
Given a sample { Y 1 , . . .
, Y m } where Y i ∈
O d,r and |{ i ∈ [ m ] | dist( Y i , V ) ≤ ε }| > m / 2 for a fixed ε > 0 , Algorithm 2 outputs
Y i ⋆ satisfying dist( Y i ⋆ , V ) ≤ 3 ε.
2.3 The ProcrustesFixing algorithm
In this section, we formally introduce the Procrustes-fixing procedure and show that it properly aligns all the non-compromised responses given the reference solution described in Section 2.2.
The procedure is described in Algorithm 3; it accepts a set of d × r matrices with orthonormal columns as well as a reference matrix Y ref of the same shape. The work [10] provides an error bound for the ProcrustesFixing algorithm under idealized conditions; namely, that the reference solution is equal to the ground truth V .
for i = 1 , . . .
, m do (cid:101) Y i :
=
Y i Z i , where Z i :=
argmin Z ∈
O r ∥
Y i Z − Y ref ∥ F ▷ Procrustes alignment return (cid:8)(cid:101) Y i | i ∈ [ m ] (cid:9) Theorem 2 (Theorem 2 in [10]) .
Let Assumption 2 hold and let
[Equation] i , where [Equation]
Then the following bound holds:
[Equation]
While the setting of Theorem 2 is idealized, when the reference chosen by Algorithm 2 is sufficiently close to V one would expect that the aligned estimates are not far from their ideal version.
The next
Lemma shows that aligning the local solutions with (cid:98) V ref is equivalent to aligning with the ground truth V , up to higher-order errors. Lemma 1. Let V i ∈ O d,r span the principal r -dimensional eigenspace of the matrix
A i and let
V ∈ O d,r span the principal r -dimensional invariant subspace of A .
Suppose that there is a
V ref ∈ O d,r satisfying dist( V ref , V ) =
ε < δ r ( A ) / 8 , and define the sets of aligned estimates [Equation] Then for any i ∈ I good the following holds: [Equation]
Putting everything together, we arrive at a deterministic characterization of the error attained by the empirical average over any subset of responses that come from non-compromised nodes and have been aligned with the robust reference estimator.
Note that this characterization does not immediately translate to an algorithm, since the set of compromised nodes is not known a-priori.
Proposition 2 (Error of clean samples) . Let (cid:98) V ref be the output of Algorithm 2 given inputs (cid:98) V 1 , . . . , (cid:98) V m . For any index set S ⊂ I good and i ∈ S , define
[Equation]
Suppose that dist( V, (cid:98) V ref ) =
ε < δ r ( A ) / 8 .
Then the following bound holds:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) .
Proof.
From the triangle inequality, Lemma 1 and Theorem 2 it follows that
[Equation]
(cid:88)
[Equation]
2.4 Analysis of robust mean estimation
We now analyze the last phase of the algorithm, which approximates V using the robust mean of the aligned samples. The mean estimation procedure used is the randomized iterative filtering method shown in Algorithm 4, which is an appropriate modification of the spectral filtering algorithm from the robust statistics literature for matrix-valued inputs. Its analysis is available in the Appendix, while Theorem 3 below summarizes its guarantees.
Algorithm 4 Filter ( S :=
{ X i } i =1 ,...,m , λ ub )
Compute empirical mean and covariance:
[Equation]
Compute leading eigenpair ( λ, v ) of Σ S .
if λ < 18 λ ub then return θ S else
Compute outlier scores τ i := v T
X i − θ S )
( X i − θ S ) T v for i ∈ S .
Sample Z from S following P ( Z
= X i ) =
[Equation]
return Filter ( S \ { Z } , λ ub )
Theorem 3.
Suppose G 0 ⊂ [ m ] and that the corruption level α and failure probability p satisfy α + 6 log(1 /p ) m ≤ and | G 0 | ≥ (1 − α ) m.
Moreover, let Σ G 0 : = 1 | G 0 | (cid:80) i {+∈ G 
 ( X i − θ G 0 ) ( X i − θ G 0 ) T+} , where θ G 0 :
= 1 | G 0 | (cid:80) i ∈ G 0 X i , and fix λ ub ≥ ∥ Σ G 0 ∥ 2 .
Then Algorithm 4 invoked with { X i } mi =1 and λ ub returns an estimate θ λ ub satisfying
P (cid:32)(cid:13)(cid:13)(cid:13) θ λ ub − 1 | G 0 | (cid:88) i ∈ G
X i (cid:13)(cid:13)(cid:13)
[Equation]
Remark 3.
The upper bound α < 1 / 12 appears to be a proof artifact and could likely be improved by optimizing choices of constants in the proof of Theorem 3.
Numerical evidence in Section 3. suggests that the breakdown point of Algorithm 4 is closer to the natural limit of 1 / 2 .
{+The error in (17) scales with the upper bound λ ub , which may be far from the “optimal” ∥ Σ G 
+}
∥ 2 .
We describe an adaptive version of Algorithm 4 that achieves this at a logarithmic additional cost.
Indeed, suppose an upper bound on α is available and the unknown parameter ∥ Σ G 0 ∥ 2 lies in an interval [ λ lb , λ ub ] .
We construct a search grid G as follows:
[Equation]
We are now in good shape to describe our estimator.
To simplify notation, we define the error proxy
[Equation]
Our estimator, θ ˆ λ , is implemented in Alg. 5 and defined as: θ ˆ λ , where ˆ λ :=
min { λ ∈ G | ∥ θ
λ − θ λ ′ ∥
[Equation]
f
[Equation]
p
[Equation]
∀
[Equation]
G
[Equation]
[Equation]
Algorithm
[Equation]
Set up search grid: j lo :=
⌊ log 2 λ lb ⌋ , j hi :=
⌈ log 2 λ ub ⌉ .
for j = j hi , . . .
, j lo do θ 2 j = Filter ( S, 2 j )
▷ Algorithmif ∃ k > j such that ∥ θ 2 j − θ 2 k ∥ > f (2 j ; p, α ) + f (2 k ; p, α ) then ▷ f defined in (19) return θ 2 j + return θ 2 j lo
If ∥ Σ G 0 ∥ 2 ∈ [ λ lb , λ ub ] , the estimator attains the optimal error up to a constant while the success probability degrades only logarithmically, as shown by the following Proposition.
Proposition 3.
If G 0 , p , and α satify (16) and ∥ Σ G 0 ∥ 2 ≤ λ ub , the estimator θ ˆ λ from (20) satisfies
[Equation]
with probability at least 1 − 2 log 2 ( λ ub / λ lb )
p.
2.5 Proof sketch of main theorem
We briefly sketch the proof of Theorem 1 here.
We decompose
[Equation]
The error ∆ 1 can be directly controlled by applying Proposition 3 with G 0 ≡ I good combined with the fact that the spectral norm of the matrix Σ I good admits the upper bound in (21); we refer the reader to the supplement for a complete derivation.
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) |I
[Equation]
| (cid:88)
i
[Equation]
good (cid:101)
[Equation]
i
[Equation]
V (cid:13)(cid:13)(cid:13)(cid:13) (21)
Finally, we control the error ∆ 2 by invoking Proposition 2 with S
=
I good , since
[Equation]
Combining the resulting upper bounds yields the error in Theorem 1.
Remark 4.
Both of the terms in (21) are typically small and can be directly controlled in concrete applications such as distributed PCA.
Moreover, while the bound in (21) is not directly computable, it is immediate that ∥ Σ I good ∥ 2 ≲ 1 and thus we may initialize Algorithm 5 with λ ub
= O (1) in the absence of a finer upper bound.
Robust distributed PCA
In this section, we specialize the results of Section 2 to robust distributed PCA for subgaussian distributions.
We first formalize the sampling model for the problem.
Assumption 3 (Subgaussian data) .
Every machine i ∈ I good draws { X ( i ) j } nj =1 iid ∼ P , where P is a zero-mean, subgaussian distribution with covariance matrix A :=
E X ∼P [ XX T ] , and forms
A i := 1 n (cid:80) nj =1 X ( i ) j
( X ( i ) j ) T .
Our main theorem follows directly from Theorem 1 and control of ∥ Σ I good ∥ 2 under Assumption 3.
Theorem 4.
Let Assumptions 1 to 3 hold and suppose that n ≳ κ 2 · ( r ⋆ + log( mn/p )) and α , m and p satisfy (9) .
Then Algorithm 1 initialized with ω = (cid:112) 1 / mn returns a ¯ V satisfying dist( ¯ V , V ) ≲
[Equation]
with probability at least 1 − 2 / n − p .
Here, r ⋆ :=
Tr ( A ) / ∥
A ∥ 2 and ϱ is given by
[Equation]
When κ ≍ 1 , high-order terms in Theorem 4 can be discarded and we arrive at the following:
Corollary 1.
Assume that the conditions of Theorem 4 hold and κ ≍ 1 .
Then:
[Equation]
with probability at least 1 − 2 / n − p .
We briefly compare the error of Algorithm 1 to that of its non-robust counterpart from [10] when κ ≍ 1 .
The latter algorithm returns an estimate ¯ V nonrobust satisfying dist( ¯ V nonrobust , V ) =
˜ O (cid:0)(cid:112) r ⋆ / mn (cid:1) .
Ignoring the √ r factors, which are likely an artifact of our proof, our algorithm also introduces an additive error of the order ˜ O ( (cid:112) α /
1 − α · ( r ⋆ r / mn ) 1 / 4 ) .
Note that for a constant absolute number of corruptions α ∝ 1 / m , this additive factor scales as
[Equation]
If m and n are comparable, this is similar to the error of the non-robust algorithm up to an ( r/r ⋆ ) 1 /factor.
Therefore, the performance of Algorithm 1 degrades gracefully as a function of the corruption level under not too restrictive assumptions on the ratio m/n .
3.1 Numerical study
We provide a brief numerical illustration of the performance of Algorithm 1 on data sampled from an unknown Gaussian distribution D := N (0 , V Λ V T + V ⊥ Λ ⊥ V T ⊥ ) , where [ V
V ⊥ ] ∈ O
d is a random d × d orthogonal matrix and Λ , Λ ⊥ are generated according to the following model:
[Equation]
where η [Equation] We simulate an adversary by replacing the first ⌊ αm ⌋ responses by the same V adv ∈ O d,r , chosen to be near-orthogonal to V .
We fix the gap δ = 0 . 25 throughout. We compare Alg. 1 (labelled Robust in our plots) against two baselines: the algorithm from [10] (labelled Naive ), which corresponds to Alg.
3 using the first response – which is always corrupted in our experiment – as the reference followed by naive averaging; and a version of Alg.
1 without the robust mean estimation step (labelled
Procrustes ).
Our implementation always removes the sample with the largest outlier score in each step of Alg. 4 and uses a simplified error proxy f ( λ ; α ) :
= √ λα instead of (19) in Alg.
Our experiment is illustrated in Figure 1.
Clearly, the baseline methods break down in the presence of corruption, yielding solutions nearly orthogonal to V as α approaches 1 / 2 .
In contrast, the error of Alg.
1 degrades gracefully with α .
We note that our algorithm yields a nontrivial solution even when almost half of the measurements are corrupted ( α = 45% ), in line with intuition suggesting that α ⋆ = 1 / 2 is a natural breakdown point for outlier-robust algorithms.
[Figure]
We report mean distance ± one standard deviation runs subspace
In addition, we study the effect of the number of per-machine samples n and total number of machines m in distributed PCA.
We fix r = 5 , r ⋆ = 2 r , κ = 5 and δ = 0 .
25 and experiment with ( m, n ) = (cid:8) (32 , 2 i ) | i = 5 , . .
, 10 (cid:9) , ( m, n ) = (cid:8) (2 i , 128) | i = 4 , . .
, 9 (cid:9) , α ∈ { 0 . 25 , 0 . 45 } .
The results can be found in Fig.
and are consistent with our theory: for fixed m , a small number of samples per machine n leads to inaccurate local solutions and inaccurate final estimates for all variants, while increasing n leads to a decrease in error for all configurations.
On the other hand, the robust algorithm produces good estimates for all attempted values of m after fixing a sufficiently large sample size n , and increasing m only decreases the error for the robust variant. Finally, we observe that the error of the non-robust variants plateaus since the corruption fraction α is bounded away from zero. [Figure]  Discussion
We presented a communication-efficient algorithm for distributed eigenspace estimation that is robust to α = 1 / 12 fraction of nodes returning structurally valid but otherwise potentially adversarial responses.
Numerical evidence suggests its breakdown point is closer to the (optimal) α ∗ = 1 / 2 , which might be achievable by an improved analysis of the filtering procedure in Alg 4. Our adaptive version of the filtering procedure in Alg. 5 trades off knowledge of (an upper bound on) the corruption level α with the need for a tight bound on ∥ Σ I good ∥ 2 .
Alternatively, one can design a version of Algorithmthat is adaptive to the corruption level α using a similar construction that evaluates the error proxy f
( λ ; p, α ) for different values of α and fixed λ ≈ ∥
Σ I good ∥ 2 instead.
Acknowledgements
We thank Jayadev Acharya and Damek Davis for their insightful comments and suggestions.
We also thank the anonymous reviewers for invaluable feedback that greatly improved the manuscript.
Appendix A Auxiliary results
In this section, we present a few supporting results.
The first result is a path independence lemma for perturbations of eigenvectors.
It first appeared in [37]; the eigengap condition in the statement of the
Lemma is justified in [10, Lemma 5]
[Equation]
Lemma 2 (Path independence) .
Let A ∈ R d × d be a fixed symmetric matrix and let ˆ A :
=
A + E , where E is a symmetric perturbation.
Suppose that we can write
[Equation]
where E 0 , E 1 , F 0 , F 1 are symmetric matrices
[Equation]
and define the intermediate matrices [Equation] Fix any V ∈ O ( d, r ) whose columns span the principal r -dimensional invariant subspace of A and construct the leading eigenvector matrices ˆ V 1 , ˆ V 2 ∈ O
( d, r ) of ˆ A 1 and ˆ A 2 such that
[Equation]
Further
[Equation]
let ˜ V 1 and ˜ V 2 be the leading eigenvector matrices of ˜
A 1 and ˜ A 2 , constructed in a similar fashion.
Then, ˆ V 2 and ˜ V 2 both span principal invariant subspaces of ˆ A
=
A + E .
Moreover, they satisfy
[Equation]
as long as A satisfies δ r ( A ) ≥ 4 ε .
Lemma 3.
Suppose that U ∈ O d,r satisfies dist( U, V ) ≤ ε < 1 / 2 , where V is the principal eigenvector matrix of a symmetric matrix A with eigengap δ r ( A ) :
= λ r ( A ) − λ r +1 ( A ) > 0 .
Then there exists a symmetric matrix B such that the following hold:
∥
A − B ∥ 2 ≤ 8 ∥ A ∥ 2 ε and δ r ( B ) =
δ r ( A ) .
U is the principal eigenvector matrix of B .
Proof.
We prove Item 1 first.
To that end, we can write A = A 1 + A 2 , where A 1 := V Σ 1 V T and
A 2 :=
V ⊥ Σ 2 V T ⊥ .
We consider the following matrix B :
[Equation]
where U T ⊥ U = 0 and U ⊥ ∈
O d,d − r .
From (24) and the condition Σ 1 ≻ Σ 2 , it follows that U is a principal eigenvector matrix for B .
Moreover, the gap condition on A immediately translates to the claimed gap condition for B .
It remains to bound the distance between A and B .
We write
[Equation]
To upper bound the first term on the right-hand side above, we use the spectral projectors P U := UU T and P U ⊥ =
I − P U to decompose it into
[Equation]
where the last inequality follows from the inequality ∥
P U ⊥ V ∥ 2 = dist( U, V ) =
{+ε and Lemma 4. A similar argument shows that ∥ U ⊥ Σ 2 U T ⊥ − A 2 ∥ 2 ≤ 2 ∥ Σ 2 ∥ 2 ε + 4 ∥ Σ 2 ∥ 
+}
ε 2 .
Taking into account the bound ε < 1 / 2 completes the proof. Lemma 4 (Modified sin θ distance) . Let U, V ∈
O
( d, r ) satisfy dist( U, V ) = α < 1 . Then the following holds: [Equation]
Proof. Let P Σ Q T be the singular value decomposition of U T V . Recall that [14, Eq.
[Equation]
where θ i ∈ [0 , π/ 2] and ∥ sin Θ ∥ 2 = α , following [14, Lemma 2.5]. From our assumptions, it follows that
[Equation]
with the last inequality following from 0 ≤ sin( θ/ 2) ≤ sin( θ ) for any θ ∈ [0 , π/ 2] . B Additional proofs This section includes proofs that were omitted from the main text.
Proof.
Define C := { i ∈ [ m ] | dist( Y i , V ) ≤ ε } with |C| > m 2 .
Now, we consider any pair ( i, j ) with i, j ∈ C . By the triangle inequality,
[Equation] [Equation]
Y [Equation] V [Equation]
ε [Equation] for all i, j ∈ C .
Now, fix i ⋆ to be any index for which dist( Y i , Y j ) ≤ 2 ε for at least m/ 2 other indices j ̸ = i ⋆ (such an index always exists because |C| ≥ m 2 + 1 ). For any such i ⋆ , there must be another index j satisfying dist( Y j , V ) ≤ ε and dist( Y j , Y i ⋆ ) ≤ 2 ε .
Therefore, dist( Y i ⋆ , V ) ≤ dist( Y i ⋆ , Y j ) + dist( Y j , V ) ≤ 3 ε. B.2 Proof of Lemma
Proof.
Recall that V is an eigenvector matrix of A that satisfies
[Equation] From Lemma 3, it follows that the columns of V ref span the principal eigenspace of a matrix B with nontrivial eigengap that satisfies ∥ A − B ∥ 2 ≲ ∥ A ∥ 2 ε . We now relate V ideal i to V corr i using the aforementioned path independence result. To that end, note that V ideal i is the leading eigenvector matrix of
[Equation]
that has been maximally aligned with V (in the sense of Frobenius distance). On the other hand, the Procrustes estimates V corr i are given by the leading eigenvector matrices of [Equation]
since V ref is the leading eigenvector of B nearest to V and V corr i is formed as the leading eigenvector of A i nearest to V ref . Applying Lemma 2 with E 0 : = A i − A , E 1 = 0 , F 0 :
= B − A and F 1 := A i − B , we obtain V corr
i [Equation] Finally [Equation]
we note the following upper bound [Equation] which concludes the proof. B.3 Proof of Proposition
Proof.
Let j crit be the smallest index for which 2 j ≥ max { λ lb , {+∥ Σ G 
 ∥ 2 } . For a fixed corruption fraction+} α and failure probability p , define the events [Equation]
From Theorem 3 in the main text and a union bound [Equation] it follows that
[Equation] Let us write θ ∗ := 1 | G 0 | (cid:80) i ∈ G 0 X i . Conditioned on the event E , for any j, j ′ ≥ j crit we have [Equation]
Consequently, it follows that 2 j crit satisfies the condition of the estimator, and therefore [Equation] Finally, the desired claim follows since
[Equation]
The next Lemma provides an upper bound on the operator norm of the empirical covariance Σ I good . Lemma 5.
Suppose that (cid:98) V ref satisfies δ r ( A ) ≥ 8 dist( (cid:98) V ref , V ) . Then we have
[Equation] Proof.
Let µ denote the empirical mean over I good .
We have µ = |I good | (cid:88) i ∈I good (cid:101) V i ,
[Equation] where V ∈ O d,r spans the principal eigenspace of A and satisfies
[Equation]
We now bound the spectral norm of Σ I good . Indeed, we have
[Equation]
+ 2 (cid:13)(cid:13)(cid:13)(cid:13)
|I
good | (cid:88) i ∈I good
(cid:101) V i − V (cid:13)(cid:13)(cid:13)(cid:13) , using the fact that (cid:101) V i (cid:101) V T i
= V i V T
i for all i ∈ I good .
In this section, we modify the proof of [33, Theorem 4] to derive guarantees for robust mean estimation with matrix-valued inputs. We recall some notation used therein: given the set of “good” samples G 0 and the initial sample S {+
+}
= { 1 , . . . , m } , we denote
S k
= { points remaining after k recursive calls to Filter } ,
[Equation]
Moreover, given any set S ⊂ [ m ] , we write
[Equation] [Equation]
In our proofs, we frequently employ the total variation distance d TV . For discrete distributions P 1 ,
P 2 on a common sample space Ω , d TV is given by [Equation] Finally, we define the [-“optimal” ∥ Σ G 
-]events E k , where k ∈ N , as below:
[Equation]
Our proof essentially traces the proof of [33, Theorem 4] but for the case of matrix-valued inputs to the Filter algorithm.
The first result has already been shown in [33], as its proof is independent of the shape of the inputs. Lemma 6 (See [33, Lemma 6]) . Let T := inf { k ∈ N | E k is true } .
Then we have:
[Equation]
The remainder of the proof is devoted to showing that, as soon as some E k is true, Filter will terminate with a good estimate. Throughout, we condition on the event
[Equation]
where T p := 3( m − | G 0 | ) + 18 log(1 /p ) ,
[Equation] [Equation]
which holds with probability at least 1 − p . Theorem 5. Suppose that α , p and N satisfy
[Equation]
Then the following hold simultaneously with probability at least 1 − p :
Filter ( S 0 , ∥ Σ G 0 ∥ 2 ) terminates after at most T p iterations; 2.
The output of Filter ( S 0 , ∥ Σ G 0 ∥ 2 ) , θ ∥ Σ G 0 ∥ 2 , satisfies [Equation]
Remark 5. While we prove the Theorem for the case λ ub = ∥ Σ G 0 ∥ 2 , a straightforward modification of the proof shows that when λ ub ≥ ∥ Σ G 0 ∥ 2 , we have [Equation]
Proof of Theorem 5. We condition on the event E from (32), which holds with probability at least 1 − p . This implies that there is some index k ≤ T p such that [Equation]
From Lemma 11, we obtain that the empirical covariance satisfies ∥ Σ S k ∥ 2 ≤ 18 ∥ Σ G 0 ∥ 2 , and thus the algorithm terminates after at most k steps. We have the following cases: The termination condition was first triggered at the k th step. In that case, Lemma 11 directly implies the desired inequality.
The algorithm terminated at some index ℓ < k .
Then it follows from Lemma 12 that
[Equation]
At the same time, Lemma 7 implies that
[Equation]
From the termination condition, we obtain that
[Equation]
Combining Eqs.
(35) to (37) yields the desired bound.
The next few Lemmas are supporting statements used in the proof of Theorem 5.
Lemma 7.
Let S = { X 1 , . . .
, X m } where X i ∈ R d × r and suppose that P 1 , P 2 are discrete distributions supported over [ m ] with d TV ( P 1 , P 2 ) =
η .
Then the following holds:
[Equation]
where the matrices Σ P i are defined as:
[Equation]
Proof.
Following the proof of [27, Lemma 2.1], we consider a coupling between P 1 and P 2 such that P ( X = X ′ ) ≥ 1 − η .
Denoting ∥
X ∥ L 2 := (cid:112) E [ X 2 ] , we have
[Equation]
Let µ 1 :=
E P 1 [ X ] and µ 2 = E P 2 [ X ] .
Since ∥·∥ L 2 is a norm
[Equation]
the triangle inequality implies that sup u,
[Equation]
∈B
[Equation]
We now upper bound the remaining terms.
For the first one, we have
[Equation]
where the penultimate equality uses linearity of the trace operator and the last equality is the definition of the spectral norm for symmetric positive semidefinite matrices.
Similar arguments also yield
[Equation]
Plugging Eqs.
(40) to (43) back into Eq.
(39) and
rearranging yields the expected result:
[Equation]
Lemma
[Equation]
Let G ⊂ S ⊂ [ m ] .
Moreover
[Equation]
let µ S and µ G be their respective empirical means
[Equation]
and let v be the leading eigenvector of Σ S so that the outlier scores satisfy [Equation] Moreover, define η := 1 − | G | / | S | and fix a γ ∈ (0 , 1 / η ) . Then, we have the implication
[Equation]
Proof.
Recall that the (normalized) sum of outlier scores over the set G is given by
[Equation]
|
[Equation]
[Equation]
We now simplify the second term.
Indeed, we have
[Equation]
For brevity, denote η := | S \ G | | S | .
Plugging (46) back into (45), we obtain
[Equation]
We now bound the second term in (47).
From [17, Lemma 2.4]
[Equation]
it follows that
[Equation]
Rearranging and multiplying by η/ (1 − η ) gives
[Equation]
Plugging back into Eq.
and using the fact that | G | = | S | (1 − η ) , we obtain
[Equation]
Finally, replacing | G | = | S | (1 − η ) in (48) and rearranging, we obtain
[Equation]
Lemma 9.
Suppose that (33) is true.
Then the following holds for any k ≤ T p :
[Equation]
Proof.
Recall that B k
= S
k \ G k and notice that
[Equation]
where the first inequality follows from the fact that | B k | ≤ | B 0 | . Lemma 10. For any integer k , the sets G k and G 0 satisfy [Equation]
Proof. We expand the definition of Σ G 0 and rewrite: [Equation] [Equation]
We now rewrite the first term in the above sum using
T 1 = [Equation] Letting v ∈ S d − 1 and using the fact that T 2 is positive semidefinite
[Equation] we arrive at [Equation] Finally, taking suprema over both sides yields the desired inequality.
Lemma 11. Suppose that (33) is true and that the following inequality holds for some index k ≤ T p : [Equation] Then the empirical means satisfy
[Equation] Proof. Let P 1 := Unif( G 0 ) and P 2 := Unif( S k ) .
From Lemma 7, it follows that [Equation] Since (51) is the reverse of (44) [Equation]
we obtain [Equation]
where the first inequality follows from the contrapositive of Lemma 8, the second inequality from γ = 3 and Lemma [Equation] and the last inequality follows by our assumption on α . Now, let K ≤ T p be the number of samples in G 0 that were removed by the algorithm.
We have | G 0 | [Equation] 1 /p ) m
From (33), we additionally have that [Equation] Substituting the above into (52) and using Lemma 12 yields the desired bound: [Equation]
Lemma 12. Suppose k ≤ T p and (33) holds. Then we have that [Equation]
Proof. We let P 1 := Unif( S k ) , P 2 := Unif( G 0 ) and P 3 := Unif( G k ) , and write K ≤ k ≤ T p for the number of samples originally in G 0 that were removed by the Filter algorithm by the k th step. From the triangle inequality, it follows that
[Equation]
where the second line follows from Lemma 13 and the last two inequalities follow from K ≤ m and m ≤ T p . Finally, using Lemma 6 and Eq. (33), we obtain | [Equation]
[Equation] Lemma 13. Consider a pair of discrete sets S, S ′ such that S ′ ⊂ S . We have: [Equation]
Proof.
Using the fact that d TV ( p, q ) = 12 ∥ p − q ∥ 1 , we have: [Equation] = [Equation]
|
[Equation] | [Equation]
|
[Equation]
| − | S
[Equation]
| | S | (cid:19)
[Equation]
B.5 Proof of Theorem
We now present the proof of the main theorem on distributed PCA.
We first recall that
[Equation]
where X ( i ) j iid ∼ P , and that the responses V i ∈ O d,r span the leading r -dimensional eigenspace of A i . Under this model, the local errors E i :=
A i − A as well as the error of the empirical average over the inliers are bounded with high probability.
We will condition on the following events for the remainder of this section:
[Equation]
Lemma 14.
Suppose that n ≳ κ 2 · ( r ⋆ + log( mn/p )) .
Then the following hold:
[Equation] Proof.
The bound on P ( E 2 ) in Eq. (56) follows from an application of [38, Exercise 9.2.5] and the assumed lower bound on n . On the other hand, the same result yields
[Equation]
for any fixed i ∈ I good .
From the lower bound on n , it follows that [Equation]
Finally, taking a union bound over I good recovers the bound on P ( E 1 ) .
An immediate corollary is a bound on the error of RobustReferenceEstimator . Corollary 2.
There is a universal constant C ref such that the output of Alg.
2 satisfies [Equation] Proof.
From the bound α < 12 and the conditioning on E 1 , we deduce the existence of an index set
S ′ such that | S ′ | > m 2 , and [Equation]
where the first bound on dist( (cid:98) V i , V ) follows from the Davis-Kahan theorem [14, Theorem 2.7] and the fact that ∥ A i − A ∥ 2 ≤ δ 8 for any i / ∈ I bad .
From Proposition 1 in the main text, it follows that [Equation] The next Proposition instantiates the bounds of Lemma 5 for for the case of distributed PCA.
Proposition 4.
In the setting of Lemma 5, the matrix Σ
I good satisfies
[Equation]
Proof.
From Lemma 5, it follows that [Equation] From Proposition 2 in the main text and conditioning E 1 and E 2 , we have [Equation]
On the other hand
[Equation]
using [
[Equation]
Theorem 2]
[Equation]
we have that
[Equation]
Putting all the bounds together yields (57) [Equation] We now invoke Proposition 3 and recall that ϱ is defined as [Equation]
From that and Proposition
[Equation]
it follows that Alg. 5 from the main text invoked with λ
lb = ω :
= (cid:112) 1 / mn and λ ub = 6 outputs an estimate satisfying
[Equation]
with failure probability at most 2 log
[Equation]
6 /ω ) p .
Finally, from Eqs.
(58) and (61) it follows that
[Equation]
In particular
[Equation]
the success probability is at least (given that ω is set as (cid:112) 1 / mn ):
[Equation]
Letting p := p ′ 2log 2 (6 mn ) and relabeling p ′
→ p yields the result.
C Experiment details
The numerical experiments in the main text were coded in Julia and run on a machine with Intel(R) Core(TM) i7-7700 CPU, 16GB of RAM and a GNU/Linux environment.
The code is attached as part of the supplementary material and consists of a library called RobustDistributedPCA.jl and a subfolder scripts/ that contains the scripts reproducing the experiments in the manuscript.
We refer the reader to the included README.md file for installation and usage instructions.
Notes on implementation.
Our implementations of the Filter and AdaptiveFilter algorithms deviate from the theory in the following ways: In Filter , we remove the point with the largest outlier score by default:
[Equation] The randomized selection can be enabled by passing randomized = true to the appropriate function.
Please see the attached README.md file for details.
The error proxy f
( λ ; p, α ) we use in AdaptiveFilter has been simplified to
[Equation]
The reason is twofold: on one hand, we suspect that many of the constants involved in the original definition of f are artifacts of our proof and are much lower in practice; on the other hand, since the default behavior of Filter is not randomized, the term owing to failure probability in f can be removed.
Finally, we change the stopping condition of AdaptiveFilter to λ ≤ λ ub in order to simplify constants.
As the value of λ ub is determined adaptively, this does not affect the output of the algorithm.