Original: Outlier-Robust Sparse Estimation via Non-Convex Optimization
Revised: Outlier-Robust Sparse Estimation via Non-Convex Optimization

Original: Abstract
Revised: Abstract

Original: We explore the connection between outlier-robust high-dimensional statistics and non-convex optimization in the presence of sparsity constraints, with a focus on the fundamental tasks of robust sparse mean estimation and robust sparse PCA.
Revised: We explore the connection between outlier-robust high-dimensional statistics and non-convex optimization in the presence of sparsity constraints, with a focus on the fundamental tasks of robust sparse mean estimation and robust sparse PCA.

Original: We develop novel and simple optimization formulations for these problems such that any approximate stationary point of the associated optimization problem yields a near-optimal solution for the underlying robust estimation task.
Revised: We develop novel and simple optimization formulations for these problems such that any approximate stationary point of the associated optimization problem yields a near-optimal solution for the underlying robust estimation task.

Original: As a corollary, we obtain that any ﬁrst-order method that efﬁciently converges to stationarity yields an efﬁcient algorithm for these tasks.
Revised: As a corollary, we obtain that any ﬁrst-order method that efﬁciently converges to stationarity yields an efﬁcient algorithm for these tasks.

Original: The obtained algorithms are simple, practical, and succeed under broader distributional assumptions compared to prior work.
Revised: The obtained algorithms are simple, practical, and succeed under broader distributional assumptions compared to prior work.

Original:  Introduction
Revised:  Introduction

Original: In several modern machine learning (ML) applications, such as ML security [BNJT10, BNL12,
Revised: In several modern machine learning (ML) applications, such as ML security [BNJT10, BNL12,

Original: SKL17, DKK + 19a] and exploratory analysis of real datasets, e g., in population genetics [RPW + 02,
Revised: SKL17, DKK + 19a] and exploratory analysis of real datasets, e g., in population genetics [RPW + 02,

Original: PLJD10, LAT + 08, DKK + 17], typical datasets contain a non-trivial fraction of arbitrary (or even adversarial) outliers.
Revised: PLJD10, LAT + 08, DKK + 17], typical datasets contain a non-trivial fraction of arbitrary (or even adversarial) outliers.

Original: Robust statistics [HRRS86, HR09] is the subﬁeld of statistics aiming to design estimators that are tolerant to a constant fraction of outliers, independent of the dimensionality of the data.
Revised: Robust statistics [HRRS86, HR09] is the subﬁeld of statistics aiming to design estimators that are tolerant to a constant fraction of outliers, independent of the dimensionality of the data.

Original: Early work in this ﬁeld, see, e.g., [Tuk60, Hub64, Tuk75] developed sample-efﬁcient robust estimators for various basic tasks, alas with runtime exponential in the dimension.
Revised: Early work in this ﬁeld, see, e.g., [Tuk60, Hub64, Tuk75] developed sample-efﬁcient robust estimators for various basic tasks, alas with runtime exponential in the dimension.

Original: During the past ﬁve years, a line of work in computer science, starting with [DKK + 16, LRV16], has developed the ﬁrst computationally efﬁcient robust high-dimensional estimators for a range of tasks.
Revised: During the past ﬁve years, a line of work in computer science, starting with [DKK + 16, LRV16], has developed the ﬁrst computationally efﬁcient robust high-dimensional estimators for a range of tasks.

Original: This progress has led to a revival of robust statistics from an algorithmic perspective (see, e.g., [DK19, DKK + 21] for surveys on the topic).
Revised: This progress has led to a revival of robust statistics from an algorithmic perspective (see, e.g., [DK19, DKK + 21] for surveys on the topic).

Original: In this work, we focus on high-dimensional estimation tasks in the presence of sparsity constraints.
Revised: In this work, we focus on high-dimensional estimation tasks in the presence of sparsity constraints.

Original: To rigorously study these problems, we need to formally deﬁne the model of data corruption.
Revised: To rigorously study these problems, we need to formally deﬁne the model of data corruption.

Original: Throughout this work, we work with the following standard contamination model.
Revised: Throughout this work, we work with the following standard contamination model.

Original: Deﬁnition 1.1 (Strong Contamination Model, see [DKK + 16]) .
Revised: Deﬁnition 1.1 (Strong Contamination Model, see [DKK + 16]) .

Original: Given a parameter 0 < (cid:15) < 1 / 2 and a distribution family D on R d , the adversary operates as follows:
Revised: Given a parameter 0 < (cid:15) < 1 / 2 and a distribution family D on R d , the adversary operates as follows:

Original: The algorithm speciﬁes a number of samples n , and n samples are drawn from some unknown D ∈ D .
Revised: The algorithm speciﬁes a number of samples n , and n samples are drawn from some unknown D ∈ D .

Original: The adversary is allowed to inspect the samples, remove up to (cid:15)n of them and replace them with arbitrary points.
Revised: The adversary is allowed to inspect the samples, remove up to (cid:15)n of them and replace them with arbitrary points.

Original: This modiﬁed set of n points is then given as input to the algorithm.
Revised: This modiﬁed set of n points is then given as input to the algorithm.

Original: We say that a set of samples is (cid:15) -corrupted if it is generated by the above process.
Revised: We say that a set of samples is (cid:15) -corrupted if it is generated by the above process.

Original: High-dimensional robust statistics is algorithmically challenging because the natural optimization formulations of such tasks are typically non-convex.
Revised: High-dimensional robust statistics is algorithmically challenging because the natural optimization formulations of such tasks are typically non-convex.

Original: The recent line of work on algorithmic robust statistics has led to a range of sophisticated algorithms.
Revised: The recent line of work on algorithmic robust statistics has led to a range of sophisticated algorithms.

Original: In some cases, such algorithms require solving large convex relaxations, rendering them computationally prohibitive for large-scale problems.
Revised: In some cases, such algorithms require solving large convex relaxations, rendering them computationally prohibitive for large-scale problems.

Original: In other cases, they involve a number of hyper-parameters that may require careful tuning.
Revised: In other cases, they involve a number of hyper-parameters that may require careful tuning.

Original: Motivated by these shortcomings of known algorithms, recent work [CDGS20, ZJS20] established an intriguing connection between high-dimensional robust estimation and non-convex optimization.
Revised: Motivated by these shortcomings of known algorithms, recent work [CDGS20, ZJS20] established an intriguing connection between high-dimensional robust estimation and non-convex optimization.

Original: The high-level idea is quite simple: Even though typical robust statistics tasks lead to non-convex formulations, it may still be possible to leverage the underlying structure to show that standard ﬁrst-order methods provably and efﬁciently reach near-optimal solutions.
Revised: The high-level idea is quite simple: Even though typical robust statistics tasks lead to non-convex formulations, it may still be possible to leverage the underlying structure to show that standard ﬁrst-order methods provably and efﬁciently reach near-optimal solutions.

Original: Indeed, [CDGS20, ZJS20] were able to prove such statements for robust mean estimation under natural distributional assumptions.
Revised: Indeed, [CDGS20, ZJS20] were able to prove such statements for robust mean estimation under natural distributional assumptions.

Original: Speciﬁcally, these works established that any (approximate) stationary point of a well-studied non-convex formulation for robust mean estimation yields a near-optimal solution for the underlying robust estimation task.
Revised: Speciﬁcally, these works established that any (approximate) stationary point of a well-studied non-convex formulation for robust mean estimation yields a near-optimal solution for the underlying robust estimation task.

Original: In this work, we continue this line of work with a focus on sparse estimation tasks.
Revised: In this work, we continue this line of work with a focus on sparse estimation tasks.

Original: Leveraging sparsity in high-dimensional datasets is a fundamental problem of signiﬁcant practical importance.
Revised: Leveraging sparsity in high-dimensional datasets is a fundamental problem of signiﬁcant practical importance.

Original: Various formalizations of this problem have been investigated in statistics and machine learning for at least the past two decades (see, e.g., [HTW15] for a textbook on the topic).
Revised: Various formalizations of this problem have been investigated in statistics and machine learning for at least the past two decades (see, e.g., [HTW15] for a textbook on the topic).

Original: We focus on robust sparse mean estimation and robust sparse PCA .
Revised: We focus on robust sparse mean estimation and robust sparse PCA .

Original: Sparse mean estimation is arguably one of the most fundamental sparse estimation tasks and is closely related to the Gaussian sequence model [Tsy08, Joh17].
Revised: Sparse mean estimation is arguably one of the most fundamental sparse estimation tasks and is closely related to the Gaussian sequence model [Tsy08, Joh17].

Original: The task of sparse PCA in the spiked covariance model, initiated in [Joh01], has been extensively investigated (see Chapter 8 of [HTW15] and references therein).
Revised: The task of sparse PCA in the spiked covariance model, initiated in [Joh01], has been extensively investigated (see Chapter 8 of [HTW15] and references therein).

Original: In the context of robust sparse mean estimation, we are given an (cid:15) -corrupted set of samples from a distribution with unknown mean µ ∈ R d where µ is k -sparse, and we want to compute a vector (cid:98) µ close to µ .
Revised: In the context of robust sparse mean estimation, we are given an (cid:15) -corrupted set of samples from a distribution with unknown mean µ ∈ R d where µ is k -sparse, and we want to compute a vector (cid:98) µ close to µ .

Original: In the context of robust sparse PCA (in the spiked covariance model), we are given an (cid:15) -corrupted set of samples from a distribution with covariance matrix
Revised: In the context of robust sparse PCA (in the spiked covariance model), we are given an (cid:15) -corrupted set of samples from a distribution with covariance matrix

Original: I + ρvv T , where v ∈ R
Revised: I + ρvv T , where v ∈ R

Original: d is k -sparse and the goal is to approximate v .
Revised: d is k -sparse and the goal is to approximate v .

Original: It is worth noting that for both problems, we have access to much fewer samples compared to the non-sparse case (roughly O ( k 2 log d ) instead of Ω( d ) ).
Revised: It is worth noting that for both problems, we have access to much fewer samples compared to the non-sparse case (roughly O ( k 2 log d ) instead of Ω( d ) ).

Original: Consequently, the design and analysis of optimization formulations for robust sparse estimation requires new ideas and techniques that signiﬁcantly deviate from the standard (non-sparse) case.
Revised: Consequently, the design and analysis of optimization formulations for robust sparse estimation requires new ideas and techniques that signiﬁcantly deviate from the standard (non-sparse) case.

Original: 1.1 Our Results and Contributions
Revised: 1.1 Our Results and Contributions

Original: We show that standard ﬁrst-order methods lead to robust and efﬁcient algorithms for sparse mean estimation and sparse PCA.
Revised: We show that standard ﬁrst-order methods lead to robust and efﬁcient algorithms for sparse mean estimation and sparse PCA.

Original: Our main contribution is to propose novel (non-convex) formulations for these robust estimation tasks, and to show that approximate stationarity sufﬁces for near-optimality .
Revised: Our main contribution is to propose novel (non-convex) formulations for these robust estimation tasks, and to show that approximate stationarity sufﬁces for near-optimality .

Original: We establish landscape results showing that any approximate stationary point of our objective function yields a near-optimal solution for the underlying robust estimation task.
Revised: We establish landscape results showing that any approximate stationary point of our objective function yields a near-optimal solution for the underlying robust estimation task.

Original: Consequently, gradient descent (or any other methods converging to stationarity) can solve these problems.
Revised: Consequently, gradient descent (or any other methods converging to stationarity) can solve these problems.

Original: Our results provide new insights and techniques in designing and analyzing (non-convex) optimization formulations of robust estimation tasks.
Revised: Our results provide new insights and techniques in designing and analyzing (non-convex) optimization formulations of robust estimation tasks.

Original: Our formulations and structural results immediately lead to simple and practical algorithms for robust sparse estimation.
Revised: Our formulations and structural results immediately lead to simple and practical algorithms for robust sparse estimation.

Original: Importantly, the gradient of our objectives can be computed efﬁciently via a small number of basic matrix operations.
Revised: Importantly, the gradient of our objectives can be computed efﬁciently via a small number of basic matrix operations.

Original: In addition to their simplicity and practicality, our methods provably succeed under more general distributional assumptions compared to prior work.
Revised: In addition to their simplicity and practicality, our methods provably succeed under more general distributional assumptions compared to prior work.

Original: For robust sparse mean estimation and robust sparse PCA, our landscape results require deterministic conditions on the original set of good samples.
Revised: For robust sparse mean estimation and robust sparse PCA, our landscape results require deterministic conditions on the original set of good samples.

Original: We refer to these conditions as stability conditions (Deﬁnitions 2.1 and 2.2, formally deﬁned in Section 2).
Revised: We refer to these conditions as stability conditions (Deﬁnitions 2.1 and 2.2, formally deﬁned in Section 2).

Original: At a high level, they state that the ﬁrst and second moments of a set of samples are stable when any (cid:15) -fraction of the samples are removed.
Revised: At a high level, they state that the ﬁrst and second moments of a set of samples are stable when any (cid:15) -fraction of the samples are removed.

Original: These stability conditions hold with high probability for a set of clean samples drawn from natural families of distributions (e.g., subgaussian).
Revised: These stability conditions hold with high probability for a set of clean samples drawn from natural families of distributions (e.g., subgaussian).

Original: For robust sparse mean estimation, we establish the following result.
Revised: For robust sparse mean estimation, we establish the following result.

Original: Theorem 1.2 (Robust Sparse Mean Estimation) .
Revised: Theorem 1.2 (Robust Sparse Mean Estimation) .

Original: Let 0 < (cid:15) < (cid:15) 0 for some universal constant (cid:15) 0 and let δ > (cid:15) .
Revised: Let 0 < (cid:15) < (cid:15) 0 for some universal constant (cid:15) 0 and let δ > (cid:15) .

Original: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ )
Revised: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ )

Original: -stable (per Deﬁnition 2.1) w.r.t.
Revised: -stable (per Deﬁnition 2.1) w.r.t.

Original: a distribution with unknown k -sparse mean µ ∈ R d .
Revised: a distribution with unknown k -sparse mean µ ∈ R d .

Original: Let S =
Revised: Let S =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 be an (cid:15) -corrupted version of G (cid:63) .
Revised: =1 be an (cid:15) -corrupted version of G (cid:63) .

Original: 1 There is an algorithm that on inputs S , k , (cid:15) , and δ , runs in polynomial time and returns a k -sparse vector (cid:98) µ ∈ R d such that (cid:107) (cid:98) µ − µ (cid:107) 2 ≤ O ( δ ) .
Revised: 1 There is an algorithm that on inputs S , k , (cid:15) , and δ , runs in polynomial time and returns a k -sparse vector (cid:98) µ ∈ R d such that (cid:107) (cid:98) µ − µ (cid:107) 2 ≤ O ( δ ) .

Original: We emphasize that a key novelty of Theorem 1.2 is that the underlying algorithm is a ﬁrst-order method applied to a novel non-convex formulation of the problem.
Revised: We emphasize that a key novelty of Theorem 1.2 is that the underlying algorithm is a ﬁrst-order method applied to a novel non-convex formulation of the problem.

Original: The major advantage of our algorithm over prior work [BDLS17, DKK + 19b] is its simplicity, practicality, and the fact that it seamlessly applies to a wider class of distributions on the clean data.
Revised: The major advantage of our algorithm over prior work [BDLS17, DKK + 19b] is its simplicity, practicality, and the fact that it seamlessly applies to a wider class of distributions on the clean data.

Original: As we will discuss in Section 3, when the ground-truth distribution D is subgaussian with unknown k -sparse mean µ ∈ R d and identity covariance, a set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from
Revised: As we will discuss in Section 3, when the ground-truth distribution D is subgaussian with unknown k -sparse mean µ ∈ R d and identity covariance, a set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from

Original: D is ( k, (cid:15), δ ) -stable
Revised: D is ( k, (cid:15), δ ) -stable

Original: (Deﬁnition 2.1) with high probability for δ
Revised: (Deﬁnition 2.1) with high probability for δ

Original: = O ( (cid:15) (cid:112) log(1 /(cid:15) )) .
Revised: = O ( (cid:15) (cid:112) log(1 /(cid:15) )) .

Original: It follows as an immediate corollary of Theorem 1.2 that, given an (cid:15) -corrupted set of samples, we can compute a vector (cid:98) µ that is O ( δ ) =
Revised: It follows as an immediate corollary of Theorem 1.2 that, given an (cid:15) -corrupted set of samples, we can compute a vector (cid:98) µ that is O ( δ ) =

Original: O ( (cid:15) (cid:112) log(1 /(cid:15) ))
Revised: O ( (cid:15) (cid:112) log(1 /(cid:15) ))

Original: close to the true mean µ .
Revised: close to the true mean µ .

Original: This sample complexity matches the known computational-statistical lower bounds [DKS17, BB20].
Revised: This sample complexity matches the known computational-statistical lower bounds [DKS17, BB20].

Original: More generally, one can relax the concentration assumption on the clean data and obtain qualitatively similar error guarantees.
Revised: More generally, one can relax the concentration assumption on the clean data and obtain qualitatively similar error guarantees.

Original: Next we state our main result for robust sparse PCA.
Revised: Next we state our main result for robust sparse PCA.

Original: Theorem 1.3 (Robust Sparse PCA) .
Revised: Theorem 1.3 (Robust Sparse PCA) .

Original: Let 0 < ρ ≤ 1 and 0 < (cid:15) < (cid:15) 0 for some universal constant (cid:15) 0 .
Revised: Let 0 < ρ ≤ 1 and 0 < (cid:15) < (cid:15) 0 for some universal constant (cid:15) 0 .

Original: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ )
Revised: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ )

Original: -stable (as in Deﬁnition 2.2) w.r.t.
Revised: -stable (as in Deﬁnition 2.2) w.r.t.

Original: a centered distribution with covariance Σ
Revised: a centered distribution with covariance Σ

Original: = I + ρvv (cid:62) , for an unknown k -sparse unit vector v ∈ R d .
Revised: = I + ρvv (cid:62) , for an unknown k -sparse unit vector v ∈ R d .

Original: Let S =
Revised: Let S =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 be an (cid:15) -corrupted version of G (cid:63) .
Revised: =1 be an (cid:15) -corrupted version of G (cid:63) .

Original: There is an algorithm that on inputs S , k , and (cid:15) , runs in polynomial time and returns a unit vector u ∈ R d such that (cid:13)(cid:13) uu (cid:62) − vv (cid:62) (cid:13)(cid:13) F = O ( (cid:112) δ/ρ ) .
Revised: There is an algorithm that on inputs S , k , and (cid:15) , runs in polynomial time and returns a unit vector u ∈ R d such that (cid:13)(cid:13) uu (cid:62) − vv (cid:62) (cid:13)(cid:13) F = O ( (cid:112) δ/ρ ) .

Original: Interestingly, our algorithm for robust sparse PCA is a ﬁrst-order method applied to a simple convex formulation of the problem.
Revised: Interestingly, our algorithm for robust sparse PCA is a ﬁrst-order method applied to a simple convex formulation of the problem.

Original: We view the existence of a convex formulation as an intriguing fact that, surprisingly, was not observed in prior work.
Revised: We view the existence of a convex formulation as an intriguing fact that, surprisingly, was not observed in prior work.

Original: As we will discuss in Section 4, when the ground-truth distribution D is centered subgaussian with covariance Σ
Revised: As we will discuss in Section 4, when the ground-truth distribution D is centered subgaussian with covariance Σ

Original: = I + ρvv (cid:62) , for an unknown k -sparse unit vector v ∈ R d , a set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from D is ( k, (cid:15), δ ) -stable
Revised: = I + ρvv (cid:62) , for an unknown k -sparse unit vector v ∈ R d , a set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from D is ( k, (cid:15), δ ) -stable

Original: (Deﬁnition 2.2) with high probability for δ
Revised: (Deﬁnition 2.2) with high probability for δ

Original: = O ( (cid:15) log(1 /(cid:15) )) .
Revised: = O ( (cid:15) log(1 /(cid:15) )) .

Original: Therefore, our algorithm outputs a vector that is O ( (cid:112) (cid:15) log(1 /(cid:15) ) /ρ ) close to the true direction v .
Revised: Therefore, our algorithm outputs a vector that is O ( (cid:112) (cid:15) log(1 /(cid:15) ) /ρ ) close to the true direction v .

Original: The sample complexity in this case nearly matches the computational-statistical lower bound of Ω( k 2 log d/(cid:15) 2 )
Revised: The sample complexity in this case nearly matches the computational-statistical lower bound of Ω( k 2 log d/(cid:15) 2 )

Original: [BR13] which holds even without corruptions.
Revised: [BR13] which holds even without corruptions.

Original: While the error guarantee of our algorithm is slightly worse compared to prior work [BDLS17, DKK + 19b] for Gaussian data (we get
Revised: While the error guarantee of our algorithm is slightly worse compared to prior work [BDLS17, DKK + 19b] for Gaussian data (we get

Original: O ( (cid:112) δ/ρ ) rather than O ( δ/ρ ) ), we note that our algorithm works for a broader family of distributions.
Revised: O ( (cid:112) δ/ρ ) rather than O ( δ/ρ ) ), we note that our algorithm works for a broader family of distributions.

Original: Prior Work on Robust Sparse Estimation.
Revised: Prior Work on Robust Sparse Estimation.

Original: We provide a detailed summary of prior work for comparison.
Revised: We provide a detailed summary of prior work for comparison.

Original: [BDLS17] obtained the ﬁrst sample-efﬁcient and polynomial-time algorithms for robust sparse mean estimation and robust sparse PCA.
Revised: [BDLS17] obtained the ﬁrst sample-efﬁcient and polynomial-time algorithms for robust sparse mean estimation and robust sparse PCA.

Original: These algorithms succeed for Gaussian inliers and inherently use the ellipsoid method.
Revised: These algorithms succeed for Gaussian inliers and inherently use the ellipsoid method.

Original: The separation oracle required for the ellipsoid algorithm turns out to be another convex program — corresponding to an SDP to solve sparse PCA.
Revised: The separation oracle required for the ellipsoid algorithm turns out to be another convex program — corresponding to an SDP to solve sparse PCA.

Original: As a consequence, the running time of these algorithms, while polynomially bounded, is impractically high.
Revised: As a consequence, the running time of these algorithms, while polynomially bounded, is impractically high.

Original: [LLC19] proposed an algorithm for robust sparse mean estimation via iterative trimmed hard thresholding, which can only tolerate a sub-constant fraction of corruptions.
Revised: [LLC19] proposed an algorithm for robust sparse mean estimation via iterative trimmed hard thresholding, which can only tolerate a sub-constant fraction of corruptions.

Original: [DKK + 19b] gave iterative spectral robust algorithms for sparse mean estimation and sparse PCA.
Revised: [DKK + 19b] gave iterative spectral robust algorithms for sparse mean estimation and sparse PCA.

Original: These algorithms are still quite complex and are only shown to succeed under Gaussian inliers.
Revised: These algorithms are still quite complex and are only shown to succeed under Gaussian inliers.

Original: 1.2 Overview of Our Approach
Revised: 1.2 Overview of Our Approach

Original: In this section, we give an overview of our approach for robust sparse mean estimation.
Revised: In this section, we give an overview of our approach for robust sparse mean estimation.

Original: At a very high level, we assign a nonnegative weight to each data point and try to ﬁnd a good set of (1 − (cid:15) ) n samples.
Revised: At a very high level, we assign a nonnegative weight to each data point and try to ﬁnd a good set of (1 − (cid:15) ) n samples.

Original: The constraint on the weight vector is that it represents at least a (fractional) set of (1 − (cid:15) ) -portion of the input dataset.
Revised: The constraint on the weight vector is that it represents at least a (fractional) set of (1 − (cid:15) ) -portion of the input dataset.

Original: Formally, given n datapoints ( X i ) ni
Revised: Formally, given n datapoints ( X i ) ni

Original: =1 , the goal is to ﬁnd a weight vector w ∈ R n such that µ w = (cid:80) i w i
Revised: =1 , the goal is to ﬁnd a weight vector w ∈ R n such that µ w = (cid:80) i w i

Original: X i is close to the true mean µ .
Revised: X i is close to the true mean µ .

Original: The constraint on w is that it belongs to
Revised: The constraint on w is that it belongs to

Original: [Equation]
Revised: [Equation]

Original: which is the convex hull of all uniform distributions over subsets S ⊆
Revised: which is the convex hull of all uniform distributions over subsets S ⊆

Original: [ n ] of size | S | =
Revised: [ n ] of size | S | =

Original: (1 − (cid:15) ) n .
Revised: (1 − (cid:15) ) n .

Original: Let Σ w = (cid:80) i w i
Revised: Let Σ w = (cid:80) i w i

Original: ( X i − µ w )( X i − µ w ) (cid:62) denote the weighted empirical covariance matrix.
Revised: ( X i − µ w )( X i − µ w ) (cid:62) denote the weighted empirical covariance matrix.

Original: It is well-known that if one can ﬁnd w ∈ ∆ n,(cid:15) that minimizes the weighted empirical variance v (cid:62) Σ w v for all k -sparse unit vectors v , then µ w must be close to µ .
Revised: It is well-known that if one can ﬁnd w ∈ ∆ n,(cid:15) that minimizes the weighted empirical variance v (cid:62) Σ w v for all k -sparse unit vectors v , then µ w must be close to µ .

Original: Unfortunately, it is NP-Hard to ﬁnd the sparse direction v with the largest variance.
Revised: Unfortunately, it is NP-Hard to ﬁnd the sparse direction v with the largest variance.

Original: To get around this issue, [BDLS17] considered the following convex relaxation, minimizing the variance for convex combinations of sparse directions:
Revised: To get around this issue, [BDLS17] considered the following convex relaxation, minimizing the variance for convex combinations of sparse directions:

Original: [Equation]
Revised: [Equation]

Original: Given w , the optimal A can be found using semideﬁnite programming (SDP).
Revised: Given w , the optimal A can be found using semideﬁnite programming (SDP).

Original: [ZJS20] observed that any stationary point w of (1) gives a good solution for robust sparse mean estimation.
Revised: [ZJS20] observed that any stationary point w of (1) gives a good solution for robust sparse mean estimation.

Original: However, solving (1) requires convex programming to compute the gradient in each iteration.
Revised: However, solving (1) requires convex programming to compute the gradient in each iteration.

Original: As explained in the proceeding discussion, our approach circumvents this shortcoming, leading to a formulation for which each gradient can be computed using only basic matrix operations .
Revised: As explained in the proceeding discussion, our approach circumvents this shortcoming, leading to a formulation for which each gradient can be computed using only basic matrix operations .

Original: In this work, we propose and analyze the following optimization formulation:
Revised: In this work, we propose and analyze the following optimization formulation:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) A (cid:107) F,k,k is the Frobenius norm of the k 2 entries of A with largest magnitude, with the additional constraint that these k 2 entries are chosen from k rows with k entries in each row.
Revised: where (cid:107) A (cid:107) F,k,k is the Frobenius norm of the k 2 entries of A with largest magnitude, with the additional constraint that these k 2 entries are chosen from k rows with k entries in each row.

Original: We prove that any stationary point of f ( w ) yields a good solution for robust sparse mean estimation.
Revised: We prove that any stationary point of f ( w ) yields a good solution for robust sparse mean estimation.

Original: Here we provide a brief overview of our proof (see Section 3 for more details).
Revised: Here we provide a brief overview of our proof (see Section 3 for more details).

Original: Given a weight vector w , we show that if w is not a good solution, then moving toward w (cid:63) (the weight vector corresponding to the uniform distribution on the clean input samples) will decrease the objective value.
Revised: Given a weight vector w , we show that if w is not a good solution, then moving toward w (cid:63) (the weight vector corresponding to the uniform distribution on the clean input samples) will decrease the objective value.

Original: Formally, we will show that, for any 0 < η < 1 ,
Revised: Formally, we will show that, for any 0 < η < 1 ,

Original: [Equation]
Revised: [Equation]

Original: We can then take (cid:107) · (cid:107) F,k,k norm on both sides (after subtracting I ) and show that the third term can be essentially ignored.
Revised: We can then take (cid:107) · (cid:107) F,k,k norm on both sides (after subtracting I ) and show that the third term can be essentially ignored.

Original: If the third term were not there, we would have
Revised: If the third term were not there, we would have

Original: [Equation]
Revised: [Equation]

Original: Therefore, if w is a bad solution with f ( w ) much larger than f ( w (cid:63) ) , then w cannot be a stationary point because f decreases when we move from w to (1 − η ) w + ηw (cid:63) .
Revised: Therefore, if w is a bad solution with f ( w ) much larger than f ( w (cid:63) ) , then w cannot be a stationary point because f decreases when we move from w to (1 − η ) w + ηw (cid:63) .

Original: Remark 1.4.
Revised: Remark 1.4.

Original: The technical overview for robust sparse PCA follows a similar high-level approach, but is somewhat more technical.
Revised: The technical overview for robust sparse PCA follows a similar high-level approach, but is somewhat more technical.

Original: It is deferred to Section 4.
Revised: It is deferred to Section 4.

Original: Roadmap.
Revised: Roadmap.

Original: In Section 2, we introduce basic notations and the deterministic stability conditions that we require on the good samples.
Revised: In Section 2, we introduce basic notations and the deterministic stability conditions that we require on the good samples.

Original: We present our algorithms and analysis for robust sparse mean estimation in Section 3 and robust sparse PCA in Section 4.
Revised: We present our algorithms and analysis for robust sparse mean estimation in Section 3 and robust sparse PCA in Section 4.

Original: In Section 5, we evaluate our algorithm on synthetic datasets and show that it achieves good statistical accuracy under various noise models.
Revised: In Section 5, we evaluate our algorithm on synthetic datasets and show that it achieves good statistical accuracy under various noise models.

Original:  Preliminaries and Background
Revised:  Preliminaries and Background

Original: Notation.
Revised: Notation.

Original: For a positive integer n , let [ n ] = { 1 , . . . , n } .
Revised: For a positive integer n , let [ n ] = { 1 , . . . , n } .

Original: For a vector v , we use (cid:107) v (cid:107) 0 , (cid:107) v (cid:107) 1 , (cid:107) v (cid:107) 2 , and (cid:107) v (cid:107) ∞ for the number of non-zeros, the (cid:96) 1 , (cid:96) 2 , and (cid:96) ∞ norm of v respectively.
Revised: For a vector v , we use (cid:107) v (cid:107) 0 , (cid:107) v (cid:107) 1 , (cid:107) v (cid:107) 2 , and (cid:107) v (cid:107) ∞ for the number of non-zeros, the (cid:96) 1 , (cid:96) 2 , and (cid:96) ∞ norm of v respectively.

Original: Let I be the identity matrix.
Revised: Let I be the identity matrix.

Original: For a matrix A , we use (cid:107) A (cid:107) 2 , (cid:107) A (cid:107) F , tr( A ) for the spectral norm, Frobenius norm, and trace of A respectively.
Revised: For a matrix A , we use (cid:107) A (cid:107) 2 , (cid:107) A (cid:107) F , tr( A ) for the spectral norm, Frobenius norm, and trace of A respectively.

Original: For two vectors x, y , let x (cid:62) y denote their inner product.
Revised: For two vectors x, y , let x (cid:62) y denote their inner product.

Original: For two matrices
Revised: For two matrices

Original: A, B , we use A • B = tr( A (cid:62) B ) for their entrywise inner product.
Revised: A, B , we use A • B = tr( A (cid:62) B ) for their entrywise inner product.

Original: A matrix A is said to be positive semideﬁnite (PSD) if x (cid:62) Ax ≥ 0 for all x .
Revised: A matrix A is said to be positive semideﬁnite (PSD) if x (cid:62) Ax ≥ 0 for all x .

Original: We write A (cid:22) B iff ( B − A ) is PSD.
Revised: We write A (cid:22) B iff ( B − A ) is PSD.

Original: For a vector w ∈ R n , let diag( w ) ∈ R n × n denote a diagonal matrix with w on the diagonal.
Revised: For a vector w ∈ R n , let diag( w ) ∈ R n × n denote a diagonal matrix with w on the diagonal.

Original: For a matrix A ∈ R n × n , let diag( A ) ∈ R n denote a column vector with the diagonal of A .
Revised: For a matrix A ∈ R n × n , let diag( A ) ∈ R n denote a column vector with the diagonal of A .

Original: For a vector v ∈ R d and a set S ⊆
Revised: For a vector v ∈ R d and a set S ⊆

Original: [ d ] , we write v S ∈ R d for a vector that is equal to v on S and zero everywhere else.
Revised: [ d ] , we write v S ∈ R d for a vector that is equal to v on S and zero everywhere else.

Original: Similarly, for a matrix A ∈ R d × d and a set S ⊆ ([ d ] × [ d ]) , we write A S for a matrix that is equal to A on S and zero everywhere else.
Revised: Similarly, for a matrix A ∈ R d × d and a set S ⊆ ([ d ] × [ d ]) , we write A S for a matrix that is equal to A on S and zero everywhere else.

Original: For a vector v , we deﬁne (cid:107) v (cid:107) 2 ,k = max | S | = k (cid:107) v S (cid:107) 2 to be the maximum (cid:96) 2 -norm of any k entries of v .
Revised: For a vector v , we deﬁne (cid:107) v (cid:107) 2 ,k = max | S | = k (cid:107) v S (cid:107) 2 to be the maximum (cid:96) 2 -norm of any k entries of v .

Original: For a matrix A , we deﬁne (cid:107) A (cid:107) F,k 2 to be the maximum Frobenius norm of any k 2 entries of A .
Revised: For a matrix A , we deﬁne (cid:107) A (cid:107) F,k 2 to be the maximum Frobenius norm of any k 2 entries of A .

Original: Moreover, we deﬁne (cid:107) A (cid:107) F,k,k to be the maximum Frobenius norm of any k 2 entries with the extra requirement that these entries must be chosen from k rows with k entries in each row.
Revised: Moreover, we deﬁne (cid:107) A (cid:107) F,k,k to be the maximum Frobenius norm of any k 2 entries with the extra requirement that these entries must be chosen from k rows with k entries in each row.

Original: Formally, (cid:107) A (cid:107) F,k 2 = max | Q | = k 2 (cid:107) A Q (cid:107) F and
Revised: Formally, (cid:107) A (cid:107) F,k 2 = max | Q | = k 2 (cid:107) A Q (cid:107) F and

Original: [Equation]
Revised: [Equation]

Original: 22 ,k where A i is i -th row of A .
Revised: 22 ,k where A i is i -th row of A .

Original: Sample Reweighting Framework.
Revised: Sample Reweighting Framework.

Original: We use n for the number of samples, d for the dimension, and (cid:15) for the fraction of corrupted samples.
Revised: We use n for the number of samples, d for the dimension, and (cid:15) for the fraction of corrupted samples.

Original: For sparse estimation, we use k for the sparsity of the ground-truth parameters.
Revised: For sparse estimation, we use k for the sparsity of the ground-truth parameters.

Original: We use G (cid:63) for the original set of n good samples.
Revised: We use G (cid:63) for the original set of n good samples.

Original: We use S =
Revised: We use S =

Original: G ∪ B for the input samples after the adversary replaced (cid:15) -fraction of G (cid:63) , where G ⊂ G (cid:63) is the set of remaining good samples and B is the set of bad samples (outliers) added by the adversary.
Revised: G ∪ B for the input samples after the adversary replaced (cid:15) -fraction of G (cid:63) , where G ⊂ G (cid:63) is the set of remaining good samples and B is the set of bad samples (outliers) added by the adversary.

Original: Note that | G | =
Revised: Note that | G | =

Original: (1 − (cid:15) ) n and | B | =
Revised: (1 − (cid:15) ) n and | B | =

Original: (cid:15)n .
Revised: (cid:15)n .

Original: Given n samples X 1 , . . .
Revised: Given n samples X 1 , . . .

Original: , X n , we write X ∈ R d × n as the sample matrix where the i -th column is X i .
Revised: , X n , we write X ∈ R d × n as the sample matrix where the i -th column is X i .

Original: For a weight vector w ∈ R n , we use µ w =
Revised: For a weight vector w ∈ R n , we use µ w =

Original: Xw = (cid:80)
Revised: Xw = (cid:80)

Original: i w i
Revised: i w i

Original: X
Revised: X

Original: i for the weighted empirical mean and Σ w = X diag( w ) X − µ w µ (cid:62) w
Revised: i for the weighted empirical mean and Σ w = X diag( w ) X − µ w µ (cid:62) w

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: i w i
Revised: i w i

Original: ( X i − µ w )( X i − µ w ) (cid:62) for the weighted empirical covariance.
Revised: ( X i − µ w )( X i − µ w ) (cid:62) for the weighted empirical covariance.

Original: Let ∆ n,(cid:15) be the convex hull of all uniform distributions over subsets S ⊆
Revised: Let ∆ n,(cid:15) be the convex hull of all uniform distributions over subsets S ⊆

Original: [ n ] of size | S | =
Revised: [ n ] of size | S | =

Original: (1 − (cid:15) ) n : ∆ n,(cid:15) = { w ∈ R n : (cid:107) w (cid:107) 1 = 1 and 0 ≤ w i ≤ 1 (1 − (cid:15) ) n ∀ i } , In other words, every w ∈ ∆ n,(cid:15) corresponds to a fractional set of (1 − (cid:15) ) n samples.
Revised: (1 − (cid:15) ) n : ∆ n,(cid:15) = { w ∈ R n : (cid:107) w (cid:107) 1 = 1 and 0 ≤ w i ≤ 1 (1 − (cid:15) ) n ∀ i } , In other words, every w ∈ ∆ n,(cid:15) corresponds to a fractional set of (1 − (cid:15) ) n samples.

Original: We use w (cid:63) to denote the uniform distribution on G (the remaining good samples in S ).
Revised: We use w (cid:63) to denote the uniform distribution on G (the remaining good samples in S ).

Original: Deterministic Stability Conditions.
Revised: Deterministic Stability Conditions.

Original: For robust sparse mean estimation and robust sparse PCA, we require the following conditions respectively.
Revised: For robust sparse mean estimation and robust sparse PCA, we require the following conditions respectively.

Original: Deﬁnition 2.1 (Stability Conditions for Sparse Mean) .
Revised: Deﬁnition 2.1 (Stability Conditions for Sparse Mean) .

Original: A set of n samples G (cid:63) =
Revised: A set of n samples G (cid:63) =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 is said to be ( k, (cid:15), δ ) -stable
Revised: =1 is said to be ( k, (cid:15), δ ) -stable

Original: (w.r.t.
Revised: (w.r.t.

Original: a distribution with mean µ ) iff for any weight vector w ∈ ∆ n, 2 (cid:15) , we have (cid:107) µ w − µ (cid:107) 2 ,k ≤ δ and (cid:107) Σ w −
Revised: a distribution with mean µ ) iff for any weight vector w ∈ ∆ n, 2 (cid:15) , we have (cid:107) µ w − µ (cid:107) 2 ,k ≤ δ and (cid:107) Σ w −

Original: I (cid:107) F,k,k ≤ δ 2 /(cid:15) , where µ w and Σ w are the weighted empirical mean and covariance matrix respectively, and the (cid:107)·(cid:107) F,k,k norm is deﬁned in Equation (2) .
Revised: I (cid:107) F,k,k ≤ δ 2 /(cid:15) , where µ w and Σ w are the weighted empirical mean and covariance matrix respectively, and the (cid:107)·(cid:107) F,k,k norm is deﬁned in Equation (2) .

Original: Deﬁnition 2.2 (Stability Conditions for Sparse PCA) .
Revised: Deﬁnition 2.2 (Stability Conditions for Sparse PCA) .

Original: A set of n samples G (cid:63) =
Revised: A set of n samples G (cid:63) =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 is ( k, (cid:15), δ ) - stable (w.r.t.
Revised: =1 is ( k, (cid:15), δ ) - stable (w.r.t.

Original: a centered distribution with covariance I + ρvv (cid:62) ) iff for any weight vector w ∈ ∆ n, 2 (cid:15) , (cid:13)(cid:13)
Revised: a centered distribution with covariance I + ρvv (cid:62) ) iff for any weight vector w ∈ ∆ n, 2 (cid:15) , (cid:13)(cid:13)

Original: M w − ( I + ρvv (cid:62) )
Revised: M w − ( I + ρvv (cid:62) )

Original: (cid:13)(cid:13) F, 2 k 2 ≤ δ , where M w = (cid:80) i w i
Revised: (cid:13)(cid:13) F, 2 k 2 ≤ δ , where M w = (cid:80) i w i

Original: X i X (cid:62)
Revised: X i X (cid:62)

Original: i and the (cid:107)·(cid:107) F, 2 k 2 norm is deﬁned in
Revised: i and the (cid:107)·(cid:107) F, 2 k 2 norm is deﬁned in

Original: Equation (2) .
Revised: Equation (2) .

Original: First-Order Stationary Points.
Revised: First-Order Stationary Points.

Original: We give a formal deﬁnition of the notion of (approximate) ﬁrstorder stationary point that we use in this paper.
Revised: We give a formal deﬁnition of the notion of (approximate) ﬁrstorder stationary point that we use in this paper.

Original: Deﬁnition 2.3 (Approximate Stationary Points) .
Revised: Deﬁnition 2.3 (Approximate Stationary Points) .

Original: Fix a convex set K and a differentiable function f .
Revised: Fix a convex set K and a differentiable function f .

Original: For γ ≥ 0 , we say that x ∈ K is a γ -stationary point of f iff the following condition holds: For any unit vector u where x + αu ∈ K for some α > 0 , we have u (cid:62) ∇ f ( x ) ≥ − γ .
Revised: For γ ≥ 0 , we say that x ∈ K is a γ -stationary point of f iff the following condition holds: For any unit vector u where x + αu ∈ K for some α > 0 , we have u (cid:62) ∇ f ( x ) ≥ − γ .

Original: We note that the objective functions studied in this paper are not everywhere differentiable.
Revised: We note that the objective functions studied in this paper are not everywhere differentiable.

Original: This is because, taking the (cid:107)·(cid:107) F,k,k norm as an example, there can be ties in choosing the largest kentries.
Revised: This is because, taking the (cid:107)·(cid:107) F,k,k norm as an example, there can be ties in choosing the largest kentries.

Original: When the function f is not differentiable, we use ∇ f informally to denote an element of the sub-differential.
Revised: When the function f is not differentiable, we use ∇ f informally to denote an element of the sub-differential.

Original: We will show in Appendix C that, while f is not differentiable, it does have a nonempty subdifferential, as it can be written as the pointwise maximum of differentiable functions.
Revised: We will show in Appendix C that, while f is not differentiable, it does have a nonempty subdifferential, as it can be written as the pointwise maximum of differentiable functions.

Original:  Robust Sparse Mean Estimation
Revised:  Robust Sparse Mean Estimation

Original: In this section, we present our non-convex approach for robust sparse mean estimation.
Revised: In this section, we present our non-convex approach for robust sparse mean estimation.

Original: We will optimize the following objective, where (cid:107)·(cid:107) F,k,k is deﬁned in Equation (2):
Revised: We will optimize the following objective, where (cid:107)·(cid:107) F,k,k is deﬁned in Equation (2):

Original: [Equation]
Revised: [Equation]

Original: We will show that the objective function (3) has no bad stationary points (Theorem 3.1).
Revised: We will show that the objective function (3) has no bad stationary points (Theorem 3.1).

Original: In other words, every ﬁrst-order stationary point of f yields a good solution for robust sparse mean estimation.
Revised: In other words, every ﬁrst-order stationary point of f yields a good solution for robust sparse mean estimation.

Original: Our algorithm is stated in Algorithm 1.
Revised: Our algorithm is stated in Algorithm 1.

Original: As a consequence of our landscape result (Theorem 3.1), we know that Algorithm 1 works no matter how we ﬁnd a stationary point of f (because any stationary point works), so we intentionally did not specify how to ﬁnd such a point.
Revised: As a consequence of our landscape result (Theorem 3.1), we know that Algorithm 1 works no matter how we ﬁnd a stationary point of f (because any stationary point works), so we intentionally did not specify how to ﬁnd such a point.

Original: As a simple illustration, we show that (projected) gradient descent can be used to minimize f .
Revised: As a simple illustration, we show that (projected) gradient descent can be used to minimize f .

Original: The convergence analysis and iteration complexity are provided in Appendix C.
Revised: The convergence analysis and iteration complexity are provided in Appendix C.

Original: Input: k > 0 , 0 < (cid:15) < (cid:15) 0 , and an (cid:15) -corrupted set of samples ( X i ) ni
Revised: Input: k > 0 , 0 < (cid:15) < (cid:15) 0 , and an (cid:15) -corrupted set of samples ( X i ) ni

Original: =1 drawn from a distribution with k -sparse mean µ .
Revised: =1 drawn from a distribution with k -sparse mean µ .

Original: Output
Revised: Output

Original: Formally, we ﬁrst prove that Algorithm 1 can output a vector (cid:98) µ ∈ R d
Revised: Formally, we ﬁrst prove that Algorithm 1 can output a vector (cid:98) µ ∈ R d

Original: that is close to µ in (cid:107)·(cid:107) 2 ,k norm, as long as the good samples satisﬁes the stability condition in Deﬁnition 2.1.
Revised: that is close to µ in (cid:107)·(cid:107) 2 ,k norm, as long as the good samples satisﬁes the stability condition in Deﬁnition 2.1.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Fix k > 0 , 0 < (cid:15) < (cid:15) 0 , and δ > (cid:15) .
Revised: Fix k > 0 , 0 < (cid:15) < (cid:15) 0 , and δ > (cid:15) .

Original: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ ) -stable
Revised: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ ) -stable

Original: (as in Deﬁnition 2.1) w.r.t.
Revised: (as in Deﬁnition 2.1) w.r.t.

Original: a distribution with unknown k -sparse mean µ ∈ R d .
Revised: a distribution with unknown k -sparse mean µ ∈ R d .

Original: Let S =
Revised: Let S =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 be an (cid:15) -corrupted version of G (cid:63) .
Revised: =1 be an (cid:15) -corrupted version of G (cid:63) .

Original: Let f ( w )
Revised: Let f ( w )

Original: =
Revised: =

Original: (cid:107) Σ w −
Revised: (cid:107) Σ w −

Original: I (cid:107) F,k,k .
Revised: I (cid:107) F,k,k .

Original: Let γ
Revised: Let γ

Original: =
Revised: =

Original: O ( n 1 / 2 δ 2 (cid:15) − 3 / 2 ) .
Revised: O ( n 1 / 2 δ 2 (cid:15) − 3 / 2 ) .

Original: Then, for any w ∈ ∆ n,(cid:15) that is a γ -stationary point of f ( w ) , we have (cid:107) µ w − µ (cid:107) 2 ,k
Revised: Then, for any w ∈ ∆ n,(cid:15) that is a γ -stationary point of f ( w ) , we have (cid:107) µ w − µ (cid:107) 2 ,k

Original: = O ( δ ) .
Revised: = O ( δ ) .

Original: Once we have a vector µ w that is O ( δ ) -close to µ in (cid:107)·(cid:107) 2 ,k norm, we can guarantee that a truncated version of µ w (the output (cid:98) µ of Algorithm 1) is O ( δ ) -close to µ in the (cid:96) 2 -norm:
Revised: Once we have a vector µ w that is O ( δ ) -close to µ in (cid:107)·(cid:107) 2 ,k norm, we can guarantee that a truncated version of µ w (the output (cid:98) µ of Algorithm 1) is O ( δ ) -close to µ in the (cid:96) 2 -norm:

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: Fix two vectors x, y with (cid:107) x (cid:107) 0 ≤ k and (cid:107) x − y (cid:107) 2 ,k ≤ δ .
Revised: Fix two vectors x, y with (cid:107) x (cid:107) 0 ≤ k and (cid:107) x − y (cid:107) 2 ,k ≤ δ .

Original: Let z be a vector that keeps the k entries of y with largest absolute values and sets the rest to 0 .
Revised: Let z be a vector that keeps the k entries of y with largest absolute values and sets the rest to 0 .

Original: We have (cid:107) x − z (cid:107) 2 ≤ √ 5 δ .
Revised: We have (cid:107) x − z (cid:107) 2 ≤ √ 5 δ .

Original: Theorem 1.2 follows immediately from Theorem 3.1 and Lemma 3.2.
Revised: Theorem 1.2 follows immediately from Theorem 3.1 and Lemma 3.2.

Original: We can apply Theorem 1.2 to get an end-to-end result for subgaussian distributions.
Revised: We can apply Theorem 1.2 to get an end-to-end result for subgaussian distributions.

Original: We show that the required stability conditions are satisﬁed with a small number of samples.
Revised: We show that the required stability conditions are satisﬁed with a small number of samples.

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: Fix k > 0 and 0 < (cid:15) < (cid:15) 0 .
Revised: Fix k > 0 and 0 < (cid:15) < (cid:15) 0 .

Original: Let G (cid:63) be a set of n samples that are drawn i.i.d.
Revised: Let G (cid:63) be a set of n samples that are drawn i.i.d.

Original: from a subgaussian distribution with mean µ and covariance I .
Revised: from a subgaussian distribution with mean µ and covariance I .

Original: If n = Ω( k 2 log d/(cid:15) 2 ) , then with probability at least 1 − exp( − Ω( k 2 log d )) , G (cid:63) is ( k, (cid:15), δ ) -stable (as in Deﬁnition 2.1) for δ
Revised: If n = Ω( k 2 log d/(cid:15) 2 ) , then with probability at least 1 − exp( − Ω( k 2 log d )) , G (cid:63) is ( k, (cid:15), δ ) -stable (as in Deﬁnition 2.1) for δ

Original: = O ( (cid:15) log(1 /(cid:15) )) .
Revised: = O ( (cid:15) log(1 /(cid:15) )) .

Original: Combining Theorem 1.2 and Lemma 3.3, we know that given an (cid:15) -corrupted set of O ( k 2 log d/(cid:15) 2 ) samples drawn from a subgaussian distribution with k -sparse mean µ , the output of Algorithm 1 is
Revised: Combining Theorem 1.2 and Lemma 3.3, we know that given an (cid:15) -corrupted set of O ( k 2 log d/(cid:15) 2 ) samples drawn from a subgaussian distribution with k -sparse mean µ , the output of Algorithm 1 is

Original: O ( (cid:15) (cid:112) log(1 /(cid:15) )) -close to µ in (cid:96) 2 -norm.
Revised: O ( (cid:15) (cid:112) log(1 /(cid:15) )) -close to µ in (cid:96) 2 -norm.

Original: In the rest of this section, we will prove Theorem 3.1.
Revised: In the rest of this section, we will prove Theorem 3.1.

Original: Omitted proofs in this section are in Appendix A.
Revised: Omitted proofs in this section are in Appendix A.

Original: We start with some intuition on why we choose our objective function (3).
Revised: We start with some intuition on why we choose our objective function (3).

Original: We would like to design f ( w ) =
Revised: We would like to design f ( w ) =

Original: g (Σ w − I ) to satisfy the following properties:
Revised: g (Σ w − I ) to satisfy the following properties:

Original: g (Σ w − I ) is an upper bound on v (cid:62) (Σ w − I ) v for all k -sparse unit vectors v ∈ R d .
Revised: g (Σ w − I ) is an upper bound on v (cid:62) (Σ w − I ) v for all k -sparse unit vectors v ∈ R d .

Original: This way, a small objective value implies that (cid:107) µ w − µ (cid:107) 2 ,k is small.
Revised: This way, a small objective value implies that (cid:107) µ w − µ (cid:107) 2 ,k is small.

Original: g (Σ w (cid:63) − I ) is small for w (cid:63) (the uniform distribution on G ).
Revised: g (Σ w (cid:63) − I ) is small for w (cid:63) (the uniform distribution on G ).

Original: This guarantees that a good w exists.
Revised: This guarantees that a good w exists.

Original: Triangle inequality on g .
Revised: Triangle inequality on g .

Original: This allows us to upper bound the objective value when we move w toward w (cid:63) by the sum of g ( · ) of each term on the right-hand side:
Revised: This allows us to upper bound the objective value when we move w toward w (cid:63) by the sum of g ( · ) of each term on the right-hand side:

Original: [Equation]
Revised: [Equation]

Original: g ( uu (cid:62) ) is close to g ( vv (cid:62) ) where v keeps only the k largest entries of u .
Revised: g ( uu (cid:62) ) is close to g ( vv (cid:62) ) where v keeps only the k largest entries of u .

Original: We want to approximate µ in (cid:107)·(cid:107) 2 ,k norm, so intuitively g (Σ w − I ) should depend only on the largest k entries of ( µ w − µ ) .
Revised: We want to approximate µ in (cid:107)·(cid:107) 2 ,k norm, so intuitively g (Σ w − I ) should depend only on the largest k entries of ( µ w − µ ) .

Original: Our choice of f ( w ) =
Revised: Our choice of f ( w ) =

Original: g (Σ w − I ) =
Revised: g (Σ w − I ) =

Original: (cid:107) Σ −
Revised: (cid:107) Σ −

Original: I
Revised: I

Original: (cid:107) F,k,k is motivated by (and satisﬁes) all these properties.
Revised: (cid:107) F,k,k is motivated by (and satisﬁes) all these properties.

Original: Lemma 3.4.
Revised: Lemma 3.4.

Original: Fix A ∈ R d × d .
Revised: Fix A ∈ R d × d .

Original: We have (cid:12)(cid:12) v (cid:62)
Revised: We have (cid:12)(cid:12) v (cid:62)

Original: Av (cid:12)(cid:12) ≤ (cid:107) A (cid:107) F,k,k for any k -sparse unit vector v ∈ R d .
Revised: Av (cid:12)(cid:12) ≤ (cid:107) A (cid:107) F,k,k for any k -sparse unit vector v ∈ R d .

Original: Lemma 3.5.
Revised: Lemma 3.5.

Original: For any vector v ∈ R d , (cid:13)(cid:13) vv (cid:62) (cid:13)(cid:13) F,k,k = (cid:107) v (cid:107) 22 ,k .
Revised: For any vector v ∈ R d , (cid:13)(cid:13) vv (cid:62) (cid:13)(cid:13) F,k,k = (cid:107) v (cid:107) 22 ,k .

Original: We now continue to present key technical lemmas for proving our main structural result (Theorem 3.1).
Revised: We now continue to present key technical lemmas for proving our main structural result (Theorem 3.1).

Original: Lemma 3.6 gives the weighted empirical covariance for a convex combination of two weight vectors.
Revised: Lemma 3.6 gives the weighted empirical covariance for a convex combination of two weight vectors.

Original: Lemma 3.6.
Revised: Lemma 3.6.

Original: Fix n samples X 1 , . . .
Revised: Fix n samples X 1 , . . .

Original: X n ∈ R d .
Revised: X n ∈ R d .

Original: Let w, (cid:98) w ∈ R n be two non-negative weight vectors with (cid:107) w (cid:107) 1 = (cid:107) (cid:98) w (cid:107) 1 = 1 .
Revised: Let w, (cid:98) w ∈ R n be two non-negative weight vectors with (cid:107) w (cid:107) 1 = (cid:107) (cid:98) w (cid:107) 1 = 1 .

Original: For any α, β ≥ 0 with α + β = 1 , letting w = αw + β (cid:98) w , we have
Revised: For any α, β ≥ 0 with α + β = 1 , letting w = αw + β (cid:98) w , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Because w = αw + β (cid:98) w and µ w is linear in w , we have µ w =
Revised: Because w = αw + β (cid:98) w and µ w is linear in w , we have µ w =

Original: αµ w + βµ (cid:98) w .
Revised: αµ w + βµ (cid:98) w .

Original: The lemma follows from the following calculations:
Revised: The lemma follows from the following calculations:

Original: [Equation]
Revised: [Equation]

Original: The last step uses α − α 2 = β − β 2 = αβ as α + β = 1 .
Revised: The last step uses α − α 2 = β − β 2 = αβ as α + β = 1 .

Original: Let w (cid:63) denote the uniform distribution on G , i.e., w (cid:63)i = 1 (1 − (cid:15) )
Revised: Let w (cid:63) denote the uniform distribution on G , i.e., w (cid:63)i = 1 (1 − (cid:15) )

Original: n if i ∈ G and w (cid:63)i = 0 otherwise.
Revised: n if i ∈ G and w (cid:63)i = 0 otherwise.

Original: By
Revised: By

Original: Lemma 3.6 for any w , if we move toward w (cid:63) , we have
Revised: Lemma 3.6 for any w , if we move toward w (cid:63) , we have

Original: [Equation]
Revised: [Equation]

Original: We will show that we can essentially ignore the last rank-one term using Lemma 3.7.
Revised: We will show that we can essentially ignore the last rank-one term using Lemma 3.7.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: Let G (cid:63) be a ( k, (cid:15), δ )
Revised: Let G (cid:63) be a ( k, (cid:15), δ )

Original: -stable set of samples with respect to the ground-truth distribution with 0 < (cid:15) ≤ δ .
Revised: -stable set of samples with respect to the ground-truth distribution with 0 < (cid:15) ≤ δ .

Original: Let S be an (cid:15) -corrupted version of G (cid:63) .
Revised: Let S be an (cid:15) -corrupted version of G (cid:63) .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: We are now ready to prove our main result (Theorem 3.1).
Revised: We are now ready to prove our main result (Theorem 3.1).

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: Fix any weight vector w ∈ ∆ n,(cid:15) .
Revised: Fix any weight vector w ∈ ∆ n,(cid:15) .

Original: We will show that if w is a bad solution, then f ( w ) decreases if w moves toward w (cid:63) , so w cannot be a stationary point.
Revised: We will show that if w is a bad solution, then f ( w ) decreases if w moves toward w (cid:63) , so w cannot be a stationary point.

Original: Let c 1 be the constant in O ( · ) in Lemma 3.7.
Revised: Let c 1 be the constant in O ( · ) in Lemma 3.7.

Original: By Lemma 3.7, if (cid:107) µ w − µ (cid:107) 2 ,k ≥ c 2 δ for a sufﬁciently large constant c 2 , then (cid:107) Σ w −
Revised: By Lemma 3.7, if (cid:107) µ w − µ (cid:107) 2 ,k ≥ c 2 δ for a sufﬁciently large constant c 2 , then (cid:107) Σ w −

Original: I (cid:107) F,k,k ≥ ( c 22 4 − c 1 ) δ 2 (cid:15) =
Revised: I (cid:107) F,k,k ≥ ( c 22 4 − c 1 ) δ 2 (cid:15) =

Original: Ω( δ 2 (cid:15) ) .
Revised: Ω( δ 2 (cid:15) ) .

Original: By Lemma 3.6, Σ
Revised: By Lemma 3.6, Σ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the triangle inequality for (cid:107)·(cid:107) F,k,k , we have
Revised: Using the triangle inequality for (cid:107)·(cid:107) F,k,k , we have

Original: [Equation]
Revised: [Equation]

Original: We know that (cid:107) Σ w
Revised: We know that (cid:107) Σ w

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ≤ δ
Revised: ≤ δ

Original: [Equation]
Revised: [Equation]

Original: by the stability condition in Deﬁnition 2.1.
Revised: by the stability condition in Deﬁnition 2.1.

Original: By Lemma 3.7 and (cid:107) Σ w −
Revised: By Lemma 3.7 and (cid:107) Σ w −

Original: I
Revised: I

Original: (cid:107) F,k,k
Revised: (cid:107) F,k,k

Original: = Ω( δ 2 /(cid:15) ) , we can show that for all 0 < η < 1 ,
Revised: = Ω( δ 2 /(cid:15) ) , we can show that for all 0 < η < 1 ,

Original: [Equation]
Revised: [Equation]

Original: The last step uses ( 12 − 4 (cid:15) )
Revised: The last step uses ( 12 − 4 (cid:15) )

Original: (cid:107) Σ w −
Revised: (cid:107) Σ w −

Original: I
Revised: I

Original: (cid:107) 2 ≥ (4 c 1 +1)
Revised: (cid:107) 2 ≥ (4 c 1 +1)

Original: δ 2 (cid:15) which holds if (cid:15) ≤ 1 / 10 and c 22 ≥ 164 c 1 +40 .
Revised: δ 2 (cid:15) which holds if (cid:15) ≤ 1 / 10 and c 22 ≥ 164 c 1 +40 .

Original: It follows immediately that w cannot be a stationary point.
Revised: It follows immediately that w cannot be a stationary point.

Original: Let u = w (cid:63) − w (cid:107) w (cid:63) − w (cid:107) 2 and h = η (cid:107) w (cid:63) − w (cid:107) 2 .
Revised: Let u = w (cid:63) − w (cid:107) w (cid:63) − w (cid:107) 2 and h = η (cid:107) w (cid:63) − w (cid:107) 2 .

Original: We have w + hu
Revised: We have w + hu

Original: =
Revised: =

Original: (1 − η ) w + ηw (cid:63) ∈ ∆ n,(cid:15) because ∆ n,(cid:15) is convex.
Revised: (1 − η ) w + ηw (cid:63) ∈ ∆ n,(cid:15) because ∆ n,(cid:15) is convex.

Original: Since (cid:107) w (cid:63) − w (cid:107) 

Revised: Since (cid:107) w (cid:63) − w (cid:107)

Original: = O ( (cid:112) (cid:15)/n ) ,
Revised: = O ( (cid:112) (cid:15)/n ) ,

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By Deﬁnition 2.3, we know w cannot be a γ -stationary point of f for some γ
Revised: By Deﬁnition 2.3, we know w cannot be a γ -stationary point of f for some γ

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: − 3 / 2 ) .
Revised: − 3 / 2 ) .

Original:  Robust Sparse PCA
Revised:  Robust Sparse PCA

Original: We consider a spiked covariance model for sparse PCA.
Revised: We consider a spiked covariance model for sparse PCA.

Original: In this model, there is a direction v ∈ R d with at most k nonzero entries.
Revised: In this model, there is a direction v ∈ R d with at most k nonzero entries.

Original: The good samples are drawn from a ground-truth distribution with covariance Σ
Revised: The good samples are drawn from a ground-truth distribution with covariance Σ

Original: = I + ρvv (cid:62) , where ρ > 0 is a parameter that intuitively measures the strength of the signal.
Revised: = I + ρvv (cid:62) , where ρ > 0 is a parameter that intuitively measures the strength of the signal.

Original: We consider the more interesting case when ρ ≤ 1 (if ρ is larger the problem becomes easier).
Revised: We consider the more interesting case when ρ ≤ 1 (if ρ is larger the problem becomes easier).

Original: To solve the sparse PCA problem, we consider the following optimization problem, where M w = (cid:80)
Revised: To solve the sparse PCA problem, we consider the following optimization problem, where M w = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: X i X
Revised: X i X

Original: [Equation]
Revised: [Equation]

Original: i and
Revised: i and

Original: [Equation]
Revised: [Equation]

Original: The objective function minimizes the Frobenius norm of the largest 2 k 2 entries of a reweighted second-moment matrix M w .
Revised: The objective function minimizes the Frobenius norm of the largest 2 k 2 entries of a reweighted second-moment matrix M w .

Original: Note that f ( w ) is actually convex in w , because the matrix M w is linear in w and the (cid:107)·(cid:107) F, 2 k 2 norm is convex.
Revised: Note that f ( w ) is actually convex in w , because the matrix M w is linear in w and the (cid:107)·(cid:107) F, 2 k 2 norm is convex.

Original: Let R be the support of vv (cid:62) .
Revised: Let R be the support of vv (cid:62) .

Original: Intuitively, the k 2 entries in R could be large due to spiked covariance.
Revised: Intuitively, the k 2 entries in R could be large due to spiked covariance.

Original: By minimizing the norm of the largest 2 k 2 entries, we hope to make the entries outside of R very small.
Revised: By minimizing the norm of the largest 2 k 2 entries, we hope to make the entries outside of R very small.

Original: Our algorithm is given in Algorithm 2.
Revised: Our algorithm is given in Algorithm 2.

Original: Algorithm 2: Robust sparse PCA.
Revised: Algorithm 2: Robust sparse PCA.

Original: ﬁrst
Revised: ﬁrst

Original: = M w −
Revised: = M w −

Original: I
Revised: I

Original: Q
Revised: Q

Original: the k 2 entries of A with largest magnitude.
Revised: the k 2 entries of A with largest magnitude.

Original: of
Revised: of

Original: Q
Revised: Q

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let 0 < ρ ≤ 1 , 0 < (cid:15) < (cid:15) 0 , and δ > (cid:15) .
Revised: Let 0 < ρ ≤ 1 , 0 < (cid:15) < (cid:15) 0 , and δ > (cid:15) .

Original: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ ) - stable (as in Deﬁnition 2.2) w.r.t.
Revised: Let G (cid:63) be a set of n samples that is ( k, (cid:15), δ ) - stable (as in Deﬁnition 2.2) w.r.t.

Original: a centered distribution with covariance I + ρvv (cid:62) for an unknown k -sparse unit vector v ∈ R d .
Revised: a centered distribution with covariance I + ρvv (cid:62) for an unknown k -sparse unit vector v ∈ R d .

Original: Let S =
Revised: Let S =

Original: ( X i ) ni
Revised: ( X i ) ni

Original: =1 be an (cid:15) -corrupted version of G (cid:63) .
Revised: =1 be an (cid:15) -corrupted version of G (cid:63) .

Original: Algorithm 2 outputs a vector u such that (cid:13) (cid:13) uu (cid:62) − vv (cid:62) (cid:13) (cid:13) F
Revised: Algorithm 2 outputs a vector u such that (cid:13) (cid:13) uu (cid:62) − vv (cid:62) (cid:13) (cid:13) F

Original: = O ( (cid:112) δ/ρ ) .
Revised: = O ( (cid:112) δ/ρ ) .

Original: Theorem 1.3 is an immediate corollary of Theorem 4.1.
Revised: Theorem 1.3 is an immediate corollary of Theorem 4.1.

Original: We can apply Theorem 4.1 to get an end-to-end result for subgaussian distributions.
Revised: We can apply Theorem 4.1 to get an end-to-end result for subgaussian distributions.

Original: Algorithmrequires the stability conditions (Deﬁnition 2.2) of the original good samples G (cid:63) .
Revised: Algorithmrequires the stability conditions (Deﬁnition 2.2) of the original good samples G (cid:63) .

Original: We show that these conditions are satisﬁed with a small number of samples.
Revised: We show that these conditions are satisﬁed with a small number of samples.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: Let 0 < ρ ≤ 1 and 0 < (cid:15) < (cid:15) 0 .
Revised: Let 0 < ρ ≤ 1 and 0 < (cid:15) < (cid:15) 0 .

Original: Let D be a centered subgaussian distribution with covariance I + ρvv (cid:62) for a k -sparse unit vector v ∈ R d .
Revised: Let D be a centered subgaussian distribution with covariance I + ρvv (cid:62) for a k -sparse unit vector v ∈ R d .

Original: Let G (cid:63) be a set of n = Ω( k 2 log d/δ 2 ) samples drawn from D .
Revised: Let G (cid:63) be a set of n = Ω( k 2 log d/δ 2 ) samples drawn from D .

Original: Then then with probability at least 1 − exp( − Ω( k 2 log d )) , G (cid:63) is ( k, (cid:15), δ ) - stable (as in Deﬁnition 2.2) w.r.t.
Revised: Then then with probability at least 1 − exp( − Ω( k 2 log d )) , G (cid:63) is ( k, (cid:15), δ ) - stable (as in Deﬁnition 2.2) w.r.t.

Original: D for δ =
Revised: D for δ =

Original: O ( (cid:15) log(1 /(cid:15) )) .
Revised: O ( (cid:15) log(1 /(cid:15) )) .

Original: Combining Theorem 4.1 and Lemma 4.2, given as input an (cid:15) -corrupted set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from a centered subgaussian distribution with covariance I + ρvv (cid:62) , Algorithmreturns a vector u with (cid:13)(cid:13) uu (cid:62) − vv (cid:62) (cid:13)(cid:13) F = O ( (cid:112)
Revised: Combining Theorem 4.1 and Lemma 4.2, given as input an (cid:15) -corrupted set of n = (cid:101) Ω( k 2 log d/(cid:15) 2 ) samples drawn from a centered subgaussian distribution with covariance I + ρvv (cid:62) , Algorithmreturns a vector u with (cid:13)(cid:13) uu (cid:62) − vv (cid:62) (cid:13)(cid:13) F = O ( (cid:112)

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We defer the proofs of Lemma 4.2 and Theorem 4.1 to Appendix B and give an overview of the proof of Theorem 4.1.
Revised: We defer the proofs of Lemma 4.2 and Theorem 4.1 to Appendix B and give an overview of the proof of Theorem 4.1.

Original: Proof Sketch of Theorem 4.1.
Revised: Proof Sketch of Theorem 4.1.

Original: We can use the stability conditions to upper bound the optimal objective value: note that for w (cid:63) (uniform distribution on the remaining good samples), we must have (cid:13)(cid:13)
Revised: We can use the stability conditions to upper bound the optimal objective value: note that for w (cid:63) (uniform distribution on the remaining good samples), we must have (cid:13)(cid:13)

Original: M w (cid:63) − ( I + ρvv (cid:62) )
Revised: M w (cid:63) − ( I + ρvv (cid:62) )

Original: (cid:13)(cid:13) F, 2 k 2 ≤ δ by the stability conditions, therefore (cid:107) M w (cid:63) − I (cid:107) F, 2 k 2 ≤ (cid:13)(cid:13)
Revised: (cid:13)(cid:13) F, 2 k 2 ≤ δ by the stability conditions, therefore (cid:107) M w (cid:63) − I (cid:107) F, 2 k 2 ≤ (cid:13)(cid:13)

Original: M w (cid:63) − ( I + ρvv (cid:62) )
Revised: M w (cid:63) − ( I + ρvv (cid:62) )

Original: (cid:13)(cid:13) F, 2 k 2 + (cid:13)(cid:13) ρvv (cid:62) (cid:13)(cid:13) F, 2 k 2 ≤ ρ + δ .
Revised: (cid:13)(cid:13) F, 2 k 2 + (cid:13)(cid:13) ρvv (cid:62) (cid:13)(cid:13) F, 2 k 2 ≤ ρ + δ .

Original: Because the objective function f ( w ) is convex, any stationary point w must be globally optimal and satisﬁes f ( w ) ≤ ρ + δ .
Revised: Because the objective function f ( w ) is convex, any stationary point w must be globally optimal and satisﬁes f ( w ) ≤ ρ + δ .

Original: Fix a stationary point w and let A
Revised: Fix a stationary point w and let A

Original: =
Revised: =

Original: M w −
Revised: M w −

Original: I .
Revised: I .

Original: Let R be the support of vv (cid:62) and let Q be the set of klargest entries of A .
Revised: Let R be the support of vv (cid:62) and let Q be the set of klargest entries of A .

Original: The stability conditions implies for any w , the projection in the v direction must be large (formally v (cid:62)
Revised: The stability conditions implies for any w , the projection in the v direction must be large (formally v (cid:62)

Original: Av ≥ ρ − δ ).
Revised: Av ≥ ρ − δ ).

Original: Because the objective function measures the norm of the largest 2 k 2 entries of A and it is not much larger than the norm of the largest k 2 entries, we can argue that
Revised: Because the objective function measures the norm of the largest 2 k 2 entries of A and it is not much larger than the norm of the largest k 2 entries, we can argue that

Original: A R and A Q are close, so v (cid:62) A Q v ≥ ρ − O ( δ ) .
Revised: A R and A Q are close, so v (cid:62) A Q v ≥ ρ − O ( δ ) .

Original: Now A Q is a matrix with Frobenius norm at most ρ + δ while v (cid:62) A Q v ≥ ρ − O ( δ ) .
Revised: Now A Q is a matrix with Frobenius norm at most ρ + δ while v (cid:62) A Q v ≥ ρ − O ( δ ) .

Original: Together these imply that the norm of A Q in direction orthogonal to vv (cid:62) is at most O ( √ ρδ ) , and then by standard matrix perturbation bounds we know the top eigenvector of A Q + A (cid:62) Q ) is O ( (cid:112) δ/ρ ) close to v .
Revised: Together these imply that the norm of A Q in direction orthogonal to vv (cid:62) is at most O ( √ ρδ ) , and then by standard matrix perturbation bounds we know the top eigenvector of A Q + A (cid:62) Q ) is O ( (cid:112) δ/ρ ) close to v .

Original:  Experiments
Revised:  Experiments

Original: We perform an experimental evaluation of our robust sparse mean estimation algorithm on synthetic datasets with a focus on statistical accuracy ( (cid:96) 2 -distance between the output and the true sparse mean).
Revised: We perform an experimental evaluation of our robust sparse mean estimation algorithm on synthetic datasets with a focus on statistical accuracy ( (cid:96) 2 -distance between the output and the true sparse mean).

Original: We evaluate our algorithm (Sparse Gradient Descent, Sparse GD ) on different noise models, and compare it to the following previous algorithms:
Revised: We evaluate our algorithm (Sparse Gradient Descent, Sparse GD ) on different noise models, and compare it to the following previous algorithms:

Original: • oracle , which is told exactly which samples are inliers, and outputs their empirical mean, • the robust sparse mean estimation algorithm RME_sp from [DKK + 19b], • NP (Naive Pruning), which removes samples far from the median and output the mean of the rest, • RANSAC , which randomly selects half of the points and computes their mean.
Revised: • oracle , which is told exactly which samples are inliers, and outputs their empirical mean, • the robust sparse mean estimation algorithm RME_sp from [DKK + 19b], • NP (Naive Pruning), which removes samples far from the median and output the mean of the rest, • RANSAC , which randomly selects half of the points and computes their mean.

Original: One solution is preferred to another if it has more points in a ball of radius O ( √ d ) around it.
Revised: One solution is preferred to another if it has more points in a ball of radius O ( √ d ) around it.

Original: For algorithms that output non-sparse vectors, we take the largest k entries before measuring the (cid:96)distance to the true mean.
Revised: For algorithms that output non-sparse vectors, we take the largest k entries before measuring the (cid:96)distance to the true mean.

Original: We evaluate the algorithms on various noise models:
Revised: We evaluate the algorithms on various noise models:

Original: • Linear-hiding noise.
Revised: • Linear-hiding noise.

Original: The inliers are drawn from N (0 , I ) .
Revised: The inliers are drawn from N (0 , I ) .

Original: Let S be a size k set.
Revised: Let S be a size k set.

Original: Then, half the outliers are drawn from N (1 S , I ) and the other half are drawn from N (0 , 2 I − I S ) .
Revised: Then, half the outliers are drawn from N (1 S , I ) and the other half are drawn from N (0 , 2 I − I S ) .

Original: • Tail-ﬂipping noise.
Revised: • Tail-ﬂipping noise.

Original: This noise model picks a k -sparse direction v and replaces the (cid:15) fraction of points farthest in the − v direction with points in the + v direction.
Revised: This noise model picks a k -sparse direction v and replaces the (cid:15) fraction of points farthest in the − v direction with points in the + v direction.

Original: • Constant-bias noise.
Revised: • Constant-bias noise.

Original: This model adds a constant to every coordinate of the outlier points.
Revised: This model adds a constant to every coordinate of the outlier points.

Original: In Figure 3, we add 2 to every coordinate of every outlier point.
Revised: In Figure 3, we add 2 to every coordinate of every outlier point.

Original: We ran our experiments on a computer with a 1.6 GHz Intel Core i5 processor and 8 GB RAM.
Revised: We ran our experiments on a computer with a 1.6 GHz Intel Core i5 processor and 8 GB RAM.

Original: We built on the codebase of [DKK + 19b] 3 and implemented our new algorithm for the experiments.
Revised: We built on the codebase of [DKK + 19b] 3 and implemented our new algorithm for the experiments.

Original: For each pair of algorithm and noise model, we repeat the same experiment 10 times and plot the median value of the measurements.
Revised: For each pair of algorithm and noise model, we repeat the same experiment 10 times and plot the median value of the measurements.

Original: We shade the interquartile region around the reported points in the ﬁgure as conﬁdence intervals.
Revised: We shade the interquartile region around the reported points in the ﬁgure as conﬁdence intervals.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Fix the sparsity k and change the number of samples n 
Revised: Fix the sparsity k and change the number of samples n .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Our experimental results are summarized in Figures 1, 2 and 3.
Revised: Our experimental results are summarized in Figures 1, 2 and 3.

Original: For the linear-hiding and constant-bias noise models, we run two experiments: 1) ﬁx the sparsity k and change the number of samples n, and 2) ﬁx n and change k .
Revised: For the linear-hiding and constant-bias noise models, we run two experiments: 1) ﬁx the sparsity k and change the number of samples n

Revised: [Figure]

Original:  For the tail-ﬂipping noise model, we run two experiments: 1) ﬁx the sparsity k and change the number of samples n , and 2) ﬁx k and n and change the fraction of corruption (cid:15) .
Revised: and 2) ﬁx n and change k . For the tail-ﬂipping noise model, we run two experiments: 1) ﬁx the sparsity k and change the number of samples n , and 2) ﬁx k and n and change the fraction of corruption (cid:15) .

Original: In terms of statistical accuracy, our algorithm ( Sparse GD ), outperforms the ﬁlter-based RME_sp
Revised: In terms of statistical accuracy, our algorithm ( Sparse GD ), outperforms the ﬁlter-based RME_sp

Original: algorithm [DKK + 19b] in the linear-hiding noise model when the number of samples or the sparsity is small, as shown in Figure 1.
Revised: algorithm [DKK + 19b] in the linear-hiding noise model when the number of samples or the sparsity is small, as shown in Figure 1.

Original: Our algorithm matches the performance of RME_sp under the tail ﬂipping and constant-bias noise models, as shown in Figures 2 and 3.
Revised: Our algorithm matches the performance of RME_sp under the tail ﬂipping and constant-bias noise models, as shown in Figures 2 and 3.

Original: Matching our theoretical results, our Sparse GD algorithm has accuracy O ( (cid:15) (cid:112) log(1 /(cid:15) )) and is within a constant factor of the Ω( (cid:15) (cid:112) log(1 /(cid:15) )) worst-case performance of oracle .
Revised: Matching our theoretical results, our Sparse GD algorithm has accuracy O ( (cid:15) (cid:112) log(1 /(cid:15) )) and is within a constant factor of the Ω( (cid:15) (cid:112) log(1 /(cid:15) )) worst-case performance of oracle .

Original: In contrast, the naive algorithms NP and RANSAC both incur error that scales as (cid:15) √ k .
Revised: In contrast, the naive algorithms NP and RANSAC both incur error that scales as (cid:15) √ k .

Original: The tail-ﬂipping noise  illustrates that Ω( (cid:15) (cid:112) log(1 /(cid:15) )) error can occur no matter which algorithm is used. In Figure 2, even the oracle has Ω( (cid:15) (cid:112) log(1 /(cid:15) )) error from the missing points, and the algorithms have twice this error from not being able to remove the newly added points.
Revised: The tail-ﬂipping noise (Figure 2) illustrates that Ω( (cid:15) (cid:112) log(1 /(cid:15) )) error can occur no matter which algorithm is used (including oracle ), because (cid:15) -fraction of the original good samples was removed.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Yu Cheng is supported in part by NSF Award CCF-2122628.
Revised: Yu Cheng is supported in part by NSF Award CCF-2122628.

Original: Ilias Diakonikolas is supported by NSF
Revised: Ilias Diakonikolas is supported by NSF

Original: Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), NSF Award AiTF-2006206, a Sloan Research Fellowship, and a DARPA Learning with Less Labels (LwLL) grant.
Revised: Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), NSF Award AiTF-2006206, a Sloan Research Fellowship, and a DARPA Learning with Less Labels (LwLL) grant.

Original: Rong Ge is supported by NSF Award CCF-1704656, NSF Award CCF-1845171 (CAREER), NSF Award
Revised: Rong Ge is supported by NSF Award CCF-1704656, NSF Award CCF-1845171 (CAREER), NSF Award

Original: CCF-1934964, a Sloan Research Fellowship, and a Google Faculty Research Award.
Revised: CCF-1934964, a Sloan Research Fellowship, and a Google Faculty Research Award.

Original: Shivam Gupta is supported by NSF Award AiTF-2006206.
Revised: Shivam Gupta is supported by NSF Award CCF-2008868, NSF Award AiTF-2006206, and the NSF AI Institute for Foundations of Machine Learning (IFML).

Original: Some of this work was done while Shivam Gupta was visiting  University of Wisconsin-Madison.
Revised: Some of this work was done while Shivam Gupta was visiting the University of Wisconsin-Madison.

Original: Daniel M. Kane is supported by NSF Award CCF- (CAREER), NSF Award CCF-2107547, a Sloan Research Fellowship, and a grant from CasperLabs.
Revised: Daniel M. Kane is supported by NSF Award CCF-1652862 (CAREER), NSF Award CCF-2107547, a Sloan Research Fellowship, and a grant from CasperLabs.

Original: Mahdi Soltanolkotabi is supported by the Packard Fellowship in Science andEngineering, a Sloan
Revised: Mahdi Soltanolkotabi is supported by the Packard Fellowship in Science and

Original:  Research Fellowship, NSF Award CCF-1846369 (CAREER), NSF Award CCF-1813877, DARPA Learning with Less Labels (LwLL) and Fast Network Interface Cards (FastNICs) grants, and Faculty Research Awards from Google and Amazon.
Revised: Engineering, a Sloan Research Fellowship, NSF Award CCF-1846369 (CAREER), NSF Award CCF1813877, DARPA Learning with Less Labels (LwLL) and Fast Network Interface Cards (FastNICs) grants, and Faculty Research Awards from Google and Amazon.