List-Decodable Sparse Mean Estimation via Difference-of-Pairs Filtering
Abstract
We study the problem of list-decodable sparse mean estimation.
Specifically, for a parameter α ∈ (0 , 1 / 2) , we are given m points in R n , ⌊ αm ⌋ of which are i.i.d. samples from a distribution D with unknown k -sparse mean µ .
No assumptions are made on the remaining points, which form the majority of the dataset.
The goal is to return a small list of candidates containing a vector ˆ µ such that ∥ ˆ µ − µ ∥ 2 is small.
Prior work had studied the problem of list-decodable mean estimation in the dense setting.
In this work, we develop a novel, conceptually simpler technique for listdecodable mean estimation.
As the main application of our approach, we provide the first sample and computationally efficient algorithm for list-decodable sparse mean estimation.
In particular, for distributions with “certifiably bounded” t -th moments in k -sparse directions and sufficiently light tails, our algorithm achieves error of (1 /α ) O
(1 /t ) with sample complexity m =
( k log( n ))
O ( t ) /α and running time poly( mn t ) .
For the special case of Gaussian inliers, our algorithm achieves the optimal error guarantee Θ( (cid:112) log(1 /α )) with quasi-polynomial complexity.
We complement our upper bounds with nearly-matching statistical query and low-degree polynomial testing lower bounds.
Introduction
It is well-established that when a dataset is corrupted by outliers, many commonly-used estimators fail to produce reliable estimates [Tuk60, ABH + 72].
The field of robust statistics was developed to perform reliable statistical inference in the presence of a constant fraction of outliers, even when the data is high-dimensional [HR09, HRRS86].
Although statistical rates of high-dimensional robust estimation problems are relatively well-understood by now [DL88, Yat85, DG92, HR09, CGR16], all of the estimators developed in these works were computationally inefficient, with runtime exponential in the dimension.
The goal of algorithmic robust statistics, beginning with the works of [DKK + 16,
LRV16], is to design computationally efficient algorithms for high-dimensional robust estimation tasks.
We refer the reader to the survey [DK19] for an introduction to this field.
The bulk of the recent progress in algorithmic robust statistics has focused on the setting where the fraction of outliers is a small constant, and the majority of samples are inliers, see, e.g., [DKK + 16,
LRV16, KKM18].
In contrast, when the fraction of outliers outnumbers the fraction of inliers, it is generally information-theoretically impossible to output a single estimate with non-vacuous error guarantees.
In such situations, we allow the algorithm to return a small list of candidates such that one of the candidates is close to the true parameter.
This list-decodable learning setting was first introduced in [BBV08] and developed in [CSV17].
We define the model below.
Definition 1.1 (List-Decodable Learning) .
Given a parameter 0 < α < 1 / 2 and a distribution family D on R n , the algorithm specifies m ∈ Z + and observes a set of m samples constructed as follows: First, a set S of ⌊ αm ⌋ i.i.d.
samples are drawn from an (unknown) distribution D ∈ D .
Then, an adversary is allowed to inspect S and choose a multiset E of m − ⌊ αm ⌋ points.
The multiset T , defined as T := S ∪ E , of m points is given as input to the algorithm.
We say that D is the distribution of inliers, the elements in S are inliers, the points in E are outliers, and T is an (1 − α ) -corrupted dataset of S .
The goal is to output a “small” list of hypotheses L at least one of which is (with high probability) close to the target parameter of D .
The list-decodable learning setting, interesting in its own right, is closely related to several wellstudied problems.
A natural example is the problem of parameter recovery from mixture models, e.g., Gaussian mixtures (see, e.g., [Das99, VW04, AK05, DS07, KK10, RV17]).
List-decodable mean estimation can serve as a key step in learning mixtures, since one can treat any component of the mixture as the set of inliers (see, e.g., [CSV17, DKS18a, KS17]).
In addition, list-decodable learning can be used to model data in important applications where mixture models are not sufficient, such as crowdsourcing (see, e.g., [SVC16, SKL17, MV18]) and community detection in stochastic block models (e.g., [CSV17]).
Prior work on list-decodable mean estimation has focused on the unstructured setting, where the target mean is an arbitrary dense vector (see, e.g., [CSV17, KS17, DKS18a, RY20a, CMY20, DKK20,
DKK + 21, DKK + 22b]).
Sparse models have proven to be useful in a wide range of statistical tasks, and thus understanding the statistical and computational aspects of sparse estimation is a fundamental problem (see, e.g., [EK12, HTW15, van16]).
Here we study list-decodable sparse mean estimation, where the target mean vector is known to be k -sparse , i.e., it has at most k non-zero coordinates.
Given an (1 − α ) -corrupted set of samples, our goal is to output, in a computationally-efficient manner, a small list of vectors containing a good approximation (cid:98) µ to the true mean µ (cf.
Definition 1.1).
Importantly, the goal is to achieve this with far fewer samples than in the dense setting.
While the dense setting would require sample size polynomial in n — the ambient dimension of the data — the goal here is to solve the problem in number of samples polynomial in k and only polylogarithmic in n .
In this paper, we present a novel and conceptually simple technique for list-decodable mean estimation (that is applicable even in the dense setting).
Combining our framework with the concentration results from [DKK + 22a], we obtain the first sample and computationally efficient algorithm for list-decodable sparse mean estimation.
We note that, while prior results [KS17, RY20a] can possibly be modified to incorporate the sparsity framework of [DKK + 22a] after sufficient effort, a notable contribution of our work is a general and conceptually simpler framework for list-decodable estimation, which can easily be adapted to incorporate various structural constraints.
1.1 Related Work
Efficient estimators for high-dimensional robust statistics with sparsity constraints have been recently developed for various problems, such as mean estimation and PCA (see, e.g., [BDLS17, DKK + 19,
Li17, DKK + 22a]).
The problem of list-decodable mean estimation was first introduced in [CSV17], which achieved an error guarantee of ˜ O
(1 / √ α ) for distributions with bounded second moment; this guarantee turned out to be optimal for this distributional assumption (see [DKS18a]).
Subsequent work improved the algorithmic guarantees for this problem (see, e.g., [CMY20, DKK20, DKK + 21]).
To achieve better error guarantees, it is necessary to make further assumptions on the distribution, e.g., Gaussianity or bounded higher moments.
In terms of the minimax optimal rate 1 , [DKS18a] showed that the optimal error for Gaussians is Θ( (cid:112) log(1 /α )) .
They also showed that any SQ algorithm that achieves the optimal error for Gaussians must take either super-polynomial time or samples, and presented an algorithm with matching guarantees.
When the distribution D has bounded t -th moment for some even t > 2 ( i.e., E [ ⟨ v, X − E [ X ] ⟩ t ] is bounded for all unit vectors v ), [DKS18a] proved that the minimax optimal rate is Θ( α − 1 /t ) .
For distributions with certifiably bounded t -th moments, [KS17, RY20a] provided algorithms obtaining an error rate of O ((1 /α )
O (1 /t ) )
with sample complexities m = poly( n t /α )
and runtimes poly( m t n t ) and poly( mn ) poly( t, 1 /α ) , respectively.
Recently, in the context of moment estimation and clustering problems, [ST21] showed how to improve the dependence on m in the runtime of algorithms that depend on certifiably bounded moments from poly( m t n t ) to poly( mn t ) .
While it is possible that their result might be applied to [KS17, RY20a], our algorithmic technique naturally lends itself to achieve runtime poly( mn t ) for the dense as well as sparse settings.
We provide detailed comparisons with (see, e.g., [DKS18a, KS17, RY20a]) in Section 1.3.
Finally, we mention that the list-decodable setting has also been studied in the context of linear regression (see, e.g., [KKK19, RY20a, DKP + 21]) and subspace recovery (see, e.g., [BK21, RY20b]).
In an independent and concurrent work [ZS22], Zeng and Shen also study list-decodable sparse mean estimation when the underlying distribution is spherical Gaussian.
1.2 Our Results
We demonstrate an algorithm to perform list-decodable sparse mean estimation with ( k log n ) O ( t ) samples, when the mean µ ∈ R n is known to be k -sparse.
For this to be possible, we will require some assumptions on the underlying distribution of inliers D .
Prior work in the dense setting ([KS17,
RY20a]) assumed that the inlier distribution D in Definition 1.1 satisfies d -certifiably bounded t -th moments in every direction (i.e., for some moment bound M > 0 , M ∥ v ∥ t 2 − E X ∼ D [ ⟨ v, X − µ ⟩ t ] can be expressed as a sum of square polynomials of degree at most d = O ( t ) in the entries of v ), and
D has light tails.
We highlight that our algorithmic technique can also be used in the dense case, under the same assumption, and provides qualitatively similar error guarantees with much simpler arguments and improved runtime.
Below we apply our technique to the sparse setting.
Our results hold when (i) the t -th moment of D is d -certifiably bounded for every v that is k -sparse, with d = O ( t ) , and (ii) D has light tails.
For ease of exposition, we state the result assuming
D has subexponential tails (i.e., for some universal constant c , for all unit vectors v and p ∈ N ,
E X ∼ D [ |⟨ v, X − µ ⟩| p ] 1 /p ≤ cp ).
Theorem 1.2 (List-Decodable Sparse Mean Estimation) .
Let t be an integer power of two.
Let D be a distribution over R n with k -sparse mean µ .
Suppose that D has t -th moments d -certifiably bounded in k -sparse directions by M for some d = O ( t ) (cf.
Definition 2.3) and subexponential tails in the standard basis directions.
There is an algorithm which, given α , M , t , k , and a (1 − α ) -corrupted set of m = ( tk log n ) O ( t ) max(1 , M − 2 ) /α
samples from D , runs in time poly( mn t ) and returns a vector ˆ µ ∈ R n such that with probability Ω( α ) it is the case that ∥ ˆ µ − µ ∥ 2 =
O t ( M 1 /t /α
O (1) /t ) .
Note that with high probability over the inliers and for any choice of outliers, with probability Ω( α ) over the internal randomness of the algorithm, the algorithm of Theorem 1.2 outputs an estimate ˆ µ close to µ .
By running our algorithm O (1 /α ) times, we can generate a list of size O (1 /α ) such that with probability 0 .
9 the list contains the desired estimate ˆ µ .
Notably, for the important special case of Gaussian N ( µ, I ) inliers, our algorithm achieves the information-theoretically optimal error rate.
This is because N ( µ, I ) has its t -th moment certifiably bounded by t t/ 2 in all directions.
Specifically, for a large enough constant C > 0 , we obtain the following result: Given α , t , and a (1 − α )-corrupted set of m ≥ ( tk log n ) Ct samples from N ( µ, I ) for a k -sparse vector µ , our algorithm runs in time poly( mn t ) and with probability Ω( α ) outputs a vector (cid:98) µ such that ∥ (cid:98) µ − µ ∥ 2 ≤ O ( √ t/α C/t ) .
Thus, by taking t =
C log(1 /α ) , we obtain the optimal error of Θ( (cid:112) log(1 /α )) in quasi-polynomial sample and time complexity.
We also note that a broad and natural class of distributions satisfying Definition 2.3 is the class of σ -Poincare distributions (see, e.g., [KS17]).
A distribution is said to be σ -Poincare if for all differentiable functions f : R n → R , we have that Var X ∼ D
[ f ( X )] ≤ σ 2 E X ∼ D [ ∥∇ f ( X ) ∥ 22 ] .
We complement our algorithm of Theorem 1.2 with a qualitatively matching lower bound in the Statistical Query (SQ) model [Kea98].
Instead of directly accessing samples, SQ algorithms are only allowed to perform adaptive queries of expectations of bounded functions of the underlying distribution, up to some desired tolerance (c.f. Definition C.1).
The class of SQ algorithms is fairly broad: a wide range of known algorithmic techniques in machine learning are known to be implementable in the SQ model (see, e.g., [FGR + 13]).
An SQ lower bound is an unconditional statement that for any SQ algorithm, either the number of queries q must be large or the tolerance, τ , of some query must be small.
Since simulating a query of tolerance τ by averaging i.i.d.
samples may need up to Ω(1 /τ 2 ) many of them, SQ lower bounds are naturally interpreted as a tradeoff between runtime Ω( q ) and sample complexity Ω(1 /τ 2 ) .
An adaptation of the result in [DKS18a] yields Theorem 1.3, which indicates that the k O ( t ) factor in the sample complexity of Theorem 1.2 might be necessary for efficient algorithms, even for Gaussian inliers.
Theorem 1.3 (SQ Lower Bound, Informal) .
Consider the problem of list-decoding the mean of
N ( µ, I ) , for a k -sparse vector µ ∈ R n , up to error better than O (( tα ) − 1 /t ) .
Any SQ algorithm that solves the problem does one of the following: (i) It returns a list of size n k Ω(1) , (ii) it uses at least one query of tolerance k − Ω( t ) exp( O ( tα ) − 2 /t ) , or (iii) it makes at least n k Ω(1) queries. Theorem 1.3 states that any SQ algorithm that runs in time less than n k c for a constant c >and achieves the minimax-optimal error of (cid:112) log(1 /α ) must use at least k Ω(log(1 /α ) / loglog(1 /α ))
samples. This follows by setting t = Ω(log(1 /α ) / log log(1 /α )) .
A similar lower bound holds for the computational model of low-degree polynomial tests, as a consequence of the recently established relationship between the two models [BBH + 21].
See Appendix C for more details.
1.3 Overview of Techniques
We begin with a brief overview of the existing techniques for dense list-decodable mean estimation.
We then highlight some of the obstacles in adapting these techniques to the sparse setting.
Then, we provide an overview of our algorithmic approach and explain how it overcomes these obstacles.
Prior Work on List-Decodable Mean Estimation In the dense setting, prior algorithmic techniques for list-decoding the mean with error better than Ω( α − 1 / 2 ) are quite complicated.
[DKS18a] uses a multifilter-based technique for list-decoding spherical Gaussians, which relies critically on knowing the higher degree moments of the inliers (and thus does not generalize to less specific distribution families). Moreover, this method runs into technical difficulties related to being unable to determine the variance of higher degree polynomials on the inliers without knowing the mean ahead of time.
The other approach in the literature (see, for example, [KS17, RY20a]) uses the Sum-of-Squares method (SoS) to find these clusters of points.
The algorithm in [RY20a] involves solving a nested SoS program and then applying a complicated rounding procedure to get the final list.
It should be noted that the runtime of [RY20a] is exponential in poly(1 /α ) , which can be quite large.
Finally, [KS17] gave an SoS based list-decodable mean estimation algorithm for the dense case with error, sample complexity, and list size similar to the ones that are obtained in our work; but significantly worse runtime.
The approach of [KS17] has some important differences.
First, the clustering relaxation is conceptually harder, involving a more complex optimization problem for each filtering step and second, after the filtering ends, the error guarantee scales with the norm of the unknown mean; thus a complicated re-clustering step that combines ideas from [SCV18] is needed to reduce the error.
Here we present a significantly cleaner method to perform the outlier removal step; we avoid problems like not knowing the mean ahead of time by simply taking pairs of differences of our samples to make their mean zero.
While it is likely that either of the prior techniques could be adapted to the sparse setting with sufficient effort, this would result in significantly more complicated algorithms. We briefly point out some difficulties below.
First, to ensure that the algorithms identify subsets of the samples that satisfy certifiably bounded moments in all k -sparse directions, requires additional variables and constraints to the algorithms. Additionally, one would need to replace the bounded moments in all directions condition by the corresponding condition for the sparse case, and ensure that all the proof steps can be modified to rely only on the latter – this would result in minor modifications of the original algorithms, such as thresholding of candidate solution vectors. Second, at the end of this process, while the algorithm might qualitatively match the error guarantee that we achieve, the runtime would continue to be ( mn ) O ( t ) for [KS17] or (1 /α ) polylog(1 /α ) n O (max { 1 /α 4 ,t } ) for [RY20a] — both of these are qualitatively worse than the runtime we achieve when α is sufficiently small.
To obtain improved runtime using these prior techniques, one would require additional ideas, e.g., from [ST21], to be adapted to this setting, overall resulting in a far more complicated algorithm. On the other hand, it seems difficult to adapt the multi-filtering technique from [DKS18a] to the setting we consider, without any introduction of an SoS component. We remind the reader that the [DKS18a] algorithm depends critically on knowing the higher moments of the inliers exactly , and does not generalize to less specific distribution families. Even in the Gaussian setting, generalizing [DKS18a] might be difficult, since it would require the design of an efficiently verifiable notion of matching higher moments in k -sparse directions.
We believe that our novel list-decoding technique is significantly simpler.
As a result of simplifying the optimization programs involved, our technique naturally improves the runtime from poly( m t n t ) in prior work to poly( mn t ) .
Novel List-Decodable Mean Estimation Algorithm All known list-decoding algorithms are based upon the following fundamental observation. Suppose that S is a dataset that contains a subset S g of samples with bounded moments.
If we can find a subset S ′ of S with bounded moments and large overlap with S g , then the means of S ′ and S g cannot be too far apart (see Lemma 3.3). The goal of a list-decoding algorithm is to find such a subset S ′ (or, more precisely, a small number of hypotheses for such a subset). In order to generalize this to the sparse mean setting, it suffices for the subsets S g and S ′ to have bounded moments only in all k -sparse directions. This will imply that |⟨ v, µ g − µ S ′
⟩| is relatively small for all k -sparse unit vectors v , which will in turn imply that truncating µ S ′ to its k largest entries will provide a suitable approximation to µ g (see Fact A.2). The basic idea behind our novel list-decoding technique, which we call the difference of pairs filter , is the following: let T be the set of differences of pairs of elements of S . The set T will contain a relatively large subset, T g , (consisting of the pairwise differences of elements of S g ) whose moments in k -sparse directions are bounded.
Our goal will be to find a subset T ′ ⊂ T that has large overlap with T g and also has bounded ( k -sparse) moments.
Naïvely, we can do this as follows.
We start with T ′ =
T .
Either this set has bounded k -sparse moments (in which case we are done) or there is some sparse direction v in which the average value of |⟨ v, x ⟩| t over T ′ is substantially larger than the average value over T g .
By throwing away points x from T ′ with probability proportional to |⟨ v, x ⟩| t , we eliminate mostly bad points.
We repeat this until T ′ has bounded sparse moments.
Unfortunately, while this approach can be shown to be correct, it does not suffice for our purposes because it is computationally infeasible in general to determine whether or not T ′ has bounded sparse moments.
However, if we assume additionally that S g has bounded k -sparse moments provable by a low-degree sum-of-squares proof (i.e., the moment bound inequality can be re-expressed as a sum of square polynomials being greater than 0 ), then so will T g .
There is an efficient algorithm to determine whether or not T ′ has certifiably bounded moments in k -sparse directions as well.
If T ′ does not have certifiably bounded moments, standard techniques for Sum-of-Squares programs imply that we can manufacture a non-negative polynomial p so that the average value of p on T ′ is substantially larger than the value of p on any set with SoS-certifiable bounded moments in k -sparse directions.
Thus, filtering out points x in T ′ with probability proportional to p ( x ) will likely remove mostly bad points.
Using this idea, we can find such a set T ′ efficiently (see Theorem 3.1).
We are thus left with a set of differences of samples rather than a set of samples.
At this point, we will need a rounding method that given a set T ′ of differences with bounded k -sparse moments guaranteed to have large overlap with T g , finds a small list of sets S ′ with bounded k -sparse moments so that at least one of them has a large overlap with S g . Note that T ′ might be the union of T i − T i , where each T i is an individual cluster drawn from a moment-bounded distribution.
This demonstrates the necessity of a rounding step to identify the means of individual clusters.
To achieve this, it is helpful to think of T ′ as consisting of a set of pairs of elements of S , or equivalently as a graph over S .
Given T ′ , our first task is to find some reasonably large subset of S with bounded moments (ideally which has a large overlap with S g ).
It is not hard to see that it suffices to find any large clique in T ′ (see Lemma 3.3).
Unfortunately, T ′ may well not have any large cliques, and even if it does, finding them may be computationally difficult.
However, we are saved here by the observation that if |⟨ v, x − y ⟩| t and |⟨ v, y − z ⟩| t are both small, then so is |⟨ v, x − z ⟩| t .
This means that if we replace T ′ by the graph H of all pairs of vertices ( v, w ) , where v and w have many common neighbors in T ′ , this new graph will also have relatively small k -sparse moments (see Lemma 3.5).
As most elements of S g are adjacent in T ′ to most other elements of S g , it is not hard to see that this new graph will have relatively large cliques; unfortunately, finding them may still be computationally difficult.
To find these large cliques efficiently, we need one final observation.
If v is a random vertex of a graph G , then there will (on average) be very few pairs of neighbors, u and w , of v so that u and w do not have a large number of common neighbors with each other in G (see Lemma 3.6).
This means that if we pick a random sample x in S , then most pairs of neighbors of x in T ′ are neighbors in H .
Using a densification procedure (see Lemma 3.7), it is not hard to find a large subset S ′ of these neighbors so that any two elements of S ′ have many neighbors in common in G .
By Lemma 3.5, this implies that S ′ will in fact have bounded k -sparse moments.
Furthermore, if we happened to pick x from S g , it is not hard to see that it is likely that S ′ has large overlap with S g , and thus its mean provides us with a good estimate.
While the above describes our algorithmic approach, we also need to consider the sample complexity of our method.
We know that the distribution D has the property of bounded moments in k -sparse directions, and our algorithm requires that the property is also satisfied by the uniform distribution over the samples S g .
In order for S g to have it as well, it suffices that the t th moment tensor of S g − µ be close to the corresponding moment tensor of D − µ .
It turns out that if these moment tensors are δk − t -close coordinate-wise, which takes O ( t log( n ) k O ( t ) /δ 2 ) samples, this suffices to get the kind of certifiable concentration we require.
This is captured by Lemma 2.4, a restatement from [DKK + 22a].
Preliminaries
Basic Notation.
We use Z + to denote positive integers.
For n ∈ Z + we denote [ n ] := { 1 , . . . , n } .
We denote by R [ x 1 , . . .
, x n ] ≤ d the set of real-valued polynomials of degree at most d in variables x 1 , . . .
, x n .
We use poly( · ) to indicate a quantity that is polynomial in its arguments.
For an ordered set of variables Q = { x 1 , . . .
, x n } , we will denote p ( Q ) to mean p ( x 1 , . . .
, x n ) .
We use I n to denote the n × n identity matrix.
For a vector v , we let ∥ v ∥ 2 denote its ℓ 2 -norm.
We call a vector k -sparse if it has at most k non-zero coordinates.
We use ⟨ v, u ⟩ for the inner product of the vectors u, v .
We use capital letters for random variables and E [ · ] for expectation.
When S is a set, we will use the notation
X ∼ S to mean that X is distributed uniformly over S .
For a graph G we denote by N G ( x ) the neighborhood of x in G .
Throughout the paper, we will use the letter n for the dimension, m for the number of samples, d for the degrees of the SoS proofs, and t for the number of bounded moments.
SoS Preliminaries
The following preliminaries are specific to the SoS part of this paper.
We refer to [BS16, FKP19] for a more complete treatment of the SoS framework.
Here, we review the basics.
Definition 2.1 (SoS Proof) .
Let x 1 , . . .
, x n be indeterminates and let A be a set of polynomial equalities { p 1 ( x ) = 0 , . . .
, p m ( x ) = 0 } .
An SoS proof of the inequality r ( x ) ≥ 0 from axioms
A is a set of polynomials { r i ( x ) } i ∈
[ m ] ∪ { r 0 ( x ) } such that r 0 is a sum of square polynomial and r i ’s are arbitrary, and r ( x )
=
r 0 ( x ) + (cid:80) i ∈ [ m ] r i ( x ) p i ( x ) .
If the set of polynomials { r i ( x ) · p i ( x ) | i ∈ [ d ] } ∪ { r 0 ( x ) } have degree at most d , we say that this proof is of degree d and denote it by
A d r ( x ) ≥ 0 .
When we need to emphasize what indeterminates are involved in a particular SoS proof, we denote it by A dx r ( x ) ≥ 0 .
When A is empty, we omit it, e.g., d r ( x ) ≥ 0 or dx r ( x ) ≥ 0 .
We will also use the objects called pseudoexpectations .
Definition 2.2 (Pseudoexpectation) .
Let x 1 , . . .
, x n be indeterminates.
A degree- d pseudoexpectation ˜ E is a linear map ˜ E : R [ x 1 , . . .
, x n ] ≤ d → R from degree- d polynomials to R such that ˜ E (cid:2) p ( x ) 2 (cid:3) ≥for any p of degree at most d/ 2 and ˜ E [1] = 1 .
If A = { p 1 ( x ) = 0 , . .
, p m ( x ) = 0 } is a set of polynomial inequalities, we say that a pseudoexpectation ˜
E satisfies A if for every i ∈ [ m ] , ˜ E
[ s ( x ) p i ( x )] = 0 for all polynomials s ( x ) such that s ( x ) p i ( x ) has degree at most d . It is well known (see, e.g., [BS16]) that pseudoexpectations are dual objects to SoS proofs in the following sense: given a set P of r polynomial equalities in n variables and a polynomial q ( x 1 , . . .
, x n ) , either there exists an SoS proof
P ℓx q ( x ) ≥
or there exists a pseudoexpectation ˜
E of degree ℓ satisfying P but having ˜ E
[ q ( x )] < 0 .
More importantly, there is an algorithm that runs in time ( rn ) O ( ℓ ) and finds that pseudoexpectation when we are in the second case.
Due to lack of space, we defer formal statements and a brief overview of SoS to Appendix A.1.
Certifiably Bounded Moments in k -sparse Directions
Our algorithm succeeds whenever the uncorrupted samples have certifiably bounded moments in k -sparse directions, defined as in [DKK + 22a]:
Definition 2.3 ( ( M, t, d ) Certifiably Bounded Moments in k -sparse Directions) .
Let Q := { v 1 , . .
. , v n , z 1 , . . .
, z n } and A k -sparse := { z 2 i
= z i } i ∈ [ n ] ∪{ v i z i = v i } i ∈ [ n ] ∪ { (cid:80) ni
{+=
+}
z i = k } ∪
(cid:8)(cid:80) ni =1 v 2 i = 1 (cid:9) .
For an M > 0 and even t ∈ N , a distribution D with mean µ satisfies ( M, t, d ) certifiably bounded moments in k -sparse directions if A k -sparse d Q E X ∼ D [ ⟨ v, X − µ ⟩ t ] 2 ≤ M 2 .
The definition of A k -sparse is based on the fact that a vector v = ( v 1 , . .
, v n ) is k -sparse if and only if there exists z = ( z 1 , . .
, z n ) such that v, z satisfy A k -sparse .
We will use the following lemma proved in [DKK + 22a] to bound the number of samples it takes to certify bounded moments in k -sparse directions.
Although this is stated for subexponential distributions, it needs the distribution to have only bounded t 2 log( d ) moments (see Lemma A.10).
Lemma 2.4 ([DKK + 22a]) .
Let D be a distribution over R n with mean µ .
Suppose that D has csub-exponential tails around µ for a constant c and that A k -sparse O ( t ) Q E X ∼ D [ ⟨ v, X − µ ⟩ t ] 2 ≤ M 2 .
Let S =
{ X 1 , . . .
, X m } be a set of m i.i.d.
samples from D , D ′ be the uniform distribution over S , and µ :=
E X ∼ D
[ X ] .
If m > C ( tk (log n )) 5 t max(1 , M − 2 ) for a sufficiently large constant C , then with probability at least 0 .
9 we have the following: (i)
A k -sparse O ( t ) Q E X ∼ D ′
[ ⟨ v, X − µ ⟩ t ] 2 ≤ 8 Mand (ii) ⟨ v, µ − µ ⟩ ≤ M 1 /t /α
6 /t for every k -sparse unit vector v .
Main Result: Proof of Theorem 1.
Recall our setting: we are given α ∈ (0 , 1 / 2) and a multiset S :
= { x 1 , . . .
, x m } such that an unknown subset of ⌊ αm ⌋ many of these points satisfy ( M, t, d ) certifiably bounded central moments in k -sparse directions, and the remaining are arbitrary.
The goal is to recover a candidate that is close to µ :=
E X ∼ D
[ X ] with probability Ω( α ) .
In what follows, t will always be 2 ℓ for some ℓ ∈ Z + .
3.1 The SoS-based Filter
Let T be the set of pairwise differences of all samples in S (similarly denote by T g the subset of
T corresponding to inliers S g ).
We would like to either detect that the moments of T are already bounded in all k -sparse directions or find a direction that violates this and filter out mostly outliers.
Unfortunately, this kind of check is computationally infeasible, but it can be done efficiently if we check for moment bounds that are certified by SoS proofs.
This is done in Algorithm 1, which takes as input the set T along with the parameters t, d, M and performs filtering until the resulting set T ′ has ( M, t, d ) -bounded moments.
Algorithm
1: function LDM EAN -S O S-F ILTER ( T := { x 1 , . .
, x m } , t, d, M ) 2:
Let Q = { v 1 , . .
. , v n , z 1 , . . .
, z n } and T ′ = T .
3: while there is no SoS proof of A k -sparse d Q (cid:80) x ∈ T ′ ⟨ v, x ⟩ t ≤ 6 M | T | do 4:
Find a degree- d ˜ E on Q satisfying A k -sparse and
˜ E
[ (cid:80) x ∈ T ⟨ v, x ⟩ t ] > 6 M | T | .
Throw out x ∈ T ′ with probability ˜ E [ ⟨ v, x ⟩ t ] / max x ∈ T ′ ˜
E [ ⟨ v, x ⟩ t ] .
6: end while 7: return T ′ 8: end function
Theorem 3.1 (Filter Identifies a Subset Satisfying Bounded Moments) .
Let T be a multiset of points in R n for which there exists a subset T g ⊂ T with | T g | = α 2 | T | for some α > 0 .
Furthermore assume that T g has zero mean and ( M, t, d ) -certifiably bounded moments in k -sparse directions for some M > 0 , d ∈ Z + , and even t .
Then Algorithm 1, given T, M, t, d , returns a subset T ′ ⊆
T in time poly( mn d ) so that, with probability at least 2 / 3 , the following holds: (i) For any k -sparse unit vector v , we have that (cid:80) x ∈ T ′ ⟨ v, x ⟩ t ≤ 6 M | T | , and (ii) | T ′ ∩
T g | ≥ | T g | / 2 .
Proof.
Let Q = { z 1 , . . .
, z n , v 1 , . . .
, v n } .
In Algorithm 1, Lines 3 and 4 use the separation oracle of Theorem A.4 with P =
A k -sparse .
Since T g has zero mean and ( M, t, d ) bounded central moments in k -sparse directions, we have that [Equation]
Thus, if A k -sparse d Q (cid:80) x ∈ T ′ ⟨ v, x ⟩ t ≤ 6 M | T | the algorithm identifies that and stops (in which case we have the desired conclusion that for any k -sparse vector v , (cid:80) x ∈ T ′ ⟨ v, x ⟩ t ≤ 6 M | T | ); otherwise it finds a degree- d pseudo-expectation ˜ E on Q satisfying A k -sparse and ˜ E (cid:2) − 6 M | T | + (cid:80) x ∈ T ′ ⟨ v, x ⟩ t (cid:3) > 0 , in which case we can create a filter: Using Equation (1) and | T g | = α 2 | T | , we have that M ≥
E X ∼ T g (cid:104) ˜
E [ ⟨ v, X ⟩ t ] (cid:105) ≥
α − 2 | T | (cid:80) x ∈ T g ∩ T ′ ˜ E [ ⟨ v, x ⟩ t ] .
Since ˜ E (cid:2) − 6 M | T | + (cid:80) x ∈ T ′ ⟨ v, x ⟩ t (cid:3) > 0 and ˜ E is linear, we see that (cid:80) x ∈ T g ∩ T ′ ˜
E [ ⟨ v,x ⟩ t ] (cid:80) x ∈ T ′ ˜
E [ ⟨ v,x ⟩ t ] ≤ α 2 / 6 .
This means if we throw out sample x with probability ˜ E
[ ⟨ v, x ⟩ t ] / max x ∈ T ′ ˜ E [ ⟨ v, x ⟩ t ] (which is indeed in [0 , 1] since ⟨ v, x ⟩ t is SoS, so its pseudoexpectation is a non-negative value), on average, only an α 2 / 6 fraction of the points that are removed will be from T g .
Since the x with the largest value of ˜ E [ ⟨ v, x ⟩ t ] will be removed, the algorithm will terminate in polynomial time.
We now analyze the size of | T ′ ∩ T g | .
By the above analysis, at each step the expected number of samples thrown out from T g is at most α 2 / 6 times the expected total number of samples removed.
Thus, the potential function ∆ =
( | T g ∩ T ′ | − ( α 2 / 6) | T ′ | ) / | T | is a submartingale.
Note that always 0 ≤ ∆ /α 2 ≤ 1 , and initially we had ∆ /α 2 ≥ 5 / 6 .
By Doob’s martimgale inequality (Proposition A.9 applied with t = 1 / 2 to the submartingale (∆ /α 2 ) ), the probability that ∆ /α 2 remains at least α 2 / 2 throughout the execution of the algorithm is at least 2 / 3 .
Thus, we will have | T g ∩ T ′ | ≥ ( α 2 / 2) | T | = | T g | / 2 throughout the execution.
3.2 Identifying a Subset of Samples with Bounded Moments
Having identified a subset T ′ ⊂
T satisfying the conclusions of Theorem 3.1, we want to extract from T ′ a vector that is close to the original mean.
Since the average of the set of differences is likely to be close to zero regardless of the true mean, we will need to use the information about the pairs that we get from T ′ to find subsets of the original samples that satisfy the appropriate concentration bounds.
We will need the following definition.
Definition 3.2.
Let S ⊂ R n .
A graph ( V, E ) on S with V = S is said to have ( M, t ) - bounded moments in k -sparse directions if for all k -sparse unit vectors v , (1 / | S | 2 ) (cid:80)
( x,y ) ∈
E ⟨ v, x − y ⟩ t ≤ M .
By the guarantee of our filter, if T ′ is the set returned, the graph G with edges ( x, y ) for which x − y or y − x belongs to T ′ will have bounded moments in the above sense.
If G contains a clique C which intersects with an α -fraction of the target samples C g , the means of C and C g are close.
Lemma 3.3.
Let S ⊂ R n and G be a graph on S satisfying Definition 3.2.
Let C ⊂ S be a clique in G .
Let C g ⊂ C be a subset with | C g | ≥ α | S | .
If µ C and µ g denote the means of C and C g respectively, then ⟨ v, µ C − µ g ⟩ t ≤ 2 M/α 2 for all k -sparse unit vectors v .
Proof.
By using the fact that t is even and the fact that G satisfies Definition 3.2, we see that
M | S | 2 ≥ (cid:80) ( x
,y ) ∈ E ⟨ v, x − y ⟩ t ≥ 12 (cid:80) x,y ∈ C ⟨ v, x − y ⟩ t ≥ 12 (cid:80) x ∈ C g ,y ∈ C ⟨ v, x − y ⟩ t .
Using Jensen’s inequality we obtain, M | S | 2 ≥ (cid:80) x ∈ C g ,y ∈ C ⟨ v,x − y ⟩ t
[Equation]
Unfortunately, even the inliers might not form a clique in G .
However, the guarantee that | T ′ ∩ T g | ≥ | T g | / 2 implies that the inliers share many neighbors in the graph G .
Thus we look at the overlap graph defined below, in the hope that this graph will be more dense.
Definition 3.4 (Overlap Graph) .
Let G =
( V, E ) be a graph and γ > 0 .
The overlap graph
R γ ( G ) is defined to be the graph with the vertex set V where each ( x, y ) is an edge in the graph iff | N G ( x ) ∩ N G ( y ) | ≥ γ | V | , where N G ( x ) denotes the neighborhood of the vertex x in G .
The following result shows that if G has bounded moments, then so does R γ ( G ) .
While this is not sufficient by itself, we will subsequently modify R γ ( G ) to ensure that we end up with a graph having bounded moments, as well as an identifiable clique.
Lemma 3.5 (If G has bounded moments, then R γ ( G ) has bounded moments) .
Let S be a set of points and G be a graph with ( M, t ) -bounded moments in k -sparse directions.
Then for γ > 0 , R γ ( G ) has (2 · 2 t M/γ, t ) -bounded moments in k -sparse directions.
Proof.
For any x, y in R γ ( G ) , the triangle inequality implies ⟨ v, ( x − a ) − ( a − y ) ⟩ t ≤ 2 t [ ⟨ v, x − a ⟩ t + ⟨ v, y − a ⟩ t ] .
By taking a sum over all a in N G ( x ) ∩ N G ( y ) , we have ⟨ v, x − y
[Equation]
t =
[Equation]
Denote Γ( α, x ) :=
{ y : neighbor of x in R γ ( G ) and neighbor of a in G } .
Summing over all the edges ( x, y ) in R γ ( G ) , and since t is even and G has ( M, t ) bounded moments in k -sparse directions, we see:
[Equation]
[Equation]
[Equation] =
[Equation] While R γ ( G ) may not have any cliques either, it is guaranteed to have fairly dense subgraphs.
We will subsequently prune out points so that the resulting final graph will have a clique.
Lemma 3.6 ( R γ ( G ) has dense subgraphs) .
Let G =
( V, E ) be a graph and γ > 0 .
If x is a randomly selected vertex of G , then the expected number of pairs y, z ∈ N G ( x ) so that y and z are not neighbors in R γ ( G ) is at most γ | V | 2 .
Proof.
The expectation in question is 1 / | V | times the number of triples x, y, z ∈ V so that y and z are not neighbors in R γ ( G ) , but are both neighbors of x in G .
By the definition of R γ ( G ) , if y and z are not neighbors in R γ ( G ) , they have at most γ | V | common neighbors in G .
Thus, the number of such triples is at most γ | V | 3 , so the expectation in question is at most γ | V | 2 .
As outlined above, the inliers in R γ ( G ) form a dense subgraph.
The next procedure (P RUNING in
Algorithm 2) prunes out points from a dense subgraph (inliers in R γ ( G ) for us) to find a clique. Algorithm
1: function P RUNING (
G =
( V, E ) , W ⊂ V ) 2:
Let W ′
=
W 3: while ∃ x ∈ W ′ that is not connected to at least 2 | W | / 3 vertices in W ′ :
Remove x from W ′ 4: return W ′ 5: end function 6: function R OUNDING ( S, G = ( V, E ) ) 7:
Let δ =
α 3 / 4608 .
Choose x ∈ S uniformly at random, let W = N G ( x ) , and let G ′ =
R δ ( G ) 9: if the number of pairs of points in W that are not connected in G ′ is more than (8 δ/α ) | V |or if | W | ≤ ( α/ 4) | V | return FAIL 10: else Run P RUNING on G ′ and W to obtain W ′ 11: return E X ∼ W ′
[ X ] .
12: end function
Lemma 3.7 (Dense subgraphs can be pruned to obtain a clique) .
Let G =
( V, E ) be a graph and let
W ⊂ V be a set of vertices with | W | = β | V | and all but γ | V | 2 pairs of vertices in W are connected in G , for β, γ > 0 with γ ≤ β 2 / 36 .
There exists an algorithm ( P RUNING in Algorithm 2) that given
G, W, β, γ runs in polynomial time and returns a W ′ ⊂
W so that | W ′ | ≥ | W | − (6 γ/β ) | V | and so that | W ′ | is a clique in R β/ 3 ( G ) .
Proof.
In Line 3 of the Algorithm, the point x which is removed satisfies | N G ( x ) ∩ W ′ | < 2 / 3 | W | .
If we also have that | W ′ | ≥ 5 | W | / 6 (something that we will verify later), the removal of x decreases the number of pairs of unconnected elements in W ′ by | W ′ | − | N G ( x ) ∩ W ′ | ≥ | W ′ | − (2 / 3) | W | ≥ | W | / [-
-]6 =
( β/ 6) | V | .
This can happen at most (6 γ/β ) | V | times before we run out of unconnected pairs of elements in W ′ , thus | W ′ | ≥ | W | − (6 γ/β ) | V | upon termination.
Also, since it holds (6 γ/β ) | V | ≤ ( β/ 6) | V | = | W | / 6 , we indeed have | W ′ | ≥ 5 | W | / 6 as claimed at the start.
Now note that each element of W ′ is connected to at least 2 | W | / 3 other elements of W ′ in G .
Thus any pair of elements of W ′ have at least | W | / 3 common neighbors, and thus are adjacent in R β/ 3 ( G ) .
We are finally ready to prove our main algorithmic result on rounding.
The basic idea is that most of the inliers in G (which are at least α -fraction of vertices) are connected to many other inliers, and thus if we start with an inlier, its neighborhood will also contain many inliers and will be dense in the overlap graph G ′ (Lemma 3.6).
Thus, we can apply the pruning of Lemma 3.7 to obtain a large clique in the overlap graph of G ′ , which also has bounded moments by two applications of Lemma 3.5.
Theorem 3.8 (Rounding) .
Let S ⊂ R n and let G =
( V, E ) be a graph with V = S and ( M, t ) bounded moments in k -sparse directions.
Suppose there is a subset S g ⊂ S with | S g | ≥ α | S | and at least half of the pairs of points in S g are connected by an edge in G .
Suppose that the S g has mean µ g and t -th moment bounded by M in k sparse directions.
Then, there exists a randomized algorithm that given G, S and α runs in polynomial time and returns a (cid:98) µ ∈ R n such that with probability Ω( α ) , for all k -sparse unit vectors v , ⟨ v, (cid:98) µ − µ g ⟩ t
= O (10 t Mα − 6 )
Proof.
The algorithm we consider is R OUNDING (Algorithm 2).
Let x and G ′ be as in Line 8.
We will claim that algorithm R OUNDING succeeds as long as the following hold: (i) x ∈ S g , (ii) x has at least S g / 4 neighbors in S g in G ′ , and (iii) the number of pairs of neighbors of x that are not neighbors in G ′ is at most (8 δ/α ) | V | 2 .
First, we show that these conditions hold with probability Ω( α ) .
The first condition holds with probability at least α over the choice of x .
Conditioned on x ∈ S g , the expected number of non-neighbors that x has in S g is at most | S g | / 2 .
Thus, the probability that it has more than 3 | S g | / 4 non-neighbors is at most 2 / 3 by Markov’s inequality.
Thus, the first two conditions both hold with probability at least α/ 3 .
Finally, the expected number of pairs of neighbors of x that are non-neighbors in G ′ is at most δ | V | 2 by Lemma 3.6.
Thus, by Markov’s inequality, there will be more than (8 δ/α ) | V | 2 such non-connected neighbors with probability at most α/ 8 .
Combining with the above, all three conditions hold with probability at least α/ 24 .
Given these assumptions, we note that | W | ≥ | S g | / 4 ≥ ( α/ 4) | V | , and at most (8 δ/α ) | V | 2 of pairs in W are not connected in G ′ .
This implies that we pass the condition in Line 9.
We will now verify the conditions in Lemma 3.7.
Since β := | W | / | V | ≥ α/ 4 and γ , the number of pairs of vertices in W that are not connected in G ′ , is at most 8 δ/α =
α 2 / 576 , we have γ ≤ β 2 / 36 , satisfying the assumptions of Lemma 3.7.
Thus the returned W ′ is a clique in R β/ 3 ( G ′ ) and satisfies
[Equation]
This means that | W ′ ∩ S g | ≥ | S g | / 4 − ( α/ 24) | V | ≥ | S g | / 6 .
On the other hand, we know that G has ( M, t ) bounded moments in k -sparse directions.
Lemma 3.5 implies that G ′ has moments bounded by O (2 t M/α 3 ) .
Applying the lemma once more implies that R β/ 3 ( G ′ ) has moments bounded by
O (4 t M/ ( α 3 β ))
=
O (4 t M/α 4 ) .
Since W ′ is a clique in R β/ 3 ( G ′ ) , we have by Lemma 3.3 that if ˜ µ is the sample mean of S g ∩ W ′ , then
[Equation]
all k -sparse unit vectors v . [Equation] Since | S g ∩ W ′ | ≥ | S g | / 6 and the S g has bounded t -th moment along k -sparse directions, we have that ⟨ v, ˜ µ − µ good ⟩ t ≤ O ( M ) (see Lemma B.1 for a proof of this fact).
Combining this with Equation (2) using triangle inequality completes the proof.
3.3 Proof Sketch of Theorem 1.
We sketch the proof here, deferring the full proof to Appendix B.1.
Let S be the (1 − α ) -corrupted set of samples, and S g be the inliers.
Let T = { x − y | x, y, ∈ S } and T g be its part due to inliers.
To every subset T ′ of T , we can associate a graph G T ′ having vertices S and edges between the pairs included in T ′ .
Because of Lemma 2.4, T g has certifiably bounded moments in k -sparse directions.
By Theorem 3.1, the filtering step will return a subset T ′ ⊂
T that has sizable overlap with T g and its graph G T ′ has bounded moments in k -sparse directions.
Finally, By Theorem 3.1, the rounding algorithm will return a ˆ µ that is close to µ in all k -sparse directions.
This ˆ µ can be truncated to yield a vector close to µ in the standard ℓ 2 -norm (Fact A.2).  Acknowledgments
ID was supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), a Sloan Research Fellowship, and a DARPA Learning with Less Labels (LwLL) grant.
DK was supported by NSF Medium Award CCF-2107547, NSF Award CCF-1553288 (CAREER), a Sloan
Research Fellowship, and a grant from CasperLabs.
SK was supported by NSF under Grant #to the Computing Research Association for the CIFellows 2021 Project.
AP was supported by NSF grants NSF Award CCF-1652862 (CAREER), DMS-1749857, and CCF-1841190.
TP was supported by NSF Award CCF-1652862 (CAREER) and NSF Medium Award CCF-2107079.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]