Semi-Random Sparse Recovery in Nearly-Linear Time
Abstract
Sparse recovery is one of the most fundamental and well-studied inverse problems.
Standard statistical formulations of the problem are provably solved by general convex programming techniques and more practical, fast (nearly-linear time) iterative methods.
However, these latter “fast algorithms” have previously been observed to be brittle in various real-world settings.
We investigate the brittleness of fast sparse recovery algorithms to generative model changes through the lens of studying their robustness to a “helpful” semi-random adversary, a framework which tests whether an algorithm overfits to input assumptions.
We consider the following basic model: let A ∈ R n ×
d be a measurement matrix which contains an unknown subset of rows
G ∈ R m × d which are bounded and satisfy the restricted isometry property (RIP), but is otherwise arbitrary.
Letting x ⋆ ∈ R
d be s -sparse, and given either exact measurements b = A x ⋆ or noisy measurements b =
A x ⋆ + ξ , we design algorithms recovering x ⋆ information-theoretically optimally in nearly-linear time.
We extend our algorithm to hold for weaker generative models relaxing our planted RIP row subset assumption to a natural weighted variant, and show that our method’s guarantees naturally interpolate the quality of the measurement matrix to, in some parameter regimes, run in sublinear time.
Our approach differs from that of prior fast iterative methods with provable guarantees under semi-random generative models [23, 54], which typically separate the problem of learning the planted instance from the estimation problem, i.e. they attempt to first learn the planted “good” instance (in our case, G ).
However, natural conditions which make sparse recovery tractable, such as RIP, are NP-hard to verify and hence first learning a sufficient row reweighting appears challenging.
We eschew this approach and design a new iterative method, tailored to the geometry of sparse recovery, which is provably robust to our semi-random model.
We hope our approach opens the door to new robust, efficient algorithms for other natural statistical inverse problems.
1. Introduction
Sparse recovery.
Sparse recovery is one of the most fundamental and well-studied inverse problems, with numerous applications in prevalent real-world settings [39].
In its most basic form, we are given an entrywise Gaussian measurement matrix
G ∈
R m × d and measurements b = G x ⋆ for an unknown s -sparse x ⋆ ∈ R d ; the goal of the problem is to recover x ⋆ .
Seminal works by Cand`es,
Romberg, and Tao [17–19] showed that even when the linear system in G is extremely underconstrained, recovery is tractable so long as m = Ω( s log d ) .
Further they gave a polynomial-time algorithm known as basis pursuit based on linear programming recovering x ⋆ in this regime.
Unfortunately, the runtime of linear programming solvers, while polynomial in the size of the input, can still be prohibitive in many high-dimensional real-world settings.
Correspondingly, a number of alternative approaches which may broadly be considered first-order methods have been developed.
These methods provably achieve similar recovery guarantees under standard generative models such as Gaussian measurements, with improved runtimes compared to the aforementioned convex programming methods.
We refer to these first-order methods through as “fast” algorithms throughout and they may roughly be placed in the following (potentially non-disjoint) categories.
• Greedy algorithms , e.g. [61, 67, 69], seek to greedily find elements in the support of the true x ⋆ using different combinatorial search criteria.
• Non-convex iterative algorithms , e
g. [15, 16, 44, 60, 66], directly optimize a (potentially non-convex) objective over a non-convex domain.
• Convex first-order methods , e.g. [2, 8, 9, 25, 27, 43, 68] quickly solve the convex objective underlying basis pursuit using first-order methods.
We note that theoretically, recent advances by [80, 81], also obtain fast runtimes for the relevant linear program.
The fastest IPM for the noiseless sparse recovery objective runs in time 1 (cid:101) O ( nd + n 2 . 5 ) which is nearly-linear when A is dense and n ≪ d 2 / 3 .
For a range of (superlogarithmic, but sublinear) n , these runtimes are no longer nearly-linear; further, these IPMs are second-order and our focus is on designing first-order methods, which are potentially more practical.
It has often been observed empirically that fast first-order methods can have large error, or fail to converge, in real-world settings [29, 48, 70] where convex programming-based algorithms (while potentially computationally cumbersome) perform well statistically [3, 84].
This may be surprising, given that in theory, fast algorithms essentially match the statistical performance of the convex programming-based algorithms under standard generative assumptions.
While there have been many proposed explanations for this behavior, one compelling argument is that fast iterative methods used in practice are more brittle to changes in modeling assumptions.
We adopt this viewpoint in this paper, and develop fast sparse recovery algorithms which achieve optimal statistical rates under a semi-random adversarial model [14, 41], a popular framework for investigating the robustness of learning algorithms under changes to the data distribution.
Semi-random adversaries.
Semi-random adversaries are a framework for reasoning about algorithmic robustness to distributional shift.
They are defined in statistical settings, and one common type of semi-random adversary is one which corresponds to generative models where data has been corrupted in a “helpful” or “monotone” way.
Such a monotone semi-random adversary takes a dataset from which learning is information-theoretically tractable, and augments it with additional information; this additional information may not break the problem more challenging from an information-theoretic perspective, 3 but may affect the performance of algorithms in other ways.
In this paper, we consider a semi-random adversary which makes the computational problem more difficult without affecting the problem information-theoretically, by returning a consistent superset of the unaugmented observations.
This contrasts with other adversarial models such as gross corruption [4, 47, 77, 78], where corruptions may be arbitrary, and the corrupted measurements incorrect.
It may be surprising that a “helpful” adversary has any implications whatsoever on a learning problem, from either an information-theoretic or computational standpoint.
Typically, convex programming methods for statistical recovery problems are robust to these sorts of perturbations — in brief, this is because constraints to a convex program that are met by an optimum point does not change the optimality of that point.
However, greedy and non-convex methods — such as popular practical algorithms for sparse linear regression — can be susceptible to semi-random adversaries.
Variants of this phenomenon have been reported in many common statistical estimation problems, such as stochastic block models and broadcast tree models [65],
PAC learning [13], matrix completion [23, 64], and principal component regression [11].
This can be quite troubling, as semi-random noise can be thought of as a relatively mild form of generative model misspecification: in practice, the true distribution is almost always different from the models considered in theory.
Consequently, an algorithm’s non-robustness to semi-random noise is suggestive that the algorithm may be more unreliable in real-world settings.
We consider a natural semi-random adversarial model for sparse recovery (see e.g. page 284 of [5]), which extends the standard restricted isometry property (RIP) assumption, which states that applying matrix
A approximately preserves the ℓ 2 norm of sparse vectors.
Concretely, throughout the paper we say matrix A satisfies the ( s, c ) -restricted isometry (RIP) property if for all s -sparse vectors v , 1 c ∥ v ∥ 22 ≤ ∥
A v ∥ 22 ≤ c ∥ v ∥ 22 .
We state a basic version of our adversarial model here, and defer the full statement to Definition 4.
4 We defer stating the notation used in the paper to Section A.
Definition 1 (pRIP matrix)
We say A ∈ R n × d is ρ -pRIP (planted RIP) if there exists G ∈ R m × d for m ≤ n such that each row of G is also a row of A , 1 √ m G is (Θ( s ) , Θ(1)) -RIP for appropriate constants, and ∥ G ∥ max ≤ ρ .
When ρ = (cid:101) O
(1) for brevity we say A is pRIP.
Under the problem parameterizations used in this paper, standard RIP matrix constructions satisfy ρ = (cid:101) O (1) with high probability.
For example, when G is entrywise Gaussian and m = Θ( s log d ) , a tail bound shows that with high probability a matrix A with G inducing a subset of its rows as in Definition 1 is ρ -pRIP, for ρ
= O ( √ log d ) .
pRIP matrices can naturally be thought of as arising from a semi-random adversarial model as follows.
First, an RIP matrix G ∈ R m × d is generated, for example from a standard ensemble (e.g.
Gaussian or subsampled Hadamard).
An adversary inspects G , and forms A ∈ R n × d by reshuffling and arbitrarily augmenting rows of G .
Whenever we refer to a “semi-random adversary” in the remainder of the introduction, we mean the adversary provides us a pRIP measurement matrix A .
The key problem we consider is recovering an unknown s -sparse vector x ⋆ ∈ R d given measurements b ∈ R n through A .
We consider both the noiseless or exact setting where b
=
A x ⋆ and the noisy setting where b
=
A x ⋆ + ξ for bounded ξ .
In the noiseless setting, the semi-random Definition 1 is the sparsity of the vector in an associated sparse recovery problem. adversary hence only gives the algorithm additional consistent measurements of the unknown ssparse vector x ⋆ .
In this sense, the adversary is “helpful,” as it returns a superset of information which is sufficient for sparse recovery (formally, this adversary cannot break the standard restricted nullspace condition which underlies the success of convex programming).
We note n may be much larger than m , i.e. we impose no constraint on how many measurements the adversary adds.
2. Our results
We devise algorithms which match the nearly-linear runtimes and optimal recovery guarantees of faster algorithms on fully random data, but which retain both their runtime and the robust statistical performances of convex programming methods against semi-random adversaries.
In this sense, our algorithms obtain the “best of both worlds.
” We compare more extensively to existing algorithms under Definition 1 in the following section.
We first state our result under noiseless observations.
Theorem 2 (informal, see Theorem 5)
Let x ⋆ ∈ R
d be an unknown s -sparse vector.
Let A ∈
R n × d be pRIP.
There is an algorithm, which given A and b =
A x ⋆ , runs in time (cid:101) O ( nd ) , and outputs x ⋆ with high probability.
Since our problem size is nd , Theorem 2’s runtime is nearly-linear in the input.
We extend our algorithm to handle noisy observations, namely perturbed linear measurements from a pRIP matrix.
Theorem 3 (informal, see Theorem 16)
Let x ⋆ ∈ R
d be an unknown s -sparse vector, and let ξ ∈
R n be arbitrary.
Let A ∈ R n × d be pRIP.
There is an algorithm, which given A and b =
A x ⋆ + ξ , runs in time (cid:101) O ( nd ) , and with high probability outputs x satisfying ∥ x − x ⋆ ∥ 2 ≤ O ( 1 √ m (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 ) , where ξ ( m ) denotes the largest m entries of ξ by absolute value, with other coordinates set to 0 .
The error scaling of Theorem 3 is optimal in the semi-random setting.
Indeed, when there is no semi-random noise, the guarantees of Theorem 3 exactly match the standard statistical guarantees in the fully-random setting for sparse recovery, up to constants; for example, when A = √ m I (which is clearly RIP, in fact an exact isometry, after rescaling), it is information-theoretically impossible to obtain a better ℓ 2 error.
The error bound of Theorem 3 is similarly optimal in the semi-random setting because in the worst case, the largest entries of ξ may correspond to the rows of the RIP matrix from which recovery is information-theoretically possible.
Performance
of existing algorithms
To contextualize Theorems 2 and 3, we discuss the performance of existing algorithms for sparse recovery under the semi-random adversarial model of
Definition 1.
First, it can be easily verified that our semi-random adversary never changes the information-theoretic tractability of sparse recovery.
In the noiseless setting for example, the performance of the minimizer to the classical convex program based on ℓ 1 minimization, min A x = b ∥ x ∥ 1 , is unchanged in the presence of pRIP matrices (as x ⋆ is still consistent with the constraint set, and in particular a RIP constraint set), and hence the semi-random problem can be solved in polynomial time via convex programming.
This suggests the main question we address: can we design a near-linear time algorithm obtaining optimal statistical guarantees under pRIP measurements?
As alluded to previously, standard greedy and non-convex methods we have discussed may fail to converge to the true solution against appropriate semi-random adversaries.
We give explicit counterexamples to several popular methods such as orthogonal matching pursuit and iterative hard thresholding in
Section E. Further, it seems likely that similar counterexamples also break other, more complex methods commonly used in practice, e.g. matching pursuit [61] and CoSaMP [66].
Additionally, while fast “convex” iterative algorithms (e.g. first-order methods for solving objectives underlying polynomial-time convex programming approaches) will never fail to converge to the correct solution given pRIP measurements, the analyses which yield fast runtimes for these algorithms [2, 68] rely on properties such as restricted smoothness and strong convexity (a specialization of standard conditioning assumptions to numerically sparse vectors).
These hold under standard generative models but again can be broken by pRIP measurements; consequently, standard convergence analyses of “convex” first-order methods may yield arbitrarily poor rates.
One intuitive explanation for why faster methods fail is that they depend on conditions such as incoherence [38] or RIP [18], which can be destroyed by a semi-random adversary.
For instance,
RIP states that if S is any subset of m = Θ( s ) columns of A , and A S is the submatrix formed by taking those columns of A , then
A ⊤ S A S is an approximate isometry (i.e. it is well-conditioned).
While it is well-known that RIP is satisfied with high probability when A consists of Θ( s log d )
Gaussian rows, it is straightforward to see that augmenting A with additional rows can ruin the condition number of submatrices of this form.
In contrast, convex methods work under weaker assumptions such as the restricted nullspace property (RNP), which cannot be destroyed by the augmentation used by pRIP matrices.
Though these weaker conditions (e.g. RNP) suffice for algorithms based on convex programming, known analyses of near-linear time “fast” algorithms require additional instance structure, such as incoherence or RIP.
Thus, it is plausible that fast algorithms for sparse recovery are less robust to the sorts of distributional changes that may occur in practice.
Beyond submatrices.
Our methods naturally extend to a more general setting (see Definition 4, wherein we define “weighted RIP” (wRIP) matrices, a generalization of Definition 1).
Rather than assuming there is a RIP submatrix G , we only assume that there is a (nonnegative) reweighting of the rows of A so that the reweighted matrix is “nice,” i.e. it satisfies RIP.
Definition 1 corresponds to the special case of this assumption where the weights are constrained to be either 0 or 1 (and hence indicates a row subset).
In our technical sections (Sections C and D), our results are stated for this more general semi-random model, i.e. sparse recovery from wRIP measurements.
Towards instance-optimal guarantees.
While the performance of the algorithms in Theoremsand 3 is already nearly-optimal in the worst case semi-random setting, one can still hope to improve our runtime and error bounds in certain scenarios.
Our formal results, Theorems 5 and 16, provide these types of fine-grained instance-optimal guarantees in several senses.
In the noiseless setting (Theorem 5), if it happens to be that the entire matrix A is RIP (and not just G ), then standard techniques based on subsampling the matrix can be used to solve the problem in time (cid:101)
O ( sd ) with high probability.
Theorem 5’s runtime smoothly interpolates between the two regimes of a worst-case adversary and an adversary which gives us additional random measurements from an RIP ensemble.
Roughly speaking, if there exists a (a priori unknown) submatrix of A of m ≫ (cid:101) Θ( s ) rows which is RIP, then we show that our algorithm runs in sublinear time (cid:101) O ( nd · sm ) , which is (cid:101)
O ( sd ) when m ≈ n .
We show this holds in our weighted semi-random model (under wRIP measurements, Definition 4) as well, where the runtime depends on the ratio of the ℓ 1 norm of the (best) weight vector to its ℓ ∞ norm, a continuous proxy for the number of RIP rows under pRIP.
We show a similar interpolation holds in the noisy measurement setting, both in the runtime sense discussed previously, and in a statistical sense.
In particular, Theorem 16 achieves (up to logarithmic factors) the same interpolating runtime guarantee of Theorem 5, but further attains a squared ℓ 2 error which is roughly the average of the m largest squared elements of the noise ξ (see
Theorem 3).
This bound thus improves as m ≫ (cid:101) Θ( s ) ; we show it naturally extends to weighted
RIP matrices (Definition 4, generalizing Definition 1), depending on the ℓ ∞ - ℓ 1 ratio of the weights.
Organization.
We provide notation used throughout the paper and supplementary material in Section A, and a brief technical overview of our algorithms in Section
B. The remaining proofs of our results, and counterexamples to existing methods, can be found in Sections C, D, E, and F.
3. Related work
Sparse recovery.
Sparse recovery, and variants thereof, are fundamental statistical and algorithmic problems which have been studied in many settings, e.g. signal processing [7, 15, 38, 53, 74], and compressed sensing [17–19, 37, 73].
A full review of the sparse recovery literature is out of the scope of the present paper; we refer the reader to e.g. [28, 39, 52, 75] for more extensive surveys.
Within the sparse recovery literature, arguably the closest line of work to ours is that which aims to design efficient algorithms which work when the restricted condition number of the sensing matrix is large.
Indeed, it is known that many non-convex methods fail when the restricted condition number of the sensing matrix is far from 1 , which is often the case in applications [48].
To address this, several works [48, 75] have designed novel non-convex methods which still converge, when the restricted condition number of the matrix is much larger than 1 .
However, these methods still require that the restricted condition number is bounded, whereas in our setting, the restricted condition number could be arbitrarily large due to the generality of our adversary.
Another related line of work considers the setting where, instead of having a sensing matrix with rows which are drawn from an isotropic Gaussian, rows are drawn from N (0 , Σ ) , for potentially ill-conditioned Σ [10, 12, 26, 48, 49, 51, 71, 79, 83].
While this setting seems potentially related to ours, there does not appear to be any concrete connection between this “ill-conditioned covariance” setting and the semi-random model we consider.
Indeed, the ill-conditioned setting appears to be qualitatively much more difficult for algorithms: [49] shows evidence that there are in fact no polynomial-time algorithms that achieve the optimal statistical rates, without additional assumptions on Σ .
In contrast in the semi-random setting, polynomial-time convex programming approaches, while having potentially undesirable superlinear runtimes, still obtain optimal statistical guarantees.
Finally as discussed earlier in the introduction, there is a large body of work on efficient algorithms for sparse recovery in an RIP matrix (or a matrix satisfying weaker or stronger analogous properties).
These works e.g. [2, 8, 9, 15–19, 25, 27, 43, 44, 60, 61, 66–69] are typically based on convex programming or different iterative first-order procedures.
Semi-random models.
Semi-random models were originally introduced in a sequence of innovative papers [14, 41] in the context of graph coloring.
In theoretical computer science, semi-random models have been explored in many settings, for instance, for various graph-structured [21, 41, 42, 55, 57] and constraint satisfaction problems [50].
More recently, they have also been studied for learning tasks such as clustering problems and community detection [22, 40, 45, 56, 58, 59, 63, 65], matrix completion [23], and linear regression [54].
We refer the reader to [72] for a more thorough overview of this vast literature.
We remark that our investigation of the semi-random sparse recovery problem is heavily motivated by two recent works [23, 54] which studied the robustness of fast iterative methods to semi-random modeling assumptions.
Appendix A. Preliminaries
General notation.
We let [ n ] :=
{ i ∈ N , 1 ≤ i ≤ n } .
The ℓ p norm of a vector is denoted ∥·∥ p , and the sparsity (number of nonzero entries) of a vector is denoted ∥·∥ 0 .
For a vector v ∈ R d and k ∈ [ d ] , we let v ( k ) be the vector equalling v on the largest k entries of v in absolute value (with other coordinates zeroed out).
The all-zeroes vector of dimension n is denoted 0 n .
The nonnegative probability simplex in dimension n (i.e. ∥ p ∥ 1 = 1 , p ∈ R n ≥ 0 ) is denoted ∆ n .
For mean µ ∈ R d and positive semidefinite covariance Σ ∈ R d × d , N ( µ, Σ ) denotes the corresponding multivariate Gaussian.
i ∼ unif.
S denotes a uniform random sample from set S .
For
N ∈ N and p ∈ ∆
n we we use Multinom ( N, p ) to denote the probability distribution corresponding to N independent draws from [ n ] as specified by p .
Sparsity.
We say v is s -sparse if ∥ v ∥ 0 ≤ s .
We define the numerical sparsity of a vector by
NS ( v ) :
= ∥ v ∥ 21 / ∥ v ∥ 22 .
Note that from the Cauchy-Schwarz inequality, if ∥ v ∥ 0 ≤ s , then NS ( v ) ≤ s .
Matrices.
Matrices are in boldface throughout.
The zero and identity matrix of appropriate dimension from context are
and I .
For a matrix A ∈ R n × d , we let its rows be A i : , i ∈ [ n ] and its columns be A : j , j ∈ [ d ] .
The set of d × d symmetric matrices is S d , and its positive definite and positive semidefinite restrictions are S d ≻ 0 and S d ⪰ 0 .
We use the Loewner partial order ⪯ on S d .
The largest entry of a matrix A ∈ R n × d is denoted ∥ A ∥ max := max i ∈ [ n ] ,j ∈ [ d ] | A ij | .
When a matrix
A ∈ R n × d is clear from context, we refer to its rows as { a i } i ∈ [ n ] .
Short-flat decompositions.
Throughout we frequently use the notion of “short-flat decompositions.”
We say v ∈ R
d has a ( C 2 , C ∞ ) short-flat decomposition if v = p + e for some e ∈ R d with ∥ e ∥ 2 ≤ C 2 and p ∈ R d with ∥ p ∥ ∞ ≤ C ∞ .
Further, we use trunc ( v, c ) ∈ R d for c ∈ R ≥ 0 to denote the vector which coordinatewise [ trunc ( v, c )] i = sgn ( v i ) max( | v i | − c, 0)
(i.e. the result of adding or subtracting at most c from each coordinate to decrease the coordinate’s magnitude).
Note that v ∈ R
d has a ( C 2 , C ∞ ) short-flat decomposition if and only if ∥ trunc ( v, C ∞ ) ∥ 2 ≤ C 2 (in which case p = trunc ( v, C ∞ ) and e = v − p is such a decomposition).
Restricted isometry property.
We say that matrix A ∈ R n × d satisfies the ( s, c ) -restricted isometry property (RIP) or (more concisely)
A is ( s, c ) -RIP , if for all s -sparse vectors v ∈ R d ,
[Equation]
Appendix B. Technical overview
Our overall approach for semi-random sparse recovery is fairly different from two recent works in the literature which designed fast iterative methods succeeding under a semi-random adversarial model [23, 54].
In particular, these two algorithms were both based on the following natural framework, which separates the “planted learning” problem (e.g. identifying the planted benign matrix) from the “estimation” task (e.g. solving a linear system or regression problem).
Compute a set of weights for the data (in linear regression for example, these are weights on each of the rows of a measurement matrix A ), such that after re-weighting , the data fits the input assumptions of a fast iterative method which performs well on a fully random instance.
Apply said fast iterative algorithm on the reweighted data in a black-box manner.
To give a concrete example, [54] studied the standard problem of overdetermined linear regression with a semi-random adversary, where a measurement matrix A is received with the promise that
A contains a “well-conditioned core” G .
The algorithm of [54] first learned a re-weighting of the rows of A by a diagonal matrix W 12 , such that the resulting system in A ⊤ WA is well-conditioned and hence can be solved using standard first-order methods.
In the case of semi-random sparse recovery, there appear to be significant barriers to reweighting approaches (which we will shortly elaborate on).
We take a novel direction that involves designing a new nearly-linear time iterative method for sparse recovery tailored to the geometry of the problem.
Why not (globally) reweight the rows?
There are several difficulties immediately encountered when trying to use the aforementioned reweighting framework for sparse recovery.
First of all, there is no effective analog of condition number for an underdetermined linear system.
The standard assumption on the measurement matrix A to make sparse recovery tractable for fast iterative methods is that A satisfies RIP, i.e. A is roughly an isometry when restricted to O ( s ) -sparse vectors.
However, RIP is NP-hard to verify [6] and this may suggest that it is computationally hard to try, say, learning a reweighting of the rows of A such that the resulting reweighted matrix is guaranteed to be
RIP (though it would be very interesting if this were achievable).
More broadly, almost all explicit conditions (e.g. RIP, incoherence etc.) which make sparse recovery tractable for fast algorithms are conditions about subsets of the columns of A .
Thus, any approach which reweights rows of A such that column subsets of the reweighted matrix satisfy an appropriate condition results in optimization problems that seems challenging to solve in nearly-linear time.
We circumvent these difficulties in two steps.
First, we propose a new analysis of an iterative method based on (reweighted) projected gradient descent, which obtains a fast convergence rate whenever each step satisfies certain locally verifiable properties.
Next, our algorithm computes a sequence of local reweightings (which can be different in each iteration) of the rows of our measurement matrix, such that each local reweighting satisfies our requisite progress conditions for that step.
We use the existence of a global reweighting satisfying RIP to demonstrate that each local reweighting subproblem has a good solution, and design an efficient method for computing each local reweighting.
Our framework of bypassing hardness of computing a global reweighting to recover planted statistical information, by instead designing an iterative method capable of exploiting local reweightings with (computationally tractable) certifiable progress conditions, is quite general, and we hope it will find uses in semi-random settings beyond our particular problem.
Short-flat decompositions: the geometry of sparse recovery.
We now explain our new approach, and how we derive deterministic conditions on the steps of an iterative method which certify progress by exploiting the geometry of sparse recovery.
We focus on the clean observation setting in this technical overview.
Suppose that we wish to solve a sparse regression problem A x ⋆ = b where x ⋆ is s -sparse, and we are given A and b .
To fix a scale, suppose for simplicity that we know ∥ x ⋆ ∥ 1 = √ s and ∥ x ⋆ ∥ 2 = 1 .
Also, assume for the purpose of conveying intuition that A is pRIP, and that the planted matrix G in Definition 1 is an entrywise random Gaussian matrix.
Our starting point is the observation that in the simpler case when we are given a Gaussian matrix G with rows { g i } i ∈ [ m ] ⊂ R d , we can compute the vector
[Equation]
We remark v is the gradient of the regression objective 12 m ∥ G x − b ∥ 22 at x = 0 .
Moreover, since each g i ∼ N (0 , I ) independently, we have E v = x ⋆ , and hence it is natural to hope v has good correlation in the x ⋆ direction which we can use to make progress.
Unfortunately, v also contains information in the subspace orthogonal to x ⋆ , and moreover it is not hard to show that most of the ℓ 2 mass of v is in fact orthogonal to x ⋆ .
In particular, it is very likely that ∥ v ∥ 22 = Ω( dn ) , whereas ∥ x ⋆ ∥ 2 = 1 (implying it is unlikely ⟨ v, x ⋆ ⟩ is superconstant).
In the underconstrained setting where d ≫ n , this suggests that the orthogonal “noise” overwhelms the signal towards x ⋆ .
To bypass this issue, we identify a key structural property of the “noise” v − x ⋆ : it is very “flat”, meaning it is likely to be spread out amongst coordinates, such that ∥ v − x ⋆ ∥
[Equation]
O ( 1 √ s ) .
More generally, we say a vector v ∈ R
d has a ( C 2 , C ∞ ) short-flat decomposition if
[Equation]
It is helpful to view p as the “signal” component towards x ⋆ , and e as the “noise” component.
As discussed, in the Gaussian setting a typical v has an ( O (1) , O ( 1 √ s ))
-short flat decomposition, as witnessed by p = x ⋆ .
Unfortunately, typically in this decomposition ∥ e [-(cid:107) 
 (cid:29) (cid:107)-]∥ 2 ≫ ∥ p ∥ 2 .
Our second main observation is that projection against an ℓ 1 ball preserves most of the signal direction p , but removes almost all of the noise direction e .
Intuitively, this is because the ℓ 1 ball is very “thick” (i.e. has large ℓ 2 diameter) in sparse directions such as the progress direction x ⋆ , due to their small ℓ 1 to ℓ 2 ratios.
On the other hand, the ℓ 1 ball is very “thin” in generic directions with large ℓ 1 to ℓ 2 ratios, which is typical of our noise.
Hence, a natural algorithm is to start from an iterate x t = 0 , and take the steps
[Equation]
where Π projects against an ℓ 1 ball of radius O ( √ s ) and η = Θ(1) is a step-size parameter.
A diagram of the induced movement is presented in Figure 1.
We show in Section C.3 that more generally, whenever G is RIP, 1 m GG ⊤ v has a short-flat decomposition whenever ∥ v [-(cid:107) 
-]∥ 1 =
O ( √ s ∥ v ∥ 2 ) (i.e. v is “numerically sparse”).
By occasionally rounding our iterate to be s -sparse, and redrawing an ℓ 1 ball accordingly, we can guarantee that the progress direction always satisfies this numerical sparsity property.
We next build upon intuition that projected gradient steps succeed when the gradient has good correlation in the progress direction (towards x ⋆ ) and has “flat” orthogonal movement.
It is not difficult to demonstrate (via ℓ 1 - ℓ ∞ H¨older) that restarted projected gradient descent against an ℓball linearly converges to x ⋆ , as long as two properties (with appropriate parameters) hold. The gradient
step has constant correlation with the x
⋆ direction.
The gradient step admits a short-flat decomposition.
We formalize this intuition in Section C.1, where we analyze our main framework, Algorithm 1, by bounding the progress made by projected descent along directions with short-flat decompositions.
We observe that the sufficiency of this set of properties is desirable for two important reasons.
First, it continues to hold in the semi-random setting (e.g. when the observation matrix we receive satisfies Definition 1).
Second, it bypasses the potential hardness of learning a globally
[Figure]
RIP reweighting, by instead requiring locally verifiable properties of a reweighting which are sufficient to guarantee progress.
By exploiting these characteristics, we extend our framework to the semi-random setting.
Finding short-flat decompositions: semi-random sparse recovery.
The last piece of our main algorithm is an optimization method for recovering the planted structure of a progress direction under semi-random observations.
Concretely, consider now the case where our observation matrix
A satisfies Definition 1.
From an iterate x t such that ∥ x t − x ⋆ ∥ 2 = 1 and ∥ x t − x [-(cid:63) (cid:107) 
-]⋆ ∥ 1 = O ( √ s ) , we consider a family of reweighted projected gradient descent algorithms, where we take
[Equation]
for some reweighting vector w t .
This is motivated by the fact that (as argued in the previous section) setting w t to be 1 m times the indicator of the rows of G recovers the fully-RIP method.
Of course, we do not know which rows belong to G , so it remains to show how to choose a “competitive” w t .
In Section C.2, we set up a potential function which captures the correlation with the x t − x ⋆ direction of our reweighted gradient step, and where approximate optimality implies the existence of a short-flat decomposition.
Finally, we design a stochastic gradient method to approximately optimize this potential function in nearly-linear time, completing our algorithm.
Appendix C. Exact recovery
In this section, we give an algorithm for solving the underconstrained linear system A x ⋆ = b given the measurement matrix A ∈ R n × d (for n ≤ d ) and responses b ∈ R n (i.e. noiseless or “exact” regression), and x ⋆ is s -sparse.
Our algorithm succeeds when A is weighted RIP (wRIP), i.e. it satisfies Definition 4, a weighted generalization of Definition 1.
Definition 4 (wRIP matrix) Let w ⋆ ∞ ∈ [0 , 1] .
We say A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP if ∥ A ∥ max ≤ ρ , and there exists a weight vector w ⋆ ∈
∆ n satisfying ∥ w ⋆ ∥ ∞ ≤ w ⋆ ∞ , such that diag ( w ⋆ ) {+
+}
A is (Θ( s ) , Θ(1)) -RIP for appropriate constants.
When ρ = (cid:101) O
(1) for brevity we say A is w ⋆ ∞ -wRIP.
As discussed after Definition 1, a wRIP matrix can be thought of as arising from a “semi-random model” because it strictly generalizes our previously-defined pRIP matrix notion in Definition 1 with w ⋆ ∞ = 1 m , by setting w ⋆ to be 1 m times the zero-one indicator vector of rows of G .
The main result of this section is the following theorem regarding sparse recovery with wRIP matrices.
Theorem 5 Let δ ∈ (0 , 1) , r > 0 , and suppose R 0 ≥ ∥ x ⋆ ∥ 2 for s -sparse x ⋆ ∈ R d .
Then with probability at least 1 − δ , Algorithm 1 using Algorithm 2 as a step oracle takes as input a ( ρ, w ⋆ ∞ ) - wRIP matrix A ∈ R n × d and b =
A x ⋆ , and computes ˆ x satisfying ∥
ˆ x − x ⋆ ∥ 2 ≤ r in time
O [Equation] (cid:19)(cid:19)
[Equation]
Under the wRIP assumption, Theorem 5 provides a natural interpolation between the fully random and semi-random generative models.
To build intuition, if a pRIP matrix contains a planted RIP matrix with (cid:101) O ( s ) rows (the information-theoretically minimum size), then by setting w ⋆ ∞
≈ (cid:101) O
( s ) , we obtain a near-linear runtime of (cid:101) O ( nd ) .
However, in the fully random regime where w ⋆ ∞ ≈ 1 n (i.e. all of A is RIP), the runtime improves (cid:101) O
( sd ) which is sublinear for n ≫ s .
The roadmap of our algorithm and its analysis are as follows.
In Section C.1, we give an algorithm (Algorithm 1) which iteratively halves an upper bound on the radius to x ⋆ , assuming that either an appropriate step oracle (see Definition 6) based on short-flat decompositions can be implemented for each iteration, or we can certify that the input radius bound is now too loose.
This algorithm is analyzed in Lemma 9.
We state in Assumption 1 a set of conditions on a matrix-vector pair ( A , ∆) centered around the notion of short-flat decompositions, which suffice to provide a sufficient step oracle implementation with high probability in nearly-linear time.
In Section C.2 we analyze this implementation (Algorithm 2) in the proof of Lemma 7 assuming the inputs satisfy Assumption 1. 3.
In Section C.3, we show Assumption 1, with appropriate parameters, follows from A being wRIP.
This is a byproduct of a general equivalence we demonstrate between RIP, restricted conditioning measures used in prior work [1] and short-flat decompositions.
C.1. Radius contraction using step oracles
In this section, we provide and analyze the main loop of our overall algorithm for proving Theorem 5.
This procedure, HalfRadiusSparse , takes as input an s -sparse vector x in and a radius bound
R ≥ ∥ x in − x ⋆ ∥ 2 and returns an s -sparse vector x out with the guarantee ∥ x out − x ⋆ ∥ 2 ≤ 12 R .
As a subroutine, it requires access to a “step oracle” O step , which we implement in Section C.2 under certain assumptions on the matrix A .
Definition 6 (Step oracle)
We say that O step is a ( C prog , C 2 , δ ) -step oracle for ∆ ∈ R n and A ∈
R n × d , if the following holds.
Whenever there is v ∈ R d with 14 ≤ ∥ v ∥ 2 ≤ 1 and ∥ v ∥ 1 ≤ 2 √ 2 s such that ∆ =
A v , with probability ≥ 1 − δ , O step returns w ∈ R n ≥ 0 such that the following two conditions hold.
First,
[Equation]
Second, there exists a ( C 2 , C prog 6 √ s ) short-flat decomposition of A ⊤ diag ( w ) ∆ :
[Equation]
Intuitively, (2) guarantees that we can write γ = p
+ e where p denotes a “progress” term which we require to be sufficiently short in the ℓ 2 norm, and e denotes an “error” term which we require to be small in ℓ ∞ .
We prove that under certain assumptions on the input A (stated in Assumptionbelow), we can always implement a step oracle with appropriate parameters.
Assumption 1 The matrix A ∈ R n × d satisfies the following.
There is a weight vector w ⋆ ∈
∆ n satisfying ∥ w ⋆ ∥ ∞ ≤ w ⋆ ∞ , a constant L , ρ ≥ 1 , and a constant K (which may depend on L ) such that for all v ∈ R d with 14 ≤ ∥ v ∥ 2 ≤ 1 and ∥ v ∥ 1 ≤ 2 √ 2 s we have, defining ∆ =
A v :
A is entrywise bounded by ± ρ ,
i
e. ∥ A ∥ max ≤ ρ . 2.
[Equation]
For W ⋆ := diag ( w ⋆ ) , there is a ( L, 1 K √ s ) short-flat decomposition of A
[Equation]
Our Assumption 1 may also be stated in a scale-invariant way (i.e. with (3), (4) scaling with ∥ v ∥ 2 ), but it is convenient in our analysis to impose a norm bound on v .
Roughly, the second property in Assumption 1 is (up to constant factors) equivalent to the “restricted strong convexity” and “restricted smoothness” assumptions of [1], which were previously shown for specific measurement matrix constructions such as random Gaussian matrices.
The use of the third property in Assumption 1 (the existence of short-flat decompositions for numerically sparse vectors) in designing an efficient algorithm is a key contribution of our work.
Interestingly, we show in Section C.3 that these assumptions are up to constant factors equivalent to RIP.
More specifically, we show that when A is wRIP, we can implement a step oracle for ∆ =
A v where v = 1 R ( x − x ⋆ ) for some iterate x of Algorithm 1, which either makes enough progress to advance the algorithm or certifies that v is sufficiently short, by using numerical sparsity properties of v .
We break this proof into two parts.
In Lemma 7, we show that Assumption 1 suffices to implement an appropriate step oracle; this is proven in Section C.2.
In Lemma 8, we then demonstrate the wRIP assumption with appropriate parameters implies our measurement matrix satisfies
Assumption 1, which we prove by way of a more general equivalence in Section C.3.
Lemma 7 Suppose A satisfies Assumption 1.
Algorithm 2 is a ( C prog , C 2 , δ ) step oracle StepOracle for (∆ , A ) with C prog = Ω(1) , C
= O (1) running in time
[Equation]
Lemma 8 Suppose A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP with a suitable choice of constants in the RIP parameters in Definition 4.
Then, A also satisfies Assumption 1.
We now give our main algorithm HalfRadiusSparse , assuming access to the step oracle O step from Section C.2 with appropriate parameters, and that A obeys Assumption 1.
: s sparse x ∈ R d , R ≥ x in − x s
O step for all (∆ , A ) with ∆ ∈ R n , δ ∈ (0 , 1) , A ∈ R n × d , b
= A x ⋆ ∈ R n ;
R with probability ≥ 1 − T δ ; 3 Set x 0 ← x in , X ← { x ∈ R d | ∥ x − x in ∥ 1 ≤ √ 2 sR } T ← (cid:108) 12 C 22 C 2 prog (cid:109) , η ← C prog 2 C 22 ; 4 for 0 ≤ t ≤ T − 1 do w t ← O step (∆ t , A ) for ∆ t ← 1 R ( A x t − b ) , γ t ← A ⊤ diag ( w t ) ∆ t = (cid:80) i ∈ [ n ] [ w t ] i [∆ t ] i a i ; if (cid:80) i ∈ [ n ] [ w t ] i [∆ t ]
i < C prog or (cid:13)(cid:13)(cid:13) trunc ( γ t , C prog 6 √ s ) (cid:13)(cid:13)(cid:13) 2 > C 2 then
Return: x out ← [ x t ] ( s ) end else x t +1 ← argmin x ∈X ∥ x − x t − ηRγ t ∥ 2 ; 10 end 11 Return: x out ← [ x t ] ( s )
Lemma 9 Assume A satisfies Assumption 1.
With probability at least 1 − T δ , Algorithm 1 succeeds (i.e. ∥ x out − x ⋆ ∥ 2 ≤ 12 R ).
Proof Throughout this proof, condition on the event that all step oracles succeed (which provides the failure probability via a union bound).
We first observe that x ⋆ ∈ X because of Cauchy-Schwarz, the 2 s -sparsity of x in − x ⋆ , and the assumption ∥ x in − x ⋆ ∥ 2 ≤ R .
Next, we show that in every iteration t of Algorithm 1,
[Equation]
As x ⋆ ∈ X , the optimality conditions of x t +1 as minimizing ∥ x − ( x
[Equation]
− ηRγ t ) ∥ 22 over X imply
[Equation]
Hence, it suffices to lower bound the right-hand side of the above expression.
Let γ t = p t + e t denote the ( C 2 , C prog 6 √ s ) short-flat decomposition of γ t which exists by Definition 6 assuming the step oracle succeeded.
We begin by observing
[Equation]
The first inequality followed from H¨older on the first term, Cauchy-Schwarz on the second term, and then applying Young’s inequality on the latter two terms in the preceding line.
The second followed from the ℓ 1 radius of X , and the bounds on e t and p t from (2).
Next, from Definition 6, for ∆ =
∆ t = 1 R ( A x t − b ) and v = 1 R ( x t − x ⋆ ) ,
[Equation]
Finally, (5) follows from combining (6), (7), and (8), with our choice of η , and the fact that inducting on this lemma implies the ℓ 2 distance to x ⋆ of the iterates is monotone decreasing.
Next, we claim that regardless of whether Algorithm 1 terminates on Line 7 or Line 11, we have ∥ x t − x ⋆ ∥ 2 ≤ 14 R .
Note that the vector v = 1 R ( x t − x ⋆ ) satisfies A v = ∆ := 1 R ( A x t − b ) .
By assumption the condition ∥ v ∥ 1 ≤ 2 √ 2 s is met (since x t , x ⋆ ∈ X ), and upon iterating (5) on our radius bound assumption, this implies that the condition ∥ v ∥ 2 ≤ 1 is met.
Hence, if the algorithm terminated on Line 7, we must have ∥ v ∥ 2 ≤ 1 4 R
[Equation]
≤ 1 4 R , as otherwise the termination condition would have been false.
On the other hand, by (5), after T steps we have
[Equation]
We conclude that at termination, ∥ x t − x ⋆ ∥ 2 ≤ 14 R .
Now, s -sparsity of x ⋆ and the definition of x out = argmin ∥ x ∥ 0 ≤ s ∥ x − x t ∥ 2 imply the desired [Equation]
[Equation]
C.2. Designing a step oracle
In this section, we design a step oracle O step (∆ , A ) (see Definition 6) under Assumption 1 on the input matrix A ∈ R n × d .
Our step oracle iteratively builds a weight vector ¯ w ∈ R n ≥ 0 .
It will be convenient to define
[Equation]
Note that a valid step oracle always exists (although it is unclear how to implement the following solution): namely, setting ¯ w = w ⋆ satisfies the oracle assumptions by the second and third conditions in Assumption 1.
In order to ensure Algorithm 5 is indeed a step oracle, we track two potentials for some µ , C we will define in Algorithm 2:
[Equation]
where sqmax µ ( x ) := µ 2 log 
[Equation]
Intuitively, Φ 2 ( ¯ w ) corresponds to progress on (1), and Φ sqmax ( ¯ w ) is intended to track the bounds (2).
We note the following fact about the sqmax function which follows from direct calculation.
Fact 1 For all x ∈ R d , ∥ x ∥ 2 ∞ ≤ sqmax µ ( x ) , and sqmax µ ( x ) ≥ µ 2 log( d ) .
Also it will be important to note that Φ sqmax ( ¯ w ) can be computed to high precision efficiently.
We state this claim in the following and defer a full proof to Section F; we give a subroutine which performs a binary search on a Lagrange multiplier on the ℓ 2 constraint on p , and then solves for each optimal p j using another binary search based on the Lagrange multiplier value.
Lemma 10 Let δ > 0 and θ ≥ 0 .
For any vector γ ∈ R d , we can solve the optimization problem
[Equation]
to additive accuracy δ in time
[Equation]
We state the full implementation of our step oracle as Algorithm 2 below.
Our main helper lemma bounds the expected increase in Φ sqmax from choosing a row of A uniformly at random, and choosing a step size according to w ⋆ .
We do not know w ⋆ , but we argue that our algorithm makes at least this much expected progress.
Define the decomposition promised by (4):
[Equation]
Furthermore, define for all i ∈ [ n ] ,
[Equation]
where p ⋆ is given above.
We use { z ( i ) } i ∈ [ n ] as certificates of Φ sqmax ’s growth in the following.
1 Input: ∆ ∈ R n , A ∈ R n × d satisfying Assumption 1, δ ∈ (0 , 1) ; 2 Output: w such that if there is v ∈ R d with 14 ≤ ∥ v ∥ 2 ≤ 1 and ∥ v ∥ 1 ≤ 2 √ 2 s such that ∆ =
A v , with probability ≥ 1 − δ , (1), (2) are satisfied with Cp = 1 , C
= O (1) .
3 C ← 200 , µ ← 1 √ Cs log d , η ← 1 Kw ⋆ ∞ sρ 2 log d , N ′ ← ⌈ log 2 1 δ ⌉ ; 4 for 0 ≤ k ≤ N ′ do w 0 ← 0 n , N ← ⌈ 5 Lnη ⌉ ; for 0 ≤ t ≤ N do if Φ 2 ( w t ) ≥ 1 then Return: w ← w t ; Sample i ∼ unif.
[ n ] ;
Compute (using Lemma 10) d t ∈ [0 , ηw ⋆ ∞ ] maximizing to additive O ( ηn )
[Equation]
(12) w t +1 ← w t
+ d t e i ; end 11 end 12 Return: w ← 0 n ;
Lemma
Assume that the constant K in Assumption 1 is sufficiently large, and that ∆ =
A v where v satisfies the norm conditions in Assumption 1.
Then for any ¯ w ∈ R n ≥ 0 such that Φ sqmax ( ¯ w ) ≤ C 2 µ 2 log d , and η ≤ 1 Kw ⋆ ∞ sρ 2 log d , we have
[Equation]
Proof We assume for simplicity L ≥ 2 √ 2 as otherwise we may set L ← max(2 √ 2 , L ) and (3) remains true.
Let p ¯ w be the minimizing argument in the definition of Φ sqmax ( ¯ w ) in (11).
For any i ∈ [ n ] , p ¯ w + ( ηw ⋆i ) p ⋆ is a valid argument for the optimization problem defining Φ sqmax ( ¯ w + ηw ⋆i ) .
This follows since ∥ p ⋆ ∥ 2 ≤ L , and since ∥ w ∥ 1 grows by ηw ⋆i .
Next, define
[Equation]
such that Φ sqmax ( ¯ w ) = µ 2 log F ( x )+ ∥ ¯ w ∥ 1 4 CLs for x = γ ¯ w − p ¯ w .
As discussed earlier, since ∥ p ¯ w + ( ηw ⋆i ) p ⋆ ∥ 2 ≤ ∥ p ¯ w ∥ 2 + ηw ⋆i L , we conclude
[Equation]
We next compute
[Equation]
We now bound the right-hand side of this expression.
For any i ∈ [ n ] and j ∈ [ d ] , recalling (13),
[Equation]
[Equation]
The second inequality used our bounds from Assumption 1; note that for ∆ =
A v where v satisfies the norm conditions in Assumption 1, | ∆ i | ≤ ρ ∥ v ∥ 1 ≤ 2 √ 2 sρ .
Hence, if we choose a sufficiently large constant K in Assumption
[Equation]
we have
[Equation]
Also by the assumption that Φ sqmax ( ¯ w ) ≤ C 2 µ 2 log
d we must have that for all j ∈ [ d ] ,
[Equation]
Now, using exp( c ) ≤ 1 + c + c 2 for | c | ≤ 1 , we get (cid:88) i ∈
[ n ] exp
[Equation]
We control the first-order term via the observation that (cid:80) i ∈ [ n ] z ( i ) = ηe ⋆ which is ℓ ∞ -bounded from (4), so taking the constant K in Assumption 1 sufficiently large, we have
[Equation]
In the last inequality we assumed K ≥ 16 C 1 .
5 L .
We control the second-order term by using ( a + b ) [-2 ≤ 2 a 
-][Equation]
, ∥ p ⋆ ∥ ∞ ≤ ∥ p ⋆ ∥ 2 ≤ L
[Equation]
and (3): (cid:88)
[Equation]
Putting together (18), (19)
[Equation]
and (20), with the definition of µ , we conclude for sufficiently large K ,
[Equation]
Hence, combining the above with (16)
[Equation]
and using log(1 + c ) ≤ c for all c , µ
[Equation]
i
[Equation]
[Equation]
Finally, we compute via (15) and concavity of log ,
[Equation]
In the last line [Equation] we used the bound (21).
Finally, we can complete the analysis of Algorithm 2.
Lemma 12 Suppose A satisfies Assumption 1.
Algorithm 2 is a ( C prog , C 2 , δ ) step oracle StepOracle for (∆ , A ) with C prog = Ω(1) , C
= O (1) running in time
[Equation]
Proof
It suffices to prove Algorithm 2 meets its output guarantees in this time.
Throughout this proof, we consider one run of Lines 5-10 of the algorithm, and prove that it successfully terminates on
Line 7 with probability ≥ 12 assuming A satisfies Assumption 1 and that ∆ =
A v for v satisfying the norm bounds in Assumption 1.
This yields the failure probability upon repeating N ′ times.
For the first part of this proof, we assume we can exactly compute ∆ t , and carry out the proof accordingly.
We discuss issues of approximation tolerance at the end, when bounding the runtime.
Correctness.
We use the notation A t :=
Φ 2 ( w t ) , B t :=
Φ sqmax ( w t ) , and Φ t :=
A t − CsB t .
We first observe that A t is 1 -Lipschitz, meaning it can only increase by 1 in any given iteration; this follows from ηw ⋆ ∞ ∆ 2 i ≤ [Equation] since ∆ [Equation]
=
[Equation]
, v [Equation] [Equation] by ℓ ∞ - ℓ 1 H¨older.
Suppose some run of Lines 5-13 terminates by returning on Line 8 in iteration T , for 0 ≤ T ≤
N .
The termination condition implies that A T ≥ 1 = C prog , so to show that the algorithm satisfies
Definition 6, it suffices to show existence of a short-flat decomposition in the sense of (2).
Clearly, Φ t is monotone non-decreasing in t , since we may always force Γ t = 0 by choosing d t = 0 .
Moreover, Φ 0 = − CsB
= − Csµ 2 log d = − 1 .
The above Lipschitz bound implies that A T ≤ 2 , since A T − 1 ≤ 1 by the termination condition; hence,
[Equation]
Note that the above inequality and nonnegativity of sqmax µ imply that ∥ w T ∥ 1 4 LCs ≤ 3 Cs , so ∥ w T ∥ 1 ≤ 12 L .
For the given value of C = 200 , and the first inequality in Fact 1, the definition of the first summand in B implies there is a short-flat decomposition meeting (2) with C [-
-]2 =
L ∥ w T [-(cid:107) 
-]∥ 1 = O (1) .
Hence, we have shown that Definition 6 is satisfied whenever the algorithm returns on Line 7.
We make one additional observation: whenever Φ t ≥ 0 , the algorithm will terminate.
This follows since on such an iteration,
[Equation]
since clearly the function B is minimized by the all-zeroes weight vector, attaining value µ 2 log d .
Success probability.
We next show that with probability at least 12 , the loop in Lines 5-10 will terminate.
Fix an iteration t .
When sampling i ∈ [ n ] , the maximum gain in Φ t for d t ∈ [0 , ηw ⋆ ∞ ] is at least that attained by setting d t = ηw ⋆i , and hence
[Equation]
Here, we used that the expected gain in A t by choosing d t = ηw ⋆i over a uniformly sampled i ∈ [ n ] is lower bounded by ηLn via (3), and the expected gain in CsB t is upper bounded by Lemma 11.
Let Z t be the random variable equal to Φ t − Φ 0 , where we freeze the value of w t ′ for all t ′ ≥ t if the algorithm ever returns on Line 8 in an iteration t .
Notice that Z t ≤ 2 always: whenever Z t ≥ 1 , we have Φ t ≥
so the algorithm will terminate, and Z t is 1 -Lipschitz because A t is.
Moreover, whenever we are in an iteration t where Pr[ A t ≥ 1] ≤ 12 , applying (22) implies
[Equation]
Clearly, Pr[ A t ≥ 1] is a monotone non-decreasing function of t , since A t is monotone.
After
N ≥ 5 Lnη iterations, if we still have Pr[ A t ≥ 1] ≤ 12 , we would obtain a contradiction since recursing the above display yields E
[ Z N ] > 2 .
This yields the desired success probability.
Runtime.
The cost of each iteration is dominated by the following computation in Line 9: we wish to find d ∈ [0 , ηw ⋆ ∞ ] maximizing to additive O ( ηn ) the following objective:
[Equation]
We claim the above function is a concave function of d .
First, we show Φ sqmax is convex (and the result will then follow from linearity of Φ 2 ).
To see this, for two values w i and w ′ i , let the corresponding maximizing arguments in the definition of Φ sqmax ( ¯ w + w i ) and Φ sqmax ( ¯ w + w ′ i ) be denoted p and p ′ .
Then, 12 ( p + p ′ ) is a valid argument for ¯ w + 12 ( w i + w ′ i ) , and by convexity of sqmax µ and linearity of the ℓ 1 portion, we have the conclusion.
Next, note that all | ∆ i | are bounded by 2 √ 2 sρ (proven after (17)) and all a ij are bounded by ρ by assumption.
It follows that the restriction of Φ 2 to a coordinate is 8 sρ 2 -Lipschitz.
Moreover the linear portion of Φ sqmax is clearly 1 4 CLs -Lipschitz in any coordinate.
Finally we bound the Lipschitz constant of the sqmax part of Φ sqmax .
It suffices to bound Lipschitzness for any fixed p of sqmax
[Equation]
) because performing the minimization over p involved in two sqmax ( γ ¯ w − p + d ∆ i a i ) and sqmax ( γ ¯ w − p + d ′ ∆ i a i ) can only bring the function values closer together.
By direct computation the derivative of the above quantity with respect to d i is (cid:88)
[Equation]
for some probability density vector q ∈ ∆ d .
Further we have
[Equation]
Here we used our earlier proof that we must only consider values of ∥ p ∥
= O (1) throughout the algorithm (since ∥ w t [-(cid:107) 
-]∥ 1 = O (1) throughout)
and this also implies no coordinate of γ ¯ w can be larger than (max i ∈ [ n ] | ∆ i | )
(max i ∈ [ n ] ,j ∈ [ d ] | a ij | ) ∥ ¯ w ∥ 1 by definition of γ ¯ w .
Combined with our bounds on linear portions this shows Φ 2 and Φ sqmax are poly ( ndρ ) -Lipschitz.
Hence, we may evaluate to the desired O ( ηn ) accuracy by approximate minimization of a Lipschitz convex function over an interval (Lemma 33, [24]) with a total cost of O ( d log 3 ( ndρ )) .
Here we use the subroutine of Lemma 10 in Lemma 33 of [24], with evaluation time O
( d log 2 ( ndρ )) .
The algorithm then runs in NN ′ iterations, each bottlenecked by the cost of approximating Γ t ; combining these multiplicative factors yields the runtime.
We note that we do not precompute ∆ =
A v ; we can compute coordinates of ∆ in time O
( d ) as they are required by Algorithm 2.
C.3. Equivalence between Assumption 1 and RIP
The main result of this section is an equivalence between Assumption 1 and the weighted restricted isometry property, which requires two helper tools to prove.
The first is a “shelling decomposition.”
Lemma 13 Let v ∈ R
d have NS ( v ) ≤ σ .
Then if we write v = (cid:80) l ∈ [ k ] v ( l ) where v (1) is obtained by taking the s largest coordinates of v , v (2) is obtained by taking the next s largest coordinates and so on (breaking ties arbitrarily so that the supports are disjoint), we have
[Equation]
Proof Note that the decomposition greedily sets v ( l ) to be the s largest coordinates (by absolute value) of v − (cid:80) l ′ ∈
[ l − 1] v ( l ′ ) , zeroing all other coordinates and breaking ties arbitrarily.
This satisfies
[Equation]
The last inequality follows since every entry of v ( l ) is larger than the largest of v ( l +1) in absolute value.
Finally, summing the above equation and using disjointness of supports yields
[Equation]
The second bounds the largest entries of image vectors from the transpose of an RIP matrix.
Lemma 14 Let A ∈ R n × d be ( s, c ) -RIP, and let u ∈ R n .
Then,
[Equation]
Proof Let v =
[ A ⊤ u ] ( s ) .
The lemma is equivalent to showing ∥ v ∥ 2 ≤ c ∥ u ∥ 2 .
Note that
[Equation]
The first inequality used Cauchy-Schwarz, and the second applied the RIP property of A to v , which is s -sparse by construction.
The conclusion follows via dividing by ∥ v ∥ 2 .
Using these helper tools, we now prove the main result of this section.
Lemma
The following statements are true.
If A satisfies Assumption 1 with weight vector w ⋆ , then ( W ⋆ ) 12 A is ( s, L ) -RIP.
If the matrix ( W ⋆ ) 12 A is RIP with parameters
[Equation]
for L ≥ 1 , and ∥ A ∥ max ≤ ρ , then A satisfies Assumption 1.
Proof We prove each equivalence in turn.
Assumption 1 implies RIP.
The statement of RIP is scale-invariant, so we will prove it for all s -sparse unit vectors v without loss of generality.
Note that such v satisfies the condition in Assumption 1, since ∥ v ∥ 2 = 1 and ∥ v ∥ 1 ≤ √ s by Cauchy-Schwarz.
Then, the second condition of
Assumption 1 implies that for ∆ =
A v , we have the desired norm preservation:
[Equation]
Boundedness and RIP imply Assumption 1.
Let v ∈ R d satisfy 14 ≤ ∥ v ∥ 2 ≤ 1 and ∥ v ∥ 1 ≤ 2 √ 2 s , and define ∆ :
= A v .
The first condition in Assumption 1 is immediate from our assumed entrywise boundedness on A , so we begin by demonstrating the lower bound in (3).
Let s ′ =
12800 L 3 K 2 s and let v (1) , . .
, v ( k ) be the shelling decomposition into s ′ -sparse vectors given by Lemma 13, where σ = 128 s from the ℓ 1 and ℓ 2 norm bounds on v .
By Lemma 13, we have
[Equation]
In particular, the triangle inequality then implies 0 .
9 ∥ v ∥ 2 ≤ (cid:13)(cid:13)
v (1) (cid:13)(cid:13) 2 ≤ ∥ v ∥ 2 .
Next, recall that (cid:80) i ∈ [ n
[Equation]
=
[Equation]
W
[Equation]
[Equation]
v
[Equation]
By applying the triangle inequality and since ( W ⋆ ) 12 A is RIP,
[Equation]
In the second inequality
[Equation]
we applied the RIP assumption to each individual term, since all the vectors are s ′ -sparse.
Similarly, to show the upper bound in (3)
[Equation]
we have
[Equation]
It remains to verify the final condition of Assumption 1.
First, for u :=
W 12 A v , by applying the shelling decomposition to v into s ′ -sparse vectors { v ( l ) } l ∈ [ k ] , [Equation] Here
[Equation]
we used our earlier proof to bound the contribution of all terms but v (1) .
Applying Lemmato the matrix ( W ⋆ ) 12 A and vector u , we have for ∆ =
A v ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] By setting the ℓ 2 bounded component in the short-flat decomposition of A ⊤ W ⋆ ∆ to be the top s ′ entries by magnitude, it remains to show the remaining coordinates are ℓ ∞ bounded by 1 K √ s .
This follows from the definition of s ′ and (23), which imply that the s ′ + 1 th largest coordinate (in magnitude) cannot have squared value larger than L 2 s ′ ≤ 1 K 2 s without contradicting (23).
Finally, it is immediate that Lemma 8 follows from Lemma 15.
C.4.
Putting it all together
At this point, we have assembled the tools to prove our main result on exact recovery.
Theorem 5 Let δ ∈ (0 , 1) , r > 0 , and suppose R 0 ≥ ∥ x ⋆ ∥ 2 for s -sparse x ⋆ ∈ R d .
Then with probability at least 1 − δ , Algorithm 1 using Algorithm 2 as a step oracle takes as input a ( ρ, w ⋆ ∞ ) - wRIP matrix A ∈ R n × d and b =
A x ⋆ , and computes ˆ x satisfying ∥ ˆ x − x ⋆ ∥ 2 ≤ r in time
[Equation]
Proof With probability at least 1 − δ , combining Lemma 7 and Lemma 8 implies that Assumptionholds for all v ∈ R d where 14 ≤ ∥ v ∥ 2 ≤ 1 and ∥ v ∥ 1 ≤ 2 √ 2 s , and that for N =
O (log R 0 r ) , we can implement a step oracle for N runs of Algorithm 1 in the allotted time, each with failure probability 1 − δ N .
Moreover, Algorithm 1 returns in O (1) iterations, and allows us to halve our radius upper bound.
By taking a union bound on failure probabilities and repeatedly running Algorithm 1 N times, we obtain a radius upper bound of r with probability ≥ 1 − δ .
Appendix D. Noisy recovery
In this section, we give an algorithm for solving a noisy sparse recovery problem in a wRIP matrix
A ∈ R n × d (where we recall Definition 4).
In particular, we assume that we receive
[Equation]
[Equation] for an arbitrary unknown ξ ∈ R n , and x ⋆ ∈ R d is s -sparse.
Throughout this section, we will define
[Equation]
where w ⋆ ∞ is an entrywise bound on w in Definition 4.
We define the (unknown) “noise floor”
[Equation]
where we defined · ( m ) in Section
A.
Our goal will be to return x such that ∥ x − x ⋆ ∥ 2 =
O ( R ξ ) .
We now formally state the main result of this section here.
Theorem 16 Let δ ∈ (0 , 1) , and suppose R 0 ≥ ∥ x ⋆ ∥ 2 for s -sparse x ⋆ ∈ R d .
Further, suppose
A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP and b =
A x ⋆ + ξ , and R 1 ≥ R ξ := (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 .
Then with probability at least 1 − δ , Algorithm 3 using Algorithm 3 as a noisy step oracle computes ˆ x satisfying
[Equation]
in time
O [Equation]
log [Equation] R
[Equation] (cid:19) log (cid:18) R R final (cid:19)(cid:19)(cid:19)
[Equation]
(cid:18) R R final (cid:19) log (cid:18) R
R final (cid:19)(cid:19) .
Similarly to Theorem 5, Theorem 16 provides a runtime guarantee which interpolates between the fully random and semi-random settings, and runs in sublinear time when e.g. the entire measurement matrix A satisfies RIP.
Theorem 16 further provides a refined error guarantee as a function of the noise vector ξ , which again interpolates based on the “quality” of the weights w .
This is captured through the parameter m = 1 w ⋆ ∞ : when m ≈ n , the squared error bound R 2 ξ scales as the average squared entry of ξ , and more generally it scales as the average of the largest m entries.
We solve the noisy variant by essentially following the same steps as Section C and making minor modifications to the analysis; we give an outline of the section here.
In Section D.1, we generalize the framework of Section C.1 to the setting where we only receive noisy observations (24), while our current radius is substantially above the noise floor.
We then implement an appropriate step oracle for this outer loop in Section D.2, and prove that the relevant Assumption 2 used in our step oracle implementation holds when A is wRIP in Section D.3.
D.1. Radius contraction above the noise floor using step oracles
In this section, we give the main loop of our overall noise-tolerant algorithm, HalfRadiusSparseNoisy , which takes as input s -sparse x in and a radius bound R ≥ ∥ x in − x ⋆ ∥ 2 .
It then returns an s -sparse vector x out with the guarantee ∥ x out − x ⋆ ∥ 2 ≤ 12 R , as long as R is larger than an appropriate multiple of R ξ .
We give the analog of Definition 6 in this setting, termed a “noisy step oracle.”
Definition 17 (Noisy step oracle)
We say that O nstep is a ( C prog , C 2 , C ξ , δ ) -noisy step oracle for (cid:101) ∆ ∈ R n and A ∈ R n × d if the following holds.
Whenever there is v ∈ R d with 112 ≤ ∥ v ∥ 2 ≤ 1 such that (cid:101) ∆ =
A v + ξ where (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) [Equation] mC ξ , with probability ≥ 1 − δ , O nstep returns w ∈ R n ≥ [-
-]0 such that the following two conditions hold.
First,
[Equation]
Second, there exists a ( C 2 , C prog 6 √ s ) short-flat decomposition of A ⊤ diag ( w ) ∆ :
[Equation]
We next characterize how a strengthened step oracle with appropriate parameters also is a noisy step oracle.
First, we will need a definition.
Definition 18 For distributions A, B on R n , we say A stochastically dominates B if there is a random variable C on R n whose coordinates are always nonnegative such that the distribution of
A is the same as the distribution of B + C (where C may depend on the realization of B ).
We now formalize the properties of the strengthened step oracle that we will construct.
Definition 19 (Strong step oracle)
We say that O step is a ( C prog , C 2 , C ξ , δ ) -strong step oracle for (cid:101) ∆ ∈ R n and A ∈ R n × d if it satisfies all the properties of a standard step oracle (Definition 6), as well as the following additional guarantees.
For the output weights w , we have
[Equation]
The distribution of w output by the oracle is stochastically dominated by the distribution
[Equation]

for some ρ ≥ 1 . 3.
Compared to Definition 6 (the step oracle definition), we have the stronger guarantees that A ⊤ diag ( w ) ∆ admits a ( C 2 , C prog 24 √ s ) short-flat decomposition in (2) , and obtains its guarantees using the bounds 112 ≤ ∥ v ∥ 2 ≤ 1 (instead of a lower bound of 14 ).
We next demonstrate that a strong step oracle is a noisy step oracle.
Lemma
Suppose O step is a ( C prog , C 2 , C ξ , δ ) -strong step oracle for (cid:101) ∆ ∈ R n and A ∈ R n × d .
Then, O step is also a ( 14 C prog , C 2 , C ξ , 2 δ ) -noisy step oracle for ( (cid:101) ∆ , A ) .
Proof In the definition of a noisy step oracle, we only need to check the condition that (cid:80) i ∈ [ n ] w i (cid:101) ∆
i ∆ i ≥ 14 C prog for an arbitrary ∆ = (cid:101) ∆ − ξ where (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 ≤ √ mC − 1 ξ , as all other conditions are immediate from Definition 19.
Note that
[Equation]
where we used a 2 − ab ≥ 12 a 2 − 12 b 2 .
The first sum above is at least 12 C prog by assumption.
To upper bound the second sum, we will use the second property in the definition of a strong step oracle.
Let
S ⊂ [ n ] be the set consisting of the m largest coordinates of ξ (with ties broken lexicographically).
Let α be drawn from the distribution
[Equation]
Note that with 1 − 0 .
1 δ probability, by a Chernoff bound, we have that (cid:80) i ∈
S
α i ≥ 15 δC prog C 2 ξ .
If this happens, then since S consists of the largest coordinates of ξ , any vector β such that β ≤ α entrywise and ∥ β ∥ 1 ≤ 14 δC prog C 2 ξ must have
[Equation]
Now note that for any S with | S | = m ,
[Equation]
Combining the above two inequalities and Markov’s inequality and the fact that the distribution of α stochastically dominates the distribution of w , we deduce that with at least 1 − δ probability, (cid:88)
[Equation]
Putting everything together, we conclude that we have
[Equation]
with failure probability at most 2 δ , completing the proof.
In Section D.2, we prove that if A satisfies Assumption 2 (a slightly different assumption than
Assumption 1) then with high probability we can implement a strong step oracle with appropriate parameters.
This is stated more formally in the following; recall m is defined in (25).
Assumption 2 The matrix A ∈ R n × d satisfies the following.
There is a weight vector w ⋆ ∈ ∆ n with ∥ w ⋆ ∥ ∞ ≤ w ⋆ ∞ = 1 m , a constants L , ρ ≥ 1 , and constants K, C ξ (which may depend on L ) such that for all v ∈ R d , ξ ∈ R n with
[Equation]
we have, defining (cid:101) ∆ =
A v + ξ : 1.
A is entrywise bounded by ± ρ , i.e.
[Equation]
[Equation]
There is a ( L, 1 K √ s ) short-flat decomposition of A ⊤ W ⋆ (cid:101) ∆ :
[Equation]
Lemma 21 Suppose A satisfies Assumption 2.
Algorithm 5 is a ( C prog , C 2 , C ξ , δ ) strong step oracle StrongStepOracle for ( (cid:101) ∆ , A ) with
[Equation]
running in time
[Equation]
Here, in contrast to the noiseless setting, we can only guarantee that the strong step oracle (and thus also the noisy step oracle) succeeds with constant probability.
In our full algorithm, we boost the success probability of the oracle by running a logarithmic number of independent trials and aggregating the outputs.
We also show that for an appropriate choice of constants in Definition 4,
Assumption 2 is also satisfied, stated in Lemma 22 and proven in Section D.3.
Lemma 22 Suppose A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP with a suitable choice of constants in the RIP parameters in Definition 4.
Then, A also satisfies Assumption 2.
Next, we give a guarantee regarding our geometric post-processing step, Algorithm 4.
Claim 1 Aggregate ( S , R ) runs in O ( k 2 d ) time and meets its output guarantees.
Proof Let T be the subset of indices i ∈
[ k ] such that ∥ y i −
z ∥ ≤ R 3 .
Whenever the algorithm tests y i for some i ∈ T , it will be returned and satisfies the desired properties.
Now consider any y i returned by the algorithm.
The ball of radius 2 R 3 around y i intersects the ball of radius R 3 around z , since otherwise it can only contain at most 0 .
49 k points.
Thus, ∥ y i − z ∥ 2 ≤ R .
The runtime is dominated by the time it takes to do k 2 distance comparisons of points in R d .
We remark that is possible that for k = Ω(log 1 δ ) as is the case in our applications, the runtime of Claim 1 can be improved to have a better dependence on k by subsampling the points and using low-rank projections for distance comparisons.
Lemma [-
-]23 Assume A satisfies Assumption 2.
Then, Algorithm 3 meets its output guarantees in time
[Equation]
Proof We claim that for each independent trial j ∈ [ N trials ] , except with probability 1 − T δ ′ , the output x jT satisfies ∥ x jT − x ⋆ ∥ 2 ≤ R 6 .
Once we prove this, by Chernoff at least 0 .
51 N trials of the trials satisfy ∥ x
jT − x ⋆ ∥ 2 ≤ R 6 except with probability at most δ , and then we are done by Claim 1.
It remains to prove the above claim.
Fix a trial j , and drop the superscript j for notational convenience.
In every iteration t , (cid:101) ∆ := 1 R ( A x t − b ) is given to O nstep .
Since b =
A x ⋆ + ξ , we have
[Equation]
1 Input noisy n 4 C
C
, A ∈ R n × d , b
A
R √ m
R ≥ C ξ R do do
; if ( w jt , γ jt ) do not meet all of (1) , (2) and the additional criteria in Definition 19 then x jt ; end x jt x − x
jt − ηRγ t
; end 16 end for ∥ v ∥ 2 ≤ 1 , ∥ v ∥ 1 ≤ 2 √ 2 s , and ∥ ξ ∥ 2 , ( m ) ≤ √ mC ξ , where the last inequality used the assumed bounds
[Equation]
Hence, by the assumptions on O nstep , it will not fail for such inputs unless ∥ v ∥ 2 ≥ 1 12 is violated, except with probability ≤ δ ′ .
If the check in Line 10 fails, then except with probability ≤ δ ′ , the conclusion ∥ x T − x ⋆ ∥ 2 ≤ R 6 follows analogously to Lemma 9, since v = 1 R ( x − x ⋆ ) .
The other case’s correctness follows identically to the proof of Lemma 9, except for one difference: to lower bound the progress term (8), we use the assumption (26) which shows
[Equation]
Hence, following the proof of Lemma 9 (and adjusting for constants), whenever the algorithm does not terminate we make at least a 50 T fraction of the progress towards x ⋆ , so in T iterations (assuming no step oracle failed) we will have ∥ x T − x ⋆ ∥ 2 ≤ R 6 .
Finally, the runtime follows from combining Lemma 21 (with constant failure probability) with a multiplicative overhead of T · N trials due to the number of calls to the step oracle, contributing one additional logarithmic factor.
We adjusted one of the log d terms to become a log dδ term to account for the runtime of Aggregate (see Claim 1).
D.2. Designing a strong step oracle
In this section, we design a strong step oracle O step ( (cid:101) ∆ , A ) under Assumption 2.
As in Section C.2, our oracle iteratively builds a weight vector ¯ w , and sets
[Equation]
We will use essentially the same potentials as in (11), defined in the following:
[Equation]
Algorithm 5 is essentially identical to Algorithm 2 except for changes in constants.
We further have the following which verifies the second property in the definition of strong step oracle.
Fact 2 The distribution of w returned by Algorithm 5 is stochastically dominated by the distribution
[Equation]
Proof Every time we inspect a row, we change the corresponding entry of w by at most ηw ∗∞ .
The result follows from the number of iterations in the algorithm and uniformity of sampling rows.
To analyze Algorithm 5, we provide appropriate analogs of Lemmas 11 and 7.
Because Algorithm 5 is very similar to Algorithm 2, we will largely omit the proof of the following statement, which follows essentially identically to the proof of Lemma 11 up to adjusting constants.
Lemma 24 Assume that the constant K in Assumption 2 is sufficiently large, and that (cid:101) ∆ =
A v + ξ where v, ξ satisfy the norm conditions in Assumption 2.
Then for any ¯ w ∈ R n ≥ 0 such that B ( ¯ w ) ≤
C 2 µ 2 log d , we have
[Equation]
Assumption 2 Output: ( w, γ ) such that γ = (cid:80) i ∈ [ n ] w i (cid:101) ∆ i a i , and if there is v ∈ R d with 112 ≤ ∥ v ∥ 2 ≤ 1 such that (cid:101) ∆ =
A v + ξ where (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 ≤ √ mC ξ , with probability ≥ 1 − δ , (1), (2) are satisfied with
C prog = 1 , C 2 =
O (1) .
Furthermore, the second condition in (2) is satisfied with the constant 24 rather than 6 , and there is C ξ
= O (1) such that (27) is also satisfied.
3 C ← 3200 , µ ← [Equation] ; 4 for 0 ≤ k ≤ N ′ do w 0 ← 0 n , N ← ⌈ 5 Lnη
; for 0 ≤ t ≤ N do if then Return Sample
Compute
Lemma O
Γ t ( d
= (cid:101)
Φ 2 ( w t + de i ) − Cs (cid:101)
Φ sqmax ( w t + de i ) w
; end 11 end 12 Return 0 n
Proof The analysis is essentially identical to that of Lemma 11; we discuss only the main difference.
To apply the Taylor expansion of the exponential, Lemma 11 required a bound that
[Equation]
for all i ∈ [ n ] and j ∈ [ d ] .
Note that in the setting of Lemma 11, we took z ( i )
j
[Equation]
ij
[Equation]
⋆
[Equation]
Here, we will take z ( i ) j
[Equation]
p ⋆j ) ; bounds on all of these terms follow identically to in
Lemma 11, except that (cid:101) ∆ i = ⟨ a i , v ⟩ + ξ i , so we need to show
[Equation]
This follows since η ≤ 1 log d and | ξ i | =
O ( √ m ) by assumption.
Hence, as w ⋆i ≤ 1 m by definition of m , this is equivalent to m =
Ω( sρ 2 ) , an explicit assumption we make.
We now give a full analysis of Algorithm 5, patterned off of Lemma 7.
Lemma 25 Suppose A satisfies Assumption 2.
Algorithm 5 is a ( C prog , C 2 , C ξ , δ ) strong step oracle
StrongStepOracle for ( (cid:101) ∆ , A ) with
[Equation]
running in time
[Equation]
Proof The analysis is essentially identical to that of Algorithm 2 in Lemma 7; we discuss differences here.
First, the stochastic domination condition follows from Fact 2 for sufficiently large C ξ , K .
For the remaining properties, since the algorithm runs N ′ ≥ log 2 2 δ times independently, it suffices to show each run meets Definition 19 with probability ≥ 12 under the events of Assumption 2, assuming there exists the desired decomposition (cid:101) ∆ =
A v + ξ in the sense of Assumption 2.
Union bounding with the failure probability in Fact 2 yields the overall failure probability.
Correctness.
As in Lemma 7, it is straightforward to see that (cid:101)
Φ 2 is 1 -Lipschitz, since the value of η is smaller than that used in Algorithm 2.
The termination condition in iteration T then again implies (cid:101) Φ 2 ( w T ) ≥ 1 , and (cid:101)
Φ sqmax ( w T ) ≤ 3 Cs .
For C = 3200 , this implies the short-flat decomposition with stronger parameters required by Definition 19, as well as the ∥ w T ∥ 1 bound.
Success probability.
As in Lemma 7, the expected growth in Φ t in any iteration where Pr[ (cid:101)
Φ 2 ( w t ) ≥ 1] ≤ 12 is ≥ η 4 Ln .
Hence, running for ≥ 5 Lnη iterations and using Φ t − Φ 0 ≤ 2 yields the claim.
Runtime.
This follows identically to the analysis in Lemma 7.
D.3. Equivalence between Assumption 2 and RIP
In this section, we prove Lemma 22, restated here for completeness.
The proof will build heavily on our previous developments in the noiseless case, as shown in Section C.3.
Lemma 26 Suppose A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP with a suitable choice of constants in the RIP parameters in Definition 4.
Then, A also satisfies Assumption 2.
Proof The analysis is largely similar to the analysis of Lemma 15; we will now discuss the differences here, which are introduced by the presence of the noise term ξ .
There are three components to discuss: the upper and lower bounds in (28), and the decomposition (29).
Regarding the bounds in (3), by changing constants appropriately in Definition 4, we can assume that A satisfies the second property in Assumption 1 with the parameters 4 L and L 4 .
In particular, for ∆ =
A v , we then have
[Equation]
Recall that (cid:101) ∆ = ∆ + ξ for some (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 ≤ √ mC ξ .
Hence,
[Equation]
, for an appropriately large C 2 ξ ≥ 4 L .
Here the first inequality used ( a + b ) 2 ≤ 2 a
+ 2 b 2 ,
and the second inequality used that the largest (cid:80) i ∈ [ n ] w ⋆i ξ {+
+} i can be subject to ∥ w ⋆ ∥ 1 = 1 and ∥ w ⋆ ∥ ∞ ≤ 1 m is attained by greedily choosing the m largest coordinates of ξ by their magnitude, and setting w ⋆i = 1 m for those coordinates.
This gives the upper bound in Assumption 2, and the lower bound follows similarly: for appropriately large C 2 ξ ≥ L 2 ,
[Equation]
Lastly, for the decomposition required by (29), we will use the decomposition of Lemma 8 for the component due to (cid:80) i ∈ [ n ] w ⋆i ∆ i a i ; in particular, assume by adjusting constants that this component has a ( L 2 , 1 2 K √ s ) short-flat decomposition.
It remains to show that
[Equation]
also admits a ( L 2 , 1 2 K √ s ) short-flat decomposition, at which point we may conclude by the triangle inequality.
Let u
= ( W ⋆ ) 12 ξ ; from earlier, we bounded
[Equation]
Hence, applying Lemma 14 using the RIP matrix ( W ⋆ ) 12 A with appropriate parameters yields the conclusion, for large enough C ξ .
In particular, the ℓ 2 -bounded part of the decomposition follows from Lemma 14, and the proof of the ℓ ∞ -bounded part is identical to the proof in Lemma 15.
D.4. Putting it all together
We now prove our main result on noisy recovery.
Theorem 16 Let δ ∈ (0 , 1) , and suppose R 0 ≥ ∥ x ⋆ ∥ 2 for s -sparse x ⋆ ∈ R d .
Further, suppose
A ∈ R n × d is ( ρ, w ⋆ ∞ )
-wRIP and b =
A x ⋆ + ξ , and R 1 ≥ R ξ := (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 .
Then with probability at least 1 − δ , Algorithm 3 using Algorithm 3 as a noisy step oracle computes ˆ x satisfying
[Equation]
in time
O (cid:18)(cid:18) ndw ⋆ ∞ s log 4 ( ndρ ) log(cid:18)
d δ · log (cid:18) R R final (cid:19) log (cid:18) R
R final (cid:19)(cid:19)(cid:19) · ρ 2 log (cid:18) R
R final (cid:19) log (cid:18) R
R final (cid:19)(cid:19) .
Proof Our algorithm will iteratively maintain a guess R guess on the value of 1 √ m (cid:13)(cid:13) ξ ( m ) (cid:13)(cid:13) 2 , initialized at R guess ← R 1 .
For each value of R guess ≥ R ξ , the hypothesis of Algorithm 3 is satisfied, and hence using a strategy similar to the proof of Theorem 5 (but terminating at accuracy R
= O
( R guess ) where the constant is large enough to satisfy the assumption R ≥ C ξ R guess ) results in an estimate at distance R with probability at least 1 − δ , with runtime
[Equation]
The runtime above follows from Lemma 23.
Our overall algorithm repeatedly halves R guess , and outputs the last point returned by a run of the algorithm where it can certify a distance bound to x ⋆ of R =
C ξ R guess .
We use R final to denote
C ξ R guess on the last run.
Clearly for any R guess ≥ R ξ this certification will succeed, so we at most lose a factor of 2 in the error guarantee as we will have R final ≤ 2 C ξ R ξ .
The final runtime follows from adjusting δ by a factor of O (log R 1 R final ) to account for the multiple runs of the algorithm.
Appendix E. Greedy and non-convex methods fail in the semi-random setting
In this section, we show how a few standard, commonly-used non-convex or greedy methods can fail (potentially quite drastically) in the semi-random adversary setting.
The two algorithms that we examine are Iterative Hard Thresholding and Orthogonal Matching Pursuit [15, 76].
We believe it is likely that similar counterexamples can be constructed for other, more complex algorithms such as
CoSaMP [66].
For simplicity in this section, we will only discuss the specific semi-random model introduced in Definition 1, where A is pRIP, i.e. it contains an unknown RIP matrix G as a subset of its rows.
E.1. Iterative hard thresholding
The iterative hard thresholding algorithm [15] involves initializing x 0 = 0 and taking
[Equation]
where H s zeroes out all but the s largest entries in magnitude (ties broken lexicographically).
We can break this algorithm in the semi-random setting by simply duplicating one row many times.
Hard semi-random adversary.
Let n =
Cm for some sufficiently large constant C .
The first m rows of A are drawn independently from N (0 , I ) .
Now draw v ∼ N (0 , I ) , except set the first entry of v to 1 .
We set the last ( C − 1) m rows of A all equal to v .
We will set the sparsity parameter s = 1 and let x ⋆ = (1 , 0 , . . . , 0) .
We let b =
A x ⋆ .
Proposition 27 With A , b generated as above, with high probability, iterative hard thresholding does not converge.
Proof With high probability, some coordinate of v is Ω( √ log d ) .
We then have that some entry of
A ⊤ b has magnitude at least Ω( n √ log d ) with high probability.
Thus, the next iterate x 1 must have exactly one nonzero entry that has magnitude at least Ω( √ log d ) and furthermore, this entry must correspond to some coordinate of v that has magnitude at least Ω( √ log d ) .
However, this means that the residuals in all of the rows that are copies of v are at least Ω(log d ) .
In the next step, by the same argument, we get that the residuals blow up even more and clearly this algorithm will never converge.
In fact, x t will never have the right support because its support will always be on one of the entries where v is large.
E.2. Orthogonal matching pursuit
The orthogonal matching pursuit algorithm
[76] involves initializing x 0 = 0 and keeping track of a set S (that corresponds to our guess of the support of x ⋆ ).
Each iteration, we choose a column c
j of
A that maximizes |⟨
c j ,r t ⟩| ∥ c j [-(cid:107) 
-]∥ 22 and then add j to S (where r t =
A x t − b is the residual).
We then add j to S and project the residual onto the orthogonal complement of all coordinates in S .
We show that we can again very easily break this algorithm in the semi-random setting.
Hard semi-random adversary.
Let n = 3 m .
First, we draw all rows of A independently from
N (0 , I ) .
Next, we modify some of the entries in the last 2 m rows.
Let s be the sparsity parameter.
Let x ⋆ =
( s − 12 , . . .
, s − 12 , 0 , . .
. , 0) be supported on the first s coordinates and set b =
A x ⋆ .
Now we modify the columns of A (aside from the first s so A x
⋆ is not affected).
We set the last 2 m entries of one of these columns c j to match those of b .
Proposition 28 With A , b generated as above, with high probability, orthogonal matching pursuit does not recover x ⋆ .
Proof With high probability (as long as s ≥ 10 ), the column c j is the one that maximizes |⟨ c j ,b ⟩| ∥ c
j ∥because its last 2 m entries exactly match those of b .
However, j is not in the support of x ⋆ so the algorithm has already failed.
We further make the following observation.
Remark
By modifying other columns of A as well, the semi-random adversary can actually make the algorithm pick all of the wrong columns in the support.
E.3. Convex methods
Now we briefly comment on how convex methods are robust, in the sense that they can still be used in the semi-random setting (but may have substantially slower rates than their fast counterparts).
In the noiseless observations case, this is clear because the additional rows of A are simply additional constraints that are added to the standard ℓ 1 minimization convex program.
In the noisy case, let the target error be θ =
[Equation] ξ [Equation] m )
[Equation] We then solve the modified problem
[Equation]
Note that the above is a convex program and thus can be solved in polynomial time by e.g. cutting plane methods [46].
Also, note that x ⋆ is indeed feasible for the second constraint.
Now for the solution (cid:98) x that we obtain, we must have ∥ (cid:98) x ∥ 1 ≤ ∥ x ⋆ ∥ 1 and
[Equation]
Let G be the set of m randomly generated rows of A under our semi-random adversarial model.
The previous two conditions imply
[Equation]
which now by restricted strong convexity of G (see [1]) implies that ∥ x ⋆ − (cid:98) x [-(cid:107) 
-]∥ 2 =
O ( θ √ m ) .
We can furthermore round (cid:98) x to s -sparse to obtain the sparse vector x ′ , and the above bound only worsens by a factor of 2 for x ′ (see Lemma 9 for this argument).
Appendix F. Deferred proofs
Lemma 30 Let δ > 0 and θ ≥ 0 .
For any vector γ ∈ R d , we can solve the optimization problem
[Equation]
to additive accuracy δ in time
[Equation]
Proof Let P ⊂ R d be the set of p such that p has the same sign as γ entrywise and | p j | ≤ | γ j | for all j ∈ [ d ] .
By symmetry of the sqmax and the ℓ 2 norm under negation, the optimal p lies in P .
Next we claim that the function smax µ ( γ − p ) is 2 ∥ γ ∥ 2 -Lipschitz in the ℓ 2 norm as a function of p , over P .
To see this, the gradient is directly computable as [Equation] p − γ ) ◦ x where x ∈ ∆ d with x i = [Equation]
∈
[Equation]
] where ◦ denotes entrywise multiplication.
Thus, the ℓ 2 norm of the derivative is bounded by 2 ∥ γ ∥over P .
In the remainder of the proof, we show how to find p ∈ P which has ℓ 2 error δ 2 ∥ γ ∥ 2 to the optimal, which implies by Lipschitzness that the function value is within additive δ of optimal.
Next, since 0 ∈ P , we may assume without loss of generality that
[Equation]
else we may just output 0 , which achieves optimality gap at most 2 ∥ γ ∥ 2 θ .
Now, by monotonicity of ln it suffices to approximately minimize
[Equation]
The sum above is always at least d .
First we check if ∥ γ ∥ 2 ≤ θ + √ δ .
If this is true then clearly we can set p so that all entries of γ − p have magnitude at most √ δ .
This gives a solution such that
[Equation]
and since the value of sqmax is always at least µ 2 log d , this solution is optimal up to additive error δ .
Thus, we can assume ∥ γ ∥ 2 ≥ θ + √ δ in the remainder of the proof.
We also assume all entries of γ are nonzero since if an entry of γ is 0 then the corresponding entry of p should also be 0 .
Finally by symmetry of the problem under negation we will assume all entries of γ are positive in the remainder of the proof, such that each entry of p is also positive.
By monotonicity of sqmax in each coordinate (as long as signs are preserved) and the assumption that ∥ γ ∥ 2 ≥ θ + √ δ , the optimal solution must have ∥ p [-(cid:107) 
-]∥ 2 = θ .
By using Lagrange multipliers, for some scalar ζ and all j ,
[Equation]
For the optimal ζ by taking ℓ 2 norms of the quantity above, we have
[Equation]
Hence taking logarithms of both sides and using both the bounds (31) and ∥ γ − p ∥ 2 ≥ √ δ at the optimum, which follows from the previous discussion, we obtain
[Equation]
We next show how to compute p to high accuracy given a guess on ζ .
Observe that if γ j > 0 , then the right-hand side of (32) is decreasing in p j and hence by the intermediate value theorem, there is a unique solution strictly between 0 and γ j for any ζ .
Also, note that the location of this solution increases with ζ .
Let p ( ζ ) be the solution obtained by exactly solving (32) for some given ζ .
We have shown for all ζ that 0 ≤ [ p ( ζ )]
j ≤ γ j entrywise and hence ∥ p ( ζ ) ∥ 2 ≤ ∥ γ ∥ 2 for all ζ .
For a fixed ζ , we claim we can estimate p ( ζ ) to ℓ 2 error β in time O ( d log ∥ γ ∥ 2 β ) .
To see this, fix some ζ , µ , and γ j , and consider solving (32) for the fixed point p j .
We can discretize [0 , γ j ] into intervals of length γ j β ∥ γ ∥ 2 and perform a binary search.
The right-hand side is decreasing in p j and the left-hand side is increasing so the binary search yields some interval of length γ j β ∥ γ ∥ 2 containing the fixed point p j via the intermediate value theorem.
The resulting ℓ 2 error along all coordinates is then β .
We also round this approximate p ( ζ ) entrywise down in the above search to form a vector ˜ p ( ζ, β ) such that ˜ p
( ζ, β ) ≤ p ( ζ ) entrywise and ∥ ˜ p ( ζ, β ) − p ( ζ ) ∥ 2 ≤ β .
We use this notation and it is well-defined as the search is deterministic.
In the remainder of the proof we choose the constants
[Equation]
We define ˜ p ( ζ ) :
=
˜ p ( ζ, β ) for short as β will be fixed.
Discretize the range [ − ∥ γ ∥ 22 µ 2 − log 2 ∥ γ [-(cid:107) 
-]∥ 22 δ , log ∥ γ ∥ 2 √ δ ] into a grid of uniform intervals of length α .
Consider the ζ such that ζ ≤ ζ ⋆ < ζ + α .
Because p ( ζ ⋆ ) is entrywise larger than p ( ζ ) and hence the logarithmic term on the right-hand side of (32) is smaller for p ( ζ ⋆ ) than p ( ζ ) , we have
[Equation]
Moreover the optimal p ( ζ ⋆ ) has ℓ 2 norm θ , so | ζ − ζ ⋆ | ≤ α and exp( α ) − 1 ≤ 2 α imply
[Equation]
Consider the algorithm which returns the ζ alg on the search grid which minimizes |∥ ˜ p ( ζ alg )
∥ 2 − θ | (we will discuss computational issues at the end of the proof).
As we have argued above, there is a choice which yields ∥ p ( ζ ) ∥ 2 ∈ [ θ − ∆ , θ + ∆] and hence
[Equation]
We next claim that
[Equation]
Suppose (34) is false and ζ alg > ζ ⋆ .
Then letting u := p ( ζ alg ) and v := p ( ζ ⋆ ) , note that u , v , and u − v are all entrywise nonnegative
and hence
[Equation]
Hence, we have by (cid:112) x
2 + y 2 ≥ x + y 2 3 x for 0 ≤ y ≤ x , (31), and θ ≤ ∥ γ ∥ 2 ,
[Equation]
So, by triangle inequality (cid:13)(cid:13)
˜ p ( ζ alg )
(cid:13)(cid:13) 2 > θ + ∆ + β
and hence we reach a contradiction with (33).
Similarly, suppose (34) is false and ζ alg < ζ ⋆ .
Then for the same definitions of u , v , and using the inequality (cid:112) x 2 − y 2 ≤ x − y 2 3 x for 0 ≤ y ≤ x , we conclude
[Equation]
So we reach a contradiction with (33) in this case as well.
In conclusion, (34) is true and we obtain by triangle inequality the desired
[Equation]
The complexity of the algorithm is bottlenecked by the cost of finding ˜ p ( ζ alg ) .
For each ζ on the grid the cost of evaluating ˜ p ( ζ ) induces a multiplicative d log( ∥ γ ∥ 22 δ ) overhead.
The cost of performing the binary search on the ζ grid is a multiplicative log( ∥ γ ∥ 22 µ √ δ ) overhead; note that a binary search suffices because (cid:13) (cid:13) ˜ p
( ζ alg )
(cid:13) (cid:13) 2 is monotonic by our consistent choice of rounding down, and hence