Global Linear and Local Superlinear Convergence of IRLS for Non-Smooth Robust Regression
Abstract
We advance both the theory and practice of robust ℓ p -quasinorm regression for p ∈ (0 , 1] by using novel variants of iteratively reweighted least-squares (IRLS) to solve the underlying non-smooth problem.
In the convex case, p = 1 , we prove that this IRLS variant converges globally at a linear rate under a mild, deterministic condition on the feature matrix called the stable range space property .
In the nonconvex case, p ∈ (0 , 1) , we prove that under a similar condition, IRLS converges locally to the global minimizer at a superlinear rate of order 2 − p ; the rate becomes quadratic as p → 0 .
We showcase the proposed methods in three applications: real phase retrieval, regression without correspondences, and robust face restoration.
The results show that (1) IRLS can handle a larger number of outliers than other methods, (2) it is faster than competing methods at the same level of accuracy, (3) it restores a sparsely corrupted face image with satisfactory visual quality.
Introduction
Given a feature matrix A ∈ R m × n with m ≫ n and a response vector y ∈ R m the problem
[Equation]
(1) of ℓ p -regression has a variety of different applications depending on the choice of ∥ v ∥ p = (cid:0) (cid:80) i | v i | p (cid:1) 1 /p .
While p = 2 corresponds to standard linear regression, the choice of p > 2 arises naturally in semi-supervised learning on graphs [1, 2, 3] and a lot of activity has been dedicated recently to the computational complexity analysis for the case p > 1 [4, 5, 6].
In this paper, we assume there is a coefficient vector x ∗ ∈ R n such that the residual r ∗ :
= Ax ∗ − y is k -sparse, in which case a choice of p ∈ (0 , 1] is of interest.
Indeed, for the convex and non-smooth case p = 1 , (1) is known as least absolute deviation , which dates back to the time of Boscovich around in the middle of the 18th century [7, 8].
Since then, it has been known intuitively that (1) ( p = 1 ) is robust to large but few measurement errors (quoting [9]), i.e., that (1) is robust to outliers.
The algorithmic and theoretical understanding of (1) for p = 1 has been of long-standing interest to statisticians, which has led to a vast literature spanning from the classic 1964 paper of Huber [10] to recent contributions [11, 12, 13, 14, 15, 16, 17, 18, 19, 20].
That being said, for p
= 1 , there is 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[Equation] [Equation]
[Equation]
[Equation]
p
[Equation]
2 Weight initialization w (0) ← [ w (0)1 ; . . .
; w (0) m ] ∈ R m ; // Instead, one can initialize some vector x (1) .
3 For t ← 0 , 1 , . . . :
[Equation]
ϵ
[Equation]
1) suitably based on x ( t +1) , A , and y
[Equation]
See Section 2.2 for details.
[Equation]
a close, but somewhat underexplored connection between (1) and the basis pursuit problem [21], and more specifically, the theoretical [22, 23, 24] and algorithmic aspects [25, 26, 27] of compressed sensing [28] (see also Section 2.1).
For p ∈ (0 , 1) , problem (1) is non-smooth and non-convex, and generally less well understood.
To the best of our knowledge, the only paper that considered (1) with p ∈ (0 , 1) is [29], where the authors presented a condition (based on restricted isometry constants ) that guarantees exact recovery of x ∗ from (1).
Other related works come either from the compressed sensing literature, where an ℓ p (and noisy) version of basis pursuit has been considered as a sparsity-promoting formulation [30, 31, 32, 26, 33, 34, 35, 36, 37], or from the literature of matrix recovery/completion, where the Schatten- p norm has come into play as a non-convex surrogate for rank minimization [38, 39, 40, 41, 42, 43].
A key message from these works is that ℓ p or Schatten- p minimization with p ∈ (0 , 1) offers better information-theoretic properties (e.g., requires fewer samples for exact recovery) than minimization with p = 1 .
Here, we study an iteratively reweighted least-squares method ( IRLS p ) to solve (1) with p ∈ (0 , 1] .
As listed in Algorithm 1, IRLS p alternates between solving a weighted least-squares problem (2) and updating the weights (3); see Section 2.2 for more elaboration on IRLS.
The simplicity of this idea (with its impressive performance) justifies its popularity in many machine learning [44, 45, 46, 47] and computer vision [48, 49, 50, 51, 52] applications.
Based on recent advances on IRLS [26, 27], we make several contributions for understanding (1) and IRLS p .
We state our contributions next.
The Stable Range Space Property (Section 3.1).
We put forward the use of the (stable) range space property (RSP) for studying the robust regression problem (1).
The stable RSP was proposed by [53] in a different context to analyze a compressed sensing algorithm for solving a weighted basis pursuit problem at each iteration.
In analogy to the nullspace property in compressed sensing [24, 28], we show that the RSP is a necessary and sufficient condition for guaranteeing that the ℓ p -regression problem in (1) admits x ∗ as its unique solution (Proposition 2).
Moreover, we show that if A ∈ R m × n has i.i.d.
N (0 , 1) entries and if m is large enough, then the (stable) RSP holds with high probability (Proposition 3).
This justifies its use as the core assumption in our analysis.
Global Linear Convergence (Section 3.2).
We prove in Theorem 1 that, under a stable RSP assumption and with ϵ ( t ) suitably updated, the IRLS p Algorithm 1 with p = 1 converges linearly to the ground-truth x ∗ from any initial weight w (0) (or equivalently from any initial point x (1) ).
Note that while (accelerated) first-order methods (e.g., (sub-)gradient descent and proximal algorithms) can also solve the convex and non-smooth problem (1) with p = 1 , they exhibit, in general, (global) sub-linear rates at best [54, 55].
To our knowledge, [17] is the only paper that claims global linear convergence of a different IRLS variant for (1) with p = 1 ; we compare our results with the ones of [17] in Section 3.2.
On the other hand, Theorem 1 is inspired by the IRLS method of [27] for basis pursuit; based on [27], we suitably modify their proof strategy, and thus obtain a faster linear rate.
Local Superlinear Convergence (Section 3.3).
We prove in Theorem 2 that, under a stable RSP assumption and with ϵ ( t ) suitably updated, the IRLS p Algorithm 1 with p ∈ (0 , 1) converges to x ∗ superlinearly, provided that x (1) falls into a certain neighborhood of x ∗ .
To the best of our knowledge, no similar result exists for ℓ p -regression (1).
While Theorem 2 is inspired by the IRLS algorithm of [26] for ℓ p -basis pursuit, their IRLS method does not work well for small p (Section 2.2), and unlike in our work, their radius of local convergence diminishes greatly as p → 0 or m → ∞ .
Applications (Section 4).
We illustrate the performance of IRLS p for real phase retrieval [56, 57], linear regression without correspondences [58, 59, 60, 61], and face restoration from sparsely corrupted measurements [62, 63, 64].
For real phase retrieval (Section 4.1), we show that IRLS 0 .needs only m = 2 n − 1 measurements to recover x ∗ up to sign, with an additional assumption.
Theoretically, even brute-force can fail to identify ± x ∗ for fewer than 2 n − 1 measurements [65].
Empirically, many methods, including Kaczmarz [66, 67, 57], PhaseLamp [68], truncated Wirtinger flow [69], and coordinate descent [70], fail with m = 2 n − 1 Gaussian measurements (Figure 2a).
For linear regression without correspondences (Section 4.2), we show in Figures 2b-2c
that, IRLS 0 .is uniformly faster (20-100x) and more accurate than PDLP [71] (merged into Google or-tools) and the commercial solver Gurobi [72], both of which solve (1) with p = 1 as a linear program, and also than subgradient descent implemented by Beck & Guttmann-Beck [73].
For face restoration (Section 4.3), we present both quantitative and qualitative results on the Extended Yale B dataset [74].
Background 2.1 Connection of Robust Regression and Compressed Sensing
The ℓ p -regression problem (1) has a natural correspondence to the sparse recovery problem
[Equation]
where D is a ( m − n ) × m matrix, z ∈ R m − n a vector and the objective ∥ r ∥ p penalizes coefficient vectors with too many non-zero coordinates.
For p = 1 , (4) is also called basis pursuit [75].
Problems (1) and (4) are known to be related in the following sense (as implicitly stated in [21, 29]):
Proposition 1.
Suppose that the range space of A is equal to the nullspace of D , and that z = − Dy .
If x 1 globally minimizes (1) , then r 1 :=
Ax 1 − y globally minimizes (4) .
On the other hand, if rglobally minimizes (4) , then there exists some x 2 with r 2 =
Ax 2 − y that globally minimizes (1) .
Proposition 1 sheds light on how we “transfer”, with new insights, from the analysis of [26, 27] for (4) to results for (1).
In what follows, we treat [26, 27] in the context of robust regression and highlight the contribution of our work relative to [26, 27] and other existing results, whenever possible.
2.2 Iteratively Reweighted Least-Squares: The Basics and New Insights
We first discuss two different updating rules for the smoothing parameter ϵ ( t ) of Algorithm 1: the fixed rule and the dynamic rule.
We then consider the weight updating strategy (3).
Along the way we use synthetic experiments to illustrate ideas; see Appendix ?? for the experimental setup.
In doing so, we intend to provide a review of the state-of-the-art on the variants of IRLS.
Fixed Smoothing Parameter.
Instead of updating ϵ ( t ) at each iteration, most works on IRLS use a fixed and small positive number, e.g., ϵ :
= ϵ ( t ) = 0 .
001 for each t [44, 76, 45, 47, 49, 51, 50, 77].
The intuition is to avoid division by the potentially very small residual | a ⊤ i x ( t ) − y
i | .
But this common practice of fixing ϵ comes with at least three issues.
First, IRLS with a fixed ϵ > 0 converges only to an “ ϵ -approximate” point, not exactly the global minimizer ([78, 79, 44, 76], Figure 1a).
Second, even if setting ϵ small (e.g., ϵ = 10 − 15 ) could lead to an accurate enough solution for p = 1 (Figure 1a), it can fail for p < 1 (Figure 1b).
Finally, it makes obtaining a global linear convergence rate guarantee difficult: We are not aware of any theory about a global linear rate of IRLS with fixed ϵ .
Dynamic Smoothing Parameter.
Researchers have used different insights to reach the consensus of dynamically updating ϵ ( t ) [26, 17, 80, 27, 81].
The first insight is that convergence to global minimizers ensues if ϵ ( t ) is suitably decreased to 0 [26].
With β ∈ (0 , 1) , one such decreasing rule is ϵ ( t +1) ← βϵ ( t ) if certain conditions are satisfied, or keep ϵ ( t +1) ← ϵ ( t ) otherwise [17, 81] .
Figure 1a depicts the performance of [17] with an arbitrary choice β = 0 .
5 and ϵ (0) = 1 .
Also shown in Figure 1a are the IRLS methods with update rules of [26] and [27], which we discuss next.
The basis pursuit paper [26] also proposed a dynamic updating rule for ϵ ( t ) .
In our context (1), it sets ϵ (0) ← ∞ , ϵ ( t +1) ← min (cid:8) ϵ ( t ) , [ r ( t +1) ] α +1 /m (cid:9) ,
[Equation]
[Equation]
[Equation]
[Figure]
where the hyper-parameter 1 α is a non-negative integer
[Figure]
and [ r ( t +1) ]
α +1 is the ( α + 1) -th largest element of the residual r ( t +1) in absolute values.
Clearly, (6) creates a non-increasing sequence of smoothing parameters ϵ ( t ) , and the decay rate of ϵ ( t ) is adaptive to the data (in this case to the residual r ( t +1) ).
While at first glance it is not clear how fast ϵ ( t ) decays, [26] showed that the decay rate of ϵ ( t ) in (6) is locally linear for p
= 1 , α = k under mild conditions, in accordance with the decay of the objective values.
The final dynamic update rule for the parameter ϵ ( t ) is from [27] 1 and improves upon (6) [26] via:
[Equation]
where σ ( t +1) ← min (cid:110)(cid:13)(cid:13) r ( t +1) − z
(cid:13)(cid:13) 1 : z ∈ R m is α -sparse (cid:111) (7)
Observe that (7) computes the ℓ 1 -norm σ ( t +1) of the best α -term approximation of the residual r ( t +1) , which is in general larger than [ r ( t +1) ] α +1 of (6), while both update rules (6) and (7) rely on the hyper-parameter α .
Ideally, if α = k , and if IRLS with (6) or (7) converges to x ∗ , then both [ r ( t +1) ] α +1 and σ ( t +1) will approach 0 for large enough t .
While update rules (6) and (7) perform similarly for ℓ 1 -regression (Figure 1a), (6) fails more easily for small p than (7) (Figure 1b).
Finally, we note that overestimating the sparsity level k by setting α > k deteriorates the performance of (7) only slightly (Figure 1c).
Summary.
In Section 2.2 we delivered two take-away messages: (1) IRLS with a fixed ϵ ( t ) finds some approximate solution, sometimes good enough, (2) dynamically updating ϵ ( t ) as per (5)-(7) leads to local [26] or global [17, 27] linear convergence guarantees.
In view of Figure 1, we will next consider the IRLS p Algorithm 1 with update rule (7) for ϵ ( t ) , and analyze its convergence rates.
Convergence Theory of IRLS for Robust Regression
In Section 3.1 we introduce the stable range space property and justify it as our core assumption.
Under this assumption, we prove the global linear convergence (Theorem 1) and local superlinear convergence (Theorem 2) of the IRLS p Algorithm 1 in Sections 3.2 and 3.3, respectively.
3.1 The Stable Range Space Property
Definition 1 (Stable Range Space Property) .
A matrix A ∈ R m × n is said to satisfy the range space property (RSP) of order k , or the k -RSP for short, if the following holds for any vector d in the range space of A and any set S ⊂ { 1 , . . .
, m } of cardinality at most k :
[Equation]
The stable RSP of A is defined in the same way as the RSP except that we now require
[Equation]
for some η ∈ (0 , 1) .
We write “ ( k, η ) -stable RSP” to emphasize the parameters k and η .
By definition, the stable RSP implies the RSP.
Note that the ( k, η ) -stable RSP was proposed in [53] to analyze a reweighted ℓ 1 -minimization algorithm for compressed sensing.
With the notation of
Proposition 1, we can see that checking (8) for all d in the range space of A is equivalent to checking it for all d in the nullspace of D , the latter being the well-known nullspace property (NSP) [24, 28].
In other words, the (stable) RSP of A is equivalent to the (stable) NSP of D .
In Sections 3.2 and 3.3, we will use the ( k, η ) -stable RSP as an assumption for analysis.
Arguably, this is a weak assumption to make as the (stable) RSP is very close to a sufficient and necessary condition for exact recovery of the coefficient vector via ℓ p -robust regression (1):
Proposition 2 (Exact Recovery ⇔ RSP) .
Let p ∈
For all x ∗ and y such that Ax ∗ − y is k -sparse x ∗ is the unique solution to (1) if and only if A ∈ R m × n satisfies the k -RSP.
It follows from [82, Theorem 5] and Proposition 1 that checking whether a given matrix A satisfies the stable RSP is co-NP-complete [83].
On the other hand, we show that random Gaussian matrices of size m × n with sufficiently large m satisfy the stable RSP with high probability, which further justifies the usage of the stable RSP as an assumption in our analysis of IRLS p :
Proposition 3 (Gaussian ⇒ RSP) .
Suppose m − n ≥ 2 k .
Let A ∈ R m × n be a matrix with i.i.d.
N (0 , 1) entries.
Let δ ∈ (0 , 1) and η ∈ (0 , 1] be fixed constants.
If it holds that
[Equation]
then A satisfies the ( k, η ) -stable RSP with probability at least 1 − δ .
The assumption m − n ≥ 2 k of Proposition 3 is necessary, as it is not hard to prove that, if m − n < 2 k and Ax ∗ − y is k -sparse, then the ℓ 0 -minimization problem min x ∈ R n ∥
Ax − y ∥ 0 has multiple solutions.
With this assumption, (10) roughly becomes m − n ≥ ck log( em/k ) for some constant c if m − n is large enough; thus, ignoring logarithmic and constant factors, condition (10) becomes m − n ≥ Θ( k ) , which is nearly optimal, as the condition m − n ≥ 2 k is necessary.
Proposition 2 follows directly from [28, Theorem 4.9], so we omit its proof.
The reader might also find Proposition 3 corresponds to [28, Corollary 9.34].
This corollary assumes D of (4) has i.i.d.
N (0 , 1) entries, from which is not immediate what the distribution of A is, thus it does not imply
Proposition 3 directly; this is why we provide a complete proof for Proposition 3 in Appendix ?? .
3.2 Global Linear Convergence of IRLS
With the background provided in Section 2, we are now ready to state the first main result:
Theorem 1 (Global Linear Convergence) .
Suppose A ∈ R m × n obeys the ( k, η ) -stable RSP with η ∈ (0 , 3 / 4) (cf.
Definition 1).
Let Ax ∗ − y be k -sparse.
If the smoothing parameter ϵ ( t ) is updated as per (7) with α = k during the execution of the IRLS 1 Algorithm 1, then it holds for any t ≥ 1 that
[Equation]
(11) meaning that the IRLS 1 Algorithm 1 converges linearly and globally in objective value.
Discussion.
The condition η ∈ (0 , 3 / 4) is better understood via Proposition 3, which asserts that this condition holds with high probability if A has i.i.d.
Gaussian entries and if m is large enough
(compared to n, k ).
Moreover, if the ( k, η ) -stable RSP holds, Proposition 2 implies that x ∗ is a global minimizer of (1) with p = 1 .
The strength of Theorem 1 is in that it guarantees linear convergence for any initialization, starting at the first iteration .
However, the price to pay is that the convergence rate seems conservative with constant 1 / 294 , and depends on the number m of samples: (11) predicts that IRLS 1 converges with accuracy δ in O ( m log(1 /δ ))
iterations and m can be large.
We argue that such dependency is not an artifact of our analysis, because similar dependencies have been established for IRLS variants for ℓ p -regression (1) with p > 2 [2, Theorem 3.1], for interior point methods in convex optimization with m inequality constraints ([84, Chapter 11], [55, Chapter 5]), and for SGD in smooth & strongly convex optimization with a sum of m function components [85,
Theorem 2.1]).
Moreover, an adversarial initialization for which the rate empirically depends on m was pointed out for IRLS for basis pursuit in [27].
That being said, we conjecture that the linear dependency on m in (11) can be improved to √ m log m for some different choice of ϵ ( t ) ; see [2] and [84, Sections 11.5 and 11.8.2] for why this conjecture makes sense.
Furthermore, Figure 1a empirically depicts that, with the least-squares initialization, i.e., with w (0) i = 1 for all i , IRLStypically converges to x ∗ in 30 iterations.
This hints to the possibility of alleviating the dependency on m by analyzing the least-squares initialization, a challenging task which we leave to future work.
Comparison to [17].
To our knowledge, [17] is the only paper that claimed the global linear convergence of an IRLS variant with ϵ ( t ) updated as per (5) and with p = 1 .
In particular, [17] sets ϵ ( t +1) ← βϵ ( t ) whenever ∥ x ( t +1) − x ( t ) ∥ 2 ≤ 2 βϵ ( t ) ; see Figure 1a for the convergence of IRLS under this rule.
The update rule ϵ ( t +1) ← βϵ ( t ) of [17] has the advantage of being faster to compute than (7).
In spite of this, and of their beautiful proof idea, their result has several disadvantages compared to Theorem 1.
First, their proof involves sophisticated probabilistic arguments, and is tailored towards a feature matrix that satisfies strong concentration properties.
Our analysis, on the other hand, uses the ( k, η ) -stable RSP, which is close to a necessary and sufficient condition for the success of ℓ 1 -regression (cf.
Proposition 2) and expected to hold for a much larger range of matrices than just (sub-)Gaussian matrices (see [86, 87] for related results in the context of basis pursuit).
Second, some statements in their proof are inaccurate (e.g., their Lemma 5 does not hold for k < n , the last paragraph in their proving Lemma 8 is not rigorous).
Third, their update rule incurs two hyper-parameters, ϵ (0) and β , while Theorem 1 is based on the decreasing rule (7) that is adaptive to data, and the only hyper-parameter α is easier to set (Figure 1c).
Connection to [27].
A related global linear convergence was established for the IRLS method for basis pursuit in [27, Theorem 3.2].
Our proof is inspired by [27], but also improves on its strategy.
For example, their Theorem 3.2 involves two parameters for the stable nullspace property , while we only have a single RSP parameter η , simplifying matters.
Also, we obtain a constant of 1 / 294 in
Theorem 1, which is better than the value of 1 / 768 for the respective constant in [27, Theorem 3.2].
It is important to note that the proofs of [17, 27] and ours heavily rely on certain rules for decreasing the smoothing parameter ϵ ( t ) ; all these proofs of global linear convergence for IRLS would break down if ϵ ( t ) were fixed.
While the main theorems of both [17] and [27] are limited to the p = 1 case, we present our Theorem 2 for p ∈ (0 , 1) next.
3.3 Local Superlinear Convergence of IRLS p
The ℓ p -regression problem (1) with p ∈ (0 , 1) is more challenging than the case p
= 1 due to the lack of convexity.
However, here we show that at least locally, IRLS p converges with a superlinear rate of order 2 − p .
Moreover, since it is valid to run IRLS p even with p = 0 , we carefully design the proof such that the following result holds not only for p ∈ (0 , 1] , but also for p = 0 , in which case IRLS p can be interpreted as an algorithm minimizing a sum-of-logarithm objective (see Appendix ?? ).
Theorem 2 (Local Superlinear Convergence) .
Run IRLS p with p ∈ [0 , 1] and update ϵ ( t ) by (7) with α = k .
Assume A satisfies the ( k, η ) -stable RSP (Definition 1).
Let c ∈
(0 , 1) be a sufficiently small constant such that 2 c 1 − p η ( η + 1) <
(1 − c ) 2 − p .
Let Ax ∗ − y be k -sparse with support S ∗ .
Define
[Equation]
If the initialization x (1) is in a neighborhood of x ∗ , in the sense that
[Equation]
then IRLS p achieves the following superlinear convergence rate of order 2 − p for every t ≥ 1 :
[Equation] [Equation]
In particular, for p ∈ [0 , 1) , the superlinear rate (14) implies
[Equation]
Example 1 ( Quadratic versus linear rates ) .
We illustrate the practical benefit of a superlinear (quadratic) convergence rate with the following simple calculation.
Suppose that the inequality in (13) is barely fulfilled such that in the case of p = 0 , x (1) satisfies µ · ∥ Ax (1) − Ax ∗ ∥ 1 ≤ 0 . 9999999 .
Then (15) implies that after 30 iterations, the residual error is far below numerical precision already since ∥ Ax (31) − Ax ∗ ∥ 1 ≤ 0 .
9999999 2 30 − 1 · ∥ Ax (1) −
Ax ∗ ∥ 1 ≈ 10 − 47 · ∥ Ax (1) −
Ax ∗ ∥ 1 .
On the other hand, for p = 1 , a linear convergence factor µ of µ ≤ 0 . 9999999 is only able to guarantee a decay of the order µ 30 ≤ 0 . 9999999
≈ 0 .
999997 after 30 iterations.
Discussion.
Even though IRLS p with p ∈ (0 , 1) is tailored to ℓ p -regression, we measure the progress of IRLS p in ℓ 1 -norm, e.g., we provide upper bounds on ∥
Ax ( t +1) − Ax ∗ ∥ 1 ; this is because doing so avoids the use of Hölder’s inequality, which allows us to give tighter results that improve on [26] (see the next paragraph).
The second point that deserves discussion is the local convergence neighborhood defined by the right-hand side of (13).
Note that, we typically assume that every outlier sample ( a i , y i ) (with i ∈ S ∗ ) would result in a relatively large residual at x ∗ , say | a ⊤ i x ∗ − y i | ≫ 0 .
Hence the minimization term of the right-hand side in (13) is well-behaved.
Next, one might ask how to obtain such an initialization x (1) that satisfies (13).
One might run IRLS 1 to produce such an initialization; note though that the upper bound of (13) can not be computed, so one could not detect when to switch from IRLS 1 to IRLS p ( p ∈ [0 , 1) ).
Or alternatively, one could run IRLS p with a least-squares initialization and count on empirical global (super)linear convergence of IRLS p ; the latter is what we did in the experiments and is what we recommend.
A final and important remark is that ℓ p -regression (1) with p ∈ (0 , 1) is in general NP-hard [28, Exercise 2.10], but this does not contradict the theoretical local superlinear convergence of Theorem 2 and the empirical global convergence in Figures 1b and 2b, and this does not mean that IRLS p solves an NP-hard problem in polynomial time.
The catch is that we operate under the assumption that x ∗ leads to a k -sparse residual and is a unique global minimizer of (1) (cf.
Proposition 2).
With this assumption and small enough k/m , ℓ p -regression is tractable and can be solved via IRLS p to high accuracy (Figure 1b).
Connection to [26].
[26, Theorem 7.9] proves the local superlinear convergence of IRLS with update rule (6) for ℓ p -basis pursuit (4), which motivates Theorem 2.
However, [26, Theorem 7.9] requires c of (13) to satisfy c
= O (1 /m 2 /p − 1 ) , which means that, as p → 0 or m → ∞ , the eligible value of c quickly becomes vanishingly small, and thus the radius of local convergence (13) diminishes greatly.
In contrast, in our condition 2 c 1 − p η ( η + 1) <
(1 − c ) 2 − p , c has no direct dependency on m and is well-behaved even if p → 0 .
The above difference is due partly to our improved proof strategy, and partly to the different update rules of the smoothing parameter ϵ ( t ) .
Figure 1b showed that rule (6) of [26] does not work well for small p , in contrast to rule (7) that we use; a possible explanation for this phenomenon is that rule (6) decreases ϵ ( t ) too fast, yielding a poor initialization for the next iteration of weighted least-squares (similarly to the interior point method, cf.
Finally, [26, Theorem 7.9] does not hold for p = 0 , for which Theorem 2 still holds and suggests a local quadratic rate.
Applications and Experiments
We now explore the performance of IRLS p in three different applications, real phase retrieval (Section 4.1), linear regression without correspondences (Section 4.2), and face restoration (Section 4.3).
Note that the first two applications are special examples of the recent algebraic-geometric framework called homomorphic sensing [88], [89, Section 4], [90, Section 1.2].
In Section 4.4, we examine the behavior of IRLS p for different values of p and under noise.
4.1 Real Phase Retrieval
In real phase retrieval [56, 57], we are given a measurement matrix A
=
[ a 1 , . .
. , a m ] ⊤ and y =
[ y 1 , . . .
, y m ] ⊤ , where y i := | a ⊤ i x ∗ | , and we need to find either x ∗ or − x ∗ .
This problem
[Figure]
is a relative of the complex phase retrieval problem [65], where all data y and A , as well as the ground-truth x ∗ , are complex-valued, and which has applications
X ray crystallography
Here we show that real phase retrieval can be solved via ℓ p -regression (1).
Consider the index sets
I + :=
{ i : a ⊤ i x ∗ > 0 } ,
I − :=
{ i : a ⊤ i x ∗ < 0 } .
We might assume a ⊤ i x ∗ ̸ = 0 for every i without loss of generality.
Then we know that either
Ax ∗ − y has its ℓ 0 norm equal to m − | I − | , or A ( − x ∗ ) − y has its ℓ 0 norm equal to m − | I + | .
With such sparsity patterns on these two residuals, we can minimize (1), which serves as a (non-convex) relaxation of ℓ 0 -minimization, to recover x ∗ or − x ∗ , whichever corresponds to a sparser residual.
Here, we essentially treat one of the two clusters defined by (16) as inliers, and the other as outliers.
This robust regression point of view on real phase retrieval seems to be known by experts [92, Section 1.2], but we have not found a paper that actually proposes to solve (1) for real phase retrieval.
Here we argue with Figure 2a that, solving (1) (by IRLS 0 . 1 ) for real phase retrieval can be beneficial.
Indeed, theoretically, IRLS 0 .
1 converges to ± x ∗ locally but superlinearly (as a corollary of
Theorem 2).
Empirically, Figure 2a shows that IRLS 0 .
1 succeeds in recovering ± x ∗ by using only m = 2 n − 1 samples, which is exactly the theoretical minimum for real phase retrieval [65, Proposition 2.5].
Figure 2a also shows that multiple other state-of-the-art methods 2 [57, 67, 68, 69, 70] fail to recover ± x ∗ in this extreme situation, even though some of them have nearly optimal sample complexity, which requires roughly O ( n ) samples up to a logarithmic factor to succeed ( caution : nearly optimal ̸ = optimal).
As suggested by Proposition 3, for | I + | fixed, minimizing (1) with a
Gaussian matrix A enjoys m =
O ( n ) sample complexity, up to also a logarithmic factor.
Despite these advantages, we also emphasize that solving (1) (via IRLS 0 . 1 ) for real phase retrieval has an inherent limit: The performance depends on the number
| I + | of positive signs and the number m of samples; we might call | I + | /m the inlier rate or outlier rate.
At n/m = 200 / 399 , we see that
IRLS 0 .
1 succeeds if | I + | ≤ 70 , and if | I + | ≥ 399 − 70 by symmetry, but it fails if | I + | and | I − | get closer (Figure 2a).
For IRLS 0 .
1 to successfully handle the case | I + | = | I − | , we need more samples, empirically, say m ≥ 5 n .
It is an interesting future direction to design an algorithm that can handle the case of minimum samples ( m = 2 n − 1 ) and balanced data ( | I + | = | I − | ).
4.2 Linear Regression without Correspondences
Compared to (real) phase retrieval, the problem of linear regression without correspondences is an also important, but less developed subject; see [58, 94, 59, 60, 61, 95, 96, 97] for recent advances.
In this problem we are given y ∈ R m and A ∈ R m × n , and we need to solve the equations
[Equation]
[Equation]
for some unknown permutation matrix Π ∈ R m × m and vector x ∈ R n , under the assumption that there exists a solution ( Π ∗ , x ∗ ) .
Solving (17) is in general NP-hard for n > 1 [94, 98].
However, if
.10% 30% 50% 70% k/m
R e l a ti v e E rr o r
Least-Squares ∗ Least-Squares
IRLS 0 .(a)
Quantitative results
Ground-Truth f i
Measurements ˜ f
i
Least-Squares ∗
Least-Squares
IRLS 0 .
[Figure]
(randomly selected images) we further assume that Π ∗ permutes at most k rows of A , then we know that the residual Ax ∗ − y is k -sparse.
This is an insight of [59], where it was proposed to estimate x ∗ by solving (1) with p = 1 .
We show that IRLS p with p ∈ (0 , 1) is more suitable than several baselines for solving this robust regression problem reduced from linear regression without correspondences.
As baselines, we use the PDLP [71] and Gurobi [72] solvers to solve (1) with p = 1 as a linear program, and we also use a subgradient descent method implemented in the FOM toolbox of Beck & Guttmann-Beck [73].
Figure 2b shows that IRLS 0 .
1 is the most accurate, and Figure 2c shows
that IRLS 0 . 1 is 30 times faster than
Gurobi and subgradient descent, and is more than 100 times faster than PDLP.
Interestingly, IRLSis not very competitive in terms accuracy (Figure 2b).
Moreover, IRLS 1 is slower than IRLS 0 .
1 , as IRLS 0 .
1 converges faster and thus terminates earlier than IRLS 1 ; we terminate IRLS p whenever the objective stops decreasing (up to a tolerance 10 − 15 ).
Finally, see also [45, 48, 49, 50, 51, 52] where their IRLS variants outperform a different set of baselines (for different problems).
4.3 Face Restoration from Sparsely Corrupted Measurements
Here we consider a simple face restoration experiment on the Extended Yale B dataset [74], downsampled as per [99].
This dataset contains the face images of 38 individuals under about n ≈ 60 different illuminations.
Let F =
[ f 1 , . .
. , f n +1 ] be the matrix of faces of the same individual, where each f i ∈ R m is a (vectorized) face image with m = 2 , 016 pixels.
We assume that F is approximately low-rank (which is true under the Lambertian reflectance assumption [100]), thus each f i can be approximately represented as a linear combination of other faces of the individual, i.e., f i ≈ F i x ∗ i for some x ∗ i ∈ R n , where F i ∈ R m × n is the same as F , except with the i -th column excluded.
In our experiments of Figure 3, we corrupt f
i by random salt & pepper noise [63] and obtain the noisy face ˜ f
i as our measurement; so F i x ∗
i − ˜ f
i is approximately sparse.
We consider three methods to estimate x ∗ i and thus to find an estimate ˆ f i of f i : (1) Least-Squares that solves (1) with A = F i , y = ˜
f
i , and p = 2 , (2) Least-Squares ∗ that solves (1) with A = F i , y = f i , and p = 2 , used as a golden baseline, and (3) IRLS 0 .
1 which solves (1) with A = F i and y =
˜ f
i .
Figure 3a shows that
IRLS 0 .
1 is more robust to salt & pepper noise, whose amount is k/m , than Least-Squares, and has lower error ∥
ˆ f i − f i ∥ 2 / ∥ f i ∥ 2 (recall though that Least-Squares is statistically optimal for Gaussian noise).
Figure 3b shows some images randomly chosen from the dataset; observe that Least-Squares tends to “average” the illumination in all images F i , while IRLS 0 .
1 delivers faithful restoration.
4.4 The Choice of p and Performance of IRLS p Under Noise
Here we outline two experiments that illustrate two different aspects of IRLS p empirically.
In the first experiment, visualized in Figure 4a, we compare the decay of the relative errors of the iterates of IRLS p for different choices of p in the case that the residual is exactly k -sparse.
We observe that IRLS 1 exhibits a linear error decay on the one hand and a superlinear decay for IRLS p with p = 0 .
5 and p = 0 .
1 that accelerates as p decreases towards 0 , confirming the rates predicted by
Theorem 1 and Theorem 2.
( p = 1 , 0 . 5 , 0 .
1 are rather arbitrary choices in our experiments.)
[Figure]
In another experiment, visualized in Figure 4b, we explore the robustness of IRLS p if the model assumption of k -sparse residuals is not satisfied, but only approximately.
In particular, the input vector y ∈ R m of IRLS p is now such that its restriction on S ∗ with | S ∗ |
=
k has i.i.d.
N (0 , 1) entries (corresponding to the sparse corruptions), but its other coordinates are distributed as independent
Gaussian variables with mean a ⊤ i x ∗ and variance 0 .
01 , where i ∈ ( S ∗ )
c (corresponding to dense noise); see also the code provided in Appendix ?? .
We observe that also for this case of only approximately k -sparse residuals, IRLS p performs well, with faster and more accurate convergence for p = 0 .
1 and p = 0 .
5 compared to p = 1 .
We note that, strictly speaking, Theorem 1 and Theorem 2 do not apply directly to this case of approximately k -sparse residuals, but it is not hard to generalize it to this case; see [27, Theorem A.1] for a similar result for IRLS applied to basis pursuit.
Discussion and Future Work
In this work, we established novel global linear and local superlinear rates of IRLS p for ℓ p -regression (1) under the assumption of the stable range space property.
Furthermore, we explored several applications for which IRLS p exhibits state-of-the-art results.
There are several directions that deserve further investigation.
Theoretically, the Gaussian assumption of Proposition 3 might be weakened; similar results would hold for sub-Gaussian distributions [86, 87].
Note also that, in experiments (Figures 1 and 2), IRLS p works well even when m − n → 2 k .
This leaves the question of whether the constant or logarithmic factors of condition (10) are too stringent for guaranteeing the stable RSP to hold; can this be improved?
Finally, it is tempting to carry out a global rate analysis of IRLS p with p ∈ (0 , 1) for ℓ p -regression; can one prove global (linear) rates under the stable RSP or other assumption?
Algorithmically, designing inexact solvers for the inner (weighted) least-squares problem (2) based on
Krylov-subspace methods [101, 102] is expected to further accelerate our current IRLS implementation (our current implementation is attached in Appendix ?? ).
Also, our empirical experiments suggest that IRLS p (in particular p ∈ (0 , 1) ) is worth being adopted and applied to many other outlier-robust estimation tasks beyond robust regression [45, 48, 49, 50, 51, 52, 103, 104, 105, 106, 107, 108].
Acknowledgments and Disclosure of Funding
This work was supported by grants NSF 1704458, NSF 1934979, NSF-IIS-1837991, ONR MURI 503405-78051, and the NSF-Simons Collaboration on the Mathematical Foundations of Deep
Learning (NSF grant 2031985).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] Furthermore, we made a lot of efforts in comparing the theoretical results of prior works and position our contribution in the literature.
(b) Did you describe the limitations of your work?
[Yes] See for example at Lines 268on real phase retrieval.
We also discussed several directions for improvements or future research at the end of the paper.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] In particular, we used an entire subsection (Section 3.1) to justify the major assumption, the stable range space property.
(b) Did you include complete proofs of all theoretical results?
[Yes] We omitted the proofs of Proposition 1 and 2 as they follow directly from existing results; see Line 83 and Line 155.
All other theoretical results are proved in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We attached a copy of our code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In the appendix, we use an entire section to describe our experimental setup.
We did not do that in the main paper in interest of space.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We did not report error bars for two reasons.
The main reason is that some of our figures are quite dense, and adding error bars on them makes it harder to read.
A slightly subjective reason is that we believe our current figures are informative enough, correctly deliver the message, and are fair for other methods.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] In the main paper we used the Extended Yale B face dataset, which is free for research purposes.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]