A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data ∗
Abstract
Nonnegative (linear) least square problems are a fundamental class of problems that is well-studied in statistical learning and for which solvers have been implemented in many of the standard programming languages used within the machine learning community.
The existing off-the-shelf solvers view the non-negativity constraint in these problems as an obstacle and, compared to unconstrained least squares, perform additional effort to address it.
However, in many of the typical applications, the data itself is nonnegative as well, and we show that the nonnegativity in this case makes the problem easier.
In particular, while the worst-case dimension-independent oracle complexity for unconstrained least squares problems necessarily scales with one of the data matrix constants (typically the spectral norm) and these problems are solved to additive error, we show that nonnegative least squares problems with nonnegative data are solvable to multiplicative error and with complexity independent of any matrix constants.
The algorithm we introduce is accelerated and based on a primal-dual perspective.
We further show how to provably obtain linear convergence using adaptive restart coupled with our method and demonstrate its effectiveness on large-scale data via numerical experiments.
Introduction
Nonnegative least squares (NNLS) problems, defined by min x ≥ 0 12 ∥
Ax − b ∥ 22 , where A ∈ R m × n and b ∈ R m , are fundamental problems and have been studied for decades in optimization and statistical learning [26, 7, 23], with various off-the-shelf solvers available in standard packages of Python (as optimize.nnls in the SciPy package), Julia (as nnls.jl ) and MATLAB (as lsqnonneg ).
Within machine learning, NNLS problems arise whenever having negative labels is not meaningful, for example, when representing prices, age, pixel intensities, chemical concentrations, or frequency counts.
NNLS is also widely used as a subroutine in nonnegative matrix factorization [10, 18, 25] to extract sparse features in applications like clustering, collaborative filtering, and community detection.
From a statistical perspective, NNLS problems can be shown to possess a regularization property that enforces sparsity similar to LASSO [46], while being comparatively simpler, without the need to tune a regularization parameter or perform cross-validation [44, 8, 17, 24, 49, 43].
From an algorithmic standpoint, the nonnegativity constraint in NNLS problems is typically viewed as an obstacle: most NNLS algorithms perform additional work to handle it, and the problem is considered harder than unconstrained least squares.
However, in many important applications of
NNLS, such as text mining [6], functional MRI [3, 20], EEG data analysis [33], pulse oximetry [22, 50], statistical procedures in observational astronomy [19], and those traditionally addressed using nonnegative matrix factorization [10], the data is also nonnegative .
We argue in this paper that when the data for NNLS is nonnegative, it is in fact possible to obtain stronger guarantees than for traditional least squares.
Our Contributions.
We study NNLS problems with (element-wise) nonnegative data matrix A , to which we refer as the NNLS+ problems, through the lens of the (equivalent) quadratic problems:
[Equation]
where c =
A ⊤ b may be assumed element-wise positive.
This assumption is without loss of generality since if c j < 0 for some j , then ∇ j ¯ f ( x ) ≥ 0 , implying that the j th coordinate of the optimal solution is zero 2 .
Hence, we could fix x j = 0 and optimize over only the remaining coordinates.
We further assume that the matrix A is non-degenerate: none of its rows or columns has all elements equal to zero.
This assumption is without loss of generality because (1) if such a row exists, we could remove it without affecting the objective, and (2) if the j th column had all elements equal to zero, the optimal value of (P) would be −∞ , obtained for x with x j → ∞ .
Having established our assumptions and setup, we now proceed to state our contributions, which are three-fold.
(1) A scale-invariant, ε -multiplicative algorithm.
We design an algorithm based on coordinate descent that, in total cost O ( nnz( A ) √ ε ) , constructs an ε - multiplicative approximate solution to (P).
Our algorithm capitalizes on structural properties of (P) that arise as a result of the nonnegativity of A .
Theorem 1.1 (Informal; see Theorem 3.5) .
Given a matrix A ∈ R m × n + and ε > 0 , define f ( x ) = 12 ∥
Ax ∥ 22 − c ⊤ x and x ⋆ ∈ argmin x ≥
f
( x ) .
Then, there exists an algorithm that in K
= O ( n log n + n √ ε ) iterations and O (cid:0)
nnz ( A ) (cid:0) log n + 1 √ ε (cid:1)(cid:1) arithmetic operations returns (cid:101) x K ∈ R n + such that
E [ ⟨∇ f ( (cid:101) x K ) , (cid:101) x
K − x ⋆ ⟩ ] ≤ ε | f ( x ⋆ ) | .
The application of our structural observations on (P) to Theorem 4 . 6 of [13] enables the recovery of our guarantee on the optimality gap; however, we provide a guarantee on the primal-dual gap, and this is stronger than the one on the optimality gap stated in Theorem 1.1.
What is significant about Theorem 1.1 is the invariance of the computational complexity to the scale of A —it does not depend on any matrix constants.
This cost stands in stark contrast to that of traditional least squares, where the dependence of (oracle) complexity on matrix constants (specifically, the spectral norm of
A in the Euclidean case) is unavoidable [35, 36], and multiplicative approximation is not possible in general .
In general, scale-invariance is a crucial feature in problems with data matrices, since a dependence on the width implies that the algorithm is technically not polynomial-time.
This feature has, in fact, been an object of extensive study in the long line of works on packing and covering linear programs [48, 1] and its variants such as a fair packing [14].
Conceptually, our algorithm is a new acceleration technique inspired by VRPDA 2 [45]. (2) Linear convergence with restart.
By incorporating adaptive restart in (P), we improve the guarantee of Theorem 1.1 to one with linear convergence (with log(1 /ε ) complexity).
Thus, we establish the first theoretical guarantee for NNLS+ that simultaneously satisfies the properties of being scale-invariant, accelerated, and linearly-convergent .
Theorem 1.2 (Informal; see Theorem 4.1) .
Consider the setup of Theorem 1.1.
Then, there is an algorithm that in expected O (nnz( A )(log n + √ nµ ) log( 1 ε )) arithmetic operations returns (cid:101) x K ∈ R n + with f ( (cid:101) x K ) − f ( x ⋆ ) ≤ ε | f
( x ⋆ ) | , where µ is the constant in a local error bound for (P) .
Proving this bound requires bounding the expected number of iterations between restarts in conjunction with careful technical work in identifying an appropriate local error bound for NNLS+. (3) Numerical experiments.
We consolidate our theoretical contributions with a demonstration of the empirical advantage of our restarted method over state-of-the-art solvers via numerical experiments on datasets from LibSVM with sizes up to 19996 × 1355191 .
Figure 1 shows that, when combined with the restart strategy, our algorithm significantly outperforms the compared algorithms.
Related work.
NNLS has seen a large body of work on the empirical front.
The first method that was widely adopted in practice (including in the lsqnonneg implementation of MATLAB) is due to the seminal work of [26] (originally published in 1974).
This method, based on active sets, solves NNLS via a sequence of (unconstrained) least squares problems and has been followed up by [7, 47, 34, 11] with improved empirical performance.
While these variants are generally effective on small to mid-scale problem instances, they are not suitable for extreme-scale problems ubiquitous in machine learning.
For example, in the experiments reported in [34], Fast NNLS [7] took 6.63 days to solve a problem of size 45000 × 45000 , while the TNT-NN algorithm [34] took 2. hours.
However, the latter requires computing the Cholesky decomposition of A ⊤ A at initialization, which can be prohibitively expensive both in computation and in memory.
Another prominent work on the empirical front is that of [23], which performs projected gradient descent with modified
Barzilai-Borwein steps [4] and step sizes a carefully designed sequence of diminishing scalars.
Another, separate, line of work concerns optimization algorithms with multiplicative error guarantees.
Of interest to us are standard first-order algorithms that run in time that is near-linear in the input size and are thus applicable to large-scale setting (for multiplicative-error algorithms applicable to broad classes of problems but that run in time that is superlinear in the input size, see [41, Chapter 7]).
Most of the existing literature in this domain concerns positive (packing and covering) linear programs (e.g., [29, 51, 1, 30]).
Results also exist for positive semidefinite programs [2], (nonlinear) fair packing and covering problems [32, 14], and fair packing problems under Schatten norms for matrices [21].
With the exception of [14] (discussed below), none of these results are directly applicable to (P).
To the best of our knowledge, theoretical guarantees explicitly for (P) have been scarce.
For instance, [23] mentioned in the preceding paragraph provides only asymptotic guarantees.
Orthogonally, the result on 1-fair covering by [14] solves the dual of NNLS+, which also gives a multiplicative guarantee for NNLS+, but with overall complexity ˜
O
( nnz( A ) ε ) , where ˜ O ( · ) hides poly-log factors.
Since our algorithm is based on the coordinate descent algorithm, we highlight some results of other coordinate descent algorithms when specialized to the closely related problem of unconstrained linear regression .
The pioneering work of [37] proposed a coordinate descent method called RCDM , which in our setting has an iteration cost O
[Equation]
(cid:1) , where ∥ A : j ∥ 2 is the Euclidean norm of the j th column of A .
This was improved by [27], in an algorithm termed
ACDM , by combining Nesterov’s estimation technique [39] and coordinate sampling, giving an iteration complexity of O (cid:0)
[Equation]
for solving (P).
The latest results in this line of work by [2, 42, 40] perform non-uniform sampling atop a framework of [37] and achieve iteration complexity of O (cid:0)
[Equation]
, with [12] dropping the dependence on max 1 ≤ j ≤ n ∥ A : j ∥ 2 .
Additionally, the work of [28] develops an accelerated randomized proximal coordinate gradient ( APCG ) method to minimize composite convex functions.
As remarked earlier, [12], coupled with insights on NNLS+ problems provided in this work, can recover our guarantee for the optimality gap from Theorem 3.5.
However, our work is the first to bring to the fore the properties of NNLS+ required to get such a guarantee, and our choice of primal-dual perspective allows for a stronger guarantee in terms of an upper bound on the primal-dual gap.
Further, our algorithm is a novel type of acceleration, with our primal-dual perspective transparently illustrating our use of the aforementioned properties.
We believe that these technical contributions, along with our techniques to obtain vastly improved theoretical guarantees with the restart strategy applied to this problem, are valuable to the broader optimization and machine learning communities.
Notation and preliminaries
Throughout the paper, we use bold lowercase letters to denote vectors and bold uppercase letters for matrices.
For vectors and matrices, the operator ′ ≥ ′ is applied element-wise, and R + is the non-negative part of the real line.
We use ⟨ a , b ⟩ to denote the inner product of vectors a and b and ∇ for gradient.
Given a matrix A , we use A : j for its j th column vector, and for a vector x , x j denotes its j th coordinate.
We use nnz( A ) for the number of non-zero entries of A .
We use x k for the vector in the k th iteration and, to disambiguate indexing, use [ x k ] j to mean the j th coordinate of x k .
The i th standard basis vector is denoted by e i .
For an n -dimensional vector x and A ∈ R m × n , we define Λ = diag([ ∥ A :1 ∥ 22 , . . .
, ∥ A : n ∥ 22 ]) and ∥ x
∥ 2 Λ = (cid:80) ni
=1 x 2 i ∥
A : i ∥ 22 .
Finally, [ n ] def = { 1 , 2 , . . . , n } .
A differentiable function f : R n → R is convex if for any x , (cid:98) x ∈ R n , we have f ( (cid:98) x ) ≥ f ( x ) + ⟨∇ f ( x ) , (cid:98) x − x ⟩ .
A differentiable function f : R n → R is said to be µ -strongly convex w.r.t.
the ℓ 2 -norm if for any x , (cid:98) x ∈ R n , we have that f ( (cid:98) x ) ≥ f ( x ) + ⟨∇ f ( x ) , (cid:98) x − x ⟩ + µ 2 ∥ x − (cid:98) x
∥ 22 .
We have analogous definitions for concave and strongly concave functions, which flip the inequalities noted.
Given a convex program min x ∈X f ( x ) , where f : R n → R is differentiable and convex and X ⊆ R n closed and convex, the first-order optimality condition of a solution x ⋆ ∈ argmin x ∈X f ( x ) is
[Equation]
[Equation]
Problem setup.
As discussed in the introduction, our goal is to solve (P), with A ∈ R m × n + .
For notational convenience, we work with the problem in the following scaled form: min
[Equation]
This assumption is w.l.o.g.
since any (P) can be brought to this form by a simple change of variable ˆ x j = c
j x j (see also e.g., [1, 14] for similar scaling ideas).
The scaling need not be explicit in the algorithm since the change of variable ˆ x j = c
j x j is easily reversible.
Properties of the objective.
To kick off our analysis, we highlight some properties inherent to the objective defined in (2).
These properies, which strongly need the non-negativity of A and x , are central to obtain a scale-invariant algorithm for (P).
Proposition 2.1.
Given f : R n + → R as defined in (2) and x ⋆ ∈ argmin x ∈ R n + f ( x ) , the following statements all hold.
a) ∇ f ( x ⋆ ) ≥ 0 .
b) f ( x ⋆ )
= − 12 ∥
Ax ⋆ ∥ 22 = − 12 1 ⊤ x ⋆ .
c) for all j ∈ [ n ] , we have x ⋆j ∈ (cid:104) 0 , 1 ∥
A : j ∥(cid:105) .
d) − 12 (cid:80) j ∈ [ n ] 1 ∥
A : j ∥ 22 ≤ f ( x ⋆ ) ≤ − 1 2min j ∈ [ n ] ∥ A : j ∥ 22 .
The validity of division by ∥ A : j ∥ 2 in the preceding proposition is by the non-degeneracy of A discussed in the introduction.
We prove this proposition in Section 6.1.
An important consequence of Proposition 2.1 (c) is that (P) can be restricted to the hyperrectangle
X = { x ∈ R n : 0 ≤ x j ≤ 1 ∥
A : j ∥ 22 } without affecting its optimal solution, but effectively reducing the search space.
Thus, going forward, we replace the constraint x ≥ 0 in (P) by x ∈ X .
Primal-dual gap perspective.
As alluded earlier, our algorithm is analyzed through a primal-dual perspective.
For this reason, it is useful to consider the Lagrangian
[Equation]
from which we can derive our rescaled problem (2) as the primal problem min x ∈X P ( x ) , where
[Equation]
Thus, the Lagrangian is constructed in a way that one can derive the primal problem from it while also localizing the matrix in the bilinear term and ensuring coordinate-wise separability of the terms involving either only the dual or only the primal variable since this greatly simplifies our analysis.
Similar to [45], we use Eq.
(3) to define the following relaxation of the primal-dual gap, for arbitrary but fixed u ∈ X , and v ∈ R m :
[Equation]
The significance of this relaxed gap function is that for a candidate solution (cid:101) x and an arbitrary (cid:101) y ∈ R m , a bound on Gap ( u , v ) L ( (cid:101) x , (cid:101) y ) translates to one on the primal error, as follows.
First select u = x ⋆ , v = A (cid:101) x .
Then, by observing that L ( (cid:101) x , A (cid:101) x ) =
f ( (cid:101) x ) and L ( x ⋆ , Ax ⋆ ) =
f ( x ⋆ ) , we have f ( (cid:101) x
[Equation]
For a fixed x , L ( x , · ) is 1-strongly concave and minimized at Ax .
Thus, L ( x ⋆ , A (cid:101) x ) ≤ L ( x ⋆ , Ax ⋆ ) − 12 ∥
A ( (cid:101) x − x ⋆ ) ∥ 2 .
Hence, we have the following primal bound:
[Equation]
In light of this connection, our algorithm for bounding the primal error is one that generates iterates that can be used to construct bounds on Gap ( u , v ) L (
[Equation]
x , (cid:101) y ) , as we detail next.
Our algorithm and convergence analysis
Our algorithm, Scale Invariant NNLS+ (SI-NNLS+), is an iterative algorithm using the estimate sequences ϕ k ( x ) and ψ k ( y ) for k ≥ 1 (see Section 3.1) giving the primal and dual updates x k = argmin x ∈X ϕ k ( x ) and y k
= argmax y ∈
[Equation]
We use our algorithm’s iterates from Eq.
(6) to construct G k , an upper estimate of Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) , where (cid:101) x k , (cid:101)
y k are convex combinations of the iterates and u ∈ X , v ∈ R m + , where u is fixed and v is arbitrary.
Our motivation for constructing G k ( u , v ) ≥ Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) is to obtain a bound on the primal error, using Inequality (5).
The main goal in the analysis is to show that G k ≤ QA k , where A k is a sequence of positive numbers and Q is bounded.
To obtain the claimed multiplicative approximation, we use Inequality (5) and argue that for u = x
⋆ , v = A (cid:101) x
k , we have Q ≤ O (cid:0) | f
( x ∗ ) | (cid:1) .
Our algorithm may be interpreted as a variant of the VRPDA 2 algorithm [45], with our analysis inspired by the approximate duality gap technique [13]; however, unlike VRPDA 2 , which uses estimate sequences, our analysis directly bounds the primal-dual gap.
Another difference from
VRPDA 2 is in our choice of primal regularizer (described shortly) and our lack of a dual regularizer.
A variant of SI-NNLS+ suitable for analysis is shown in Algorithm 1, with proofs in Section 6.2 and Section 6.3.
We give an equivalent implementation version (“ Lazy SI-NNLS+”, which updates as few coordinates as is possible to improve cost per iteration) in Algorithm 2 and its analysis in Section 7.
1: Input: Matrix A ∈ R m × n + with n ≥ 4 , accuracy ε , initial point x2:
Initialize: (cid:101) x 0 = x 0 , y
= y
= Ax 0 , K = 52 n log n + 6 n √ ε , a 1 = [Equation]
[-A 1 = a 1 , ϕ 0 ( x ) = 
 ∥ x − x 0 ∥ 2 Λ .-][Equation]
3: for k = 1 , 2 , . . .
, K do 4:
Sample j k uniformly at random from { 1 , 2 , . . . , n } 5: x k ← arg min x ∈X ϕ k ( x ) , for ϕ k ( x ) defined by Eq.
(12) and Eq.
(15) 6: y k ← arg max y ∈
R m ψ k ( y ) , for ψ k ( y ) defined by Eq.
[Equation]
[-(9) 7: (cid:101) x k = 
-][Equation] A [Equation] ←
[Equation] [Equation] na
[Equation]
[-+
-]√
[Equation] [Equation] } 10: end for 11: Return (cid:101) x K
3.1 Gap estimate construction
The gap estimate G k is constructed as the difference G k ( u , v ) =
U k
( v ) − L k ( u ) , where U k ( v ) ≥
L ( (cid:101) x k , v ) and L k ( u ) ≤ L ( u , (cid:101) y k ) are, respectively, upper and lower bounds we construct on the
Lagrangian.
It then follows by Eq.
(4) that G k ( u , v ) is a valid upper estimate of Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) .
We first introduce a technical component our constructions L k and U k crucially hinge on: we define two positive sequences of numbers { a i } i ≥ 1 and { a ki } 1 ≤ i ≤ k , with one of their properties being that both sum up to A k > 0 for k ≥ 1 .
Specifically, we define A 0 = 0 and { a i } i ≥ 1 as a i =
A i − A
i − 1 .
The sequence { a ki } changes with k and for k = 1 is defined by a 11 = a 1 , while for k ≥ 2 : a
[Equation]
Summing over i ∈ [ k ] verifies that A k = (cid:80) ki
=1 a ki .
For the sequence { a ki } 1 ≤ i ≤ k to be non-negative, we further require that a 1 − ( n − 1) a 2 ≥ 0 and ∀ i ≥ 2 , na i − ( n − 1) a i +1 ≥ 0 .
The significance of these two sequences lies in defining the algorithm’s primal-dual output pair by (cid:101) [Equation] and
[Equation]
The intricate interdependence of { a i } and { a ki } enables expressing (cid:101) x k in terms of only { a i } .
This expression further simplifies to a cheaper recursive one, which is used in Algorithm 1.
With the sequences { a i } i ≥ 1 and { a ki } 1 ≤ i ≤ k in tow, we are now ready to show the construction of an upper bound U k ( v ) on L ( (cid:101) x k , v ) and a lower bound L k ( u ) on L ( u , (cid:101) y k ) .
Upper bound.
To construct an upper bound, first observe that by Eq.
(3) and Eq.
[Equation]
[Equation]
Consider the primal estimate sequence defined for k = 0 as ψ 0 = 0 and for k ≥ 1 by
[Equation]
which ensures that L ( (cid:101) x k , v )
=
1 A k ψ k ( v ) .
A key upshot of constructing ψ k ( v ) as in Eq.
(9) is that the quadratic term implies A k -strong concavity of ψ k for k ≥ 1 , which in turn ensures that
the vector y k = arg max y ∈ R m ψ k ( y ) from Eq.
(6) is unique.
This property, coupled with the first-order optimality condition in Inequality (1), gives that for any y ∈ R m , ψ k ( y ) ≤ ψ k
[Equation] k 2 ∥ y [Equation]
y k ∥ 22 .
We are now ready to define the following upper bound by:
[Equation]
The preceding discussion immediately implies that U k is a valid upper bound for the Lagrangian.
Lemma 3.1.
For U k as defined in Eq.
(10), Lagrangian defined in Eq.
(3) and (cid:101) x k ∈ R n + in Eq.
(8), we have, for all y ∈ R m , the upper bound U k ( y ) ≥ L ( (cid:101) x k , y ) .
Lower bound.
Analogous to the preceding section, we now obtain a lower bound on the Lagrangian, completing the bound on the gap estimate.
However, the construction becomes more technical.
We start with the same approach as for the upper bound.
Since L ( u , (cid:101) y k ) is concave in (cid:101) y k , by Jensen’s inequality: L ( u , (cid:101) y k ) ≥
A k (cid:80) i ∈
[ k ] a i (cid:16) ⟨ Au , y i ⟩ − 1 ⊤ u − 12 ∥
y i ∥ 22 (cid:17) .
Were we to define the dual estimate sequence ϕ k in the same way as we did for the primal estimate sequence ψ k , we would now simply define it as A k times the right-hand side in the last inequality.
However, doing so would make ϕ k depend on y k , which is updated after x k , which in Eq.
(6) is defined as the minimizer of the ϕ k .
To avoid such a circular dependency, we add and subtract a linear term (cid:80) i ∈
[ k ] ⟨ A ⊤ y
i − 1 , u ⟩ , where y i − 1 , defined later, are extrapolation points that depend only on y 1 , . . .
y i − 1 .
We thus have
[Equation]
If we now defined ϕ k based on the first term in the above inequality, we run into another obstacle: the linearity of the resulting estimate sequence is insufficient for cancelling all the error terms in the analysis.
Hence, as is common, we introduce strong convexity by adding and subtracting an appropriate strongly convex function.
Our chosen strongly convex function is motivated by the box-constrained property of the optimum from Proposition 2.1 (c) and crucial in bounding the initial gap estimate.
It coincides with ϕ 0 : for any x ∈ R n + , define the function
[Equation]
[Equation]
This function is 1 -strongly convex with respect to ∥ · ∥ Λ and used in defining ϕ 1 as:
[Equation]
[Equation]
The definition of ϕ 1 is driven by the purpose of cancelling initial error terms.
Next, we choose ϕ k so that for any fixed u ∈ X , we have
[Equation]
where the expectation is with respect to all the randomness in the algorithm.
This construction is used to reduce the per-iteration complexity, for which we employ a randomized coordinate update on x k for k ≥ 2 .
To support such updates, we relax the lower bound to hold only in expectation .
Concretely, let j i be the coordinate sampled uniformly at random from [ n ] in the i th iteration of
SI-NNLS+, independent of history.
Fix y
i for i = 1 , . . .
, k − 1 and for k ≥ 2 and x ∈ X , define
[Equation]
[Equation]
For k ≥ 2 , ϕ k ( u ) can also be defined recursively via
[Equation]
[Equation]
The function ϕ
k inherits the strong convexity of ϕ 0 .
This property, together with Eq.
(6) and first-order optimality from Inequality (1), give
[Equation]
Along with strong convexity, our choice of ϕ k in Eq.
(15) leads to the following properties essential to our analysis: (1)
ϕ k is separable in its coordinates; (2) the primal variable x k is updated only at its j th k coordinate; (3) Eq.
(13) is true.
These are formally stated in Proposition 6.2.
With the dual estimate sequence ϕ k defined in Eq.
(15), we now define the sequence L k by
[Equation]
We conclude this section by justifying our choice of L k as a valid expected lower bound on E L ( x , (cid:101) y k ) .
Lemma 3.2.
For L k defined in Eq.
(17), for the Lagrangian in Eq.
(3) and (cid:101) y k in Eq.
(8), we have, for a fixed u ∈ X , the lower bound E L ( u , (cid:101)
y k ) ≥
E L k ( u ) , where the expectation is with respect to all the random choices of coordinates in Algorithm 1.
3.2 Bounding the gap estimate
With the gap estimate G k constructed as in the preceding section and combining Eq.
(10) and Eq.
(17), we now achieve our goal of bounding A k G k (to obtain a convergence rate of the order 1 /A k ) by bounding the change in A k G k and the initial scaled gap A 1 G 1 .
Lemma 3.3.
Consider the iterates { x k } and { y k } evolving according to Algorithm 1.
Let n ≥and assume that a 1 = 1 √ 2 n 1 . 5 and a 1 ≥ ( n − 1) a 2 , while for k ≥ 3 ,
[Equation]
Then, for fixed u ∈ X , any v ∈ R m , and all k ≥ 2 , the gap estimate G k
= U k − L k satisfies
[Equation]
Lemma 3.4.
Given a fixed u ∈ X , any v ∈ R m , ¯ y
=
y 0 , and x 1 and y 1 from Algorithm 1, we have
[Equation]
Combining the two lemmas, we now bound G K and deduce our final result on the primal error.
Theorem 3.5.
[Main Result] Assume that n ≥ 4 .
Given a matrix A ∈ R m × n + , ε > 0 , an arbitrary x 0 ∈ X and ¯ y
= y
= Ax 0 , let x k and A k evolve according to SI-NNLS+ (Algorithm 1) for k ≥ 1 .
For f defined in (2), define x ⋆ ∈ argmin x ≥
f
( x ) .
Then, for all K ≥ 2 , we have
[Equation]
When K ≥ 52 n log n , we have A K ≥ ( K − 52 n log n ) 2 36 n.
If ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | , then for K ≥ 52 n log n + 6 n √ ε , we have E [ f ( (cid:101) x K ) − f ( x ⋆ )] ≤ ε | f
( x ⋆ ) | .
The total cost is O (cid:0) nnz
( A ) (cid:0) log n + 1 √ ε
(cid:1)(cid:1) .
The assumption ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | above is satisfied by x 0 =
(c.f.
Proposition 2.1 and Eq.
We reiterate that the reason ∥
A ∥ does not show up in the final bounds (thereby rendering our algorithm “scale-invariant”) is because Proposition 2.1 allows bounding ∥ x 0 − x ⋆ ∥ 2 Λ by | f ( x ⋆ ) | , where we crucially used the non-negativity of A and x ; this does not seem possible for general A .
Remark 3.6 .
SI-NNLS+ (Algorithm 1) and Theorem 3.5 also generalize to a mini-batch version.
Increasing the batch size grows our bounds and number of data passes by a factor of at most squareroot of the batch size s , by relating the spectral norm of the s columns of A corresponding to a batch to the Euclidean norms of individual columns of A from the same batch.
However, due to efficient available implementations of vector operations, mini-batch variants of our algorithm with small batch sizes can have lower total runtimes on some datasets (see Section 5).
Adaptive restart
We now describe how SI-NNLS+ can be combined with adaptive restart to obtain linear convergence rate.
To apply the restart strategy, we need suitable upper and lower bounds on the measure of convergence rate.
Our measure of optimality is the natural residual r ( x ) =
∥ R ( x ) ∥ Λ [31] for
[Equation]
where Π R n + is the projection operator onto R n + and Λ is as defined in Section 2.
For Λ
= I , R ( x ) is the natural map as defined in, e.g., [15].
Due to space constraints, we only state the main result of this section in the following theorem, while full technical details are deferred to Section 6.4.
Theorem 4.1.
Given an error parameter ε > 0 and x 0 = 0 , consider the following algorithm A :
Initialize Initialize
Run Lazy output
[Figure]
Then, the expected number of arithmetic operations of A is O (cid:0) nnz
( A ) (cid:0) log n + √ nµ (cid:1) log (cid:0) r ( x 0 ) ε (cid:1)(cid:1) .
As a consequence, given ¯ ε > 0 , the total expected number of arithmetic operations until a point with f ( x ) − f ( x ⋆ ) ≤ ¯ ε | f ( x ⋆ ) | can be constructed by A is O (cid:16) nnz
( A ) (cid:16) log n + √ nµ (cid:17) log (cid:16) nµ ¯ ε (cid:17)(cid:17) .
Numerical experiments and discussion
We conclude our paper by presenting the numerical performance of SI-NNLS+ and its restart versions (see the efficient implementation version in Algorithm 2) against FISTA with restart [5, 38], a generalpurpose large-scale optimization algorithm, OA+DS with restart designed by [23] specifically for large-scale NNLS problems, and lazy implemented APCG [16] with restart.
We use the same restart strategy for all the algorithms, proposed in Section 4.
As an accelerated algorithm, FISTA has the optimal 1 /k 2 convergence rate; OA+DS, while often efficient in practice, has only an asymptotic convergence guarantee.
For FISTA, we compute the tightest Lipschitz constant (i.e., the spectral norm ∥ A ∥ ); for OA+DS, we follow the best practices laid out by [23].
For our SI-NNLS+ algorithm and its restart version with batch size bs = 1 , we follow Algorithm 2 and the restart strategy in
Section 4.[-
-]
For the restart version with batch size larger than 1 , we choose the best batch size in { 10 , 50 , 300 , 500 } and compute the block coordinate Lipschitz constants as the spectral norms of the corresponding block matrices.
All algorithms were implemented in Julia and run on a server with Intel(R) Xeon(R)
Silver 4110 32-Core Processors.
We evaluated the performance of the algorithms on the large-scale sparse datasets real-sim , news20 , and E2006train from the LibSVM library [9].
Both real-sim and news20 datasets have nonnegative data matrices, but the labels may be negative.
When there exist negative labels, it is possible for the elements of A ⊤ b to be negative.
In such cases, per the discussion from the introduction, we can simply remove the corresponding columns of A and solve an equivalent problem with smaller dimension.
On the other hand, the data matrix in E2006train dataset is not non-negative, which means that this dataset does not satisfy the assumption required for the analysis of Algorithm 1.
[Figure]
However, Algorithm 1 can still be run by keeping only the non-negativity constraints for primal updates.
This example is provided solely for illustration of empirical performance.
Results.
To compare all implemented algorithms, we plot the natural residual/objective value gap versus number of data passes/time in Figure 1 for all algorithms implemented with restart and in
Figure 2 for all algorithms implemented without restart.
As can be observed from the two figures, our proposed restart speeds up all the algorithms.
Figure 1(a)-(d) shows that SI-NNLS+ is better than FISTA and OA+DS in terms of number of data passes on the real-sim dataset and better than APCG in terms of time.
While the proposed restart strategy speeds up all the algorithms to linear convergence, variants of SI-NNLS+ remain competitive in all the settings.
In terms of the performance of different variants of SI-NNLS+, with bs = 1 , we have a much better coordinate
Lipschitz constant than for bs = 10 and thus the case of bs = 1 dominates bs
= 10 in terms of data passes.
As FISTA and OA+DS take less time accessing the full dataset once, they have lower runtimes than SI-NNLS+ but are beaten by SI-NNLS+ with restart and bs = 1 .
In Figure 1(e)-(h), on the news20 dataset, in terms of number of data passes, restarted APCG and SI-NNLS+ with bs =are dominant.
However, as news20 is a very sparse dataset, letting bs = 1 significantly increases the total time to access the full data once due to the overhead per iteration.
As a result, single-coordinate methods have the worst runtimes, while restarted OA+DS is the fastest but SI-NNLS+ with bs =remains competitive.
Figure 1(i)-(l) shows the performance comparison on the E2006train dataset.
On this dataset, both restarted FISTA and restarted OA+DS ran for 4 hours without visibly reducing the function value. SI-NNLS+ outperforms FISTA and OA+DS in both number of data passes and time, and outperforms APCG in terms of time.
Further experiments are left for future work.
Acknowledgements
CS was supported in part by the NSF grant 2023239.
JD was supported in part by the NSF grant 2007757, by the Office of Naval Research under contract number N00014-22-1-2348, and by the
Wisconsin Alumni Research Foundation.
Part of this work was done while JD was visiting Simons
Institute for the Theory of Computing.
Part of this work was done when SP was visiting JD at the
University of Wisconsin-Madison.
Appendix: Omitted Technical Details
In this section, we provide proofs for all the claims made in the main body of the paper, additional supporting propositions and lemmas, and also omitted details of the implementation of our algorithm.
The proofs are provided in the order in which the corresponding statement appears in the main body.
In Section 6.1, we prove the properties of our problem.
We again emphasize that these properties are crucial to achieving our goal of a scale-invariant algorithm for (P).
Section 6.2 contains the proofs of all our results pertaining to the convergence analysis of Algorithm 1, with proofs of the growth rate of the scalar sequences { a i } , { a ki } , and A k grouped separately in Section 6.3, owing to their more technical nature.
6.1 Omitted Proof from Section 2: Properties of Our Objective
Proposition 2.1.
Given f : R n + → R as defined in (2) and x ⋆ ∈ argmin x ∈ R n + f ( x ) , the following statements all hold.
a) ∇ f ( x ⋆ ) ≥ 0 .
b) f ( x ⋆ )
= − 12 ∥
Ax ⋆ ∥ 22 = − 12 1 ⊤ x ⋆ . c) for all j ∈ [ n ] , we have x ⋆j ∈ (cid:104) 0 , 1 ∥
A : j ∥(cid:105) .
d) − 12 (cid:80) j ∈ [ n ] 1 ∥
A : j ∥ 22 ≤ f ( x ⋆ ) ≤ − 1 2min j ∈ [ n ] ∥ A : j ∥ 22 .
Proof.
We recall the first-order optimality condition stated in Inequality (1): for all x ≥ 0 , we have ⟨∇ f ( x ⋆ ) , x − x ⋆ ⟩ ≥ 0 ; we repeatedly invoke this inequality in the proof below.
Suppose there exists a coordinate j at which Proposition 2.1 (a) does not hold and instead, we have ∇ j f ( x ⋆ ) < 0 .
Consider x ≥
such that x i =
x ⋆i for all i ̸
=
j and let x j = x ⋆j + ε for some ε > 0 .
Then, Inequality (1) becomes ∇ j f ( x ⋆ ) · ε ≥ 0 .
Under the assumption ∇ j f ( x ⋆ ) < 0 , this is an invalid inequality, thus contradicting our assumption.
From Proposition 2.1 (a), we know that ∇ f ( x ⋆ ) ≥ 0 .
If ∇ i f ( x ⋆ ) > 0 , and if x ⋆i > 0 , then by picking a vector x such that x j = x ⋆j for j ̸ = i and x i = x ⋆i − γ for any γ ∈ (0 , x ⋆i ) , we violate Inequality (1).
Therefore it must be the case that if ∇ i f ( x ⋆ ) > 0 , then x ⋆i = 0 .
Thus we have
[Equation]
Therefore, f ( x ⋆ ) =
[Equation]
From the proof of Proposition 2.1 (b), we have ⟨ x ⋆ , ∇ f ( x ⋆ ) ⟩ = 0 .
We also have x ⋆ ≥and, from Proposition 2.1 (a), that ∇ f ( x ⋆ ) ≥ 0 .
Therefore, if x ⋆i > 0 for some coordinate i then it must be that ∇ i f ( x ⋆ ) = 0 .
That is, 1 = ⟨ A : i , Ax ⋆ ⟩ .
Combining this equality with the fact that A and x ⋆ are both coordinate-wise non-negative gives
[Equation]
which implies x ⋆i ≤
[Equation] for all coordinates i .
The lower bound follows immediately by combining Proposition 2.1 (b) and Proposition 2.1 (c).
For the upper bound, we need to find a feasible point (cid:98) x and compute the function value at (cid:98) x , since f ( x ⋆ ) =
min y ≥ 0 f ( y ) ≤ f ( (cid:98) x ) .
Let (cid:98) x
= γ e k for some γ > 0 .
Then,
[Equation]
Let γ = 1 ∥
A : k ∥ 22 .
Then, f ( (cid:98) x ) = − 1 2 ∥
A : k ∥ 22 .
We pick k = arg min i ∈ [ n ] ∥ A : i ∥ 2 , therefore f ( x ⋆ ) ≤ − 1 2min i ∈ [ n ] ∥ A : i ∥ 22 as claimed.
Section
on Upper and Lower Estimates
We first show the results stating U k and L k are indeed valid upper and lower (respectively) estimates of the Lagrangian.
Lemma 3.1.
For U k as defined in Eq.
(10), Lagrangian defined in Eq.
(3) and (cid:101) x k ∈ R n + in Eq.
(8), we have, for all y ∈ R m , the upper bound U k ( y ) ≥ L ( (cid:101) x k , y ) .
Proof.
By evaluating the Lagrangian described by Eq.
(3) at x = (cid:101) x k , and by definition of ψ k from
Eq.
(9), we obtain the following upper bound on the Lagrangian at ( (cid:101) x k , y ) .
[Equation]
where the final steps are by strong convexity of ψ k and by Eq.
We emphasize that y here can be random since this is a deterministic statement.
Lemma 3.2.
For L k defined in Eq.
(17), for the Lagrangian in Eq.
(3) and (cid:101) y k in Eq.
(8), we have, for a fixed u ∈ X , the lower bound E L ( u , (cid:101)
y k ) ≥
E L k ( u ) , where the expectation is with respect to all the random choices of coordinates in Algorithm 1.
Proof.
First, evaluating Eq.
(3) at (cid:101) y k gives
[Equation]
Taking the expectation on both sides
[Equation]
applying the definition of (cid:101) y k , convexity of 12 ∥ · ∥ 2 , and Jensen’s inequality [Equation] and adding and subtracting 1 A k E (cid:80)
i ∈
[ k ] a i ⟨ Ax , y i − 1 ⟩ + 1 A k ϕ 0 ( u ) gives
[Equation]
We continue the analysis as
[Equation]
the first step comes from Eq.
(13), the second step comes from Eq.
(16), and the final step comes from Eq.
6.2.2 Proofs from Section [-3.
-]3.
We now describe three technical propositions that bound terms that show up in the proof of our result on bounding the scaled gap estimate.
Proposition 6.1.
For ψ k defined in Eq.
(9), with { a ki } defined in Eq.
(7), we have for all k ≥ 1 ,
[Equation]
Proof.
Evaluating ψ k and ψ k − 1 as defined in Eq.
(9) at y k and subtracting, we have
[Equation]
Next, applying strong convexity of ψ k − 1 at y k and y k − 1 while using the fact that y k − 1 minimizes ψ k − 1 gives
[Equation]
To complete the proof, it remains to add Eq.
(20) and Inequality (21).
Proposition 6.2.
The random function ϕ k : X → R , k ≥ 2 , defined in Eq.
(15) satisfies the following properties, with x k , y k , and y k evolving as per Algorithm 1.
a) It is separable in its coordinates: ϕ k ( x )
= (cid:80) j ∈ [ n ] ϕ k,j ( x j ) , where, for each j ∈ [ n ] , we define ϕ 0 ,j ( x j ) =
∥
A
[Equation] j [Equation] ( x j − [ x 0 ] j ) 2 , ϕ 1 ,j ( x j ) =
a 1 x j ( A ⊤ y 0 − 1 ) j + ϕ 0 ,j ( x j ) , and for k ≥ 2 ,
[Equation]
b)
The primal variable x k is updated only on the j th k coordinate in each iteration:
x k = x
k − 1 + γ e j k for some γ , and [ x k ] j = [ x k − 1 ] j for j ̸ = j k . c)
For a fixed x ∈ X and for k ≥ 1 , we have, over all the randomness in the algorithm,
[Equation]
Proof.
In the statement of Proposition 6.2 (a) [Equation] the claim about separability of ϕ 0 and ϕ 1 can be checked just from the definitions of ϕ 0 ,j and ϕ 1 ,j .
We prove the claim of coordinate-wise separability for k ≥ 2 by summing over j ∈ [ n ] both sides of Eq.
We can compute this sum via following observation, which concludes the proof of Proposition 6.2 (a).
[Equation]
From Proposition 6.2 (a) [Equation] we may therefore define, for j ̸ = j k ,
[Equation]
Therefore, [ x k ] j = [ x k − 1 ] j for all j ̸ = j k , thus proving Proposition 6.2 (b).
To prove Proposition 6.2 (c), we use induction on k .
The base case holds for k = 1 by the definition of ϕ 1 ( x ) .
Let
Proposition 6.2 (c) hold for k ≥ 1 .
Then, by the definition of ϕ k as in Eq.
(15), we have ϕ k
( x ) =
ϕ k − 1 ( x ) + na k ⟨ A ⊤ y k − 1 − 1 , x j k e j k ⟩ , for all k ≥ 2 .
Let F k − 1 be the natural filtration, containing all the randomness in the algorithm up to and including iteration k − 1 .
Taking expectations with respect to all the randomness until iteration k and invoking linearity of expectation, the inductive hypothesis, and the tower rule E [ · ] =
E [ E [ · |F k − 1 ]] , we have
[Equation]
which finishes the proof of Proposition 6.2 (c).
Proposition 6.3.
For all k ≥ 2 , the random function ϕ
k : X → R , k ≥ 2 , defined in Eq.
(15) satisfies the following inequality, where x k and y k evolve according to Algorithm 1.
[Equation]
Proof.
We have, using x = x k in Eq.
(15), that
[Equation]
Applying Eq.
(16) to ϕ k − 1 at x k gives
[Equation]
Adding these inequalities completes the proof of the claim.
We now use the preceding technical results to bound the change in scaled gap.
Lemma 3.3.
Consider the iterates { x k } and { y k } evolving according to Algorithm 1.
Let n ≥and assume that a 1 = 1 √ 2 n 1 . 5 and a 1 ≥ ( n − 1) a 2 , while for k ≥ 3 ,
[Equation]
Then, for fixed u ∈ X , any v ∈ R m , and all k ≥ 2 , the gap estimate G k
= U k − L k satisfies
[Equation]
Proof.
Using G k
= U k − L k , U k from Eq.
(10), and L k from Eq.
(17), we have
[Equation]
Therefore, the difference in scaled gap between successive iterations is
[Equation]
Based on the above expression
[Equation]
to prove the lemma
[Equation]
it suffices to bound the expectation of
T
[Equation]
First
[Equation]
we take expectations on both sides of the inequality in Proposition 6.3 by invoking E [ · ] =
E [ E [ · |F k − 1 ]] as before, where F k − 1 denotes the natural filtration.
By using the fact that x k − 1 is updated only at coordinate j k (as stated in Proposition 6.2 (b))
[Equation]
we observe the following for the term from the right hand side of Proposition 6.3.
[Equation]
Therefore
[Equation]
we have from Proposition 6.3 and scaling Eq.
(26) by − na k that
[Equation]
We now bound the expectation of the term involving differences of ψ k by taking expectations of both sides of Proposition 6.1.
[Equation]
By taking expectations on both sides of Eq.
(25), we have
[Equation]
Combining Eq.
(27), Eq.
[Equation] and Eq.
(29) then gives
[Equation]
Recall that by the assumption in the statement of the lemma,
[Equation]
Plugging into Eq.
(30) and rearranging, we have
[Equation]
To complete the proof [Equation] we need to bound the terms from the first two lines on the right-hand side of
Eq.
First, observe that, by the coordinate update of x k and Young’s inequality, ∀ β > 0 ,
[Equation]
By the same token, ∀ γ > 0 ,
[Equation]
[Equation]
Recalling that, by the choice of step sizes, ( n − 1) a k ≤ na k − 1 and a k ≤ √ A k
[Equation]
2 n , we can verify that for β = 2( n − 1) a k and γ = 2 na k − 1 , the following inequalities hold:
[Equation]
Combining Equations (31)–(34),
[Equation]
It remains to combine Eq.
(24), Eq.
(25), and Eq.
Lemma 3.4.
Given a fixed u ∈ X , any v ∈ R m , ¯ y
=
y 0 , and x 1 and y 1 from Algorithm 1, we have
[Equation]
Proof.
Evaluating Eq.
(10) and Eq.
(17) at k = 1 gives
[Equation]
[Equation]
By definition of ψ 1 from Eq.
(9), ϕ 1 from Eq.
(12), and the assignment a 11 = a 1 , we have
[Equation]
where we have used that ¯ y
= y 0 , which holds by assumption.
To complete the proof, it remains to subtract Eq.
(37) from Eq.
(36) and combine with the last equality.
Theorem 3.5.
[Main Result] Assume that n ≥ 4 .
Given a matrix A ∈ R m × n + , ε > 0 , an arbitrary x 0 ∈ X and ¯ y
= y
= Ax 0 , let x k and A k evolve according to SI-NNLS+ (Algorithm 1) for k ≥ 1 .
For f defined in (2), define x ⋆ ∈ argmin x ≥
f
( x ) .
Then, for all K ≥ 2 , we have
[Equation]
When K ≥ 52 n log n , we have A K ≥ ( K − 52 n log n ) 2 36 n.
If ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | , then for K ≥ 52 n log n + 6 n √ ε , we have E [ f ( (cid:101) x K ) − f ( x ⋆ )] ≤ ε | f
( x ⋆ ) | .
The total cost is O (cid:0) nnz
( A ) (cid:0) log n + 1 √ ε
(cid:1)(cid:1) .
Proof.
Observe that, by the choice of step sizes, n 2 a 2 k − 1 ≤ A k − 2 4 , ∀ k ≥ 3 .
Thus, telescoping the bound in Lemma 3.3 and combining with Lemma 3.4, we have E
[Equation]
[Equation]
We first show how to cancel out the inner product term with the negative quadratic terms.
Observe that, ∀ β > 0 ,
[Equation]
where the last line is by Young’s inequality.
In particular, choosing β = 2 a K , we have 12 a K nβ = na 2 K , which is at most A K − 1 4 , by the choice of step sizes in SI-NNLS+.
Thus, since ϕ 0 ( x 1 ) = 12 ∥ x 1 − x 0 ∥ 2 Λ , Eq.
(38) simplifies to
[Equation]
Since − 14 E [ ∥ u − x K ∥ 2 Λ ] ≤ 0 , we can ignore it.
Let us now bound n 2 a 12 ∥
y 1 − y
∥ 22 − ϕ
( x 1 ) .
By definition of x 1 and ϕ 1 , we have x 1 = x 0 − a 1 Λ − 1 ( A ⊤ y 0 − 1 ) .
Further, from Eq.
(9) and Eq.
(6), as we have y [-
-]1 =
Ax 1 and y
= Ax 0 , we can simplify the terms to bound as follows.
[Equation]
where the reasoning behind the first inequality follows from the definition of spectral norm and that ∥ AΛ − 1 / 2 ∥ 22 = λ max ( Λ − 1 / 2 A ⊤ AΛ − 1 / 2 ) ; the last inequality follows as the matrix Λ − 1 / [-
-]2 A ⊤ AΛ − 1 / 2 has all ones on the main diagonal, and thus its trace is at most n, and since it is positive semidefinite, its spectral norm is at most its trace.
As a 1 = 1 √ 2 n 1 . 5 , we conclude that n [Equation] − y
∥ 22 − ϕ
( x 1 ) ≤ 0 .
Thus, Eq.
(39) simplifies to
[Equation]
By construction, Gap u , v L ( (cid:101) x K , (cid:101) y K ) ≤ G K ( u , v ) .
Further, by Inequality (5), f ( (cid:101) x K ) − f ( x ⋆ ) + 12 ∥
A ( (cid:101) x
K − x ⋆ ) ∥ 2 ≤ Gap u , v L ( (cid:101) x K , (cid:101) y K ) .
Hence, we can conclude from Eq.
(40) that
[Equation]
On the other hand, for u = x ⋆ and v = y ⋆ =
Ax ⋆ , Gap u , v L ( (cid:101) x K , (cid:101) y K ) ≥ 0 , and, recalling from
Eq.
(6), Eq.
(8), and Eq.
(9) that
y K
=
A (cid:101) x K , we can also conclude from Eq.
(40) that
[Equation]
By Proposition 2.1 (b), f ( x ⋆ )
= − 12 1 ⊤ x ⋆ [Equation] − 12 ∥
Ax ⋆ ∥ 22 .
Using this identity, one can verify that, ∀ x ,
[Equation]
Hence, summing Eq.
(41) and Eq.
(42), we also have
[Equation]
Finally, the bound on the rate of growth of A k is provided in Appendix 6.3.
The reason A does not show up in the final bounds (thereby rendering our algorithm “scale-invariant”) is because Proposition 2.1 allows bounding ∥ x 0 − x ⋆ ∥ 2 Λ by | f ( x ⋆ ) | where we crucially use the nonnegativity of A and x .
This does not seem possible for general A .
However, an additive (as opposed to multiplicative) error bound can be obtained even with the more general A with only small updates to the analysis.
This bound would necessarily depend on the scale of A .
The choice of the regularizer 12 ∥ · − x 0 ∥ 2 Λ is also crucial here.
6.3 Omitted Proofs from Section 3: Growth of Scalar Sequences
In this section, we use the properties of { a i } and { a ki } to obtain our claimed rate of growth of A k .
Note that in any iteration k ≥ 2 of Algorithm 1, there are two possible updates to a k , which we name as follows.
[Equation]
Obtaining a handle on the growth rate of A k requires controlling the number of updates of both types specified above.
At a high level, the idea behind obtaining such a bound is that if the algorithm had only Type II updates, we would have A k ≥ Ω( k 2 n 2 ) ; we then go on to show that we cannot have more than 52 n log n Type I updates since those make a k grow too fast.
We formalize this intuition in the following lemmas.
Lemma 6.4.
In Algorithm 1, we have, for k ≥ 2 , that a k +1 ≥ a k and A k
+1 >
A k .
Proof.
Notice that for all k , we have a k > 0 , which implies that A k +1 def =
A k + a k +1 satisfies
A k +1 > A k .
To check the non-decreasing nature of a k , we recall that a k +1 = min (cid:16) na k n − 1 , √
A k 2 n (cid:17) .
In the case that √ A k 2 n ≥ na k n − 1 , we have a k +1 =
na k n − 1 > a k , as claimed.
Consider the other case with a k +1 = √
A k 2 n , and suppose, for the sake of contradiction, that a k +1 < a k .
Chaining this inequality with the assumed expression for a k +1 , scaling appropriately, and squaring both sides gives
A k < 4 n 2 a 2 k .
Plugging this into A k =
A k − 1 + a k and solving for a k from this quadratic inequality (and further invoking the nonnegativity of a k ), yields a k > 1+
[Equation]
However, this contradicts a k = min
[Equation]
Lemma 6.5. Consider the iterations { s k } in which Algorithm 1 performs a Type II update a s k +1 = √
A sk 2 n .
Then we have A s k ≥ k 2 c 1 n
and a s k ≥ k − 1 2 √ c 1 n 2 for c 1 = 36 .
Proof.
We prove this claim by induction.
First, notice that s k ≥ k + 1 for any k .
Recall our initialization a 1 =
A 1 = 1 √ 2 n 1 .
By combining this with the monotonicity property stated in
Lemma 6.4, we have a s 1 ≥ a 2 = 1 √ 2 n 2 .
5 ≥ 0 .
By using Lemma 6.4 again, we have, in a similar fashion, that A s 1 ≥ A 2 = 1 √ 2 n 2 . 5 + 1 √ 2 n 1 .
5 ≥ 1 c 1 n 2 , which proves the base case for induction.
Assume that for some k > 1 , we have the induction hypothesis A s k ≥ k 2 c 1 n
and a s k ≥ k − 1 2 √ c 1 n 2 .
Then, combining the monotonicity of A k from Lemma 6.4 with the fact that the algorithm performs a Type
II update on a s k , we have a s k {++
+} =
[Equation] ≥ √ A sk
[Equation] n ≥ k 2 √ c 1 n 2 .
By again applying monotonicity of A k and the induction hypothesis about a k , we have A s k +
=
A s k
[Equation] k [-+
-][Equation]
2 k 2 + √ c 1 k 2 c 1 n> ( k +1) 2 c 1 n 2 .
Lemma 6.6.
If at some k th0 iteration of Algorithm 1, we have that
[Equation]
then for all k ≥ k 0 , we have that
[Equation]
for c 1 = 36 .
Proof.
We prove the claim by induction.
First, the base case is true for k = k 0 by our assumption on a k 0 and A k 0 .
Assume the induction hypothesis a k ≥ k − 1 − k 0 + n 2 √ c 1 nand A k ≥ ( k − k 0 + n ) 2 c 1 nfor k ≥ k 0 .
We now discuss how a k changes with the two types of updates.
If the algorithm performs a Type I update on a k , then, by definition, a k +1 = na k n − 1 .
Now applying the assumed lower bound on a k , we have, when k > k 0 , that
[Equation]
Similarly, given that A k ≥ ( k − k 0 + n ) 2 c 1 n, we have,
[Equation]
If, on the other hand, the algorithm performs a Type II update on a k , then we have
[Equation]
This completes the induction.
As we saw in Lemma 6.6, after the k th0 iteration - starting at which Inequality (45) holds - A k grows fast.
We therefore need to estimate the number of Type I updates before the k th0 iteration.
Lemma 6.7.
There are at most 32 n log n Type I updates (Equation (43) ) performed before the k th iteration (the first iteration at which Inequality (45) holds).
Proof.
Suppose there are n 1 Type I updates performed by Algorithm 1 before the k th0 iteration, when
Inequality (45) starts to hold.
Further, by Lemma 6.4, a k is monotonically increasing (for both types of updates).
Then, when considering Type I updates (Equation (43)), we have a k 0 ≥ (cid:16) nn − 1 (cid:17) n 1 a 2 = (cid:16) nn − 1 (cid:17) n 1 · 1 √ 2 n 2 . In order for a k 0 > n − 1 12 n 2 , we only need to have n 1 > log nn − 1 (cid:16) √ n ( n − 1)
[Equation]
√(cid:17) .
In a similar fashion, combining the monotonicity of A k from Lemma 6.4 with the Type I update rule, we have
[Equation]
So, in order to have A k 0 > 136 per Inequality (45), we only need to have n 1 ≥ log nn − 1 ( n 2 .
5 18 √ 2 ) .
By using the approximation 1 + x ≤ e x and combining the above two bounds, we get as soon as n 1 ≥ 32 n log n , the inequality (45) holds.
Proposition 6.8.
[Rate of change of A k ] When k ≥ 52 n log n , we have A k ≥ ( k − 52 n log n ) 2 36 n.
Proof.
Let there be t 1 Type I updates and t 2 Type II updates before the first iteration at which
Inequality (45) holds, and let us call this iteration k 0 .
By the result of Lemma 6.7, we have t 1 ≤ 32 n log n .
By the result of Lemma 6.5, we must have A k 0 ≥ t 22 c 1 n 2 and a k 0 ≥ t 2 − [Equation] √ c 1 n 2 .
To meet the requirement in Inequality (45) then, we can see that t 2 ≤ n .
Therefore, k 0 = t 1 + t 2 ≤ 32 n log n + n ≤ 52 n log n .
Having reached the k th0 iteration, the result of Lemma 6.6 applies, and we have A k ≥ ( k − k 0 ) 2 c 1 n 2 .
6.4 Omitted Proofs from Section 4: Restart Strategy
To establish local error bounds, we start with the observation that (P) is equivalent to a linear complementarity problem.
Proposition 6.9.
Problem (P) is equivalent to the following linear complementarity problem, denoted by LCP( M , q ) .
[Equation]
where Λ − 1 M =
A ⊤ A and q = − Λ − 1 1 .
Proof.
Observe first that, as Λ − 1 is a diagonal matrix with positive elements on the diagonal, the stated linear complementarity problem is equivalent to
[Equation]
By Proposition 2.1, these conditions hold for any solution of (P).
In the opposite direction, suppose that the conditions from Eq.
(48) hold for some x .
Then applying these conditions for any u ≥gives
[Equation]
But ⟨∇ f ( x ) , u − x ⟩ ≥ 0 is the first-order optimality condition for (P), and so x solves (P).
For r ( x ) =
∥ R ( x ) ∥ Λ , a quantity termed natural residual [31], local error bound is obtained as a corollary of the following theorem.
Theorem 6.10 ([31], Theorem 2.1) .
Let M ∈ R n × n be such that LCP( M , 0 ) has 0 as its unique solution.
Then there exists µ > 0 such that for each x ∈ R n , we have r ( x ) ≥ µ ∥ x − x ⋆ ∥ , where x ⋆ is a solution to LCP( M , q ) that is closest to x under the norm ∥ · ∥ .
Theorem 6.10 applies to our problem due to the nonnegativity (and nondegeneracy) of A and choosing ∥ · ∥ = ∥ · ∥ Λ .
By arguing that Theorem 3.5 provides an upper bound on r ( (cid:101) x K ) , in expectation, we then obtain our final result below.
Proposition 6.11.
For any x ∈ R n + , r ( x ) ≤ (cid:112) 2 n ( f ( x ) − f ( x ⋆ )) , where x ⋆ ∈ argmin u ∈ R n + f ( u ) .
Proof.
Given x ∈
R n + , consider ˆ x defined as ˆ x j ⋆ = x
j ⋆ − R j ⋆ ( x ) , where j ⋆ = argmax 1 ≤ j ≤ n | R j ( x ) | · ∥ A : j ∥ 2 , and ˆ x j = x j for j ̸ = j ⋆ .
Then observing that
[Equation]
and combining with f ( ˆ x ) ≥ f ( x ⋆ ) , r ( x ) =
∥ R ( x ) ∥ Λ , the claimed bound follows after a simple rearrangement.
Theorem 4.1.
Given an error parameter ε > 0 and x 0 = 0 , consider the following algorithm A :
A : SI-NNLS+ with Restarts
Initialize: k = 1 .
Initialize Lazy SI-NNLS+ at x k − 1 .
Run Lazy SI-NNLS+ until the output (cid:101) x
kK satisfies r ( (cid:101) x kK ) ≤ 12 r ( x k − 1 ) .
Restart Lazy SI-NNLS+ initializing at x k = (cid:101) x kK .
Increment k.
Repeat until r ( (cid:101) x k K ) ≤ ε.
Then, the expected number of arithmetic operations of A is O (cid:0) nnz
( A ) (cid:0) log n + √ nµ (cid:1) log (cid:0) r ( x 0 ) ε (cid:1)(cid:1) .
As a consequence, given ¯ ε > 0 , the total expected number of arithmetic operations until a point with f ( x ) − f ( x ⋆ ) ≤
Proof.
Because each restart halves the natural residual r ( x ) , it is immediate that the total number of restarts until r ( (cid:101) x kK ) ≤ ε is bounded by log( r ( x 0 ) ε ) .
Thus, to prove the first (and main) part of the theorem, we only need to bound the number of iterations (and the overall number of arithmetic operations) of (Lazy) SI-NNLS+ in expectation.
Hence, in the following, we only consider one run of SI-NNLS+ until the natural residual is halved.
To keep the notation simple, we let x 0 denote the initial point of SI-NNLS+ and (cid:101) x
k denote the output of SI-NNLS+ at iteration k.
If r ( x 0 ) = 0 , A halts immediately and the bound on the number of iterations holds trivially, so assume r ( x 0 ) > 0 .
Using Theorem 3.5, we have that ∀ k ≥ 2 ,
[Equation]
As r 2 ( · ) is nonnegative, we can use Markov’s inequality to bound the total number of iterations K until r ( (cid:101) x K ) ≤ r ( x 0 ) 2 .
In particular, using Eq.
(49), we get by Markov’s inequality
that Pr[ K > k ] ≤
Pr[ r 2 ( (cid:101) x k ) ≥ r 2 ( x 0 )] ≤ 4 n µ 2 A k .
As K is nonnegative, we can estimate its expectation using
[Equation]
[Equation]
where in the last inequality we use the rate of A k from Proposition 6.8.
In the lazy implementation of SI-NNLS+, as argued in Appendix 7, the expected cost of an iteration is nnz( A ) n , which leads to the claimed bound on the number of arithmetic operations until r ( x ) ≤ ε.
By using that r ( x 0 ) ≤ (cid:112) [Equation] f
[Equation]
x
[Equation]
1)+ n +1 µ (cid:1) r 2 ( (cid:101) x K 1 ) (argued below), the bound on the number of iterations until f (cid:0)(cid:101) x K 1 − R ( (cid:101) x K 1 ) (cid:1) − f ( x ⋆ ) ≤ ¯ ε | f ( x ⋆ ) | have [Equation] ≤
[Equation]
and by setting K 1 = 12 log 2 2 n (cid:0) ( n − [Equation] + n +1 µ [Equation]
¯
[Equation] , we have this bound. Finally, it remains to argue that f (cid:0) (cid:101) x K 1 − R ( (cid:101) x K 1 ) (cid:1) − f ( x ⋆[-) ≤ (cid:0) ( n − 1) + n +1 µ (cid:1) r 
-]
[Equation] ≤ (cid:0) ( n − 1) + n +1 µ (cid:1) r 2 ( (cid:101) x K 1 ) .
Observe that the definition of R ( x ) is equivalent to x − ¯ x , where
[Equation]
By the first-order optimality of ¯ x based on the equivalent definition of R ( x ) above, we have ⟨∇ f ( x )+ Λ (¯ x − x ) , x ⋆ − ¯ x ⟩ ≥ 0 .
Rearranging, and using the definition of convexity of f, we have f (¯ x ) − f ( x ⋆ ) ≤ [Equation] f (¯ x ) , ¯ x − x ⋆ ⟩
[Equation]
where in the last inequality we have used the error bound from Theorem 6.10.
Implementation Version of SI-NNLS+
Since Algorithm 1 explicitly updates (cid:101) x k and (cid:101)
y k
(of lengths n and m respectively), the per iteration cost is O ( m + n ) , which is unnecessarily high when the matrix A is sparse.
In this section, we show that by using a lazy update strategy, we can efficiently implement Algorithm 1 with overall complexity independent of the ambient dimension.
To attain this result, we maintain implicit representations for (cid:101) x k , y k , and ¯ y k by introducing two auxiliary variables that are amenable to efficient updates.
Efficiently Updating the Primal Variable.
In Lemma 7.1, we show that we can work with an implicit representation of (cid:101) x k by introducing r k .
Lemma 7.1.
For { (cid:101) x k } defined in Eq.
(8) (and simplified in Algorithm 1), we have, for k ≥ 1 ,
[Equation]
where x k evolves as per Algorithm 1, r 1 =
and, when k ≥ 1 , r k = r k − 1 + (( n − 1) a k −
A k − 1 )( x k − x k − 1 ) .
Proof.
We prove the lemma by induction.
Using the facts that x 0 = 0 , x 1 = (cid:101) x 1 , s 1 = 0 ,
a 1 = A 1 , and A
= 0 , we have
[Equation]
Assume for certain k ≥ 2 , that Eq.
(50) holds for k − 1 .
Then, using the recursion of ˜ x k in Algorithm 1, we have that for k ≥ 3 ,
[Equation]
as required.
The expression for r k in Lemma 7.1 shows that it can be updated at cost O (1) as x k differs from x k − 1 only at one coordinate.
Therefore, by Eq.
(50) we need not compute ˜ x k in all iterations and can instead maintain r k .
Along the same lines, we give an efficient implementation strategy for y k and ¯ y k in the following discussion.
Efficiently Updating the Dual Variable.
We now show how to update the dual variable efficiently.
Lemma 7.2.
Consider { y k } and { x k } evolving as per Algorithm 1.
Then, for k = 1 , we {+have y 
+} =
Ax 1 ; for k ≥ 2 , we have
[Equation]
Proof.
The proof is directly from the definition of y k in Algorithm 1.
Lemma 7.3.
Consider { y k } and { x k } evolving as per Algorithm 1.
Then for all k ≥ 1 , we have
[Equation]
where s 1 = 0 and s k
= s k − 1 + (( n − 1) a k − A k − 1 )
A ( x k − x k − 1 ) when k ≥ 2 .
Proof.
We prove the lemma by induction.
For the base case of k = 1 , we have, by the choice of s 1 = 0 , that y
=
Ax
=
Ax 1 + 1 A 1 s 1 .
Then for some k ≥ 2 , assume Eq.
(54) holds for k − 1 , then we have,
[Equation]
where the first step is by Lemma 7.2, second step is by the induction hypothesis, third step is by adding and subtracting A k − 1 Ax k , fourth step is by rearranging terms appropriately, and the final step uses the recursive definition of s k stated in the lemma.
Dividing throughout by A k then finishes the proof.
n 2: Output: Vector x K ∈ R n + such f
x K ) ≤ (1 − ε )
f ( x ⋆ ) . n , a 2 = nn − 1 , A 1 = a 1 , ϕ 0 ( x
[Equation]
=
[Equation]
x 0 ∥
[Equation]
{+, y 
 = y 
 = Ax 0 , p 0 = 0 , q 
 = Ax 0 , t 0 = 0 , s 1 = 0 , r 1 = 0 .+}
4: for k = 1 , 2 , . . . , K do 5:
Sample j k uniformly at random from { 1 , 2 , . . . , n } 6:
if k = 1 then 7: ¯ y 0 = q8: else
if k = 2 then 9: ¯ y 1 = q 1 + a 1 a 2 t10: else if k ≥ 3 then 11: ¯ y k − 1 = q k − 1 + 1 A k −(cid:16)
1 − a 2 k − 1 a k A k − 2 (cid:17) s k − 1 + ( n − 1) a 2 k − 1 a k A k − 2 t k −12: end if 13: p k,i = (cid:26) p k − 1 ,i , i ̸
=
j k p k − 1 ,i + na k (cid:0) A T : i ¯ y k − 1 − 1 (cid:1) , i =
j k .
14: x k,i = (cid:40) x k − 1 ,i , i ̸
=
j k max (cid:8) 0 , min (cid:8) x 0 ,i −
∥ A : i ∥ 2 · p k,i , 1 ∥
A : i ∥ 2 (cid:9)(cid:9) , i =
j k 15: t k =
A ( x k − x k − 1 ) 16:
if k ≥ 2 then 17: r k = r k − 1 + (( n − 1) a k − A k − 1 )( x k − x k − 1 ) 18: s k
= s k −
+ (( n − 1) a k − A k − 1 ) t k 19: end if 20: q k = q k − 1 + t k 21:
A k +1 =
A k + a k + 22: a k +2 = min { na k +1 n − 1 , [Equation] A k + 2 n } 23: end for 24: return x K + 1 A K r K
Lemma 7.4.
Consider { x k } , { y k } , and { y k } evolving as per Algorithm 1.
Then we have that
[Equation]
and
[Equation]
[Equation]
Proof.
From the definition of ¯ y k , the initializations for x 0 , y 0 , and y 0 , and Lemma 7. [Equation] we have
[Equation]
For k ≥ [Equation] by Lemma [Equation]
[Equation]
we have
[Equation]
As a result
[Equation]
So for k ≥ [Equation] it follows that
[Equation]
where the first step is by the definition of y k in Algorithm 1, the second step is by Eq.
(58), the third step is by rearranging, and the final step is by Lemma 7.3.
Based on the above lemmas, we give our efficient lazy implementation version of Algorithm 1 in
Algorithm 2.
In Algorithm 2, we also introduce other auxiliary variables p k , q k and t k .
Based on Lemmas 7.1-7.4, it is easy to verify the equivalence between Algorithms 1 and 2.
With this implementation, by updating only the dual coordinates corresponding to the nonzero coordinates of the selected column of A , the per-iteration cost is proportional to the number of nonzero elements of the selected row in the iteration.
As a result, the overall complexity result will depend only on the number of nonzero elements of A .