A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data
Anonymous Author(s)
AffiliationAddress email
Abstract
Nonnegative (linear) least square problems are a fundamental class of problemsthat is well-studied in statistical learning and for which solvers have been implemented in many of the standard programming languages used within the machinelearning community.
The existing off-the-shelf solvers view the non-negativityconstraint in these problems as an obstacle and, compared to unconstrained leastsquares, perform additional effort to address it.
However, in many of the typicalapplications, the data itself is nonnegative as well, and we show that the nonnegativity in this case makes the problem easier.
In particular, while the worst-casedimension-independent oracle complexity for unconstrained least squares problemsnecessarily scales with one of the data matrix constants (typically the spectral norm)and these problems are solved to additive error, we show that nonnegative leastsquares problems with nonnegative data are solvable to multiplicative error andwith complexity that is independent of any matrix constants.
The algorithm weintroduce is accelerated and based on a primal-dual perspective.
We further showhow to provably obtain linear convergence using adaptive restart coupled withour method and demonstrate its effectiveness on large-scale data via numericalexperiments.
Introduction
Nonnegative least squares (NNLS) problems, defined by min x ≥ 0 12 ∥
Ax − b ∥ 22 , where A ∈ R m × n and b ∈ R m , are fundamental problems and have been studied for decades in optimization and statistical learning [25, 7, 22], with various off-the-shelf solvers available in standard packages of Python(as optimize.nnls in the SciPy package), Julia (as nnls.jl ) and MATLAB (as lsqnonneg ).
Within machine learning, NNLS problems arise whenever having negative labels is not meaningful,for example, when representing prices, age, pixel intensities, chemical concentrations or frequencycounts.
NNLS is also widely used as a subroutine in nonnegative matrix factorization [10, 17, 24] toextract sparse features in applications like clustering, collaborative filtering, and community detection.
From a statistical perspective, NNLS problems can be shown to possess a regularization propertythat enforces sparsity similar to LASSO [45], while being comparatively simpler, without the need totune a regularization parameter or perform cross-validation [43, 8, 16, 23, 47, 42].
From an algorithmic standpoint, the nonnegativity constraint in NNLS problems is typically viewedas an obstacle: most NNLS algorithms perform additional work to handle it, and the problem isconsidered harder than unconstrained least squares.
However, in many important applications of
NNLS, such as text mining [6], functional MRI [3, 19], EEG data analysis [32], pulse oximetry [21,48], statistical procedures in observational astronomy [18], and those traditionally addressed usingnonnegative matrix factorization [10], the data is also nonnegative .
We argue in this paper that when the data for NNLS is nonnegative, it is in fact possible to obtain stronger guarantees than fortraditional least squares.
Contributions.
We study NNLS problems with (element-wise) nonnegative data matrix A , towhich we refer as the NNLS+ problems, through the lens of the (equivalent) quadratic problems:
[Equation]
where c =
A ⊤ b may be assumed element-wise positive.
This assumption is without loss of generalitysince if c j < 0 for some j , then ∇ j ¯ f ( x ) ≥ 0 , implying that the j th coordinate of the optimal solutionis zero 1 .
Hence, we could fix x j = 0 and optimize over only the remaining coordinates.
We further assume that the matrix A is non-degenerate: none of its rows or columns has all elementsequal to zero.
This assumption is without loss of generality because (1) if such a row exists, wecould remove it without affecting the objective, and (2) if the j th column had all elements equal tozero, the optimal value of (P) would be −∞ , obtained for x with x j → ∞ .
Having established ourassumptions and setup, we now proceed to state our contributions, which are three-fold.(1)
A scale-invariant, ε -multiplicative algorithm.
We design an algorithm based on coordinatedescent that, in total cost O ( nnz( A ) √ ε ) , constructs an ε - multiplicative approximate solution to (P).
Ouralgorithm capitalizes on structural properties of (P) that arise as a result of the nonnegativity of A .
Theorem 1.1 (Informal; see Theorem 3.5) .
Given a matrix A ∈ R m × n + and ε > 0 , define f ( x ) =12 ∥
Ax ∥ 22 − c ⊤ x and x ⋆ ∈ argmin x ≥
f
( x ) .
Then, there exists an algorithm that in K
= O ( n log n +n √ ε ) iterations and O (cid:0)
nnz ( A ) (cid:0) log n + 1 √ ε (cid:1)(cid:1) arithmetic operations returns (cid:101) x K ∈ R n + such that
E [ ⟨∇ f ( (cid:101) x K ) , (cid:101) x
K − x ⋆ ⟩ ] ≤ ε | f ( x ⋆ ) | .
The application of our structural observations on (P) to Theorem 4 . 6 of [13] enables the recoveryof our guarantee on the optimality gap; however, we provide a guarantee on the primal-dual gap,and this is stronger than the one on the optimality gap stated in Theorem 1.1.
What is significantabout Theorem 1.1 is the invariance of the computational complexity to the scale of A —it does notdepend on any matrix constants.
This cost stands in stark contrast to that of traditional least squares,where the dependence of (oracle) complexity on matrix constants (specifically, the spectral norm of
A in the Euclidean case) is unavoidable [34, 35], and multiplicative approximation is not possible ingeneral .
Conceptually, our algorithm is a new acceleration technique inspired by VRPDA 2 [44].(2) Linear convergence with restart.
By incorporating adaptive restart in (P), we improve theguarantee of Theorem 1.1 to one with linear convergence (with log(1 /ε ) complexity).
Thus, weestablish the first theoretical guarantee for NNLS+ that simultaneously satisfies the properties ofbeing scale-invariant, accelerated, and linearly-convergent .
Theorem 1.2 (Informal; see Theorem 4.1) .
Consider the setup of Theorem 1.1.
Then, there is analgorithm that in expected O (nnz( A )(log n + √ nµ ) log( 1 ε )) arithmetic operations returns (cid:101) x K ∈ R n +with f ( (cid:101) x K ) − f ( x ⋆ ) ≤ ε | f
( x ⋆ ) | , where µ is the constant in a local error bound for (P) .
Proving this bound requires bounding the expected number of iterations between restarts in conjunction with careful technical work in identifying an appropriate local error bound for NNLS+.(3) Numerical experiments.
We consolidate our theoretical contributions with a demonstration of theempirical advantage of our restarted method over state-of-the-art solvers via numerical experimentson datasets from LibSVM with sizes up to 19996 × 1355191 .
Figure 1 shows that, when combinedwith the restart strategy, our algorithm significantly outperforms the compared algorithms.
Related work.
NNLS has seen a large body of work on the empirical front.
The first methodthat was widely adopted in practice (including in the lsqnonneg implementation of MATLAB)is due to the seminal work of [25] (originally published in 1974).
This method, based on activesets, solves NNLS via a sequence of (unconstrained) least squares problems and has been followed up by [7, 46, 33, 11] with improved empirical performance.
While these variants are generallyeffective on small to mid-scale problem instances, they are not suitable for extreme-scale problemsubiquitous in machine learning.
For example, in the experiments reported in [33], Fast NNLS [7] took6.63 days to solve a problem of size 45000 × 45000 , while the TNT-NN algorithm [33] took 2.45hours.
However, the latter requires computing the Cholesky decomposition of A ⊤ A at initialization,which can be prohibitively expensive both in computation and in memory.
Another prominentwork on the empirical front is that of [22], which performs projected gradient descent with modified
Barzilai-Borwein steps [4] and step sizes a carefully designed sequence of diminishing scalars.
Another, separate, line of work concerns optimization algorithms with multiplicative error guarantees.
Of interest to us are standard first-order algorithms that run in time that is near-linear in the input sizeand are thus applicable to large-scale setting (for multiplicative-error algorithms applicable to broadclasses of problems but that run in time that is superlinear in the input size, see [40, Chapter 7]).
Mostof the existing literature in this domain concerns positive (packing and covering) linear programs (e.g.,[28, 49, 1, 29]).
Results also exist for positive semidefinite programs [2], (nonlinear) fair packing andcovering problems [31, 14], and fair packing problems under Schatten norms for matrices [20].
Withthe exception of [14] (discussed below), none of these results are directly applicable to (P).
To the best of our knowledge, theoretical guarantees explicitly for (P) have been scarce.
For instance, [22] mentioned in the preceding paragraph provides only asymptotic guarantees.
Orthogonally,the result on 1-fair covering by [14] solves the dual of NNLS+, which also gives a multiplicativeguarantee for NNLS+, but with overall complexity ˜
O
( nnz( A ) ε ) , where ˜ O ( · ) hides poly-log factors.
Since our algorithm is based on the coordinate descent algorithm, we highlight some results ofother coordinate descent algorithms when specialized to the closely related problem of unconstrained linear regression .
The pioneering work of [36] proposed a coordinate descent methodcalled RCDM , which in our setting has an iteration cost O (cid:0) (cid:80) nj =1 ∥ A [Equation] ∥
[Equation]
x [Equation] ε (cid:1) , where ∥ A : j ∥ 2 isthe Euclidean norm of the j th column of A .
This was improved by [26], in an algorithm termed
ACDM , by combining Nesterov’s estimation technique [38] and coordinate sampling, giving an iteration complexity of O (cid:0)
[Equation]
for solving (P).
The latest results in this line ofwork by [2, 41, 39] perform non-uniform sampling atop a framework of [36] and achieve iterationcomplexity of O (cid:0)
[Equation]
, with [12] dropping the dependence on max 1 ≤ j ≤ n ∥ A : j ∥ 2 .
Additionally, the work of [27] develops an accelerated randomized proximal coordinate gradient( APCG ) method to minimize composite convex functions.
As remarked earlier, [12], coupled with insights on NNLS+ problems provided in this work, canrecover our guarantee for the optimality gap from Theorem 3.5.
However, our work is the first to bringto the fore the properties of NNLS+ required to get such a guarantee, and our choice of primal-dualperspective allows for a stronger guarantee in terms of an upper bound on the primal-dual gap.
Further, our algorithm is a novel type of acceleration, with our primal-dual perspective transparentlyillustrating our use of the aforementioned properties.
We believe that these technical contributions,along with our techniques to obtain vastly improved theoretical guarantees with the restart strategyapplied to this problem, are valuable to the broader optimization and machine learning communities.
Notation and preliminaries
Throughout the paper, we use bold lowercase letters to denote vectors and bold uppercase lettersfor matrices.
For vectors and matrices, the operator ′ ≥ ′ is applied element-wise, and R + is thenon-negative part of the real line.
We use ⟨ a , b ⟩ to denote the inner product of vectors a and b and ∇for gradient.
Given a matrix A , we use A : j for its j th column vector, and for a vector x , x j denotesits j th coordinate.
We use nnz( A ) for the number of non-zero entries of A .
We use x k for the vectorin the k th iteration and, to disambiguate indexing, use [ x k ] j to mean the j th coordinate of x k .
Thei th standard basis vector is denoted by e i .
For an n -dimensional vector x and A ∈ R m × n , we defineΛ = diag([ ∥ A :1 ∥ 22 , . . .
, ∥ A : n ∥ 22 ]) and ∥ x
∥ 2 Λ = (cid:80) ni
=1 x 2 i ∥
A : i ∥ 22 .
Finally, [ n ] def = { 1 , 2 , . . . , n } .
A differentiable function f : R n → R is convex if for any x , (cid:98) x ∈ R n , we have f ( (cid:98) x ) ≥ f ( x ) +⟨∇ f ( x ) , (cid:98) x − x ⟩ .
A differentiable function f : R n → R is said to be µ -strongly convex w.r.t.
the ℓ 2 -norm if for any x , (cid:98) x ∈ R n , we have that f ( (cid:98) x ) ≥ f ( x ) + ⟨∇ f ( x ) , (cid:98) x − x ⟩ + µ 2 ∥ x − (cid:98) x
∥ 22 .
We haveanalogous definitions for concave and strongly concave functions, which flip the inequalities noted.
Given a convex program min x ∈X f ( x ) , with f : R n → R is differentiable and convex and X ⊆
R nclosed and convex, the first-order optimality condition of a solution x ⋆ ∈ argmin x ∈X f ( x ) is( ∀ x ∈ X ) :
⟨∇ f ( x ⋆ ) , x − x ⋆ ⟩ ≥ 0 .
Problem setup.
As discussed in the introduction, our goal is to solve (P), with A ∈ R m × n + .
Fornotational convenience, we work with the problem in the following scaled form:
[Equation]
This assumption is w.l.o.g.
since any (P) can be brought to this form by a simple change of variableˆ x j = c
j x j (see also e.g., [1, 14] for similar scaling ideas).
The scaling need not be explicit in thealgorithm since the change of variable ˆ x j = c
j x j is easily reversible.
Properties of the objective.
To kick off our analysis, we highlight some properties inherent to theobjective defined in (2).
These are central to obtain a scale-invariant algorithm for (P).
Proposition 2.1.
Given f : R n + → R as defined in (2) and x ⋆ ∈ argmin x ∈ R n + f ( x ) , the followingstatements all hold.
a) ∇ f ( x ⋆ ) ≥ 0 . b) f ( x ⋆ )
= − 12 ∥
Ax ⋆ ∥ 22 = − 12 1 ⊤ x ⋆ . c) for all j ∈ [ n ] , we have x ⋆j ∈ (cid:104) 0 , 1 ∥
A : j ∥(cid:105) .d)
− 12 (cid:80) j ∈ [ n ] 1 ∥
A : j ∥ 22 ≤ f ( x ⋆ ) ≤ − 1 2min j ∈ [ n ] ∥ A : j ∥ 22 .
The validity of division by ∥ A : j ∥ 2 in the preceding proposition is by the non-degeneracy of Adiscussed in the introduction.
We prove this proposition in Section 6.1.
An important consequence of Proposition 2.1 (c) is that (P) can be restricted to the hyperrectangle
X = { x
[Equation] n : [Equation] ≤ x j ≤ 1 ∥
A : j ∥ 22 } without affecting its optimal solution, but effectively reducingthe search space.
Thus, going forward, we replace the constraint x ≥ 0 in (P) by x ∈ X .
Primal-dual gap perspective.
As alluded earlier, our algorithm is analyzed through a primal-dualperspective.
For this reason, it is useful to consider the Lagrangian
[Equation]
from which we can derive our rescaled problem (2) as the primal problem min x ∈X P ( x ) , where
[Equation]
Similar to [44], we use Eq.
(3) to define the following relaxation of the primal-dual gap, for arbitrarybut fixed u ∈ X , and v ∈ R m :
[Equation]
The significance of this relaxed gap function is that for a candidate solution (cid:101) x and an arbitrary(cid:101) y ∈ R m , a bound on Gap ( u , v ) L ( (cid:101) x , (cid:101) y ) translates to one on the primal error, as follows.
First selectu = x ⋆ , v = A (cid:101) x .
Then, by observing that L ( (cid:101) x , A (cid:101) x ) =
f ( (cid:101) x ) and L ( x ⋆ , Ax ⋆ ) =
f ( x ⋆ ) , we havef
[Equation]
For a fixed x , L ( x , · ) is 1-strongly concave and minimized at Ax .
Thus, L ( x ⋆ , A (cid:101) x ) ≤ L ( x ⋆ , Ax ⋆ ) −1 2 ∥
A ( (cid:101) x − x ⋆ ) ∥ 2 .
Hence, we have the following primal bound:
[Equation]
In light of this connection, our algorithm for bounding the primal error is one that generates iteratesthat can be used to construct bounds on Gap ( u , v ) L ( [Equation] x , (cid:101) y ) , as we detail next.
Our algorithm and convergence analysis
Our algorithm, Scale Invariant NNLS+ (SI-NNLS+), is an iterative algorithm using the estimatesequences
ϕ k ( x ) and ψ k ( y ) for k ≥ 1 (see Section 3.1) giving the primal and dual updatesx k =
argmin x ∈X ϕ k
( x ) and y k = argmax y
[Equation]
k ( y ) .
We use our algorithm’s iterates from Eq.
(6) to construct G k , an upper estimate of Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) , 167 where (cid:101) x k , (cid:101)
y k are convex combinations of the iterates and u ∈ X , v ∈ R m + , where u is fixed and v isarbitrary.
Our motivation for constructing G k ( u , v ) ≥ Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) is to obtain a bound on theprimal error, using Inequality (5).
The main goal in the analysis is to show that G k ≤ QA k , where A k isa sequence of positive numbers and Q is bounded.
To obtain the claimed multiplicative approximation,we use Inequality (5) and argue that for u = x
⋆ , v = A (cid:101) x
k , we have Q ≤ O (cid:0) | f
( x ∗ ) | (cid:1) .
Our algorithm may be interpreted as a variant of the VRPDA 2 algorithm [44], with our analysistechnique inspired by the approximate duality gap technique [13]; however, unlike VRPDA 2 , whichuses estimate sequences, our analysis directly bounds the primal-dual gap.
Another difference from
VRPDA 2 is in our choice of primal regularizer (described shortly) and our lack of a dual regularizer.
A variant of SI-NNLS+ suitable for the analysis is provided in Algorithm 1, with proofs in Section 6.2and Section 6.3.
We give an (equivalent) implementation version (“ Lazy SI-NNLS+” 3 ) in Algorithm2 and its analysis in Section 7.
1: Input: Matrix A ∈ R m × n + with n ≥ 4 , accuracy ε , initial point x2:
Initialize: (cid:101) x 0 = x 0 , y
= y
= Ax 0 , K = 52 n log n + 6 n √ ε , a 1 = 1 √ 2 n 1 . 5 , a 2 = a 1 n − 1 , A 0 = 0 ,
A 1 = a 1 , ϕ 0 ( x ) =
∥ x − x 0 ∥ 2 Λ .
3: for k = 1 , 2 , . . .
, K do 4:
Sample j k uniformly at random from { 1 , 2 , . . . , n } 5: x k ← arg min x ∈X ϕ k ( x ) , for ϕ k ( x ) defined by Eq.
(12) and Eq.
(15) 6: y k ← arg max y ∈
R m ψ k ( y ) , for ψ k ( y ) defined by Eq.
(9) 7: (cid:101) x k =
A k (cid:2)
A k − 1 (cid:101) x
k − 1 + a k (cid:0) n x k − ( n − 1) x k − 1 (cid:1)(cid:3) .
8: y k ← y k + a k a k
+
( y k − y k − 1 ) 9:
A k +1 ← A k + a k +1 , a k +2 = min { na k +1 n − 1 , √ A k + 2 n } 10: end for 11: Return (cid:101) x K
3.1 Gap estimate construction
The gap estimate G k is constructed as the difference G k ( u , v ) =
U k
( v ) − L k ( u ) , where U k ( v ) ≥
L ( (cid:101) x k , v ) and L k ( u ) ≤ L ( u , (cid:101) y k ) are, respectively, upper and lower bounds we construct on the
Lagrangian.
It then follows by Eq.
(4) that G k ( u , v ) is a valid upper estimate of Gap ( u , v ) L ( (cid:101) x k , (cid:101) y k ) .
We first introduce a technical component our constructions L k and U k crucially hinge on: we definetwo positive sequences of numbers { a i } i ≥ 1 and { a ki } 1 ≤ i ≤ k , with one of their properties being thatboth sum up to A k > 0 for k ≥ 1 .
Specifically, we define A 0 = 0 and { a i } i ≥ 1 as a i =
A i − A
i − 1 .
The sequence { a ki } changes with k and for k = 1 is defined by a 11 = a 1 , while for k ≥ 2 :
[Equation]
Summing over i ∈ [ k ] verifies that A k = (cid:80) ki
=1 a ki .
For the sequence { a ki } 1 ≤ i ≤ k to be non-negative,we further require that a 1 − ( n − 1) a 2 ≥ 0 and ∀ i ≥ 2 , na i − ( n − 1) a i +1 ≥ 0 . , to improve iteration, we update few coordinates as is possible.
The significance of these two sequences lies in defining the algorithm’s primal-dual output pair by
[Equation]
The intricate interdependence of { a i } and { a ki } enables expressing (cid:101) x k in terms of only { a i } .
Thisexpression further simplifies to a cheaper recursive one, which is used in Algorithm 1.
With the sequences { a i } i ≥ 1 and { a ki } 1 ≤ i ≤ k in tow, we are now ready to show the construction of anupper bound U k ( v ) on L ( (cid:101) x k , v ) and a lower bound L k ( u ) on L ( u , (cid:101) y k ) .
Upper bound.
To construct an upper bound, first observe that by Eq.
(3) and Eq.
[Equation]
Consider the primal estimate sequence defined for k = 0 as ψ 0 = 0 and for k ≥ 1 by
[Equation]
which ensures that L ( (cid:101) x k , v )
=
1 A k ψ k ( v ) .
A key upshot of constructing ψ k ( v ) as in Eq.
(9) is thatthe quadratic term implies A k -strong concavity of ψ k for k ≥ 1 , which in turn ensures that thevector y k =
[Equation]
( y ) from Eq.
(6) is unique.
This property, coupled with the first-orderoptimality condition in Inequality (1), gives that for any y ∈ R m , ψ k ( y ) ≤ ψ k ( y k )
− A k 2 ∥ y − y k
∥ 22 .
We are now ready to define the following upper bound by:
[Equation]
The preceding discussion immediately implies that U k is a valid upper bound for the Lagrangian.
Lemma 3.1.
For U k as defined in Eq.
(10), Lagrangian defined in Eq.
(3) and (cid:101) x k ∈ R n + in Eq.
(8),we have, for all y ∈ R m , the upper bound U k ( y ) ≥ L ( (cid:101) x k , y ) .
Lower bound.
Analogous to the preceding section, we now obtain a lower bound on the Lagrangian,completing the bound on the gap estimate.
However, the construction becomes more technical.
Westart with the same approach as for the upper bound.
Since L ( u , (cid:101) y k ) is concave in (cid:101) y k , by Jensen’sinequality: L ( u , (cid:101) y k ) ≥
A k (cid:80) i ∈
[ k ] a i (cid:16) ⟨ Au , y i ⟩ − 1 ⊤ u − 12 ∥
y i ∥ 22 (cid:17) .
Were we to define the dualestimate sequence ϕ k in the same way as we did for the primal estimate sequence ψ k , we would nowsimply define it as A k times the right-hand side in the last inequality.
However, doing so would makeϕ k depend on y k , which is updated after x k , which in Eq.
(6) is defined as the minimizer of the ϕ k .
To avoid such a circular dependency, we add and subtract a linear term (cid:80) i ∈
[ k ] ⟨ A ⊤ y
i − 1 , u ⟩ , wherey i − 1 , defined later, are extrapolation points that depend only on y 1 , . . .
y i − 1 .
We thus have
[Equation]
If we now defined ϕ k based on the first term in the above inequality, we run into another obstacle:the linearity of the resulting estimate sequence is insufficient for cancelling all the error terms inthe analysis.
Hence, as is common, we introduce strong convexity by adding and subtracting anappropriate strongly convex function.
Our chosen strongly convex function is motivated by thebox-constrained property of the optimum from Proposition 2.1 (c) and crucial in bounding the initialgap estimate.
It coincides with ϕ 0 : for any x ∈ R n + , define the function
[Equation]
This function is 1 -strongly convex with respect to ∥ · ∥ Λ and used in defining ϕ 1 as:
[Equation]
The definition of ϕ 1 is driven by the purpose of cancelling initial error terms.
Next, we choose ϕ k sothat for any fixed u ∈ X , we have
[Equation]
where the expectation is with respect to all the randomness in the algorithm.
This construction isused to reduce the per-iteration complexity, for which we employ a randomized coordinate update onx k for k ≥ 2 .
To support such updates, we relax the lower bound to hold only in expectation .
Concretely, let j i be the coordinate sampled uniformly at random from [ n ] in the i th iteration of
SI-NNLS+, independent of history.
Fix y
i for i = 1 , . . .
, k − 1 and for k ≥ 2 and x ∈ X , define
[Equation]
For k ≥ 2 , ϕ k ( u ) can also be defined recursively via
[Equation]
The function ϕ
k inherits the strong convexity of ϕ 0 .
This property, together with Eq.
(6) and first-orderoptimality from Inequality (1), give
[Equation]
[Equation]
Along with strong convexity, our choice of ϕ k in Eq.
(15) leads to the following properties essentialto our analysis: (1)
ϕ k is separable in its coordinates; (2) the primal variable x k is updated only at itsj th k coordinate; (3) Eq.
(13) is true.
These are formally stated in Proposition 6.2.
With the dual estimate sequence ϕ k defined in Eq.
(15), we now define the sequence L k by
[Equation]
We conclude this section by justifying our choice of L k as a valid expected lower bound on E L ( x , (cid:101) y k ) .
Lemma 3.2.
For L k defined in Eq.
(17), for the Lagrangian in Eq.
(3) and (cid:101) y k in Eq.
(8), we have,for a fixed u ∈ X , the lower bound E L ( u , (cid:101)
y k ) ≥
E L k ( u ) , where the expectation is with respect toall the random choices of coordinates in Algorithm 1.
3.2 Bounding the gap estimate
With the gap estimate G k constructed as in the preceding section and combining Eq.
(10) and Eq.
(17),we now achieve our goal of bounding A k G k (to obtain a convergence rate of the order 1 /A k ) bybounding the change in A k G k and the initial scaled gap
A 1 G 1 .
Lemma 3.3.
Consider the iterates { x k } and { y k } evolving according to Algorithm 1.
Let n ≥ 2and assume that a 1 = 1 √ 2 n 1 . 5 and a 1 ≥ ( n − 1) a 2 , while for k ≥ 3 ,
[Equation]
Then, for fixed u ∈ X , any v ∈ R m , and all k ≥ 2 , the gap estimate G k
= U k − L k satisfies
[Equation]
Lemma 3.4.
Given a fixed u ∈ X , any v ∈ R m , ¯ y
=
y 0 , and x 1 and y 1 from Algorithm 1, we have
[Equation]
Combining the two lemmas, we now bound G K and deduce our final result on the primal error.
Theorem 3.5.
[Main Result] Assume that n ≥ 4 .
Given a matrix A ∈ R m × n + , ε > 0 , an arbitraryx 0 ∈ X and ¯ y
= y
= Ax 0 , let x k and A k evolve according to SI-NNLS+ (Algorithm 1) for k ≥ 1 .
For f defined in (2), define x ⋆ ∈ argmin x ≥
f
( x ) .
Then, for all K ≥ 2 , we have
[Equation]
When K ≥ 52 n log n , we have A K ≥ ( K − 52 n log n ) 2 36 n.
If ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | , then for K ≥ 52 n log n +6 n √ ε , we have E [ f ( (cid:101) x K ) − f ( x ⋆ )] ≤ ε | f
( x ⋆ ) | .
The total cost is O (cid:0) nnz
( A ) (cid:0) log n + 1 √ ε
(cid:1)(cid:1) .
The assumption ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | above is satisfied by x 0 =
(c.f.
Proposition 2.1 and Eq.
Remark 3.6 .
SI-NNLS+ (Algorithm 1) and Theorem 3.5 also generalize to a mini-batch version.
Increasing the batch size grows our bounds and number of data passes by a factor of at most squareroot of the batch size s , by relating the spectral norm of the s columns of A corresponding to a batchto the Euclidean norms of individual columns of A from the same batch.
However, due to efficientavailable implementations of vector operations, mini-batch variants of our algorithm with small batchsizes can have lower total runtimes on some datasets (see Section 5).
Adaptive restart
We now describe how SI-NNLS+ can be combined with adaptive restart to obtain linear convergencerate.
To apply the restart strategy, we need suitable upper and lower bounds on the measure ofconvergence rate.
Our measure of optimality is the natural residual r ( x ) =
∥ R ( x ) ∥ Λ [30] for
[Equation]
) where Π R n + is the projection operator onto R n + and Λ is as defined in Section 2.
For Λ
= I , R ( x ) isthe natural map as defined in, e.g., [15].
Due to space constraints, we only state the main result ofthis section in the following theorem, while full technical details are deferred to Section 6.4.
Theorem 4.1.
Given an error parameter ε > 0 and x 0 = 0 , consider the following algorithm A :
A : SI-NNLS+ with Restarts
Initialize: k = 1 .
Initialize Lazy SI-NNLS+ at x k − 1 .
Run Lazy SI-NNLS+ until the output (cid:101) x
kK satisfies r ( (cid:101) x kK ) ≤ 12 r ( x k − 1 ) .
Restart Lazy SI-NNLS+ initializing at x k = (cid:101) x kK .
Increment k.
Repeat until r ( (cid:101) x kK ) ≤ ε.
Then, the expected number of arithmetic operations of A is O (cid:0) nnz
( A ) (cid:0) log n + √ nµ (cid:1) log (cid:0) r ( x 0 ) ε (cid:1)(cid:1) .
As a consequence, given ¯ ε > 0 , the total expected number of arithmetic operations until a point withf ( x ) − f ( x ⋆ ) ≤ ¯ ε | f ( x ⋆ ) | can be constructed by A is O (cid:16) nnz
( A ) (cid:16) log n + √ nµ (cid:17) log (cid:16) nµ ¯ ε (cid:17)(cid:17) .
Numerical experiments and discussion
We conclude our paper by presenting the numerical performance of SI-NNLS+ and its restart versions(see the efficient implementation version in Algorithm 2) against FISTA with restart [5, 37], a generalpurpose large-scale optimization algorithm, OA+DS with restart designed by [22] specifically forlarge-scale NNLS problems, and APCG [27] with restart.
As an accelerated algorithm, FISTA hasthe optimal 1 /k 2 convergence rate; OA+DS, while often efficient in practice, has only an asymptoticconvergence guarantee.
For FISTA and APCG, we compute the tightest Lipschitz constant (i.e., thespectral norm ∥ A ∥ ); for OA+DS, we follow the best practices laid out by [22].
For our SI-NNLS+algorithm and its restart version with batch size bs = 1 , we follow Algorithm 2 and the restart strategy
[Figure]
in Section 4.
For the restart version with batch size larger than 1 , we choose the best batch size in{ 10 , 50 , 300 , 500 } and compute the block coordinate Lipschitz constants as the spectral norms of thecorresponding block matrices.
All algorithms were implemented in Julia.
We evaluated the performance of the algorithms on the large-scale sparse datasets real-sim , news20 ,and E2006train from the LibSVM library [9].
Both real-sim and news20 datasets have nonnegative data matrices, but the labels may be negative.
When there exist negative labels, it is possiblefor the elements of A ⊤ b to be negative.
In such cases, per the discussion from the introduction, wecan simply remove the corresponding columns of A and solve an equivalent problem with smallerdimension.
On the other hand, the data matrix in E2006train dataset is not non-negative, whichmeans that this dataset does not satisfy the assumption required for the analysis of Algorithm 1.
However, Algorithm 1 can still be run by keeping only the non-negativity constraints for primalupdates.
This example is provided solely for illustration of empirical performance.
Results.
To compare all implemented algorithms, we plot the natural residual/objective value gapversus number of data passes/time in Figure 1.
Figure 1(a)-(d) shows that SI+NNLS+ is betterthan FISTA and OA+DS in terms of number of data passes on the real-sim dataset and betterthan APCG in terms of time.
While the proposed restart strategy speeds up all the algorithms tolinear convergence, variants of SI-NNLS+ remain competitive in all the settings.
In terms of theperformance of different variants of SI-NNLS+, with bs = 1 , we have a much better coordinate
Lipschitz constant than for bs = 10 and thus the case of bs = 1 dominates bs = 10 in terms ofdata passes.
As FISTA and OA+DS take less time accessing the full dataset once, they have lowerruntimes than SI-NNLS+ but are beaten by SI-NNLS+ with restart and bs = 1 .
In Figure 1(e)-(h), onthe news20 dataset, in terms of number of data passes, restarted APCG and SI-NNLS+ with bs = 1are dominant.
However, as news20 is a very sparse dataset, letting bs = 1 significantly increases thetotal time to access the full data once due to the overhead per iteration.
As a result, single-coordinatemethods have the worst runtimes, while restarted OA+DS is the fastest but SI-NNLS+ with bs = 10remains competitive.
Figure 1(i)-(l) shows the performance comparison on the E2006train dataset.
On this dataset, both FISTA and OA+DS ran for 4 hours without visibly reducing the function value,whereas our (block) coordinate algorithm outperforms them in both number of data passes and time.
Further experiments are left for future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We summarized the contributions and reviewedrelated work in Introduction.
Section 3 and 6 contain the full technical details forour theoretical results.
Section 5 provides full details for numerical experiments,while Section 7 provides full technical details for the implementation version of ouralgorithm, including the associated discussion about maintaining the claimed theoreticalguarantees.(b)
Did you describe the limitations of your work?
[Yes] The main examples are: (i) thatsimilar to ours (although weaker) results can be obtained with existing acceleratedcoordinate descent algorithms once the structural properties we prove are taken intoaccount, in Introduction (Section 1); (ii) the discrepancy between the theoretical andempirical guarantee depending on the batch size as a consequence of modern processorarchitecture, in Remark 3.6 and Section 5; and (iii) that further experiments are left forfuture work, in Section 5.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
This isa theoretical optimization paper for generic non-negative least squares problems, andas such it is not tied to any specific application that could lead to a negative societalimpact. (d) Have you read the ethics
review
guidelines
and
ensured
that your paper conforms to them
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We stated all the assumptions in Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] We provided a detailed
technical overview in Sections 3 and 4, and included complete proofs in Sections 6 and7.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We includedthe code in the supplementary material.
The data source used in this paper is cited inSection 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We explained the data preprocessing in Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] The performance of all the algorithms was consistent over
many runs and thus the error bars would not have been particularly informative.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We included this information in
Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we cited [9]; see
Section 5.
(b) Did you mention the license of the assets?
[N/A]
The datasets we use are from LibSVM,
which is an open source library under the 3-clause BSD license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We included the code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The data we use is from LibSVM, which is an open resource
database.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
According to the data’s description, there is
no such issue.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix: Omitted Technical Details
In this section, we provide proofs for all the claims made in the main body of the paper, additionalsupporting propositions and lemmas, and also omitted details of the implementation of our algorithm.
The proofs are provided in the order in which the corresponding statement appears in the main body.
In Section 6.1, we prove the properties of our problem.
We again emphasize that these properties arecrucial to achieving our goal of a scale-invariant algorithm for (P).
Section 6.2 contains the proofs ofall our results pertaining to the convergence analysis of Algorithm 1, with proofs of the growth rateof the scalar sequences { a i } , { a ki } , and A k grouped separately in Section 6.3, owing to their moretechnical nature.
6.1 Omitted Proof from Section 2: Properties of Our Objective
Proposition 2.1.
Given f : R n + → R as defined in (2) and x ⋆ ∈ argmin x ∈ R n + f ( x ) , the followingstatements all hold.
a) ∇ f ( x ⋆ ) ≥ 0 .
b) f ( x ⋆ )
= − 12 ∥
Ax ⋆ ∥ 22 = − 12 1 ⊤ x ⋆ .c) for all j ∈ [ n ] , we have x ⋆j ∈ (cid:104) 0 , 1 ∥
A : j ∥(cid:105) .
d) − 12 (cid:80) j ∈ [ n ] 1 ∥
A : j ∥ 22 ≤ f ( x ⋆ ) ≤ − 1 2min j ∈ [ n ] ∥ A : j ∥ 22 .
Proof.
We recall the first-order optimality condition stated in Inequality (1): for all x ≥ 0 , we have⟨∇ f
( x ⋆ ) , x − x ⋆ ⟩ ≥ 0 ; we repeatedly invoke this inequality in the proof below.1.
Suppose there exists a coordinate j at which Proposition 2.1 (a) does not hold and instead,we have ∇ j f ( x ⋆ ) < 0 .
Consider x ≥
such that x i =
x ⋆i for all i ̸
=
j and let x j = x ⋆j + εfor some ε > 0 .
Then, Inequality (1) becomes ∇ j f ( x ⋆ ) · ε ≥ 0 .
Under the assumption∇ j f ( x ⋆ ) < 0 , this is an invalid inequality, thus contradicting our assumption.2.
From Proposition 2.1 (a), we know that ∇ f ( x ⋆ ) ≥ 0 .
If ∇ i f ( x ⋆ ) > 0 , and if x ⋆i > 0 , thenby picking a vector x such that x j = x ⋆j for j ̸ = i and x i = x ⋆i − γ for any γ ∈ (0 , x ⋆i ) , weviolate Inequality (1).
Therefore it must be the case that if ∇ i f ( x ⋆ ) > 0 , then x ⋆i = 0 .
Thuswe have
[Equation]
Therefore, f ( x ⋆ ) = [Equation] x ⋆ ,
[Equation]
x ⋆
[Equation]
− 12 ⟨ x ⋆ , [Equation] ⊤ Ax ⋆ ⟩ = − 12 1 ⊤ x ⋆ .3.
From the proof of Proposition 2.1 (b), we have ⟨ x ⋆ , ∇ f ( x ⋆ ) ⟩ = 0 .
We also have x ⋆ ≥ 0and, from Proposition 2.1 (a), that ∇ f ( x ⋆ ) ≥ 0 .
Therefore, if x ⋆i > 0 for some coordinate ithen it must be that ∇ i f ( x ⋆ ) = 0 .
That is, 1 = ⟨ A : i , Ax ⋆ ⟩ .
Combining this equality withthe fact that A and x ⋆ are both coordinate-wise non-negative gives
[Equation]
which implies x ⋆i ≤ 1 ∥
A : i ∥ 22 for all coordinates i .4.
The lower bound follows immediately by combining Proposition 2.1 (b) and Proposition 2.1 (c).
For the upper bound, we need to find a feasible point (cid:98) x and compute the function value at (cid:98) x , since f ( x ⋆ ) =
min y ≥ 0 f ( y ) ≤ f ( (cid:98) x ) .
Let (cid:98) x
= γ e k for some γ > 0 .
Then,
[Equation]
Let γ = 1 ∥
A : k ∥ 22 .
Then, f ( (cid:98) x ) = − 1 2 ∥
A : k ∥ 22 .
We pick k = arg min i ∈ [ n ] ∥ A : i ∥ 2 , therefore f
from Section
Estimates We first show the results stating U k and L k are indeed valid upper and lower (respectively) estimatesof the Lagrangian.
Lemma 3.1.
For U k as defined in Eq.
(10), Lagrangian defined in Eq.
(3) and (cid:101) x k ∈ R n + in Eq.
(8),we have, for all y ∈ R m , the upper bound U k ( y ) ≥ L ( (cid:101) x k , y ) .
Proof.
By evaluating the Lagrangian described by Eq.
(3) at x = (cid:101) x k , and by definition of ψ k from
Eq.
(9), we obtain the following upper bound on the Lagrangian at ( (cid:101) x k , y ) .
[Equation]
where the final steps are by strong convexity of ψ k and by Eq.
We emphasize that y here can be random since this is a deterministic statement.
Lemma 3.2.
For L k defined in Eq.
(17), for the Lagrangian in Eq.
(3) and (cid:101) y k in Eq.
(8), we have,for a fixed u ∈ X , the lower bound E L ( u , (cid:101)
y k ) ≥
E L k ( u ) , where the expectation is with respect toall the random choices of coordinates in Algorithm 1.
Proof.
First, evaluating Eq.
(3) at (cid:101) y k gives
[Equation]
Taking the expectation on both sides
[Equation]
applying the definition of (cid:101) y k , convexity of 12 ∥ · ∥ 2 , and Jensen’sinequality, and adding and subtracting 1 A k E (cid:80)
i ∈
[ k ] a i ⟨ Ax , y i − 1 ⟩ + 1 A k ϕ 0 ( u ) gives
[Equation]
We continue the analysis as
[Equation]
6.2.2 Proofs from Section 3.
We now describe three technical propositions that bound terms that show up in the proof of our resulton bounding the scaled gap estimate.
Proposition 6.1.
For ψ k defined in Eq.
(9), with { a ki } defined in Eq.
(7), we have for all k ≥ 1 ,
[Equation]
Proof.
Evaluating ψ k and ψ k − 1 as defined in Eq.
(9) at y k and subtracting, we have
[Equation]
[Equation]
Next, applying strong convexity of ψ k − 1 at y k and y k − 1 while using the fact that y k − 1 minimizes 546 ψ k − 1 gives
[Equation]
To complete the proof, it remains to add Eq.
(20) and Inequality (21).
Proposition 6.2.
The random function ϕ k : X → R , k ≥ 2 , defined in Eq.
(15) satisfies the followingproperties, with x k , y k , and y k evolving as per Algorithm 1.
a) It is separable in its coordinates: ϕ k ( x )
= (cid:80) j ∈ [ n ] ϕ k,j ( x j ) , where
, for each j ∈ [ n ] , wedefine ϕ 0 ,j ( x j ) =
∥
A : j ∥ 22( x j − [ x 0 ] j ) 2 , ϕ 1 ,j ( x j ) =
a 1 x j ( A ⊤ y 0 − 1 ) j + ϕ 0 ,j ( x j ) , and fork ≥ 2 ,
[Equation]
b)
The primal variable x k is updated only on the j th k coordinate in each iteration:
x k =x
k − 1 + γ e j k for some γ , and [ x k ] j = [ x k − 1 ] j for j ̸ = j k .c)
For a fixed x ∈ X and for k ≥ 1 , we have, over all the randomness in the algorithm,
[Equation]
Proof.
In the statement of Proposition 6.2 (a), the claim about separability of ϕ 0 and ϕ 1 can be checked just from the definitions of ϕ 0 ,j and ϕ 1 ,j .
We prove the claim of coordinate-wise separability for k ≥ 2 by summing over j ∈ [ n ] both sides of Eq.
We can compute this sum via following observation, which concludes the proof of Proposition 6.2 (a).
[Equation]
From Proposition 6.2 (a), we may therefore define, for j ̸ = j k ,
[Equation]
u ) =
[ x k − 1 ] j .
Therefore, [ x k ] j = [ x k − 1 ] j for all j ̸ = j k , thus proving Proposition 6.2 (b).
To prove Proposition 6.2 (c), we use induction on k .
The base case holds for k = 1 by the definition of ϕ 1 ( x ) .
Let
Proposition 6.2 (c) hold for k ≥ 1 .
Then, by the definition of ϕ k as in Eq.
(15), we have ϕ k
( x ) =
ϕ k − 1 ( x ) + na k ⟨ A ⊤ y k − 1 − 1 , x j k e j k ⟩ , for all k ≥ 2 .
Let F k − 1 be the natural filtration, containing all the randomness in the algorithm up to and includingiteration k − 1 .
Taking expectations with respect to all the randomness until iteration k and invokinglinearity of expectation, the inductive hypothesis, and the tower rule E [ · ] =
E
[ E [ · |F k − 1 ]] , we have
[Equation]
which finishes the proof of Proposition 6.2 (c).
Proposition 6.3.
For all k ≥ 2 , the random function ϕ
k : X → R , k ≥ 2 , defined in Eq.
(15)satisfies the following inequality, where x k and y k evolve according to Algorithm 1.
[Equation]
Proof.
We have, using x = x k in Eq.
(15), that
[Equation]
Applying Eq.
(16) to ϕ k − 1 at x k gives
[Equation]
Adding these inequalities completes the proof of the claim.
We now use the preceding technical results to bound the change in scaled gap.
Lemma 3.3.
Consider the iterates { x k } and { y k } evolving according to Algorithm 1.
Let n ≥ 2and assume that a 1 = 1 √ 2 n 1 . 5 and a 1 ≥ ( n − 1) a 2 , while for k ≥ 3 ,
[Equation]
Then, for fixed u ∈ X , any v ∈ R m , and all k ≥ 2 , the gap estimate G k
= U k − L k satisfies
[Equation]
Proof.
Using G k
= U k − L k , U k from Eq.
(10), and L k from Eq.
(17), we have
[Equation]
Therefore
[Equation]
the difference in scaled gap between successive iterations is
[Equation]
Based on the above expression
[Equation]
to prove the lemma
[Equation]
it suffices to bound the expectation of
[Equation] [Equation]
First [Equation] we take expectations on both sides of the inequality in Proposition 6.3 by invoking E [ · ] = 574 E [ E [ · |F k − 1 ]] as before
[Equation]
where F k − 1 denotes the natural filtration.
By using the fact that x k − 1 isupdated only at coordinate j k (as stated in Proposition 6.2 (b))
[Equation]
we observe the following for the termfrom the right hand side of Proposition 6.3.
[Equation]
Therefore
[Equation]
we have from Proposition 6.3 and scaling Eq.
(26) by − na k that
[Equation] [Equation]
We now bound the expectation of the term involving differences of ψ k by taking expectations of bothsides of Proposition 6.1.
[Equation]
By taking expectations on both sides of Eq.
(25), we have
[Equation]
Combining Eq.
(27), Eq.
(28), and Eq.
(29) then gives
[Equation]
Recall that by the assumption in the statement of the lemma,
[Equation]
Plugging into Eq.
(30) and rearranging, we have
[Equation]
To complete the proof, we need to bound the terms from the first two lines on the right-hand side of
Eq.
First, observe that, by the coordinate update of x k and Young’s inequality, ∀ β > 0 ,
[Equation]
By the same token, ∀ γ > 0 ,
[Equation]
[Equation]
Recalling that, by the choice of step sizes, ( n − 1) a k ≤ na k − 1 and a k ≤ √ A
[Equation]
2 n , we can verify thatfor β
= 2( n − 1) a k and γ = 2 na k − 1 , the following inequalities hold:
[Equation]
Combining Equations (31)–(34),
[Equation]
It remains to combine Eq.
(24), Eq.
(25), and Eq.
Lemma 3.4.
Given a fixed u ∈ X , any v ∈ R m , ¯ y
=
y 0 , and x 1 and y 1 from Algorithm 1, we have
[Equation]
Proof.
Evaluating Eq.
(10) and Eq.
(17) at k = 1 gives
[Equation]
By definition of ψ 1 from Eq.
(9), ϕ 1 from Eq.
(12), and the assignment a 11 = a 1 , we have
[Equation]
where we have used that ¯ y
= y 0 , which holds by assumption.
To complete the proof, it remains tosubtract Eq.
(37) from Eq.
(36) and combine with the last equality.
Theorem 3.5.
[Main Result] Assume that n ≥ 4 .
Given a matrix A ∈ R m × n + , ε > 0 , an arbitraryx 0 ∈ X and ¯ y
= y
= Ax 0 , let x k and A k evolve according to SI-NNLS+ (Algorithm 1) for k ≥ 1 .
For f defined in (2), define x ⋆ ∈ argmin x ≥
f
( x ) .
Then, for all K ≥ 2 , we have
[Equation]
When K ≥ 52 n log n , we have A K ≥ ( K − 52 n log n ) 2 36 n.
If ϕ 0 ( x ⋆ ) ≤ | f
( x ⋆ ) | , then for K ≥ 52 n log n +6 n √ ε , we have E [ f ( (cid:101) x K ) − f ( x ⋆ )] ≤ ε | f
( x ⋆ ) | .
The total cost is O (cid:0) nnz
( A ) (cid:0) log n + 1 √ ε
(cid:1)(cid:1) .
Proof.
Observe that, by the choice of step sizes, n 2 a 2 k − 1 ≤ A k − 2 4 , ∀ k ≥ 3 .
Thus, telescoping thebound in Lemma 3.3 and combining with Lemma 3.4, we have
[Equation]
We first show how to cancel out the inner product term with the negative quadratic terms.
Observethat, ∀ β > 0 ,
[Equation]
where the last line is by Young’s inequality.
In particular, choosing β = 2 a K , we have 12 a K nβ =na 2 K , which is at most A K − 1 4 , by the choice of step sizes in SI-NNLS+.
Thus, since ϕ 0 ( x 1 ) =12 ∥ x 1 − x 0 ∥ 2 Λ , Eq.
(38) simplifies to
[Equation]
Since − 14 E [ ∥ u − x K ∥ 2 Λ ] ≤ 0 , we can ignore it.
Let us now bound n 2 a 12 ∥
y 1 − y
∥ 22 − ϕ
( x 1 ) .
Bydefinition of x 1 and ϕ 1 , we have x 1 = x 0 − a 1 Λ − 1 ( A ⊤ y 0 − 1 ) .
Further, from Eq.
(9) and Eq.
(6),as we have y
=
Ax 1 and y
= Ax 0 , we can simplify the terms to bound as follows.
[Equation]
where the reasoning behind the first inequality follows from the definition of spectral normand that ∥ AΛ − 1 / 2 ∥ 22 = λ max ( Λ − 1 / 2 A ⊤ AΛ − 1 / 2 ) ; the last inequality follows as the matrixΛ − 1 /
A ⊤ AΛ − 1 / 2 has all ones on the main diagonal, and thus its trace is at most n, and sinceit is positive semidefinite, its spectral norm is at most its trace.
As a 1 = 1 √ 2 n 1 . 5 , we conclude thatn 2 a 12 ∥ y 1 − y
∥ 22 − ϕ
( x 1 ) ≤ 0 .
Thus, Eq.
(39) simplifies to
[Equation]
By construction, Gap u , v L ( (cid:101) x K , (cid:101) y K ) ≤ G K ( u , v ) .
Further, by Inequality (5), f ( (cid:101) x K ) − f ( x ⋆ ) +12 ∥
A ( (cid:101) x
K − x ⋆ ) ∥ 2 ≤ Gap u , v L ( (cid:101) x K , (cid:101) y K ) .
Hence, we can conclude from Eq.
(40) that
[Equation] [Equation]
On the other hand, for u = x ⋆ and v = y ⋆ =
Ax ⋆ , Gap u , v L ( (cid:101) x K , (cid:101) y K ) ≥ 0 , and, recalling from
Eq.
(6), Eq.
(8), and Eq.
(9) that
y K
=
A (cid:101) x K , we can also conclude from Eq.
(40) that
[Equation]
By Proposition 2.1 (b), f ( x ⋆ )
= − 12 1 ⊤ x ⋆ = − 12 ∥
Ax ⋆ ∥ 22 .
Using this identity, one can verify that,∀ x ,
[Equation]
Hence, summing Eq.
(41) and Eq.
(42), we also have
[Equation]
Finally, the bound on the rate of growth of A k is provided in Appendix 6.3.
6.3 Omitted Proofs from Section
: Growth of
Scalar Sequences
In this section, we use the properties of { a i } and { a ki } to obtain our claimed rate of growth of A k .
Note that in any iteration k ≥ 2 of Algorithm 1, there are two possible updates to a k , which we nameas follows.
[Equation]
Obtaining a handle on the growth rate of A k requires controlling the number of updates of both typesspecified above.
At a high level, the idea behind obtaining such a bound is that if the algorithm hadonly Type II updates, we would have A k ≥ Ω( k 2 n 2 ) ; we then go on to show that we cannot have morethan 52 n log n Type I updates since those make a k grow too fast.
We formalize this intuition in thefollowing lemmas.
Lemma 6.4.
In Algorithm 1, we have, for k ≥ 2 , that a k +1 ≥ a k and A k
+1 >
A k .
Proof.
Notice that for all k , we have a k > 0 , which implies that A k +1 def =
A k + a k +1 satisfies
A k +1 > A k .
To check the non-decreasing nature of a k , we recall that a k +1 = min (cid:16) na k n − 1 , √
A k 2 n (cid:17) .
In the case that √ A k 2 n ≥ na k n − 1 , we have a k +1 =
na k n − 1 > a k , as claimed.
Consider the other casewith
a k +1 = √ A k 2 n , and suppose, for the sake of contradiction,
that a k +1 < a k .
Chaining thisinequality with the assumed expression for a k +1 , scaling appropriately, and squaring both sides gives
A k < 4 n 2 a 2 k .
Plugging this into A k =
A k − 1 + a k and solving for a k from this quadratic inequality(and further invoking the nonnegativity of a k ), yields a k > 1+ √
[Equation]
16 n
[Equation]
n
[Equation]
> √
[Equation]
However,this contradicts a k = min
[Equation]
Lemma 6.5. Consider the iterations { s k } in which Algorithm 1 performs a Type II update a s k +1 =√
A sk 2 n .
Then we have A s k ≥ k 2 c 1 n
and a s k ≥ k − 1 2 √ c 1 n 2 for c 1 = 36 .
Proof.
We prove this claim by induction.
First, notice that s k ≥ k + 1 for any k .
Recall ourinitialization a 1 =
A 1 = 1 √ 2 n 1 .
By combining this with the monotonicity property stated in
Lemma 6.4, we have a s 1 ≥ a 2 = 1 √ 2 n 2 .
5 ≥ 0 .
By using Lemma 6.4 again, we have, in a similarfashion, that A s 1 ≥ A 2 = 1 √ 2 n 2 . 5 + 1 √ 2 n 1 .
5 ≥ 1 c 1 n 2 , which proves the base case for induction.
Assume that for some k > 1 , we have the induction hypothesis A s k ≥ k 2 c 1 n
and a s k ≥ k − 1 2 √ c 1 n 2 .
Then,combining the monotonicity of A k from Lemma 6.4 with the fact that the algorithm performs a Type
II update on a s k , we have a s k +1 = √
A sk +1 − 1 2 n ≥ √
A sk 2 n ≥ k 2 √ c 1 n 2 .
By again applying monotonicityof A k and the induction hypothesis about a k , we have A s k +
=
A s k +1 − 1 + a s k +1 ≥
A s k
+ a s k +
≥2 k 2 + √ c 1 k 2 c 1 n> ( k + c n
Lemma 6.6.
If at some k th0 iteration of Algorithm 1, we have that
[Equation] [Equation]
then for all k ≥ k 0 , we have that
[Equation] [Equation]
for c 1 = 36 .
Proof.
We prove the claim by induction.
First, the base case is true for k = k 0 by our assumption ona k 0 and A k 0 .
Assume the induction hypothesis a k ≥ k − 1 − k 0 + n 2 √ c 1 nand A k ≥ ( k − k 0 + n ) 2 c 1 nfor k ≥ k 0 .
We now discuss how a k changes with the two types of updates.
If the algorithm performs a Type I update on a k , then, by definition, a k +1 = na k n − 1 .
Now applying theassumed lower bound on a k , we have, when k > k 0 , that
[Equation]
Similarly, given that A k ≥ ( k − k 0 + n ) 2 c 1 n, we have,
[Equation]
If, on the other hand, the algorithm performs a Type II update on a k , then we have
[Equation]
This completes the induction.
As we saw in Lemma 6.6, after the k th0 iteration - starting at which Inequality (45) holds - A k growsfast.
We therefore need to estimate the number of Type I updates before the k th0 iteration.
Lemma 6.7.
There are at most 32 n log n Type I updates (Equation (43) ) performed before the k th0iteration (the first iteration at which Inequality (45) holds).
Proof.
Suppose there are n 1 Type I updates performed by Algorithm 1 before the k th0 iteration, when
Inequality (45) starts to hold.
Further, by Lemma 6.4, a k is monotonically increasing (for both types of updates).
Then, when considering Type I updates (Equation (43)), we have a k 0 ≥ (cid:16) nn − 1 (cid:17) n 1 a
[Equation]
(cid:16) nn − 1 (cid:17) n 1 · 1 √ 2 n 2 . 5 .
In order for a k 0 > n − 1 12 n 2 , we only need to have n 1 > log nn − 1 (cid:16) √ n ( n − 1) 6 √(cid:17) .
In a similar fashion, combining the monotonicity of A k from Lemma 6.4 with the Type I update rule, we have
[Equation]
So, in order to have A k 0 > 136 per Inequality (45), we only need to have n 1 ≥ log nn − 1 ( n 2 .
5 18 √ 2 ) .
By using the approximation 1 + x ≤ e x and combining the above two bounds, we get as soon asn 1 ≥ 32 n log n , the inequality (45) holds.
Proposition 6.8.
[Rate of change of A k ] When k ≥ 52 n log n , we have A k ≥ ( k − 52 n log n ) 2 36 n.
Proof.
Let there be t 1 Type I updates and t 2 Type II updates before the first iteration at which
Inequality (45) holds, and let us call this iteration k 0 .
By the result of Lemma 6.7, we havet 1 ≤ 32 n log n .
By the result of Lemma 6.5, we must have A k 0 ≥ t 22 c 1 n 2 and a k 0 ≥ t 2 − 1 2 √ c 1 n 2 .
Tomeet the requirement in Inequality (45) then, we can see that t 2 ≤ n .
Therefore, k 0 = t 1 + t 2 ≤32 n log n + n ≤ 52 n log n .
Having reached the k th0 iteration, the result of Lemma 6.6 applies, and wehave A k ≥ ( k − k 0 ) 2 c 1 n 2 .
6.4 Omitted Proofs from Section 4: Restart Strategy
To establish local error bounds, we start with the observation that (P) is equivalent to a linearcomplementarity problem.
Proposition 6.9.
Problem (P) is equivalent to the following linear complementarity problem, denotedby LCP( M , q ) .
[Equation]
where Λ − 1 M =
A ⊤ A and q = − Λ − 1 1 .
Proof.
Observe first that, as Λ − 1 is a diagonal matrix with positive elements on the diagonal, thestated linear complementarity problem is equivalent to
[Equation]
By Proposition 2.1, these conditions hold for any solution of (P).
In the opposite direction, supposethat the conditions from Eq.
(48) hold for some x .
Then applying these conditions for any u ≥ 0gives
[Equation]
But ⟨∇ f ( x ) , u − x ⟩ ≥ 0 is the first-order optimality condition for (P), and so x solves (P).
For r ( x ) =
∥ R ( x ) ∥ Λ , a quantity termed natural residual [30], local error bound is obtained as acorollary of the following theorem.
Theorem 6.10 ([30], Theorem 2.1) .
Let M ∈ R n × n be such that LCP( M , 0 ) has 0 as its uniquesolution.
Then there exists µ > 0 such that for each x ∈ R n , we have r ( x ) ≥ µ ∥ x − x ⋆ ∥ , where x ⋆is a solution to LCP( M , q ) that is closest to x under the norm ∥ · ∥ .
Theorem 6.10 applies to our problem due to the nonnegativity (and nondegeneracy) of A andchoosing ∥ ·
∥ = ∥ · ∥ Λ .
By arguing that Theorem 3.5 provides an upper bound on r ( (cid:101) x K ) , inexpectation, we then obtain our final result below.
Proposition 6.11.
For any x ∈ R n + , r ( x ) ≤ (cid:112) 2 n ( f ( x ) − f ( x ⋆ )) , where x ⋆ ∈ argmin u ∈ R n + f ( u ) .
Proof.
Given x ∈
R n + , consider ˆ x defined as ˆ x j ⋆ = x
j ⋆ − R j ⋆ ( x ) , where j ⋆ =argmax 1 ≤ j ≤ n | R j ( x ) | · ∥ A : j ∥ 2 , and ˆ x j = x j for j ̸ = j ⋆ .
Then observing that
[Equation]
and combining with f (ˆ x ) ≥ f ( x ⋆ ) , r ( x ) =
∥ R ( x ) ∥ Λ , the claimed bound follows after a simplerearrangement.
Theorem 4.1.
Given an error parameter ε > 0 and x 0 = 0 , consider the following algorithm A :
A : SI-NNLS+ with Restarts
Initialize: k = 1 .
Initialize Lazy SI-NNLS+ at x k − 1 .
Run Lazy SI-NNLS+ until the output (cid:101) x
kK satisfies r ( (cid:101) x kK ) ≤ 12 r ( x k − 1 ) .
Restart Lazy SI-NNLS+ initializing at x k = (cid:101) x kK .
Increment k.
Repeat until r ( (cid:101) x kK ) ≤ ε.
Then, the expected number of arithmetic operations of A is O (cid:0) nnz
( A ) (cid:0) log n + √ nµ (cid:1) log (cid:0) r ( x 0 ) ε (cid:1)(cid:1) .
As a consequence, given ¯ ε > 0 , the total expected number of arithmetic operations until a point withf
− f ( x ⋆ ≤ ¯ ε | f ( x ⋆ ) | can be constructed by A is O (cid:16) nnz( ) (cid:16) log n + √ nµ (cid:17) log (cid:16) nµ ¯ ε (cid:17)(cid:17) .
Proof.
Because each restart halves the natural residual r ( x ) , it is immediate that the total numberof restarts until r ( (cid:101) x kK ) ≤ ε is bounded by log( r ( x 0 ) ε ) .
Thus, to prove the first (and main) part ofthe theorem, we only need to bound the number of iterations (and the overall number of arithmeticoperations) of (Lazy) SI-NNLS+ in expectation.
Hence, in the following, we only consider one run of SI-NNLS+ until the natural residual is halved.
To keep the notation simple, we let x 0 denote theinitial point of SI-NNLS+ and (cid:101) x
k denote the output of SI-NNLS+ at iteration k.
If r ( x 0 ) = 0 , Ahalts immediately and the bound on the number of iterations holds trivially, so assume r ( x 0 ) > 0 .
Using Theorem 3.5, we have that ∀ k ≥ 2 ,
[Equation] [Equation]
As r 2 ( · ) is nonnegative, we can use Markov’s inequality to bound the total number of iterations Kuntil r ( (cid:101) x K ) ≤ r ( x 0 ) 2 .
In particular, using Eq.
(49), we get by Markov’s inequality
that Pr[ K > k ] ≤
Pr[ r 2 ( (cid:101) x k ) ≥ r 2 ( x 0 )] ≤ 4 n µ 2 A k .
As K is nonnegative, we can estimate its expectation using
E [ K ] =
[Equation]
where in the last inequality we use the rate of A k from Proposition 6.8.
In the lazy implementation of SI-NNLS+, as argued in Appendix 7, the expected cost of an iterationis nnz( A ) n , which leads to the claimed bound on the number of arithmetic operations until r ( x ) ≤ ε.
By using that r ( x 0 )
[Equation]
n
[Equation]
1)+ n +1 µ (cid:1) r 2 ( (cid:101) x K 1 ) (argued below), the bound on the number of iterations until f (cid:0)(cid:101) x K 1 − R ( (cid:101) x K 1 ) (cid:1)
−f ( x ⋆ ) ≤ ¯ ε | f ( x ⋆ ) | have
[Equation]
| and by setting K 1 = 12 log 2 2 n (cid:0) ( n − 1)+ n +1 µ (cid:1) ¯ ε , we have this bound.
Finally, it remains to argue that f (cid:0)(cid:101) x K 1 − R ( (cid:101) x K 1 ) (cid:1) − f ( x ⋆ ) ≤ (cid:0) ( n − 1) + n +1 µ (cid:1) r
( (cid:101) x K 1 ) .
Observe that the definition of R ( x ) is equivalent to x − ¯ x , where
[Equation]
By the first-order optimality of ¯ x based on the equivalent definition of R ( x ) above, we have ⟨∇ f ( x )+Λ (¯ x − x ) , x ⋆ − ¯ x ⟩ ≥ 0 .
Rearranging, and using the definition of convexity of f, we have
[Equation]
where in the last inequality we have used the error bound from Theorem 6.10.
Implementation Version of SI-NNLS+
Since Algorithm 1 explicitly updates (cid:101) x k and (cid:101)
y k
(of lengths n and m respectively), the per iterationcost is O ( m + n ) , which is unnecessarily high when the matrix A is sparse.
In this section, we showthat by using a lazy update strategy, we can efficiently implement Algorithm 1 with overall complexityindependent of the ambient dimension.
To attain this result, we maintain implicit representations for(cid:101) x k , y k , and ¯ y k by introducing two auxiliary variables that are amenable to efficient updates.
Efficiently Updating the Primal Variable.
In Lemma 7.1, we show that we can work with animplicit representation of (cid:101) x k by introducing r k .
Lemma 7.1.
For { (cid:101) x k } defined in Eq.
(8) (and simplified in Algorithm 1), we have, for k ≥ 1 ,
[Equation]
where x k evolves as per Algorithm 1, r 1 =
and, when k ≥ 1 , r k = r k − 1 + (( n − 1) a k −
A k − 1 )( x k − x k − 1 ) .
Proof.
We prove the lemma by induction.
Using the facts that x 0 = 0 , x 1 = (cid:101) x 1 , s 1 = 0 ,
a 1 = A 1 ,and A
= 0 , we have
[Equation]
Assume for certain k ≥ 2 , that Eq.
(50) holds for k − 1 .
Then, using the recursion of ˜ x k in Algorithm1, we have that for k ≥ 3 ,
[Equation]
as required.
The expression for r k in Lemma 7.1 shows that it can be updated at cost O (1) as x k differs fromx k − 1 only at one coordinate.
Therefore, by Eq.
(50) we need not compute ˜ x k in all iterations and caninstead maintain r k .
Along the same lines, we give an efficient implementation strategy for y k
and¯ y k in the following discussion.
Efficiently Updating the Dual Variable.
We now show how to update the dual variable efficiently.
Lemma 7.2.
Consider { y k } and { x k } evolving as per Algorithm 1.
Then, for k = 1 , we havey 1 =
Ax 1 ; for k ≥ 2 , we have
[Equation]
Proof.
The proof is directly from the definition of y k in Algorithm 1.
Lemma 7.3.
Consider { y k } and { x k } evolving as per Algorithm 1.
Then for all k ≥ 1 , we have
[Equation]
(54) where s 1 = 0 and s k
= s k − 1 + (( n − 1) a k − A k − 1 )
A ( x k − x k − 1 ) when k ≥ 2 .
Proof.
We prove the lemma by induction.
For the base case of k = 1 , we have, by the choice ofs 1 = 0 , that y
=
Ax
=
Ax 1 + 1 A 1 s 1 .
Then for some k ≥ 2 , assume Eq.
(54) holds for k − 1 ,then we have,
[Equation]
where the first step is by Lemma 7.2, second step is by the induction hypothesis, third step is byadding and subtracting A k − 1 Ax k , fourth step is by rearranging terms appropriately, and the finalstep uses the recursive definition of s k stated in the lemma.
Dividing throughout by A k then finishesthe proof.
Algorithm
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation] [Equation]
[Equation] [Equation]
Sample j k uniformly at random from { 1 , 2 , . . . , n } 6:
if k = 1 then 7: ¯ y 0 = q8: else
if k = 2 then 9: ¯ y 1 = q 1 + a 1 a 2 t10: else if k ≥ 3 then 11: ¯ y k − 1 = q k − 1 + 1 A k −(cid:16)
1 − a 2 k − 1 a k A k − 2 (cid:17) s k − 1 + ( n − 1) a 2 k − 1 a k A k − 2 t k −12: end if 13: p k,i = (cid:26) p k − 1 ,i , i ̸
=
j k p k − 1 ,i + na k (cid:0) A T : i ¯ y k − 1 − 1 (cid:1) , i =
j k .
14: x k,i = (cid:40) x k − 1 ,i , i ̸
=
j k max (cid:8) 0 , min (cid:8) x 0 ,i −
∥ A : i ∥ 2 · p k,i , 1 ∥
A : i ∥ 2 (cid:9)(cid:9) , i =
j k 15: t k =
A ( x k − x k − 1 ) 16:
if k ≥ 2 then 17: r k = r k − 1 + (( n − 1) a k − A k − 1 )( x k − x k − 1 ) 18: s k
= s k −
+ (( n − 1) a k − A k − 1 ) t k 19: end if 20: q k = q k − 1 + t k 21:
A k +1 =
A k + a k + 22: a k +2 = min { na k +1 n − 1 , √ A k + 2 n } 23: end for 24: return x K + 1 A K r K
Lemma 7.4.
Consider { x k } , { y k } , and { y k } evolving as per Algorithm 1.
Then we have that
[Equation]
56) and
[Equation]
Proof.
From the definition of ¯ y k , the initializations for x 0 , y 0 , and y 0 , and Lemma 7.2, we have
[Equation]
For k ≥ 2 , by Lemma 7.2, we have
[Equation]
As a result,
[Equation]
So for k ≥ 2 , it follows that
[Equation]
where the first step is by the definition of y k in Algorithm 1, the second step is by Eq.
(58), the thirdstep is by rearranging, and the final step is by Lemma 7.3.
Based on the above lemmas, we give our efficient lazy implementation version of Algorithm 1 in
Algorithm 2.
In Algorithm 2, we also introduce other auxiliary variables p k , q k and t k .
Basedon Lemmas 7.1-7.4, it is easy to verify the equivalence between Algorithms 1 and 2.
With thisimplementation, by updating only the dual coordinates corresponding to the nonzero coordinates ofthe selected column of A , the per-iteration cost is proportional to the number of nonzero elements ofthe selected row in the iteration.
As a result, the overall complexity result will depend only on thenumber of nonzero elements of A .