L OCAL S EARCH A LGORITHMS FOR R ANKC ONSTRAINED C ONVEX O PTIMIZATION
A BSTRACT
We propose greedy and local search algorithms for rank-constrained convex optimization, namely solving min rank( A ) ≤ r ∗ R
( A )
given a convex function R : R m × n →
R and a parameter r ∗ .
These algorithms consist of repeating two steps: (a) adding a new rank-1 matrix to A and (b) enforcing the rank constraint on A .
We reﬁne and improve the theoretical analysis of Shalev-Shwartz et al.
and show that if the rank-restricted condition number of R is κ , a solution A with rank O ( r ∗ · min { κ log R ( 0 ) − R ( A ∗ ) (cid:15) , κ 2 } ) and R ( A ) ≤ R ( A ∗ ) + (cid:15) can be recovered, where A ∗ is the optimal solution.
This signiﬁcantly generalizes associated results on sparse convex optimization, as well as rank-constrained convex optimization for smooth functions.
We then introduce new practical variants of these algorithms that have superior runtime and recover better solutions in practice.
We demonstrate the versatility of these methods on a wide range of applications involving matrix completion and robust principal component analysis
I NTRODUCTION
Given a real-valued convex function R : R m × n → R on real matrices and a parameter r ∗ ∈ N , the rank-constrained convex optimization problem consists of ﬁnding a matrix A ∈ R m × n that minimizes R ( A ) among all matrices of rank at most r ∗ :
[Equation]
Even though R is convex, the rank constraint makes this problem non-convex.
Furthermore, it is known that this problem is NP-hard and even hard to approximate (Natarajan (1995); Foster et al.
In this work, we propose efﬁcient greedy and local search algorithms for this problem.
Our contribution is twofold:
We provide theoretical analyses that bound the rank and objective value of the solutions returned by the two algorithms in terms of the rank-restricted condition number , which is the natural generalization of the condition number for low-rank subspaces.
The results are signiﬁcantly stronger than previous known bounds for this problem.
We experimentally demonstrate that, after careful performance adjustments, the proposed general-purpose greedy and local search algorithms have superior performance to other methods, even for some of those that are tailored to a particular problem.
Thus, these algorithms can be considered as a general tool for rank-constrained convex optimization and a viable alternative to methods that use convex relaxations or alternating minimization
The rank-restricted condition number Similarly to the work in sparse convex optimization, a restricted condition number quantity has been introduced as a reasonable assumption on R .
If we let ρ + r be the maximum smoothness bound and ρ − r be the minimum strong convexity bound only along rank- r directions of R (these are called rank-restricted smoothness and strong convexity respectively), the rank-restricted condition number is deﬁned as κ r = ρ + r ρ − r .
If this quantity is bounded, one can efﬁciently ﬁnd a solution A with R ( A ) ≤ R ( A ∗ )+ (cid:15) and rank r =
O ( r ∗ · κ r + r ∗ R ( 0 ) (cid:15) ) using a greedy algorithm (Shalev-Shwartz et al.
However, this is not an ideal bound since the rank scales linearly with R ( 0 ) (cid:15) , which can be particularly high in practice.
Inspired by the analogous literature on sparse convex optimization by Natarajan (1995); Shalev-Shwartz et al.
; Zhang (2011); Jain et al.
(2014) and more recently Axiotis & Sviridenko (2020), one would hope to achieve a logarithmic dependence or no dependence at all on R ( 0 ) (cid:15) .
In this paper we achieve this goal by providing an improved analysis showing that the greedy algorithm of Shalev-Shwartz et al.
(2011) in fact returns a matrix of rank of r = O ( r ∗ · κ r + r ∗ log R ( 0 ) (cid:15) ) .
We also provide a new local search algorithm together with an analysis guaranteeing a rank of r = O ( r ∗ · κ 2 r + r ∗ ) .
Apart from signiﬁcantly improving upon previous work on rank-restricted convex optimization, these results directly generalize a lot of work in sparse convex optimization, e.g. Natarajan (1995); Shalev-Shwartz et al.
(2010); Jain et al.
Our algorithms and theorem statements can be found in Section 2.
Runtime improvements Even though the rank bound guaranteed by our theoretical analyses is adequate, the algorithm runtimes leave much to be desired.
In particular, both the greedy algorithm of Shalev-Shwartz et al.
and our local search algorithm have to solve an optimization problem in each iteration in order to ﬁnd the best possible linear combination of features added so far.
Even for the case that R ( A ) = 1(cid:80) ( i,j ) ∈ Ω ( M − A ) 2 ij , this requires solving a least squares problem on | Ω | examples and r 2 variables.
For practical implementations of these algorithms, we circumvent this issue by solving a related optimization problem that is usually much smaller.
This instead requires solving n least squares problems with total number of examples | Ω | , each on r variables.
This not only reduces the size of the problem by a factor of r , but also allows for a straightforward distributed implementation.
Interestingly, our theoretical analyses still hold for these variants.
We propose an additional heuristic that reduces the runtime even more drastically, which is to only run a few (less than 10) iterations of the algorithm used for solving the inner optimization problem.
Experimental results show that this modiﬁcation not only does not signiﬁcantly worsen results, but for machine learning applications also acts as a regularization method that can dramatically improve generalization.
These matters, as well as additional improvements for making the local search algorithm more practical, are addressed in Section 2.3.
Roadmap
In Section 2, we provide the descriptions and theoretical results for the algorithms used, along with several modiﬁcations to boost performance.
In Section 3, we evaluate the proposed greedy and local search algorithms on optimization problems like robust PCA.
Then, in Sectionwe evaluate their generalization performance in machine learning problems like matrix completion.
A LGORITHMS & T HEORETICAL G UARANTEES
In Sections 2.1 and 2.2 we state and provide theoretical performance guarantees for the basic greedy and local search algorithms respectively.
Then in Section 2.3 we state the algorithmic adjustments that we propose in order to make the algorithms efﬁcient in terms of runtime and generalization performance.
A discussion regarding the tightness of the theoretical analysis is deferred to Appendix A.4.
When the dimension is clear from context, we will denote the all-ones vector by 1 , and the vector that is 0 everywhere and 1 at position i by 1 i .
Given a matrix A , we denote by im( A ) its column span.
One notion that we will ﬁnd useful is that of singular value thresholding .
More speciﬁcally, given a rank- k matrix A ∈ R m × n with SVD k
(cid:80) i [-=
-]=
[Equation] σ [Equation] u i v i (cid:62) such that σ 1 ≥ · · · ≥ σ k , as well as an integer parameter r ≥ 1 , we deﬁne H r ( A )
=
r (cid:80) i =
σ i u i v i (cid:62) to be the operator that truncates to the r highest singular values of A .
2. G REEDY
Algorithm 1 (Greedy) was ﬁrst introduced in Shalev-Shwartz et al.
(2011) as the GECO algorithm.
It works by iteratively adding a rank-1 matrix to the current solution.
This matrix is chosen as the rank-1 matrix that best approximates the gradient, i.e. the pair of singular vectors corresponding to the maximum singular value of the gradient.
In each iteration, an additional procedure is run to optimize the combination of previously chosen singular vectors.
In Shalev-Shwartz et al.
(2011) guarantee on the rank of the solution returned by the algorithm is r ∗ κ r + r ∗ R ( 0 ) (cid:15) .
The main bottleneck in order to improve on the R ( 0 ) (cid:15) factor is the fact that the analysis is done in terms of the squared nuclear norm of the optimal solution.
As the worst-case discrepancy between the squared nuclear norm and the rank is R ( 0 ) /(cid:15) , their bounds inherit this factor.
Our analysis works directly with the rank, in the spirit of sparse optimization results (e.g. Shalev-Shwartz et al.
(2011); Jain et al.
(2014); Axiotis & Sviridenko (2020)).
A challenge compared to these works is the need for a suitable notion of “intersection” between two sets of vectors.
The main technical contribution of this work is to show that the orthogonal projection of one set of vectors into the span of the other is such a notion, and, based on this, to deﬁne a decomposition of the optimal solution that is used in the analysis.
Algorithm
1: procedure G REEDY ( r ∈ N : target rank) 2: function to be minimized R : R m × n → R 3:
U ∈ R m ×(cid:46)
Initially rank is zero 4:
V ∈ R n ×5: for t = 0 . .
r − 1 do 6: σuv (cid:62) ← H 1 ( ∇ R ( UV (cid:62) ))
(cid:46)
Max singular value σ and corresp.
singular vectors u, v 7:
U ← ( U u ) (cid:46) Append new vectors as columns 8:
V ← ( V v ) 9:
U, V ← O PTIMIZE ( U, V ) 10: return UV (cid:62) 11: procedure O PTIMIZE ( U ∈ R m × r , V ∈ R n × r ) 12:
X ← arg min X ∈ R r × r R ( UXV (cid:62) ) 13: return UX, V
Theorem 2.1 (Algorithm 1 (greedy) analysis) .
Let A ∗ be any ﬁxed optimal solution of (1) for some function R and rank bound r ∗ , and let (cid:15) > 0 be an error parameter.
For any integer r ≥ 2 r ∗ · κ r + r ∗ log R ( 0 ) − R ( A ∗ ) (cid:15) , if we let A = G REEDY ( r ) be the solution returned by Algorithm 1, then
R ( A ) ≤ R ( A ∗ ) + (cid:15) .
The number of iterations is r .
The proof of Theorem 2.1 can be found in Appendix A.2.
2. L OCAL S EARCH
One drawback of Algorithm 1 is that it increases the rank in each iteration.
Algorithm 2 is a modiﬁcation of Algorithm 1, in which the rank is truncated in each iteration.
The advantage of Algorithm 2 compared to Algorithm 1 is that it is able to make progress without increasing the rank of A, while Algorithm 1 necessarily increases the rank in each iteration.
More speciﬁcally, because of the greedy nature of Algorithm 1, some rank-1 components that have been added to A might become obsolete or have reduced beneﬁt after a number of iterations.
Algorithm 2 is able to identify such candidates and remove them, thus allowing it to continue making progress.
Theorem 2.2 (Algorithm 2 (local search) analysis) .
Let A ∗ be any ﬁxed optimal solution of (1) for some function R and rank bound r ∗ , and let (cid:15) > 0 be an error parameter.
For any integer r ≥ r ∗ · (1 + 8 κ 2 r + r ∗ ) , if we let A
=
L OCAL S EARCH ( r ) be the solution returned by Algorithm 2, then R ( A ) ≤ R ( A ∗ )
+ (cid:15) .
The number of iterations is O (cid:16) r ∗ κ r + r ∗ log R ( 0 ) − R
( A ∗ ) (cid:15) (cid:17) .
The proof of Theorem 2.2 can be found in Appendix A.3.
1: procedure L OCAL S EARCH ( r ∈ N : target rank) 2: function to be minimized R : R m × n → R 3:
U ← 0 m × r (cid:46) Initialize with all-zero solution 4:
V ← 0 n × r 5: for t = 0 . . .
L − 1 do (cid:46) iterations 6: σuv (cid:62) ← H 1 ( ∇ R ( UV (cid:62) ))
(cid:46)
Max singular value σ and corresp.
singular vectors u, v 7:
U, V ← T RUNCATE ( U, V ) (cid:46) Reduce rank of UV (cid:62) by one 8:
U ← ( U u ) (cid:46) Append new vectors as columns 9:
V ← ( V v ) 10:
U, V ← O PTIMIZE ( U, V ) 11: return UV (cid:62) 12: procedure T RUNCATE ( U ∈ R m × r , V ∈ R n × r ) 13:
U Σ V (cid:62) ← SVD( H r − 1 ( UV (cid:62) ))
(cid:46)
Keep all but minimum singular value 14: return U Σ , V 2.
A LGORITHMIC ADJUSTMENTS
Inner optimization problem
The inner optimization problem that is used in both greedy and local search is:
[Equation]
It essentially ﬁnds the choice of matrices U (cid:48) and V (cid:48) , with columns in the column span of U and V respectively, that minimizes R ( U (cid:48) V (cid:48) (cid:62) ) .
We, however, consider the following problem instead:
[Equation]
Note that the solution recovered from (3) will never have worse objective value than the one recovered from (2), and that nothing in the analysis of the algorithms breaks.
Importantly, (3) can usually be solved much more efﬁciently than (2).
As an example, consider the following objective that appears in matrix completion: R ( A ) =(cid:80) ( i,j ) ∈ Ω ( M − A ) 2 ij for some Ω ⊆ [ m ] × [ n ] .
If we let Π Ω ( · ) be an operator that zeroes out all positions in the matrix that are not in Ω , we have ∇ R ( A )
= − Π Ω
( M − A ) .
The optimality condition of (2) now is U (cid:62) Π Ω ( M − UXV (cid:62) )
V =
and that of (3) is U (cid:62) Π Ω ( M − UV (cid:62) )
= 0 .
The former corresponds to a least squares linear regression problem with | Ω | examples and r 2 variables, while the latter can be decomposed into n independent systems
U
[Equation]
M
[Equation]
j ) , where the variable is V j which is the j -th column of V .
The j -th of these systems corresponds to a least squares linear regression problem with |{ i : ( i, j ) ∈ Ω }| examples and r variables.
Note that the total number of examples in all systems is (cid:80)
[Equation]
|{ i :
[Equation]
i, j ) ∈ Ω }| = | Ω | .
The choice of V here as the variable to be optimized is arbitrary.
In particular, as can be seen in Algorithm 3, in practice we alternate between optimizing U and V in each iteration.
It is worthy of mention that the O PTIMIZE F AST procedure is basically the same as one step of the popular alternating minimization procedure for solving low-rank problems.
As a matter of fact, when our proposed algorithms are viewed from this lens, they can be seen as alternating minimization interleaved with rank-1 insertion and/or removal steps.
Singular value decomposition As modern methods for computing the top entries of a singular value decomposition scale very well even for large sparse matrices (Martinsson et al.
(2011); Szlam et al.
; Tulloch (2014)), the “insertion” step of greedy and local search, in which the top entry of the SVD of the gradient is determined, is quite fast in practice.
However, these methods are not suited for computing the smallest singular values and corresponding singular vectors, a step required for the local search algorithm that we propose.
Therefore, in our practical implementations we opt to perfom the alternative step of directly removing one pair of vectors from the representation
UV (cid:62) .
A simple approach is to go over all r possible removals and pick the one that increases the
1: procedure O PTIMIZE F AST ( U ∈ R m × r , V ∈ R n r , t ∈ N : iteration index of algorithm) 2: if t mod 2 = 0 then 3:
5: else 6: objective by the least amount.
A variation of this approach has been used by Shalev-Shwartz et al.
However, a much faster approach is to just pick the pair of vectors U 1 i , V 1 i that minimizes (cid:107)
U 1 i (cid:107) 2 (cid:107) V 1 i (cid:107) 2 .
This is the approach that we use, as can be seen in Algorithm 4.
i ← arg min i ∈ [ r ] (cid:107)
U 1 i (cid:107) 2 (cid:107) V 1 i (cid:107)3: return (cid:0) U m [1 ,i − 1]
U , [ i 1 ,r (cid:1) , (cid:0) V [ n ] , [1 ,i − 1]
V [ n ] , [ i +1 ,r ] (cid:1) (cid:46)
Remove column i
After the previous discussion, we are ready to state the fast versions of Algorithm 1 and Algorithmthat we use for our experiments.
These are Algorithm 2.3 and Algorithm 5.
Notice that we initialize
Algorithm 5 with the solution of Algorithm 2.3 and we run it until the value of R ( · ) stops decreasing rather than for a ﬁxed number of iterations.
Algorithm 2.3 (Fast Greedy) .
The Fast Greedy algorithm is deﬁned identically as Algorithm 1, with the only difference that it uses the O PTIMIZE F AST routine as opposed to the O PTIMIZE routine.
1: procedure F AST L OCAL S EARCH ( r ∈ N : target rank) 2: function to be minimized R : R m × n → R 3:
U, V ← solution returned by F AST G REEDY ( r ) 4: do 5:
U prev , V prev ← U, V 6: σuv (cid:62) ← H 1 ( ∇ R ( UV (cid:62) ))
(cid:46)
Max singular value σ and corresp.
singular vectors u, v 7:
U, V ← T RUNCATE F AST ( U, V ) (cid:46) Reduce rank of UV (cid:62) by one 8:
U ← ( U u ) (cid:46) Append new vectors as columns 9:
V ← ( V v ) 10:
U, V ← O PTIMIZE F AST ( U, V, t ) 11: while R ( UV (cid:62) ) <
R ( U prev V (cid:62) prev ) 12: return U prev V (cid:62) prev
O PTIMIZATION A PPLICATIONS
An immediate application of the above algorithms is in the problem of low rank matrix recovery .
Given any convex distance measure between matrices d : R m × n × R m × n → R ≥ 0 , the goal is to ﬁnd a low-rank matrix A that matches a target matrix M as well as possible in terms of d : min rank( A ) ≤ r ∗ d ( M, A )
This problem captures a lot of different applications, some of which we go over in the following sections.
3. L OW - RANK APPROXIMATION ON OBSERVED SET
A particular case of interest is when d ( M, A ) is the Frobenius norm of M − A , but only applied to entries belonging to some set Ω .
In other words, d ( M, A ) = 12 (cid:107) Π Ω ( M − A ) (cid:107) 2 F .
We have compared our Fast Greedy and Fast Local Search algorithms with the SoftImpute algorithm of Mazumder et al.
(2010) as implemented by Rubinsteyn & Feldman (2016), on the same experiments as in Mazumder et al.
We have solved the inner optimization problem required by our algorithms by the
LSQR algorithm Paige & Saunders (1982).
More speciﬁcally, M = UV (cid:62) + η ∈ R 100 × 100 , where η is some noise vector.
We let every entry of U, V, η be i.i.d.
normal with mean 0 and the entries of Ω are chosen i.i.d.
uniformly at random over the set [100] × [100] .
The experiments have three parameters: The true rank r ∗ (of UV (cid:62) ), the percentage of observed entries p =
| Ω | / 10 4 , and the signal-to-noise ratio SNR.
We measure the normalized MSE, i.e. (cid:107) Π Ω ( M − A ) (cid:107) 2 F / (cid:107) Π Ω ( M ) (cid:107) 2 F .
The results can be seen in Figure 1, where it is illustrated that Fast Local Search sometimes returns signiﬁcantly more accurate and lower-rank solutions than Fast Greedy, and Fast Greedy generally returns signiﬁcantly more accurate and lower-rank solutions than SoftImpute.
[Figure]
3. R OBUST PRINCIPAL COMPONENT ANALYSIS (RPCA)
The robust PCA paradigm asks one to decompose a given matrix M as L + S , where L is a lowrank matrix and S is a sparse matrix.
This is useful for applications with outliers where directly computing the principal components of M is signiﬁcantly affected by them.
For a comprehensive survey on Robust PCA survey one can look at Bouwmans et al.
The following optimization problem encodes the above-stated requirements:
[Equation]
where (cid:107) X (cid:107) 0 is the sparsity (i.e. number of non-zeros) of X .
As neither the rank constraint or the (cid:96)function are convex, Cand`es et al.
(2011) replaced them by their usual convex relaxations, i.e. the nuclear norm (cid:107) · (cid:107) ∗ and (cid:96) 1 norm respectively.
However, we opt to only relax the (cid:96) 0 function but not the rank constraint, leaving us with the problem:
[Equation]
In order to make the objective differentiable and thus more well-behaved, we further replace the (cid:96)norm by the Huber loss H δ
( x ) =
[Equation]
, thus getting:
[Equation]
L ) ij .
This is a problem on which we can directly apply our algorithms.
We solve the inner optimization problem by applying 10 L-BFGS iterations.
In Figure 2 one can see an example of foreground-background separation from video using robust
PCA.
The video is from the BMC 2012 dataset Vacavant et al.
In this problem, the lowrank part corresponds to the background and the sparse part to the foreground.
We compare three algorithms: Our Fast Greedy algorithm, standard PCA with 1 component (the choice of 1 was picked to get the best outcome), and the standard Principal Component Pursuit (PCP) algorithm (Cand`es et al.
(2011)), as implemented in Lin et al.
(2010), where we tuned the regularization parameter λ to achieve the best result.
We ﬁnd that Fast Greedy has the best performance out of the three algorithms in this sample task.
M ACHINE L EARNING A PPLICATIONS 4. R EGULARIZATION TECHNIQUES
In the previous section we showed that our proposed algorithms bring down different optimization objectives aggressively.
However, in applications where the goal is to obtain a low generalization error, regularization is needed.
We considered two different kinds of regularization.
The ﬁrst method is to run the inner optimization algorithm for less iterations, usually 2-3.
Usually this is straightforward since an iterative method is used.
For example, in the case R ( A ) = 12 (cid:107) Π Ω
( M − A ) (cid:107) 2 F the inner optimization is a least squares linear regression problem that we solve using the LSQR algorithm.
The second one is to add an (cid:96) 2 regularizer to the objective function.
However, this option did not provide a substantial performance boost in our experiments, and so we have not implemented it.
4. M ATRIX C OMPLETION WITH R ANDOM N OISE
In this section we evaluate our algorithms on the task of recovering a low rank matrix UV (cid:62) after observing Π Ω ( UV (cid:62) + η ) , i.e. a fraction of its entries with added noise.
As in Section 3.1, we use the setting of Mazumder et al.
and compare with the SoftImpute method.
The evaluation metric is the normalized MSE, deﬁned as ( (cid:80)
[Equation]
) , where A is the predicted matrix and UV (cid:62) the true low rank matrix.
A few example plots can be seen in Figure 3 and a table of results in Table 1.
We have implemented the Fast Greedy and Fast Local Search algorithms with 3 inner optimization iterations.
In the ﬁrst few iterations there is a spike in the relative MSE of the algorithms that use the O PTIMIZE F AST routine.
We attribute this to the aggressive alternating minimization steps of this procedure and conjecture that adding a regularization term to the objective might smoothen the spike.
However, the Fast Local Search algorithm still gives the best overall performance in terms of how well it approximates the true low rank matrix UV (cid:62) , and in particular with a very small rank—practically the same as the true underlying rank.
[Figure]
Algorithm random errorrandom errorrandom error
SoftImpute (Mazumder et al. (2010)) 0 .
Fast Greedy (Algorithm [Table]
[Table]
4. R ECOMMENDER S YSTEMS
In this section we compare our algorithms on the task of movie recommendation on the Movielens datasets Harper & Konstan (2015).
In order to evaluate the algorithms, we perform random 80%- 20% train-test splits that are the same for all algorithms and measure the mean RMSE in the test set.
If we let Ω ⊆ [ m ] × [ n ] be the set of user-movie pairs in the training set, we assume that the true user-movie matrix is low rank, and thus pose (1) with R ( A )
= 12 (cid:107)
Π Ω ( M − A ) (cid:107) 2 F .
We make the following slight modiﬁcation in order to take into account the range of the ratings [1 , 5] : We clip the entries of A between 1 and 5 when computing ∇ R ( A ) in Algorithm 2.3 and Algorithm 5.
In other words, instead of Π Ω ( A − M )
we compute the gradient as Π Ω (clip( A, 1 , 5) − M ) .
This is similar to replacing our objective by a Huber loss, with the difference that we only do so in the steps that we mentioned and not the inner optimization step, mainly for runtime efﬁciency reasons.
The results can be seen in Table 2.
We do not compare with Fast Local Search, as we found that it only provides an advantage for small ranks ( < 30 ), and otherwise matches Fast Greedy.
For the inner optimization steps we have used the LSQR algorithm with 2 iterations in the 100K and 1M datasets, and with 3 iterations in the 10M dataset.
Note that even though the SVD algorithm by Koren et al.
(2009) as implemented by Hug (2020) (with no user/movie bias terms) is a highly tuned algorithm for recommender systems that was one of the top solutions in the famous Netﬂix prize, it has comparable performance to our general-purpose Algorithm 2.3.
Finally, Table 3 demonstrates the speedup achieved by our algorithms over the basic greedy implementation.
It should be noted that the speedup compared to the basic greedy of Shalev-Shwartz et al.
(2011) (Algorithm 1) is larger as rank increases, since the fast algorithms scale linearly with rank, but the basic greedy scales quadratically.
[Table]
[Table]
and tested on an Intel Skylake CPU with 16 vCPUs.
It is important to note that our goal here is not to be competitive with the best known algorithms for matrix completion, but rather to propose a general yet practically applicable method for rankconstrained convex optimization.
For a recent survey on the best performing algorithms in the
Movielens datasets see Rendle et al.
It should be noted that a lot of these algorithms have signiﬁcant performance boost compared to our methods because they use additional features (meta information about each user, movie, timestamp of a rating, etc.) or stronger models (user/movie biases, ”implicit” ratings).
A runtime comparison with these recent approches is an interesting avenue for future work.
As a rule of thumb, however, Fast Greedy has roughly the same runtime as
SVD (Koren et al.
(2009)) in each iteration, i.e. O ( | Ω
| r ) , where Ω is the set of observable elements and r is the rank.
As some better performing approaches have been reported to be much slower than SVD (e.g. SVD++ is reported to be 50-100x slower than SVD in the Movielens 100K and 1M datasets (Hug (2020)), this might also suggest a runtime advantage of our approach compared to some better performing methods.
C ONCLUSIONS
We presented simple algorithms with strong theoretical guarantees for optimizing a convex function under a rank constraint.
Although the basic versions of these algorithms have appeared before, through a series of careful runtime, optimization, and generalization performance improvements that we introduced, we have managed to reshape the performance of these algorithms in all fronts.
Via our experimental validation on a host of practical problems such as low-rank matrix recovery with missing data, robust principal component analysis, and recommender systems, we have shown that the performance in terms of the solution quality matches or exceeds other widely used and even specialized solutions, thus making the argument that our Fast Greedy and Fast Local Search routines can be regarded as strong and practical general purpose tools for rank-constrained convex optimization.
Interesting directions for further research include the exploration of different kinds of regularization and tuning for machine learning applications, as well as a competitive implementation and extensive runtime comparison of our algorithms.
A A PPENDIX A. P RELIMINARIES AND N OTATION
Given an positive integer k we denote [ k ] = { 1 , 2 , . . .
k } .
Given a matrix A we denote by (cid:107) A (cid:107) F its Frobenius norm i e. the (cid:96) 2 norm of the entries of A (or equivalently of the singular values of A ).
The following lemma is a simple corollary of the deﬁnition of the Frobenius norm:
Lemma A.1.
Given two matrices A ∈ R m × n , B ∈ R m × n , we have (cid:107) A + B (cid:107) 2 F ≤ 2 (cid:0) (cid:107) A (cid:107) 2 F + (cid:107) B (cid:107) 2 F (cid:1) .
Proof.
(cid:107) A +
[Equation]
Deﬁnition A.2 (Rank-restricted smoothness, strong convexity, condition number) .
Given a convex function R ∈ R m × n → R and an integer parameter r , the rank-restricted smoothness of R at rank r is the minimum constant ρ + r ≥ 0 such that for any two matrices A ∈ R m × n , B ∈ R m × n such that rank( A − B ) ≤ r , we have
[Equation]
Similarly, the rank-restricted strong convexity of R at rank r is the maximum constant ρ − r ≥ 0 such that for any two matrices A ∈ R m × n , B ∈ R m × n such that rank( A − B ) ≤ r , we have
[Equation]
Given that ρ + r , ρ − r exist and are nonzero, the rank-restricted condition number of R at rank r is then deﬁned as
[Equation]
Note that ρ + r is increasing and ρ − r is decreasing in r .
Therefore, even though our bounds are proven in terms of the constants ρ +1 ρ − r
and ρ +2 ρ − r , these quantities are always at most ρ + r ρ − r = κ r as long as r ≥ 2 , and so they directly imply the same bounds in terms of the constant κ r .
Deﬁnition A.3 (Spectral norm) .
Given a matrix A ∈ R m × n , we denote its spectral norm by (cid:107) A (cid:107) 2 .
The spectral norm is deﬁned as
[Equation]
Deﬁnition A.4 (Singular value thresholding operator) .
Given a matrix A ∈ R m × n of rank k , a singular value decomposition A = U Σ V (cid:62) such that Σ 11 ≥ Σ 22 ≥ · · · ≥ Σ kk , and an integer 1 ≤ r ≤ k , we deﬁne H r ( A ) =
U Σ (cid:48) V (cid:62) , here Σ (cid:48) is a diagonal matrix with
[Equation]
if i
[Equation]
otherwise
In other words, H r ( · ) is an operator that eliminates all but the top r highest singular values of a matrix.
Lemma A.5 (Weyl’s inequality) .
For any matrix A and integer i ≥ 1 , let σ i ( A ) be the i -th largest singular value of A or
if i > rank( A ) .
Then, for any two matrices A, B and integers i ≥ 1 , j ≥ 1 :
[Equation]
A proof of the previous fact can be found e.g. in Fisk (1996).
Lemma A.6 ( H r ( · ) optimization problem) .
Let A ∈ R m × n be a rank- k matrix and r ∈ [ k ] be an integer parameter.
Then M =
λ H r
( A ) is an optimal solution to the following optimization problem:
[Equation]
Proof.
Let U Σ V (cid:62)
= (cid:80)
[Equation]
i V (cid:62) i be a singular value decomposition of A .
We note that (6) is equivalent to
[Equation]
Now, note that f ( 1 λ H
r
( A ))
= (cid:107)
A − H r ( A ) (cid:107) 2 F = k (cid:80) i =
r +1 Σ 2 ii .
On the other hand, by applying Weyl’s inequality (Lemma A.5) for j = r + 1 ,
[Equation]
where the last equality follows from the fact that rank( A ) =
k and rank( M ) ≤ r .
Therefore,
M = 1 λ H r ( A ) minimizes (7) and thus maximizes (6).
A. P ROOF OF T HEOREM 2.1 ( GREEDY )
We will start with the following simple lemma about the Frobenius norm of a sum of matrices with orthogonal columns or rows:
Lemma A.7.
Let U ∈ R m × r , V ∈ R n × r , X ∈ R m × r , Y ∈ R n × r be such that the columns of U are orthogonal to the columns of X or the columns of V are orthogonal to the columns of Y .
Then (cid:107) UV (cid:62) +
[Equation]
Proof.
If the columns of U are orthogonal to those of X , then U (cid:62) X =
and if the columns of V are orthogonal to those of Y , then Y (cid:62) V = 0 .
Therefore in any case (cid:104) UV (cid:62) , XY (cid:62) (cid:105) =
Tr( V U (cid:62) XY (cid:62) ) =
Tr( U (cid:62) XY (cid:62) V ) = 0 , implying (cid:107) UV (cid:62) + XY (cid:62) (cid:107) 2 F = (cid:107) UV (cid:62) (cid:107) 2 F + (cid:107) XY (cid:62) (cid:107) 2 F + 2 (cid:104) UV (cid:62) , XY (cid:62) (cid:105) =
(cid:107) UV (cid:62) (cid:107) 2 F + (cid:107) XY (cid:62) (cid:107) 2 F
Additionally, we have the following lemma regarding the optimality conditions of (2):
Lemma A.8.
Let A = UXV (cid:62)
where U ∈ R m × r , X ∈ R r × r , and V ∈ R n × r , such that X is the optimal solution to (2)
Then for any u ∈ im( U ) and v ∈ im( V ) we have that (cid:104)∇ R ( A ) , uv (cid:62) (cid:105) = 0 .
Proof.
By the optimality condition of 2, we have that
[Equation]
Now, for any u =
Ux and v =
V y
we have
[Equation]
We are now ready for the proof of Theorem 2.1.
Proof.
Let A t − 1 be the current solution UV (cid:62) before iteration t − 1 ≥ 0 .
Let u ∈ R m and v ∈ R m be left and right singular vectors of matrix ∇ R ( A ) , i.e. unit vectors maximizing |(cid:104)∇ R ( A ) , uv (cid:62) (cid:105)| .
Let
B
[Equation]
By smoothness we have
[Equation]
where (cid:107) · (cid:107) 2 is the spectral norm (i.e. maximum magnitude of a singular value).
On the other hand, by strong convexity and noting that
[Equation]
Let A t − 1 = UV (cid:62) and A ∗ =
U ∗ V ∗(cid:62) .
We let Π im( U ) =
U ( U (cid:62) U ) + U (cid:62) and Π im( V ) =
V ( V (cid:62) V ) + V (cid:62) denote the orthogonal projections onto the images of U and V respectively.
We now write
[Equation]
where U 1 = Π im( U ) U ∗ is a matrix where every column of U ∗ is replaced by its projection on im( U ) and U 2 =
U ∗ − U 1 and similarly V
= Π im( V ) V ∗ is a matrix where every column of V ∗ is replaced by its projection on im( V ) and V
=
V ∗ − V 1 .
By setting U (cid:48)
= ( − U | U 1 ) and V (cid:48) =
( V | V 1 ) we can write
[Equation]
(cid:48) V (cid:48)(cid:62) + U 1 V 2 (cid:62) + U 2 V ∗(cid:62) where im( U (cid:48) )
=
im( U ) and im( V (cid:48) )
=
im( V ) .
Also, note that rank( U 1 V 2 (cid:62) ) ≤ rank( V 2 ) ≤ rank( V ∗ )
= rank( A ∗ ) ≤ r ∗ and similarly rank( U 2 V ∗(cid:62) )
≤ r ∗ .
So now the right hand side of (8) can be reshaped as
[Equation]
Now, note that since by deﬁnition the columns of U (cid:48) are in im( U ) and the columns of V (cid:48) are in im( V ) , Lemma A.8 implies that (cid:104)∇ R ( A
[Equation]
1 ) , U [Equation]
V (cid:48)(cid:62) (cid:105) = 0 .
Therefore the above is equal to
[Equation]
where the ﬁrst equality follows by noticing that the columns of V (cid:48) and V 1 are orthogonal to those of
V 2 and the columns of U (cid:48) and U 1 are orthogonal to those of U 2 , and applying Lemma A.7.
The last equality is a direct application of Lemma A.6 and the last inequality states that the largest squared singular value is not smaller than the average of the top r ∗ squared singular values.
Therefore we have concluded that
[Equation]
Plugging this back into the smoothness inequality, we get
[Equation]
or equivalently
[Equation]
Therefore after L = 2 r ∗ κ log R ( A 0 ) − R
( A ∗ ) (cid:15) iterations we have
[Equation]
Since A 0 = 0 , the result follows.
A. P ROOF OF T HEOREM 2.2 ( LOCAL SEARCH )
Proof.
Similarly to Section A.3, we let A t − 1 be the current solution before iteration t − 1 ≥ 0 .
Let u ∈ R m and v ∈ R m be left and right singular vectors of matrix ∇ R ( A ) , i.e. unit vectors maximizing |(cid:104)∇ R ( A ) , uv (cid:62) (cid:105)| and let
[Equation]
where σ min xy (cid:62)
=
A t − 1 − H r − 1 ( A t − 1 ) is the rank-1 term corresponding to the minimum singular value of A t − 1 .
By smoothness we have
[Equation]
where in the last inequality we used the fact that (cid:104)∇ R ( A t − 1 ) , xy (cid:62) (cid:105) = 0 following from Lemma A.8, as well as Lemma A.
[Equation]
On the other hand, by strong convexity,
[Equation]
Let A t − 1 = UV (cid:62) and A ∗ =
U ∗ V ∗(cid:62) .
We write
[Equation]
where U 1 is a matrix where every column of U ∗ is replaced by its projection on im( U ) and U 2 =
U ∗ − U 1 and similarly V 1 is a matrix where every column of V ∗ is replaced by its projection on im( V ) and V
=
V ∗ − V 1 .
By setting U (cid:48)
= ( − U | U 1 ) and V (cid:48) =
( V | V 1 ) we can write
[Equation]
where im( U (cid:48) )
=
im( U ) and im( V (cid:48) )
=
im( V ) .
Also, note that rank( U 1 V 2 (cid:62) ) ≤ rank( V 2 ) ≤ rank( V ∗ )
= rank( A ∗ ) ≤ r ∗ and similarly rank( U 2 V ∗(cid:62) )
≤ r ∗ .
So we now have
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the second equality follows from the fact that (cid:104)∇ R ( A t − 1 ) , uv (cid:62) (cid:105) = 0 for any u ∈
im( U ) , v ∈ im( V ) , the third equality from the fact that im( U 2 ) ⊥
im( U (cid:48) ) ∪ im( U 1 ) and im( V 2 ) ⊥
im( V (cid:48) ) and by applying Lemma A.7, and the last inequality from the fact that the largest squared singular value is not smaller than the average of the top r ∗ squared singular values.
Now, note that since
[Equation]
where we used the fact that rank( U 1 V 1 (cid:62) ) ≤ r ∗ together with Lemma A.5.
Therefore we have concluded that
[Equation]
Plugging this back into the smoothness inequality and setting (cid:101) κ = ρ +2 ρ − r + r ∗ , we get
R [Equation] A t − 1 ) − R
[Equation]
A t ) ≥
[Equation]
as long as r ≥ r ∗ (1 + 8 (cid:101) κ 2 ) , or equivalently, [Equation]
[Equation]
Therefore after L = 4 r ∗ (cid:101) κ log R ( A 0 ) − R ( A ∗ ) (cid:15) iterations we have [Equation] Since A 0 = 0 and (cid:101) κ ≤ κ r + r ∗ , the result follows.
A. T IGHTNESS OF THE ANALYSIS
It is important to note that the κ r + r ∗ factor that appears in the rank bounds of both Theorems 2. and 2.2 is inherent in these algorithms and not an artifact of our analysis.
In particular, such lower bounds based on the restricted condition number have been previously shown for the problem of sparse linear regression.
More speciﬁcally, Foster et al.
(2015) showed that there is a family of instances in which the analogues of Greedy and Local Search for sparse optimization require the sparsity to be Ω( s ∗ κ (cid:48) ) for constant error (cid:15) > 0 , where s ∗ is the optimal sparsity and κ (cid:48) is the sparsity -restricted condition number.
These instances can be easily adjusted to give a rank lower bound of Ω( r ∗ κ r + r ∗ ) for constant error (cid:15) > 0 , implying that the κ dependence in Theorem 2.1 is tight for Greedy.
Furthermore, speciﬁcally for Local Search, Axiotis & Sviridenko (2020) additionally showed that there is a family of instances in which the analogue of Local Search for sparse optimization requires a sparsity of Ω( s ∗ ( κ (cid:48) ) 2 ) .
Adapting these instances to the setting of rank-constrained convex optimization is less trivial, but we conjecture that it is possible, which would lead to a rank lower bound of Ω( r ∗ κ 2 r + r ∗ ) for Local Search.
We present the following lemma, which essentially states that sparse optimization lower bounds for Orthogonal Matching Pursuit (OMP, Pati et al.
(resp.
Orthogonal Matching Pursuit with Replacement (OMPR, Jain et al.
(2011))) in which the optimal sparse solution is also a global optimum, immediately carry over (up to constants) to rank-constrained convex optimization lower bounds for Greedy (resp.
Local Search).
Lemma A.9.
Let f ∈ R n → R and x ∗ ∈ R n be an s ∗ -sparse vector that is also a global minimizer of f .
Also, let f have restricted smoothness parameter β at sparsity level s + s ∗ for some s ≥ s ∗ and restricted strong convexity parameter α at sparsity level s + s ∗ .
Then we can deﬁne the rankconstrained problem, with R : R n × n → R ,
[Equation]
(9) where diag( A ) is a vector containing the diagonal of A .
R has rank-restricted smoothness at rank s + s ∗ at most 2 β and rank-restricted strong convexity at rank s + s ∗ at least α .
Suppose that we run t iterations of OMP (resp.
OMPR) starting from a solution x , to get solution x (cid:48) , and similarly run t iterations of Greedy (resp.
Local Search) starting from solution A = diag( x ) (where diag( x ) is a diagonal matrix with x on the diagonal) to get solution A (cid:48) .
Then A (cid:48) is diagonal and diag( A (cid:48) )
=
x (cid:48) .
In other words, in this scenario OMP and Greedy (resp.
OMPR and Local Search) are equivalent.
Proof.
Note that for any solution (cid:98) A of R we have R ( (cid:98) A ) ≥ f (diag( (cid:98) A )) ≥ f ( x ∗ ) , with equality only if (cid:98) A is diagonal.
Furthermore, rank(diag( x ∗ )) ≤ s ∗ , meaning that diag( x ∗ ) is an optimal solution of (9) .
Now, given any diagonal solution A of (9) such that A = diag( x ) , we claim that one step of either Greedy or Local Search keeps it diagonal.
This is because
[Equation]
Therefore the largest eigenvalue of ∇ R ( A ) has corresponding eigenvector 1 i for some i , which implies that the rank-1 component which will be added is a multiple of 1 i 1 (cid:62) i .
For the same reason the rank-1 component removed by Local Search will be a multiple of 1 j 1 (cid:62) j for some j .
Therefore running Greedy (resp.
Local Search) on such an instance is identical to running OMP (resp.
OMPR) on the diagonal.
Together with the lower bound instances of Foster et al. (2015) (in which the global minimum property is true), it immediately implies a rank lower bound of Ω( r ∗ κ r + r ∗ ) for getting a solution with constant error for rank-constrained convex optimization.
On the other hand, the lower bound instances of Axiotis & Sviridenko (2020) give a quadratic lower bound in κ for OMPR.
The above lemma cannot be directly applied since the sparse solutions are not global minima, but we conjecture that a similar proof will give a rank lower bound of Ω( r ∗ κ 2 r + r ∗ ) for rank-constrained convex optimization with Local Search.
A. A DDENDUM TO S ECTION
[Figure] [Figure]