Paper: Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time (ICLR-2024)

Reviewer comment (Presentation):
In my opinion, the presentation of the paper can be improved. The current paper spends too much space on providing the intuition of the proposed results, which makes the paper too dry to understand.

Author response:
We appreciate your valuable and insightful comments on our manuscript, particularly regarding its structure. We have significantly condensed the discussion in Section 1 and now present our informal main result at its conclusion. Additionally, we have enriched Sections 4.1 to 4.3 with more detailed discussions, aiming to enhance clarity about our techniques and overall proof strategy.

Reviewer comment (Theory):
I think the authors need to include more technical details in the main body of the paper. Due to the time limit, I do not have time to check the appendix, so I cannot be sure about the correctness of the theoretical results given the limited information in the main manuscript.

Author response:
We would like to emphasize that our decision to focus on informal intuitions and technique overviews in the main body of the paper is a deliberate choice, given its technical nature. The paper's results are highly technical, and deriving a major result requires many intricate components. Including these in the main body could make the paper overly technical and challenging for readers. Therefore, we have opted for a format that conveys the core ideas behind the proofs while minimizing too complicated technical details.

Reviewer comment (Presentation):
Theorem 4.2: I think it might be better to provide the main results (Theorem 4.2) a little earlier in the paper. The preparation for the statement of the main results is too long. The contents of Section 4 are basically a more detailed version of Section 1. Given the page limit, I feel that it is more efficient to simplify the discussion of techniques in Section 1, but include more details in Sections 4.1–4.3.

Author response:
We appreciate your valuable and insightful comments on our manuscript, particularly regarding its structure. We have significantly condensed the discussion in Section 1 and now present our informal main result at its conclusion. Additionally, we have enriched Sections 4.1 to 4.3 with more detailed discussions, aiming to enhance clarity about our techniques and overall proof strategy.

Reviewer comment (Theory):
Though the runtime of the proposed algorithm is nearly linear in verification time and improves on the runtime compared to previous algorithms, without any discussion on computational limitations or lower bounds, it is hard to judge if this is indeed a significant theoretical result for this problem. Some discussion on runtime or sample complexity lower bound could be useful to understand what is the runtime one should be aiming for this problem.

Author response:
Sample complexity lower bound. For this problem, it is known that $\Omega(nk)$ samples are needed by a simple parameter counting argument. Beyond that, Hardt et al., 2014 has shown that even if the ground truth is rank-4, has incoherent rows and columns, we can observe 90\% of entries and in additional, we are allowed to output a higher constant rank matrix, then this problem is still NP-hard (via a reduction from 4-coloring). From an upper bound side, it is known that methods based on convex relaxation (Candes and Tao, 2010; Recht, 2011) provide a sample complexity upper bound $\widetilde O(nk)$, but these methods require $O(n^\omega)$ time to solve a semidefinite program. For non-convex programming methods, current best algorithm is due to Kelner et al., 2023 which requires $O(nk^{2+o(1)})$ samples. However, as we have discussed in our paper, it is not clear whether their algorithm can provide a good practical performance as their method involves using many techniques with good theoretical runtime guarantees but lack practical validations. See the end of Section 4 of our revised draft for a more detailed comparison.

Reviewer comment (Writing):
The proofs in the appendix seem to have some confusing notations and sometimes use certain notations without defining them first, which causes problems while reading the proofs:  
i) For example, in Lemma B.6, in some places $D_{W_i}$ seems to indicate a matrix with $W_i$ on the diagonal and in other places, a constant? When defining $\| z\|_w^2$ it should be just ||z ||_w^2=\sum_{i=1}^n  w_i^2, I think for a vector $w$?  
ii) Also, in Definition 3.5, M is factorized as $U \Sigma V^T$ while in the appendix, it seems $U^* \Sigma^* V^*$ is used?

Author response:
Proof notations: for a non-negative vector $w\in \mathbb{R}^n$, we use  $\\|v\\|_w^2=\sum_i w_iv_i^2=\sum_i (D_W)_i v_i^2$ to denote the weighted norm, where the second equality follows from we can rewrite a vector by a diagonal matrix that puts the vector on the diagonal. In definition 3.5, the factorization $M=U\Sigma V^\top$ is a *local* definition, as we will use the incoherent assumptions on other matrices. The factorization $M=U^* \Sigma^* (V^*)^\top$ is a *global* definition of the ground truth.

Reviewer comment (Theory):
In the forward error bound of Lemma B.5, there is confusion about why the step ||Ax'-b-(Ax_{OPT}-b)||_2^2=||Ax'-b||_2^2-||Ax_{OPT}-b||_2^2 should be true. Why should the Pythagorean theorem hold in this case? Is Ax_{OPT}-b orthogonal to Ax'-b-(Ax_{OPT}-b) due to some condition? Also, it seems A is assumed to have full column rank for the proof. Is A guaranteed to have full column rank whenever this result is applied, i.e., in all iterations whenever B.7 is invoked?

Author response:
Lemma B.5. We have revised the lemma statement and provided an alternative proof without assuming $A$ has full column rank. Further, we have provided a comprehensive explanation why using the Pythagorean theorem is valid. For more details, see page 25-26 in our revised draft.

Reviewer comment (Theory):
In Lemma G.3, there is uncertainty about how $\sigma_{min}(B) \geq 0.5\sigma_{min}(A)$ follows from $||A-B|| \leq \epsilon_o \leq \sigma_{min}(A)$.

Author response:
Lemma G.3. We have added an additional assumption $\epsilon_0 \leq \frac{1}{2}\sigma_{\min}(A)$, note this won’t affect the overall analysis. We also significantly simplify and streamline the proof so that it’s easier to understand. For more details, see page 51-52 in our revised draft.

Reviewer comment (Novelty):
The main weakness might be in the significance of the final running time result in the context of the recent result by Kelner et. al (2023). Given that Kelner et. al achieve a significantly lower sample complexity of $\tilde{O}(n k^2)$ (with no dependence on $\kappa$), the novelty of this result could be questioned.

Author response:
We appreciate your insightful and valuable comments. We would like to highlight the significance of our results in comparison to Kelner et al., 2023b. Indeed, Kelner et al., 2023b offers a theoretically state-of-the-art sample complexity and runtime for a non-convex programming-based algorithm in matrix completion. Under standard incoherence assumptions, they achieve a sample complexity of $|\Omega|=O(nk^{2+o(1)})$ and a runtime of $\widetilde O(|\Omega|k)$. However, their approach significantly diverges from the algorithms commonly used in matrix completion practice.

Their algorithm, broadly speaking, is grounded in the 'short-flat decomposition' developed in their earlier work (Kelner et al., 2023a) on semi-random sparse recovery. This decomposition separates the observation into a useful term and a noisy term with 'flat' singular values. The algorithm begins with initial matrices $U_0, V_0=0$, and processes a set of observations from M by first removing large rows and columns from $P_{\Omega_t}(M-U_tV_t^\top)$, followed by running a top-k SVD procedure from Musco and Musco, 2015. This procedure is iterated for ${\rm polylog}(1/\epsilon)$ rounds. Due to dropping rows and columns with large norms, they need to recover them back in the post-processing. This can be achieved through regression 'tests' to determine if a set of columns can linearly form a targeted column. The recovery algorithm then proceeds by sampling columns, tests for linear independence, and this process repeats for ${\rm polylog}(n)$ times. Ultimately, the selected columns help fill missing entries for a partially completed matrix for $U$, and the matrix $V$ is completed via Nesterov’s accelerated gradient descent (AGD). They employ Nesterov’s AGD for its good condition number dependence. Of course, they also need a bound on the initial error, which is done via estimating $\log(1/\epsilon)$ many spectral norms over independent samples of $M$, and outputting the median using Chernoff bound.

From a practical standpoint, their algorithm is likely slower than alternating minimization or SGD due to the need for top-k SVD computation and linear independence testing for ${\rm polylog}(n, 1/\epsilon)$ rounds, along with an additional $\log(1/\epsilon)$ rounds for multiple response regression using Nesterov’s AGD. The randomized top-k SVD algorithm they employ, based on block Krylov subspace iteration, is known for numerical instability. While Nesterov’s AGD offers theoretical advantages, it is challenging to achieve fast practical implementation, often making Polyak’s momentum a preferred choice for its simplicity and practical efficacy.

In contrast, alternating minimization and stochastic gradient descent are the most widely applied approaches in practice for matrix completion, despite theoretically inferior sample complexity. These methods are practically efficient, generally involving a sequence of least squares problems solvable by iterative methods. Our robust alternating minimization framework provides a theoretical justification for the practical success of alternating minimization, benefiting from faster solvers. We also believe that further tightening the sample complexity bound for alternating minimization could lead to direct runtime improvements for our algorithm.

Kelner et al., 2023a. Kelner, Li, Liu, Sidford and Tian. Semi-random sparse recovery in nearly-linear time. COLT’2023.

Kelner et al., 2023b. Kelner, Li, Liu, Sidford and Tian. Matrix completion in almost-verification time. FOCS’2023.

Musco and Musco, 2015. Randomized block krylov methods for stronger and faster approximate singular value decomposition. NeurIPS’2015.

Reviewer comment (Novelty):
Can you speak more to the significance of your result as compared to that of Kelner et. al (2023)? Specifically to the significance of the running time result given they achieve an asymptotically smaller running time result (please correct me if that is incorrect).

Author response:
We appreciate your insightful and valuable comments. We would like to highlight the significance of our results in comparison to Kelner et al., 2023b. Indeed, Kelner et al., 2023b offers a theoretically state-of-the-art sample complexity and runtime for a non-convex programming-based algorithm in matrix completion. Under standard incoherence assumptions, they achieve a sample complexity of $|\Omega|=O(nk^{2+o(1)})$ and a runtime of $\widetilde O(|\Omega|k)$. However, their approach significantly diverges from the algorithms commonly used in matrix completion practice.

Their algorithm, broadly speaking, is grounded in the 'short-flat decomposition' developed in their earlier work (Kelner et al., 2023a) on semi-random sparse recovery. This decomposition separates the observation into a useful term and a noisy term with 'flat' singular values. The algorithm begins with initial matrices $U_0, V_0=0$, and processes a set of observations from M by first removing large rows and columns from $P_{\Omega_t}(M-U_tV_t^\top)$, followed by running a top-k SVD procedure from Musco and Musco, 2015. This procedure is iterated for ${\rm polylog}(1/\epsilon)$ rounds. Due to dropping rows and columns with large norms, they need to recover them back in the post-processing. This can be achieved through regression 'tests' to determine if a set of columns can linearly form a targeted column. The recovery algorithm then proceeds by sampling columns, tests for linear independence, and this process repeats for ${\rm polylog}(n)$ times. Ultimately, the selected columns help fill missing entries for a partially completed matrix for $U$, and the matrix $V$ is completed via Nesterov’s accelerated gradient descent (AGD). They employ Nesterov’s AGD for its good condition number dependence. Of course, they also need a bound on the initial error, which is done via estimating $\log(1/\epsilon)$ many spectral norms over independent samples of $M$, and outputting the median using Chernoff bound.

From a practical standpoint, their algorithm is likely slower than alternating minimization or SGD due to the need for top-k SVD computation and linear independence testing for ${\rm polylog}(n, 1/\epsilon)$ rounds, along with an additional $\log(1/\epsilon)$ rounds for multiple response regression using Nesterov’s AGD. The randomized top-k SVD algorithm they employ, based on block Krylov subspace iteration, is known for numerical instability. While Nesterov’s AGD offers theoretical advantages, it is challenging to achieve fast practical implementation, often making Polyak’s momentum a preferred choice for its simplicity and practical efficacy.

In contrast, alternating minimization and stochastic gradient descent are the most widely applied approaches in practice for matrix completion, despite theoretically inferior sample complexity. These methods are practically efficient, generally involving a sequence of least squares problems solvable by iterative methods. Our robust alternating minimization framework provides a theoretical justification for the practical success of alternating minimization, benefiting from faster solvers. We also believe that further tightening the sample complexity bound for alternating minimization could lead to direct runtime improvements for our algorithm.

Kelner et al., 2023a. Kelner, Li, Liu, Sidford and Tian. Semi-random sparse recovery in nearly-linear time. COLT’2023.

Kelner et al., 2023b. Kelner, Li, Liu, Sidford and Tian. Matrix completion in almost-verification time. FOCS’2023.

Musco and Musco, 2015. Randomized block krylov methods for stronger and faster approximate singular value decomposition. NeurIPS’2015.