Paper: Fast Updating Truncated SVD for Representation Learning with Sparse Matrices (ICLR-2024)

Reviewer comment (Experiments):
The method of Zha and Simon should be more accurate than the algorithms of Vecharynski and Yamazaki, yet it seems that all three methods basically give the same accuracy. One reason for that is when ‘l’ is too large. What are the results when ‘l’ is smaller?

Author response:
**[Accuracy of smaller $\phi$]**

We have included the Frobenius norm values of the original and reconstructed matrices for various $\phi$ values in Appendix E.3. Experimental results demonstrate that our proposed approach significantly enhances efficiency while maintaining comparable precision to previous methods.

Comparatively, for smaller values of $\phi$ , the algorithms of Vecharynski and Yamazaki demonstrate a higher degree of accuracy loss compared to the algorithm of Zha & Simon. We suspect that this discrepancy could be attributed to larger values of $s$  when $\phi$ is smaller in our experiment. These two methods approximate a higher-dimensional space with a significantly lower-dimensional basis. From the perspective discussed in [1],  it is possible that these methods might have difficulty accurately capturing the $range(U_k)$, leading to a loss of accuracy.

Reviewer comment (Writing):
“trunacated” -> “truncated”

Author response:
**[Writing]**

In the revised version of the paper, we have addressed and corrected several typos and grammatical errors. Additionally, we have made clarifications in areas that may have caused ambiguity.

Reviewer comment (Writing):
There are issues of grammar and English usage throughout the paper, making it more difficult to read and understand. Examples include: the title and first sentence of the abstract are not grammatically correct; phrases such as "truncated SVD" should be "the truncated SVD," "augment matrix" should be "augmented matrix," and "augment procedure" should be "augmentation procedure"; subject–verb and pronoun agreement errors (e.g., "has" should be "have" and "its" should be "their" in a sentence about a series of methods). Even with grammatical fixes, some sentences remain difficult to read.

Author response:
We appreciate the reviewer's valuable comments. Thank you for the suggestion, and we have revised the writing and addressed the grammar issues in the paper revision.

Reviewer comment (Writing):
There are structural and clarity issues in the writing. Examples include: "augment matrix" (which should be "augmented matrix") is introduced before the term is defined or the problem stated; acronyms "RPI" and "GKL" in section 3.3 were only briefly defined earlier in section 3.1 and would benefit from backward references for clarity; the precise meaning of "isometric" in lemma 3 is unclear; the average precision metric is never defined. The paper could benefit from reorganization to improve the narrative and sequencing of ideas.

Author response:
We have partially restructured the paper, and in the revised version, we supplemented missing definitions, including those for augmented matrix, isometric, and average precision, which are highlighted in blue.

Reviewer comment (Theory):
The proposed orthogonalization procedure uses the Gram–Schmidt process, which is known to suffer from potentially significant loss of orthogonality, raising concerns about numerical stability.

Author response:
We have replaced the Gram-Schmidt process in Section 3.1 with the modified Gram-Schmidt process to enhance the numerical stability. 

The GS process demonstrates relative stability when the value of $s$ is small. In practical applications, Lanczos vectors are commonly employed to approximate the basis for efficiency considerations, especially when the value of *s* is large. Therefore, we recommend utilizing the proposed variant method when dealing with large values of $s$.

Reviewer comment (Theory):
The updating rules decompose a matrix $U_k$ with orthonormal columns into the product of two matrices, both of which could become significantly ill-conditioned, potentially leading to stability degradations.

Author response:
The condition number of the k-by-k matrix in the extended decomposition is close to $1$ in our experiments. In the following results, we present the maximum condition number observed throughout the entire update process for different numbers of update batches (i.e. $\phi$) on the Slashdot dataset. (Please note that in this experiment, the value of $s$ is inversely proportional to $\phi$.)

| $\phi$   | $10^1$ | $10^2$ | $10^3$ | $10^4$ |
| -------- | ------ | ------ | ------ | ------ |
| ours     | 1.745  | 1.671  | 1.630  | 1.620  |
| ours-GKL | 1.518  | 1.250  | 1.226  | 1.167  |
| ours-RPI | 1.137  | 1.138  | 1.143  | 1.149  |

The experimental results show that the condition numbers of the matrices remain relatively stable even after numerous updates, exhibiting no exponential or significant growth.

In the body of the Section 3.2, we have provided further elaboration by discussing potential errors that could be caused by this aspect. In practice, the problem of matrices with large condition numbers can be solved by multiplying $U^\prime$ and $U^{\prime \prime}$ and resetting the resulting $k$-by-$k$ matrix to the identity. Considering the time efficiency improvement provided by the proposed method, we believe this error is acceptable.

Reviewer comment (Theory):
Assuming these stability issues are real, they appear to be intrinsic limitations of the method and would be difficult to resolve while maintaining competitive runtime. Although numerical results suggest the method may be sufficiently stable for some data analysis tasks, the authors should comment on numerical stability as a potential issue in any revised version.

Author response:
The condition number of the k-by-k matrix in the extended decomposition is close to $1$ in our experiments. In the following results, we present the maximum condition number observed throughout the entire update process for different numbers of update batches (i.e. $\phi$) on the Slashdot dataset. (Please note that in this experiment, the value of $s$ is inversely proportional to $\phi$.)

| $\phi$   | $10^1$ | $10^2$ | $10^3$ | $10^4$ |
| -------- | ------ | ------ | ------ | ------ |
| ours     | 1.745  | 1.671  | 1.630  | 1.620  |
| ours-GKL | 1.518  | 1.250  | 1.226  | 1.167  |
| ours-RPI | 1.137  | 1.138  | 1.143  | 1.149  |

The experimental results show that the condition numbers of the matrices remain relatively stable even after numerous updates, exhibiting no exponential or significant growth.

In the body of the Section 3.2, we have provided further elaboration by discussing potential errors that could be caused by this aspect. In practice, the problem of matrices with large condition numbers can be solved by multiplying $U^\prime$ and $U^{\prime \prime}$ and resetting the resulting $k$-by-$k$ matrix to the identity. Considering the time efficiency improvement provided by the proposed method, we believe this error is acceptable.

Reviewer comment (Writing):
There are small typos and minor issues throughout the paper. Examples include: on the top of page 3, $U_kU_k$ and $V_kV_k$ should have transposes on the second factors; "Orthogonalization" is misspelled in a section header on page 3; in equation (3), the middle line should have $U_kC$ not $U_k^\top C$; there are inconsistencies in typesetting, with some A's set in upright bold font and others in slanted bold font. The authors should read the paper carefully to correct these.

Author response:
We appreciate the reviewer's valuable comments. Thank you for the suggestion, and we have revised the writing and addressed the grammar issues in the paper revision.

Reviewer comment (Theory):
In the case of updating (adding) columns to \(A\), the manuscript assumes \(E\) is sparse, but the improvement is actually limited to a narrower regime. If \(E\) has a constant number of non-zeros per row, the algorithms discussed are no more efficient than just forming \(E - U(U^TE)\) and computing the dense QR factorization \(\mathcal{O}(ns^2)\). The only setting of interest is if \(E\) has a constant number of non-zeros per column (\(\mathcal{O}(s)\) non-zeros). The manuscript lacks a clear articulation of such tradeoffs, including in the numerical experiments.

Author response:
**[Sparsity of $BF_k[k:]$]**

The number of non-zero entries of the $n$-by-$k$ matrix $BF_k[k:]$ is less than $nnz(B)\cdot k$. 

Here, 'sparse' is used to indicate that this addition should be a *sparse addition* to ensure the asymptotic time complexity, i.e., the time complexity of this addition step is $O(nnz(B) \cdot k)$, not $O(nk)$. For relatively dense matrices, the utilization of *sparse addition* in this context can still maintain the asymptotic time complexity.

Reviewer comment (Novelty):
Much of section 3.1 seems unnecessarily complicated for what appears to be a simple choice. Under the constant non-zeros per column sparsity assumption, one could compute the Cholesky factorization \(E^TE - (E^TU_k)(U_k^TE) = RR^T\) (\(\mathcal{O}(s^3)\)) and then store the QR factorization as \((I-U_kU_k^T)(ER^{-1})\) (\(\mathcal{O}(s^3)\) under the sparsity assumption). The manuscript appears to propose this with unnecessary additional machinery, obscuring the message. Use of Cholesky could be faster with standard libraries. Theorem 1 may not be particularly novel or interesting.

Author response:
**[About Section 3.1]**

In Section 3.1, we present the fundamental operations in an inner product space, which enable the proposed structure to enhance the efficiency of various previous works on SVD update within a unified framework. We dedicate the entire Section 3.1 to present these operations, highlighting that our approach not only enables efficient QR decomposition but also serves as a generalized framework for approximating space bases, such as the GKL and RPI processes. We believe it would be interesting to provide more fundamental definitions and examine how a series of processes within the space, such as GS, Modified GS, and Lanczos vectors, are optimized from a unified perspective. Due to the page limit of ICLR, we provide detailed explanations of these methods in the appendix. 

A faster QR process is not the only contribution of this paper. In fact, apart from enhancing efficiency, another important motivation for proposing the structure is to accommodate the extended decomposition presented in Section 3.2. By merging the low-dimensional components of the proposed structure through Equation (3), we achieve a reduction in the overall asymptotic time complexity.

Reviewer comment (Experiments):
The numerical experiments would be stronger if designed to explicitly show the tradeoffs between methods, for instance using synthetic examples where sparsity can be controlled. This is more important than the current "plausible" situations since accuracy should be similar; the key is to show when to use which method based on the sparsity of \(E\). The numerical experiments are not illuminating in this regard.

Author response:
**[In what setting the proposed method is faster]**

The insight of this paper is the elimination of the $O(n)$ and $O(m)$ terms from the time complexity (where n and m are the length and width of the matrix, respectively). We adopt this approach because, when updating large-scale matrices, the $O(n)$ or $O(m)$ term typically dominates the overall time complexity compared to other factors such as $k$ (the dimension of the decomposition), $s$ (the rank of the update), and $nnz$ (the number of non-zero rows or columns in the update matrix). Therefore, the proposed method exhibits significant improvement when the values of $s$ or $nnz$ are small. However, when $s$ or $nnz$ becomes comparable to $O(n)$ or $O(m)$ (i.e., the terms we aim to eliminate), the optimization effect diminishes.

We conduct numerical experiments with synthetic matrices in **Appendix E.2**. Specifically, we generate several random sparse matrices with a fixed size of 100,000 * 100,000. The number of non-zero elements in these matrices ranges from 1,000,000 to 1,000,000,000 (i.e., density from 0.01% to 10%). Experimental results with various $s$ values show that: 1) The proposed method exhibits more substantial efficiency improvements when the matrix is relatively sparse. 2) When the rank of the update matrix is higher, the variant using Lanczos vectors for space approximation achieves faster performance.

Reviewer comment (Writing):
The manuscript does not clearly articulate the contribution, compounded by a lack of clarity throughout, which significantly weakens the presentation and understanding of what is being contributed.

Author response:
**[In what setting the proposed method is faster]**

The insight of this paper is the elimination of the $O(n)$ and $O(m)$ terms from the time complexity (where n and m are the length and width of the matrix, respectively). We adopt this approach because, when updating large-scale matrices, the $O(n)$ or $O(m)$ term typically dominates the overall time complexity compared to other factors such as $k$ (the dimension of the decomposition), $s$ (the rank of the update), and $nnz$ (the number of non-zero rows or columns in the update matrix). Therefore, the proposed method exhibits significant improvement when the values of $s$ or $nnz$ are small. However, when $s$ or $nnz$ becomes comparable to $O(n)$ or $O(m)$ (i.e., the terms we aim to eliminate), the optimization effect diminishes.

We conduct numerical experiments with synthetic matrices in **Appendix E.2**. Specifically, we generate several random sparse matrices with a fixed size of 100,000 * 100,000. The number of non-zero elements in these matrices ranges from 1,000,000 to 1,000,000,000 (i.e., density from 0.01% to 10%). Experimental results with various $s$ values show that: 1) The proposed method exhibits more substantial efficiency improvements when the matrix is relatively sparse. 2) When the rank of the update matrix is higher, the variant using Lanczos vectors for space approximation achieves faster performance.

Reviewer comment (Writing):
The paper contains numerous grammatical mistakes and typos, such as in the title, abstract, and first paragraph of the introduction. Examples include unclear or incorrect phrasing (“in sparse matrix,” “updating truncated singular value decomposition,” “maintaining precision comparing”) and awkward statements about the SVD’s interpretability. The manuscript requires a careful editing pass to address these issues.

Author response:
Thank you for dedicating your time to reviewing our paper and providing feedback on the grammatical issues. We apologize for any confusion caused during your review. We have taken your comments seriously and made significant revisions to address issues highlighted in your review. We meticulously reviewed the entire manuscript, correcting grammatical errors and ensuring sentence clarity and accuracy.

Reviewer comment (Writing):
After the author response, while some clarifications about the scope of the contribution were offered, these are not fully reflected in the manuscript. Phrases like “input sparsity” time need follow-up contextualization in terms of the sparsity of the update, which is not clearly present in the updated manuscript.

Author response:
**[In what setting the proposed method is faster]**

The insight of this paper is the elimination of the $O(n)$ and $O(m)$ terms from the time complexity (where n and m are the length and width of the matrix, respectively). We adopt this approach because, when updating large-scale matrices, the $O(n)$ or $O(m)$ term typically dominates the overall time complexity compared to other factors such as $k$ (the dimension of the decomposition), $s$ (the rank of the update), and $nnz$ (the number of non-zero rows or columns in the update matrix). Therefore, the proposed method exhibits significant improvement when the values of $s$ or $nnz$ are small. However, when $s$ or $nnz$ becomes comparable to $O(n)$ or $O(m)$ (i.e., the terms we aim to eliminate), the optimization effect diminishes.

We conduct numerical experiments with synthetic matrices in **Appendix E.2**. Specifically, we generate several random sparse matrices with a fixed size of 100,000 * 100,000. The number of non-zero elements in these matrices ranges from 1,000,000 to 1,000,000,000 (i.e., density from 0.01% to 10%). Experimental results with various $s$ values show that: 1) The proposed method exhibits more substantial efficiency improvements when the matrix is relatively sparse. 2) When the rank of the update matrix is higher, the variant using Lanczos vectors for space approximation achieves faster performance.

Reviewer comment (Theory):
On a technical note, while modified Gram-Schmidt (MGS) helps address potential numerical issues when \(B\) is close to the column space of \(U_k\), it does not address potential issues from persistent decoupled storage of vectors in the sparse + low-dimensional subspace. It is unclear whether Cholesky QR would actually be worse in this setting, and given some reported condition numbers, it might be fine for simple tasks.

Author response:
**[Numerical Issue of Gram-Schmidt]**

Thank you for your suggestion. In the revised version, we have replaced Gram-Schmidt with the modified Gram-Schmidt. When a vector is parallel or nearly parallel to the preceding span, the result of the orthogonalized vector can be set to the zero vector. From the perspective of the Rayleigh-Ritz process, this situation can be regarded as the current space being capable of approximating the singular space of the new matrix well without requiring the addition of new dimensions. We have included this observation as a remark in the revised version.