Paper: Optimal Sketching for Residual Error Estimation for Matrix and Vector Norms (ICLR-2024)

Reviewer comment (Evaluation):
The authors suggest data streams as a potential application. While the proposed approach does give a way of estimating the residual error from a stream in low space, allowing one to decide whether or not low-rank approximation is worth it, this raises objections: (1) If one is going to compute a low-rank approximation anyway, then one will have to incur the space cost of storing such an approximation eventually. The proposed low-space diagnostic doesn't seem useful if, conditional on a positive outcome, actually computing the approximation needs more space. (2) Often in the streaming setting, we assume we get only a single pass over the data. The authors’ approach requires two passes: one to compute the diagnostic and a second to compute a low-rank approximation.

Author response:
On Use Cases.

> I understand that your algorithm has lower space than computing a low-rank approximation. My question is What use is a low-space algorithm for testing the residual error if, should the low-rank approximation error be small, you'll need higher-space to actually compute the low-rank approximation? "

One of the reviewer's points seems to be that since the downstream task needs more space if we test a smaller error, then the memory savings is not very useful as we would ultimately need a larger memory. However, note that the data analysis in practice is often conducted simultaneously among different groups of data. Hence, the smaller space requirement in the first stage allows us to do a larger amount of testing among different groups of data during the same period, which results also in a time savings.

> I have never encountered or heard of such a scenario occurring in practice. Do the authors have a reference for such a scenario in an applied context?

A typical application of low-rank approximation is to recommendation systems (for more details, see the textbook [Agg16]). In this model, we consider an $m \times n $ preference matrix $A$, where $A_{ij}$ denotes user $i$'s preference for item $j$. Given a matrix preference $A$ with a subset of known entries, the goal is to predict large entries of $A$, representing good recommendations. The method based on low-rank matrix completion works well in practice for this problem.
For this task, it is reasonable to first check whether the current collection of items and users indeed have a good low-rank representation and use them as part of the data in future predictions, which we believe should result in a runtime savings. The $\mathrm{poly}(k)$ space of the algorithm is also beneficial here as usually the prediction is performed simultaneously across different user/item groups, and hence a smaller working space allows for computing more groups at the same time.

Another potential use of our algorithm is as a sub-routine of more complicated optimization problems. Consider the low-rank tensor decomposition problems in [SGL+20]. At a high level, for each mode of the tensor $X^{(n)}$, we compute a decomposition of $X^{(n)}$, however, if the tail error of $\|X^{(n)} - X^{(n)}_k\|_F^2$ is close to $\|X^{(n)}\|_F^2$, then we could just use an arbitrary orthonormal basis, which will only increase the error by a small factor and we then gain some savings in runtime / memory by not having to explicitly compute a low rank approximation of most flattenings of the input tensor.

Reviewer comment (Experiments):
The author’s use of CountSketch is questionable, particularly in the numerical experiments. In practice, an OSNAP (even with just 2 entries of sparsity per column) is dramatically more safe and effective than CountSketch. For example, with $A = \operatorname{diag}( i^{-6} : i=1,2,\ldots,10^4)$, $m = 50$, and $k = 20$, CountSketch gives an estimate $\| SAT - (SAT)_k \|_F$ that is a factor of **58** smaller than the true residual error $\| A - A_k \|_F$, whereas OSNAP with just 2 nonzeros per column underestimates by only a factor of 2. For screening whether significant compression is possible by low-rank approximation, an estimate within a factor of two is useful; one wrong by more than an order of magnitude is not.

Author response:
- The experiments: the reason we use a plain CountSketch matrix is that we found it has good performance for the two real-world datasets. We agree with the reviewer's point that OSNAP has a better worst-case guarantee. Hence, we have replaced CountSketch with ONSAP in our new version of the experiments. (Note that the new experiments are conducted on a different device, so the runtime is incomparable with that of the previous version.)

Reviewer comment (Experiments):
Since this paper is about theoretical computer science with rigorous analysis of computational methods to work even on worst‑case inputs, the use of a plain CountSketch in numerical experiments, shown to severely underestimate residual error in practice, should be removed in favor of OSNAP or perhaps the composite CountSketch–Gaussian sketch.

Author response:
- The experiments: the reason we use a plain CountSketch matrix is that we found it has good performance for the two real-world datasets. We agree with the reviewer's point that OSNAP has a better worst-case guarantee. Hence, we have replaced CountSketch with ONSAP in our new version of the experiments. (Note that the new experiments are conducted on a different device, so the runtime is incomparable with that of the previous version.)

Reviewer comment (Experiments):
The runtime of a full, dense SVD is not the appropriate comparison in Table 1. The appropriate comparison is a randomized low‑rank approximation such as the randomized SVD or generalized Nyström (with OSNAP’s, if the latter).

Author response:
The Numerical Experiments.

We note that the reason we mentioned the time of the SVD is just as a reference time - we did not treat it as a baseline to beat. Indeed, the main point we want to illustrate from the previous version of the experiments is that the sparse sketch we use has a similar accuracy while having a much faster running time than the dense sketch. Following the reviewer's advice, we compare the following two approaches in our new version of the experiments:

- Randomized SVD: here we use the implementation in the sklearn Library in Python. Note that this method does not support the streaming setting and has a larger space requirement. 

- The algorithm in [CW13]:  which supports the streaming setting and computes a low-rank approximation directly.

Taking the parameters $m = 50$ and $k = 20$ for the KOS dataset as an example, we have:
- 0.120s  for our approach
- 0.230s for randomized SVD
- 0.660s for the streaming LRA 

Hence, we believe that our algorithm still has a runtime benefit when compared to more baselines for low-rank approximation. (The result of our reports is slightly inconsistent than that of the reviewer's. One possible reason is if the reviewer is using Matlab, then the built-in method is already heavily optimized, which could result in a closer runtime gap).

Reviewer comment (Novelty):
The residual error for $k$‑sparse approximation of vectors in $\ell_p$ norms for $p > 2$ is of interest to theoretical computer science but lacks interest for practical problems in machine learning.

Author response:
> “But noting that  heavy hitters can detect lighter heavy hitters, by itself, does not provide justification that the 
 residual estimation problem is of interest to machine learning. Is super-constant space cost of $n^{1-2/p}$
 acceptable in practice? "
 
Note that our algorithm indeed also outputs a vector $\hat{x}$ which satisfies the requirements for the $\ell_p$ sparse recovery problem, which was unknown for $p > 2$. We also provide a matching lower bound for the $\ell_p$ recovery problem.  As mentioned, our algorithm for $\ell_p$ is able to find heavy hitters that are even ``less heavy" than those for $\ell_2$, which is beneficial to, e.g., the standard Gaussian white noise model in practice for which the underlying vector is a sparse vector plus Gaussian noise (see. e.g., [WHXF11]), where now we can handle a smaller variance on the Gaussians. The ability to detect the existence of a large signal in compressed sensing is also useful, as we can decide to use more measurements only if we notice the existence of such a signal; otherwise we can use fewer. This would make the sensing adaptive, but this is also a common model in compressed sensing (see, e.g., [IPW12, MN13]).

Reviewer comment (Experiments):
Overall, this paper seems more suited to a dedicated TCS publication venue. The reviewer is unsure when they would run the proposed method in practice, and notes that the improvement from $O(k/\varepsilon^3)$ to $O(k/\varepsilon^2)$ has limited practical impact. The use of plain CountSketch in the numerical experiments is a significant concern; if this were fixed, they would consider a soft recommendation for acceptance.

Author response:
- The experiments: the reason we use a plain CountSketch matrix is that we found it has good performance for the two real-world datasets. We agree with the reviewer's point that OSNAP has a better worst-case guarantee. Hence, we have replaced CountSketch with ONSAP in our new version of the experiments. (Note that the new experiments are conducted on a different device, so the runtime is incomparable with that of the previous version.) 

- On use cases: there are natural settings in which we do not necessarily need to compute a low-rank approximation of all of the input matrices $A_1, A_2, \cdots, A_m$. Instead, we only need to compute an $\alpha$-fraction of them, based on which of them has a good low-rank approximation. In the case when $\alpha$ is small (such as $0.1$) and computing an actual low-rank approximation is much more expensive than estimating its cost, our algorithm will achieve a factor of roughly $\alpha$ savings in its resource requirements. In the streaming model, the space we need for our algorithm is $\mathrm{poly}(k)$, while if we want to compute a rank-$k$ approximation in a stream, the space we need would be $\Omega((n + m) \cdot k)$, which is larger than $\mathrm{poly}(k)$ when $k \ll m, n$. Also, the overall runtime of our algorithm for our rank-$k$ residual estimation algorithm is $O(\mathrm{nnz}(A) + \mathrm{poly}(k))$, while for a rank-$k$ approximation, the overall runtime is $O(\mathrm{nnz}(A) + (n + m) \cdot \mathrm{poly}(k))$. Hence, if the matrix $A$ is sparse enough, the second term of $\mathrm{poly}(k)$ will save a factor of $n$ from the latter complexity of $(n + m) \cdot \mathrm{poly}(k)$. 
For the $\ell_p$ sparse recovery problem, when $p \ge 2$, our algorithm is able to find heavy hitters that are even ``less heavy" than those for $\ell_2$, e.g., imagine a single coordinate $i^*$ has value $n^{1/3}$ while all other coordinates have value $1$. Then $i^*$ must be found by an $\ell_3$ sparse recovery algorithm, but not by an $\ell_2$ sparse recovery algorithm.

Reviewer comment (Theory):
The proof of Lemma 3.2 is not convincing. It is unclear why $d_{TV}(S(G), S(G+\beta \sqrt{\varepsilon/k} \cdot wz^\top) \le 1/10$ and $\beta \sqrt{\varepsilon/k} \cdot \|w\|\|z\|$ imply $d_{TV}(S(G), S(G + c\alpha\sqrt{\varepsilon} \cdot uv^\top)) \le 1/9$. The reviewer believes the result is probably true but does not follow the reasoning.

Author response:
- The proof of Lemma 3.2: We thank the reviewer for pointing this out. We have added more details to the proof of this lemma. Please check our new version of the submission. At a high level, here we have that $d_{TV}(X,Y)$ is small and wish to prove that $d_{TV}(X+Z,Y+Z)$ is small, where $Z$ is independent of $X$ but not of $Y$. In fact, we can show that $d_{TV}(X+Z,Y+Z) = d_{TV}(X,Y)$ is small when conditioned on a fixed $Z$ (the distribution of $Y$ depends on the choice of $Z$) over an overwhelming number of choices of $Z$. This is enough to conclude that the unconditioned value $d_{TV}(X+Z,Y+Z)$ is small.

Reviewer comment (Theory):
It is questionable whether data processing and linearity of the sketch are enough to conclude $d_{TV}(\mathcal{L}_1,\mathcal{L}_2)$. The perturbation $c\sqrt{\varepsilon}B$ is applied to both $G$ and $G + c\alpha \sqrt{\varepsilon} uv^\top$, and is dependent with the latter. The claim that if $d_{\rm TV}(X,Y) \le \varepsilon$, then $d_{\rm TV}(X+Z,Y+Z)\le \varepsilon$ is not generally true, as shown by a counterexample. The reviewer is unsure how the authors’ argument concludes the proof and suggests revising it to make the correctness unassailable.

Author response:
- The proof of Lemma 3.2: We thank the reviewer for pointing this out. We have added more details to the proof of this lemma. Please check our new version of the submission. At a high level, here we have that $d_{TV}(X,Y)$ is small and wish to prove that $d_{TV}(X+Z,Y+Z)$ is small, where $Z$ is independent of $X$ but not of $Y$. In fact, we can show that $d_{TV}(X+Z,Y+Z) = d_{TV}(X,Y)$ is small when conditioned on a fixed $Z$ (the distribution of $Y$ depends on the choice of $Z$) over an overwhelming number of choices of $Z$. This is enough to conclude that the unconditioned value $d_{TV}(X+Z,Y+Z)$ is small.

Reviewer comment (Writing):
There are some typos etc. in the paper which should be corrected: 1) In definition 3.4, $S$ should be $s \times n$ 2) The proof of theorem 3.7 should refer to lemma 3.5 instead of 3.4.

Author response:
We thank the reviewer for their thoughtful and detailed comments. We have fixed the typos mentioned and uploaded a new version of the paper.

Reviewer comment (Theory):
Can anything be said regarding how good $SAT$ is as compared to $A_k$, the best rank-k approximation to A?

Author response:
- Question 1: Note that the size of $SAT$ is different from that of $A_k$, so they are not comparable as matrices. We have shown though in Theorem 3.7 that the residual error of $SAT$ is a good approximation to the residual error of using $A_k$ for $A$. Note though that $SAT$ is a much smaller sketch of $A$ and so gives a low memory way of approximating the residual that one could have found by using $A_k$. Also, in the low-rank approximation problem, using matrices $SA, AR, SAT$, we can more efficiently (than computing $A_k$ directly) compute two matrices $U$ and $V$ such that
$
||A - UV||_F^2 \le (1 + \epsilon)||A - A_k||_F^2 \;.
$
(See the survey [1] for more details.)

Reviewer comment (Theory):
Confusion regarding the proof of Lemma 4.3: It seems we should have $(I \cap J) \cup T_1 \cup \ldots T_{m+1} \subseteq I$ instead of exactly being equal to $I$ as it could happen that $i \in I \backslash J$ and  $x_i > || x_{-k}||_p $  in which case $i$ wouldn't belong to any of the sets $ T_1, \ldots T_{m+1}$. It seems instead of decomposing I, decomposing J as $ J= (I \cap J) \cup T_1 \cup \ldots T_{m+1}$ where each $T_i$ contains elements of $J$ not in $I$ in the given interval and then doing the analysis should give the same result.

Author response:
- Question 2: We thank the reviewer for pointing this out. In fact, one can show that $\vert x_i\vert ^p \leq 2 \Vert x_{-k}\Vert_p^p$ for all $x_i \in I \setminus J$. Defining a new magnitude level for coordinates between $\Vert x_{-k} \Vert_p^p$ and $2\Vert x_{-k} \Vert_p^p$ is enough and the remainder of the proof is almost the same. The updated proof has been included in the new version which we have uploaded.

Reviewer comment (Novelty):
For the matrix case, the improvement compared to Andoni and Nguyen seem a bit incremental.

Author response:
On our improvement over Andoni and Nguyen: we improve the upper bound of Andoni and Nguyen from $O(k^2/\epsilon^6)$ to $O(k^2/\epsilon^4)$.  
Note that the $O(k^2/\epsilon^6)$ bound in their work was also only shown to hold for Gaussian matrices. In our work, we give a very different analysis than that in Andoni and Nguyen, and instead directly show that any sketching matrix with the so-called projection cost-preserving (PCP) property suffices, which allows us to directly use extremely sparse sketching matrices with a much faster running time, and also allows us to improve the bound to $O(k^2/\epsilon^4)$ using advances in PCPs. We also show an $\Omega(k^2/\epsilon^4)$ lower bound, which shows that our upper bound is optimal up to a constant factor.

Reviewer comment (Novelty):
While I think it makes sense to estimate the residual error of matrix approximations, I think there is little use to estimate the residual error of sparse recovery under $\ell_p$ norm.

Author response:
For the $\ell_p$ sparse recovery problem, when $p \ge 2$, our algorithm is able to find heavy hitters that are even ``less heavy" than those for $\ell_2$, e.g., imagine a single coordinate $i^*$ has value $n^{1/3}$ while all other coordinates have value $1$. Then $i^*$ must be found by an $\ell_3$ sparse recovery algorithm, but not by an $\ell_2$ sparse recovery algorithm.