Paper: Improved algorithm and bounds for successive projection (ICLR-2024)

Reviewer comment (Novelty):
This paper seems to be a incremental work based on Gillis & Vavasis (published in 2013).

Author response:
Dear All, 

1. Reviewer WFSc said "This paper seems to be a incremental work based on Gillis & Vavasis (published in 2013)".
2. A couple of minutes ago,  Review KsqF posted a comment saying that the above comment by WFSc is incorrect, where he/she further said "he proofs are clearly not the same. Working on the same problem/algorithm is not incremental work no matter how you spin it".

We thank all reviewers for their valuable time and input, and we especially thank Review KsqF for his/her timely comment. While we are preparing a more thorough response letter, we just wish to point out that 

-- We have two contributions: a new non-asymptotic bound (which is what review WFSc commented on) and a new algorithm and a set of new results. Gillis and Vavasis's work is only related to the first one (the second contribution goes far beyond Gilis and Vavasis work and is not related. 

-- Even our first contribution is not an incremental work based on Gillis and Vavasis: you will agree if you read our statement and proofs. Here is a simple way to see it. Our bound says 

max_{1 leq i \leq K} |\hat{v}_k - v_k| <= C / s_{K-1}(\tilde{V}), 

while Gillis and Vavasis' bound says 

max_{1 leq i leq K} |hat{v}_k - v_k| <= C / s_k (V). 

In the denominator of Gillis and Vavasis bound, s_K(V) does not depend on the volume of the simplex (in fact, it depends on the location of the simplex). Therefore, even when the volume is large (but the location is bad), s_K(V) is 0. In such a case, the RHS of their bound is infinity, and so the bound is not useful. In our bound, s_{K-1}(tilde{V}) can be proved to be lower bounded by a simple function of the volume, so as long as the volume of the simplex is large, s_{K-1}(tilde{V}) is lower bounded from 0. Therefore, our bound is significantly better in this sense. Note that an optimal bound should depend on the volume but not the location of the simplex (a simplex with a small volume is nearly flat so vertex hunting is impossible; however, the location of the simplex should not matter if we have the right algorithm or the right proof). 

Given such a significant difference, I think you will agree that no one can deduce our results from Gillis and Vavasis (otherwise, 
I think Gillis and Vavasis will improve their bounds a long time ago). 

Thanks again for your valuable time and input. We will be in touch soon!

Reviewer comment (Experiments):
There is only limited numerical experiments to validate the peformance of the proposed algorithms. (c.f. Figure 2).

Author response:
Thank you for your comments and for appreciating that "this paper proposes a novel algorithm" and "the analysis seems to be solid". We already posted a quick response earlier. This current response is to supplement the earlier one with more details. We also encourage you to read our Response-to-All-Reviewers for other clarifications. 

### Main Points: 

**1. Comparison with [GV13]**: *"This paper seems to be an incremental work based on Gillis & Vavasis (published in 2013)."*

We are sorry to hear this comment, but we believe it is due to some misunderstanding. It is our fault that our initial submission did not explain why the proof is non-trivial. We hope that you will change your mind after seeing the clarification here.   

Our paper has 3 major contributions: 
- (a) A new algorithm 
- (b) the non-asymptotic bound for SPA, 
- (c) the error bound for pp-PSA. 

Only (b) is related to Gillis & Vavasis (2013) [GV13]. Even without (b), the contributions in (a) and (c) are already significant. For example, our analysis in pp-SPA uses theory of extreme order statistics to analyze the effect of KNN denoise, which is subtle. 

Regarding the contribution in (b), we guess you were actually asking: "Is the proof a simple modification of the proof in [GV13]?" 

The proof is completely different. The proof in [GV13] is based on controlling matrix norms. Their proof does not use explicitly the geometry behind this problem. To see this, we recall that $\beta(X)$ is equal to the maximum of $ \Vert\epsilon_i\Vert$ among all points,  and $\beta_{\text{new}}(X)$ is roughly the maximum of $\Vert\epsilon_i\Vert$ among those points outside the true simplex. In the proof of [GV13], they frequently control the difference of replacing $X$ by $\mathbb{E}[X]$ in many intertermediate matrices. Such steps are feasible only if one has a bound for the maximum of *all* $\Vert\epsilon_i\Vert$'s - i.e., the quantity $\beta(X)$. However, it is hard to repeat these steps by using $\beta_{\text{new}}(X)$. Similarly, if one has a lower bound for $s_K(V)$, one can control the inverse of $V'V$, but a lower bound for $s_{K-1}(V)$ cannot do so. 
To tackle these difficulties, we use a different proof approach. Our proof relies more on the geometry behind this problem. We define a "simplicial neighborhood" for each true vertex and prove that each estimated vertex must fall into one of these simplicial neighborhoods. This is a new proof strategy. 

In addition, we have added a numerical example in this revised draft, where we calculate the two bounds explicitly. In some cases, the new bound is only 0.01 times the old bound in [GV13]. 

In summary, we agree that [GV13] is an influential work, but our paper is not incremental of [GV13]. 

**2. More simulations**: *"There is only limited numerical experiments to validate the performance of the proposed algorithms"*

Thanks for your comment. We have significantly expanded our simulations:
- We have included settings for every $d=1,2,..., 48, 49, 59$. 
- We have added the comparison with robust SPA in all simulation settings. 

The updated Figure 3 contains nearly 90 different settings. We hope that the numerical experiments are sufficiently rich now.

Reviewer comment (Experiments):
Will more samples contribute to the reconstruction performance? My intuition is that the reconstruction performance will improve with more observations, that is, the reconstruction error will decrease with an increasing sample $n$. However, the relation between sample number $n$ and the reconstruction error is largely missing.

Author response:
### Other questions: 

1. The relation between sample number $n$ and the reconstruction error. 

The reconstruction error is $O(\sqrt{d+\log(n)})$ for SPA and $O(\sqrt{\log(n)})$ for pp-SPA. In addition, in some special cases (e.g., Theorem 2), the error of pp-SPA can be further reduced to nearly $O(1)$. 

At first glance, it is a bit counter-intuitive that increasing $n$ does not decrease the rate. The main reason is that the configuration of population points $r_1, r_2, \ldots, r_n$ can be arbitrary. Here is a toy example. Suppose 
$$
r_k = v_k, \qquad 1\leq k\leq K, \qquad r_i = \bar{v}:= (1/K)\sum_kv_k, \qquad\mbox{for all the remaining $(n-K)$ points}. 
$$
In this example, no matter how large $n$ is, there is only one point, $X_k$, that is useful for estimating $v_k$. This is why increasing $n$ does not help reduce the error bound.   

This insight is further supported by simulations. On the right panel of Figure 3, we plot the error curve as $n$ increases. It shows that the error does not decrease with $n$, which is consistent with our theory.

Reviewer comment (Experiments):
How is the algorithm compared with the matrix completion solution? To be more specific, we can stack the observations and obtain the sensing relation $X = V\Pi + N$, where observation $X_i$ is the $i$th column of matrix $X$, and $N$ is the sensing noise. Easily, we can verify that $V\Pi$ is a matrix with rank at most $K$. Hence, we can view matrix $X$ as a linear combination of low-rank matrix and noise matrix. Then, we exploit the research on low-rank matrix reconstruction and solve the problem.

Author response:
### Other questions: 

2. How is the algorithm compared with the matrix completion solution?

This is an interesting comment. The main difference is: We are interested in recovering $V$, but the matrix completion literature is interested in recovering $V\Pi$. We can find $\widetilde{V}\in\mathbb{R}^{d\times K}$ and $\widetilde{\Pi}\in\mathbb{R}^{K\times n}$ such that $\widetilde{V}\widetilde{\Pi}=V\Pi$ but $\widetilde{V}\neq V$. 

For example, let's consider a special case with $K=3$ and $d=2$. The true simplex is a triangle in $\mathbb{R}^2$. We now consider an arbitrary triangle that contains the true triangle in its interior. Let $\widetilde{V}$ contain vertices of this new triangle, and let $\widetilde{\Pi}$ contain the barycentric coordinates of $r_i$'s in this new triangle. Then, $V\Pi=R=\widetilde{V}\widetilde{\Pi}$. However, $V\neq \widetilde{V}$. 

This example suggests that it is impossible to naively apply matrix completion. However, there may be a way to borrow some ideas from matrix completion literature to further improve the current algorithms. We leave this interesting direction to future work.

Reviewer comment (Theory):
In comparing the performance between Theorem 1 over Lemma 4, what if we have $\gamma(V) \ll s_{K-1}(V)$? In such case, lemma 4 may be more tight then Theorem 1.

Author response:
### Other questions: 

3. What if we have $\gamma(V)\ll s_{K-1}(V)$?

Thanks for the comment. This will never happen, as a result of Lemma D in the supplementary material. 

For your convenience, we re-prove Lemma D here. By definition,  $\gamma^2(V)$ is the maximum diagonal entry of $V'V$. Note that
$$
K\gamma^2(V)=K\max_{1\leq k\leq K}\Vert v_k\Vert^2\geq \sum_{k=1}^K \Vert v_k\Vert^2
=  \mathrm{trace}(V'V).
$$
In addition, let $s_k(V)$ be the $k$th singular value of $V$, so that $s_k^2(V)$ is the $k$th largest eigenvalue of $V'V$. It follows that
$$
 \mathrm{trace}(V'V)  = \sum_{k=1}^K s_k^2(V)\geq \sum_{k=1}^{K-1} s_k^2(V)\geq \sum_{k=1}^{K-1} s_{K-1}^2(V) = (K-1) s_{K-1}^2(V). 
$$ 
Combining the above, we have
$
 \gamma(V) \geq \sqrt{\frac{K-1}{K}} s_{K-1}(V)\geq \frac{1}{\sqrt{2}}s_{K-1}(V).
$

Reviewer comment (Presentation):
It is not clear what the illustration in Figure 1 shows. It is unclear whether it is obvious that one of these is better and what "idea simplex" denotes.

Author response:
**1. Figure 1**: *"It is not clear to me what the illustration in Figure 1 shows? What does 'idea simplex' denote?"*

Thanks for the comment. In this revision, we have combined two panels of Figure 1 into a single plot, to make it easier to compare the true vertices and the estimates from SPA and pp-SPA. The  "ideal simple" means the ground-truth simplex formed by the true vertices $v_1,v_2,\ldots,v_K$. We have changed the terminology "ideal simplex" to "the simplex formed by the true vertices" to make it clear.

Reviewer comment (Writing):
At times a statement is given without citation. For example, in "Our contributions" on page 2, you state: "since the SPA is greedy algorithm, it tends to be biased outward bound," but it is not clear where this can be seen.

Author response:
**2. Citations to the biased outward bound**: *"At times a statement is given without citation,... pg 2: 'since the SPA is greedy algorithm, it tends to be biased outward bound.', but it is not clear where this can be seen."*

We realized that since we put SPA and pp-SPA in two separate panels in the old Figure 1, the "biased outward bound" was not easy to see. In the updated Figure 1, the two simplexes are put together, so this phenomenon is more obvious to see. 

This phenomenon is caused by the way SPA is designed: At each iteration, it selects $\hat{v}_k$ by finding the point furthest to the original. No matter the original is inside or outside the true simplex, this will yield "biased outward bound". We think this phenomenon was already recognized in the robust SPA literature, although not using the exact phrase. In this revision, we added a reference Gillis (2019) in this paragraph.

Reviewer comment (Theory):
It is very difficult to follow the motivation of Theorem 1 and how it shows that the previous results of (Gillis & Vavasis, 2013) are non-satisfactory. It is also very difficult to see the comparison with the bound in (5) for $\beta_{new}$.

Author response:
**3. The bound in Theorem 1**: *"How it shows that the previous results of (Gillis & Vavasis, 2013) is non-satisfactory. It is very difficult to see the comparison with the bound in (5) for $\beta_{new}$."*

Your first question is about why the bound in [GV13] is non-satisfactory. We now give a toy example where we fix $(K, d) = (3,3)$, $v_1=(20,20,a)$, $v_2=(20,30,a)$, and $v_3=(30,20,a)$, for a parameter $a\in\mathbb{R}$. 
Recall their bound and our bound are  $g(V) \cdot \beta(X)$ and $g_{\text{new}}(V) \cdot \beta_{new}(X,V)$, respectively, where $\beta_{new}(X,V)\leq \beta(X)$ is easy to see. We now compare $g_{\text{new}}(V)$ and $g(V)$. By direct calculations, we get the following table. It demonstrates that the improvement is substantial. 
$$
\begin{array}{ccccccc}
    a & 10 & 110 & 210 & 310 & 410 & 510 & 610 & 710 & 810 & 910\cr
    \frac{g(V)}{g_{\text{new}}(V)} &114.8 & 11.2 & 27.6 &27.2 &14.6 &9.7 &7.9 &14.4 & 10&24.6 
  \end{array}
$$
The main reason is that $s_{K-1}(V)$ is driven by the volume of the simplex, but $s_K(V)$ is driven by the location (center) of the simplex. As $a$ varies, the location of the simplex can change significantly, making $g(V)$ unsatisfactory sometimes. 

Your second question is about how to interpret $\beta_{\text{new}}(X,V)$. In this revision, we added another toy example and a new figure, Figure 2, to illustrate this quantity. Intuitively speaking, 
- $\beta(X)$ is the maximum noise in all all data points, no matter whether the data points fall inside or outside the true simplex. 
- $\beta_{\text{new}}(X, V)$ only cares about the maximum noise of data points outside the true simplex or near the boundary of the true simplex.  

As a result, we can add a large noise to any point deeply in the interior of the true simplex without changing $\beta_{\text{new}}(X,V)$, but $\beta(V)$ will be heavily increased. This improvement of $\beta(V)$ to $\beta_{\text{new}}(X,V)$ is critical in the analysis of the new algorithm, pp-SPA. The KNN denoise idea only helps decrease the noise at those points outside the true simplex. If we stick to the original non-asymptotic bound in [GV13], then we will not be able to derive the sub-sequent bounds for pp-SPA.

Reviewer comment (Experiments):
The numerical experiments are done only for very small sizes, and it is unclear how significant the effect of denoising using low-dimensional PCA projection is in this case.

Author response:
**4. Simulations for larger $d$**: *"The numerical experiments are done only for very small sizes. I am not sure how big is the effect of denoising using low-dimensional PCA projection in this case."*

This is a great point. We have expanded Experiment 1 by considering every $d$ from $1$ to $50$. The effect of low-dimensional PCA projection can be seen by comparing pp-SPA and D-SPA (a simplified version with the projection step skipped). We see that the estimation error of D-SPA deteriorates as  $d$ gets larger. However, the estimation error of pp-SPA is almost invariant with respect to the increase of $d$. This shows that the hyperplane projection is crucial for the performance, especially when $d$ is large.

Reviewer comment (Theory):
How does the theory differ in the case of 3-simplex in 2D? Is there a difference in the $K$ vs $K-1$ singular value?

Author response:
*"How does the theory differ in the case of 3-simplex in 2D? Is there a difference in the $K$ v.s. $K-1$ singular value?"*

Thanks for the interesting question. Let's consider two cases: (a) The triangle lives in $\mathbb{R}^2$. Then, $V$ is a $2\times 3$ matrix that has only $2$ singular values. The 3rd singular value is exactly zero. (b) The triangle lives in a hyperplane in $\mathbb{R}^3$. In this case, $V$ is a $3\times 3$ matrix, where 2nd singular value is nonzero as long as the triangle is non-degenerate (i.e., it does not collapse to a line segment). However, the 3rd singular value depends on the location of the hyperplane. When the hyperplane crosses the origin, the 3rd singular value can be zero. 

The above conclusions can be deduced from Lemma 3 in the revised draft.

Reviewer comment (Experiments):
How does the algorithm perform for higher dimensional problems?

Author response:
*"How does the algorithm perform for higher dimensional problems?"*

Please see our response to your Main Point 4 (in Part 1 of our responses).

Reviewer comment (Experiments):
For some reason, the authors do not compare pp-SPA with any robust SPA. I found many results in the literature on robust SPA, so it seems bizare it is not included in the experimental section here. Please try to include at least one robust variant of SPA/similar algorithm in the experimental section.

Author response:
**1. Robust SPA**: *"Please try to include at least one robust variant of SPA/similar algorithm in the experimental section."*   

This is a great point. In the revision we have added the comparison with robust-SPA (Gillis, 2019) in all simulations. We used the Matlab code downloaded from $\texttt{bit.ly/robustSPA}$. 
The results are included in the updated Figure 3. We observe that pp-SPA outperforms robust-SPA, especially when $d$ is large. 

We also added references to recent variants of SPA, including robust SPA, pre-conditioned SPA, rank-K approximate SPA, etc.. The hyperplane projection and KNN denoise idea have never been used in these algorithms. Furthermore, none of existing works has got as tight bounds as in our work.

Reviewer comment (Writing):
The writing can be dramatically improved. See below minor points, but if this is accepted, please spent a few iterations on improving the writing. It's extremely terse at time.

Author response:
**2. Writing**: *"If this is accepted, please spent a few iterations on improving the writing. It's extremely terse at time."*   

In this revision, we have already re-written some sections (a summary of changes is in our response-to-all-reviewers). We tried to use figures/examples/words to explain the math and notations. Below are a few examples.  
- New figure 2 and text therein: They provide an intuitive explanation of $\beta_{\text{new}}(X,V)$ in the non-asymptotic bound. 
- The numerical example below Theorem~1: We explicitly calculate the bound in Gillis and Vavasis (2013) and our new bound in a numerical example, to show why the improvement can be substantial in some cases. 
- Table 1: This table is a summary of rate comparison and has been moved to the beginning of Section 4. 

If the paper is accepted, we are willing to spend more time improving the writing.

Reviewer comment (Novelty):
You reprove several "elementary" results that I am almost certain must exist in the literature. Please don't do this, and spend some time to search for the result you need and cite appropriately. You don't need to prove everything from first principles! For example, Lemma 1 can be given as a HW question for an honors linear algebra class....

Author response:
**3. Whether to include the proof of Lemma 1**: *"You don't need to prove everything from first principles! For example, Lemma 1 can be given as a HW question for an honors linear algebra class."*

We agree that Lemma 1 is a standard result regarding subspace approximation via SVD, which appears in tutorial materials. However, we hope to prove this lemma in the supplementary materials for completion, and we think doing so will increase the readability of our paper. We did not intend to have any credit for such results. 

In this revision, we added a sentence in the first line of proof saying that "this is a quite standard result, which can be found at tutorial materials (e.g., https://people.math.wisc.edu/~roch/mmids/roch-mmids-llssvd-6svd.pdf). We include a proof here only for convenience of readers."

Reviewer comment (Writing):
The statements of Theorem 2/3 are an absolute nightmare, and I struggle to see how anyone would cite/use your result without a ton of heavy lifting in understanding your proofs.

Author response:
**4. Statements of Theorems 2-3**: *"I struggle to see how anyone would cite/use your result without a ton of heavy lifting in understanding your proofs."*

We recognized where the confusion came from. In order to make the results as general as possible, we use $\alpha_n$, $b_n$, and $t_n$ in the theorem statements. To get the simple expressions of rates, one still needs to plug in the orders of $(\alpha_n, b_n, t_n)$ from the theorem body. We agree that this is not reader-friendly. 

In the revision, we have explicitly explained how to plug in the order of $(\alpha_n, b_n, t_n)$ to get simplified bounds. For example, in the newly added paragraph below Theorem 2, we showed carefully how to use Theorem 2 to get a bound like
$$
\max_k \|\hat{v}\_k -v_k\|\leq \sigma g_{\text{new}}(V)\cdot  \begin{cases}
\sqrt{\log\log(n)} & \text{ if $d/n$ is properly small};\cr
\sqrt{[2+o(1)] \log (n)} & \text{ if $d/n$ is properly large}.
\end{cases}
$$
In addition, we have included a table in the beginning of Section 4 to summarize all the bounds.

Reviewer comment (Writing):
Page 2 "and so on and so forth" Akward, please change.

Author response:
- Editing of text: We have made all the changes as you suggested.

Reviewer comment (Theory):
"However, since the simplex lies on a hyperplane of (K − 1)-dimension, it is inefficient if we directly apply SPA to X1, X2, . . . , Xn" — This needs more explanation.

Author response:
- An explanation of "it is inefficient if we directly apply SPA":  We added more explanations on page 2 as follows: "However, since the true simplex vertices $v_1,\ldots,v_K$ lie on a hyperplane of $(K-1)$-dimension, if we directly apply SPA to $X_1, X_2, \ldots, X_n$, the resultant hyperplane formed by the estimated simplex vertices is very likely to deviate from the true hyperplane, due to noise corruption. This will cause inefficiency of SPA."

Reviewer comment (Evaluation):
"The results suggest that the SPA may be significantly biased outward bound, and there is a large room for improvement" — No! This result suggests that SPA is biased outward bound for this example! Why would this suggest it is a general phenomenon?! Actually, how do you know your own algorithm doesn't display this phenomenon for other examples?

Author response:
- If bias outward of SPA is a general phenomenon: This is indeed a general phenomenon (e.g., see Gillis (2019)), and it is caused by the way SPA is designed.  At each iteration, the estimated vertex is (by definition) the point in the projected data cloud that is *furthest away* from the origin. Therefore,  the output vertices are likely those points that lie on the furthest positions away from the center of the data cloud. This is why we always have the biased outward bound phenomenon. It is now seen more clearly in the updated Figure 1.

Reviewer comment (Writing):
"he triangles in black, green, and cyan are the true simplex (which is a triangle since (K −1) = 2), the simplex estimated by SPA, and the simplex estimated by D-SPA (D- SPA is a special case of pp-SPA where the projection step is skipped and pp-SPA is a new approach to be introduced; see below), respectively." — Rewrite this sentence!

Author response:
- Editing of text: We have made all the changes as you suggested.

Reviewer comment (Writing):
Page 2 "Roughly say" --> Informally, ...

Author response:
- Editing of text: We have made all the changes as you suggested.

Reviewer comment (Novelty):
"successive projection algorithm (SPA) (Ara´ujo et al., 2001)" — The "term" successive projection algorithm might've originated from the Araujo paper from 2001, but I can't believe that this algorithm was first discovered in 2001. To start off, it has similarities to the Frank-Wolfe minimum norm point algorithm, and several polyhedral algorithms. Can you please verify where the original **idea** was derived, to keep historical accuracy?

Author response:
We realize that we did not respond to your minor point about Frank-Wolfe minimum norm point (MNP) algorithm, and we add it here. 

The Frank-Wolfe (Rank and Wolfe, 1956) and MNP (Wolfe, 1976) seemed to be two algorithms. The MNP algorithm is more relevant, which aims to find the smallest Euclidean norm in a polytope. The major difference is however, in Wolf's setting, the polytope is known but in our setting, the simplex is unknown. In a later round, we are happy to add some references of Wolfe's works and connect it to the history of SPA.

Reviewer comment (Novelty):
Lemma 1 (Best-fit hyperplane) — There is no way this result isn't known. I can literally prove this result without even looking at your Appendix with Lagrangian multipliers and straightforward linear algebra identities. Please search for where this result was first proved, and don't reprove old things unless your proof is starkly different (which is absolutely not the case here)

Author response:
**3. Whether to include the proof of Lemma 1**: *"You don't need to prove everything from first principles! For example, Lemma 1 can be given as a HW question for an honors linear algebra class."*

We agree that Lemma 1 is a standard result regarding subspace approximation via SVD, which appears in tutorial materials. However, we hope to prove this lemma in the supplementary materials for completion, and we think doing so will increase the readability of our paper. We did not intend to have any credit for such results. 

In this revision, we added a sentence in the first line of proof saying that "this is a quite standard result, which can be found at tutorial materials (e.g., https://people.math.wisc.edu/~roch/mmids/roch-mmids-llssvd-6svd.pdf). We include a proof here only for convenience of readers."