Paper: Asymptotically Free Sketched Ridge Ensembles: Risks, Cross-Validation, and Tuning (ICLR-2024)

Reviewer comment (Reproducibility):
It would be beneficial to provide more details regarding computational aspects in the main text. For example, it may not be necessary to compute $\hat{\beta}_{\lambda}^k$ alone in equation (1).

Author response:
We agree with you and the other reviewers regarding this point and have now added a new Appendix H providing a detailed computational comparison between unsketched GCV, $k$-fold CV, and the ensemble trick, clarifying that the computational demands decrease for smaller sketch sizes. 
Indeed, as you have written, it is not necessary to "broadcast" $\hat{\beta}_\lambda^{S_k}$ back to $p$-dimensional space, and all computation can (and should) be done only in the sketched domain; our presentation is simply made for mathematical convenience. 
In fact, for sketches based on hash functions such as CountSketch, computing the adjoint of the sketching operator is in general intractable, so this point is critical in sketched machine learning practice.

Reviewer comment (Reproducibility):
It is only necessary to compute the predicted values $X \hat{\beta}_{\lambda}^k$ rather than $\hat{\beta}_{\lambda}^k$ alone.

Author response:
We agree with you and the other reviewers regarding this point and have now added a new Appendix H providing a detailed computational comparison between unsketched GCV, $k$-fold CV, and the ensemble trick, clarifying that the computational demands decrease for smaller sketch sizes. 
Indeed, as you have written, it is not necessary to "broadcast" $\hat{\beta}_\lambda^{S_k}$ back to $p$-dimensional space, and all computation can (and should) be done only in the sketched domain; our presentation is simply made for mathematical convenience. 
In fact, for sketches based on hash functions such as CountSketch, computing the adjoint of the sketching operator is in general intractable, so this point is critical in sketched machine learning practice.

Reviewer comment (Presentation):
It may not be necessary to explicitly premultiply $S_k$ to $\hat{\beta}_{\lambda}^{S_k}$ in equation (1). If correct, this point can be emphasized at the end of section 2 when discussing that matrix inversions are inexpensive after precomputing $X S_k$.

Author response:
We agree with you and the other reviewers regarding this point and have now added a new Appendix H providing a detailed computational comparison between unsketched GCV, $k$-fold CV, and the ensemble trick, clarifying that the computational demands decrease for smaller sketch sizes. 
Indeed, as you have written, it is not necessary to "broadcast" $\hat{\beta}_\lambda^{S_k}$ back to $p$-dimensional space, and all computation can (and should) be done only in the sketched domain; our presentation is simply made for mathematical convenience. 
In fact, for sketches based on hash functions such as CountSketch, computing the adjoint of the sketching operator is in general intractable, so this point is critical in sketched machine learning practice.

Reviewer comment (Reproducibility):
Generally speaking, it would be helpful to provide more details regarding computational aspects.

Author response:
We agree with you and the other reviewers regarding this point and have now added a new Appendix H providing a detailed computational comparison between unsketched GCV, $k$-fold CV, and the ensemble trick, clarifying that the computational demands decrease for smaller sketch sizes. 
Indeed, as you have written, it is not necessary to "broadcast" $\hat{\beta}_\lambda^{S_k}$ back to $p$-dimensional space, and all computation can (and should) be done only in the sketched domain; our presentation is simply made for mathematical convenience. 
In fact, for sketches based on hash functions such as CountSketch, computing the adjoint of the sketching operator is in general intractable, so this point is critical in sketched machine learning practice.

Reviewer comment (Presentation):
In figure 1, it would be good to indicate that SRDCT refers to subsampled randomized discrete cosine transform because SRDCT first appears toward the end of page 2.

Author response:
We have revised the caption of Figure 1 to include a definition for SRDCT, ensuring that it is clear on its first appearance in the text.

Reviewer comment (Theory):
In proposition 6, $S_k^T S_k$ is assumed to be invertible. How strong is this assumption? Some further remarks might be helpful.

Author response:
The invertibility assumption for $\mathbf{S}_k^\top \mathbf{S}_k$ is merely technical.
If it is not invertible, it is straightforward to verify that the sketched ridge regression problem is unchanged if we replace $\mathbf{S}_k \in \mathbb{R}^{p \times q}$ by its projection $\widetilde{\mathbf{S}} \in \mathbb{R}^{p \times r}$ onto its right principal singular vectors, where $r = \mathrm{rank}(\mathbf{S}_k)$ takes the place of $q$. 
Thus, Proposition 7 holds for any sketch if we replace $\alpha$ by $r/p$, but we thought it would be better to tie into the same $\alpha$ used in the rest of the paper. 
Additionally, to use a rank-deficient sketch in practice would generally be wasteful, so we don't anticipate this to be a common case.

Reviewer comment (Writing):
Improvment of clarity in Assumption 2 statement and explanation. The paper has the merit to introduce elements of free probability theory useful in Assumption 2. I regret not to have more intuition here: I can easily imagine that a form of independence (I've read the appendix) between $X^TX$ and $SS^T$ is useful but the notion of limiting S-transform is not at all discussed at this stage (page 4) and Assumption 2 remains not clear at all when beginning to read what follows. Same thing for Theorem 1, describe the $|ambda^+$ function.

Author response:
We acknowledge that Assumption 2 (now Assumption A after our revision) and Theorem 1 could benefit from additional intuition, especially concerning the S-transform and its role in our results. 
We have added a new footnote on page 4 to ground the assumption concretely with Gaussian and orthogonal sketches, which are commonly studied and proven to be asymptotically free.
To clarify regarding freeness, the independence between $\mathbf{X}$ and $\mathbf{S}$ is not the only requirement; what is crucial is a form of incoherence, often achieved through rotational invariance of the sketching matrix.
This incoherence is captured by the concept of asymptotic freeness in free probability theory.

The S-transform, another key concept from free probability theory, is indeed opaque and difficult to give intuition for. Despite this, it is a standard notion in free probability, and the main thing that matters for our results is that is a function only of the eigenvalue distribution of the sketching matrix $\mathbf{S} \mathbf{S}^\top$.
We assume analyticity of the S-transform, which holds for both Gaussian and orthogonal sketches, but we do not know in general what requirements this places on sketching matrices.

As we define immediately before Theorem 1, $\lambda_{\min}^+$ is the minimum non-zero eigenvalue of the matrix. 
This arises because the multiplication by $\mathbf{X}$ effectively nullifies the zero eigenvalues, leaving only the non-zero spectrum relevant for determining singular inversion, and enabling us to consider zero and negative regularization.

Reviewer comment (Evaluation):
A simple analysis of the complexity in time and memory for the full aproach in constrat to other estimators (CV) would be welcome.

Author response:
We have now included a new Appendix H performing such a comparison. In particular, we demonstrate that the ensemble trick is always competitive with unsketched GCV (which is in turn generally more efficient than $k$-fold CV) as long as $q < p / 2$, and can be competitive even more generally in the right circumstances.

Reviewer comment (Evaluation):
Bonus: Is it interesting to come back on other risk estimators (Bootstrap) and clearly identify what could be done with this estimator or not with ensemble of sketched ridge regression.

Author response:
We agree that contrasting our approach with other risk estimators, such as the bootstrap, could be valuable. 
The bootstrap, however, does not provide an estimator for prediction risk
directly and has its computational challenges when dealing with large datasets. 
Our approach offers both computational efficiency and statistical consistency, which are crucial for practical applications.

Reviewer comment (Experiments):
Asymtpotic freeness seems to be an essential assumption in the paper. Although this assumption is experimentally supported by artifical datasets, I wonder if one can observe similar matching on real-world dataset.

Author response:
> **Although [asymptotic freeness] is experimentally supported by artificial datasets, I wonder if one can observe similar matching on real-world dataset.**

Given the importance of the asymptotic freeness assumption, we agree that it is crucial to verify this assumption in realistic settings, such as with real-world data and practical sketches.

Recall that for Gaussian sketches and uniformly random orthogonal sketches, asymptotic freeness is theoretically guaranteed for any data matrix $X$ independent of $S$. For other types of sketches, such as CountSketch or SRDCT, the validation of the asymptotic freeness assumption is indeed more complex.

However, as you suggest, we can employ further empirical tests to support our claims on real-world data. We have added a new experiment in Figure A.2 in Section A.3.2, where we showcase empirical subordination relations on the real-world datasets we use in Figure 2 (RCV1 and RNA-Seq), similar to the previous experiment in Figure A.1 on artificial data. These results show that the resolvent relation given in Theorem 1, which underpins our theoretical results, holds with good accuracy on real datasets as well.

We hope that these empirical validations, along with the theoretical underpinnings for more readily analyzable Gaussian and orthogonal sketches, offer a compelling argument for the practical relevance of our theoretical contributions.