Paper: Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning (ICLR-2024)

Reviewer comment (Presentation):
I do not see any precise discussion of Barlow Twins or VICReg in the main paper, however Proposition 1 informally states a result related to these formulations. I wonder what is the value in this --- a reader who knows about these formulations might dismiss the informal result (looking at the Appendix instead), and a reader who does not know about these formulations is left completely in the dark. Perhaps section 3.4 could be removed and replaced with a sentence referring to appendix D. (Presentation -)

Author response:
Thank you for raising section 3.4.
On reflection, we agree that (what was) Proposition 1 could be better integrated with the main text, and that it may be confusing to readers who do not know about Barlow twins and VICReg.

We have therefore taken the opportunity to rewrite the section.
Our proposals include:
- Focusing on VICReg - and writing the VICReg loss explicitly. (there was not enough space to state both losses, and we have more complete theoretical analysis for VICReg than Barlow twins).
- Replacing the 'informal' proposition with a sequence of bullet point conclusions.
- Brief sentences to better motivate why these results are of interest to the SSL community.

Do you agree that these changes improve the presentation? (and is there more we can do to improve this section?)

Reviewer comment (Experiments):
I checked the appendix I.3.4 but was not able to properly understand basic properties of dataset UK Biobank. Are the features in this data categorical, integer, real-valued, mixed, ... ? What are the modelling and "convenience" considerations when applying your method? In section 5.4, it is mentioned that "this can reveal novel phenotypes of interest and uncover genetic mechanisms of disease and brain morphometry." What is the precise task that is being performed here, with respect to the PLS formulation (e.g. with reference to the symbols in equations 4, 5, 6, 7)? (Presentation -, and perhaps Soundness -)

Author response:
We have also made some small improvements in response to your question about the Partial Least Squares task - thank you for raising this.
- *features:* the UK BioBank consists of real-valued continuous brain volumes and ordinal, integer genetic variants. We have added this information to appendix I.3.4.
- *modelling and "convenience" considerations:* could you please clarify what you are referring to here - we do not believe we used the word "convenience" in this UK Biobank context. 
- *precise task:* it is now standard in Imaging-Genetics research to perform (full sample) CCA or PLS on multi-view datasets, in order to interpret the estimated directions or to use transformed variables for downstream tasks. We use stochastic PLS in this scenario because the dataset is extremely high dimensional and has far more features than samples ($D \gg N$) so un-regularised CCA would break down - moreover, it is more convenient to run a single PLS than to perform tuning parameter selection for ridge CCA.
  With regard to equations 4, 5, 6, 7: we (hoped to) define PLS through our 'Unified GEP formulation' (equation 7) with $\alpha_i = 1 \, \forall i$, and here there are two views ($I=2$).
  In this case equation 7 specialises to give the GEP defined by
$$A = \begin{pmatrix} 0 &\operatorname{Cov}(X^{(1)}, X^{(2)}) \\\\ \operatorname{Cov}(X^{(2)}, X^{(2)}) & 0 \end{pmatrix}, \quad
	B = \begin{pmatrix}I_{D_1} & 0 \\\\ 0 & I_{D_2} \end{pmatrix}, \quad
	u =\begin{pmatrix}	u^{(1)} \\\\ u^{(2)} \end{pmatrix}.$$
	We have also added this equation to appendix I.3.4, with a short discussion.
	
Does that answer your question?

Reviewer comment (Theory):
It is not clear what is meant by "which should be true in general" in footnote 3. Perhaps if the observations are drawn from a continuous probability distribution, with probability 1 they are linearly independent?

Author response:
Yes, assuming the observations are from a continuous probability distribution is sufficient to give linear independence with probability 1. We have modified the footnote accordingly.

Reviewer comment (Writing):
Check the references. E.g. "stochastic optimization for pca and pls" should be "stochastic optimization for PCA and PLS". Perhaps the authors could also seek advice from the area chair (if the paper is accepted) about the most appropriate way to cite the multiple papers with huge author lists (this might be common in other fields where these papers are published, but in ML it looks really bizarre).

Author response:
References: we have fixed the capitalisation and will seek advice from the area chair regarding the long author lists.

Reviewer comment (Theory):
I am having trouble with Lemma 3.2. When we assume that there is a final linear layer in each neural net, does this parameter $\theta$ only relate to this final layer, or to the whole neural net? How then is it intuitively possible that $\theta$ could be a local optimum?

Author response:
Thank you for raising Lemma 3.2.
On reflection, we agree that the proof was unclear - and have taken the opportunity to rewrite it with more careful notation.
Unfortunately, this took us over the 9 page limit, so we only give a sketched summary in the main text and give full details appendix C.1 (referenced in the main-text sketched summary).

To directly answer your first question: the parameter $\theta$ relates to the whole neural network. 
To better explain the local-optimum argument we introduce notation: decompose $\theta = (U, \phi)$ into a set of final-layer weights $U$, and a set of weights for the previous layers $\phi$. Then if $\hat{\theta} = (\hat{U}, \hat{\phi})$ is a local optimum of $\mathcal{L}(\theta)$, then $\hat{U}$ is a local optimum of the function $l(U) = \mathcal{L}((U,\hat{\phi}))$.
Though in practice we would not reach a local optimum in finite time with SGD, one may expect the final estimates to be near a local optimum, and therefore for the conclusions of the lemma to apply at least approximately.

Does that answer your question? (and are you satisfied with our proposed change? or is there more we can do?)

Reviewer comment (Theory):
Ridge-regularized CCA. Does the ridge term also change the aforementioned "notions of uniqueness" to a stronger notion?

Author response:
Your question about uniqueness is a good one - and a detail that we had hoped to 'brush under the carpet' (we wanted to focus on computation, and not get bogged down in regularisation).
We will give a very short answer here due to space constraints, but hope to expand upon this later in the discussion.

For GEPs with $B$ strictly positive definite there is non-uniqueness up to choice of basis within each generalised eigenspace. 
In high dimensional *sample* CCA there is extreme degeneracy - all canonical correlations are 1 (as discussed in the main text) and there is additional degeneracy in the canonical directions due to the low-rank structure of the sample covariances (the full-batch $\hat{B}$ has non-trivial kernel).
This degeneracy is overcome by sample ridge CCA by enforcing that $\hat{B}_\alpha$ is strictly positive definite (note however, that the generalised eigenvalues no longer correspond to canonical correlations, and need not be in [0,1], complicating interpretation).
This phenomenon is analogous to the phenomenon that ridge regression will find unique solutions even in underdetermined systems where OLS gives a degenerate family of solutions (which we suspect motivated your question).

Does this clarify matters?

Reviewer comment (Novelty):
Justification of the novelty of the proposed method.

Author response:
Thank you for agreeing that the results of our extensive experiments prove the effectiveness of our proposed method.
We appreciate this opportunity to address the concerns raised in your review, particularly regarding the novelty of our approach compared to prior work such as [1].

We have addressed your concern about our decision not to perform PCA preprocessing in our common response. We hope that you find this helpful.

In the remainder of this response, we will reiterate the novel aspects of our proposed method, with particular emphasis on comparisons to Gemp et al. (2022) [1], as you requested.
Comparing directly to [1] we will argue that our work is of much broader scope and that even in the narrow scope where our work is comparable to [1], our method is very different (and in fact has a number of advantages, which we hope to convince you of).
We will also argue that our work is novel within the literature more broadly.
By elaborating on these points, we hope to persuade you of the novelty of our contribution.

We sincerely hope that this response successfully addresses your concerns; if so we kindly ask that you consider raising your score. Regardless of the outcome, we thank you again for the time you have taken to review our work.

## Novelty of our Proposed Methods

Firstly, we address novelty more broadly.
Our Eckhart-Young inspired characterisation for GEPs is novel - it does not appear elsewhere in the literature (to our knowledge).
Our presentation of a single framework and class of algorithms to tackle stochastic CCA, PLS and Deep CCA is also novel, as is the application of Deep CCA to SSL.
Even our stochastic GEP algorithm is very different to all previous proposals - it is the first such algorithm formulated with an explicit loss function that permits unbiased gradient updates.

The work of Gemp et al. (2022) is narrower in that it only proposes algorithms for stochastic GEPs.
Their algorithms do not have an explicit loss function, and instead use update steps which are inspired by their previous work on an 'Eigengame' for PCA (a very different motivation).
Crucially, this means that their method cannot be extended to Deep CCA, let alone SSL; these deep extensions were a key focus of our work, and comprised the majority of our experiments (3 out of 5).

Even if we only compare proposals for stochastic GEPs, the $\gamma$-eigengame method of Gemp et al. [1] is very different to our method, and is in fact problematic in a number of ways.
We reiterate that our method is derived by applying SGD to a population loss function that characterises GEPs, has unbiased update steps and only involves a single learning rate parameter for SGD.
By contrast $\gamma$-eigengame (Algorithm 2 in [1]):
- **Does not have an explicit loss function.** Instead motivates update directions by a heuristic argument.
- **Has biased update steps,** because it involves normalisation by sample quantities.
- **Has flimsier theoretical support:** they use an error-propagation argument to justify their claim that their algorithm converges with probability 1. However, this proof appears incomplete (red flags include "Assume $\epsilon_i'$ is $\mathcal{O}(\epsilon_i)$ to ease the exposition"). By contrast, we can directly apply results from the literature on SGD to deduce convergence with probability 1.
- **Most importantly, their method is much more complicated.** It requires "introducing an auxiliary variable, denoted $[B\hat{v}]_j$ to track the running averages of $B\hat{v}_j$" and an additional step to "manually clip the result to be greater than or equal to $\rho$, the minimum singular value of $B$". Their method therefore requires knowledge of a lower bound $\rho$ to the singular values of $B$, which is hard to justify in practice. Also, they require an additional learning rate parameter / step size sequence $\gamma_t$ for updates of $[B\hat{v}]_j$. The performance of their method depends delicately on the choice of this parameter, and tuning it induces a significant computational overhead.
Finally, we highlight the practical advantages of our method over $\gamma$-eigengame, as illustrated in our experiments.

If anything, our proposal is most closely related to the update rule of SGHA [2]; yet because this cannot be written as the gradient of a population loss function it cannot be easily extended to Deep CCA, and again has flimsier theoretical support.

[1] Gemp, Ian, Charlie Chen, and Brian McWilliams. "The generalized eigenvalue problem as a Nash equilibrium." arXiv preprint arXiv:2206.04993 (2022).

[2] Chen, Zhehui, et al. "On constrained nonconvex stochastic optimization: A case study for generalized eigenvalue decomposition." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

Reviewer comment (Theory):
Although the literature is extensively cited across the paper, and well detailed, and although the experiments show a clear advantage to the presented method, it is still a little bit hard to compare the theoretical and computational guarantees of the presented method versus the other methods from the benchmark. It might be useful, for instance in the Appendix, to provide some short structured summary (perhaps as a table), on all the properties of the related methods in the literature versus the one presented (e.g., biasedness/unbiasedness of the gradients, complexity of computing one iteration as a function of the batch-size, dimension, guarantees of convergence towards global/local optimum/saddle point, etc.).

Author response:
We sincerely appreciate your positive feedback on our work. 
Thank you for acknowledging its originality, quality, clarity, and significance, especially in the context of CCA-type problems and self-supervised learning methods.

Thank you also for your constructive criticism.
We have corrected the typo in section 3.3.
We agree with you that it would be valuable to have a clearer comparison of properties of the presented methods with the other methods in our benchmarks.
You are right in that it is not straightforward to decide what information is most relevant to display.
We are still working on this, and hope to include such a summary in the appendix shortly.

We hope that our detailed responses to the other reviews will reinforce your confidence in our work.

Reviewer comment (Theory):
Such a comparison may not always be doable, in particular because the guarantees of convergence, for instance, are not always simple to express, so this is just a suggestion, and the paper in its current form may already provide the necessary information to the reader to study such aspects.

Author response:
We sincerely appreciate your positive feedback on our work. 
Thank you for acknowledging its originality, quality, clarity, and significance, especially in the context of CCA-type problems and self-supervised learning methods.

Thank you also for your constructive criticism.
We have corrected the typo in section 3.3.
We agree with you that it would be valuable to have a clearer comparison of properties of the presented methods with the other methods in our benchmarks.
You are right in that it is not straightforward to decide what information is most relevant to display.
We are still working on this, and hope to include such a summary in the appendix shortly.

We hope that our detailed responses to the other reviews will reinforce your confidence in our work.

Reviewer comment (Writing):
In section 3.3, there is a typo: “we this flesh out” should be corrected to “we flesh this out.”

Author response:
We sincerely appreciate your positive feedback on our work. 
Thank you for acknowledging its originality, quality, clarity, and significance, especially in the context of CCA-type problems and self-supervised learning methods.

Thank you also for your constructive criticism.
We have corrected the typo in section 3.3.
We agree with you that it would be valuable to have a clearer comparison of properties of the presented methods with the other methods in our benchmarks.
You are right in that it is not straightforward to decide what information is most relevant to display.
We are still working on this, and hope to include such a summary in the appendix shortly.

We hope that our detailed responses to the other reviews will reinforce your confidence in our work.