Paper: Preventing Model Collapse in Deep Canonical Correlation Analysis by Noise Regularization (ICLR-2024)

Reviewer comment (Novelty):
Important references and discussions are missing. The fact that powerful neural networks could lead to degenerate feature space is known. In the original deep CCA paper [Andrew et al, 2013] there was already ridge regularization of the auto-covariance matrices (see their eqn 5) to avoid degeneracy and improve numerical stability. And that regularization technique was already studied for linear CCA, see  
- De Bie and De Moor. On the regularization of canonical correlation analysis, 2003.  
and a probabilistic interpretation where a Gaussian observation model leads to regularized covariance matrices  
- Bach and Jordan. A Probabilistic Interpretation of Canonical Correlation Analysis. 2005.  
While the authors' proposal appears different, it is still important to discuss the connections to existing methods and compare with them, both theoretically and empirically.

Author response:
Thanks for the thoughtful comment. The reviewer mentioned the ridge regularization, which is a widely used technique to regularize the features (W_kX_k in CCA and f_k(X_k) in DCCA using our notations), and it is a default setting across almost all (D)CCA methods. In our experiments, **we have actually included ridge regularization during hyper-parameter tuning for all the methods with and without noise regularization**. 

However, the ridge regularization **mainly regularizes the features (i.e., W_kX_k in CCA and f_k(X_k) in DCCA), rather than the transformations (i.e., W_k in CCA and f_k in DCCA)**. W_k in CCA does not need any extra regularization as it is automatically full-rank, while the f_k is still unconstrained and unregulated in DCCA. Therefore, our study mainly focuses on the regularization of the transformation in DCCA. That is why we did not compare with the ridge regularization in the paper.

The reviewer agreed that “powerful neural networks could lead to degenerate feature space”. However, the degenerated features produced by neural networks in DCCA can still be full-rank, and simply enforcing the covariance matrix to be full-rank by adding ridge regularization is not even helping, because the collapse of neural networks is intricate. Therefore, we develop the concept of the “full-rank” property of f_k (note this is an analogy of the full-rank property of W_k), and it is different from the full-rank of the features f_k(X_k). We apologize for the potential confusion raised by this definition. 

Overall, we believe that ridge regularization is mainly for combating noise and improving numerical stability, it cannot prevent the neural networks from degenerating (i.e., model collapse). In contrast, we should directly study the “full-rank” property of the neural networks f_k and we believe the relevant studies are lacking. 

To further support our arguments, we provide the experimental results with different ridge parameters on a real-world dataset CUB at 500-th epoch as follows:

| Ridge parameter | Performance | Square sum of feature covariance |
|-----------------|-------------|---------------------------------|
| 0               | 0.805       | 0.019                           |
| 1e-5            | 0.786       | 0.028                           |
| 1e-4            | 0.776       | 0.210                           |
| 1e-3            | 0.731       | 7.336                           |

One can see that the ridge regularization **even damages the performance** of DCCA. In our NR-DCCA, we actually set the ridge parameter to **zero**. We conjecture the reason is that the large ridge parameter could make the neural network even “lazier” to actively project the data into a better feature space, as the full-rank property of features and covariance matrix are already guaranteed, and this is also evidenced by the “Square sum of feature covariance” shown in the table.

We apologize that we should have included the above discussion in the paper, and we have now incorporated the above results in Appendix A.6.1 for clarification.

Reviewer comment (Theory):
The intuition behind the proposal is not super clear. If the goal is to have full-rank f(X), so that the covariances are better conditioned, the abovementioned covariance regularization approach already achieves the same effect. Proposition 4 essentially says CCA is invariant to linear transformations of input; but this is well-known and easy to see from the original formulation, even without complicated linear algebra. The more interesting analysis would be to explain why the correlation between signal and random noise is a good quantity for deep neural networks to mimic; there must be more structure than saying that fk is full-rank.

Author response:
We apologize for the potential misunderstanding caused, and we have updated the paper to make it clearer (in Section 3.4). Actually, **we define the “full-rank” property of f_k and it is different from the full-rank property of features  f_k(X_k)**. In CCA, the two concepts are nearly the same, as features W_kX_k are forced to be full-rank by the loss function, and W_k is naturally full-rank (refer to Lemma 2 in our paper). However, it is difficult to define what is a “full-rank” property of neural networks since it is not a matirx. Because the current DCCA has no regularization on the neural networks, we believe **such inconsistency is the potential reason for the model collapse in DCCA**.

We agree with the reviewer that the Proposition looks straightforward. To be precise, the correlation with noise is invariant to full-rank linear transformation. However, it is the first time that the connection between the correlation with noise and the full-rank property of W_k is built. In terms of DCCA, as it is difficult to directly define the “full-rank” property of f_k, we first find the equivalent condition of the full-rank property of W_k in CCA, and then we can transplant this condition to DCCA. Specifically, because of Theorem 1 (Proposition 3 + Proposition 4), we can equivalently define the “full-rank” property of  f_k based on the noise in Definition 2.

We do agree with that reviewer that more structure of the neural networks should be explored to further understand what has been regularized by our proposed Noise Regularization. However, we have to admit that interpreting the parameters in neural networks is challenging. Our way to tackle this problem is to look at the behavior of the neural networks. Through numerical experiments, we find that the CCA never collapses and the DCCA always collapses. Then we design the experiments more extremely to let both CCA and DCCA learn the correlation between two independent random noises. The results indicate that CCA can identify the invariant correlation between the noise, while DCCA cannot (we have moved related experiments into Section 5.2 ). We believe this is mainly because **the powerful neural networks have “created” correlation itself**. Therefore, we enforce the neural networks to preserve the correlation for random noise in DCCA, and this motivates our idea of noise regularization. 

**We call f_k “full-rank” to help the reader associate with the full-rank property of W_k, as the full-rank transformation matrix would not “create” correlation**. But this name might cause the reviewer’s misunderstanding. We apologize and we have incorporated more discussions to elaborate on the “full-rank” property of neural networks in the updated paper to eliminate confusion (in Section 4.2).

Reviewer comment (Experiments):
The paper is not purely about optimization and numerical stability, and different inductive bias from the proposed regularization is expected. There should be an investigation of the feature quality on real-world datasets, as shown by prior deep CCA-based papers.

Author response:
Thanks for the comment. Indeed, our work is not about optimization and numerical stability. We **focus on imposing the inductive bias on the neural networks for DCCA**. The correlation with random noise is invariant to full-rank linear transformation, and hence we hope the correlation is also invariant to “full-rank” non-linear transformation. Hence we define such “full-rank” property as the inductive bias of the neural networks.  

In terms of the investigation of the **feature quality** on real-world datasets, our evaluation of feature quality is mainly guided by **visualization techniques** (as demonstrated by TSNE in Appendix A.9 ). Additionally, the **performances** across a multitude of downstream tasks, encompassing both synthetic and real-world datasets, demonstrate the consistent outperformance of the proposed method. Moreover, guided by our theory, we reported the **correlation between the outputs of unrelated data**, which provides another way to observe feature quality and model collapse issues.