Paper: Exploring the Combined Power of Covariance and Hessian Matrices Eigenanalysis for Binary Classification (ICLR-2024)

Reviewer comment (Theory):
The description of the key technique is unclear. How to integrate the covariance matrix and the Hessian matrix can be confusing. The authors only offer some description in the Section 2.1 and it lacks more detailed theoretical explanation.

Author response:
We acknowledge the need for a more explicit presentation of the algorithm for the proposed method. In response to this, we have added additional clarity and detail in Section 2.1 of the revised manuscript, providing a more thorough description of the key technique.

In the revised Section 2.1:

1. **Covariance matrix eigenanalysis:**
   We clarified the eigenanalysis of the covariance matrix ($Cov(\boldsymbol{\theta})$) and its role in capturing the principal directions with the highest variances. The leading eigenvector ($\mathbf{v}_1$) associated with the largest eigenvalue ($\lambda_1$) represents the principal direction with the highest variance.

   The eigen-equation is given by $Cov(\boldsymbol{\theta}) \cdot \mathbf{v}_1 = \lambda_1 \cdot \mathbf{v}_1$.

2. **Hessian matrix eigenanalysis:**
   We explained the eigenanalysis of the Hessian matrix ($H_{\boldsymbol{\theta}}$) in detail, including the utilization of a deep neural network with binary cross-entropy loss during training. The leading eigenvector ($\mathbf{v}_1'$) associated with the largest eigenvalue ($\lambda_1'$) represents the direction corresponding to the sharpest curvature.


   The eigen-equation is given by $H_{\boldsymbol{\theta}} \cdot \mathbf{v}_1' = \lambda_1' \cdot \mathbf{v}_1'$.

3. **Integration of matrices and projection of data:**
   We explicitly presented the integration of matrices and projection of data into a 2D space with mathematical formulations.

   $$ \mathbf{U} = [\mathbf{v}_1, \mathbf{v}_1'] $$

   The 2D projection of the data is obtained by:

   $$ \mathbf{X}_{\text{proj}} = \mathbf{X} \cdot \mathbf{U} $$
   
   where $\mathbf{X}$ is the original data matrix, and $\mathbf{X}_{\text{proj}}$ represents the final output of the proposed method.

We want to emphasize that this summary provides the core mathematical ideas, and the complete and detailed explanations, along with additional context, can be found in the revised manuscript.

Reviewer comment (Experiments):
The comparison methods are insufficient. To clarify the superiority of the proposed method, the authors compare the method with four data projection techniques including PCA, Hessian, UMAP and LDA. However, some other dimensionality reduction and data projection techniques should be included. For example, kernel based methods including kernel PCA and kernel LDA and manifold based methods like locally linear embedding (LLE) and t-distributed Stochastic Neighbor Embedding (t-SNE) are also representative methods in this problem.

Author response:
-	To address the concern regarding the comparison methods, we have expanded our evaluation to include a broader spectrum of contemporary techniques. In the revised manuscript, we compare our proposed method with nine distinct approaches, encompassing kernel-based methods like KPCA and KDA, manifold based methods like UMAP and LLE as well as LOL [1]. This extended comparison aims to provide a more comprehensive view of the proposed method's performance.
-	We have introduced nonlinearity through distinct kernels determined by grid search in KPCA and KDA for each dataset.
-	The results, discussed in the revised manuscript, demonstrate the consistent superiority of our approach across various datasets and metrics.
-	These extended results are completely reproducible using the same Colab notebooks whose links are provided for transparency and validation.
-	The relevant discussion for these additional comparisons has been appropriately incorporated into the Discussion section of the revised manuscript.
-	The abstract has also been updated to reflect the implications of these extended comparisons.

Reviewer comment (Reproducibility):
The authors should provide a specific algorithm of the proposed method. The procedure of the algorithm remains highly unclear.

Author response:
We acknowledge the need for a more explicit presentation of the algorithm for the proposed method. In response to this, we have added additional clarity and detail in Section 2.1 of the revised manuscript, providing a more thorough description of the key technique.

In the revised Section 2.1:

1. **Covariance matrix eigenanalysis:**
   We clarified the eigenanalysis of the covariance matrix ($Cov(\boldsymbol{\theta})$) and its role in capturing the principal directions with the highest variances. The leading eigenvector ($\mathbf{v}_1$) associated with the largest eigenvalue ($\lambda_1$) represents the principal direction with the highest variance.

   The eigen-equation is given by $Cov(\boldsymbol{\theta}) \cdot \mathbf{v}_1 = \lambda_1 \cdot \mathbf{v}_1$.

2. **Hessian matrix eigenanalysis:**
   We explained the eigenanalysis of the Hessian matrix ($H_{\boldsymbol{\theta}}$) in detail, including the utilization of a deep neural network with binary cross-entropy loss during training. The leading eigenvector ($\mathbf{v}_1'$) associated with the largest eigenvalue ($\lambda_1'$) represents the direction corresponding to the sharpest curvature.


   The eigen-equation is given by $H_{\boldsymbol{\theta}} \cdot \mathbf{v}_1' = \lambda_1' \cdot \mathbf{v}_1'$.

3. **Integration of matrices and projection of data:**
   We explicitly presented the integration of matrices and projection of data into a 2D space with mathematical formulations.

   $$ \mathbf{U} = [\mathbf{v}_1, \mathbf{v}_1'] $$

   The 2D projection of the data is obtained by:

   $$ \mathbf{X}_{\text{proj}} = \mathbf{X} \cdot \mathbf{U} $$
   
   where $\mathbf{X}$ is the original data matrix, and $\mathbf{X}_{\text{proj}}$ represents the final output of the proposed method.

We want to emphasize that this summary provides the core mathematical ideas, and the complete and detailed explanations, along with additional context, can be found in the revised manuscript.

Reviewer comment (Experiments):
The authors should offer more experimental results to validate the effectiveness of the proposed method, which are not limited to more comparison methods and more convincing classification results. As mentioned above, some other dimensionality reduction and data projection techniques should be included. Besides, the performance on the classification problem is not outstanding.

Author response:
-	To address the concern regarding the comparison methods, we have expanded our evaluation to include a broader spectrum of contemporary techniques. In the revised manuscript, we compare our proposed method with nine distinct approaches, encompassing kernel-based methods like KPCA and KDA, manifold based methods like UMAP and LLE as well as LOL [1]. This extended comparison aims to provide a more comprehensive view of the proposed method's performance.
-	We have introduced nonlinearity through distinct kernels determined by grid search in KPCA and KDA for each dataset.
-	The results, discussed in the revised manuscript, demonstrate the consistent superiority of our approach across various datasets and metrics.
-	These extended results are completely reproducible using the same Colab notebooks whose links are provided for transparency and validation.
-	The relevant discussion for these additional comparisons has been appropriately incorporated into the Discussion section of the revised manuscript.
-	The abstract has also been updated to reflect the implications of these extended comparisons.

Reviewer comment (Experiments):
This work is limited in the problem of binary classification, and it would be better if the authors could devise a multi-class classifier considering the real-world applications.

Author response:
While our work focuses on binary classification, we appreciate your suggestion to extend our methodology to multiclass classification as a potential avenue for future research. We have explicitly mentioned in the Discussion section of the revised manuscript that the binary classification focus in this work stems from foundational aspects guiding our formal proof, which is designed around binary assumptions to facilitate a streamlined and elegant derivation process. In particular, the use of binary cross-entropy as the loss function and the utilization of a linear SVM for evaluation inherently adhere to binary classification. Moving forward, careful exploration is needed to adapt our approach to multiclass scenarios to ensure its applicability and effectiveness across a broader range of classification tasks.

Reviewer comment (Writing):
The part 3 of Methodologies session (Section 2.1) can be written more clearly.

Author response:
Dear Reviewer LZ1V,

Thank you for your thoughtful review and valuable feedback on our manuscript, Submission7389. We appreciate the time and effort you invested in providing constructive comments.

We would like to inform you that the revised and clarified content of Section 2.1 has been incorporated into the updated manuscript that we have submitted. We hope that the revisions address your concerns and enhance the clarity of the proposed approach.

In the revised Section 2.1:

1. **Covariance matrix eigenanalysis:**
   We clarified the eigenanalysis of the covariance matrix ($Cov(\boldsymbol{\theta})$) and its role in capturing the principal directions with the highest variances. The leading eigenvector ($\mathbf{v}_1$) associated with the largest eigenvalue ($\lambda_1$) represents the principal direction with the highest variance.

   The eigen-equation is given by $Cov(\boldsymbol{\theta}) \cdot \mathbf{v}_1 = \lambda_1 \cdot \mathbf{v}_1$.

2. **Hessian matrix eigenanalysis:**
   We explained the eigenanalysis of the Hessian matrix ($H_{\boldsymbol{\theta}}$) in detail, including the utilization of a deep neural network with binary cross-entropy loss during training. The leading eigenvector ($\mathbf{v}_1'$) associated with the largest eigenvalue ($\lambda_1'$) represents the direction corresponding to the sharpest curvature.


   The eigen-equation is given by $H_{\boldsymbol{\theta}} \cdot \mathbf{v}_1' = \lambda_1' \cdot \mathbf{v}_1'$.

3. **Integration of matrices and projection of data:**
   We explicitly presented the integration of matrices and projection of data into a 2D space with mathematical formulations.

   $$ \mathbf{U} = [\mathbf{v}_1, \mathbf{v}_1'] $$

   The 2D projection of the data is obtained by:

   $$ \mathbf{X}_{\text{proj}} = \mathbf{X} \cdot \mathbf{U} $$
   
   where $\mathbf{X}$ is the original data matrix, and $\mathbf{X}_{\text{proj}}$ represents the final output of the proposed method.

We want to emphasize that this summary provides the core mathematical ideas, and the complete and detailed explanations, along with additional context, can be found in the revised manuscript.

If you have any specific points or further inquiries, we are more than willing to address them. We look forward to hearing your thoughts on the revised manuscript.

Reviewer comment (Writing):
I wonder if the part 3 of Methodologies session (Section 2.1) can be written more clearly?

Author response:
Dear Reviewer LZ1V,

Thank you for your thoughtful review and valuable feedback on our manuscript, Submission7389. We appreciate the time and effort you invested in providing constructive comments.

We would like to inform you that the revised and clarified content of Section 2.1 has been incorporated into the updated manuscript that we have submitted. We hope that the revisions address your concerns and enhance the clarity of the proposed approach.

In the revised Section 2.1:

1. **Covariance matrix eigenanalysis:**
   We clarified the eigenanalysis of the covariance matrix ($Cov(\boldsymbol{\theta})$) and its role in capturing the principal directions with the highest variances. The leading eigenvector ($\mathbf{v}_1$) associated with the largest eigenvalue ($\lambda_1$) represents the principal direction with the highest variance.

   The eigen-equation is given by $Cov(\boldsymbol{\theta}) \cdot \mathbf{v}_1 = \lambda_1 \cdot \mathbf{v}_1$.

2. **Hessian matrix eigenanalysis:**
   We explained the eigenanalysis of the Hessian matrix ($H_{\boldsymbol{\theta}}$) in detail, including the utilization of a deep neural network with binary cross-entropy loss during training. The leading eigenvector ($\mathbf{v}_1'$) associated with the largest eigenvalue ($\lambda_1'$) represents the direction corresponding to the sharpest curvature.


   The eigen-equation is given by $H_{\boldsymbol{\theta}} \cdot \mathbf{v}_1' = \lambda_1' \cdot \mathbf{v}_1'$.

3. **Integration of matrices and projection of data:**
   We explicitly presented the integration of matrices and projection of data into a 2D space with mathematical formulations.

   $$ \mathbf{U} = [\mathbf{v}_1, \mathbf{v}_1'] $$

   The 2D projection of the data is obtained by:

   $$ \mathbf{X}_{\text{proj}} = \mathbf{X} \cdot \mathbf{U} $$
   
   where $\mathbf{X}$ is the original data matrix, and $\mathbf{X}_{\text{proj}}$ represents the final output of the proposed method.

We want to emphasize that this summary provides the core mathematical ideas, and the complete and detailed explanations, along with additional context, can be found in the revised manuscript.

If you have any specific points or further inquiries, we are more than willing to address them. We look forward to hearing your thoughts on the revised manuscript.

Reviewer comment (Theory):
As I read the paper, the authors talk about *the* Hessian matrix.  However, the paper is about a Hessian estimated using a specific deep net, which is of course *a* Hessian, but not *the* Hessian.  The discussion implied (to me) much more general claims than were warranted, imho, given the actual theoretical and empirical results. I would have expected a specific mention of Fisher's Information Matrix, which is closely related to the Hessian, as it includes it, and is a known bound of the variance for any random variable.

Author response:
You rightly pointed out that in the original manuscript, there was a usage of "the Hessian matrix," which has been corrected in the revised version. We acknowledge this oversight, and the updated manuscript now consistently refers to "a Hessian matrix." We have also explicitly mentioned Fisher's information as an approximation of the Hessian in Section 2.2 of the revised manuscript. This modification ensures a clear acknowledgment of the connection between the Hessian matrix and Fisher's information matrix, providing a more accurate representation of our methodology.

Reviewer comment (Novelty):
LDA is well known to find the projection that balances maximizing across-variance while minimizing within-variance.  It was never clear to me why we would want another method to do something like that?  What is missing in LDA that this method achieves? I can imagine a desire to embed in multiple dimensions, rather than just 1, but see my next point about that.

Author response:
Our initial motivation was not driven by a desire to embed in multiple dimensions, as mentioned. Instead, it originated from the intention to explore the relationship between two familiar concepts: covariance and Hessian matrices. Unexpectedly, this exploration led to their unification under another familiar concept, namely the LDA concept. The elegance of this relationship, as highlighted in the revised manuscript, allows us to introduce a method that outperforms established methods, both linear and non-linear, including Linear DA and Kernel DA. The method's superiority is not based on a need for multiple dimensions but rather on the unique insights provided by this unification.

We appreciate the opportunity to clarify why our approach outperforms LDA. Experimental results unequivocally demonstrate the superior performance of our method across all cases, supported by a robust theoretical explanation. Our method surpasses LDA by leveraging higher-dimensional feature spaces, aligning with Cover’s theorem, which favors linear separability in higher dimensions.

Reviewer comment (Novelty):
Under the Gaussian model, the direction that captures the variance across classes is simply the difference of means vector (after 'whitening'), and the direction that maximizes the variance within is the class-centered covariance.  Reduced Rank LDA essentially combines those two: it projects the data onto the matrix which is the product of the difference of means with the low-rank estimate of the pooled covariance.  So, we already have a standard/classical approach to embedding into these two dimensions.  How is your approach better than this?

Author response:
Our initial motivation was not driven by a desire to embed in multiple dimensions, as mentioned. Instead, it originated from the intention to explore the relationship between two familiar concepts: covariance and Hessian matrices. Unexpectedly, this exploration led to their unification under another familiar concept, namely the LDA concept. The elegance of this relationship, as highlighted in the revised manuscript, allows us to introduce a method that outperforms established methods, both linear and non-linear, including Linear DA and Kernel DA. The method's superiority is not based on a need for multiple dimensions but rather on the unique insights provided by this unification.

We appreciate the opportunity to clarify why our approach outperforms LDA. Experimental results unequivocally demonstrate the superior performance of our method across all cases, supported by a robust theoretical explanation. Our method surpasses LDA by leveraging higher-dimensional feature spaces, aligning with Cover’s theorem, which favors linear separability in higher dimensions.

Reviewer comment (Novelty):
The defined Hessian is closely related to another standard thing called the Pointwise Mutual Information (https://en.wikipedia.org/wiki/Pointwise_mutual_information), which is very commonly used in language processing and embedding.  A discussion/comparison with this method would be desirable.

Author response:
We appreciate your suggestion regarding the potential connection between the defined Hessian and Pointwise Mutual Information (PMI). However, after careful consideration, we believe that PMI might not be directly relevant to the goals and scope of our work, which focuses on binary classification tasks and the interplay between covariance and Hessian matrices. While PMI is indeed a valuable tool in language processing and embedding, the objectives and underlying principles of our work differ significantly.

If you have further questions or if there are specific aspects of PMI that you believe could contribute meaningfully to our study, please provide additional details, and we will gladly consider them in our response or future work.

Reviewer comment (Novelty):
The paper does not clearly delineate its unique contributions. The authors should explicitly state what differentiates their work from existing literature, aiding readers in understanding the novelty and significance of the proposed method.

Author response:
- The contributions of our paper are explicitly stated in both the original and revised versions. In the Conclusion section, we highlight the multifaceted contributions, showcasing the theoretical insight and practical method presented in our work. 

- Additionally, we have added a paragraph at the beginning of the Discussion section in the revised manuscript to further underscore the uniqueness and significance of our contributions. In this paragraph we emphasize that our work provides a compelling theoretical insight and a powerful, practical method, demonstrating the strength of simplicity in achieving remarkable results. Firstly, we provide a profound theoretical insight, revealing a subtle yet powerful relationship between covariance and Hessian matrices. Our formal proof seamlessly links covariance eigenanalysis with the first LDA criterion while Hessian eigenanalysis with the second one. This unification under LDA criteria offers a fresh and intuitive perspective on their interplay. Secondly, capitalizing on this theoretical elegance and simplicity, we introduce a novel method that consistently outperforms established techniques across diverse datasets. The unexpected efficacy of our method, rooted in the straightforward relationship between covariance, Hessian, and LDA, showcases the effectiveness of simplicity in addressing complex challenges.

Reviewer comment (Theory):
The theoretical results presented in Section 2.2 need a more formal presentation. The authors should use mathematical statements and rigorous proofs to enhance the credibility and clarity of these results.

Author response:
We understand your concern about the formal presentation of theoretical results in Section 2.2. The full proof, including mathematical statements and rigorous proofs, is provided in Appendix A due to space constraints. The proof sketches in Section 2.2 serve as concise summaries, with the complete and detailed proofs available in the appendix.

Reviewer comment (Theory):
The proposed approach, which integrates the eigenanalysis of the covariance and Hessian matrices based on binary cross-entropy loss, raises questions about its applicability to other loss functions. The authors should clarify this point or extend their methodology to include different loss functions.

Author response:
The adaptability of our methodology to various loss functions is acknowledged as a potential avenue for future research. We have explicitly mentioned in the Discussion section of the revised manuscript that the mathematical derivation in our current work relies on the elegant relationship between (the Hessian of) binary cross-entropy loss and within-class variances. Exploring the adaptability of our method to different loss functions requires careful scrutiny in the future work to establish analogous connections.