Paper: Matrix Information Theory for Self-Supervised Learning (ICLR-2024)

Reviewer comment (Novelty):
According to Propositions 3.1 and 5.2, it can be established that the Matrix-KL-uniformity loss is synonymous with the von Neumann entropy loss of I-VNE+ as proposed in [1]. This similarity diminishes the novelty of this paper. Therefore, it is imperative to substantiate, either through theoretical or empirical means, the superiority of Matrix-SSL in comparison to I-VNE+.

Author response:
- Thank you for your reminder. Our starting point is to use $MKL( \frac{1}{d} I  + \lambda I || C_{auto} + \lambda I )$ to present **a unified framework to understand many SSL methods through uniformity pursuit,** where $C_{auto}$ is the auto-correlation matrix. Specifically, we demonstrate in this case that the difference between MCE and KL is constant, and we technically considered introducing MCE because using it can easily see the connections with previous pioneer works. Note that there is an asymmetry with matrix KL, i.e. In general $MKL(P || Q) \neq MKL(Q || P)$. Interestingly, although generally asymmetric, when $P=Q$, the two are equal and equal to 0. This non-trivial result explains the phenomenon of the increase in "Entropy". **Regarding the KL uniformity you mentioned, we have shown its closed-form relationship with "Entropy" by noticing the quantity $MKL(C_{auto}||I)$, which is novel, because it established the connection between the two quantities and this is only a special case of $\lambda=0$.**

- In addition, In our revised paper, we **generalize** the definition of von Neumann entropy on density matrix (with unit trace) into any positive semi-definite matrices, which are differentiable, convex, and can even generalize to any complex-valued Hermitian matrices. The von Neumann entropy mainly arises from the Quantum Information realm, but we certainly generalize the concept of it, removing the constraints of the measurement axiom (the capstone of Quantum Mechanics) presented in the unit trace density matrix.

- Besides, we generalize the quantum and kernel KL divergence defined in Bach's paper into positive semi-definite matrices as **matrix KL divergence**, and also introduce the definition of matrix cross-entropy, which forms the **exact mirror** of **Shannon information theory** and **Quantum Information theory**, into the PSD realm. And it has a nice expression as follows:
$\operatorname{MCE} (P, Q) =  \operatorname{MKL} (P || Q) + \operatorname{ME} (P)$

Reviewer comment (Evaluation):
The results presented in this paper are not significant to substantiate the effectiveness of Matrix-SSL. Notably, Table 1 does not incorporate the official performance metrics of SwAV, as reported in [2], which report values of 71.99, 73.85, and 74.81 for 100, 200, and 400 training epochs, respectively. Additionally, Table 2 lacks the inclusion of performance data as reported in [1]. When both Table 1 and Table 2 are appropriately updated, it becomes evident that the performance of Matrix-SSL is not state-of-the-art.

Author response:
Thank you for your careful reading.

- **The performance issue of SwAV and the related discussion on Multi-crop**. As you mentioned, the official results reported by SwAV are indeed significantly higher than those of other methods. The main reason is that **SwAV uses a Multi-crop setting when pretraining**, augmenting the images in a scheme of $2 \times 224$ + $6 \times 96$, totaling to 8 views. Our implementation, however, only considered augmenting the images twice (i.e., without the six smaller augmentations). One very important point is that in the ablation study done by SwAV (https://arxiv.org/pdf/2006.09882.pdf) on Multi-crop (Figure 3), it can be seen that **Multi-crop brings a very substantial improvement to self-supervised methods**. **Thus we reported the performance of SwAV without Multi-crop**. It should be noted that **our method can also use Multi-crop** as an add-on, but due to the lack of computational resources, we have not had the opportunity to test MCE + Multi-crop. We will promptly carry out the experiments of the multi-crop case once we have sufficient computational resources. And we believe that MCE + Multi-crop could achieve even better results than what we have now. 

- Thank you for pointing out that including I-VNE+(https://github.com/jaeill/CVPR23-VNE/) as baseline should be considered. We attempted to include I-VNE+ in the table, but we noticed that **the official open-source code implementation of I-VNE+ uses Multi-crop** (can be seen from lines 170 to 189 at https://github.com/jaeill/CVPR23-VNE/blob/main/examples/i-vne%2B/train_ivne_imagenet100.py ). Recall the discussion of multi-crop in (1), **we believe that adding I-VNE+ to the current table would not be a fair comparison at this stage**. We will include I-VNE+ in the table after conducting experiments of MCE + multi-crop. However, we note that even so, the result of I-VNE+'s Linear evaluation on Imagenet-1K is 72.1 (as presented in Table 10 in Section F of the paper), which is significantly below the results of our method in linear evaluation, indicating from another aspect that there is a considerable difference between our work and I-VNE+.

- Thank you for your suggestion to add more downstream tasks. In this short period of time, we have conducted experiments on 'Semi-supervised Learning' and 'Transfer learning: Image classification'. For experiment results, see General Response.

Reviewer comment (Experiments):
This paper does not provide comprehensive benchmark tables, including but not limited to "Semi-supervised learning on ImageNet" and "Transfer learning: image classification," as elaborated in Table 2 and Table 3 of [3].

Author response:
Thank you for your careful reading.

- **The performance issue of SwAV and the related discussion on Multi-crop**. As you mentioned, the official results reported by SwAV are indeed significantly higher than those of other methods. The main reason is that **SwAV uses a Multi-crop setting when pretraining**, augmenting the images in a scheme of $2 \times 224$ + $6 \times 96$, totaling to 8 views. Our implementation, however, only considered augmenting the images twice (i.e., without the six smaller augmentations). One very important point is that in the ablation study done by SwAV (https://arxiv.org/pdf/2006.09882.pdf) on Multi-crop (Figure 3), it can be seen that **Multi-crop brings a very substantial improvement to self-supervised methods**. **Thus we reported the performance of SwAV without Multi-crop**. It should be noted that **our method can also use Multi-crop** as an add-on, but due to the lack of computational resources, we have not had the opportunity to test MCE + Multi-crop. We will promptly carry out the experiments of the multi-crop case once we have sufficient computational resources. And we believe that MCE + Multi-crop could achieve even better results than what we have now. 

- Thank you for pointing out that including I-VNE+(https://github.com/jaeill/CVPR23-VNE/) as baseline should be considered. We attempted to include I-VNE+ in the table, but we noticed that **the official open-source code implementation of I-VNE+ uses Multi-crop** (can be seen from lines 170 to 189 at https://github.com/jaeill/CVPR23-VNE/blob/main/examples/i-vne%2B/train_ivne_imagenet100.py ). Recall the discussion of multi-crop in (1), **we believe that adding I-VNE+ to the current table would not be a fair comparison at this stage**. We will include I-VNE+ in the table after conducting experiments of MCE + multi-crop. However, we note that even so, the result of I-VNE+'s Linear evaluation on Imagenet-1K is 72.1 (as presented in Table 10 in Section F of the paper), which is significantly below the results of our method in linear evaluation, indicating from another aspect that there is a considerable difference between our work and I-VNE+.

- Thank you for your suggestion to add more downstream tasks. In this short period of time, we have conducted experiments on 'Semi-supervised Learning' and 'Transfer learning: Image classification'. For experiment results, see General Response.

Reviewer comment (Novelty):
In Section 5, this paper demonstrates that enhancing uniformity leads to an increased effective rank through matrix entropy. However, this result is not groundbreaking. In [1], the authors have previously presented these mathematical findings and have empirically shown that von Neumann entropy regulates uniformity, thereby influencing the effective rank.

Author response:
Thank you for your careful reading of our paper. 

We want to clarify that [1] didn't previously present the **general** mathematical findings of "enhancing uniformity leads to an increased effective rank through matrix entropy". Firstly, [1] shows that under some conditions, when the autocorrelation matrix $C_{auto}$ 's von Neumann entropy is **maximized**, the representation achieves **Isotropy**. Note **isotropy** is not **uniformity**, they do not prove anything about uniformity. Another important **missing** point is that they **do not** show that many self-supervised learning methods can be seen as minimizing $\operatorname{MKL}( (\lambda + \frac{1}{d}) I || C_{auto} + \lambda I )$, this is what we have shown in section 4. Thus, from our derivation, at the optimal point of loss, uniformity is achieved. Secondly, [1] **does not introduce the concept of effective rank.** So we think [1] **does not** show "have empirically shown that von Neumann entropy regulates uniformity, thereby influencing the effective rank." But we give a **closed-form** relationship among effective rank, matrix entropy, and matrix KL divergence. Our derivation can give an understanding of the rank increasing, due to the the optimal point of SSL loss and the closed-form relationship of effective rank and matrix entropy.

Reviewer comment (Theory):
How was Lemma 3.4 obtained? I understand the purpose of this Lemma, but it's better to give the proof or the corresponding reference. On the other hand, Lemma 3.4 shows that minimizing matrix cross-entropy between the Identity diagonal matrix and the covariance matrix can achieve a uniformity target. However, starting from the fourth page, the zero-mean assumption is disregarded. Does this have any impact on the theoretical analysis results?

Author response:
**A2**: We appreciate your request for clarity on Lemma 3.4. We have updated our paper with the proofs presented in Appendix A. Regarding the zero-mean assumption, it's important to note that Lemma 3.4 (now renumbered as Lemma 4.4 following a restructuring of the related work section) is an analysis of the optimal point of uniformity pursuit. Our theoretical analysis in subsequent sections does not rely on this assumption, allowing for broader applications derived from the matrix KL divergence perspective.

Reviewer comment (Experiments):
Although matrix-KL and matrix-CE share similar optimization properties and theoretical results, are they consistent in practical experiments? I recommend that the authors conduct a set of experiments to validate this.

Author response:
**A5**: Your question raises an important point about the empirical validation of theoretical properties. While matrix-KL and matrix-CE indeed share similar optimization properties and theoretical results, their practical performance can vary due to different characteristics in optimization landscapes and convergence behaviors. To address this, we plan to conduct a comprehensive set of experiments comparing these two approaches in various settings.

In these experiments, we will assess the performance of matrix-KL and matrix-CE on standard benchmarks, focusing on metrics such as accuracy, convergence speed, and robustness to hyperparameter variations. This will allow us to determine not only their theoretical consistency but also their practical applicability and effectiveness in self-supervised learning tasks.

We believe that these additional experiments will provide valuable insights into the nuances of matrix-KL and matrix-CE, further enriching our understanding of their roles in self-supervised learning frameworks.

Reviewer comment (Novelty):
The reference [a] proposes utilizing "correlative information" maximization for self-supervised learning. Analyzing this paper within the context of the proposed matrix information framework would be interesting, especially since the authors of [a] assert that maximizing correlative information between the representations of augmentations establishes a linear dependence rather than an arbitrary nonlinear one.

Author response:
A1: Thanks to your advice, we are glad to share our new finding with you that the ColorInfomax loss function can also incorporated within our framework. For the linear dependence part, according to "Information Geometry and Its Applications" by Amari and [2], only under Gaussian distribution, it is precise. However, we do not think the representation space should obey Gaussian distribution, this assumption is not mild. 

[1] https://arxiv.org/pdf/2209.07999v1.pdf, page 5: we apply the first-order Taylor series approximation.

[2] https://arxiv.org/pdf/2102.05485.pdf, On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions, page 2, Equation (3).

Reviewer comment (Writing):
In Section 2.1, the first paragraph is based on SimCLR and SimSiam, however, the authors introduce a generic SSL architecture. Furthermore, what is meant by dual networks is not clear at this point.

Author response:
A3: Thanks to your advice, we have revised our paper.

Reviewer comment (Presentation):
Consider categorizing Section 2.1 into subheadings like "Contrastive SSL Approaches" and "Non-Contrastive SSL Approaches" to enhance clarity.

Author response:
A4: Thanks to your advice, we have revised our paper.

Reviewer comment (Theory):
For Equation (3), clarify whether there is an underlying assumption about the normalization of the encoder, such as $\||z_i^{(k)}\||_2=1$.

Author response:
A6: Yes, we have emphasized it in our revised paper with \textcolor{blue}.

Reviewer comment (Writing):
In Section 2.2, provide a citation detailing Matrix Entropy, discuss its interpretation, and perhaps its existing applications especially within the context of machine learning/SSL.

Author response:
A7: 
- We have added references to the von Neumann Entropy definition (defined on density matrices with unit trace): Edward Witten. A mini-introduction to information theory, Springer, 2020.

- It seems that we are the first (as far as we know), to generalize the von Neumann Entropy definition with density matrices constraints (with unit trace) into positive semi-definite (PSD) matrices realm, and the trio of matrix entropy, matrix KL divergence, and matrix cross-entropy exactly mirror the classical Shannon information-theoretic quantities, respectively.

- Its existing applications: I think the VNE (https://arxiv.org/pdf/2304.01434.pdf) paper would be the first attempt, but they mainly focus on the von Neumann Entropy part, they haven't introduced the matrix entropy defined on PSD matrices, nor the **effective rank**. In addition, the matrix cross-entropy, and the matrix KL divergence based alignment loss are proposed by us first.

- From Theorem 4.5, we discovered that the renowned work Total Coding Rate, EMP-SSL, etc. by Yi Ma, Yann Lecun, et. al, is essentially equivalent to matrix cross entropy between $\frac{d}{B \epsilon^2} \mathbf{Z} \mathbf{Z}^{\top}$ and $\frac{1}{d}\mathbf{I}_d$, So we think that the effectiveness of matrix entropy regularization should be attributed to them.

  [1] Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29  (9):1546–1562, 2007.

  [2] Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprint arXiv:2201.10000, 2022.

  [3] Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in one training epoch. arXiv preprint arXiv:2304.03977, 2023.

Reviewer comment (Theory):
Regarding the Matrix Entropy definition, clarify if there is a specific assumption about the trace of $\mathbf{A}$ ensuring its eigenvalues form a probability mass function.

Author response:
A8: In our revised paper, we generalize the definition of von Neumann entropy, without such assumption.

Reviewer comment (Writing):
Provide interpretations and potential applications of Matrix KL Divergence and MCE.

Author response:
A9: We have conducted experiments on MCE loss for fine-tuning large language models, with results presented in Appendix A and also the General Response:

Reviewer comment (Theory):
Bach 2022's KL divergence doesn't seem to incorporate the $-\mathbf{P}+\mathbf{Q}$ terms within trace. This discrepancy should be addressed.

Author response:
A10: It seems that we have addressed this problem in the revised paper, we certainly generalize Bach's KL divergence onto PSD matrices, and the answers A7 and A8 are relevant to this problem.

Reviewer comment (Theory):
From the presented definitions, it appears that, unlike Shannon Entropy, MCE does not equate to the sum of Matrix KL and Matrix Entropy. Clarify whether there should be a $\text{Tr}(\mathbf{P})$ term included in the matrix entropy definition.

Author response:
A11: Thanks for your suggestion, we have revised our paper.

Reviewer comment (Writing):
Include a brief discussion explaining the relevance of these definitions to the SSL problem.

Author response:
A12: 
- We appreciate your suggestions, we have moved the effective rank part into the background section, and we think the section **DIMENSIONAL COLLAPSE, EFFECTIVE RANK, AND MATRIX ENTROPY** presents the closed-form relevance among effective rank, matrix KL divergence, and matrix entropy. 

- In addition, inspired by your suggestion, our new findings on interpreting ColorInfoMax loss within our framework (presented in Section 5) are pretty interesting. 

- Besides, the original version contains the matrix information-theoretic interpretation on uniformity pursuit, including several renowned methods.

- We also revised our paper to further clarify the SimCLR loss part presented in Proposition 4.3.

Reviewer comment (Theory):
Proposition 3.3 statement is ambiguous: clarify which matrices the InfoNCE cost in (1) is the MCE of, and whether InfNCE cost and SimCLR cost are identical. The cost function obtained in the proof in terms of MCE does not match (1).

Author response:
A14: We have revised our formula, to make it aligned with the original InfoNCE loss.

Reviewer comment (Writing):
The paragraph after Proposition 3.3 appears convoluted: Clarify whether $p_{data}=p_{\mathbf{x}}$ and $\mathcal{X}$ is the support set of $p_{\mathbf{x}}$. Define $f$ clearly since $f_\Theta$ and $f_\phi$ were mentioned without clear definitions before. Ensure consistent bolding of $f$. Clarify the assumption about layer normalization $\||f(\mathbf{x})\||_2^2=1$. Instead of "straightforward calculation", provide a proof of Lemma 3.4 with proper notation in Appendix A. This paragraph and Lemma 3.4 are not properly motivated.

Author response:
A15: We have revised our paper according to your suggestion.

Reviewer comment (Writing):
Lemma 3.4: Suggestion "Let $\sigma$ represent the uniform distribution on $S^{d-1}$...".

Author response:
A15: We have revised our paper according to your suggestion.

Reviewer comment (Writing):
Quoting the sentence "From Proposition 3.3, we find that SimCLR (InfoNCE) loss is not canonical for achieving matrix information-theoretic uniformity unless the covariance matrix is diagonal": Clarify what "loss being not canonical" means, whether matrix information-theoretic uniformity has been defined yet, and if this simply means that SimCLR or InfoNCE does not enforce feature whitening.

Author response:
- We have revised our paper to fix this typo. The main reason why we introduce the $\lambda$regularization is because we want to interpret existing SSL approaches, $\lambda$ in our MCE formulation can be set as 0, but in other methods the result would be not satisfying. In addition, we conducted ablation studies, and find that $\lambda = 1$ has exactly the same performance, which make our formulation more robust to hyperparamters, or even simply without such a hyperparameter.

Reviewer comment (Writing):
The paragraph before Theorem 3.5: Suggestion "This MCE based uniformity loss definition (or $\mathcal{L}_{UMCE}$) and its Matrix-KL divergence based counterpart are closely related... as outlined by the following theorem:".

Author response:
A17: Thanks for your suggestion, we have revised our paper based on your insightful comments.

Reviewer comment (Theory):
Theorem 3.6: Suggest "... under the constraint $\||\mathbf{z}_i\||_2^2=1$, for $i=1, \ldots, n$". The proof of Theorem 3.6 should be provided in the Appendix.

Author response:
A18: Thanks for your suggestion, we have revised this Theorem, and added a self-contained proof of it to the appendix.

Reviewer comment (Writing):
Suggest revising the sentence "Our formulation interestingly recovers the Maximal Entropy Coding (MEC) loss..." to "The Maximal Entropy Coding (MEC) loss in ... can be formulated in terms of Matrix MCE..." as per the given formulation.

Author response:
A19: Thanks for your suggestion, we have incorporated this part in our revised paper.

Reviewer comment (Writing):
For $\mathcal{L}_{EMP-TCR}$, $\bar{\mathbf{Z}}$ is not defined — only $\bar{\mathbf{Z}}_i$ is defined. There is confusion of index representations relative to Equation (1). It appears $\mathbf{z}_k^{i}$ vectors were defined as row vectors. The article should set up the proper data model and notation at the beginning and stick with it.

Author response:
A20: Thanks for your suggestion, we have fixed the notation inconsistency.

Reviewer comment (Experiments):
Consider providing an MCE-based representation for the Corinfomax SSL in reference [a].

Author response:
A21: Yes, we have incorporated this part in our revised paper.

Reviewer comment (Writing):
In Section 4, first sentence: "... we would like embeddings to have zero mean and covariance ...".

Author response:
A23:  Thanks for your suggestion, we have revised our paper.

Reviewer comment (Theory):
Theorem 4.1 needs to be clearly reworded with proper references to the objective function and constraints. Define "effective rank" and explain how it differs from rank. Clarify whether the theorem is about the given optimization problem. Rewrite the proof in Appendix with explicit mathematical assertions and a clearly stated optimization problem.

Author response:
A24: Our revised content addresses the issues by clarifying the notation, explaining the concept of effective rank, and presenting the theorem in the context of an optimization problem. The theorem is now more explicitly connected to the matrix uniformity loss, and the relationship between this optimization and other similar methods is clearly outlined.

Reviewer comment (Novelty):
Lemma 4.2 is typically well known.

Author response:
A25: Thanks for your suggestion, we moved it to the appendix.

Reviewer comment (Writing):
For $\mathcal{L}_{Matrix-KL-uniformity}$, MCE is used rather than the Matrix-KL measure. Clarify why it is called this way.

Author response:
A26: We have added supplementary content to address this problem:
$$\begin{aligned}
\mathcal{L}_{\text{Matrix-KL-Uniformity}} = \operatorname{MCE}(\frac{1}{d}\mathbf{I}_d, \mathbf{C}\left(\mathbf{Z}_1, \mathbf{Z}_2\right)) &= \operatorname{MKL}(\frac{1}{d}\mathbf{I}_d || \mathbf{Z}\left(\mathbf{Z}_1, \mathbf{Z}_2\right)) + \operatorname{ME}(\frac{1}{d}\mathbf{I}_d)\\
&= \operatorname{MKL}(\frac{1}{d}\mathbf{I}_d || \mathbf{Z}\left(\mathbf{Z}_1, \mathbf{Z}_2\right)) + \text{Const.}\\
\end{aligned}$$

Reviewer comment (Theory):
The fact that covariance matrices of two matrices are aligned with respect to MCE or KL does not necessarily imply that representations for the same image are aligned in direction. Euclidean distance or cosine angle based approaches aim to ensure sample-wise alignment. Justify why $\mathcal{L}_{Matrix-KL-allignment}$ should be a better choice.

Author response:
A28: We have updated our paper. Matrix-KL-Alignment loss has a large feasible solution, such that the alignment part and the uniformity part may not have inherent contradiction, since element-wise euclidean alignment has large possibility leads to collapsed solution (SimSiam w/o stop-gradient, presented in Figure 2 in Appendix)

In addition, we have conducted ablation studies, and found that previous SOTA method MEC with our newly added matrix-KL-alignment loss can have 0.3\% improvement on 100-epoch linear probing tasks and 0.8\% improvement on 400-epoch COCO transfer learning tasks.

Reviewer comment (Experiments):
In experiments, it would be interesting to include Tong et.al, 2023 and [a] for comparison.

Author response:
A30: Thank you for your suggestion. We will include the performance of [a](CorInfoMax) on ImageNet-1k in the table in future version. As for Tong et al., 2023 (Emp-SSL), we have noticed that they do not report linear evaluation performance on ImageNet-1k or the other benchmarks we use. Unfortunately, we currently cannot include their results in our paper. However, if we add more experiments on additional benchmarks in the future, we will include Emp-SSL in the Baselines which Emp-SSL reports performance.

Reviewer comment (Experiments):
Since the proposed Matrix-SSL method provides superior performance, clarify whether the authors reproduced the accuracy of other algorithms to calibrate their simulation and evaluation models.

Author response:
A31: Thank you for your question. We primarily replicated the 100-epoch experiment of MEC to ensure that our code is correct. Currently, all baseline data in the table come from MEC as well as those baselines' own papers. If sufficient computational resources become available later on, we will consider replicating more baselines in our experimental environment.

Reviewer comment (Presentation):
The Related Work section typically follows the Introduction section. Moreover, it should not only summarize literature but should also state the contributions of the article relative to these works.

Author response:
A32: Thanks for your suggestion, we have revised our paper.