Paper: Information Flow in Self-Supervised Learning (ICLR-2024)

Reviewer comment (Theory):
Theorems 1 and 2 do not seem to be proved clearly. The proof to theorem 1 in the appendix (the one that includes "lemma 1"!) does not seem to proof things by following a clear mathematical reasoning. Can you clarify how the last sentences leads to a valid proof? It it the same for Theorem 2 (the proof on page 15 given the other one seems to refer to Proposition 2).

Author response:
>Q1: Theorems 1 and 2 do not seem to be proved clearly. The proof to theorem 1 in the appendix (the one that includes "lemma 1"!) does not seem to proof things by following a clear mathematical reasoning. Can you clarify how the last sentences leads to a valid proof? It it the same for Theorem 2 (the proof on page 15 given the other one seems to refer to Proposition 2).

A1: Thank you for your feedback. **We stated in the general response that "the last sentence" $K_1 = Z_1$ and $K_2 = Z_2$ is a typo and we have corrected it to $K_1=Z^T_1Z_1$ and $K_2=Z^T_2Z_2$ in the revised version of our manuscript.**

Reviewer comment (Reproducibility):
Can you provide additional technical details about the experimental setup. The appendix does not seem to contain any information related to that and only limited information is given in the main paper (what are exactly the training objectives and hyper-parameters such as batch size, etc.).

Author response:
>Q2: Can you provide additional technical details about the experimental setup. The appendix does not seem to contain any information related to that and only limited information is given in the main paper (what are exactly the training objectives and hyper-parameters such as batch size, etc.).

A2: **In our initial manuscript, the loss of M-MAE is given by equation (11) and the definition of TCR is given by definition 3.4 and hyper-parameters like batch size are given in section 5.1.** We appreciate the reviewer's question and their valuable input. However, based on the information provided, we are unable to fully understand the specific implementation details that are lacking. If possible, we kindly request the reviewer to provide further clarification or specific requirements, so that we can better assist and address their concerns. We are more than happy to provide additional guidance and clarification to ensure that the reviewer receives the necessary and accurate information they require.

Reviewer comment (Theory):
It is not fully clear how Theorems 1 and 2 are not going against each other given Definition 2. Can you provide some information about that?

Author response:
>Q3: It is not fully clear how Theorems 1 and 2 are not going against each other given Definition 2. Can you provide some information about that?

A3: **Maximizing mutual information is not contradictory to maximizing joint entropy.** For example, in traditional information theory $X=Y$ $\sim$ uniform distribution, then $H(X)=H(Y)=H(X, Y)=I(X, Y)$ and are all maximized. We have similar cases here in matrix information theory, if $K_1=K_2=I_d$, then $H_2(K_1)=H_2(K_2)=H_2(K_1, K_2)=I_2(K_1, K_2)$ and are all maximized.

Reviewer comment (Theory):
There is no detailed discussion between the introduced matrix entropy (definition 1) and the original Shannon's information entropy. In fact, The Renyi entropy is defined by a special matrix family named density matrix. It is not defined for an arbitrary matrix, but only for a Gram matrix. However, this paper misuses the concept of matrix entropy throughout the whole paper. For example, In page 14, the last paragraph says that "Take K1 = Z1 and K2 = Z2, the results follow similarly". However, K1 and K2 should be Gram matrices, while Z1 and Z2 are feature matrices, that are not a Gram matrix. I think this paper should clarify the relationship between the matrix entropy (defined by a Gram matrix of the samples from a probability distribution) and the original Shannon's information entropy (defined by a random variable following a probability density function).

Author response:
Thank you for taking your time to review our paper. As your concerns (W1, W2, W4, W5) are caused by our typo. We will make further clarification here. As our matrix information quantities are defined on **square** matrices whose diagonals are all $1$. Taking $K_1=Z_1$ and $K_2=Z_2$ is a typo, the correct one should be $K_1=Z^T_1Z_1$ and $K_2=Z^T_2Z_2$. **This is why we say in the proof part that it is similar to Barlow twins loss proof where the $K_1$ and $K_2$ defined there are square matrices whose diagonals are all $1$.** We have corrected this typo in the revised manuscript.

We propose to use a unified matrix information-theoretic view on self-supervised learning. We want to further stress that our main contributions are given a theoretical understanding of both contrastive and non-contrastive methods and we also use matrix information theory to improve masked image modelling.

Reviewer comment (Experiments):
The proposed M-MAE is sensitive to the choice of the hyperparameter, as shown in Table 2. The gap between each $\mu$ varies a lot, and it means that we need to access the original target labels to tune the hyperparemeter. It violates the spirit of self-supervised learning; we should not access the original labels.

Author response:
>Q3: The proposed M-MAE is sensitive to the choice of the hyperparameter $\mu$, as shown in Table 2. The gap between each 
 varies a lot, and it means that we need to access the original target labels to tune the hyperparemeter. It violates the spirit of self-supervised learning; we should not access the original labels

A3: Robustness may not be our primary focus; rather, we are highlighting the potential for improvement with matrix information theory. Additionally, this theoretically motivated improvement is closely connected to previous method U-MAE. **Moreover, the closely related method U-MAE is also sensitive to hyper-parameter.**

Reviewer comment (Theory):
The results in Section 3 are not generalizable to the generic self-supervised learning methods; these results are only applicable to Barlow twins and spectral contrastive learning. In fact, as mentioned in my previous comment, the proof is wrong for spectral contrastive learning because K1 and K2 should be a Gram matrix. In other words, the proof only works for a special case of self-supervised learning, where the objective function coincides with "Proposition 1" and K1 and K2 are Gram matrices of the feature matrices Z1 and Z2.

Author response:
Thank you for taking your time to review our paper. As your concerns (W1, W2, W4, W5) are caused by our typo. We will make further clarification here. As our matrix information quantities are defined on **square** matrices whose diagonals are all $1$. Taking $K_1=Z_1$ and $K_2=Z_2$ is a typo, the correct one should be $K_1=Z^T_1Z_1$ and $K_2=Z^T_2Z_2$. **This is why we say in the proof part that it is similar to Barlow twins loss proof where the $K_1$ and $K_2$ defined there are square matrices whose diagonals are all $1$.** We have corrected this typo in the revised manuscript.

We propose to use a unified matrix information-theoretic view on self-supervised learning. We want to further stress that our main contributions are given a theoretical understanding of both contrastive and non-contrastive methods and we also use matrix information theory to improve masked image modelling.

Reviewer comment (Theory):
There is no discussion of why the mutual information maximization (or joint entropy) of Z1, Z2 is a good measure of a good self-supervised learning method. As there is no connection between mutual information maximization and goodness of self-supervised learning methods, the motivation of M-MAE is somewhat weak. What is the benefit of making an MAE model maximize mutual information? (Note that, even more it is actually not about mutual information. See comment 2)

Author response:
Thank you for taking your time to review our paper. As your concerns (W1, W2, W4, W5) are caused by our typo. We will make further clarification here. As our matrix information quantities are defined on **square** matrices whose diagonals are all $1$. Taking $K_1=Z_1$ and $K_2=Z_2$ is a typo, the correct one should be $K_1=Z^T_1Z_1$ and $K_2=Z^T_2Z_2$. **This is why we say in the proof part that it is similar to Barlow twins loss proof where the $K_1$ and $K_2$ defined there are square matrices whose diagonals are all $1$.** We have corrected this typo in the revised manuscript.

We propose to use a unified matrix information-theoretic view on self-supervised learning. We want to further stress that our main contributions are given a theoretical understanding of both contrastive and non-contrastive methods and we also use matrix information theory to improve masked image modelling.

Reviewer comment (Experiments):
The experimental results only show the comparisons between MAE, U-MAE and M-MAE. There are a lot of self-supervised learning methods. I think this paper needs more comparisons with other self-supervised learning methods (e.g., BalowTwins, MoCo, SimCLR, BYOL, DeepClustering, Swav, Data2Vec, DINO, iBot, SimMIM, ...) in terms of both information quantity and performance. I also think that it would be good for this paper to compare with other MAE variants (or MIM methods, such as SimMIM), but it could depend on the scope of this paper; as I think this paper needs a heavy non-trivial revision, as of now, I don't argue that M-MAE should be compared with other MAE variants, but I think additional comparisons with MAE variants will make the submission stronger. I recommend this survey paper to search more recent MAE variants: "A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond"

Author response:
>Q6: The experimental results only show the comparisons between MAE, U-MAE and M-MAE. There are a lot of self-supervised learning methods. I think this paper needs more comparisons with other self-supervised learning methods (e.g., BalowTwins, MoCo, SimCLR, BYOL, DeepClustering, Swav, Data2Vec, DINO, iBot, SimMIM, ...) in terms of both information quantity and performance. I also think that it would be good for this paper to compare with other MAE variants (or MIM methods, such as SimMIM), but it could depend on the scope of this paper; as I think this paper needs a heavy non-trivial revision, as of now, I don't argue that M-MAE should be compared with other MAE variants, but I think additional comparisons with MAE variants will make the submission stronger. I recommend this survey paper to search more recent MAE variants: "A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond"

A6: There may be unfair comparisons when comparing our method with other SSL approaches. The MAE and its variants have different underlying principles compared to other self-supervised methods. Additionally, U-MAE only compares with MAE variants. Note we plan to add a reproduction of SimMIM in the future. We agree with you that "I don't argue that M-MAE should be compared with other MAE variants" and our comparisons already cover the most related and canonical methods MAE and U-MAE. **It's worth noting that we have cited the paper "A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond" to provide a broader context for our work.**

Reviewer comment (Writing):
There are some definitions used before they are defined (e.g., TCR is defined in the appendix). It would be great to at least point to them.

Author response:
Thank you for your feedback. **The TCR is already defined in definition 3.4 of the initial manuscript, which is before they are used.**

Reviewer comment (Evaluation):
The TCR loss on the latent codes, how does it contribute to the entropy of the model? Is there a similar plot one can show as the training of M-MAE proceeds to Figure 1/2?

Author response:
Thank you for your question. In our paper, the entropy of the model means the matrix entropy of the latent codes' covariance matrix. As we show in the paper, matrix entropy is directly related with effective rank. This definition of model's entropy aligns with the literatures that use effective rank to measure the dimension of the model's representation [1] [2]. For the matrix entropy plot you required, we plot it along our pretraining on CIFAR-100 (Figure is shown in https://i.postimg.cc/tT48m774/matrix-entropy.png). **From the Figure, it is clear that M-MAE has an increasing entropy during training, which aligns well with our theory.**




[1]: How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders, Qi Zhang, Yifei Wang, Yisen Wang, NeurIPS 2022

[2]: RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank, Quentin Garrido, Randall Balestriero, Laurent Najman, Yann LeCun, ICML 2023