Paper: Gaussian Mutual Information Maximization for Graph Self-supervised Learning: Bridging Contrastive-based to Decorrelation-based (ICLR-2024)

Reviewer comment (Novelty):
The theoretical contributions seem rather straightforward, as most can be found or deduced from other papers without these being clearly mentioned.

Author response:
**Question 1:** The theoretical contributions seem rather straight-forward as most can be found or deduced from other papers.

**Answer:** Here, we will respond point by point to the questions raised by the reviewer in the "**Redundant theoretical analysis**" part.

(1) Mutual Information between multivariate Gaussians is already well-studied and can be found in several classical books of Information Theory and Statistics.

*Answer:* We are grateful to the reviewer for providing the references. Indeed, as the reviewer pointed out, there have been some researches on Gaussian Mutual Information (GMI) in the fields of statistics and information theory. In our revision, we have included citations involving GMI. Simultaneously, we will preserve the derivation process of Gaussian mutual information in Appendix A to ensure the self-completeness of our paper. This allows interested readers to directly browse the derivation of GMI without searching for related literature. To be honest, we have not found any existing literature that provides a complete derivation process for Gaussian mutual information.

(2) The relations and differences between our paper and CorInfoMax should be clearly stated in terms of design. The distinctions between them mainly lie in how to handle instability.

*Answer:* In the design of objective function, the ways in addressing the numerical instability issue during the optimization process are a significant distinction between the two works. Besides, compared with the ways in addressing the instability, more important difference we think is that our approach does not employ additional projection heads. Our objective function directly operates on the output representations of the encoder, while CorInfoMax first employs a projector of 3-layer MLP to map the outputs of the encoder to a new embedding space and then calculate loss function in the new space. This distinction is not only the discrepancy in network architecture but, more importantly, it actually indicates the difference in motivations between the two works. The purpose of our research is to directly calculate mutual information under the Gaussian assumption without reliance on neural estimators or additional designs. *If our method could only work with the blessing of the additional projection heads, we would not complete this paper, which is contrary to our original motivation.*  In fact, we found the paper CorInfoMax after we had almost finished our work. There are indeed some similarities between the two works. Hence, we cited this paper and spent considerable space on relations and differences between the two works in Appendix F. We have done our best to make these statements comprehensive and objective. 

(3) The analysis of $I_G$-based solution clearly connects with the information theory point of view of CCA-SSG already developed in this paper.

*Answer:* The $I_G$-based variant is obtained from the original formula of Gaussian mutual information. In our paper, the analysis about it mainly involves two aspects. Firstly, we point out that directly optimizing $I_G$-based objective will result in numerical instability and thus introduce a strategy of offsetting and scaling eigenvalues. Secondly, we decompose Gaussian mutual information maximization into two components (conditional entropy minimization and information entropy maximization). Further, we introduce identity constraint to realizing conditional entropy minimization, which aligns with the design of our shared encoder across two views. Besides, the discussion focuses on the relationship between the encoded representations from two views *A* and *B*, that is, the two variables in the formula are respectively linked to the representations from two views.

Different from ours, CCA-SSG interprets its objective from the view of information theory, which focuses on explaining the roles of the two items in the objective function. The relationship discussed in CCA-SSG involves the encoded representations and the inputs of the encoder, that is, the two variables in the formula are respectively related to the inputs and outputs of the neural networks.

In summary, both works have analyzed their methods from the perspective of information theory, but they differ in two aspects including purposes and objects (variables).

Reviewer comment (Novelty):
Mutual Information between multivariate Gaussians is already well-studied and documented in several classical books of Information Theory and Statistics, and it clearly connects with closed-form solutions for the KL divergence (e.g., [A], [B]).

Author response:
(1) Mutual Information between multivariate Gaussians is already well-studied and can be found in several classical books of Information Theory and Statistics.

*Answer:* We are grateful to the reviewer for providing the references. Indeed, as the reviewer pointed out, there have been some researches on Gaussian Mutual Information (GMI) in the fields of statistics and information theory. In our revision, we have included citations involving GMI. Simultaneously, we will preserve the derivation process of Gaussian mutual information in Appendix A to ensure the self-completeness of our paper. This allows interested readers to directly browse the derivation of GMI without searching for related literature. To be honest, we have not found any existing literature that provides a complete derivation process for Gaussian mutual information.

Reviewer comment (Novelty):
The relations and differences to the CorInfoMax paper should be clearly stated in terms of theoretical design to provide a more comprehensive overview of MI variants. CorInfoMax relies on log-determinant MI (LDMI), which connects directly with the entropy of Gaussian distributions. Differences with GMIM mostly concern handling instability via different offset and scaling techniques. Some theoretical analyses of LDMI-based papers also overlap with results in the current paper.

Author response:
(2) The relations and differences between our paper and CorInfoMax should be clearly stated in terms of design. The distinctions between them mainly lie in how to handle instability.

*Answer:* In the design of objective function, the ways in addressing the numerical instability issue during the optimization process are a significant distinction between the two works. Besides, compared with the ways in addressing the instability, more important difference we think is that our approach does not employ additional projection heads. Our objective function directly operates on the output representations of the encoder, while CorInfoMax first employs a projector of 3-layer MLP to map the outputs of the encoder to a new embedding space and then calculate loss function in the new space. This distinction is not only the discrepancy in network architecture but, more importantly, it actually indicates the difference in motivations between the two works. The purpose of our research is to directly calculate mutual information under the Gaussian assumption without reliance on neural estimators or additional designs. *If our method could only work with the blessing of the additional projection heads, we would not complete this paper, which is contrary to our original motivation.*  In fact, we found the paper CorInfoMax after we had almost finished our work. There are indeed some similarities between the two works. Hence, we cited this paper and spent considerable space on relations and differences between the two works in Appendix F. We have done our best to make these statements comprehensive and objective.

Reviewer comment (Reproducibility):
There are no clear descriptions of the architectures, validated hyperparameters, or best configurations reported in Table 1, which prevents assessment of the fairness of these benchmarks.

Author response:
(1) There are no clear descriptions of the architectures and hyperparameters for the results reported in Table 1.

*Answer:* The network architectures are implemented through Graph Convolutional Networks (GCNs). The hyperparameters for GMIM are as follows, where "lr" denotes learning rate and "wd" indicates weight decay:

|   Dataset   | Layers | Representation dim |  lr  |  wd  | $p_f$ | $p_e$ |
| :---------: | :----: | :----------------: | :--: | :--: | :---: | :---: |
|    Cora     |   2    |        512         | 1e-3 |  0   |  0.4  |  0.5  |
|  Citeseer   |   1    |         64         | 1e-3 |  0   |  0.4  |  0.5  |
|   Pubmed    |   2    |        512         | 1e-3 |  0   |  0.2  |  0.6  |
|  Computers  |   2    |        512         | 1e-3 |  0   |  0.1  |  0.3  |
|    Photo    |   2    |        512         | 1e-3 |  0   |  0.2  |  0.3  |
| Coauthor-CS |   2    |        512         | 1e-3 |  0   |  0.2  |  1.0  |

The hyperparameters for GMIM-IC are as follows:

|   Dataset   | Layers | Representation dim | $\beta$ |  lr  |  wd  | $p_f$ | $p_e$ |
| :---------: | :----: | :----------------: | :-----: | :--: | :--: | :---: | :---: |
|    Cora     |   2    |        512         |   1.0   | 1e-3 |  0   |  0.1  |  0.5  |
|  Citeseer   |   1    |         64         |   0.5   | 1e-3 |  0   |  0.0  |  0.6  |
|   Pubmed    |   2    |        512         |    3    | 1e-3 |  0   |  0.3  |  0.5  |
|  Computers  |   2    |        512         |    4    | 1e-3 |  0   |  0.1  |  0.3  |
|    Photo    |   2    |        512         |    7    | 1e-3 |  0   |  0.2  |  0.3  |
| Coauthor-CS |   2    |        512         |    1    | 1e-3 |  0   |  0.2  |  1.0  |

Reviewer comment (Experiments):
Gains in terms of performance seem rather marginal compared to CCA-SSG. As briefly mentioned in the conclusion, it would have been relevant to explore different prior distributions.

Author response:
**Question 5:** Isotropic Gaussian distributions are probably the best ones to promote uniformity in the embedding space so it does not seem so obvious that other prior distributions would perform better. Could the authors further discuss this point?

**Answer:** We highly agree with the reviewer's opinion. We also do not think that other prior distributions would perform better than the Gaussian distribution. As a basic knowledge in statistics, the Gaussian distribution is the most common distribution in life and nature. Besides, the Gaussian assumption is justifiable and extensively employed in numerous disciplines such as economics, data science, and physics. More importantly, although the hypothesis test indicates that node representations do not exhibit a purely Gaussian distribution, the visualization results suggest that they present a Gaussian appearance. This phenomenon contributes to the study under the Gaussian assumption. Considering the above factors, in terms of performance and practical value, we are not optimistic that other priors can outperform the Gaussian distribution. However, deriving a closed-form solution for mutual information under specific distributions is not an effortless task. Therefore, please forgive us for not being able to provide empirical results for other prior distributions here. These deserve further exploration in a new research. In fact, the derivation of Gaussian mutual information has been a laborious endeavor in our paper, even though it might be available in existing literature. In the *future work* section of our paper, we also demonstrate the prospect of extending the Gaussian assumption to other priors, aiming to inspire interested readers.

Reviewer comment (Evaluation):
There is a lack of clarity or justification regarding the evaluation: there are no clear reasons provided for the choice of supervised evaluation, no experiments in semi-supervised settings, and no fully unsupervised evaluations (e.g., using KL-based clustering methods such as KL-quantization methods).

Author response:
(3) The paper lacks justifications for the choice of supervised evaluation. Besides, there are no experiments in semi-supervised settings or fully unsupervised evaluations.

*Answer:* After self-supervised pre-training for graph $G(\mathbf{A}, \mathbf{X})$, we obtain node representations $\mathbf{H} = [\mathbf{h}_1,...,\mathbf{h}_N]^{\top}$ and evaluate their qualities with simple linear classifier $y = \mathbf{W} x + \mathbf{b}$ under supervised settings. Good node representations should exhibit discriminability in space according to their true categories. The linear classifier under the supervised manner maps node representations to label space, and the linear classification results can well reflect the linear separability and quality of node representations. Besides, the supervised linear evaluation is a widely adopted practice in the field of self-supervised learning, including graph and computer vision. The self-supervised baselines in their papers almost universally employ linear evaluation for comparison. Thus, we also follow this practice.

Here, we evaluate the learned representations on node clustering task in an unsupervised manner, where the number of categories serves as a known parameter. The clustering results are evaluated through two metrics: Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI). The compared baselines include k-means, spectral clustering, DeepWalk [1], DNGR [2], MGAE [3], DAEGC [4], DGI [5], and GRACE [6]. The experimental results are as follows:

|       Dataset       | Cora  | Cora  | Citeseer | Citeseer | Pubmed | Pubmed |
| :-----------------: | :---: | :---: | :------: | :------: | :----: | :----: |
|     **Metrics**     |  NMI  |  ARI  |   NMI    |   ARI    |  NMI   |  ARI   |
|       k-means       | 0.377 | 0.149 |  0.241   |  0.154   | 0.263  | 0.247  |
| spectral clustering | 0.265 | 0.158 |  0.082   |  0.075   | 0.136  | 0.091  |
|      DeepWalk       | 0.401 | 0.254 |  0.238   |  0.085   | 0.251  | 0.203  |
|        DNGR         | 0.318 | 0.142 |  0.180   |  0.043   | 0.153  | 0.059  |
|        MGAE         | 0.489 | 0.436 |  0.416   |  0.425   | 0.271  | 0.224  |
|        DAEGC        | 0.528 | 0.496 |  0.397   |  0.410   | 0.266  | 0.278  |
|         DGI         | 0.565 | 0.529 |  0.431   |  0.434   | 0.281  | 0.263  |
|        GRACE        | 0.556 | 0.518 |  0.377   |  0.369   | 0.251  | 0.230  |
|       GMIM-IC       | 0.578 | 0.533 |  0.441   |  0.452   | 0.303  | 0.281  |

For DeepWalk, the number of random walks is 10, the path length of each walk is 20, the context size is 10, and the number of embedding dimension is 256. For DNGR, the encoder is built with two 512-dim hidden layers and a 256-dim embedding layer. For RMSC, the trade-off parameter is set to 0.005. For MGAE, we set the number of layers, corruption level $p$, and regularization coefficient $\lambda$ to 3, 0.4, and $10^{-5}$ respectively. For DAEGC, the encoder is constructed with a 256-dim hidden layer and a 16-dim embedding layer, and the clustering coefficient is set to 10. Due to time limitation, we currently only present the results in the table. In the future, we will conduct more clustering-related experiments.

Reviewer comment (Experiments):
There is no sensitivity analysis with respect to the encoder, presumably a GNN backbone, nor a clear comparison between the performances of this GNN backbone in a fully supervised setting versus the two-step strategy used to evaluate GMIM embeddings. Such analysis could connect to common concerns in the GNN literature, such as (i) expressivity — for example, considering several GNN layers using Jumping Knowledge-based backbones, and (ii) homophily vs. heterophily — e.g., [C], whose supervised models exhibit much higher classification performances than those reported in Table 1.

Author response:
(4) There is no sensitivity analysis w.r.t the encoder.

*Answer:* We adopt simple Graph Convolutional Networks (GCNs) [7] as the encoder, which is stated in Section 6.1. The relevant supervised counterpart is listed in Table 1. The paper [8] adopts different splits of the datasets from ours, so we kindly argue that directly comparing the two works is somewhat unreasonable. The work [8] is a great job with good performance. However, in our study, we primarily focus on the performance improvement brought by self-supervised learning strategies rather than the backbone network itself.

Reviewer comment (Evaluation):
The analysis of the results appears incomplete or arguable.

Author response:
(5) The analysis of the results is incomplete or arguable.

*Answer:* Thank you for your guidance.  In our revision, we have conducted a more detailed analysis and discussion of the experimental results.

Reviewer comment (Experiments):
The main contribution of this paper is the finding of tractable surrogate for maximizing mutual information under the Gaussian assumption, resulting in an efficient SSL method. The proposed method demonstrates effectiveness and efficiency on datasets that adhere to or approximate a Gaussian distribution. Although the author mentions that the method maintains performance under non-Gaussian conditions, this claim is based on datasets that did not pass the hypothesis testing for Gaussian distribution. While Fig. 6 shows some similarity between the dataset distribution and Gaussian distribution, the slight disparity is unlikely to significantly impact model performance. Thus, this claim might be over-sell, additional evidence with alternative datasets would be valuable.

Author response:
**Answer:** In our revision, we visualize the histograms for three additional datasets: Ogbn-Arxiv, CS, and Computers. On the CS and Computers, the representations deviate noticeably from the Gaussian distribution, but our proposed method still exhibits satisfactory performance on these two datasets. This can serve as empirical evidence for the effectiveness of our method in non-Gaussian scenarios.

Below, we analyze from the perspective of multi-view self-supervised learning why our method remains effective in non-Gaussian scenarios. Current multi-view self-supervised learning methods can be decomposed into two complementary components: a) enhancing the consistency between representations from different views, and b) employing specific strategies to avoid collapsed solutions. The former usually brings representations from different views closer in the representation space. Among existing methods, the latter includes strategies such as negative sampling (GRACE, SimCLR, and MVGRL), decorrelation strategies (CCA-SSG and Barlow Twins), and asymmetric network architectures (BYOL). In our paper, we demonstrate that our method can indeed achieve both effects a) and b) in Sections 3.2 and 5.1, and this analysis is not conducted under the Gaussian prior, which is a crucial factor contributing to the effectiveness of our approach in non-Gaussian scenarios.

If our method indeed exhibits suboptimal performance in non-Gaussian scenarios in practical applications, a coping strategy can be to impose probabilistic constraints on the representations to encourage them to approximate a Gaussian distribution. For a representation matrix $\mathbf{H} \in \mathbb{R}^{N \times d}$ from view A or B, we first calculate mean $u_i$ and variance $v_i$ of $N$ points from the $i$-th dimension. Then, a 1-dim Gaussian distribution $p_i(x)$ based on $u_i$ and $v_i$ can be constructed. Further, we calculate the product of probabilities of $N$ samples in the $i$-th dim from $p_i(x)$, that is, $\prod_{k=1}^{N} p_i(H_{ki})$. In practice, we utilize its logarithm $\sum_{k=1}^{N} \log p_i(H_{ki})$. Considering all dimensions, we construct a loss $\mathcal{L}\_{gau} = - \sum_{i=1}^{d} \sum_{k=1}^{N} \log p_i(H\_{ki})$ and make representations tend towards a Gaussian distribution by minimizing $\mathcal{L}\_{gau}$. It can be realized by attaching $\mathcal{L}_{gau}$ to the original loss. After applying the above constraints, we observed that the performance nearly did not improve on the experimental datasets. The reason is that our method is not influenced by the non-Gaussian conditions on these datasets.

Reviewer comment (Evaluation):
The results in Table 2 demonstrate a notable enhancement in the efficiency of the proposed method. The author attributes this improvement to the reduced computational cost of mutual information estimators. However, there seems to be a similarity in the calculation process between infoNCE and the proposed GMI outlined in Algorithm 3. Therefore, it is necessary to provide further clarification on the underlying reason for the efficiency difference between the two methods. Additional explanation would be beneficial in addressing this question.

Author response:
**Answer:** In addition to the objective function, our method and InfoNCE-based methods also differ significantly in the overall network architecture. Our architecture only comprises one encoder, which maps the input space to the representation space. In contrast, InfoNCE-based methods include an additional projection head following the encoder, which maps node representations to a new embedding space, adding computational burden and decreasing efficiency. In addition, we directly compute the loss function in the representation space while InfoNCE-based methods operate in the new embedding space. Compared to InfoNCE-based methods, our encoder parameters are closer to the source of gradients, allowing for better optimization and, consequently, faster convergence.

Reviewer comment (Evaluation):
Given that the objective of SSL methods is to acquire a discriminative and robust representation, it is important to include visualizations of the learned representations and conduct comparisons. Additionally, it is crucial to further investigate whether the generalization performance of the representation is influenced by the Gaussian assumption. These aspects require additional verification to provide a comprehensive evaluation of the proposed method.

Author response:
**Answer:** Thank you for your suggestions. In Appendix H.6 of our revision, we have included t-SNE visualizations of our method under different configurations and compared them with other methods, offering a more profound understanding of our approach.

Reviewer comment (Novelty):
The Gaussian assumption approach seems well motivated since it is not about the data, but rather about the embedding of data. However, there should be more references to non-linear independent component analysis and disentanglement methods for the latent space of auto-encoders, such as beta-VAE. Foundational references to decorrelation methods for independent component analysis could also be mentioned.

Author response:
**Question 6:** More discussions of uncorrelated Gaussians in auto-encoders, independent component analysis, and blind source separation are expected.

**Answer:** Our method expects to enforce the representations to achieve an isotropic Gaussian distribution with the aim of decoupling various dimensions and learning diverse representations.

Conventional Independent Component Analysis (ICA) or Blind Source Separation (BSS) thinks that the observed mixed signals are obtained through a linear combination of source signals and aims to recover latent variables from observations. The traditional ICA assumes that the source signals follow non-Gaussian distributions and are statistically independent. The assumption of statistical independence among latent variables is, in fact, a decoupling or independence constraint. Non-linear ICA posits that the observed signals are obtained through a nonlinear transformation of the source signals. 

In variational auto-encoders (VAE), there is a term used to minimize the KL divergence between the variational posterior and the prior distribution. The chosen prior distribution is typically selected to satisfy certain independent properties, such as an isotropic Gaussian distribution. As a result, the KL divergence term potentially imposes a decoupling constraint on the latent variables. The $\beta$-VAE [4] multiplies the KL divergence term by a coefficient to enhance the decoupling effect on the latent variables. The KL divergence term and the entropy maximization term in our paper share a similar underlying principle. Similar to decorrelation-based self-supervised learning methods such as CCA-SSG, DIP-VAE [5] directly regularizes the elements in the covariance matrix of the posterior distribution, making it approach the identity matrix. Generally speaking, research on disentangled representation learning in VAEs mainly focuses on improving the KL divergence term or finding its alternatives.

We are currently organizing the content related to this question. We are conducting more extensive research and discussions, and  will integrate the relevant content into our revision.

Reviewer comment (Presentation):
The organization of results is not straightforward. Property 2 is a statement of PCA, and the reasoning that "Property 2 potentially suggests that the unevenness of the eigenvalues of the covariance matrix leads to the issue of dimensional collapse" is unclear. Theorem 1 is talking about an empirical version but could be combined with Proposition 2. It seems easier to say that maximizing the determinant ensures maximum entropy and isotropic covariance.

Author response:
Dear Reviewer qTvu,

Thank you very much for your response. 

Firstly, we are sorry for having misunderstandings regarding some of your initial reviews. Indeed, the statement regarding "the unevenness of eigenvalues potentially leads to dimensional collapse issue" is not a new concept and has been well-known to the reviewer. However, in order to maintain the self-completeness of our article and facilitate a smoother understanding of the conclusions presented in our paper by readers, the relevant descriptions in Section 5.1 still hold value and significance and would be retained after modification. Specifically, to streamline and clarify the relevant statements, we have removed Theorem 1 in our new revision and reorganized our statements of Section 5.1 in a more natural manner based on your comments in "**Organization of results is not straightforward**".

Thank you for providing the reference. In our revision, we have cited the article "Elements of Information Theory". After a careful browsing, we find that this paper contains the derivation of Gaussian entropy, which is crucial for deriving Gaussian mutual information. To be honest, when deriving the Gaussian mutual information, we did not find a complete derivation process in the existing literature. The derivations presented in the paper were conducted independently by us, which consumed a significant amount of efforts on our part.

Lastly, please allow us to restate the contributions of our paper. On the one hand, in most existing methods, the presence of negative samples and additional projection heads adds redundancy and complexity of self-supervised learning (SSL) methods. Therefore, we aim to simplify SSL approaches under some specific settings. Specifically, under the Gaussian assumption, we propose a SSL method based on the feasible closed-form formula of mutual information. The advantage in efficiency is a notable strength that should not be overlooked. On the other hand, it is an important issue to establish theoretical connections and a unified perspective for different self-supervised learning methods, which has received relatively limited research attention. Our study contributes to this aspect and provides insights into bridging various self-supervised learning methods.

We would also like to express our gratitude to the other two reviewers for their efforts. We have made our best to address the weaknesses pointed out by the reviewer jkcF and eagerly await his feedback.

We sincerely appreciate your valuable comments and active discussion. Based on your feedback, we have further improved our paper and sincerely look forward to further discussion with you.

Reviewer comment (Experiments):
It is not clear why the method is not directly compared with the work by Ozsoy et al. in the main body. As mentioned in an appendix, this prior work already developed decorrelation-based self-supervision and has a similar objective. Reference: Ozsoy, Serdar, Shadi Hamdan, Sercan Arik, Deniz Yuret, and Alper Erdogan. "Self-supervised learning with an information maximization criterion." Advances in Neural Information Processing Systems 35 (2022): 35240-35253.

Author response:
**Question 3:** The authors should directly compare the proposed method with CorInfoMax [1].

**Answer:** In Table 1 of our revision, we add CorInfoMax as a baseline. When conducting related experiments, we adopt the same encoder architecture for CorInfoMax as ours. CorInfoMax falls behind our method in performance. Our method directly optimizes Gaussian mutual information in the representation space, while CorInfoMax adds an additional projection head following the encoder. This component is somewhat redundant and hinders the direct optimization of representations.

Reviewer comment (Reproducibility):
Details about hyper-parameter search are unclear. How are hyper-parameters searched? What are the details of the linear classifier, given that many are possible with differing regularization or penalty hyper-parameters, and how are these selected? To be truly self-supervised, a validation set would not be available. Is it a grid search on dimension and trade-off parameter? The augmentation intensities \( p_f \) and \( p_e \) must be selected, and the results in Figure 10 show these have a large effect. The variation across hyper-parameters seems greater than the performance gains of the proposed methodology. CCA-SSG outperforms GMIM, and GMIM-IC (with a hyper-parameter) outperforms CCA-SSG.

Author response:
**Question 4:** Some questions and issues regarding comparison to previous works and experimental settings.

(1) To be truly self-supervised learning, a validation set would not be available. Is a grid search employed for hyper-parameter selection via a validation?

*Answer:* Based on a validation set in the downstream node classification task, we conduct a grid search for hyper-parameters of the proposed method in our experiments. In our paper, all self-supervised baselines go through such a process. *During the self-supervised pretraining process, no labels are involved in the training.*

Indeed, as you mentioned, truly self-supervised learning should not have access to a validation set. However, in most existing literature including classical works in both **vision and graph fields** (e.g., MAE, GRACE, and CCA-SSG), a hyper-parameter search is performed based on a validation of downstream task. In other words, they also do not achieve truly self-supervised learning. We think that this issue is primarily due to the huge gap between self-supervised proxy tasks and downstream tasks such as classification in the graph and vision fields. A well-performed self-supervised proxy task (e.g., a small InfoNCE loss) does not guarantee that the pre-trained model will still perform well on downstream classification tasks. Therefore, it is often necessary to perform hyper-parameter search based on the validation set of downstream tasks. Without relying on a validation set, constructing specific metrics or strategies to measure the effectiveness of self-supervised pretraining remains an important under-explored research problem. This is also a significant reason why the research on self-supervised learning in the fields of vision and graph lags behind that in the NLP. In NLP, proxy tasks like masked language modeling align well with downstream applications such as question-answering. Generally speaking, the issue the reviewer points out is actually one that the entire self-supervised learning community for graph and vision is facing and actively seeking solutions for.

Reviewer comment (Experiments):
It is unclear whether competing baselines use defaults or also undergo a hyper-parameter search. There is uncertainty as to whether the hyper-parameter search is fair: the statement "to get relevant results based on the officially released source code" does not clarify if hyper-parameter selection is used for all methods.

Author response:
(4) Are competing baselines using defaults or do they also enjoy a hyper-parameter search?

*Answer:* Based on the official codes of the baselines, we perform a hyper-parameter search to obtain their best results. For instance, in the case of GRACE, the official code is used to conduct experimental evaluations on random splits of Cora, Citeseer, and Pubmed. The given hyper-parameters perform poorly on the public splits, and we obtained the reported results in our paper after conducting a hyper-parameter search.

Reviewer comment (Writing):
The phrase "as a pivot to elucidate" is unclear.

Author response:
(2) What does "as a pivot to elucidate" means?

*Answer:*  GMIM-IC is a contrastive objective derived from maximizing Gaussian Mutual Information. Here, "as a pivot to elucidate" means that we will demonstrate the relationship between contrastive methods and decorrelation-based methods using GMIM-IC as an *intermediary*.