From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel unsupervised domain adaptation method that stands in the covariate shift for Graph Convolutional Networks model. The paper discusses the relevance of covariate shift assumption using contextual stochastic block model (CSBM) on node classification task. Then an optimal transport based approach is presented which does not rely on the covariate-shift assumption.

Strengths:
-The paper studies an important problem
-The method is novel and brings interesting ideas


Weaknesses:
-The paper lacks of clarity and a rewriting appears necessary
-Some choices and claims are sufficiently justified
-The experimental validation is limited (significance of the results unclear, no code provided).
  
Overall, there is clear consensus for saying that this contribution is not ready and a major revision is necessary. I propose then rejection. 


justification_for_why_not_higher_score: All reviewers agreed that the paper lacks clarity and that a major rewriting is needed, in addition to other weaknesses. During rebuttal, authors' answers were rather short, no paper update has been proposed.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5913/Authors
title: Author's response to Reviewer chHv
comment: We thank the reviewer for thoughtful comments and time spent reviewing our paper.

### 1. Difference with the related work mentioned by the reviewer.
We would like to note the difference between image domain and graph domain. The paper mentioned in the review conducts UDA on MRI data, which consists of convolutional neural networks across different. In image domain, various domain invariant learning algorithms are all valid because of 

### 2. The reason behind adopting CSBM graphs for analysis in the paper.
The reasons are twofold: (1) the distribution after graph neural network (e.g. GCN) follows a gaussian mixture model, which stands for (2) CSBM can easily illustrate how covariate shift assumption is not satisfied with structure p/q changes


### 3. Better organization of the paper.
We will move current section 3.1 into appendix because the main contribution of our work lies in overcoming the reliance on covariate shift by optimal transport. Current section 3.1 originally aims to justify the out-of-distribution problem on graph neural networks. We understand it might distract the audience and decide to make the aforementioned change.

From: ICLR.cc/2023/Conference/Paper5913/Authors
title: Author's response to Reviewer rFzG
comment: We thank the reviewer for thoughtful comments and time spent reviewing our paper. 

### 1. On the clarity of Section 3.1
In this section, we use the popular data model for graph neural networks - contextual stochastic block model (CSBM) to describe our observations on distribution shift. According to definition 3.1, in two-class CSBM, p,q denotes edge probability between same and different class nodes, respectively. We add the introduction of homophily ratio to definition 3.1 for clarity. 

### 2. Discussion of graph setup in Section 3.2
We think this is a great suggestion for our work. In the future revision, we will revise Theorem 3.2 by introducing the VC dimension of graph neural networks. In addition, in the footnote or the main paper, we explain HΔH as a symmetric difference of H-divergence in domain adaptation. Specifically, H-divergence [1] is defined as the probability divergence of hypothesis class h \in H on source D and target data D’.

### 3. The limitation of covariate shift assumption on graph domain adaptation.
We would think of adding the original CSBM graph to this figure to clarify how graph structure causes more challenges for distribution shifts.

### 4. Embedding h missing in Lemma 4.1.
We apologize for the confusion here. In Lemma 4.1, we intend to be consistent with our definition Definition 4.1. In fact, your interpretation is correct, here h is the \mu \sim Ps(X, Y). We modified the Definition and Lemma 4.1into h in the revision.

### 4. Additional discussion on synthetic experiments (Figure 5)
In literature[2], graph neural networks have shown better theoretical and empirical generalization to structural shifts on CSBM graphs. It leads to a smaller gap between in-distribution and out-of-distribution training data. Meanwhile, in our analysis corollaries 3.1.1 and 3.1.2, the feature shifts would be more severe for CSBM graphs. Hence, it is reasonable to see smaller gaps on structural shifts. Our experiments in 5.3 shows on non-CSBM graphs, structural shifts can also affect the performance a lot (i.e. >10% test ROC_AUC). 

We will fix all of the typos in the revision. 

[1] Ben-David, Shai, et al. "A theory of learning from different domains." Machine learning 79.1 (2010): 151-175.
[2] Baranwal, Aseem, Kimon Fountoulakis, and Aukosh Jagannath. "Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization." arXiv preprint arXiv:2102.06966 (2021).

From: ICLR.cc/2023/Conference/Paper5913/Authors
title: Author's response to Reviewer 1ebR
comment: ### 1. The claim that ``covariate shift assumption rarely holds’' in general is not well supported and should be better substantiated. The provided evidence is only for the CSBM.

We would like to note that not only CSBM experiments but our sections 5.3 and 6.2 all support the sub-optimal performance of unsupervised domain adaptation algorithms. But we agree with the reviewer that we should support this argument more clearly in multiple scenarios. In our appendix, we will add figures on syn-cora/syn-product to support that discrepancy measure based on covariate shift cannot reflect the performance gap properly. 

### 2. More comprehensive introduction of the CSBM model. The overall organization and presentation of the paper should be improved.

We appreciate the suggestion to introduce the data model more comprehensively. The reason for the CSBM model being widely used in theoretical study of graph neural networks is its closeness to the gaussian mixture model. Before we gain insight into sophisticated feature space, we aim to conduct the first theoretical study of distribution shift on graph domain in this paper.
We also agree with the reviewer that the organization of the sections can be improved. For example, we plan to move discussion on OOD generalization of graph convolution into appendix, so the discussion on domain adaptation will be more coherent from motivation to methodology. 


### 3. Experiments on real datasets (Table 1 and 2) are not convincing. No significance testing is reported (results from Cora/Citeseer Table 1 ACMtime/ACM-DBLPlarge Table 2 show small improvement with high standard deviation).

We will report the standard deviation in the revision. The reason for high deviation is caused by the small training size of multiple different biased training seeds used in semi-supervised experiments.

From: ICLR.cc/2023/Conference/Paper5913/Authors
title: Author Response to Reviewer fttP 
comment: We thank the reviewer for thoughtful comments and time spent reviewing our paper. 

### 1. Theoretical contribution of the paper
The theoretical contribution of the paper is on the topic of graph domain adaptation. To this end, we first conduct the first theoretical analysis of graph convolution OOD generalization on feature and structure shift. Then we analyze the insufficiency of domain invariant representation learning and discuss the background of optimal transport. Thirdly, we prove the graph optimal transport cost can be used as a divergence measure in UDA. We believe these three contributions are novel and solid enough. We would like to address your concerns as the following:

### 2. Questions about UDA setting of the paper

In graph learning, both supervised and semi-supervised settings are popular and widely studied.
We note that when we mentioned supervised or semi-supervised settings, they are meant to be on training domain. In other words, we only utilized unlabeled target data in all of the experiments, which is exactly the UDA setting.

### 3. Comparison against f-domain adversarial learning
First, we would like to thank the reviewer for pointing out an interesting relevant work. We would like to provide a comparison of applying f-domain adversarial learning on Graph Neural Networks. Nevertheless, in our existing analysis, this work stresses the limitation of learning invariant representation in graph structured data and [1] also falls in this category. 

[1] Acuna, David, et al. "f-domain adversarial learning: Theory and algorithms." International Conference on Machine Learning. PMLR, 2021.

From: ICLR.cc/2023/Conference/Paper5913/Reviewer_rFzG
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper focuses on the performance of a specific family of GNNs i.e., GCNs, under domain shift. The authors propose a domain adaptation algorithm based on the notion of Optimal Transform that searches for a transportation plan that minimizes the cost of the joint feature and estimated label distribution between source and target domains. The method is validated on synthetic and real dataset.

strength_and_weaknesses: Strength: The problem is very relevant, and timely. Solving it with an OT approach is interesting and promising.

Weakness: From the methodology viewpoint, the results are valid only for a specific class of GNNs, I.e, GCNs, and for only one layer. The graph is assumed to follow a continual stochastic block model. From the presentation viewpoint, the paper is quite technical and not easy to follow.

clarity,_quality,_novelty_and_reproducibility: The paper deals with a relatively novel approach to a well-known problem in ML. It is quite dense and difficult to follow. The results should be reproducible once the authors release the code. The hyperparements are well-documented in the appendix. 
summary_of_the_review: The paper needs a major rewriting in order to be more comprehensive and insightful. Below are a few comments that could help in that direction.

- Discussion on the corollaries 3.1.1 and 3.1.2:. What does homophily ratio mean? How does it change with respect to the density/connectivity of the graph? That is key for the rest of the paper. Same for 3.1.2.

- Definition 3.2: Given that we are discussing about a graph setup, the graph should also appear there.

- Define VC dimension, even at a high level. It would be useful to make the paper accessible to a wider audience. 

- The terms $H\Delta H$ is used after equation (3) but it is not defined/explained adequately. 

- g is used for both embeddings and functions 

- The figure of the covariate shift assumption (p.4)  is informative to demonstrate condition shift but at the same time it is confusing as it is not obvious what is the underlying graph and how these embeddings are computed. 

- In lemma 4, the authors define a cost function as a metric on the GCN representations h, but that h does not appear in that function.

- Can you please discuss the results of Fig.5? Why the gain is less obvious for structural shifts?

- Please define the abbreviations such as CDF. 

- The improvement that we see in terms of semi-supervised classification is marginal. Can you please comment on why?


Various typos exists in the text: 
- p.3 ‘netwprk’
- p.4 ‘barawal2022effects’ probably corresponds to a citation (?)
- theorem 3.2: please remove redundant ]
- p.4 'different classes' instead of 'different class'
- p.6 ‘1-Wasserstein’ instead of '1-wasserstein'
- p.7 ‘the density of the graph’ instead of ‘the density of graph’
- p.7 ‘GDOT yields’ instead of 'GDOT yield'




correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No flags to report. 
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5913/Reviewer_fttP
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to explore the generalization ability of GCNs on out-of-distribution data.
The authors observe that the widely used covariate shift assumption rarely holds for graph data and thus propose to address bias in graph models using domain adaptation with optimal transport.
Experiments are conducted to show the effectiveness of their proposed method.
strength_and_weaknesses: machine-learning community and is barely well-solved. Let alone the graph data cases. However, there are still some issues in this work needed to be improved before publishing:
1)	The contributions of this work seem limited, particularly the theoretical parts. The theoretical results are just summarized by the existing literature, and the authors might not present some new conclusion or need to provide more discussions motivated by their results for this work.
2） Another concern is that the VC dimension of Thm3.2 in (3) would be very large since the backbone is a graph neural network (May be linear/square with the parameters). Thus, I wonder about the persuasiveness of the bound. Or I hope the authors could give me some discussions about this.
3）The experiments are not convincing:
	a) The UDA competitors for non-graph data are out-of-date, and the authors should consider more SOTA methods such as [1].
	b) The authors conduct the experiments under both supervised and semi-supervised settings. But in most cases, the DA problem is unsupervised. Moreover, I notice that this paper focuses on developing Unsupervised DA. So, I believe the empirical studies under UDA should be included to show the effectiveness.
Ref:
[1] f-domain adversarial learning: Theory and algorithms. 

clarity,_quality,_novelty_and_reproducibility: Quality: good
Novelty: fair
Reproducibility: fair
summary_of_the_review: This work attacks a challenging problem in the machine learning community. However, there are still some issues in this work needed to be improved before being accepted, such as clear contributions and more convincing empirical studies.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5913/Reviewer_1ebR
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work considers the problem of OOD generalization of GNNs for node classification.
The paper models domain shift with different Contexual Stochastic Block Models (CSBM) and show that the covariate shift assumption does not hold in that setting.
As a result, this work proposes an efficient method based on optimal transport for unsupervised graph domain adaptation that does not rely on the covariate shift assumption.
This work presents experiment on synthetic and real datasets.

strength_and_weaknesses: Strengths:
1) This paper tackles an important problem with a principled approach. Using optimal transport for unsupervised graph domain adaptation is novel.
2) The synthetic numerical experiments are convincing. The 3rd novelty point : ``On synthetic and real datasets, GDOT is able to successfully mitigate the domain shift’' is well supported for the synthetic experiment.

Weak points:
1) The claim that ``covariate shift assumption rarely holds’' in general is not well supported and should be better substantiated. The provided evidence is only for the CSBM. 
2) Considering that the majority of this work’s contribution is tied to the CSBM, this modeling assumption should be justified. In particular, a better justification for the parameterization chosen (changing p/q ratio) and why it is representative of a plausible domain shift for graph data should be provided.
3) The overall organization and presentation of the paper should be improved.
4) Experiments on real datasets (Table 1 and 2) are not convincing. No significance testing is reported (results from Cora/Citeseer Table 1 ACMtime/ACM-DBLPlarge Table 2  show small improvement with high standard deviation).

Additional details and questions:
1.1 Please clarify what is meant by: ``Additionally, these approaches require either covariate shift or multiple training environments which are not justified on structured data’' Why are multiple training environments not justified?
1.2 The claim in Section 4 : ``In the previous section, we demonstrated the limitations of DIRL on graph data when covariate shift does not hold.‘’ needs more substantiation.
In that previous section, the authors gave a convincing example of how and why it could fail using a figure, but this was tied to the CSBM model and did not provide evidence that the covariate shift assumption does not (approximately) apply in real-world graph data.

3.1 A clear specification of the considered problem setting would greatly help the reader understand the purpose of this work.
3.2 ``We use a slightly different version here for analysis’' What are the consequences of using a different version in the analysis? Do the results follow for this version as well?
3.3 Appendix A.1 states that : ``Without loss of generality, we let two graphs have same amount of nodes n and focus on one class (i.e. y_i = 1) of graph G’‘. I am not convinced that focusing on the binary case does not lead to a loss of generality. Can the authors provide an argument as to why this is the case?
3.4 The proof needs some polishing. Why is the assumption ``On testing graph, we assume only one class centroid moves or graph structure (p,q) changes’' sensible?
3.5 The loss equation is presented as one expression, but the algorithm seems to refer to a part of this loss. This should be clearly expressed in two equations.
3.6 The symbols Ps(Y |X) = Pt(Y |X) should be defined before use. Could you comment on where the graph symbol would appear?
3.7 More details about Figure 3 should be provided. How are MMD and CMD computed?
3.8 The notation of CSBM(u, −u, p, q) should be defined in the CSBM definition.
3.9 In Definition 3.2, the phrase ``that generalizes well on target domain’' should be replaced by a more formal definition. The concept of “generalizing well” is too vague. 
3.10 In Definition 3.1 ``is a Gaussian random variable’' with what mean and variance?
3.11 The reference section needs improvement. There are inconsistencies in the bibliography and many published works are listed as arxiv references.
For example: Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017, is an ICLR paper.
3.12 Please clarify the sentence: ``Calculating the thresholds from corollary 3.1.1, we get 5376 > n = 128 (p/q=1) and 91.5 < n (p/q=2) which aligns well with result in Figure 4a.” How does this align with Figure 4a?

Typo after Eq.2: linear netwprk,
Typos after Corollary 3.1.2 ``recently baranwal2022effects’’.

clarity,_quality,_novelty_and_reproducibility: Clarity/Quality : The overall organization and presentation of the work lack clarity as outlined above.

Novelty : This work is novel. Both the theoretical analysis and the proposed optimal transport idea are interesting contributions that are distinct from existing work. 

Reproducibility: Code as well as a description of model architecture and hyperparameters are provided.
summary_of_the_review: The paper present an interesting idea and makes a novel contribution. The presentation of the paper could be improved significantly to better convey the key ideas of the work. The key weaknesses lie in the absence of justifications of assumptions and claims. There is a general claim that the paper demonstrates that covariate shift assumption rarely holds in graph data, but the paper does not seem to provide evidence to support this, with the arguments focusing on the special case of the CSBM. The CSBM underpins the majority of the contributions but there is little supporting evidence to suggest that this is a good model for distribution shift in graph data. The theoretical analysis contains claims about generality and specific assumptions about the test graph that do not appear to be justified. 


correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
details_of_ethics_concerns: Not applicable.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5913/Reviewer_chHv
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces a novel graph domain adaptation framework based on optimal transport. Using Context Stochastic Block Models, the authors provide limits of the out-of-distribution generalisation of GNN for feature and structural changes. The algorithm considered is a mini-batch method optimal transport domain adaptation (called GDOT). 
strength_and_weaknesses: Strengths. The paper contains a number of technical details. The results of the numerical experiments report that the method performs well. 

Weaknesses. The paper is in general difficult to follow. High level ideas are missing, and it is not clear why the CSBM were chosen to be considered.
Some abbreviations are never explained, OOD (in Introduction), what GDOT stands for?   
A number of statements need clarification. So, what is meant by  "an algorithm does not assume covariate shift" (in Introduction): is it meant that there is not any covariate shift, or the algorithm copes with the shift well in any case? What is a semi-supervised distribution shift?

In Introduction, it is stated that "our results show when the generalisation of a single-layer graph convolutional network would be worse than that of a linear classifier". Is this result (Theorem 3.1) stands for the case of CSBM only?

In Introduction: why a covariate shift is not justified on structural data? 

It seems that the authors switch between GNN and GCN all the time in the paper. Is there any reason for that? 

The results of Corollaries (3.1.1 and 3.1.2) are used in the paper? Is \delta from Corollary 3.1.2 and \delta in Theorem 3.2 the same?

I do not understand the motivation for figure on page 4 illustrating the conditional probabilities in a well-specified and ill-specified models, especially after quite technical results (two theorems and corollaries). This figure suits much more in Introduction, to motivate the paper. 

In Section 4: what is meant under "when covariate shift does not hold"? There is not any covariate shift, or this means that in addition to P_s(X) \neq P_t(X) we have that P_s(Y|X) \neq P_t(Y|X), i.e. that the model is misspecified (ill-specified)?

In Algorithm 1: there is no any $g$ is equation 4. 

clarity,_quality,_novelty_and_reproducibility: I have some concerns about the novelty. The GDOT algorithm resembles a lot one introduced in "Unsupervised Domain Adaptation With Optimal Transport in Multi-Site Segmentation of Multiple Sclerosis Lesions From MRI Data", Front Comput Neurosci., 2020. This paper is not cited in the current submission. 
The loss function of the current submission (eq. 4) is the same as one in the above mentioned paper (eq. 6 in the published article). Algorithm 1 of the current submission is exactly the same as one from the above mentioned paper. 
I guess that the novelty (limited in my opinion) is that graphs (contextual SBMs) are considered. 

As far as I can see, the code is not provided, and it is not obvious to reproduce the results based on the submission only. 

summary_of_the_review: A number of points need to be clarified. 

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough