Original: Learning Invariant Graph Representations for Out-of-Distribution Generalization
Revised: Learning Invariant Graph Representations for Out-of-Distribution Generalization

Original: Abstract
Revised: Abstract

Original: Graph representation learning has shown effectiveness when testing and training graph data come from the same distribution, but most existing approaches fail to generalize under distribution shifts.
Revised: Graph representation learning has shown effectiveness when testing and training graph data come from the same distribution, but most existing approaches fail to generalize under distribution shifts.

Original: Invariant learning, backed by the invariance principle from causality, can achieve guaranteed generalization under distribution shifts in theory and has shown great successes in practice.
Revised: Invariant learning, backed by the invariance principle from causality, can achieve guaranteed generalization under distribution shifts in theory and has shown great successes in practice.

Original: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.
Revised: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.

Original: To solve this problem, we propose Graph Invariant Learning ( GIL ) model capable of learning generalized graph representations under distribution shifts.
Revised: To solve this problem, we propose Graph Invariant Learning ( GIL ) model capable of learning generalized graph representations under distribution shifts.

Original: Our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three tailored modules.
Revised: Our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three tailored modules.

Original: Specifically, we first design a GNN-based subgraph generator to identify invariant subgraphs.
Revised: Specifically, we first design a GNN-based subgraph generator to identify invariant subgraphs.

Original: Then we use the variant subgraphs, i.e., complements of invariant subgraphs, to infer the latent environment labels.
Revised: Then we use the variant subgraphs, i.e., complements of invariant subgraphs, to infer the latent environment labels.

Original: We further propose an invariant learning module to learn graph representations that can generalize to unknown test graphs.
Revised: We further propose an invariant learning module to learn graph representations that can generalize to unknown test graphs.

Original: Theoretical justifications for our proposed method are also provided.
Revised: Theoretical justifications for our proposed method are also provided.

Original: Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our method against state-of-the-art baselines under distribution shifts for the graph classification task.
Revised: Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our method against state-of-the-art baselines under distribution shifts for the graph classification task.

Original:  Introduction
Revised:  Introduction

Original: Graph structured data is ubiquitous in the real world, e g., social networks, biology networks, chemical molecules, etc.
Revised: Graph structured data is ubiquitous in the real world, e g., social networks, biology networks, chemical molecules, etc.

Original: Graph representation learning, which encodes graphs into vectorized representations, has been the central topic in graph machine learning in the last decade.
Revised: Graph representation learning, which encodes graphs into vectorized representations, has been the central topic in graph machine learning in the last decade.

Original: For example, graph neural networks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information and are shown to be successful in a variety of applications.
Revised: For example, graph neural networks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information and are shown to be successful in a variety of applications.

Original: Despite the enormous success the existing approaches for learning graph representations heavily rely on the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from an identical distribution.
Revised: Despite the enormous success, the existing approaches for learning graph representations heavily rely on the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from an identical distribution.

Original: However, distribution shifts of graph data widely exist in real-world scenarios and are usually inevitable due to the uncontrollable underlying data generation mechanism [4].
Revised: However, distribution shifts of graph data widely exist in real-world scenarios and are usually inevitable due to the uncontrollable underlying data generation mechanism [4].

Original: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.
Revised: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.

Original: One critical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely on correlations that are variant for graphs from different environments.
Revised: One critical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely on correlations that are variant for graphs from different environments.

Original: Therefore, it is of paramount significance to learn graph representations under distribution shifts and develop methods capable of out-of-distribution (OOD) generalization.
Revised: Therefore, it is of paramount significance to learn graph representations under distribution shifts and develop methods capable of out-of-distribution (OOD) generalization.

Original: Such studies are particularly critical for high-stake graph applications such as medical diagnosis [5], financial analysis [6], molecular prediction [7], etc.
Revised: Such studies are particularly critical for high-stake graph applications such as medical diagnosis [5], financial analysis [6], molecular prediction [7], etc.

Original: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.
Revised: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.

Original: Invariant learning, which aims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spurious correlations, can provably achieve satisfactory OOD generalization under distribution shifts [8–10].
Revised: Invariant learning, which aims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spurious correlations, can provably achieve satisfactory OOD generalization under distribution shifts [8–10].

Original: Though invariant learning has been studied for images and texts [8, 11], it remains largely unexplored in the literature of graph representation learning.
Revised: Though invariant learning has been studied for images and texts [8, 11], it remains largely unexplored in the literature of graph representation learning.

Original: However, invariant graph representation learning is non-trivial due to the following challenges.
Revised: However, invariant graph representation learning is non-trivial due to the following challenges.

Original: First, graph data usually comes from a mixture of latent
Revised: First, graph data usually comes from a mixture of latent

Original: [Figure]
Revised: [Figure]

Original: environments without accurate environment labels, as shown in Figure 1.
Revised: environments without accurate environment labels, as shown in Figure 1.

Original: Since most invariant methods require multiple training environments with explicit environment labels, these existing methods cannot be directly applied to graphs.
Revised: Since most invariant methods require multiple training environments with explicit environment labels, these existing methods cannot be directly applied to graphs.

Original: Second, the formation process of graphs is affected by the complex interaction of both invariant and variant patterns.
Revised: Second, the formation process of graphs is affected by the complex interaction of both invariant and variant patterns.

Original: How to identify the invariant patterns among latent environments is even more challenging.
Revised: How to identify the invariant patterns among latent environments is even more challenging.

Original: Last but not least, even after having obtained the environmental labels, how to design a theoretically grounded learning scheme to generate graph representations capable of OOD generalization remains largely unexplored.
Revised: Last but not least, even after having obtained the environmental labels, how to design a theoretically grounded learning scheme to generate graph representations capable of OOD generalization remains largely unexplored.

Original: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL ) which is able to capture invariant graph patterns in a mixture of latent environments and capable of OOD generalization under distribution shifts.
Revised: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL ) which is able to capture invariant graph patterns in a mixture of latent environments and capable of OOD generalization under distribution shifts.

Original: As shown in Figure 2, our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three mutually promoting modules, with each module tackling one aforementioned challenge.
Revised: As shown in Figure 2, our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three mutually promoting modules, with each module tackling one aforementioned challenge.

Original: Specifically, in the invariant subgraph identification module, we design a GNN-based subgraph generator to identify potentially invariant subgraphs from the complex interaction between invariant and variant patterns.
Revised: Specifically, in the invariant subgraph identification module, we design a GNN-based subgraph generator to identify potentially invariant subgraphs from the complex interaction between invariant and variant patterns.

Original: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering these environment-discriminative features.
Revised: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering these environment-discriminative features.

Original: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.
Revised: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.

Original: Lastly, in the invariant learning module, we propose to optimize the maximal invariant subgraph generator criterion given the identified invariant subgraphs and inferred environments to generate graph representations capable of OOD generalization under distribution shifts.
Revised: Lastly, in the invariant learning module, we propose to optimize the maximal invariant subgraph generator criterion given the identified invariant subgraphs and inferred environments to generate graph representations capable of OOD generalization under distribution shifts.

Original: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generator of our GIL , and further prove that our GIL satisfies permutation invariance.
Revised: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generator of our GIL , and further prove that our GIL satisfies permutation invariance.

Original: We conduct extensive experiments on both synthetic graph datasets and real graph benchmarks for the graph classification task.
Revised: We conduct extensive experiments on both synthetic graph datasets and real graph benchmarks for the graph classification task.

Original: The results show that the representations learned from GIL achieve substantial performance gains on the unseen OOD testing graphs compared with various state-of-the-art baselines.
Revised: The results show that the representations learned from GIL achieve substantial performance gains on the unseen OOD testing graphs compared with various state-of-the-art baselines.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows

Original: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized graph representations under distribution shifts.
Revised: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized graph representations under distribution shifts.

Original: To the best of our knowledge, we are the first to study invariant learning for graph representation learning under a mixture of latent environments.
Revised: To the best of our knowledge, we are the first to study invariant learning for graph representation learning under a mixture of latent environments.

Original: • Our proposed method can automatically infer the environment label of graphs from a mixture of latent environments without supervision.
Revised: • Our proposed method can automatically infer the environment label of graphs from a mixture of latent environments without supervision.

Original: • We propose maximal invariant subgraph generator criterion to learn graph representations capable of OOD generalization under distribution shifts.
Revised: • We propose maximal invariant subgraph generator criterion to learn graph representations capable of OOD generalization under distribution shifts.

Original: • We theoretically show that finding a maximal invariant subgraph generator of GIL can solve the OOD generalization problem.
Revised: • We theoretically show that finding a maximal invariant subgraph generator of GIL can solve the OOD generalization problem.

Original: Extensive empirical results demonstrate the effectiveness of GIL on various synthetic and benchmark datasets under distribution shifts.
Revised: Extensive empirical results demonstrate the effectiveness of GIL on various synthetic and benchmark datasets under distribution shifts.

Original:  Notations and Problem Formulation
Revised:  Notations and Problem Formulation

Original: Notations.
Revised: Notations.

Original: Let G and Y be the graph and label space.
Revised: Let G and Y be the graph and label space.

Original: We consider a graph dataset G = {
Revised: We consider a graph dataset G = {

Original: ( G i , Y i ) } Ni =
Revised: ( G i , Y i ) } Ni =

Original: where G i ∈ G and Y i ∈ Y .
Revised: where G i ∈ G and Y i ∈ Y .

Original: Following the OOD convention [11, 8, 9], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } N e i = denotes the dataset from environment e , supp( E tr ) is the support of the environmental variable in the training data.
Revised: Following the OOD convention [11, 8, 9], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } N e i = denotes the dataset from environment e , supp( E tr ) is the support of the environmental variable in the training data.

Original: We use G and Y to denote the random variables of graph and label, and G e and Y e tospecify random variables from environment e .
Revised: We use G and Y to denote the random variables of graph and label, and G e and Y e to

Original: The environment label for graphs is unobserved since it is prohibitively expensive to collect graph environment labels for most real scenarios.

Original: [Figure]
Revised: [Figure]

Original: Prediction Φ(⋅)
Revised: Prediction Φ(⋅)

Original: Clustervariantsubgraphs {" - } toinferenvironments …
Revised: Clustervariantsubgraphs {" - } toinferenvironments …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B % =
Revised: B % =

Original: Tree …
Revised: Tree …

Original: B & = Wheel
Revised: B & = Wheel

Original: Environment Inference 0 /.

Revised: Environment Inference 0 /.

Original: = 1 /2 3 ⋅

Revised: = 1 /2 3 ⋅

Original: GNN
Revised: GNN

Original: Invariant Subgraph Identification
Revised: Invariant Subgraph Identification

Original: Noderepresentation G (*) … … Graphdataset
Revised: Noderepresentation G (*) … … Graphdataset

Original: EdgeMask R ..0 0 .… 0 .
Revised: EdgeMask R ..0 0 .… 0 .

Original: Invariant&variantsubgraphs {( 1 },{( ( } 2 4 =
Revised: Invariant&variantsubgraphs {( 1 },{( ( } 2 4 =

Original: Top 5 (0⊙2) 2 6 = 2 - 

Revised: Top 5 (0⊙2) 2 6 = 2 -

Original: Graphsfromamixtureoflatent environments
Revised: Graphsfromamixtureoflatent environments

Original: [Figure]
Revised: [Figure]

Original: Training Dataset
Revised: Training Dataset

Original: Invariant subgraphs {" 7 }
Revised: Invariant subgraphs {" 7 }

Original: Variant subgraphs {" - }
Revised: Variant subgraphs {" - }

Original: I n f e rr ed en v i r on m en t s ℰ VWXYZ
Revised: I n f e rr ed en v i r on m en t s ℰ VWXYZ

Original: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =
Revised: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =

Original: Wheel
Revised: Wheel

Original: Labels: 9 = 9 = …
Revised: Labels: 9 = 9 = …

Original: Testing Dataset % ∘ '(⋅)
Revised: Testing Dataset % ∘ '(⋅)

Original: Invarianceregularizer
Revised: Invarianceregularizer

Original: Shared H∘J(⋅)
Revised: Shared H∘J(⋅)

Original: H∘J(⋅)
Revised: H∘J(⋅)

Original: H∘J(⋅) Shared ∇ , ℛ - ! ∇ , ℛ - " ∇ , ℛ - # \trace(Var(∇ , ℛ - ))
Revised: H∘J(⋅) Shared ∇ , ℛ - ! ∇ , ℛ - " ∇ , ℛ - # \trace(Var(∇ , ℛ - ))

Original: Invariant Learning …
Revised: Invariant Learning …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B & = Wheel …
Revised: B & = Wheel …

Original: B % =
Revised: B % =

Original: Tree ℛ - ! ℛ - " ℛ - # Classificationloss [ - (ℛ - ) + =
Revised: Tree ℛ - ! ℛ - " ℛ - # Classificationloss [ - (ℛ - ) + =

Original: Objective function () =
Revised: Objective function () =

Original: + ,
Revised: + ,

Original: =
Revised: =

Original: % ∘ ' ∘ Φ(,)
Revised: % ∘ ' ∘ Φ(,)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: specify random variables from environment e .

Revised: The environment label for graphs is unobserved since it is prohibitively expensive to collect graph environment labels for most real scenarios.

Original: Problem Formulation.
Revised: Problem Formulation.

Original: We formulate the generalization under distribution shifts on graphs as:
Revised: We formulate the generalization under distribution shifts on graphs as:

Original: Problem 1.
Revised: Problem 1.

Original: Let E denote the random variable on indices of all possible environments.
Revised: Let E denote the random variable on indices of all possible environments.

Original: Our goal is to find an optimal predictor f ∗ ( · ) :
Revised: Our goal is to find an optimal predictor f ∗ ( · ) :

Original: G → Y that performs well on all environments:
Revised: G → Y that performs well on all environments:

Original: [Equation]
Revised: [Equation]

Original: where R ( f | e )
Revised: where R ( f | e )

Original: =
Revised: =

Original: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :
Revised: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :

Original: Y × Y → R denotes a loss function.
Revised: Y × Y → R denotes a loss function.

Original: We further decompose f ( · ) =
Revised: We further decompose f ( · ) =

Original: w ◦ h , where h ( · ) : G → R d is the representation learning function, d is the dimensionality, and w ( · ) : R d → Y is the classifier.
Revised: w ◦ h , where h ( · ) : G → R d is the representation learning function, d is the dimensionality, and w ( · ) : R d → Y is the classifier.

Original: Note that supp( E tr ) ⊂ supp( E ) .
Revised: Note that supp( E tr ) ⊂ supp( E ) .

Original: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =
Revised: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =

Original: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,
Revised: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,

Original: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and the corresponding label is different for training and testing graph data.
Revised: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and the corresponding label is different for training and testing graph data.

Original: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [8, 9].
Revised: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [8, 9].

Original: In addition, for most graph datasets, we do not have access to accurate environment labels or environment partitions.
Revised: In addition, for most graph datasets, we do not have access to accurate environment labels or environment partitions.

Original: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good
Revised: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good

Original: OOD generalization performance under the inferred environments.
Revised: OOD generalization performance under the inferred environments.

Original: The problem is formulated as:
Revised: The problem is formulated as:

Original: Problem 2.
Revised: Problem 2.

Original: Given a graph dataset G collected from a mixture of latent environments but without environment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) , and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achieve good OOD generalization performance.
Revised: Given a graph dataset G collected from a mixture of latent environments but without environment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) , and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achieve good OOD generalization performance.

Original:  Method
Revised:  Method

Original: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.
Revised: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.

Original: We first present the invariant subgraph identification module.
Revised: We first present the invariant subgraph identification module.

Original: Then, we infer environment labels by clustering the variant subgraphs.
Revised: Then, we infer environment labels by clustering the variant subgraphs.

Original: Next, we introduce the maximal invariant subgraph generator criterion to generate graph representations which can generalize to test graphs under distribution shifts.
Revised: Next, we introduce the maximal invariant subgraph generator criterion to generate graph representations which can generalize to test graphs under distribution shifts.

Original: Lastly, we provide some discussions of our proposed method.
Revised: Lastly, we provide some discussions of our proposed method.

Original: 3.1 Invariant Subgraph Identification
Revised: 3.1 Invariant Subgraph Identification

Original: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationship with the label is invariant across different environments.
Revised: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationship with the label is invariant across different environments.

Original: We refer to the rest of the graph, i.e., the complement of G I , as the variant subgraph and denote it as G V .
Revised: We refer to the rest of the graph, i.e., the complement of G I , as the variant subgraph and denote it as G V .

Original: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.
Revised: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.

Original: Therefore, the model will have a better OOD generalization ability if it can identify the invariant subgraph and only uses structural information from G I .
Revised: Therefore, the model will have a better OOD generalization ability if it can identify the invariant subgraph and only uses structural information from G I .

Original: We denote a generator to obtain the invariant subgraph as G I =
Revised: We denote a generator to obtain the invariant subgraph as G I =

Original: Φ( G ) .
Revised: Φ( G ) .

Original: Following the invariant learning literature [12], we make an assumption on Φ( G ) as follows:
Revised: Following the invariant learning literature [12], we make an assumption on Φ( G ) as follows:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:
Revised: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:

Original: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))
Revised: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))

Original: =
Revised: =

Original: P e ′
Revised: P e ′

Original: ( Y | Φ ∗ ( G )) .
Revised: ( Y | Φ ∗ ( G )) .

Original: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))
Revised: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))

Original: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation learning function, w ∗ is the classifier, ⊥ indicates statistical independence, and
Revised: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation learning function, w ∗ is the classifier, ⊥ indicates statistical independence, and

Original: ϵ
Revised: ϵ

Original: is
Revised: is

Original: random
Revised: random

Original: noise.
Revised: noise.

Original: The invariance assumption means that there exists a subgraph generator such that it can generate invariant subgraphs across different environments.
Revised: The invariance assumption means that there exists a subgraph generator such that it can generate invariant subgraphs across different environments.

Original: The sufficiency assumption means that the generated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.
Revised: The sufficiency assumption means that the generated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.

Original: Under this assumption, we instantiate Φ( · ) with learnable parameters.
Revised: Under this assumption, we instantiate Φ( · ) with learnable parameters.

Original: Consider an input graph instance G with n nodes.
Revised: Consider an input graph instance G with n nodes.

Original: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where
Revised: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where

Original: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.
Revised: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.

Original: To split the input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × n on the adjacency matrix A .
Revised: To split the input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × n on the adjacency matrix A .

Original: However, directly optimizing a discrete matrix M is intractable as G has exponentially many subgraph candidates [13].
Revised: However, directly optimizing a discrete matrix M is intractable as G has exponentially many subgraph candidates [13].

Original: Besides, learning M for each graph G separately hinders the method from handling unseen test graphs [14].
Revised: Besides, learning M for each graph G separately hinders the method from handling unseen test graphs [14].

Original: Therefore, we adopt a shared learnable
Revised: Therefore, we adopt a shared learnable

Original: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:
Revised: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:

Original: [Equation]
Revised: [Equation]

Original: where Z ( m ) is the node representation.
Revised: where Z ( m ) is the node representation.

Original: Then, we obtain the invariant and variant subgraphs as:
Revised: Then, we obtain the invariant and variant subgraphs as:

Original: [Equation]
Revised: [Equation]

Original: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largest values.
Revised: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largest values.

Original: The parameters of GNN M are trained on all available graphs to generate the corresponding
Revised: The parameters of GNN M are trained on all available graphs to generate the corresponding

Original: G I and G V .
Revised: G I and G V .

Original: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs, as opposed to directly optimizing mask matrices.
Revised: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs, as opposed to directly optimizing mask matrices.

Original: 3.2 Environment Inference
Revised: 3.2 Environment Inference

Original: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .
Revised: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .

Original: The intuition is that since the invariant subgraph captures invariant relationships between predictive graph structural information and labels, the variant subgraphs in turn capture variant correlations under different distributions, which are environment-discriminative features.
Revised: The intuition is that since the invariant subgraph captures invariant relationships between predictive graph structural information and labels, the variant subgraphs in turn capture variant correlations under different distributions, which are environment-discriminative features.

Original: Therefore, we can use the variant subgraphs to infer the latent environments.
Revised: Therefore, we can use the variant subgraphs to infer the latent environments.

Original: We adopt another GNN encoder, whose parameters are also shared among different graphs, to generate the representation of the variant subgraph G V :
Revised: We adopt another GNN encoder, whose parameters are also shared among different graphs, to generate the representation of the variant subgraph G V :

Original: [Equation]
Revised: [Equation]

Original: where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .
Revised: where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .

Original: The representation of all the variant subgraphs is denoted as H =
Revised: The representation of all the variant subgraphs is denoted as H =

Original: [ h V 1 , ..., h V N ] .
Revised: [ h V 1 , ..., h V N ] .

Original: After obtaining H , we use an off-the-shelf clustering algorithm to infer the environment label E infer .
Revised: After obtaining H , we use an off-the-shelf clustering algorithm to infer the environment label E infer .

Original: In this paper, we adopt k-means [15] as our clustering algorithm:
Revised: In this paper, we adopt k-means [15] as our clustering algorithm:

Original: [Equation]
Revised: [Equation]

Original: means ( H ) .
Revised: means ( H ) .

Original: [Equation]
Revised: [Equation]

Original: Using E infer , we can partition the graph dataset into multiple training environments, i.e., G = {G e } e ∈ supp( E infer ) .
Revised: Using E infer , we can partition the graph dataset into multiple training environments, i.e., G = {G e } e ∈ supp( E infer ) .

Original: The environment inference module is purely unsupervised without needing any ground-truth environment labels.
Revised: The environment inference module is purely unsupervised without needing any ground-truth environment labels.

Original: 3.3 Invariant Learning
Revised: 3.3 Invariant Learning

Original: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariant learning module which can generate OOD generalized graph representations under distribution shifts.
Revised: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariant learning module which can generate OOD generalized graph representations under distribution shifts.

Original: Recall that both the invariant subgraph identification module and environment inference module heavily depend on the generator Φ .
Revised: Recall that both the invariant subgraph identification module and environment inference module heavily depend on the generator Φ .

Original: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.
Revised: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.

Original: First, following the invariant learning literature [9], we give the following definition.
Revised: First, following the invariant learning literature [9], we give the following definition.

Original: Definition 1.
Revised: Definition 1.

Original: The invariant subgraph generator set I with respect to E is defined as:
Revised: The invariant subgraph generator set I with respect to E is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.
Revised: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only if it is the maximal invariant subgraph generator, i.e.,
Revised: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only if it is the maximal invariant subgraph generator, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where I ( · ; · ) is the mutual information between the label and the generated subgraph.
Revised: where I ( · ; · ) is the mutual information between the label and the generated subgraph.

Original: The proof is provided in Appendix.
Revised: The proof is provided in Appendix.

Original: Eq.
Revised: Eq.

Original: (7) provides us an objective function to optimize the subgraph generator.
Revised: (7) provides us an objective function to optimize the subgraph generator.

Original: However, directly solving Eq.
Revised: However, directly solving Eq.

Original: (7) for a non-linear Φ is difficult [9].
Revised: (7) for a non-linear Φ is difficult [9].

Original: Following the invariant learning literature [9], we transform Eq.
Revised: Following the invariant learning literature [9], we transform Eq.

Original: (7) into an invariance regularizer:
Revised: (7) into an invariance regularizer:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnable parameters.
Revised: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnable parameters.

Original: Recall that g ( · ) is the representation learning function of the invariant subgraphs and w ( · ) is the classifier.
Revised: Recall that g ( · ) is the representation learning function of the invariant subgraphs and w ( · ) is the classifier.

Original: We instantiate g as another GNN as follows:
Revised: We instantiate g as another GNN as follows:

Original: [Equation]
Revised: [Equation]

Original: and h I are the node-level and graph-level representations of invariant subgraph G I , respectively.
Revised: and h I are the node-level and graph-level representations of invariant subgraph G I , respectively.

Original: w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.
Revised: w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.

Original: By optimizing Eq.
Revised: By optimizing Eq.

Original: (8), we can get our desired generator Φ and the subgraph representation learning function g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ .
Revised: (8), we can get our desired generator Φ and the subgraph representation learning function g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ .

Original: 3.4 Discussions
Revised: 3.4 Discussions

Original: Training Procedure.
Revised: Training Procedure.

Original: We present the pseudocode of GIL in Appendix.
Revised: We present the pseudocode of GIL in Appendix.

Original: Time Complexity.
Revised: Time Complexity.

Original: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotes the number of nodes and edges, respectively, and d is the dimensionality of the representations.
Revised: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotes the number of nodes and edges, respectively, and d is the dimensionality of the representations.

Original: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexity of O ( | E | d + | V | d 2 ) .
Revised: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexity of O ( | E | d + | V | d 2 ) .

Original: Since we only need to generate mask for the existing edges in graphs, the time complexity of generating invariant and variant subgraphs and further obtaining their representations is O ( | E | d + | V | d 2 ) .
Revised: Since we only need to generate mask for the existing edges in graphs, the time complexity of generating invariant and variant subgraphs and further obtaining their representations is O ( | E | d + | V | d 2 ) .

Original: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B| is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes the number of inferred environments.
Revised: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B| is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes the number of inferred environments.

Original: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) , as the number of parameters for most GNNs is O ( d 2 ) .
Revised: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) , as the number of parameters for most GNNs is O ( d 2 ) .

Original: Since |B| , |E infer | , and T are small constants, the overall time complexity of GIL is O ( | E
Revised: Since |B| , |E infer | , and T are small constants, the overall time complexity of GIL is O ( | E

Original: | d + | V | d 2 ) .
Revised: | d + | V | d 2 ) .

Original: In comparison, the time complexity of other
Revised: In comparison, the time complexity of other

Original: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .
Revised: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .

Original: Therefore, the time complexity of our proposed GIL is on par with the existing methods.
Revised: Therefore, the time complexity of our proposed GIL is on par with the existing methods.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically analyze our GIL model by showing that the maximal invariant subgraph generator can achieve OOD optimal.
Revised: In this section, we theoretically analyze our GIL model by showing that the maximal invariant subgraph generator can achieve OOD optimal.

Original: The proofs are provided in Appendix.
Revised: The proofs are provided in Appendix.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote the complement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.
Revised: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote the complement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.

Original: Then, we can obtain the optimal predictor under distribution shifts, i.e., the solution to Problem 1, as follows:
Revised: Then, we can obtain the optimal predictor under distribution shifts, i.e., the solution to Problem 1, as follows:

Original: [Equation]
Revised: [Equation]

Original: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,
Revised: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,

Original: ∃ e ′ ∈ supp
Revised: ∃ e ′ ∈ supp

Original: ( E ) such that P e ′ ( G , Y ) =
Revised: ( E ) such that P e ′ ( G , Y ) =

Original: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))
Revised: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))

Original: =
Revised: =

Original: P e (Φ( G )) .
Revised: P e (Φ( G )) .

Original: The theorem shows that we can transform the OOD generalization problem into finding the optimal invariant subgraphs while maintaining the optimality.
Revised: The theorem shows that we can transform the OOD generalization problem into finding the optimal invariant subgraphs while maintaining the optimality.

Original: We also prove that our GIL satisfies permutation invariance in Appendix.
Revised: We also prove that our GIL satisfies permutation invariance in Appendix.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the effectiveness of our GIL on both synthetic and real-world datasets.
Revised: In this section, we evaluate the effectiveness of our GIL on both synthetic and real-world datasets.

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We adopt one synthetic dataset with controllable ground-truth environments and four real-world benchmark datasets for the graph classification task.
Revised: We adopt one synthetic dataset with controllable ground-truth environments and four real-world benchmark datasets for the graph classification task.

Original: • SP-Motif :
Revised: • SP-Motif :

Original: Following [13, 16], we generate a synthetic dataset where each graph consists of one variant subgraph and one invariant subgraph,
Revised: Following [13, 16], we generate a synthetic dataset where each graph consists of one variant subgraph and one invariant subgraph,

Original: i
Revised: i

Original: e., motif.
Revised: e., motif.

Original: The variant subgraph includes Tree, Ladder, and Wheel (denoted by V = 0, 1, 2, respectively) and the invariant subgraph includes Cycle, House, and Crane (denoted by I = 0, 1, 2).
Revised: The variant subgraph includes Tree, Ladder, and Wheel (denoted by V = 0, 1, 2, respectively) and the invariant subgraph includes Cycle, House, and Crane (denoted by I = 0, 1, 2).

Original: The ground-truth label Y only depends on the invariant subgraph I , which is sampled uniformly.
Revised: The ground-truth label Y only depends on the invariant subgraph I , which is sampled uniformly.

Original: The spurious correlation between V and Y is injected by controlling the variant subgraphs distribution as: P ( V ) =
Revised: The spurious correlation between V and Y is injected by controlling the variant subgraphs distribution as: P ( V ) =

Original: r if V = I and P ( V ) =
Revised: r if V = I and P ( V ) =

Original: (1 − r ) / 2 if V ̸
Revised: (1 − r ) / 2 if V ̸

Original: = I .
Revised: = I .

Original: Intuitively, r controls the strength of the spurious correlation.
Revised: Intuitively, r controls the strength of the spurious correlation.

Original: We set r to different values in the testing and training set to simulate the distribution shifts.
Revised: We set r to different values in the testing and training set to simulate the distribution shifts.

Original: • MNIST-75sp [17]: The task is to classify each graph that is converted from an image in MNIST [18] into the corresponding handwritten digit.
Revised: • MNIST-75sp [17]: The task is to classify each graph that is converted from an image in MNIST [18] into the corresponding handwritten digit.

Original: Distribution shifts exist on node features by adding random noises in the testing set.
Revised: Distribution shifts exist on node features by adding random noises in the testing set.

Original: • Graph-SST2 [19]: Each graph is converted from a text sequence.
Revised: • Graph-SST2 [19]: Each graph is converted from a text sequence.

Original: Graphs are split into different sets based on average node degrees to create distribution shifts.
Revised: Graphs are split into different sets based on average node degrees to create distribution shifts.

Original: • Open Graph Benchmark (OGB) [
Revised: • Open Graph Benchmark (OGB) [

Original: ]: We consider two datasets, MOLSIDER and MOLHIV.
Revised: ]: We consider two datasets, MOLSIDER and MOLHIV.

Original: The default split separates structurally different molecules with different scaffolds into different subsets.
Revised: The default split separates structurally different molecules with different scaffolds into different subsets.

Original: Baselines.
Revised: Baselines.

Original: We compare our GIL with some representative state-of-the-art methods.
Revised: We compare our GIL with some representative state-of-the-art methods.

Original: The first group of these methods generates masks on graph structures to filter out important subgraphs using different
Revised: The first group of these methods generates masks on graph structures to filter out important subgraphs using different

Original: GNNs, including Attention [2], Top-k Pool [21], SAGPool [22], and ASAP [23].
Revised: GNNs, including Attention [2], Top-k Pool [21], SAGPool [22], and ASAP [23].

Original: The second group is invariant learning methods, including standard ERM, GroupDRO [24], IRM [8], V-REx [25],
Revised: The second group is invariant learning methods, including standard ERM, GroupDRO [24], IRM [8], V-REx [25],

Original: DIR [16].
Revised: DIR [16].

Original: We also consider a recent interpretable graph learning method GSAT [26].
Revised: We also consider a recent interpretable graph learning method GSAT [26].

Original: For a fair comparison, we use the same GNN backbone as GIL for the baselines.
Revised: For a fair comparison, we use the same GNN backbone as GIL for the baselines.

Original: Optimization and Hyper-parameters.
Revised: Optimization and Hyper-parameters.

Original: The adopted GNNs and READOUT functions including
Revised: The adopted GNNs and READOUT functions including

Original: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.
Revised: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.

Original: The hyperparameter λ in Eq.
Revised: The hyperparameter λ in Eq.

Original: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .
Revised: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .

Original: The number of clusters in Eq.
Revised: The number of clusters in Eq.

Original: (5) is chosen from [2 , 4] .
Revised: (5) is chosen from [2 , 4] .

Original: They are tuned on the validation set.
Revised: They are tuned on the validation set.

Original: We report the mean results and standard deviations of five runs.
Revised: We report the mean results and standard deviations of five runs.

Original: More details on the datasets, baselines and implementations are in Appendix.
Revised: More details on the datasets, baselines and implementations are in Appendix.

Original: 5.2 Experiments on SP-Motif
Revised: 5.2 Experiments on SP-Motif

Original: Settings.
Revised: Settings.

Original: To simulate different degrees of distribution shifts, we vary r in both the training and testing datasets.
Revised: To simulate different degrees of distribution shifts, we vary r in both the training and testing datasets.

Original: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: A larger r train indicates a higher spurious correlation between Y and G V in the training set, while r train = 1 /implies that the training set is balanced without any spurious correlation.
Revised: A larger r train indicates a higher spurious correlation between Y and G V in the training set, while r train = 1 /implies that the training set is balanced without any spurious correlation.

Original: For the testing set, we consider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variant subgraphs are randomly attached without spurious correlations; (2) r test = 0 .
Revised: For the testing set, we consider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variant subgraphs are randomly attached without spurious correlations; (2) r test = 0 .

Original: 2 , which indicates that the testing set has reversed spurious correlations and thus is more challenging.
Revised: 2 , which indicates that the testing set has reversed spurious correlations and thus is more challenging.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We have the following observations.
Revised: We have the following observations.

Original: Our proposed GIL model consistently and significantly outperforms the baselines and achieves the best performance on all settings.
Revised: Our proposed GIL model consistently and significantly outperforms the baselines and achieves the best performance on all settings.

Original: The results demonstrate that our proposed method can well handle graph distribution shifts and have a remarkable out-of-distribution generalization ability.
Revised: The results demonstrate that our proposed method can well handle graph distribution shifts and have a remarkable out-of-distribution generalization ability.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: EnvironmentsTreeLadderWheel
Revised: EnvironmentsTreeLadderWheel

Original: As r train grows larger, the performance of all the methods tends to decrease since there exists a larger degree of distribution shift.
Revised: As r train grows larger, the performance of all the methods tends to decrease since there exists a larger degree of distribution shift.

Original: Nevertheless, our proposed method is able to maintain the most relatively stable performance.
Revised: Nevertheless, our proposed method is able to maintain the most relatively stable performance.

Original: In fact, the performance gap between GIL and the baselines becomes more significant as the degree of distribution shift increases.
Revised: In fact, the performance gap between GIL and the baselines becomes more significant as the degree of distribution shift increases.

Original: For example, when r test = 1 / 3 , the accuracy of all baselines drops by more than 7% when r train changes from 0 .
Revised: For example, when r test = 1 / 3 , the accuracy of all baselines drops by more than 7% when r train changes from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 8 , indicating their poor OOD generalization ability.
Revised: 8 , indicating their poor OOD generalization ability.

Original: In contrast, our method only has 3% performance drop.
Revised: In contrast, our method only has 3% performance drop.

Original: When the degree of distribution shift is relatively small, GNNs with different pooling methods to generate subgraphs generally report better results.
Revised: When the degree of distribution shift is relatively small, GNNs with different pooling methods to generate subgraphs generally report better results.

Original: On the other hand, when the degree of distribution shift is large, invariant baselines show more stable performance.
Revised: On the other hand, when the degree of distribution shift is large, invariant baselines show more stable performance.

Original: Among them, DIR, which is a recently proposed invariant method specifically designed for graphs, is one competitive baseline.
Revised: Among them, DIR, which is a recently proposed invariant method specifically designed for graphs, is one competitive baseline.

Original: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classification accuracy in most cases.
Revised: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classification accuracy in most cases.

Original: GSAT achieves promising gains over the other baselines, but our GIL still performs better than GSAT.
Revised: GSAT achieves promising gains over the other baselines, but our GIL still performs better than GSAT.

Original: When r test =
Revised: When r test =

Original: r train = 1 / 3 , i.e., no distribution shifts, our proposed method also achieves the best results, indicating that learning invariant subgraphs is also beneficial.
Revised: r train = 1 / 3 , i.e., no distribution shifts, our proposed method also achieves the best results, indicating that learning invariant subgraphs is also beneficial.

Original: Analysis.
Revised: Analysis.

Original: To analyze whether our proposed method can accurately capture the invariant subgraph, we compare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.
Revised: To analyze whether our proposed method can accurately capture the invariant subgraph, we compare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.

Original: The evaluation metric is Precision@5.
Revised: The evaluation metric is Precision@5.

Original: We report the results in Figure 3.
Revised: We report the results in Figure 3.

Original: The results show that GIL has a clear advantage in discovering invariant subgraphs under latent environments, while the other baselines cannot handle distribution shifts well.
Revised: The results show that GIL has a clear advantage in discovering invariant subgraphs under latent environments, while the other baselines cannot handle distribution shifts well.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless the world 's best actors auteuil ,
Revised: delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless the world 's best actors auteuil ,

Original: [Table]
Revised: [Table]

Original: a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,
Revised: a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,

Original: po s i t i v e nega t
Revised: po s i t i v e nega t

Original: i v e
Revised: i v e

Original: Train set
Revised: Train set

Original: Test set ! ! ! "
Revised: Test set ! ! ! "

Original: Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .and
Revised: Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .and

Original: r test = 0 . 2 ) in Figure 6.
Revised: r test = 0 . 2 ) in Figure 6.

Original: The figure shows that the learned invariant subgraph of our method is more accurate than baselines.
Revised: The figure shows that the learned invariant subgraph of our method is more accurate than baselines.

Original: 5.3 Experiments on Real-world Graphs
Revised: 5.3 Experiments on Real-world Graphs

Original: We further evaluate the effectiveness of our method on real-world graph datasets.
Revised: We further evaluate the effectiveness of our method on real-world graph datasets.

Original: The experimental results are presented in Table 2.
Revised: The experimental results are presented in Table 2.

Original: Our GIL achieves the best performance on all four datasets, indicating that GIL can well handle distribution shifts on real-world graphs.
Revised: Our GIL achieves the best performance on all four datasets, indicating that GIL can well handle distribution shifts on real-world graphs.

Original: For example, GIL increases the classification accuracy by 1.8% on MNIST-75sp and ROC-AUC by 2.0% on MOLHIV against the strongest baselines respectively.
Revised: For example, GIL increases the classification accuracy by 1.8% on MNIST-75sp and ROC-AUC by 2.0% on MOLHIV against the strongest baselines respectively.

Original: On MOLHIV, the results of most baselines are worse than ERM, indicating that they fail to achieve OOD generalization in this dataset.
Revised: On MOLHIV, the results of most baselines are worse than ERM, indicating that they fail to achieve OOD generalization in this dataset.

Original: Besides, different datasets have different distribution shifts, e.g., Graph-SST2 has different node degrees, the distribution shift of MNIST-75sp is on node features, and OGB is split based on scaffold.
Revised: Besides, different datasets have different distribution shifts, e.g., Graph-SST2 has different node degrees, the distribution shift of MNIST-75sp is on node features, and OGB is split based on scaffold.

Original: Therefore, the results show that our proposed method can well handle diverse types of distribution shifts in real graph datasets.
Revised: Therefore, the results show that our proposed method can well handle diverse types of distribution shifts in real graph datasets.

Original: [Table]
Revised: [Table]

Original: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is also compatible with the other popular GNNs.
Revised: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is also compatible with the other popular GNNs.

Original: We try using HIG and CIN [27] (Rank #2 and #8 on the MOLHIV leaderboard 1 ) as the backbone since these models are orthogonal to ours.
Revised: We try using HIG and CIN [27] (Rank #2 and #8 on the MOLHIV leaderboard 2 ) as the backbone since these models are orthogonal to ours.

Original: Tableshows that our GIL can consistently improve these models.
Revised: Tableshows that our GIL can consistently improve these models.

Original: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL on both the train and test set of Graph-SST2.
Revised: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL on both the train and test set of Graph-SST2.

Original: This dataset consists of sentences with positive/negative sentiments and is more understandable for humans.
Revised: This dataset consists of sentences with positive/negative sentiments and is more understandable for humans.

Original: Figure 7 shows that our method can learn invariant subgraphs by consistently focusing on the positive/negative words that are salient for sentiments and distinguishing invariant/variant parts under distribution shifts.
Revised: Figure 7 shows that our method can learn invariant subgraphs by consistently focusing on the positive/negative words that are salient for sentiments and distinguishing invariant/variant parts under distribution shifts.

Original: For example, the subgraph “The world’s best actors” identified by GIL has a predictive and invariant relationship with the positive sentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in different sentences.
Revised: For example, the subgraph “The world’s best actors” identified by GIL has a predictive and invariant relationship with the positive sentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in different sentences.

Original: These results validate: (1) the invariant and variant subgraphs widely exist in real-world datasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and further make predictions with high accuracy based on the learned invariant subgraphs.
Revised: These results validate: (1) the invariant and variant subgraphs widely exist in real-world datasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and further make predictions with high accuracy based on the learned invariant subgraphs.

Original: 5.4 Analysis of Environment Inference
Revised: 5.4 Analysis of Environment Inference

Original: In our proposed model, all components are jointly optimized.
Revised: In our proposed model, all components are jointly optimized.

Original: To show that the environment inference module and invariant learning module can mutually enhance each other, we record the test accuracy and the Silhouette score [28], which is a commonly used evaluation metric for clustering, as the model is trained.
Revised: To show that the environment inference module and invariant learning module can mutually enhance each other, we record the test accuracy and the Silhouette score [28], which is a commonly used evaluation metric for clustering, as the model is trained.

Original: The results on SP-Motif ( r train = 0 .
Revised: The results on SP-Motif ( r train = 0 .

Original: 8 , r test = 1 / 3 ) are shown in Figure 4.
Revised: 8 , r test = 1 / 3 ) are shown in Figure 4.

Original: We can observe that the test accuracy and the clustering performance improve synchronously over training.
Revised: We can observe that the test accuracy and the clustering performance improve synchronously over training.

Original: A plausible reason is that, as the training stage progresses, the invariant subgraph generator is optimized so that it can generate more informative invariant subgraphs and therefore improve the performance on the testing set.
Revised: A plausible reason is that, as the training stage progresses, the invariant subgraph generator is optimized so that it can generate more informative invariant subgraphs and therefore improve the performance on the testing set.

Original: On the other hand, accurately discovering invariant subgraphs can also promote identifying variant subgraphs, which capture the environment-discriminate features and better infer the latent environments.
Revised: On the other hand, accurately discovering invariant subgraphs can also promote identifying variant subgraphs, which capture the environment-discriminate features and better infer the latent environments.

Original: To verify that GIL can infer the environments accurately, we use t-SNE [29] to plot the discovered environments on a 2D-plane when the optimization is finished.
Revised: To verify that GIL can infer the environments accurately, we use t-SNE [29] to plot the discovered environments on a 2D-plane when the optimization is finished.

Original: Figure 5 shows that the variant subgraphs perfectly capture the environment-discriminate features.
Revised: Figure 5 shows that the variant subgraphs perfectly capture the environment-discriminate features.

Original: Notice that GIL achieves such results without needing any ground-truth environment label.
Revised: Notice that GIL achieves such results without needing any ground-truth environment label.

Original: 5.5 Hyper-parameter Sensitivity
Revised: 5.5 Hyper-parameter Sensitivity

Original: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask t in Eq.
Revised: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask t in Eq.

Original: For simplicity, we only report the results on SP-Motif ( r train = 0 .
Revised: For simplicity, we only report the results on SP-Motif ( r train = 0 .

Original: 8 and r test = 1 / 3 ) and
Revised: 8 and r test = 1 / 3 ) and

Original: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.
Revised: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.

Original: First, the number of environments has a moderate impact on the model performance.
Revised: First, the number of environments has a moderate impact on the model performance.

Original: For SP-Motif, the performance reaches a peak when |E infer | = 3 , showing that GIL achieves the best result when the number of environments matches the ground truth.
Revised: For SP-Motif, the performance reaches a peak when |E infer | = 3 , showing that GIL achieves the best result when the number of environments matches the ground truth.

Original: [Figure]
Revised: [Figure]

Original: The number of
Revised: The number of

Original: [Figure]
Revised: [Figure]

Original: |E infer |
Revised: |E infer |

Original: [Figure]
Revised: [Figure]

Original: For MNIST-75sp, the best of environments is |E infer | = 2 .
Revised: For MNIST-75sp, the best of environments is |E infer | = 2 .

Original: A plausible reason is that a large number of environments will bring difficulty in inferring the latent environment, leading to sub-optimal performance.
Revised: A plausible reason is that a large number of environments will bring difficulty in inferring the latent environment, leading to sub-optimal performance.

Original: Second, the coefficient λ also has a slight influence on the performance, indicating that we need to properly balance the classification loss and the invariance regularizer term.
Revised: Second, the coefficient λ also has a slight influence on the performance, indicating that we need to properly balance the classification loss and the invariance regularizer term.

Original: Finally, a proper value of the mask size t is important.
Revised: Finally, a proper value of the mask size t is important.

Original: A very large t will result in too many edges in the invariant subgraph and bring in variant structures, while a small t may let the invariant subgraph become too small to capture enough structural information.
Revised: A very large t will result in too many edges in the invariant subgraph and bring in variant structures, while a small t may let the invariant subgraph become too small to capture enough structural information.

Original: Although an appropriate choice of hyper-parameters can further improve the performance, our method is not very sensitive to hyperparameters.
Revised: Although an appropriate choice of hyper-parameters can further improve the performance, our method is not very sensitive to hyperparameters.

Original: Figure 8 shows that GIL can outperform the best baselines with a wide range of hyper-parameters choices.
Revised: Figure 8 shows that GIL can outperform the best baselines with a wide range of hyper-parameters choices.

Original:  Related Works
Revised:  Related Works

Original: Graph neural networks.
Revised: Graph neural networks.

Original: Recently, graph neural networks (GNNs) have shown enormous success in graph representation learning [1–3], demonstrating their strength in various tasks [30–36].
Revised: Recently, graph neural networks (GNNs) have shown enormous success in graph representation learning [1–3], demonstrating their strength in various tasks [30–36].

Original: GNNs generally adopt a neighborhood aggregation (message passing) paradigm, i.e., the representations of nodes are iteratively updated by aggregating representations of their neighbors.
Revised: GNNs generally adopt a neighborhood aggregation (message passing) paradigm, i.e., the representations of nodes are iteratively updated by aggregating representations of their neighbors.

Original: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 22].
Revised: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 22].

Original: However, most existing GNN models do not consider the out-of-distribution generalization ability [37] so that their performances can drop substantially on testing graphs with distribution shifts.
Revised: However, most existing GNN models do not consider the out-of-distribution generalization ability [37] so that their performances can drop substantially on testing graphs with distribution shifts.

Original: Generalization of GNNs.
Revised: Generalization of GNNs.

Original: Early works [38–41] for analyzing the generalization ability of GNNs do not consider distribution shifts [37, 42, 43].
Revised: Early works [38–41] for analyzing the generalization ability of GNNs do not consider distribution shifts [37, 42, 43].

Original: More recently, the generalization ability of GNNs under distribution shifts starts to receive research attention [44, 16, 45, 46].
Revised: More recently, the generalization ability of GNNs under distribution shifts starts to receive research attention [44, 16, 45, 46].

Original: [47] find that encoding task-specific non-linearities in the architecture or features can improve GNNs in extrapolating graph algorithmic tasks.
Revised: [47] find that encoding task-specific non-linearities in the architecture or features can improve GNNs in extrapolating graph algorithmic tasks.

Original: [17, 48, 49] try to encourage GNNs to perform well on testing graphs with different sizes.
Revised: [17, 48, 49] try to encourage GNNs to perform well on testing graphs with different sizes.

Original: Some works [50, 51] are proposed to deal with node-level tasks.
Revised: Some works [50, 51] are proposed to deal with node-level tasks.

Original: EERM [52] studies the OOD generalization in node classification.
Revised: EERM [52] studies the OOD generalization in node classification.

Original: However, little attention has been paid to learning graph-level representations under distribution shifts from the invariant learning perspective.
Revised: However, little attention has been paid to learning graph-level representations under distribution shifts from the invariant learning perspective.

Original: One exception is the work DIR [16], which conducts interventions on graphs to create interventional distributions.
Revised: One exception is the work DIR [16], which conducts interventions on graphs to create interventional distributions.

Original: However, performing causal intervention relies on strong assumptions [53] that could be violated and expensive to satisfy in practice [54].
Revised: However, performing causal intervention relies on strong assumptions [53] that could be violated and expensive to satisfy in practice [54].

Original: GSAT [26] applies graph information bottleneck criteria for generalization, but its goal is mainly to build inherently interpretable GNNs.
Revised: GSAT [26] applies graph information bottleneck criteria for generalization, but its goal is mainly to build inherently interpretable GNNs.

Original: Invariant Learning.
Revised: Invariant Learning.

Original: Invariant learning aims to exploit the invariant relationships between features and labels across distribution shifts, while filtering out the variant spurious correlations.
Revised: Invariant learning aims to exploit the invariant relationships between features and labels across distribution shifts, while filtering out the variant spurious correlations.

Original: Backed by causal theory, invariant learning model can lead to OOD optimal models under some assumptions [11,
Revised: Backed by causal theory, invariant learning model can lead to OOD optimal models under some assumptions [11,

Original: However, most existing methods heavily rely on multiple environments that have to be explicitly provided in the training dataset.
Revised: However, most existing methods heavily rely on multiple environments that have to be explicitly provided in the training dataset.

Original: Such annotation is not only prohibitively expensive for graphs, but also inherently problematic as the environment split could be inaccurate, rendering these invariant learning methods inapplicable.
Revised: Such annotation is not only prohibitively expensive for graphs, but also inherently problematic as the environment split could be inaccurate, rendering these invariant learning methods inapplicable.

Original: A few works study OOD generalization on latent environments in computer vision [55, 56] or raw feature data [57], which cannot be directly applied to graphs.
Revised: A few works study OOD generalization on latent environments in computer vision [55, 56] or raw feature data [57], which cannot be directly applied to graphs.

Original: In summary, how to learn invariant graph representations without explicit environment label under distribution shits remains largely unexplored in the literature.
Revised: In summary, how to learn invariant graph representations without explicit environment label under distribution shits remains largely unexplored in the literature.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper we propose the graph invariant learning ( GIL ) model to tackle the problem of learning invariant graph representations under distribution shifts.
Revised: In this paper we propose the graph invariant learning ( GIL ) model to tackle the problem of learning invariant graph representations under distribution shifts.

Original: Three tailored modules are jointly optimized to encourage the graph representations to capture the invariant relationships between predictive graph structural information and labels.
Revised: Three tailored modules are jointly optimized to encourage the graph representations to capture the invariant relationships between predictive graph structural information and labels.

Original: Theoretical analysis and extensive experiments on both synthetic and real-world datasets demonstrate the superiority of GIL .
Revised: Theoretical analysis and extensive experiments on both synthetic and real-world datasets demonstrate the superiority of GIL .

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported in part by the National Key Research and Development Program of China
Revised: This work was supported in part by the National Key Research and Development Program of China

Original: No. 2020AAA0106300 and National Natural Science Foundation of China (No. 62250008, 62222209, 62102222, 62206149), China National Postdoctoral Program for Innovative Talents
Revised: No. 2020AAA0106300 and National Natural Science Foundation of China (No. 62250008, 62222209, 62102222, 62206149), China National Postdoctoral Program for Innovative Talents

Original: No. BX20220185,
Revised: No. BX20220185,

Original: China Postdoctoral Science Foundation No. 

Revised: China Postdoctoral Science Foundation No.

Original: M
Revised: M

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]