Original: Learning Invariant Graph Representations Under Distribution Shifts Anonymous Author(s)
Revised: Learning Invariant Graph Representations for Out-of-Distribution Generalization

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Graph representation learning has shown effectiveness when testing and traininggraph data come from the same distribution, but most existing approaches fail togeneralize under distribution shifts.
Revised: Graph representation learning has shown effectiveness when testing and training graph data come from the same distribution, but most existing approaches fail to generalize under distribution shifts.

Original: Invariant learning, backed by the invarianceprinciple from causality, can achieve guaranteed generalization under distributionshifts in theory and has shown great successes in practice.
Revised: Invariant learning, backed by the invariance principle from causality, can achieve guaranteed generalization under distribution shifts in theory and has shown great successes in practice.

Original: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.
Revised: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.

Original: Tosolve this problem, we propose Graph Invariant Learning ( GIL ) model capable oflearning generalized graph representations under distribution shifts.
Revised: To solve this problem, we propose Graph Invariant Learning ( GIL ) model capable of learning generalized graph representations under distribution shifts.

Original: Our proposedmethod can capture the invariant relationships between predictive graph structuralinformation and labels in a mixture of latent environments through jointly optimizing three tailored modules.
Revised: Our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three tailored modules.

Original: Specifically, we first design a GNN-based subgraphgenerator to identify invariant subgraphs.
Revised: Specifically, we first design a GNN-based subgraph generator to identify invariant subgraphs.

Original: Then we use the variant subgraphs, i.e.,complements of invariant subgraphs, to infer the latent environment labels.
Revised: Then we use the variant subgraphs, i.e., complements of invariant subgraphs, to infer the latent environment labels.

Original: Wefurther propose an invariant learning module to learn graph representations thatcan generalize to unknown test graphs.
Revised: We further propose an invariant learning module to learn graph representations that can generalize to unknown test graphs.

Original: Theoretical justifications for our proposedmethod are also provided.
Revised: Theoretical justifications for our proposed method are also provided.

Original: Extensive experiments on both synthetic and real-worlddatasets demonstrate the superiority of our method against state-of-the-art baselinesunder distribution shifts for the graph classification task.
Revised: Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our method against state-of-the-art baselines under distribution shifts for the graph classification task.

Original:  Introduction
Revised:  Introduction

Original: Graph structured data is ubiquitous in the real world, e.g., social networks, biology networks, chemicalmolecules, etc.
Revised: Graph structured data is ubiquitous in the real world, e g., social networks, biology networks, chemical molecules, etc.

Original: Graph representation learning, which encodes graphs into vectorized representations,has been the central topic in graph machine learning in the last decade.
Revised: Graph representation learning, which encodes graphs into vectorized representations, has been the central topic in graph machine learning in the last decade.

Original: For example, graph neuralnetworks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information andare shown to be successful in a variety of applications.
Revised: For example, graph neural networks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information and are shown to be successful in a variety of applications.

Original: Despite the enormous success, the existing approaches for learning graph representations heavily relyon the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from anidentical distribution.
Revised: Despite the enormous success the existing approaches for learning graph representations heavily rely on the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from an identical distribution.

Original: However, distribution shifts of graph data widely exist in real-world scenariosand are usually inevitable due to the uncontrollable underlying data generation mechanism [4].
Revised: However, distribution shifts of graph data widely exist in real-world scenarios and are usually inevitable due to the uncontrollable underlying data generation mechanism [4].

Original: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.
Revised: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.

Original: Onecritical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely oncorrelations that are variant for graphs from different environments.
Revised: One critical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely on correlations that are variant for graphs from different environments.

Original: Therefore, it is of paramountsignificance to learn graph representations under distribution shifts and develop methods capable ofout-of-distribution (OOD) generalization.
Revised: Therefore, it is of paramount significance to learn graph representations under distribution shifts and develop methods capable of out-of-distribution (OOD) generalization.

Original: Such studies are especially critical for high-stake graphapplications such as medical diagnosis [5],criminal justice [6], financial analysis  [7],molecularprediction [8], etc.[Figure]
Revised: Such studies are particularly critical for high-stake graph applications such as medical diagnosis [5], financial analysis [6], molecular prediction [7], etc.

Original: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.
Revised: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.

Original: Invariant learning, whichaims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spuriouscorrelations, can provably achieve satisfactory OOD generalization under distribution shifts [9–11].
Revised: Invariant learning, which aims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spurious correlations, can provably achieve satisfactory OOD generalization under distribution shifts [8–10].

Original: Though invariant learning has been studied for images and texts [9, 12], it remains largelyunexplored in the literature of graph representation learning.
Revised: Though invariant learning has been studied for images and texts [8, 11], it remains largely unexplored in the literature of graph representation learning.

Original: However, invariant graph representationlearning is non-trivial due to the following challenges.
Revised: However, invariant graph representation learning is non-trivial due to the following challenges.

Original: First, graph data usually comes from a mixtureof latent  environments without accurate environment labels, as shown in Figure 1.
Revised: First, graph data usually comes from a mixture of latent [Figure] environments without accurate environment labels, as shown in Figure 1.

Original: Since mostinvariant methods require multiple training environments with explicit environment labels, theseexisting methods cannot be directly applied to graphs.
Revised: Since most invariant methods require multiple training environments with explicit environment labels, these existing methods cannot be directly applied to graphs.

Original: Second, the formation process of graphsis affected by the complex interaction of both invariant and variant patterns.
Revised: Second, the formation process of graphs is affected by the complex interaction of both invariant and variant patterns.

Original: How to identify theinvariant patterns among latent environments is even more challenging.
Revised: How to identify the invariant patterns among latent environments is even more challenging.

Original: Last but not least, even afterhaving obtained the environmental labels, how to design a theoretically grounded learning scheme togenerate graph representations capable of OOD generalization remains largely unexplored.
Revised: Last but not least, even after having obtained the environmental labels, how to design a theoretically grounded learning scheme to generate graph representations capable of OOD generalization remains largely unexplored.

Original: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL ) whichis able to capture invariant graph patterns in a mixture of latent environments and capable of OODgeneralization under distribution shifts.
Revised: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL ) which is able to capture invariant graph patterns in a mixture of latent environments and capable of OOD generalization under distribution shifts.

Original: As shown in Figure 2, our proposed method can capturethe invariant relationships between predictive graph structural information and labels in a mixtureof latent environments through jointly optimizing three mutually promoting modules, with eachmodule tackling one aforementioned challenge.
Revised: As shown in Figure 2, our proposed method can capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments through jointly optimizing three mutually promoting modules, with each module tackling one aforementioned challenge.

Original: Specifically, in the invariant subgraph identificationmodule, we design a GNN-based subgraph generator to identify potentially invariant subgraphsfrom the complex interaction between invariant and variant patterns.
Revised: Specifically, in the invariant subgraph identification module, we design a GNN-based subgraph generator to identify potentially invariant subgraphs from the complex interaction between invariant and variant patterns.

Original: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering theseenvironment-discriminative features.
Revised: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering these environment-discriminative features.

Original: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.
Revised: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.

Original: Lastly, in theinvariant learning module, we propose to optimize the maximal invariant subgraph generator criteriongiven the identified invariant subgraphs and inferred environments to generate graph representationscapable of OOD generalization under distribution shifts.
Revised: Lastly, in the invariant learning module, we propose to optimize the maximal invariant subgraph generator criterion given the identified invariant subgraphs and inferred environments to generate graph representations capable of OOD generalization under distribution shifts.

Original: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generatorof our GIL , and further prove that our GIL satisfies permutation invariance.
Revised: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generator of our GIL , and further prove that our GIL satisfies permutation invariance.

Original: We conduct extensiveexperiments on both synthetic graph datasets and real graph benchmarks for the graph classificationtask.
Revised: We conduct extensive experiments on both synthetic graph datasets and real graph benchmarks for the graph classification task.

Original: The results show that the representations learned from GIL achieve substantial performancegains on the unseen OOD testing graphs compared with various state-of-the-art baselines.
Revised: The results show that the representations learned from GIL achieve substantial performance gains on the unseen OOD testing graphs compared with various state-of-the-art baselines.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows.

Original: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized graph representations under distribution shifts.
Revised: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized graph representations under distribution shifts.

Original: To the best of our knowledge, we are the first to study invariant learning for graph representation learning under a mixture of latent 
Revised: To the best of our knowledge, we are the first to study invariant learning for graph representation learning under a mixture of latent environments.

Original: environments.• Our proposed method can automatically infer the environment label of graphs from a mixture oflatent environments without supervision.•
Revised: • Our proposed method can automatically infer the environment label of graphs from a mixture of latent environments without supervision.

Original:  We propose maximal invariant subgraph generator criterion to learn graph representations capable of OOD-generalization under distribution 
Revised: • We propose maximal invariant subgraph generator criterion to learn graph representations capable of OOD generalization under distribution shifts.

Original: shifts.• We theoretically show that finding a maximal invariant subgraph generator of GIL can solve theOOD generalization problem.
Revised: • We theoretically show that finding a maximal invariant subgraph generator of GIL can solve the OOD generalization problem.

Original: Extensive empirical results demonstrate the effectiveness of GIL onvarious synthetic and benchmark datasets under distribution shifts.
Revised: Extensive empirical results demonstrate the effectiveness of GIL on various synthetic and benchmark datasets under distribution shifts.

Original:  Notations and Problem Formulation
Revised:  Notations and Problem Formulation

Original: Notations.
Revised: Notations.

Original: Let G and Y be the graph and label space.
Revised: Let G and Y be the graph and label space.

Original: We consider a graph dataset G = {
Revised: We consider a graph dataset G = {

Original: ( G i , Y i ) } Ni =
Revised: ( G i , Y i ) } Ni =

Original: 1where G i ∈ G and Y i ∈ Y .
Revised: where G i ∈ G and Y i ∈ Y .

Original: Following the OOD convention [12, 9, 10], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } denote variables graph label G
Revised: Following the OOD convention [11, 8, 9], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } N e i = denotes the dataset from environment e , supp( E tr ) is the support of the environmental variable in the training data.

Original:  Y to 
Revised: We use G and Y to denote the random variables of graph and label, and G e and Y e to specify random variables from environment e . The environment label for graphs is unobserved since it is prohibitively expensive to collect graph environment labels for most real scenarios.

Original: [Figure]
Revised: [Figure]

Original: Prediction [Figure]
Revised: Prediction Φ(⋅)

Original: Clustervariantsubgraphs {" - } toinferenvironments …
Revised: Clustervariantsubgraphs {" - } toinferenvironments …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B % =
Revised: B % =

Original: Tree …
Revised: Tree …

Original: B & = Wheel
Revised: B & = Wheel

Original: Environment Inference 0 /.

Revised: Environment Inference 0 /.

Original: = 1 /2 3 ⋅

Revised: = 1 /2 3 ⋅

Original: GNN
Revised: GNN

Original: Invariant Subgraph Identification
Revised: Invariant Subgraph Identification

Original: Noderepresentation G (*) … … Graphdataset
Revised: Noderepresentation G (*) … … Graphdataset

Original: EdgeMask R ..0 0 .… 
Revised: EdgeMask R ..0 0 .… 0 .

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Invariant&variantsubgraphs {( 1 },{( ( } 2 4 =

Original: [Figure] [Figure] [Figure]{+Top 5 (0⊙2) 2 6 = 2 - 
+}
Revised: {+Top 5 (0⊙2) 2 6 = 2 - 
+}

Original: Invariant
Revised: Graphsfromamixtureoflatent environments

Original: [Figure]
Revised: [Figure]

Revised: Training Dataset

Revised: Invariant subgraphs {" 7 }

Original: Variant subgraphs {" - }
Revised: Variant subgraphs {" - }

Original: I n f e rr ed en v i r on m en t s ℰ VWXYZ
Revised: I n f e rr ed en v i r on m en t s ℰ VWXYZ

Original: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =
Revised: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =

Original: Wheel
Revised: Wheel

Original: Labels: 9= [Figure]
Revised: Labels: 9 = 9 = …

Original: [Figure] (⋅)
Revised: Testing Dataset % ∘ '(⋅)

Original: Invarianceregularizer
Revised: Invarianceregularizer

Original: Shared H∘J(⋅)
Revised: Shared H∘J(⋅)

Original: H∘J(⋅)H∘J(⋅) Shared [Figure]
Revised: H∘J(⋅)

Original:  ∇ , ℛ - " ∇ , ℛ - # \ [Figure] (Var(∇ , ℛ - ))
Revised: H∘J(⋅) Shared ∇ , ℛ - ! ∇ , ℛ - " ∇ , ℛ - # \trace(Var(∇ , ℛ - ))

Original: Invariant Learning …
Revised: Invariant Learning …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B & = Wheel …
Revised: B & = Wheel …

Original: B % =
Revised: B % =

Original: Tree ℛ - ! [Figure] [Figure] [Figure]
Revised: Tree ℛ - ! ℛ - " ℛ - # Classificationloss [ - (ℛ - ) + =

Original: [Figure]

Original: (ℛ - )

Original: [Figure]

Original: Objective function () 
Revised: Objective function () =

Original: [Figure]
Revised: + , =

Original: [Figure] ∘Φ(,)
Revised: % ∘ ' ∘ Φ(,)

Original: [Figure]
Revised: [Figure]

Original: (⋅)

Original: [Figure]
Revised: [Figure]

Original: specify random variables from environment e .

Original: The environment label for graphs is unobserved sinceit is prohibitively expensive to collect graph environment labels for most real scenarios.

Original: Problem Formulation.
Revised: Problem Formulation.

Original: We formulate the generalization under distribution shifts on graphs as:
Revised: We formulate the generalization under distribution shifts on graphs as:

Original: Problem 1.
Revised: Problem 1.

Original: Let E denote the random variable on indices of all possible environments.
Revised: Let E denote the random variable on indices of all possible environments.

Original: Our goal is tofind an optimal predictor f ∗ ( · ) :
Revised: Our goal is to find an optimal predictor f ∗ ( · ) :

Original: G → Y that performs well on all environments:
Revised: G → Y that performs well on all environments:

Original: [Equation]
Revised: [Equation]

Original: where R ( f | e )
Revised: where R ( f | e )

Original: =
Revised: =

Original: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :
Revised: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :

Original: Y × Y → R denotes a loss function.
Revised: Y × Y → R denotes a loss function.

Original: We further decompose f ( · ) =
Revised: We further decompose f ( · ) =

Original: w ◦ h , where h ( · ) : G → R d isthe representation learning function , d is the dimensionality, and w ( · ) : R d → Y is the classifier.
Revised: w ◦ h , where h ( · ) : G → R d is the representation learning function, d is the dimensionality, and w ( · ) : R d → Y is the classifier.

Original: Note that supp( E tr ) ⊂ supp( E ) .
Revised: Note that supp( E tr ) ⊂ supp( E ) .

Original: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =
Revised: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =

Original: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,
Revised: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,

Original: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and thecorresponding label is different for training and testing graph data.
Revised: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and the corresponding label is different for training and testing graph data.

Original: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [9, 10].
Revised: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [8, 9].

Original: In addition, formost graph datasets, we do not have access to accurate environment labels or environment partitions.
Revised: In addition, for most graph datasets, we do not have access to accurate environment labels or environment partitions.

Original: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good
Revised: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good

Original: OOD generalization performance under the inferred environments.
Revised: OOD generalization performance under the inferred environments.

Original: The problem is formulated as:
Revised: The problem is formulated as:

Original: Problem 2.
Revised: Problem 2.

Original: Given a graph dataset G collected from a mixture of latent environments but withoutenvironment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) ,and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achievegood OOD generalization performance.
Revised: Given a graph dataset G collected from a mixture of latent environments but without environment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) , and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achieve good OOD generalization performance.

Original:  Method
Revised:  Method

Original: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.
Revised: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.

Original: We first present the invariant subgraph identification module.
Revised: We first present the invariant subgraph identification module.

Original: Then, we infer environment labelsby clustering the variant subgraphs.
Revised: Then, we infer environment labels by clustering the variant subgraphs.

Original: Next, we introduce the maximal invariant subgraph generatorcriterion to generate graph representations which can generalize to test graphs under distributionshifts.
Revised: Next, we introduce the maximal invariant subgraph generator criterion to generate graph representations which can generalize to test graphs under distribution shifts.

Original: Lastly, we provide some discussions of our proposed method.
Revised: Lastly, we provide some discussions of our proposed method.

Original: 3.1 Invariant Subgraph Identification
Revised: 3.1 Invariant Subgraph Identification

Original: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationshipwith the label is invariant across different environments.
Revised: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationship with the label is invariant across different environments.

Original: We refer to the rest of the graph, i.e., thecomplement of G I , as the variant subgraph and denote it as G V .
Revised: We refer to the rest of the graph, i.e., the complement of G I , as the variant subgraph and denote it as G V .

Original: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.
Revised: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.

Original: Therefore, the model will have a better OOD generalization ability if it can identify the invariantsubgraph and only uses structural information from G I .
Revised: Therefore, the model will have a better OOD generalization ability if it can identify the invariant subgraph and only uses structural information from G I .

Original: We denote a generator to obtain the invariant subgraph as G I =
Revised: We denote a generator to obtain the invariant subgraph as G I =

Original: Φ( G ) .
Revised: Φ( G ) .

Original: Following the invariantlearning literature [13], we make an assumption on Φ( G ) as follows:
Revised: Following the invariant learning literature [12], we make an assumption on Φ( G ) as follows:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:
Revised: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:

Original: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))
Revised: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))

Original: =
Revised: =

Original: P e ′
Revised: P e ′

Original: ( Y | Φ ∗ ( G )) .
Revised: ( Y | Φ ∗ ( G )) .

Original: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))
Revised: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))

Original: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation learning function , w ∗ is the classifier
Revised: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation learning function, w ∗ is the classifier, ⊥ indicates statistical independence, and

Original: , ⊥ indicates

Original: statistical independence,

Original: and ϵ is
Revised: ϵ is

Original: random noise.
Revised: random noise.

Original: The invariance assumption means that there exists a subgraph generator such that it can generateinvariant subgraphs across different environments.
Revised: The invariance assumption means that there exists a subgraph generator such that it can generate invariant subgraphs across different environments.

Original: The sufficiency assumption means that thegenerated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.
Revised: The sufficiency assumption means that the generated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.

Original: Under this assumption, we instantiate Φ( · ) with learnable parameters.
Revised: Under this assumption, we instantiate Φ( · ) with learnable parameters.

Original: Consider an input graphinstance G with n nodes.
Revised: Consider an input graph instance G with n nodes.

Original: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where
Revised: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where

Original: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.
Revised: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.

Original: To splitthe input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × non the adjacency matrix A .
Revised: To split the input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × n on the adjacency matrix A .

Original: However, directly optimizing a discrete matrix M is intractable as Ghas exponentially many subgraph candidates [14].
Revised: However, directly optimizing a discrete matrix M is intractable as G has exponentially many subgraph candidates [13].

Original: Besides, learning M for each graph G separatelyhinders the method from handling unseen test graphs [15].
Revised: Besides, learning M for each graph G separately hinders the method from handling unseen test graphs [14].

Original: Therefore, we adopt a shared learnable
Revised: Therefore, we adopt a shared learnable

Original: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:
Revised: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:

Original: [Equation]
Revised: [Equation]

Original: where Z ( m ) is the node representation.
Revised: where Z ( m ) is the node representation.

Original: Then, we obtain the invariant and variant subgraphs as:
Revised: Then, we obtain the invariant and variant subgraphs as:

Original: [Equation]
Revised: [Equation]

Original: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largestvalues.
Revised: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largest values.

Original: The parameters of GNN M are trained on all available graphs to generate the corresponding
Revised: The parameters of GNN M are trained on all available graphs to generate the corresponding

Original: G I and G V .
Revised: G I and G V .

Original: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs,as opposed to directly optimizing mask matrices.
Revised: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs, as opposed to directly optimizing mask matrices.

Original: 3.2 Environment Inference
Revised: 3.2 Environment Inference

Original: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .
Revised: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .

Original: Theintuition is that since the invariant subgraph captures invariant relationships between predictive graphstructural information and labels, the variant subgraphs in turn capture variant correlations underdifferent distributions, which are environment-discriminative features.
Revised: The intuition is that since the invariant subgraph captures invariant relationships between predictive graph structural information and labels, the variant subgraphs in turn capture variant correlations under different distributions, which are environment-discriminative features.

Original: Therefore, we can use thevariant subgraphs to infer the latent environments.
Revised: Therefore, we can use the variant subgraphs to infer the latent environments.

Original: We adopt another GNN encoder, whose parametersare also shared among different graphs, to generate the representation of the variant subgraph G V :
Revised: We adopt another GNN encoder, whose parameters are also shared among different graphs, to generate the representation of the variant subgraph G V :

Original: [Equation]
Revised: [Equation]

Original: ) where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .
Revised: where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .

Original: The representation of all the variant subgraphs is denotedas H =
Revised: The representation of all the variant subgraphs is denoted as H =

Original: [ h V 1 , ..., h V N ] .
Revised: [ h V 1 , ..., h V N ] .

Original: After obtaining H , we use an off-the-shelf clustering algorithm to infer theenvironment label E infer .
Revised: After obtaining H , we use an off-the-shelf clustering algorithm to infer the environment label E infer .

Original: In this paper, we adopt k-means [16] as our clustering algorithm:
Revised: In this paper, we adopt k-means [15] as our clustering algorithm:

Original: E infer =
Revised: [Equation]

Original: k-means ( H ) .
Revised: means ( H ) .

Original:  Using E infer , we can partition the graph dataset into multiple training environments, i.e., G ={G e } e ∈ supp( E infer ) .
Revised: [Equation] Using E infer , we can partition the graph dataset into multiple training environments, i.e., G = {G e } e ∈ supp( E infer ) .

Original: The environment inference module is purely unsupervised without needing anyground-truth environment labels.
Revised: The environment inference module is purely unsupervised without needing any ground-truth environment labels.

Original: 3.3 Invariant Learning
Revised: 3.3 Invariant Learning

Original: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariantlearning module which can generate OOD generalized graph representations under distribution shifts.
Revised: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariant learning module which can generate OOD generalized graph representations under distribution shifts.

Original: Recall that both the invariant subgraph identification module and environment inference moduleheavily depend on the generator Φ .
Revised: Recall that both the invariant subgraph identification module and environment inference module heavily depend on the generator Φ .

Original: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.
Revised: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.

Original: First,following the invariant learning literature [10], we give the following definition.
Revised: First, following the invariant learning literature [9], we give the following definition.

Original: Definition 1.
Revised: Definition 1.

Original: The invariant subgraph generator set I with respect to E is defined as:
Revised: The invariant subgraph generator set I with respect to E is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.
Revised: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only ifit is the maximal invariant subgraph generator, i.e.,
Revised: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only if it is the maximal invariant subgraph generator, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where I ( · ; · ) is the mutual information between the label and the generated subgraph.
Revised: where I ( · ; · ) is the mutual information between the label and the generated subgraph.

Original: The proof is provided in Appendix.
Revised: The proof is provided in Appendix.

Original: Eq.
Revised: Eq.

Original: (7) provides us an objective function to optimize the subgraphgenerator.
Revised: (7) provides us an objective function to optimize the subgraph generator.

Original: However, directly solving Eq.
Revised: However, directly solving Eq.

Original: (7) for a non-linear Φ is difficult [10].
Revised: (7) for a non-linear Φ is difficult [9].

Original: Following theinvariant learning literature [10], we transform Eq.
Revised: Following the invariant learning literature [9], we transform Eq.

Original: (7) into an invariance regularizer:
Revised: (7) into an invariance regularizer:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnableparameters.
Revised: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnable parameters.

Original: Recall that g ( · ) is the representation learning function of the invariant subgraphs andw ( · ) is the classifier.
Revised: Recall that g ( · ) is the representation learning function of the invariant subgraphs and w ( · ) is the classifier.

Original: We instantiate g as another GNN as follows:
Revised: We instantiate g as another GNN as follows:

Original: [Equation]
Revised: [Equation]

Original: Z I and h I are the node-level and graph-level representations of invariant subgraph G I , respectively.w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.
Revised: and h I are the node-level and graph-level representations of invariant subgraph G I , respectively. w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.

Original: Byoptimizing Eq.
Revised: By optimizing Eq.

Original: (8), we can get our desired generator Φ and the subgraph representation learningfunction g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ [-.3.
-]
Revised: (8), we can get our desired generator Φ and the subgraph representation learning function g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ [-.3.
-].

Original:  Discussions
Revised: 3.4 Discussions

Original: Training Procedure.
Revised: Training Procedure.

Original: We present the pseudocode of GIL in Appendix.
Revised: We present the pseudocode of GIL in Appendix.

Original: Time Complexity.
Revised: Time Complexity.

Original: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotesthe number of nodes and edges, respectively, and d is the dimensionality of the representations.
Revised: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotes the number of nodes and edges, respectively, and d is the dimensionality of the representations.

Original: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexityof O ( | E | d + | V | d 2 ) .
Revised: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexity of O ( | E | d + | V | d 2 ) .

Original: Since we only need to generate mask for the existing edges in graphs, the timecomplexity of generating invariant and variant subgraphs and further obtaining their representationsis O ( | E | d + | V | d 2 ) .
Revised: Since we only need to generate mask for the existing edges in graphs, the time complexity of generating invariant and variant subgraphs and further obtaining their representations is O ( | E | d + | V | d 2 ) .

Original: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B|is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes thenumber of inferred environments.
Revised: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B| is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes the number of inferred environments.

Original: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) ,as the number of parameters for most GNNs is O ( d 2 ) .
Revised: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) , as the number of parameters for most GNNs is O ( d 2 ) .

Original: Since |B| , |E infer | , and T are small constants,the overall time complexity of GIL is O ( | E
Revised: Since |B| , |E infer | , and T are small constants, the overall time complexity of GIL is O ( | E

Original: | d + | V | d 2 ) .
Revised: | d + | V | d 2 ) .

Original: In comparison, the time complexity of other
Revised: In comparison, the time complexity of other

Original: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .
Revised: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .

Original: Therefore, the time complexityof our proposed GIL is on par with the existing methods.
Revised: Therefore, the time complexity of our proposed GIL is on par with the existing methods.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically analyze our GIL model by showing that the maximal invariantsubgraph generator can achieve OOD optimal.
Revised: In this section, we theoretically analyze our GIL model by showing that the maximal invariant subgraph generator can achieve OOD optimal.

Original: The proofs are provided in Appendix.
Revised: The proofs are provided in Appendix.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote thecomplement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.
Revised: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote the complement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.

Original: Then, we can obtain the optimalpredictor under distribution shifts, i.e., the solution to Problem 1, as follows:
Revised: Then, we can obtain the optimal predictor under distribution shifts, i.e., the solution to Problem 1, as follows:

Original: [Equation]
Revised: [Equation]

Original: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,
Revised: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,

Original: ∃ e ′ ∈ supp
Revised: ∃ e ′ ∈ supp

Original: ( E ) suchthat P e ′ ( G , Y ) =
Revised: ( E ) such that P e ′ ( G , Y ) =

Original: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))
Revised: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))

Original: =
Revised: =

Original: P e (Φ( G )) .
Revised: P e (Φ( G )) .

Original: The theorem shows that we can transform the OOD generalization problem into finding the optimalinvariant subgraphs while maintaining the optimality.
Revised: The theorem shows that we can transform the OOD generalization problem into finding the optimal invariant subgraphs while maintaining the optimality.

Original: We also prove that our GIL satisfies permutation invariance in Appendix.[Table] [Figure] [Table]
Revised: We also prove that our GIL satisfies permutation invariance in Appendix.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the effectiveness of our GIL on both synthetic and real-world datasets.
Revised: In this section, we evaluate the effectiveness of our GIL on both synthetic and real-world datasets.

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We adopt one synthetic dataset with controllable ground-truth environments and fourreal-world benchmark datasets for the graph classification task.
Revised: We adopt one synthetic dataset with controllable ground-truth environments and four real-world benchmark datasets for the graph classification task.

Original: • SP-Motif :
Revised: • SP-Motif :

Original: Following [14, 17], we

Original: generate

Original:  a synthetic dataset where each graph consists of one variant subgraph and one invariant subgraph,
Revised: Following [13, 16], we generate a synthetic dataset where each graph consists of one variant subgraph and one invariant subgraph,

Original: i
Revised: i

Original: e., motif.
Revised: e., motif.

Original: The variant subgraph includes Tree , Ladder, and Wheel (denoted by V = 0, 1, 2, 
Revised: The variant subgraph includes Tree, Ladder, and Wheel (denoted by V = 0, 1, 2, respectively) and the invariant subgraph includes Cycle, House, and Crane (denoted by I = 0, 1, 2).

Original: respectively) and

Original: the

Original: invariant

Original: subgraph

Original: includes Cycle

Original: House

Original: ,and

Original: Crane

Original: denoted

Original: by

Original: I = 0, 1,

Original: The

Original:  ground-truth label Y only depends on the invariant subgraph 
Revised: The ground-truth label Y only depends on the invariant subgraph I , which is sampled uniformly.

Original: I ,

Original: which is

Original: sampled uniformly.

Original: The

Original:  spurious correlation between V and Y is injected by controllingthe variant subgraphs 
Revised: The spurious correlation between V and Y is injected by controlling the variant subgraphs distribution as: P ( V ) =

Original: distribution

Original: as: P ( V ) = r if V = I and P ( V ) =
Revised: r if V = I and P ( V ) =

Original: (1 − r ) / 2 if V ̸
Revised: (1 − r ) / 2 if V ̸

Original: = I .
Revised: = I .

Original: Intuitively, r controls the strength of the spurious correlation.
Revised: Intuitively, r controls the strength of the spurious correlation.

Original: We set r to different values in thetesting and training set to simulate the distribution shifts.
Revised: We set r to different values in the testing and training set to simulate the distribution shifts.

Original: • MNIST-75sp [18]: The task is to classify each graph that is converted from an image in MNIST [19]into the corresponding handwritten digit.
Revised: • MNIST-75sp [17]: The task is to classify each graph that is converted from an image in MNIST [18] into the corresponding handwritten digit.

Original: Distribution shifts exist on node features by adding randomnoises in the testing set.• Graph-SST2 [20]: Each graph is converted from a text sequence.
Revised: Distribution shifts exist on node features by adding random noises in the testing set. • Graph-SST2 [19]: Each graph is converted from a text sequence.

Original: Graphs are split into differentsets based on average node degrees to create distribution 
Revised: Graphs are split into different sets based on average node degrees to create distribution shifts.

Original: shifts.• Open Graph Benchmark (OGB) 
Revised: • Open Graph Benchmark (OGB) [

Original: [21]: We consider two datasets, MOLSIDER and MOLHIV.
Revised: ]: We consider two datasets, MOLSIDER and MOLHIV.

Original: Thedefault split separates structurally different molecules with different scaffolds into different subsets.
Revised: The default split separates structurally different molecules with different scaffolds into different subsets.

Original: Baselines.
Revised: Baselines.

Original: We compare our GIL with some representative state-of-the-art methods.
Revised: We compare our GIL with some representative state-of-the-art methods.

Original: The first groupof these methods generates masks on graph structures to filter out important subgraphs using different
Revised: The first group of these methods generates masks on graph structures to filter out important subgraphs using different

Original: GNNs, including Attention [2], Top-k Pool [22], SAGPool [23], and ASAP [24].
Revised: GNNs, including Attention [2], Top-k Pool [21], SAGPool [22], and ASAP [23].

Original: The second groupis invariant learning methods, including standard ERM, GroupDRO [25], IRM [9], V-REx [26],
Revised: The second group is invariant learning methods, including standard ERM, GroupDRO [24], IRM [8], V-REx [25],

Original: HRM [27], EERM [28], DIR [17].
Revised: DIR [16].

Original: We also consider a recent interpretable graph learning method GSAT [29].
Revised: We also consider a recent interpretable graph learning method GSAT [26].

Original: For a fair comparison, we use the same GNN backbone as GIL for the baselines.
Revised: For a fair comparison, we use the same GNN backbone as GIL for the baselines.

Original: Optimization and Hyper-parameters.
Revised: Optimization and Hyper-parameters.

Original: The adopted GNNs and READOUT functions including
Revised: The adopted GNNs and READOUT functions including

Original: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.
Revised: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.

Original: The hyperparameter λ in Eq.
Revised: The hyperparameter λ in Eq.

Original: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .
Revised: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .

Original: The number of clusters in Eq.
Revised: The number of clusters in Eq.

Original: (5) ischosen from [2 , 4] .
Revised: (5) is chosen from [2 , 4] .

Original: They are tuned on the validation set.
Revised: They are tuned on the validation set.

Original: We report the mean results and standarddeviations of five runs.
Revised: We report the mean results and standard deviations of five runs.

Original: More details on the datasets, baselines and implementations are in Appendix.
Revised: More details on the datasets, baselines and implementations are in Appendix.

Original: 5.2 Experiments on SP-Motif
Revised: 5.2 Experiments on SP-Motif

Original: Settings.
Revised: Settings.

Original: To simulate different degrees of distribution shifts, we vary r in both the training and testingdatasets.
Revised: To simulate different degrees of distribution shifts, we vary r in both the training and testing datasets.

Original: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: A larger r trainindicates a higher spurious correlation between Y and G V in the training set, while r train = 1 / 3implies that the training set is balanced without any spurious correlation.
Revised: A larger r train indicates a higher spurious correlation between Y and G V in the training set, while r train = 1 /implies that the training set is balanced without any spurious correlation.

Original: For the testing set, weconsider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variantsubgraphs are randomly attached without spurious correlations; (2) r test = 0 .
Revised: For the testing set, we consider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variant subgraphs are randomly attached without spurious correlations; (2) r test = 0 .

Original: 2 , which indicates thatthe testing set has reversed spurious correlations and thus is more challenging.[Figure] GIL GSAT
Revised: 2 , which indicates that the testing set has reversed spurious correlations and thus is more challenging.

Original: DIR

Original: SAG Pool Top-k Pool

Original: ASAP

Original: Attention

Original: [Figure]

Original: EnvironmentsTreeLadderWheel

Original: Quantitative Results.
Revised: Quantitative Results.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We have the following observations.
Revised: We have the following observations.

Original: Ourproposed GIL model consistently and significantly outperforms the baselines and achieves the bestperformance on all settings.
Revised: Our proposed GIL model consistently and significantly outperforms the baselines and achieves the best performance on all settings.

Original: The results demonstrate that our proposed method can well handle graphdistribution shifts and have a remarkable out-of-distribution generalization ability. 
Revised: The results demonstrate that our proposed method can well handle graph distribution shifts and have a remarkable out-of-distribution generalization ability. [Table] [Figure] [Figure]

Revised: EnvironmentsTreeLadderWheel

Original: As r train grows larger, the performance of all the methods tends to decrease since there exists alarger degree of distribution shift.
Revised: As r train grows larger, the performance of all the methods tends to decrease since there exists a larger degree of distribution shift.

Original: Nevertheless, our proposed method is able to maintain the mostrelatively stable performance.
Revised: Nevertheless, our proposed method is able to maintain the most relatively stable performance.

Original: In fact, the performance gap between GIL and the baselines becomesmore significant as the degree of distribution shift increases.
Revised: In fact, the performance gap between GIL and the baselines becomes more significant as the degree of distribution shift increases.

Original: For example, when r test = 1 / 3 , theaccuracy of all baselines drops by more than 7% when r train changes from 0 .
Revised: For example, when r test = 1 / 3 , the accuracy of all baselines drops by more than 7% when r train changes from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 8 , indicatingtheir poor OOD generalization ability.
Revised: 8 , indicating their poor OOD generalization ability.

Original: In contrast, our method only has 3% performance drop.
Revised: In contrast, our method only has 3% performance drop.

Original: When the degree of distribution shift is relatively small, GNNs with different pooling methods togenerate subgraphs generally report better results.
Revised: When the degree of distribution shift is relatively small, GNNs with different pooling methods to generate subgraphs generally report better results.

Original: On the other hand, when the degree of distributionshift is large, invariant baselines show more stable performance.
Revised: On the other hand, when the degree of distribution shift is large, invariant baselines show more stable performance.

Original: Among them, DIR, which is arecently proposed invariant method specifically designed for graphs, is one competitive baseline.
Revised: Among them, DIR, which is a recently proposed invariant method specifically designed for graphs, is one competitive baseline.

Original: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classificationaccuracy in most cases.
Revised: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classification accuracy in most cases.

Original: EERM, as one node-level OOD generalization method, cannot well handlegraph-level distribution shifts for accurate predictions.

Original: Directly adopting HRM that is proposedfor raw feature data on more complex graph-structured data produces poor results.

Original: GSAT achievespromising gains over the other baselines, but our GIL still performs better than GSAT.
Revised: GSAT achieves promising gains over the other baselines, but our GIL still performs better than GSAT.

Original: Whenr test =
Revised: When r test =

Original: r train = 1 / 3 , i.e., no distribution shifts, our proposed method also achieves the best results,indicating that learning invariant subgraphs is also beneficial.
Revised: r train = 1 / 3 , i.e., no distribution shifts, our proposed method also achieves the best results, indicating that learning invariant subgraphs is also beneficial.

Original: Analysis.
Revised: Analysis.

Original: To analyze whether our proposed method can accurately capture the invariant subgraph, wecompare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.
Revised: To analyze whether our proposed method can accurately capture the invariant subgraph, we compare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.

Original: The evaluation metric is Precision@5.
Revised: The evaluation metric is Precision@5.

Original: We report the results in Figure 3.
Revised: We report the results in Figure 3.

Original: The results show that GILhas a clear advantage in discovering invariant subgraphs under latent environments, while the otherbaselines cannot handle distribution shifts well.
Revised: The results show that GIL has a clear advantage in discovering invariant subgraphs under latent environments, while the other baselines cannot handle distribution shifts well.

Original: Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .
Revised: [Figure] [Table] delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless the world 's best actors auteuil , [Table]

Original: 8and r test = 0 . 2 ) in Figure 6.
Revised: a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , po s i t i v e nega t i v e Train set

Original: More showcases are provided in Appendix.
Revised: Test set ! ! ! " Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .and r test = 0 . 2 ) in Figure 6.

Original: The figure shows that thelearned invariant subgraph of our method is more accurate than baselines.[Figure]
Revised: The figure shows that the learned invariant subgraph of our method is more accurate than baselines.

Original: 5.3 Experiments on Real-world Graphs
Revised: 5.3 Experiments on Real-world Graphs

Original: We further evaluate the effectiveness of our method on real-world graph datasets.
Revised: We further evaluate the effectiveness of our method on real-world graph datasets.

Original: The experimentalresults are presented in Table 2.
Revised: The experimental results are presented in Table 2.

Original: Our GIL achieves the best performance on all four datasets, indicatingthat GIL can well handle distribution shifts on real-world graphs.
Revised: Our GIL achieves the best performance on all four datasets, indicating that GIL can well handle distribution shifts on real-world graphs.

Original: For example, GIL increases theclassification accuracy by 1.8% on MNIST-75sp and ROC-AUC by 2.0% on MOLHIV against thestrongest baselines respectively.
Revised: For example, GIL increases the classification accuracy by 1.8% on MNIST-75sp and ROC-AUC by 2.0% on MOLHIV against the strongest baselines respectively.

Original: On MOLHIV, the results of most baselines are worse than ERM,indicating that they fail to achieve OOD generalization in this dataset.
Revised: On MOLHIV, the results of most baselines are worse than ERM, indicating that they fail to achieve OOD generalization in this dataset.

Original: Besides, different datasetshave different distribution shifts, e.g., Graph-SST2 has different node degrees, the distribution shift [Table] 79.08 ± 0.
Revised: Besides, different datasets have different distribution shifts, e.g., Graph-SST2 has different node degrees, the distribution shift of MNIST-75sp is on node features, and OGB is split based on scaffold.

Original: delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,

Original: po s i t i v e nega t

Original: i v e

Original: Train set

Original: Test set ! ! !

Original: " of MNIST-75sp is on node features, and OGB is split based on scaffold.

Original: Therefore, the results showthat our proposed method can well handle diverse types of distribution shifts in real graph datasets.
Revised: Therefore, the results show that our proposed method can well handle diverse types of distribution shifts in real graph datasets.

Original: [Table]
Revised: [Table]

Original: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is alsocompatible with the other popular GNNs.
Revised: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is also compatible with the other popular GNNs.

Original: Wetry using HIG and CIN [30] (Rank #2 and #8 onthe MOLHIV leaderboard 1 ) as the backbone since these models are orthogonal to ours.
Revised: We try using HIG and CIN [27] (Rank #2 and #8 on the MOLHIV leaderboard 1 ) as the backbone since these models are orthogonal to ours.

Original: Table 3 showsthat our GIL can consistently improve these models and achieve new SOTA on the leaderboard.
Revised: Tableshows that our GIL can consistently improve these models.

Original: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL onboth the train and test set of Graph-SST2.
Revised: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL on both the train and test set of Graph-SST2.

Original: This dataset consists of sentences with positive/negativesentiments and is more understandable for humans.
Revised: This dataset consists of sentences with positive/negative sentiments and is more understandable for humans.

Original: Figure 7 shows that our method can learn invariantsubgraphs by consistently focusing on the positive/negative words that are salient for sentimentsand distinguishing invariant/variant parts under distribution shifts.
Revised: Figure 7 shows that our method can learn invariant subgraphs by consistently focusing on the positive/negative words that are salient for sentiments and distinguishing invariant/variant parts under distribution shifts.

Original: For example, the subgraph “Theworld’s best actors” identified by GIL has a predictive and invariant relationship with the positivesentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in differentsentences.
Revised: For example, the subgraph “The world’s best actors” identified by GIL has a predictive and invariant relationship with the positive sentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in different sentences.

Original: These results validate: (1) the invariant and variant subgraphs widely exist in real-worlddatasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and furthermake predictions with high accuracy based on the learned invariant subgraphs.
Revised: These results validate: (1) the invariant and variant subgraphs widely exist in real-world datasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and further make predictions with high accuracy based on the learned invariant subgraphs.

Original: 5.4 Analysis of Environment Inference
Revised: 5.4 Analysis of Environment Inference

Original: In our proposed model, all components are jointly optimized.
Revised: In our proposed model, all components are jointly optimized.

Original: To show that the environment inferencemodule and invariant learning module can mutually enhance each other, we record the test accuracyand the Silhouette score [31], which is a commonly used evaluation metric for clustering, as themodel is trained.
Revised: To show that the environment inference module and invariant learning module can mutually enhance each other, we record the test accuracy and the Silhouette score [28], which is a commonly used evaluation metric for clustering, as the model is trained.

Original: The results on SP-Motif ( r train = 0 .
Revised: The results on SP-Motif ( r train = 0 .

Original: 8 , r test = 1 / 3 ) are shown in Figure 4.
Revised: 8 , r test = 1 / 3 ) are shown in Figure 4.

Original: We canobserve that the test accuracy and the clustering performance improve synchronously over training.
Revised: We can observe that the test accuracy and the clustering performance improve synchronously over training.

Original: Aplausible reason is that, as the training stage progresses, the invariant subgraph generator is optimizedso that it can generate more informative invariant subgraphs and therefore improve the performanceon the testing set.
Revised: A plausible reason is that, as the training stage progresses, the invariant subgraph generator is optimized so that it can generate more informative invariant subgraphs and therefore improve the performance on the testing set.

Original: On the other hand, accurately discovering invariant subgraphs can also promoteidentifying variant subgraphs, which capture the environment-discriminate features and better inferthe latent environments.
Revised: On the other hand, accurately discovering invariant subgraphs can also promote identifying variant subgraphs, which capture the environment-discriminate features and better infer the latent environments.

Original: To verify that GIL can infer the environments accurately, we use t-SNE [32]to plot the discovered environments on a 2D-plane when the optimization is finished.
Revised: To verify that GIL can infer the environments accurately, we use t-SNE [29] to plot the discovered environments on a 2D-plane when the optimization is finished.

Original: Figure 5 showsthat the variant subgraphs perfectly capture the environment-discriminate features.
Revised: Figure 5 shows that the variant subgraphs perfectly capture the environment-discriminate features.

Original: Notice that GILachieves such results without needing any ground-truth environment label.
Revised: Notice that GIL achieves such results without needing any ground-truth environment label.

Original: 5.5 Hyper-parameter Sensitivity
Revised: 5.5 Hyper-parameter Sensitivity

Original: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask tin Eq.
Revised: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask t in Eq.

Original: For simplicity, we only report the results on SP-Motif ( r train = 0 .
Revised: For simplicity, we only report the results on SP-Motif ( r train = 0 .

Original: 8 and r test = 1 / 3 ) and
Revised: 8 and r test = 1 / 3 ) and

Original: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.
Revised: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.

Original: First, the number of environments has a moderate impact on the model performance.
Revised: First, the number of environments has a moderate impact on the model performance.

Original: For SP-Motif, the performance reaches a peak when |E infer | = 3 , showing that GILachieves the best result when the number of environments matches the ground truth.
Revised: For SP-Motif, the performance reaches a peak when |E infer | = 3 , showing that GIL achieves the best result when the number of environments matches the ground truth.

Original: [Figure]
Revised: [Figure]

Original: The number ofenvironments
Revised: The number of

Original: [Figure]
Revised: [Figure]

Original:  For MNIST-75sp, the bestnumber of environments is|E infer | = 2 .
Revised: |E infer | [Figure] For MNIST-75sp, the best of environments is |E infer | = 2 .

Original: A plausible reason is that a large number ofenvironments will bring difficulty in inferring the latentenvironment, leading to sub-optimal performance.
Revised: A plausible reason is that a large number of environments will bring difficulty in inferring the latent environment, leading to sub-optimal performance.

Original: Second, the coefficient λ also has a slight influence on theperformance, indicating that we need to properly balancethe classification loss and the invariance regularizer term.
Revised: Second, the coefficient λ also has a slight influence on the performance, indicating that we need to properly balance the classification loss and the invariance regularizer term.

Original: Finally, a proper value of the mask size t is important.
Revised: Finally, a proper value of the mask size t is important.

Original: A very large t will result in too many edges in the invariant subgraph and bring in variant structures, while asmall t may let the invariant subgraph become too smallto capture enough structural information.
Revised: A very large t will result in too many edges in the invariant subgraph and bring in variant structures, while a small t may let the invariant subgraph become too small to capture enough structural information.

Original: Although an appropriate choice of hyper-parameters can further improvethe performance, our method is not very sensitive to hyperparameters.
Revised: Although an appropriate choice of hyper-parameters can further improve the performance, our method is not very sensitive to hyperparameters.

Original: Figure 8 shows that GIL can outperformthe best baselines with a wide range of hyper-parameterschoices.
Revised: Figure 8 shows that GIL can outperform the best baselines with a wide range of hyper-parameters choices.

Original:  Related Works
Revised:  Related Works

Original: Graph neural networks.
Revised: Graph neural networks.

Revised: Graph structured data is ubiquitous in the real world.

Revised: Recently, graph neural networks (GNNs) have shown enormous success in graph representation learning [1–3], demonstrating their strength in various tasks [30–36].

Original: GNNs generally adopt a neighborhood aggregation (message passing)paradigm, i.e., the representations of nodes are iteratively updated by aggregating representationsof their neighbors [1, 2, 33].
Revised: GNNs generally adopt a neighborhood aggregation (message passing) paradigm, i.e., the representations of nodes are iteratively updated by aggregating representations of their neighbors.

Original: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 23].
Revised: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 22].

Original: However, most existing GNN modelsdo not consider the out-of-distribution generalization ability  so that their performances can dropsubstantially on testing graphs with distribution shifts.
Revised: However, most existing GNN models do not consider the out-of-distribution generalization ability [37] so that their performances can drop substantially on testing graphs with distribution shifts.

Original: Generalization of GNNs.
Revised: Generalization of GNNs.

Original: Early works  for analyzing the generalization ability of GNNs do not consider distribution shifts [34–37].
Revised: Early works [38–41] for analyzing the generalization ability of GNNs do not consider distribution shifts [37, 42, 43].

Original: More recently, the generalization ability of GNNs under distributionshifts starts to receive research attention.
Revised: More recently, the generalization ability of GNNs under distribution shifts starts to receive research attention [44, 16, 45, 46].

Original: [38] find that encoding task-specific non-linearities in thearchitecture or features can improve GNNs in extrapolating graph algorithmic tasks.
Revised: [47] find that encoding task-specific non-linearities in the architecture or features can improve GNNs in extrapolating graph algorithmic tasks.

Original: [18, 39, 40] tryto encourage GNNs to perform well on testing graphs with different sizes.
Revised: [17, 48, 49] try to encourage GNNs to perform well on testing graphs with different sizes.

Original: Some works [41, 42] areproposed to deal with node-level tasks.
Revised: Some works [50, 51] are proposed to deal with node-level tasks.

Original: EERM [28] studies the OOD generalization in node classification.
Revised: EERM [52] studies the OOD generalization in node classification.

Original: However, little attention has been paid to learning graph-level representations under distributionshifts from the invariant learning perspective.
Revised: However, little attention has been paid to learning graph-level representations under distribution shifts from the invariant learning perspective.

Original: One exception is thevery recent pioneering work DIR [17], which conducts interventions on graphs to create interventional distributions.
Revised: One exception is the work DIR [16], which conducts interventions on graphs to create interventional distributions.

Original: However,performing causal intervention relies on strong assumptions [43] that could be violated and expensiveto satisfy in practice [44].
Revised: However, performing causal intervention relies on strong assumptions [53] that could be violated and expensive to satisfy in practice [54].

Original: GSAT [29] applies graph information bottleneck criteria for generalization,but its goal is mainly to build inherently interpretable GNNs.
Revised: GSAT [26] applies graph information bottleneck criteria for generalization, but its goal is mainly to build inherently interpretable GNNs.

Original: Invariant Learning.
Revised: Invariant Learning.

Original: Invariant learning aims to exploit the invariant relationships between featuresand labels across distribution shifts, while filtering out the variant spurious correlations.
Revised: Invariant learning aims to exploit the invariant relationships between features and labels across distribution shifts, while filtering out the variant spurious correlations.

Original: Backed bycausal theory, invariant learning model can lead to OOD optimal models under some assumptions [12,9, 10].
Revised: Backed by causal theory, invariant learning model can lead to OOD optimal models under some assumptions [11, 8, 9].

Original: However, most existing methods heavily rely on multiple environments that have to beexplicitly provided in the training dataset.
Revised: However, most existing methods heavily rely on multiple environments that have to be explicitly provided in the training dataset.

Original: Such annotation is not only prohibitively expensive forgraphs, but also inherently problematic as the environment split could be inaccurate, renderingthese invariant learning methods inapplicable.
Revised: Such annotation is not only prohibitively expensive for graphs, but also inherently problematic as the environment split could be inaccurate, rendering these invariant learning methods inapplicable.

Original: A few works study OOD generalization on latentenvironments in computer vision [45, 46] or raw feature data [27], which cannot be directly appliedto graphs.
Revised: A few works study OOD generalization on latent environments in computer vision [55, 56] or raw feature data [57], which cannot be directly applied to graphs.

Original: In summary, how to learn invariant graph representations without explicit environmentlabel under distribution shits remains largely unexplored in the literature.
Revised: In summary, how to learn invariant graph representations without explicit environment label under distribution shits remains largely unexplored in the literature.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we propose the graph invariant learning ( GIL ) model to tackle the problem of learninginvariant graph representations under distribution shifts.
Revised: In this paper, we propose the graph invariant learning ( GIL ) model to tackle the problem of learning invariant graph representations under distribution shifts.

Original: Three tailored modules are jointly optimizedto encourage the graph representations to capture the invariant relationships between predictive graphstructural information and labels.
Revised: Three tailored modules are jointly optimized to encourage the graph representations to capture the invariant relationships between predictive graph structural information and labels.

Original: Theoretical analysis and extensive experiments on both syntheticand real-world datasets demonstrate the superiority of GIL .
Revised: Theoretical analysis and extensive experiments on both synthetic and real-world datasets demonstrate the superiority of GIL .

Revised: Acknowledgements

Revised: This work was supported in part by the National Key Research and Development Program of China

Revised: No. 2020AAA0106300 and National Natural Science Foundation of China (No. 62250008, 62222209, 62102222, 62206149), China National Postdoctoral Program for Innovative Talents

Revised: No. BX20220185,

Revised: China Postdoctoral Science Foundation No. 2022M711813.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]