Original: Learning Invariant Graph Representations Under Distribution Shifts
Revised: Learning Invariant Graph Representations Under Distribution Shifts

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Graph representation learning has shown effectiveness when testing and traininggraph data come from the same distribution, but most existing approaches fail togeneralize under distribution shifts.
Revised: Graph representation learning has shown effectiveness when testing and traininggraph data come from the same distribution, but most existing approaches fail togeneralize under distribution shifts.

Original: Invariant learning, backed by the invarianceprinciple from causality, can achieve guaranteed generalization under distributionshifts in theory and has shown great successes in practice.
Revised: Invariant learning, backed by the invarianceprinciple from causality, can achieve guaranteed generalization under distributionshifts in theory and has shown great successes in practice.

Original: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.
Revised: However, invariant learning for graphs under distribution shifts remains unexplored and challenging.

Original: Tosolve this problem, we propose Graph Invariant Learning ( GIL ) model capable oflearning generalized graph representations under distribution shifts.
Revised: Tosolve this problem, we propose Graph Invariant Learning ( GIL ) model capable oflearning generalized graph representations under distribution shifts.

Original: Our proposedmethod can capture the invariant relationships between predictive graph structuralinformation and labels in a mixture of latent environments through jointly optimizing three tailored modules.
Revised: Our proposedmethod can capture the invariant relationships between predictive graph structuralinformation and labels in a mixture of latent environments through jointly optimizing three tailored modules.

Original: Specifically, we first design a GNN-based subgraphgenerator to identify invariant subgraphs.
Revised: Specifically, we first design a GNN-based subgraphgenerator to identify invariant subgraphs.

Original: Then we use the variant subgraphs, i.e.,complements of invariant subgraphs, to infer the latent environment labels.
Revised: Then we use the variant subgraphs, i.e.,complements of invariant subgraphs, to infer the latent environment labels.

Original: Wefurther propose an invariant learning module to learn graph representations thatcan generalize to unknown test graphs.
Revised: Wefurther propose an invariant learning module to learn graph representations thatcan generalize to unknown test graphs.

Original: Theoretical justifications for our proposedmethod are also provided.
Revised: Theoretical justifications for our proposedmethod are also provided.

Original: Extensive experiments on both synthetic and real-worlddatasets demonstrate the superiority of our method against state-of-the-art baselinesunder distribution shifts for the graph classification task.
Revised: Extensive experiments on both synthetic and real-worlddatasets demonstrate the superiority of our method against state-of-the-art baselinesunder distribution shifts for the graph classification task.

Original:  Introduction
Revised:  Introduction

Original: Graph structured data is ubiquitous in the real world, e.g., social networks, biology networks, chemicalmolecules, etc.
Revised: Graph structured data is ubiquitous in the real world, e.g., social networks, biology networks, chemicalmolecules, etc.

Original: Graph representation learning, which encodes graphs into vectorized representations,has been the central topic in graph machine learning in the last decade.
Revised: Graph representation learning, which encodes graphs into vectorized representations,has been the central topic in graph machine learning in the last decade.

Original: For example, graph neuralnetworks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information andare shown to be successful in a variety of applications.
Revised: For example, graph neuralnetworks (GNNs) [1–3] design end-to-end learning schemes to extract useful graph information andare shown to be successful in a variety of applications.

Original: Despite the enormous success, the existing approaches for learning graph representations heavily relyon the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from anidentical distribution.
Revised: Despite the enormous success, the existing approaches for learning graph representations heavily relyon the I.I.D. assumption, i.e., the testing and training graph data are independently drawn from anidentical distribution.

Original: However, distribution shifts of graph data widely exist in real-world scenariosand are usually inevitable due to the uncontrollable underlying data generation mechanism [4].
Revised: However, distribution shifts of graph data widely exist in real-world scenariosand are usually inevitable due to the uncontrollable underlying data generation mechanism [4].

Original: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.
Revised: Most existing approaches fail to generalize to out-of-distribution (OOD) testing graph data.

Original: Onecritical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely oncorrelations that are variant for graphs from different environments.
Revised: Onecritical bottleneck is that the existing methods ignore the invariant graph patterns and tend to rely oncorrelations that are variant for graphs from different environments.

Original: Therefore, it is of paramountsignificance to learn graph representations under distribution shifts and develop methods capable ofout-of-distribution (OOD) generalization.
Revised: Therefore, it is of paramountsignificance to learn graph representations under distribution shifts and develop methods capable ofout-of-distribution (OOD) generalization.

Original: Such studies are especially critical for high-stake graphapplications such as medical diagnosis [5], criminal justice [6], financial analysis [7], molecularprediction [8], etc.
Revised: Such studies are especially critical for high-stake graphapplications such as medical diagnosis [5], criminal justice [6], financial analysis [7], molecularprediction [8], etc.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.
Revised: In this work, we propose a brand new methodology to learn invariant graph representation under distribution shifts.

Original: Invariant learning, whichaims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spuriouscorrelations, can provably achieve
Revised: Invariant learning, whichaims to exploit the invariant relationships between features and labels across different distributions while disregarding the variant spuriouscorrelations, can provably achieve

Original: satisfactory
Revised: satisfactory

Original: OOD generalization under distribution shifts [9–11].
Revised: OOD generalization under distribution shifts [9–11].

Original: Though invariant learning has been studied for images and texts [9, 12], it remains largelyunexplored in the literature of graph representation learning.
Revised: Though invariant learning has been studied for images and texts [9, 12], it remains largelyunexplored in the literature of graph representation learning.

Original: However, invariant graph representationlearning is non-trivial due to the following challenges.
Revised: However, invariant graph representationlearning is non-trivial due to the following challenges.

Original: First, graph data usually comes from a mixtureof latent environments without accurate environment labels, as shown in Figure 1.
Revised: First, graph data usually comes from a mixtureof latent environments without accurate environment labels, as shown in Figure 1.

Original: Since mostinvariant methods require multiple training environments with explicit environment labels, theseexisting methods cannot be directly applied to graphs.
Revised: Since mostinvariant methods require multiple training environments with explicit environment labels, theseexisting methods cannot be directly applied to graphs.

Original: Second, the formation process of graphsis affected by the complex interaction of both invariant and variant patterns.
Revised: Second, the formation process of graphsis affected by the complex interaction of both invariant and variant patterns.

Original: How to identify theinvariant patterns among latent environments is even more challenging.
Revised: How to identify theinvariant patterns among latent environments is even more challenging.

Original: Last but not least, even afterhaving obtained the environmental labels, how to design a theoretically grounded learning scheme togenerate graph representations capable of OOD generalization remains largely unexplored.
Revised: Last but not least, even afterhaving obtained the environmental labels, how to design a theoretically grounded learning scheme togenerate graph representations capable of OOD generalization remains largely unexplored.

Original: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL )
Revised: To tackle these challenges, in this paper, we propose Graph Invariant Learning method ( GIL )

Original: whichis able to capture invariant graph patterns in a mixture of latent environments and capable of OODgeneralization under distribution shifts.
Revised: whichis able to capture invariant graph patterns in a mixture of latent environments and capable of OODgeneralization under distribution shifts.

Original: As shown in Figure 2, our proposed method can capturethe invariant relationships between predictive graph structural information and labels in a mixtureof latent environments through jointly optimizing three mutually promoting modules, with eachmodule tackling one aforementioned challenge.
Revised: As shown in Figure 2, our proposed method can capturethe invariant relationships between predictive graph structural information and labels in a mixtureof latent environments through jointly optimizing three mutually promoting modules, with eachmodule tackling one aforementioned challenge.

Original: Specifically, in the invariant subgraph identificationmodule, we design a GNN-based subgraph generator to identify potentially invariant subgraphsfrom the complex interaction between invariant and variant patterns.
Revised: Specifically, in the invariant subgraph identificationmodule, we design a GNN-based subgraph generator to identify potentially invariant subgraphsfrom the complex interaction between invariant and variant patterns.

Original: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering theseenvironment-discriminative features.
Revised: Then, we use the variant subgraphs, i.e., the complement of invariant subgraphs, to infer environment labels by clustering theseenvironment-discriminative features.

Original: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.
Revised: The variant subgraphs capture variant correlations under different distributions and therefore contain informative features to infer environment labels.

Original: Lastly, in theinvariant learning module, we propose to optimize the maximal invariant subgraph generator criteriongiven the identified invariant subgraphs and inferred environments to generate graph representationscapable of OOD generalization under distribution shifts.
Revised: Lastly, in theinvariant learning module, we propose to optimize the maximal invariant subgraph generator criteriongiven the identified invariant subgraphs and inferred environments to generate graph representationscapable of OOD generalization under distribution shifts.

Original: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generatorof our GIL , and further prove that our GIL satisfies permutation invariance.
Revised: We theoretically show that the OOD generalization problem on graphs can be formulated as finding a maximal invariant subgraph generatorof our GIL , and further prove that our GIL satisfies permutation invariance.

Original: We conduct extensiveexperiments on both synthetic graph datasets and real graph benchmarks for the graph classificationtask.
Revised: We conduct extensiveexperiments on both synthetic graph datasets and real graph benchmarks for the graph classificationtask.

Original: The results show that the representations learned from GIL achieve substantial performancegains on the unseen OOD testing graphs compared with various state-of-the-art baselines.
Revised: The results show that the representations learned from GIL achieve substantial performancegains on the unseen OOD testing graphs compared with various state-of-the-art baselines.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows.

Original: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized
Revised: • We propose a novel Graph Invariant Learning method ( GIL ) to learn invariant and OOD generalized

Original: graph representations under distribution shifts.
Revised: graph representations under distribution shifts.

Original: To the best of our knowledge, we are the first to
Revised: To the best of our knowledge, we are the first to

Original: study invariant learning for graph representation learning under a mixture of latent
Revised: study invariant learning for graph representation learning under a mixture of latent

Original: environments.•
Revised: environments.•

Original: Our proposed method can automatically infer the environment label of graphs from a mixture oflatent environments without supervision.•
Revised: Our proposed method can automatically infer the environment label of graphs from a mixture oflatent environments without supervision.•

Original: We propose maximal invariant subgraph generator criterion to learn graph representations capable
Revised: We propose maximal invariant subgraph generator criterion to learn graph representations capable

Original: of OOD-generalization
Revised: of OOD-generalization

Original: under distribution
Revised: under distribution

Original: shifts.• We theoretically show that finding a maximal invariant subgraph generator of GIL can solve theOOD generalization problem.
Revised: shifts.• We theoretically show that finding a maximal invariant subgraph generator of GIL can solve theOOD generalization problem.

Original: Extensive empirical results demonstrate the effectiveness of GIL onvarious synthetic and benchmark datasets under distribution shifts.
Revised: Extensive empirical results demonstrate the effectiveness of GIL onvarious synthetic and benchmark datasets under distribution shifts.

Original:  Notations and Problem Formulation
Revised:  Notations and Problem Formulation

Original: Notations.
Revised: Notations.

Original: Let G and Y be the graph and label space.
Revised: Let G and Y be the graph and label space.

Original: We consider a graph dataset G = {
Revised: We consider a graph dataset G = {

Original: ( G i , Y i ) } Ni =
Revised: ( G i , Y i ) } Ni =

Original: 1where G i ∈ G and Y i ∈ Y .
Revised: 1where G i ∈ G and Y i ∈ Y .

Original: Following the OOD convention [12, 9, 10], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } denote variables graph label
Revised: Following the OOD convention [12, 9, 10], we assume the dataset is collected from multiple training environments, i.e., G = {G e } e ∈ supp( E tr ) , where G e = { ( G ei , Y ei ) } denote variables graph label

Original: G
Revised: G

Original: Y to
Revised: Y to

Original: [Figure]
Revised: [Figure]

Original: Prediction
Revised: Prediction

Original: [Figure]
Revised: [Figure]

Original: Clustervariantsubgraphs {" - } toinferenvironments …
Revised: Clustervariantsubgraphs {" - } toinferenvironments …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B % =
Revised: B % =

Original: Tree …
Revised: Tree …

Original: B & = Wheel
Revised: B & = Wheel

Original: Environment Inference 0 /.

Revised: Environment Inference 0 /.

Original: = 1 /2 3 ⋅

Revised: = 1 /2 3 ⋅

Original: GNN
Revised: GNN

Original: Invariant Subgraph Identification
Revised: Invariant Subgraph Identification

Original: Noderepresentation G (*) … … Graphdataset
Revised: Noderepresentation G (*) … … Graphdataset

Original: EdgeMask R ..0 0 .…
Revised: EdgeMask R ..0 0 .…

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Invariant
Revised: Invariant

Original: [Figure]
Revised: [Figure]

Original: Variant subgraphs {" - }
Revised: Variant subgraphs {" - }

Original: I n f e rr ed en v i r on m en t s ℰ VWXYZ
Revised: I n f e rr ed en v i r on m en t s ℰ VWXYZ

Original: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =
Revised: Latentenvironments: 8 $ = Ladder , 8 ( = Tree , 8 ) =

Original: Wheel
Revised: Wheel

Original: Labels: 9=
Revised: Labels: 9=

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (⋅)
Revised: (⋅)

Original: Invarianceregularizer
Revised: Invarianceregularizer

Original: Shared H∘J(⋅)
Revised: Shared H∘J(⋅)

Original: H∘J(⋅)
Revised: H∘J(⋅)

Original: H∘J(⋅) Shared
Revised: H∘J(⋅) Shared

Original: [Figure]
Revised: [Figure]

Original: ∇ , ℛ - " ∇ , ℛ - # \
Revised: ∇ , ℛ - " ∇ , ℛ - # \

Original: [Figure]
Revised: [Figure]

Original: (Var(∇ , ℛ - ))
Revised: (Var(∇ , ℛ - ))

Original: Invariant Learning …
Revised: Invariant Learning …

Original: B $ = Ladder …
Revised: B $ = Ladder …

Original: B & = Wheel …
Revised: B & = Wheel …

Original: B % =
Revised: B % =

Original: Tree ℛ - !
Revised: Tree ℛ - !

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (ℛ - )
Revised: (ℛ - )

Original: [Figure]
Revised: [Figure]

Original: Objective function ()
Revised: Objective function ()

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ∘Φ(,)
Revised: ∘Φ(,)

Original: [Figure]
Revised: [Figure]

Original: (⋅)
Revised: (⋅)

Original: [Figure]
Revised: [Figure]

Original: specify random variables from environment e .
Revised: specify random variables from environment e .

Original: The environment label for graphs is unobserved sinceit is prohibitively expensive to collect graph environment labels for most real scenarios.
Revised: The environment label for graphs is unobserved sinceit is prohibitively expensive to collect graph environment labels for most real scenarios.

Original: Problem Formulation.
Revised: Problem Formulation.

Original: We formulate the generalization under distribution shifts on graphs as:
Revised: We formulate the generalization under distribution shifts on graphs as:

Original: Problem 1.
Revised: Problem 1.

Original: Let E denote the random variable on indices of all possible environments.
Revised: Let E denote the random variable on indices of all possible environments.

Original: Our goal is tofind an optimal predictor f ∗ ( · ) :
Revised: Our goal is tofind an optimal predictor f ∗ ( · ) :

Original: G → Y that performs well on all environments:
Revised: G → Y that performs well on all environments:

Original: [Equation]
Revised: [Equation]

Original: where R ( f | e )
Revised: where R ( f | e )

Original: =
Revised: =

Original: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :
Revised: E e G , Y [ ℓ ( f ( G ) , Y )] is the risk of the predictor f on the environment e , and ℓ ( · , · ) :

Original: Y × Y → R denotes a loss function.
Revised: Y × Y → R denotes a loss function.

Original: We further decompose f ( · ) =
Revised: We further decompose f ( · ) =

Original: w ◦ h , where h ( · ) : G → R d isthe representation learning function
Revised: w ◦ h , where h ( · ) : G → R d isthe representation learning function

Original: , d is the dimensionality, and w ( · ) : R d → Y is the classifier.
Revised: , d is the dimensionality, and w ( · ) : R d → Y is the classifier.

Original: Note that supp( E tr ) ⊂ supp( E ) .
Revised: Note that supp( E tr ) ⊂ supp( E ) .

Original: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =
Revised: Besides, distribution shifts indicate that P e ( G , Y ) ̸ =

Original: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,
Revised: P e ′ ( G , Y ) , e ∈ supp( E tr ) ,

Original: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and thecorresponding label is different for training and testing graph data.
Revised: e ′ ∈ supp( E ) \ supp( E tr ) , i.e., the joint distribution of the graph and thecorresponding label is different for training and testing graph data.

Original: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [9, 10].
Revised: Problem 1 is difficult to be solved since supp( E ) is unobserved or latent [9, 10].

Original: In addition, formost graph datasets, we do not have access to accurate environment labels or environment partitions.
Revised: In addition, formost graph datasets, we do not have access to accurate environment labels or environment partitions.

Original: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good
Revised: Therefore, we focus on jointly inferring the environments of the graph dataset G and achieving good

Original: OOD generalization performance under the inferred environments.
Revised: OOD generalization performance under the inferred environments.

Original: The problem is formulated as:
Revised: The problem is formulated as:

Original: Problem 2.
Revised: Problem 2.

Original: Given a graph dataset G collected from a mixture of latent environments but withoutenvironment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) ,and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achievegood OOD generalization performance.
Revised: Given a graph dataset G collected from a mixture of latent environments but withoutenvironment labels, the task is to jointly infer graph environments E infer , i.e., G = {G e } e ∈ supp( E infer ) ,and learn a graph predictor f ∗ ( · ) in Problem 1 under the inferred environments E infer to achievegood OOD generalization performance.

Original:  Method
Revised:  Method

Original: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.
Revised: In this section, we introduce our proposed method in detail, whose framework is shown in Figure 2.

Original: We first present the invariant subgraph identification module.
Revised: We first present the invariant subgraph identification module.

Original: Then, we infer environment labelsby clustering the variant subgraphs.
Revised: Then, we infer environment labelsby clustering the variant subgraphs.

Original: Next, we introduce the maximal invariant subgraph generatorcriterion to generate graph representations which can generalize to test graphs under distributionshifts.
Revised: Next, we introduce the maximal invariant subgraph generatorcriterion to generate graph representations which can generalize to test graphs under distributionshifts.

Original: Lastly, we provide some discussions of our proposed method.
Revised: Lastly, we provide some discussions of our proposed method.

Original: 3.1 Invariant Subgraph Identification
Revised: 3.1 Invariant Subgraph Identification

Original: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationshipwith the label is invariant across different environments.
Revised: We assume that each input graph G ∈ G has an invariant subgraph G I ⊂ G so that its relationshipwith the label is invariant across different environments.

Original: We refer to the rest of the graph, i.e., thecomplement of G I , as the variant subgraph and denote it as G V .
Revised: We refer to the rest of the graph, i.e., thecomplement of G I , as the variant subgraph and denote it as G V .

Original: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.
Revised: G V represents the graph part whose relationship with the label is variant across different environments, e.g., spurious correlations.

Original: Therefore, the model will have a better OOD generalization ability if it can identify the invariantsubgraph and only uses structural information from G I .
Revised: Therefore, the model will have a better OOD generalization ability if it can identify the invariantsubgraph and only uses structural information from G I .

Original: We denote a generator to obtain the invariant subgraph as G I =
Revised: We denote a generator to obtain the invariant subgraph as G I =

Original: Φ( G ) .
Revised: Φ( G ) .

Original: Following the invariantlearning literature [13], we make an assumption on Φ( G ) as follows:
Revised: Following the invariantlearning literature [13], we make an assumption on Φ( G ) as follows:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:
Revised: Given G , there exists an optimal invariant subgraph generator Φ ∗ ( G ) satisfying:

Original: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))
Revised: a. Invariance property : ∀ e, e ′ ∈ supp( E ) , P e ( Y | Φ ∗ ( G ))

Original: =
Revised: =

Original: P e ′
Revised: P e ′

Original: ( Y | Φ ∗ ( G )) .
Revised: ( Y | Φ ∗ ( G )) .

Original: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))
Revised: b. Sufficiency property : Y = w ∗ ( g ∗ (Φ ∗ ( G )))

Original: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation
Revised: + ϵ, ϵ ⊥ G , where g ∗ ( · ) denotes a representation

Original: learning function
Revised: learning function

Original: , w ∗ is the
Revised: , w ∗ is the

Original: classifier
Revised: classifier

Original: , ⊥ indicates
Revised: , ⊥ indicates

Original: statistical independence,
Revised: statistical independence,

Original: and ϵ is
Revised: and ϵ is

Original: random noise.
Revised: random noise.

Original: The invariance assumption means that there exists a subgraph generator such that it can generateinvariant subgraphs across different environments.
Revised: The invariance assumption means that there exists a subgraph generator such that it can generateinvariant subgraphs across different environments.

Original: The sufficiency assumption means that thegenerated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.
Revised: The sufficiency assumption means that thegenerated invariant subgraphs should have sufficient predictive abilities in predicting the graph labels.

Original: Under this assumption, we instantiate Φ( · ) with learnable parameters.
Revised: Under this assumption, we instantiate Φ( · ) with learnable parameters.

Original: Consider an input graphinstance G with n nodes.
Revised: Consider an input graphinstance G with n nodes.

Original: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where
Revised: The corresponding adjacency matrix is denoted as A = { 0 , 1 } n × n , where

Original: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.
Revised: A i,j = 1 represents that there exists an edge between node i and j , and A i,j = 0 otherwise.

Original: To splitthe input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × non the adjacency matrix A .
Revised: To splitthe input graph G into G I and G V , a common strategy is to use a binary mask matrix M = { 0 , 1 } n × non the adjacency matrix A .

Original: However, directly optimizing a discrete matrix M is intractable as Ghas exponentially many subgraph candidates [14].
Revised: However, directly optimizing a discrete matrix M is intractable as Ghas exponentially many subgraph candidates [14].

Original: Besides, learning M for each graph G separatelyhinders the method from handling unseen test graphs [15].
Revised: Besides, learning M for each graph G separatelyhinders the method from handling unseen test graphs [15].

Original: Therefore, we adopt a shared learnable
Revised: Therefore, we adopt a shared learnable

Original: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:
Revised: GNN (denoted as GNN M ) to generate a soft mask matrix M = R n × n as follows:

Original: [Equation]
Revised: [Equation]

Original: where Z ( m ) is the node representation.
Revised: where Z ( m ) is the node representation.

Original: Then, we obtain the invariant and variant subgraphs as:
Revised: Then, we obtain the invariant and variant subgraphs as:

Original: [Equation]
Revised: [Equation]

Original: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largestvalues.
Revised: where A I and A V denotes the adjacency matrix of G I and G V , respectively, ⊙ means the elementwise matrix multiplication, and Top t ( · ) selects the top t -percentage of elements with the largestvalues.

Original: The parameters of GNN M are trained on all available graphs to generate the corresponding
Revised: The parameters of GNN M are trained on all available graphs to generate the corresponding

Original: G I and G V .
Revised: G I and G V .

Original: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs,as opposed to directly optimizing mask matrices.
Revised: Using the inductive learning ability of GNNs, it can also be used to unseen test graphs,as opposed to directly optimizing mask matrices.

Original: 3.2 Environment Inference
Revised: 3.2 Environment Inference

Original: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .
Revised: After obtaining the invariant and variant subgraphs, we can infer the environment label E infer .

Original: Theintuition is that since the invariant subgraph captures invariant relationships between predictive graphstructural information and labels, the variant subgraphs in turn capture variant correlations underdifferent distributions, which are environment-discriminative features.
Revised: Theintuition is that since the invariant subgraph captures invariant relationships between predictive graphstructural information and labels, the variant subgraphs in turn capture variant correlations underdifferent distributions, which are environment-discriminative features.

Original: Therefore, we can use thevariant subgraphs to infer the latent environments.
Revised: Therefore, we can use thevariant subgraphs to infer the latent environments.

Original: We adopt another GNN encoder, whose parametersare also shared among different graphs, to generate the representation of the variant subgraph G V :
Revised: We adopt another GNN encoder, whose parametersare also shared among different graphs, to generate the representation of the variant subgraph G V :

Original: [Equation]
Revised: [Equation]

Original: ) where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .
Revised: ) where READOUT is a permutation-invariant readout function that aggregates node-level representation Z V into graph-level representation h V .

Original: The representation of all the variant subgraphs is denotedas H =
Revised: The representation of all the variant subgraphs is denotedas H =

Original: [ h V 1 , ..., h V N ] .
Revised: [ h V 1 , ..., h V N ] .

Original: After obtaining H , we use an off-the-shelf clustering algorithm to infer theenvironment label E infer .
Revised: After obtaining H , we use an off-the-shelf clustering algorithm to infer theenvironment label E infer .

Original: In this paper, we adopt k-means [16] as our clustering algorithm:
Revised: In this paper, we adopt k-means [16] as our clustering algorithm:

Original: E infer =
Revised: E infer =

Original: k-means ( H ) .
Revised: k-means ( H ) .

Original: Using E infer , we can partition the graph dataset into multiple training environments, i.e., G ={G e } e ∈ supp( E infer ) .
Revised: Using E infer , we can partition the graph dataset into multiple training environments, i.e., G ={G e } e ∈ supp( E infer ) .

Original: The environment inference module is purely unsupervised without needing anyground-truth environment labels.
Revised: The environment inference module is purely unsupervised without needing anyground-truth environment labels.

Original: 3.3 Invariant Learning
Revised: 3.3 Invariant Learning

Original: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariantlearning module which can generate OOD generalized graph representations under distribution shifts.
Revised: After obtaining the inferred invariant subgraphs and environment labels, we propose the invariantlearning module which can generate OOD generalized graph representations under distribution shifts.

Original: Recall that both the invariant subgraph identification module and environment inference moduleheavily depend on the generator Φ .
Revised: Recall that both the invariant subgraph identification module and environment inference moduleheavily depend on the generator Φ .

Original: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.
Revised: Therefore, we aim to learn the optimal generator Φ ∗ in Assumption 3.1 by proposing and optimizing the maximal invariant subgraph generator criterion.

Original: First,following the invariant learning literature [10], we give the following definition.
Revised: First,following the invariant learning literature [10], we give the following definition.

Original: Definition 1.
Revised: Definition 1.

Original: The invariant subgraph generator set I with respect to E is defined as:
Revised: The invariant subgraph generator set I with respect to E is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.
Revised: Then, we show that the optimal generator Φ ∗ satisfies the following theorem.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only ifit is the maximal invariant subgraph generator, i.e.,
Revised: A generator Φ( G ) is the optimal generator that satisfies Assumption 3.1 if and only ifit is the maximal invariant subgraph generator, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where I ( · ; · ) is the mutual information between the label and the generated subgraph.
Revised: where I ( · ; · ) is the mutual information between the label and the generated subgraph.

Original: The proof is provided in Appendix.
Revised: The proof is provided in Appendix.

Original: Eq.
Revised: Eq.

Original: (7) provides us an objective function to optimize the subgraphgenerator.
Revised: (7) provides us an objective function to optimize the subgraphgenerator.

Original: However, directly solving Eq.
Revised: However, directly solving Eq.

Original: (7) for a non-linear Φ is difficult [10].
Revised: (7) for a non-linear Φ is difficult [10].

Original: Following theinvariant learning literature [10], we transform Eq.
Revised: Following theinvariant learning literature [10], we transform Eq.

Original: (7) into an invariance regularizer:
Revised: (7) into an invariance regularizer:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnableparameters.
Revised: where f ( · ) = w ◦ g ◦ Φ , E infer is the infered environment label, and θ denotes all the learnableparameters.

Original: Recall that g ( · ) is the representation learning function of the invariant subgraphs andw ( · ) is the classifier.
Revised: Recall that g ( · ) is the representation learning function of the invariant subgraphs andw ( · ) is the classifier.

Original: We instantiate g as another GNN as follows:
Revised: We instantiate g as another GNN as follows:

Original: [Equation]
Revised: [Equation]

Original: Z I and h
Revised: Z I and h

Original: I are the node-level and graph-level representations of invariant subgraph G I , respectively.w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.
Revised: I are the node-level and graph-level representations of invariant subgraph G I , respectively.w ( · ) is instantiated as a multilayer perceptron followed by the softmax activation function.

Original: Byoptimizing Eq.
Revised: Byoptimizing Eq.

Original: (8), we can get our desired generator Φ and the subgraph representation learningfunction g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ .3.

Revised: (8), we can get our desired generator Φ and the subgraph representation learningfunction g ( · ) , which collectively serve as our representation learning method h ( · ) , i.e., h = g ◦ Φ .3.

Original: Discussions
Revised: Discussions

Original: Training Procedure.
Revised: Training Procedure.

Original: We present the pseudocode of GIL in Appendix.
Revised: We present the pseudocode of GIL in Appendix.

Original: Time Complexity.
Revised: Time Complexity.

Original: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotesthe number of nodes and edges, respectively, and d is the dimensionality of the representations.
Revised: The time complexity of our GIL is O ( | E | d + | V | d 2 ) , where | V | and | E | denotesthe number of nodes and edges, respectively, and d is the dimensionality of the representations.

Original: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexityof O ( | E | d + | V | d 2 ) .
Revised: Specifically, we adopt message-passing GNNs to instantiate our GNN components, which has a complexityof O ( | E | d + | V | d 2 ) .

Original: Since we only need to generate mask for the existing edges in graphs, the timecomplexity of generating invariant and variant subgraphs and further obtaining their representationsis O ( | E | d + | V | d 2 ) .
Revised: Since we only need to generate mask for the existing edges in graphs, the timecomplexity of generating invariant and variant subgraphs and further obtaining their representationsis O ( | E | d + | V | d 2 ) .

Original: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B|is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes thenumber of inferred environments.
Revised: The time complexity of environment inference is O ( |B||E infer | Td ) , where |B|is the batch size, T is the number of iterations for the k-means algorithm, and |E infer | denotes thenumber of inferred environments.

Original: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) ,as the number of parameters for most GNNs is O ( d 2 ) .
Revised: The time complexity of the invariance regularizer is O ( |E infer | d 2 ) ,as the number of parameters for most GNNs is O ( d 2 ) .

Original: Since |B| , |E infer | , and T are small constants,the overall time complexity of GIL is O ( | E
Revised: Since |B| , |E infer | , and T are small constants,the overall time complexity of GIL is O ( | E

Original: | d + | V | d 2 ) .
Revised: | d + | V | d 2 ) .

Original: In comparison, the time complexity of other
Revised: In comparison, the time complexity of other

Original: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .
Revised: GNN-based graph representation methods is also O ( | E | d + | V | d 2 ) .

Original: Therefore, the time complexityof our proposed GIL is on par with the existing methods.
Revised: Therefore, the time complexityof our proposed GIL is on par with the existing methods.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically analyze our GIL model by showing that: (1) the maximal invariantsubgraph generator can achieve OOD optimal; (2) our GIL satisfies permutation invariance.
Revised: In this section, we theoretically analyze our GIL model by showing that the maximal invariantsubgraph generator can achieve OOD optimal.

Original: Theproofs are provided in Appendix.
Revised: The proofs are provided in Appendix.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote thecomplement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.
Revised: Let Φ ∗ be the optimal invariant subgraph generator in Assumption 3.1 and denote thecomplement as G \ Φ ∗ ( G ) , i.e., the corresponding variant subgraph.

Original: Then, we can obtain the optimalpredictor under distribution shifts, i.e., the solution to Problem 1, as follows:
Revised: Then, we can obtain the optimalpredictor under distribution shifts, i.e., the solution to Problem 1, as follows:

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,
Revised: if the following conditions hold: (1) Φ ∗ ( G ) ⊥ G \ Φ ∗ ( G ) ; and (2) ∀ Φ ∈ I E ,

Original: ∃ e ′ ∈ supp
Revised: ∃ e ′ ∈ supp

Original: ( E ) suchthat P e ′ ( G , Y )
Revised: ( E ) suchthat P e ′ ( G , Y )

Original: =
Revised: =

Original: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))
Revised: P e ′ (Φ( G ) , Y ) P e ′ ( G \ Φ( G )) and P e ′ (Φ( G ))

Original: =
Revised: =

Original: P e (Φ( G )) .
Revised: P e (Φ( G )) .

Original: The theorem shows that we can transform the OOD generalization problem into finding the optimalinvariant subgraphs while maintaining the optimality.
Revised: The theorem shows that we can transform the OOD generalization problem into finding the optimalinvariant subgraphs while maintaining the optimality.

Original: Theorem 4.2. Our proposed GIL model is permutation-invariant if GNN M and GNN I arepermutation-equivariant and READOUT I is permutation-invariant.
Revised: We also prove that our GIL satisfies permutation invariance in Appendix.

Original: The theorem shows that while learning invariant graph representations under distribution shifts, ourproposed method naturally holds permutation-invariance as other GNNs.
Revised: [Table] [Figure] [Table]

Original:  Experiments
Revised:  Experiments

Original: In this section, weempirically evaluate the effectiveness of the proposed GIL on both synthetic andreal-world datasets.
Revised: In this section, we evaluate the effectiveness of our GIL on both synthetic and real-world datasets.

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We adopt one synthetic dataset with controllable ground-truth environments and fourreal-world benchmark datasets for the graph classification task.•
Revised: We adopt one synthetic dataset with controllable ground-truth environments and fourreal-world benchmark datasets for the graph classification task.

Original:  SP-Motif 
Revised: • SP-Motif :

Revised: Following [14, 17], we

Revised: generate

Original: : Following [14, 17], we generate a synthetic dataset where each graph consists of onevariant subgraph and one invariant subgraph,i.e., motif.
Revised: a synthetic dataset where each graph consists of one variant subgraph and one invariant subgraph,

Revised: i

Revised: e., motif.

Original: The variant subgraph includes Tree, Ladder,and Wheel (denoted by V = 0, 1, 2,respectively) and the invariant subgraph includes Cycle, House,and Crane (denoted by I = 0, 1, 2).
Revised: The variant subgraph includes Tree , Ladder, and Wheel (denoted by V = 0, 1, 2,

Revised: respectively) and

Original: The ground-truth label Y only depends on the invariant subgraph 
Revised: the invariant subgraph includes Cycle

Revised: House

Revised: ,and

Revised: Crane

Revised: denoted

Revised: by

Revised: I = 0, 1,

Revised: The

Revised: ground-truth label Y

Revised: only

Revised: depends on the invariant

Revised: subgraph

Original: I , which is sampled uniformly.
Revised: I , which is sampled uniformly.

Revised: The

Original: The spurious correlation between V and Y is injected by controllingthe variant subgraphsdistribution as: P ( V ) =
Revised: spurious correlation between V and Y is injected by controllingthe variant subgraphs

Revised: distribution

Original:  r if V = I and P ( V ) =
Revised: as: P ( V ) = r if V = I and P ( V ) =

Original: (1 − r ) / 2 if V ̸
Revised: (1 − r ) / 2 if V ̸

Original: = I .
Revised: = I .

Original: Intuitively, r controls the strength of the spurious correlation.
Revised: Intuitively, r controls the strength of the spurious correlation.

Original: We set r to different values in thetesting and training set to simulate the distribution shifts.
Revised: We set r to different values in thetesting and training set to simulate the distribution shifts.

Original: • MNIST-75sp [18]: The task is to classify each graph that is converted from an image in MNIST [19]into
Revised: • MNIST-75sp [18]: The task is to classify each graph that is converted from an image in MNIST [19]into

Original: the corresponding handwritten digit.
Revised: the corresponding handwritten digit.

Original: Distribution shifts exist on node features by adding randomnoises in the testing set.• Graph-SST2 [20]: Each graph is converted from a text sequence.
Revised: Distribution shifts exist on node features by adding randomnoises in the testing set.• Graph-SST2 [20]: Each graph is converted from a text sequence.

Original: Graphs are split into differentsets based on average node degrees to create distributionshifts.
Revised: Graphs are split into differentsets based on average node degrees to create distribution

Original: • Open Graph Benchmark (OGB) [21]: We consider two datasets, MOLSIDER and MOLHIV.
Revised: shifts.• Open Graph Benchmark (OGB) [21]: We consider two datasets, MOLSIDER and MOLHIV.

Original: Thedefault split separates structurally different molecules with different scaffolds into different subsets.
Revised: Thedefault split separates structurally different molecules with different scaffolds into different subsets.

Original: Baselines.
Revised: Baselines.

Original: We compare our GIL with two groups of representative state-of-the-art methods.
Revised: We compare our GIL with some representative state-of-the-art methods.

Original: Thefirst group of these methods generates masks on graph structures to filter out important subgraphsusing different GNNs, including Attention [2], Top-k Pool [22], SAGPool [23], and ASAP [24].
Revised: The first groupof these methods generates masks on graph structures to filter out important subgraphs using different GNNs, including Attention [2], Top-k Pool [22], SAGPool [23], and ASAP [24].

Original: Thesecond group is invariant learning methods, including standard ERM, GroupDRO [25], IRM [9], 
Revised: The second groupis invariant learning methods, including standard ERM, GroupDRO [25], IRM [9], V-REx [26],

Original: V-REx [26], DIR [17].
Revised: HRM [27], EERM [28], DIR [17].

Revised: We also consider a recent interpretable method GSAT [29].

Original: For the second group of baselines, we use the same GNN backbone as GIL .
Revised: For afair comparison, we use the same GNN backbone as GIL for the baselines.

Original: Optimization and Hyper-parameters.
Revised: Optimization and Hyper-parameters.

Original: The adopted GNNs and READOUT functions including
Revised: The adopted GNNs and READOUT functions including

Original: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.
Revised: GNN M , GNN V , GNN I , READOUT V , and READOUT I are listed in Appendix.

Original: The hyperparameter λ in Eq.
Revised: The hyperparameter λ in Eq.

Original: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .
Revised: (8) is chosen from { 10 − 5 , 10 − 3 , 10 − 1 } .

Original: The number of clusters in Eq.
Revised: The number of clusters in Eq.

Original: (5) ischosen from [2 , 4] .
Revised: (5) ischosen from [2 , 4] .

Original: They are tuned on the validation set.
Revised: They are tuned on the validation set.

Original: We report the mean results and standarddeviations of five runs.
Revised: We report the mean results and standarddeviations of five runs.

Original: More details on the datasets, baselines and implementations are in Appendix.
Revised: More details on the datasets, baselines and implementations are in Appendix.

Original: 5.2 Experiments on SP-Motif
Revised: 5.2 Experiments on SP-Motif

Original: Settings.
Revised: Settings.

Original: To simulate different degrees of distribution shifts, we vary r in both the training and testingdatasets.
Revised: To simulate different degrees of distribution shifts, we vary r in both the training and testingdatasets.

Original: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: For the training set, we select r train from { 1 / 3 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: A larger r trainindicates a higher spurious correlation between Y and G V in the training set, while r train = 1 / 3implies that the training set is balanced without any spurious correlation.
Revised: A larger r trainindicates a higher spurious correlation between Y and G V in the training set, while r train = 1 / 3implies that the training set is balanced without any spurious correlation.

Original: For the testing set, weconsider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variantsubgraphs are randomly attached without spurious correlations; (2) r test = 0 .
Revised: For the testing set, weconsider two settings: (1) r test = 1 / 3 , which simulates that the invariant subgraphs and variantsubgraphs are randomly attached without spurious correlations; (2) r test = 0 .

Original: 2 , which indicates thatthe testing set has reversed spurious correlations and thus is more challenging. 
Revised: 2 , which indicates thatthe testing set has reversed spurious correlations and thus is more challenging. [Figure] GIL GSAT

Revised: DIR

Original: Quantitative Results.
Revised: SAG Pool Top-k Pool ASAP Attention [Figure]

Revised: EnvironmentsTreeLadderWheel

Revised: Quantitative Results.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We have the following observations.
Revised: We have the following observations.

Original: Ourproposed GIL model consistently and significantly outperforms the baselines and achieves the bestperformance on all settings.
Revised: Ourproposed GIL model consistently and significantly outperforms the baselines and achieves the bestperformance on all settings.

Original: The results demonstrate that our proposed method can well handle graphdistribution shifts and have a remarkable out-of-distribution generalization ability.
Revised: The results demonstrate that our proposed method can well handle graphdistribution shifts and have a remarkable out-of-distribution generalization ability.

Original: As r train grows larger, the performance of all the methods tends to decrease since there exists alarger degree of distribution shift.
Revised: As r train grows larger, the performance of all the methods tends to decrease since there exists alarger degree of distribution shift.

Original: Nevertheless, our proposed method is able to maintain the most [Table] [Figure] [Figure]
Revised: Nevertheless, our proposed method is able to maintain the mostrelatively stable performance.

Original: [Figure]

Original: relatively stable performance.

Original: In fact, the performance gap between GIL and the baselines becomesmore significant as the degree of distribution shift increases.
Revised: In fact, the performance gap between GIL and the baselines becomesmore significant as the degree of distribution shift increases.

Original: For example, when r test = 1 / 3 , theaccuracy of all baselines drops by more than 7% when r train changes from 0 .
Revised: For example, when r test = 1 / 3 , theaccuracy of all baselines drops by more than 7% when r train changes from 0 .

Original: 5 to 0 .
Revised: 5 to 0 .

Original: 8 , indicatingtheir poor OOD generalization ability.
Revised: 8 , indicatingtheir poor OOD generalization ability.

Original: In contrast, our method only has 3% performance drop.
Revised: In contrast, our method only has 3% performance drop.

Original: When the degree of distribution shift is relatively small, GNNs with different pooling methods togenerate subgraphs generally report better results.
Revised: When the degree of distribution shift is relatively small, GNNs with different pooling methods togenerate subgraphs generally report better results.

Original: On the other hand, when the degree of distributionshift is large, invariant baselines show more stable performance.
Revised: On the other hand, when the degree of distributionshift is large, invariant baselines show more stable performance.

Original: Among them, DIR, which is arecently proposed invariant method specifically designed for graphs, is the most competitive baseline.
Revised: Among them, DIR, which is arecently proposed invariant method specifically designed for graphs, is one competitive baseline.

Original: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classificationaccuracy in most cases.
Revised: Nevertheless, our proposed method outperforms DIR by more than 3% in terms of the classificationaccuracy in most cases.

Revised: EERM, as one node-level OOD generalization method, cannot well handlegraph-level distribution shifts for accurate predictions.

Revised: Directly adopting HRM that is proposedfor raw feature data on more complex graph-structured data produces poor results.

Revised: GSAT achievespromising gains over the other baselines, but our GIL still performs better than GSAT.

Original: When r test =
Revised: Whenr test =

Original: r train = 1 / 3 , i.e., no distribution shifts, our proposed methodalso achieves the best results, indicating that learning invariant subgraphs is also beneficial.
Revised: r train = 1 / 3 , i.e., no distribution shifts, our proposed method also achieves the best results,indicating that learning invariant subgraphs is also beneficial.

Original: Analysis.
Revised: Analysis.

Original: To analyze whether our proposed method can accurately capture the invariant subgraph, wecompare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.
Revised: To analyze whether our proposed method can accurately capture the invariant subgraph, wecompare GIL with baselines that also output subgraphs using the ground-truth invariant subgraphs.

Original: The evaluation metric is Precision@5.
Revised: The evaluation metric is Precision@5.

Original: We report the results in Figure 3.
Revised: We report the results in Figure 3.

Original: The results show that GILhas a clear advantage in discovering invariant subgraphs under latent environments. DIR, which alsoaims at discovering invariant subgraphs, reports the second-best result. Other baselines, includingdifferent pooling methods, cannot handle distribution shifts well.
Revised: The results show that GILhas a clear advantage in discovering invariant subgraphs under latent environments, while the otherbaselines cannot handle distribution shifts well.

Original: Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .
Revised: Besides the quantitative evaluation, we plot a showcase from the testing set of SP-Motif ( r train = 0 .

Original: 8and r test = 0 . 2 ) in Figure 6.
Revised: 8and r test = 0 . 2 ) in Figure 6.

Original: More showcases are provided in Appendix.
Revised: More showcases are provided in Appendix.

Original: The figure shows that thelearned invariant subgraph of our method is more accurate than baselines.
Revised: The figure shows that thelearned invariant subgraph of our method is more accurate than baselines.

Original: [Figure]
Revised: [Figure]

Original: 5.3 Experiments on Real-world Graphs
Revised: 5.3 Experiments on Real-world Graphs

Original: We further evaluate the effectiveness of our method on real-world graph datasets.
Revised: We further evaluate the effectiveness of our method on real-world graph datasets.

Original: The experimentalresults are presented in Table 2.
Revised: The experimentalresults are presented in Table 2.

Original: Our proposed method achieves the best performance on all fourdatasets, indicating that GIL can well handle distribution shifts on real-world graphs.
Revised: Our GIL achieves the best performance on all four datasets, indicatingthat GIL can well handle distribution shifts on real-world graphs.

Original: For example, GIL increases the classification accuracy by 2.6% on MNIST-75sp and ROC-AUC by 2.0% on [Table]
Revised: For example, GIL increases theclassification accuracy by 1.8% on MNIST-75sp and ROC-AUC by 2.0% on MOLHIV against thestrongest baselines respectively.

Original: delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,
Revised: On MOLHIV, the results of most baselines are worse than ERM,indicating that they fail to achieve OOD generalization in this dataset. Besides, different datasetshave different distribution shifts, e.g., Graph-SST2 has different node degrees, the distribution shift

Original: [Table]
Revised: [Table]

Original: 's
Revised: 79.08 ± 0.

Original:  best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,
Revised: delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil , delightful a romantic comedy unbelievably an stupid film onto loud the screen , daniel violent , mindless and the world 's best actors auteuil ,

Original: po s i t i v e nega t
Revised: po s i t i v e nega t

Original: i v e
Revised: i v e

Original: Train set
Revised: Train set

Original: Test set ! ! !"
Revised: Test set ! ! !

Original: MOLHIV against the strongest baselines respectively. On MOLHIV, the results of all baselinesexcept DIR are worse than ERM, indicating that they fail to achieve OOD generalization in thisdataset. Besides, different datasets have different distribution shifts, e.g., Graph-SST2 has differentnode degrees, the distribution shift of MNIST-75sp is on node features, and OGB is split based onscaffold.
Revised: " of MNIST-75sp is on node features, and OGB is split based on scaffold.

Original: Therefore, the results show that our proposed method can well handle diverse types ofdistribution shifts in real graph datasets.
Revised: Therefore, the results showthat our proposed method can well handle diverse types of distribution shifts in real graph datasets.

Original: [Table]
Revised: [Table]

Original: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is alsocompatible with the other popular GNNs.
Revised: For MOLHIV, besides adopting GIN [3] as backbone (shown in Table 2), our method is alsocompatible with the other popular GNNs.

Original: Wetry using HIG and CIN [27] (Rank #2 and #8 onthe MOLHIV leaderboard 1 ) as the backbone since these models are orthogonal to ours.
Revised: Wetry using HIG and CIN [30] (Rank #2 and #8 onthe MOLHIV leaderboard 1 ) as the backbone since these models are orthogonal to ours.

Original: Table 3 showsthat our GIL can consistently improve these models and achieve new SOTA on the leaderboard.
Revised: Table 3 showsthat our GIL can consistently improve these models and achieve new SOTA on the leaderboard.

Original: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL onboth the train and test set of Graph-SST2.
Revised: In addition, we present some showcases of the learned invariant subgraph of the proposed GIL onboth the train and test set of Graph-SST2.

Original: This dataset consists of sentences with positive/negativesentiments and is more understandable for humans.
Revised: This dataset consists of sentences with positive/negativesentiments and is more understandable for humans.

Original: Figure 7 shows that our method can learn invariantsubgraphs by consistently focusing on the positive/negative words that are salient for sentimentsand distinguishing invariant/variant parts under distribution shifts.
Revised: Figure 7 shows that our method can learn invariantsubgraphs by consistently focusing on the positive/negative words that are salient for sentimentsand distinguishing invariant/variant parts under distribution shifts.

Original: For example, the subgraph “Theworld’s best actors” identified by GIL has a predictive and invariant relationship with the positivesentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in differentsentences.
Revised: For example, the subgraph “Theworld’s best actors” identified by GIL has a predictive and invariant relationship with the positivesentiment label, while the subgraph “daniel auteuil” may reflect variant sentiments in differentsentences.

Original: These results validate: (1) the invariant and variant subgraphs widely exist in real-worlddatasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and furthermake predictions with high accuracy based on the learned invariant subgraphs.
Revised: These results validate: (1) the invariant and variant subgraphs widely exist in real-worlddatasets, and (2) our GIL can well identify invariant subgraphs under distribution shifts and furthermake predictions with high accuracy based on the learned invariant subgraphs.

Original: 5.4 Analysis of Environment Inference
Revised: 5.4 Analysis of Environment Inference

Original: In our proposed model, all components are jointly optimized.
Revised: In our proposed model, all components are jointly optimized.

Original: To show that the environment inferencemodule and invariant learning module can mutually enhance each other, we record the test accuracyand the Silhouette score [28], which is a commonly used evaluation metric for clustering, as themodel is trained.
Revised: To show that the environment inferencemodule and invariant learning module can mutually enhance each other, we record the test accuracyand the Silhouette score [31], which is a commonly used evaluation metric for clustering, as themodel is trained.

Original: The results on SP-Motif ( r train = 0 .
Revised: The results on SP-Motif ( r train = 0 .

Original: 8 , r test = 1 / 3 ) are shown in Figure 4.
Revised: 8 , r test = 1 / 3 ) are shown in Figure 4.

Original: We canobserve that the test accuracy and the clustering performance improve synchronously over training.
Revised: We canobserve that the test accuracy and the clustering performance improve synchronously over training.

Original: Aplausible reason is that, as the training stage progresses, the invariant subgraph generator is optimizedso that it can generate more informative invariant subgraphs and therefore improve the performanceon the testing set.
Revised: Aplausible reason is that, as the training stage progresses, the invariant subgraph generator is optimizedso that it can generate more informative invariant subgraphs and therefore improve the performanceon the testing set.

Original: On the other hand, accurately discovering invariant subgraphs can also promoteidentifying variant subgraphs, which capture the environment-discriminate features and better inferthe latent environments.
Revised: On the other hand, accurately discovering invariant subgraphs can also promoteidentifying variant subgraphs, which capture the environment-discriminate features and better inferthe latent environments.

Original: To verify that GIL can infer the environments accurately, we use t-SNE [29]to plot the discovered environments on a 2D-plane when the optimization is finished.
Revised: To verify that GIL can infer the environments accurately, we use t-SNE [32]to plot the discovered environments on a 2D-plane when the optimization is finished.

Original: Figure 5 showsthat
Revised: Figure 5 showsthat

Original: the variant subgraphs perfectly capture the environment-discriminate features.
Revised: the variant subgraphs perfectly capture the environment-discriminate features.

Original: Notice that GILachieves such results without needing any ground-truth environment label.
Revised: Notice that GILachieves such results without needing any ground-truth environment label.

Original: 5.5 Hyper-parameter Sensitivity
Revised: 5.5 Hyper-parameter Sensitivity

Original: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask tin Eq.
Revised: We investigate the sensitivity of hyper-parameters of our method, including the number of environments |E infer | , the invariance regularizer coefficient λ , and the size of the invariant subgraph mask tin Eq.

Original: For simplicity, we only report the results on SP-Motif ( r train = 0 .
Revised: For simplicity, we only report the results on SP-Motif ( r train = 0 .

Original: 8 and r test = 1 / 3 ) and
Revised: 8 and r test = 1 / 3 ) and

Original: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.
Revised: MNIST-75sp in Figure 8, while the results on other datasets show similar patterns.

Revised: First, the number of environments has a moderate impact on the model performance.

Revised: For SP-Motif, the performance reaches a peak when |E infer | = 3 , showing that GILachieves the best result when the number of environments matches the ground truth.

Original: [Figure]
Revised: [Figure]

Original: The number of environments
Revised: The number of environments

Original: [Figure]
Revised: [Figure]

Original: E

Original: [Figure]

Original: | .

Original: [Figure]

Original: First, the number of environments has a moderate impacton the model performance.

Original: For SP-Motif, the performancereaches a peak when |E infer | = 3 , showing that GILachieves the best result when the number of environmentsmatches the ground truth.

Original: For MNIST-75sp, the best number of environments is |E infer | = 2 .
Revised: For MNIST-75sp, the best number of environments is|E infer | = 2 .

Original: A plausible reason isthat a large number of environments will bring difficultyin inferring the latent environment, leading to sub-optimalperformance.
Revised: A plausible reason is that a large number ofenvironments will bring difficulty in inferring the latentenvironment, leading to sub-optimal performance.

Original: Second, the coefficient λ also has a slightinfluence on the performance, indicating that we need toproperly balance the classification loss and the invarianceregularizer term.
Revised: Second, the coefficient λ also has a slight influence on theperformance, indicating that we need to properly balancethe classification loss and the invariance regularizer term.

Original: Finally, a proper value of the mask size tis important.
Revised: Finally, a proper value of the mask size t is important.

Original: A very large t will result in too many edgesin the invariant subgraph and bring in variant structures,while a small t may let the invariant subgraph become toosmall to capture enough structural information.
Revised: A very large t will result in too many edges in the invariant subgraph and bring in variant structures, while asmall t may let the invariant subgraph become too smallto capture enough structural information.

Original: Althoughan appropriate choice of hyper-parameters can further improve the performance, our method is not very sensitive to hyper-parameters.
Revised: Although an appropriate choice of hyper-parameters can further improvethe performance, our method is not very sensitive to hyperparameters.

Original: Figure 8 shows that GIL can outperform the best baselines with a wide range of hyper-parameters choices.
Revised: Figure 8 shows that GIL can outperformthe best baselines with a wide range of hyper-parameterschoices.

Original:  Related Works
Revised:  Related Works

Original: Graph neural networks.
Revised: Graph neural networks.

Original: GNNs generally adopt a neighborhood aggregation (message passing)paradigm, i.e., the representations of nodes are iteratively updated by aggregating representationsof their neighbors [1, 2, 30].
Revised: GNNs generally adopt a neighborhood aggregation (message passing)paradigm, i.e., the representations of nodes are iteratively updated by aggregating representationsof their neighbors [1, 2, 33].

Original: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 23].
Revised: The representation of the whole graph is summarized on node representations through the readout function (i.e., pooling) [3, 23].

Original: However, most existing GNN modelsdo not consider the out-of-distribution generalization ability so that their performances can dropsubstantially on testing graphs with distribution shifts.
Revised: However, most existing GNN modelsdo not consider the out-of-distribution generalization ability so that their performances can dropsubstantially on testing graphs with distribution shifts.

Original: Generalization of GNNs.
Revised: Generalization of GNNs.

Original: Early works for analyzing the generalization ability of GNNs do not consider distribution shifts [31–34].
Revised: Early works for analyzing the generalization ability of GNNs do not consider distribution shifts [34–37].

Original: More recently, the generalization ability of GNNs under distributionshifts starts to receive research attention.
Revised: More recently, the generalization ability of GNNs under distributionshifts starts to receive research attention.

Original: [35] find that encoding task-specific non-linearities in thearchitecture or features can improve GNNs in extrapolating graph algorithmic tasks.
Revised: [38] find that encoding task-specific non-linearities in thearchitecture or features can improve GNNs in extrapolating graph algorithmic tasks.

Original: [18, 36, 37] tryto encourage GNNs to perform well on testing graphs with different sizes.
Revised: [18, 39, 40] tryto encourage GNNs to perform well on testing graphs with different sizes.

Original: Some works [38, 39] areproposed to deal with node-level tasks.
Revised: Some works [41, 42] areproposed to deal with node-level tasks.

Revised: EERM [28] studies the OOD generalization in node classification.

Original: However, little attention has been paid to learning graph-levelrepresentations under distribution shifts from the invariant learning perspective.
Revised: However, little attention has been paid to learning graph-level representations under distributionshifts from the invariant learning perspective.

Original: One exception is thevery recent pioneering work DIR [17], which conducts interventions on graphs to create interventionaldistributions.
Revised: One exception is the very recent pioneering work DIR [17], which conducts interventions on graphs to create interventional distributions.

Original: However, performing causal intervention relies on strong assumptions [40] that couldbe violated and expensive to satisfy in practice [41].
Revised: However,performing causal intervention relies on strong assumptions [43] that could be violated and expensiveto satisfy in practice [44].

Original: We also compare with DIR in the experiments.
Revised: GSAT [29] applies graph information bottleneck criteria for generalization,but its goal is mainly to build inherently interpretable GNNs.

Original: Invariant Learning.
Revised: Invariant Learning.

Original: Invariant learning aims to exploit the invariant relationships between featuresand labels across distribution shifts, while filtering out the variant spurious correlations.
Revised: Invariant learning aims to exploit the invariant relationships between featuresand labels across distribution shifts, while filtering out the variant spurious correlations.

Original: Backed bycausal theory, invariant learning model can lead to OOD optimal models under some assumptions [12,9, 10].
Revised: Backed bycausal theory, invariant learning model can lead to OOD optimal models under some assumptions [12,9, 10].

Original: However, most existing methods heavily rely on multiple environments that have to beexplicitly provided in the training dataset.
Revised: However, most existing methods heavily rely on multiple environments that have to beexplicitly provided in the training dataset.

Original: Such annotation is not only prohibitively expensive forgraphs, but also inherently problematic as the environment split could be inaccurate, renderingthese invariant learning methods inapplicable.
Revised: Such annotation is not only prohibitively expensive forgraphs, but also inherently problematic as the environment split could be inaccurate, renderingthese invariant learning methods inapplicable.

Original: A few works study OOD generalization on latentenvironments in computer vision [42, 43], which cannot be directly applied to graphs.
Revised: A few works study OOD generalization on latentenvironments in computer vision [45, 46] or raw feature data [27], which cannot be directly appliedto graphs.

Original: In summary,how to learn invariant graph representations without explicit environment label under distributionshits remains largely unexplored in the literature.
Revised: In summary, how to learn invariant graph representations without explicit environmentlabel under distribution shits remains largely unexplored in the literature.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we propose the graph invariant learning ( GIL ) model to tackle the problem of learninginvariant graph representations under distribution shifts.
Revised: In this paper, we propose the graph invariant learning ( GIL ) model to tackle the problem of learninginvariant graph representations under distribution shifts.

Original: Three tailored modules are jointly optimizedto encourage the graph representations to capture the invariant relationships between predictive graphstructural information and labels.
Revised: Three tailored modules are jointly optimizedto encourage the graph representations to capture the invariant relationships between predictive graphstructural information and labels.

Original: Theoretical analysis and extensive experiments on both syntheticand real-world datasets demonstrate the superiority of GIL .
Revised: Theoretical analysis and extensive experiments on both syntheticand real-world datasets demonstrate the superiority of GIL .

Original: Checklist
Revised: Checklist

Original: For all authors...(
Revised: For all authors...

Original: a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]