Original: Debiasing Graph Neural Networks via Learning Disentangled Causal Substructure
Revised: Debiasing Graph Neural Networks via Learning Disentangled Causal Substructure

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Most Graph Neural Networks (GNNs) predict the labels of unseen graphs by learning the correlation between the input graphs and labels.
Revised: Most Graph Neural Networks (GNNs) predict the labels of unseen graphs by learning the correlation between the input graphs and labels.

Original: However, by presentinga graph classification investigation on the training graphs with severe bias, surprisingly, we discover that GNNs always tend to explore the spurious correlationsto make decision, even if the causal correlation always exists.
Revised: However, by presentinga graph classification investigation on the training graphs with severe bias, surprisingly, we discover that GNNs always tend to explore the spurious correlationsto make decision, even if the causal correlation always exists.

Original: This implies thatexisting GNNs trained on such biased datasets will suffer from poor generalizationcapability.
Revised: This implies thatexisting GNNs trained on such biased datasets will suffer from poor generalizationcapability.

Original: By analyzing this problem in a causal view, we find that disentanglingand decorrelating the causal and bias latent variables from the biased graphs areboth crucial for debiasing.
Revised: By analyzing this problem in a causal view, we find that disentanglingand decorrelating the causal and bias latent variables from the biased graphs areboth crucial for debiasing.

Original: Inspiring by this, we propose a general disentangled
Revised: Inspiring by this, we propose a general disentangled

Original: GNN framework to learn the causal substructure and bias substructure, respectively.
Revised: GNN framework to learn the causal substructure and bias substructure, respectively.

Original: Particularly, we design a parameterized edge mask generator to explicitly split theinput graph into causal and bias subgraphs.
Revised: Particularly, we design a parameterized edge mask generator to explicitly split theinput graph into causal and bias subgraphs.

Original: Then two GNN modules supervisedby causal/bias-aware loss functions respectively are trained to encode causal andbias subgraphs into their corresponding representations.
Revised: Then two GNN modules supervisedby causal/bias-aware loss functions respectively are trained to encode causal andbias subgraphs into their corresponding representations.

Original: With the disentangledrepresentations, we synthesize the counterfactual unbiased training samples tofurther decorrelate causal and bias variables.
Revised: With the disentangledrepresentations, we synthesize the counterfactual unbiased training samples tofurther decorrelate causal and bias variables.

Original: Moreover, to better benchmark thesevere bias problem, we construct three new graph datasets, which have controllable bias degrees and are easier to visualize and explain.
Revised: Moreover, to better benchmark thesevere bias problem, we construct three new graph datasets, which have controllable bias degrees and are easier to visualize and explain.

Original: Experimental results welldemonstrate that our approach achieves superior generalization performance overexisting baselines.
Revised: Experimental results welldemonstrate that our approach achieves superior generalization performance overexisting baselines.

Original: Furthermore, owing to the learned edge mask, the proposedmodel has appealing interpretability and transferability.
Revised: Furthermore, owing to the learned edge mask, the proposedmodel has appealing interpretability and transferability.

Original:  Introduction
Revised:  Introduction

Original: Graph Neural Networks (GNNs) have exhibited powerful performance on graph data with variousapplications [16, 32, 12].
Revised: Graph Neural Networks (GNNs) have exhibited powerful performance on graph data with variousapplications [15, 33, 11].

Original: One major category of applications are the graph classification task,such as molecular graph property prediction [14, 19, 41], superpixel graph classification [13] andsocial network category classification [43, 41].
Revised: One major category of applications are the graph classification task,such as molecular graph property prediction [13, 18, 42], superpixel graph classification [12] andsocial network category classification [44, 42].

Original: It is well known that graph classification is usuallydetermined by a relevant substructure but not the whole graph structure [40, 23, 42].
Revised: It is well known that graph classification is usuallydetermined by a relevant substructure but not the whole graph structure [41, 24, 43].

Original: For example,for MNIST superpixel graph classification task, the digit subgraphs are causal ( i.e., deterministic) forlabels [33].
Revised: For example,for MNIST superpixel graph classification task, the digit subgraphs are causal ( i.e., deterministic) forlabels [34].

Original: The mutagenic property of a molecular graph depends on the functional groups ( i.e.,nitrogen dioxide (NO 2 )), rather than the irrelevant patterns ( i.e., carbon rings) [24].
Revised: The mutagenic property of a molecular graph depends on the functional groups ( i.e.,nitrogen dioxide (NO 2 )), rather than the irrelevant patterns ( i.e., carbon rings) [25].

Original: Therefore, it is afundamental requirement for GNNs to identify causal substructures, so as to make correct prediction.
Revised: Therefore, it is afundamental requirement for GNNs to identify causal substructures, so as to make correct prediction.

Original: Ideally, when the graphs are unbiased, i.e. , only the causal substructures are related with the graphlabels, the GNNs are able to utilize such substructure to predict the labels.
Revised: Ideally, when the graphs are unbiased, i.e. , only the causal substructures are related with the graphlabels, the GNNs are able to utilize such substructure to predict the labels.

Original: However, due to theuncontrollable data collection process, the graphs are inevitably biased, i.e. , existing meaninglesssubstructures spuriously correlates with labels.
Revised: However, due to theuncontrollable data collection process, the graphs are inevitably biased, i.e. , existing meaninglesssubstructures spuriously correlates with labels.

Original: Taking a colored MNIST superpixel graph dataset in
Revised: Taking a colored MNIST superpixel graph dataset in

Original: Sec.
Revised: Sec.

Original: 3.1 as an example (illustrated in Fig. 1(a)), each category of digit subgraphs mainly correspond to one kind of color background subgraphs, e.g. , digit 0 subgraph is related with red backgroundsubgraph.
Revised: 3.1 as an example (illustrated in Fig. 1(a)), each category of digit subgraphs mainly correspond to one kind of color background subgraphs, e.g. , digit 0 subgraph is related with red backgroundsubgraph.

Original: Therefore, the color background subgraph will be treated as bias information, which highlycorrelates with labels but does not determines them in the training set.
Revised: Therefore, the color background subgraph will be treated as bias information, which highlycorrelates with labels but does not determines them in the training set.

Original: Under this situation, will
Revised: Under this situation, will

Original: GNNs still stably utilize the causal substructure to make decision?
Revised: GNNs still stably utilize the causal substructure to make decision?

Original: To investigate the impact of bias on GNNs, we conduct an experimental investigation to demonstratethe impact of bias (especially in the severe bias scenarios) on the generalization capability of GNNs(Sec. 3.1).
Revised: To investigate the impact of bias on GNNs, we conduct an experimental investigation to demonstratethe impact of bias (especially in the severe bias scenarios) on the generalization capability of GNNs(Sec. 3.1).

Original: We find that GNNs actually utilize both bias and causal substructures to make prediction.
Revised: We find that GNNs actually utilize both bias and causal substructures to make prediction.

Original: However, with severer bias correlation, even bias substructure still could not exactly determine labelslike causal substructure, GNNs majorly utilize bias substructure as shortcuts to make prediction,causing a large generalization performance degradation.
Revised: However, with severer bias correlation, even bias substructure still could not exactly determine labelslike causal substructure, GNNs majorly utilize bias substructure as shortcuts to make prediction,causing a large generalization performance degradation.

Original: Why this happens?
Revised: Why this happens?

Original: We analyze the datagenerating process and model prediction mechanism behind the graph classification using a causalgraph (Sec. 3.2).
Revised: We analyze the datagenerating process and model prediction mechanism behind the graph classification using a causalgraph (Sec. 3.2).

Original: The casual graph illustrates that the observed graphs are generated by the causal andbias latent variables and existing GNNs could not distinguish the causal substructure from entangledgraphs.
Revised: The casual graph illustrates that the observed graphs are generated by the causal andbias latent variables and existing GNNs could not distinguish the causal substructure from entangledgraphs.

Original: How can we disentangle the causal and bias substructures from observed graphs, so that
Revised: How can we disentangle the causal and bias substructures from observed graphs, so that

Original: GNNs can only utilize the causal substructures to make stable prediction when severe bias appears?
Revised: GNNs can only utilize the causal substructures to make stable prediction when severe bias appears?

Original: To address the question, two challenges need to be faced.
Revised: To address the question, two challenges need to be faced.

Original: 1) How to identify the causal substructureand bias substructure in the severe biased graphs?
Revised: 1) How to identify the causal substructureand bias substructure in the severe biased graphs?

Original: In the severe bias scenarios, bias substructure willbe “easier to learn” for GNNs and finally dominate the prediction.
Revised: In the severe bias scenarios, bias substructure willbe “easier to learn” for GNNs and finally dominate the prediction.

Original: Using the normal cross-entropyloss, like DIR [37], could not fully capture such aggressive property of bias.
Revised: Using the normal cross-entropyloss, like DIR [37], could not fully capture such aggressive property of bias.

Original: 2) How to extractthe causal substructure from an entangled graph?
Revised: 2) How to extractthe causal substructure from an entangled graph?

Original: The statistically causal substructure is usuallydetermined by the global property of the entire graph population, rather than a single graph.
Revised: The statistically causal substructure is usuallydetermined by the global property of the entire graph population, rather than a single graph.

Original: Whenextracting causal substructure from a graph, we need to establish the relations among all the graphs.
Revised: Whenextracting causal substructure from a graph, we need to establish the relations among all the graphs.

Original: In this paper, we propose a novel debiasing framework for GNNs via learning Dis entangled C ausalsubstructure, called DisC .
Revised: In this paper, we propose a novel debiasing framework for GNNs via learning Dis entangled C ausalsubstructure, called DisC .

Original: Given an input biased graph, we propose to explicitly filter edges intocausal and bias subgraphs by a parameterized edge mask generator, whose parameters are sharedacross entire graph population.
Revised: Given an input biased graph, we propose to explicitly filter edges intocausal and bias subgraphs by a parameterized edge mask generator, whose parameters are sharedacross entire graph population.

Original: As a result, the edge masker is naturally capable to specify theimportance for each edge and extract causal and bias subgraphs from a global view of the entireobservations.
Revised: As a result, the edge masker is naturally capable to specify theimportance for each edge and extract causal and bias subgraphs from a global view of the entireobservations.

Original: Then, a “casual”-aware (weighted cross-entropy) loss and a “bias”-aware (generalizedcross-entropy) loss are respectively utilized to supervise two functional GNN modules.
Revised: Then, a “casual”-aware (weighted cross-entropy) loss and a “bias”-aware (generalizedcross-entropy) loss are respectively utilized to supervise two functional GNN modules.

Original: Based onthe supervision, the edge mask generator could generate corresponding subgraphs and the GNNscould encode corresponding subgraphs into their disentangled embeddings.
Revised: Based onthe supervision, the edge mask generator could generate corresponding subgraphs and the GNNscould encode corresponding subgraphs into their disentangled embeddings.

Original: With the disentangledembeddings, we randomly permute the latent vectors extracted from different graphs to generatemore unbiased counterfactual samples in embedding space.
Revised: With the disentangledembeddings, we randomly permute the latent vectors extracted from different graphs to generatemore unbiased counterfactual samples in embedding space.

Original: The new generated samples still containboth causal and bias information, while their correlation has been decorrelated.
Revised: The new generated samples still containboth causal and bias information, while their correlation has been decorrelated.

Original: In this time, there isonly correlation between causal variables with labels, so that the model could concentrate on the truecorrelation between the causal subgraphs and labels.
Revised: In this time, there isonly correlation between causal variables with labels, so that the model could concentrate on the truecorrelation between the causal subgraphs and labels.

Original: Our major contributions are as follows:
Revised: Our major contributions are as follows:

Original: • To our knowledge, we first study the generalization problem of GNNs in a more challenging yetpractical scenario, i.e., the graphs are with severe bias.
Revised: • To our knowledge, we first study the generalization problem of GNNs in a more challenging yetpractical scenario, i.e., the graphs are with severe bias.

Original: We systematically analyze the bias impacton GNNs from both experimental study and causal analysis.
Revised: We systematically analyze the bias impacton GNNs from both experimental study and causal analysis.

Original: We find that the bias substructure,compared with causal substructure, is much easier to dominate the training of GNNs.
Revised: We find that the bias substructure,compared with causal substructure, is much easier to dominate the training of GNNs.

Original: • To debias GNNs, we develop a novel GNN framework for disentangling causal substructure,which is flexible to build upon various GNNs for improving generalization ability while enjoying
Revised: • To debias GNNs, we develop a novel GNN framework for disentangling causal substructure,which is flexible to build upon various GNNs for improving generalization ability while enjoying

Original: inherent interpretability, robustness and transferability.
Revised: inherent interpretability, robustness and transferability.

Original: • We construct three new datasets with various properties and controllable bias degrees, which can
Revised: • We construct three new datasets with various properties and controllable bias degrees, which can

Original: better benchmark the new problem.
Revised: better benchmark the new problem.

Original: Our model outperforms the corresponding base models witha large margin (from 4.47% to 169.
Revised: Our model outperforms the corresponding base models witha large margin (from 4.47% to 169.

Original: % average improvements).
Revised: % average improvements).

Original: Various investigation studiesdemonstrate that our model could discover and leverage causal substructure for prediction.
Revised: Various investigation studiesdemonstrate that our model could discover and leverage causal substructure for prediction.

Original:  Related Works
Revised:  Related Works

Original: Generalization of GNNs in wild environments.
Revised: Generalization of GNNs in wild environments.

Original: Most existing GNN methods are proposed underthe IID hypothesis, i.e., training and testing set are independently sampled from the identicaldistribution [30, 16, 32, 12, 22].
Revised: Most existing GNN methods are proposed underthe IID hypothesis, i.e., training and testing set are independently sampled from the identicaldistribution [32, 15, 33, 11, 22].

Original: However, in reality, thus ideal assumption is hard to be satisfied.
Revised: However, in reality, thus ideal assumption is hard to be satisfied.

Original: Recently, several methods have been proposed to improve the generalization ability of GNNs inwild OOD environments.
Revised: Recently, several methods have been proposed to improve the generalization ability of GNNs inwild OOD environments.

Original: Several works [25, 8, 36] study the OOD problem of node classification.
Revised: Several works [27, 7, 36] study the OOD problem of node classification.

Original: For OOD graph classification task, StableGNN [7] propose to learn the stable causal relationshipin graphs.
Revised: For OOD graph classification task, StableGNN [6] propose to learn the stable causal relationship ingraphs.

Revised: OOD-GNN [20] propose to constrain each dimension of learned embedding independent.

Original: DIR [37] discovers the invariant rationales for generalizing GNNs.
Revised: DIR [37] discovers the invariant rationales for generalizing GNNs.

Original: Although they haveachieved better OOD performance, they are not designed for the datasets with severe bias, which ismore challenging for guaranteeing the generalization ability of GNNs.
Revised: Although they have achievedbetter OOD performance, they are not designed for the datasets with severe bias, which is morechallenging for guaranteeing the generalization ability of GNNs.

Revised: Disentangled graph neural networks.

Original: And we first try to solve thisproblem by disentangling method, which fundamentally conforms with the data generation process.
Revised: Recently, there are a couple of methods that study thedisentangled GNNs. DisenGCN [26] utilizes neighbourhood routing mechanism to divide theneighbours of the node into several mutually exclusive parts. IPGDN [23] promotes DisenGCNby constraining the different parts of the embedding feature independent. DisenGCN and IPGDNare node-level disentanglement, thus FactorGCN [40] considers the whole graph information anddisentangles the target graph into several factorized graphs.

Revised: Despite results of the previous works,they do not consider disentangling the causal and bias information for graphs.

Original: General debiasing methods.
Revised: General debiasing methods.

Original: Recently, debiasing problem has drawn much attention in machinelearning community [15, 21, 29, 1, 2, 10].
Revised: Recently, debiasing problem has drawn much attention in machinelearning community [14, 21, 31, 1, 2, 9].

Original: One category of these methods is pre-defining a certainbias type explicitly to mitigate [15, 21, 29, 1, 34].
Revised: One category of these methods is pre-defining a certainbias type explicitly to mitigate [14, 21, 31, 1, 35].

Original: For example, Wang et al.
Revised: For example, Wang et al.

Original: [34] and Bahng etal.
Revised: [35] and Bahng etal.

Original: [1] design a texture- and color-guided model to adversarially train a debiased neural networkagainst the biased one.
Revised: [1] design a texture- and color-guided model to adversarially train a debiased neural networkagainst the biased one.

Original: Instead of defining certain types of bias, recent approaches [26, 6, 20] relyon the straightforward assumption that models are prone to exploit the bias as shortcuts to makeprediction [9].
Revised: Instead of defining certain types of bias, recent approaches [28, 5, 19] relyon the straightforward assumption that models are prone to exploit the bias as shortcuts to makeprediction [8].

Original: In the line with the recent studies, our study belongs to the second category.
Revised: In the line with the recent studies, our study belongs to the second category.

Original: However,most of existing methods are designed for image datasets and could not effectively extract causalsubstructure from graph data.
Revised: However,most of existing methods are designed for image datasets and could not effectively extract causalsubstructure from graph data.

Original: Distinctly, we first study the severe bias problem on graph data, andour method could effectively extract causal substructure from graph data.
Revised: Distinctly, we first study the severe bias problem on graph data, andour method could effectively extract causal substructure from graph data.(a) Examples

Original: [Figure]
Revised: [Figure]

Original:  Preliminary Study and Analysis
Revised:  Preliminary Study and Analysis

Original: [Figure] In this section, we first illustrate the existing GNNs tend to exploit the bias substructure as shortcutsfor prediction through a motivating experiment.
Revised: In this section, we first illustrate the existing GNNs tend to exploit the bias substructure as shortcutsfor prediction through a motivating experiment.

Original: Then we analyze the prediction process of GNNs incausal view.
Revised: Then we analyze the prediction process of GNNs incausal view.

Original: Based on this causal view, it motivates our solution to relieve the impact of bias.
Revised: Based on this causal view, it motivates our solution to relieve the impact of bias.

Original: 3.1 Motivating Example
Revised: 3.1 Motivating Example

Original: [Figure] To measure the generalization ability of GNNs with the effect of bias, we construct a graph classification dataset with controllable bias degrees, called Colored MNIST-75sp.
Revised: To measure the generalization ability of GNNs with the effect of bias, we construct a graph classification dataset with controllable bias degrees, called Colored MNIST-75sp.

Original: We first construct a biased
Revised: We first construct a biased

Original: MNIST image dataset like [1], where each category of digit highly correlates with a pre-defined colorin their background.
Revised: MNIST image dataset like [1], where each category of digit highly correlates with a pre-defined colorin their background.

Original: For example, in the training set, 90% of 0 digits are with red background ( i.e., biased samples), and remaining 10% images are with random background color ( i.e., unbiased samples),whose the bias degree is 0.9 in this situation.
Revised: For example, in the training set, 90% of 0 digits are with red background ( i.e., biased samples), and remaining 10% images are with random background color ( i.e., unbiased samples),whose the bias degree is 0.9 in this situation.

Original: We consider four bias degrees { 0 . 8 , 0 . 85 , 0 . 9 , 0 . 95 } .
Revised: We consider four bias degrees { 0 . 8 , 0 . 85 , 0 . 9 , 0 . 95 } .

Original: For the testing set, we construct both biased testing set and unbiased testing set.
Revised: For the testing set, we construct both biased testing set and unbiased testing set.

Original: The biased testingset has the same bias degree with training set, aiming to measure the extent of models relying onbias.
Revised: The biased testingset has the same bias degree with training set, aiming to measure the extent of models relying onbias.

Original: The unbiased testing set, where the digit labels uncorrelate with the background colors, aims totest whether the model could utilize the inherent digit signals for prediction.
Revised: The unbiased testing set, where the digit labels uncorrelate with the background colors, aims totest whether the model could utilize the inherent digit signals for prediction.

Original: Note that training setand testing set have the same pre-defined color set.
Revised: Note that training setand testing set have the same pre-defined color set.

Original: Then, we convert the biased MNIST images intosuperpixel graphs with at most 75 nodes each graph using [17], where the edges are constructed bythe KNN method based on the 2D coordinates of superpixels and node features are the concatenationof coordinates and average color of superpixels.
Revised: Then, we convert the biased MNIST images intosuperpixel graphs with at most 75 nodes each graph using [16], where the edges are constructed bythe KNN method based on the 2D coordinates of superpixels and node features are the concatenation of coordinates and average color of superpixels.

Original: Each graph is labeled by its digit class, so that itsdigital subgraph is deterministic for label and background subgraph is spuriously correlated withlabels but not deterministic.
Revised: Each graph is labeled by its digit class, so that itsdigital subgraph is deterministic for label and background subgraph is spuriously correlated withlabels but not deterministic.

Original: The examples of graphs are illustrated in Fig. 1(a).
Revised: The examples of graphs are illustrated in Fig. 1(a).

Original: We perform three popular GNN methods: GCN [16], GIN [39], and GCNII [3] on Colored MNIST75sp and the results are shown in Fig. 1(b).
Revised: We perform three popular GNN methods: GCN [15], GIN [39], and GCNII [3] on Colored MNIST75sp and the results are shown in Fig. 1(b).

Original: The same color of dashed line and solid line represent theresults of the corresponding methods on the biased testing set and the unbiased testing set respectively.
Revised: The same color of dashed line and solid line represent theresults of the corresponding methods on the biased testing set and the unbiased testing set respectively.

Original: Overall, the GNNs achieve much better performance on biased testing set than unbiased testing set.
Revised: Overall, the GNNs achieve much better performance on biased testing set than unbiased testing set.

Original: The phenomenon indicates that although GNNs could still learn some causal signals for prediction,the unexpected bias information is also being utilized for prediction.
Revised: The phenomenon indicates that although GNNs could still learn some causal signals for prediction,the unexpected bias information is also being utilized for prediction.

Original: More specifically, with biasdegree becoming larger, the performance of GNNs on biased testing set is increased and the valueof accuracy is nearly in line with the bias degree, while the performance on unbiased testing dropsdramatically.
Revised: More specifically, with biasdegree becoming larger, the performance of GNNs on biased testing set is increased and the valueof accuracy is nearly in line with the bias degree, while the performance on unbiased testing dropsdramatically.

Original: Hence, although causal substructure could determine labels perfectly, in severe biasscenarios, the GNNs lean to utilize the easier to learn bias information to make prediction rather thanthe inherent causal signals, and bias substructure will finally dominate the prediction.
Revised: Hence, although causal substructure could determine labels perfectly, in severe biasscenarios, the GNNs lean to utilize the easier to learn bias information to make prediction rather thanthe inherent causal signals, and bias substructure will finally dominate the prediction.

Original: 3.2 Problem Analysis
Revised: 3.2 Problem Analysis

Original: Debiasing GNNs for unbiased prediction requires understanding the natural mechanisms of graphclassification task.
Revised: Debiasing GNNs for unbiased prediction requires understanding the natural mechanisms of graphclassification task.

Original: We present a causal view of the union of the data-generating process and the modelprediction process behind the task.
Revised: We present a causal view of the union of the data-generating process and the modelprediction process behind the task.

Original: Here we formalize the causal view as a Structure Causal Model(SCM) or causal graph [11, 27] by inspecting on the causalities among five variables: unobservedcausal variable C , unobserved bias variable B , observed graph G , graph embedding E , and groundtruth label / prediction Y1 .
Revised: Here we formalize the causal view as a Structure Causal Model(SCM) or causal graph [10, 29] by inspecting on the causalities among five variables: unobservedcausal variable C , unobserved bias variable B , observed graph G , graph embedding E , and groundtruth label / prediction Y .

Original: Fig.
Revised: Fig.

Original: 2(a) illustrates the SCM, where each link denotes a causal relationship.
Revised: 2(a) illustrates the SCM, where each link denotes a causal relationship.

Original: • C → G ← B.
Revised: • C → G ← B

Original: The observed graph data is generated by two unobserved latent variables: the causal variable C and the bias variable B , such as digit subgraphs and background subgraphs in
Revised: The observed graph data is generated by two unobserved latent variables: thecausal variable C and the bias variable B , such as digit subgraphs and background subgraphs in

Original: the Colored MNIST-75sp dataset.
Revised: the Colored MNIST-75sp dataset.

Original: And all bellow relations are illustrated by Colored MNIST-75sp.
Revised: And all bellow relations are illustrated by Colored MNIST-75sp.

Original: • C → Y .
Revised: • C → Y .

Original: This link means that the causal variable C is the only endogenous parent to determine
Revised: This link means that the causal variable C is the only endogenous parent to determine

Original: the generation of ground-truth label Y .
Revised: the generation of ground-truth label Y .

Original: For example, C is the oracle digit subgraph, which exactly explains why the label is labeled as Y .
Revised: For example, C is the oracle digit subgraph, which exactlyexplains why the label is labeled as Y .

Original: • C ⇠⇢ B .
Revised: • C ⇠⇢ B .

Original: This link indicates the spurious correlation between C and B .
Revised: This link indicates the spurious correlation between C and B .

Original: Such probabilistic
Revised: Such probabilistic

Original: dependencies is usually caused by the direct cause or unobserved confounder [28].
Revised: dependencies is usually caused by the direct cause or unobserved confounder [30].

Original: Here we do
Revised: Here we do

Original: not distinguish these scenarios and only observe the spurious correlation between B and C , such
Revised: not distinguish these scenarios and only observe the spurious correlation between B and C , such

Original: as the spurious correlation between the color background subgraphs and digit subgraphs.
Revised: as the spurious correlation between the color background subgraphs and digit subgraphs.

Original: • G
Revised: • G

Original: → E → Y .
Revised: → E → Y .

Original: Existing GNNs usually learn the graph embedding E based on the observed graph
Revised: Existing GNNs usually learn the graph embedding E based on the observed graph

Original: G and make the prediction Y based on the learned embedding E .
Revised: G and make the prediction Y based on the learned embedding E .

Original: [Figure]

Original: According to the SCM, GNNs will utilize both information to make prediction.
Revised: According to the SCM, GNNs will utilize both information to make prediction.

Original: As bias substructure( e.g., background subgraph) usually has simpler structure than meaningful causal substructure ( e.g.,digit subgraph), if GNN utilizes such simple substructure, it could achieve low loss very fast.
Revised: As bias substructure( e.g., background subgraph) usually has simpler structure than meaningful causal substructure ( e.g.,digit subgraph), if GNN utilizes such simple substructure, it could achieve low loss very fast.

Original: Hence,
Revised: Hence,

Original: GNN inclines to utilizes bias information when most graphs are biased.
Revised: GNN inclines to utilizes bias information when most graphs are biased.

Original: Based on the SCM in
Revised: Based on the SCM in

Original: Fig. 2(a), according to d -connection theory [27] (see App.
Revised: Fig. 2(a), according to d -connection theory [29] (see App.

Original: A): two variables are dependent if they areconnected by at least one unblocked path, we could find two paths that would induce the spuriouscorrelation between the bias variable B and label Y : (1) B → G → E → Y and (2) B
Revised: A): two variables are dependent if they areconnected by at least one unblocked path, we could find two paths that would induce the spuriouscorrelation between the bias variable B and label Y : (1) B → G → E → Y and (2) B

Original: ↔ C → Y .
Revised: ↔ C → Y .

Original: To make the prediction Y being uncorrelated with the bias B , we need to intercept the two unblockedpaths.
Revised: To make the prediction Y being uncorrelated with the bias B , we need to intercept the two unblockedpaths.

Original: For this purpose, we propose to debias GNNs in causal view, as in Fig. 2(b).
Revised: For this purpose, we propose to debias GNNs in causal view, as in Fig. 2(b).

Original: • C ← G → B and C → Y .
Revised: • C ← G → B and C → Y .

Original: To intercept the path (1), we should disentangle the latent variablesC
Revised: To intercept the path (1), we should disentangle the latent variables

Original:  and B from the observed graph G and make prediction only based on the causal variable C.
Revised: C and B from the observed graph G and make prediction only based on the causal variable C

Original: • C ⇠ (cid:8)(cid:8)(cid:72)(cid:72) −− ⇢ B .
Revised: • C ⇠ (cid:8)(cid:8)(cid:72)(cid:72) −− ⇢ B .

Original: To intercept the path (2), as we cannot change the link between C and Y , one
Revised: To intercept the path (2), as we cannot change the link between C and Y , one

Original: possible solution is to make C and B uncorrelated.
Revised: possible solution is to make C and B uncorrelated

Original: [Figure]

Original:  Methodology
Revised:  Methodology

Original: [Figure] Motivated by the above causal analysis, in this section, we present our proposed debiasing GNNframework DisC , to remove the spurious correlation.
Revised: Motivated by the above causal analysis, in this section, we present our proposed debiasing GNNframework DisC , to remove the spurious correlation.

Original: The overall framework is shown in Fig.
Revised: The overall framework is shown in Fig.

Original: First,an edge mask generator is learnt to mask the edges of original input graphs into causal subgraphsand bias subgraphs.
Revised: First,an edge mask generator is learnt to mask the edges of original input graphs into causal subgraphs [Figure]

Original:  Second, two separate GNN modules with their corresponding masked subgraphsare trained to encode corresponding causal substructure and bias substructure into disentangledrepresentations, respectively.
Revised: C C B and bias subgraphs. Second, two separate GNN modules with their corresponding masked subgraphsare trained to encode corresponding causal substructure and bias substructure into disentangledrepresentations, respectively.

Original: Last, after the disentangled representations are well-trained, we permutethe bias representations among the training graphs to generate counterfactual unbiased samples, sothat the correlation between causal representations and bias representations is removed.
Revised: Last, after the disentangled representations are well-trained, we permutethe bias representations among the training graphs to generate counterfactual unbiased samples, sothat the correlation between causal representations and bias representations is removed.

Original: [Figure]
Revised: [Figure]

Original: 4.1 Causal and Bias Substructure Generator
Revised: 4.1 Causal and Bias Substructure Generator

Original:  Given a mini-batch of biased graphs G = { G 1 , ⋯ , G n } , our idea is that: we take a collection of graphinstances and design a generative probabilistic model to learn to mask the edges into causal subgraphor bias subgraph.
Revised: [Figure] Given a mini-batch of biased graphs G = { G 1 , ⋯ , G n } , our idea is that: we take a collection of graphinstances and design a generative probabilistic model to learn to mask the edges into causal subgraphor bias subgraph.

Original: Particularly, given a graph G = { A , X } , where A is the adjacency matrix and X isthe node feature matrix, we utilize a multi-layer perceptron (MLP) upon the concatenation of nodefeatures
Revised: Particularly, given a graph G = { A , X } , where A is the adjacency matrix and X isthe node feature matrix, we utilize a multi-layer perceptron (MLP) upon the concatenation of nodefeatures

Original: x i of node i and x j of node j to measure the importance of edge ( i, j ) for causal subgraph:
Revised: x i of node i and x j of node j to measure the importance of edge ( i, j ) for causal subgraph:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then a sigmoid function σ ( ⋅ ) is employed to project α ij into the range of (0,1), which indicates theprobability of edge ( i, j ) being the edge in the causal subgraph as follows:[Figure]
Revised: Then a sigmoid function σ ( ⋅ ) is employed to project α ij into the range of (0,1), which indicates theprobability of edge ( i, j ) being the edge in the causal subgraph as follows:

Original: [Equation]
Revised: [Equation]

Original: Naturally, we could get the probability of edge ( i, j ) being the edge in the bias subgraph by:b ij = 1 − c ij .
Revised: Naturally, we could get the probability of edge ( i, j ) being the edge in the bias subgraph by:b ij = 1 − c ij .

Original: Now we could construct the causal edge mask M c =
Revised: Now we could construct the causal edge mask M c =

Original: [ c ij ] and bias edge mask
Revised: [ c ij ] and bias edge mask

Original: M b =
Revised: M b =

Original: [ b ij ] .
Revised: [ b ij ] .

Original: Finally, we decompose the original graph G into causal subgraph G c
Revised: Finally, we decompose the original graph G into causal subgraph G c

Original: =
Revised: =

Original: { M c ⊙ A , X }and bias subgraph G b = { M b ⊙ A , X } .
Revised: { M c ⊙ A , X }and bias subgraph G b = { M b ⊙ A , X } .

Original: Intuitively, the edge mask could highlight different part ofstructure information of original graphs, thus GNNs built on the different subgraphs could encodedifferent parts of graph information.
Revised: Intuitively, the edge mask could highlight different part ofstructure information of original graphs, thus GNNs built on the different subgraphs could encodedifferent parts of graph information.

Original: Moreover, the mask generator has two advantages.
Revised: Moreover, the mask generator has two advantages.

Original: (1) Globalview:
Revised: (1) Globalview:

Original: In individual graph level, the mask generator ( i.e. , MLP), whose parameters are shared byall the edges in a graph, take a global view of all the edges in a graph, which enables us to identifycommunity in graph.
Revised: In individual graph level, the mask generator ( i.e. , MLP), whose parameters are shared byall the edges in a graph, take a global view of all the edges in a graph, which enables us to identifycommunity in graph.

Original: It is well known that the effect of an edge cannot be judged independently,because edges usually collaborate with each other, forming a community, to make prediction.
Revised: It is well known that the effect of an edge cannot be judged independently,because edges usually collaborate with each other, forming a community, to make prediction.

Original: Thus,it is critical to evaluate an edge in a global view.
Revised: Thus,it is critical to evaluate an edge in a global view.

Original: In whole graph population level, the mask generatortakes a global view of all the graphs in the training set, which enables us to identify causal/biassubgraph.
Revised: In whole graph population level, the mask generatortakes a global view of all the graphs in the training set, which enables us to identify causal/biassubgraph.

Original: Particularly, as the causal/bias is the statistical information in the population level, itis necessary to view all the graphs to identify the causal/bias substructure.
Revised: Particularly, as the causal/bias is the statistical information in the population level, itis necessary to view all the graphs to identify the causal/bias substructure.

Original: Considering both suchcoalition effects and population-level statistical information, the generator is able to measure theimportance of edges more accurately.
Revised: Considering both suchcoalition effects and population-level statistical information, the generator is able to measure the importance of edges more accurately.

Original: (2) Generalization : The mask generator can generalize themechanism of mask generation to new graphs without retraining, so it is capable and efficient toprune unseen graphs.
Revised: (2) Generalization : The mask generator can generalize themechanism of mask generation to new graphs without retraining, so it is capable and efficient toprune unseen graphs.

Original: 4.2 Learning Disentangled Graph Representations
Revised: 4.2 Learning Disentangled Graph Representations

Original: Given G c and G b , how to ensure they are causal subgraph and bias subgraph, respectively?
Revised: Given G c and G b , how to ensure they are causal subgraph and bias subgraph, respectively?

Original: Inspiredby [20], our approach simultaneously trains a pair of GNNs ( g b , g c ) with linear classifiers ( C b , C c )as follows: (1) Motivated by the observation in Sec.
Revised: Inspiredby [19], our approach simultaneously trains a pair of GNNs ( g b , g c ) with linear classifiers ( C b , C c )as follows: (1) Motivated by the observation in Sec.

Original: 3.1 that bias substructure is easier to learn, weutilize a bias-aware loss to train a bias GNN g b and a bias classifier C b and (2) in contrast, we train acausal GNN g c and a causal classifier C c on the training graphs that the bias GNN struggles to learn.
Revised: 3.1 that bias substructure is easier to learn, weutilize a bias-aware loss to train a bias GNN g b and a bias classifier C b and (2) in contrast, we train acausal GNN g c and a causal classifier C c on the training graphs that the bias GNN struggles to learn.

Original: Next, we would present each component in detail.
Revised: Next, we would present each component in detail.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, GNN g c and g b embed the corresponding subgraphs into causal embeddingz c
Revised: 3, GNN g c and g b embed the corresponding subgraphs into causal embeddingz c

Original: = g c ( G c ; γ c ) and bias embedding z b = g b ( G b ; γ b ) , respectively, where γ is the parameters of
Revised: = g c ( G c ; γ c ) and bias embedding z b = g b ( G b ; γ b ) , respectively, where γ is the parameters of

Original: GNNs.
Revised: GNNs.

Original: Subsequently, concatenated vector z =
Revised: Subsequently, concatenated vector z =

Original: [ z c ; z b ] is fed into linear classifiers C c and C b topredict the target label y .
Revised: [ z c ; z b ] is fed into linear classifiers C c and C b topredict the target label y .

Original: To train g b and C b as bias extractor, we utilize the generalized cross entropy(GCE) [44] loss to amplify the bias of the bias GNN and classifier:
Revised: To train g b and C b as bias extractor, we utilize the generalized cross entropy(GCE) [45] loss to amplify the bias of the bias GNN and classifier:

Original: [Equation]
Revised: [Equation]

Original: where C b ( z ; α b ) and C yb ( z ; α b ) are softmax output of the bias classifier and its probability belongingto the target category y , respectively, and α is the parameters of classifier.
Revised: where C b ( z ; α b ) and C yb ( z ; α b ) are softmax output of the bias classifier and its probability belongingto the target category y , respectively, and α is the parameters of classifier.

Original: Here q ∈ ( 0 , 1 ] is ahyperparameter that controls the degree of amplifying bias.
Revised: Here q ∈ ( 0 , 1 ] is ahyperparameter that controls the degree of amplifying bias.

Original: Given θ b =
Revised: Given θ b =

Original: [ γ b , α b ] , the gradient of the
Revised: [ γ b , α b ] , the gradient of the

Original: GCE loss up-weights the gradient of the standard cross entropy (CE) loss for the samples with a highconfidence C yb of predicting the correct target category as follows:
Revised: GCE loss up-weights the gradient of the standard cross entropy (CE) loss for the samples with a highconfidence C yb of predicting the correct target category as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, compared with CE loss, GCE loss will amplify the gradients of θ b on samples by theconfidence score ( C yb ) q .
Revised: Therefore, compared with CE loss, GCE loss will amplify the gradients of θ b on samples by theconfidence score ( C yb ) q .

Original: Based on our observation that the bias information is usually easier to belearned, so the biased graphs will have higher ( C yb )
Revised: Based on our observation that the bias information is usually easier to belearned, so the biased graphs will have higher ( C yb )

Original: q than unbiased graphs.
Revised: q than unbiased graphs.

Original: Therefore, the modelg b and C b trained by GCE loss will focus on bias information and finally get the bias subgraph.
Revised: Therefore, the modelg b and C b trained by GCE loss will focus on bias information and finally get the bias subgraph.

Original: Note that, to ensure that C b predicts target labels mainly based on this z b , the loss from C b is notbackpropagated to g c , i.e., only update θ b in Eq.
Revised: Note that, to ensure that C b predicts target labels mainly based on this z b , the loss from C b is notbackpropagated to g c , i.e., only update θ b in Eq.

Original: (4), and vice versa.
Revised: (4), and vice versa.

Original: Meanwhile, we also train a causal GNN simultaneously with the weighted CE loss.
Revised: Meanwhile, we also train a causal GNN simultaneously with the weighted CE loss.

Original: The graphs withhigh CE loss from C b can be regarded as the unbiased samples compared with the samples with low
Revised: The graphs withhigh CE loss from C b can be regarded as the unbiased samples compared with the samples with low

Original: CE loss.
Revised: CE loss.

Original: In this regard, we could obtain the unbias score of each graph as
Revised: In this regard, we could obtain the unbias score of each graph as

Original: [Equation]
Revised: [Equation]

Original: Large value of W implies the graph is an unbiased sample, hence we could use these weights toreweight the loss of these graphs to train g c and C c , enforcing them to learn the unbiased information.
Revised: Large value of W implies the graph is an unbiased sample, hence we could use these weights toreweight the loss of these graphs to train g c and C c , enforcing them to learn the unbiased information.

Original: Thus, the objective function for learning disentangled representation is:
Revised: Thus, the objective function for learning disentangled representation is:

Original: [Equation]
Revised: [Equation]

Original: 4.3 Counterfactual Unbiased Sample Generation
Revised: 4.3 Counterfactual Unbiased Sample Generation

Original: Until now, we have achieved the first goal analyzed in Sec. 3.2 that is the disentanglement ofcausal and bias substructures.
Revised: Until now, we have achieved the first goal analyzed in Sec. 3.2 that is the disentanglement ofcausal and bias substructures.

Original: Next, we will show how to achieve the second goal that makes thecausal variable z c and bias variable z b uncorrelated.
Revised: Next, we will show how to achieve the second goal that makes thecausal variable z c and bias variable z b uncorrelated.

Original: Although we have disentangled causal and biasinformation, they are disentangled from the biased observed graphs.
Revised: Although we have disentangled causal and biasinformation, they are disentangled from the biased observed graphs.

Original: Hence, there will exist statisticalcorrelation between causal and bias variables inheriting from the biased observed graphs.
Revised: Hence, there will exist statisticalcorrelation between causal and bias variables inheriting from the biased observed graphs.

Original: To furtherdecorrelate z c and z b , according to the causal relation of data-generating process: C → G ← B , wepropose to generate the counterfactual unbiased samples in embedding space by swapping z b .
Revised: To furtherdecorrelate z c and z b , according to the causal relation of data-generating process: C → G ← B , wepropose to generate the counterfactual unbiased samples in embedding space by swapping z b .

Original: Morespecifically, we randomly permute bias vectors in each mini-batch and obtain z unbiased =
Revised: Morespecifically, we randomly permute bias vectors in each mini-batch and obtain z unbiased =

Original: [ z c ; ˆ z b ] ,where ˆ z b represents the randomly permuted bias vectors of z b .
Revised: [ z c ; ˆ z b ] ,where ˆ z b represents the randomly permuted bias vectors of z b .

Original: As z c and ˆ z b in z unbiased are randomlycombined from different graphs, they will have much less correlation than z =
Revised: As z c and ˆ z b in z unbiased are randomly combined from different graphs, they will have much less correlation than z =

Original: [ z c ; z b ] where bothare from the same graph.
Revised: [ z c ; z b ] where bothare from the same graph.

Original: To make g b and C b still focus on the bias information, we also swap label yas ˆ y along with ˆ z b , so that the spurious correlation between ˆ z b and ˆ y still exists.
Revised: To make g b and C b still focus on the bias information, we also swap label yas ˆ y along with ˆ z b , so that the spurious correlation between ˆ z b and ˆ y still exists.

Original: With the generatedunbiased samples, we utilize the following loss function to train two GNN modules:
Revised: With the generatedunbiased samples, we utilize the following loss function to train two GNN modules:

Original: [Equation]
Revised: [Equation]

Original: Together with the disentanglement loss, total loss function is defined as:
Revised: Together with the disentanglement loss, total loss function is defined as:

Original: [Equation]
Revised: [Equation]

Original: (8) where λ G is a hyperparameter for weighting the importance of generation component.
Revised: where λ G is a hyperparameter for weighting the importance of generation component.

Original: Moreover,training with more diverse samples would also benefit with better generalization on unseen testingscenarios.
Revised: Moreover,training with more diverse samples would also benefit with better generalization on unseen testingscenarios.

Original: Our approach is summarized in App.
Revised: Our approach is summarized in App.

Original: B. Note that, as we need well-disentangled representations to generate the high-quality unbiased samples, in the early stage of training, we only train themodel with L D .
Revised: B. Note that, as we need well-disentangled representations to generate the high-quality unbiased samples, in the early stage of training, we only train themodel with L D .

Original: After certain epochs, we train the model with L .
Revised: After certain epochs, we train the model with L .

Original:  Experiment
Revised:  Experiment

Original: Datasets.
Revised: Datasets.

Original: We construct three datasets with various properties and bias ratios to benchmark thisnew problem, where the datasets have clear causal subgraphs making the results explainable 2 .
Revised: We construct three datasets with various properties and bias ratios to benchmark thisnew problem, where the datasets have clear causal subgraphs making the results explainable.

Original: Following Colored MNIST-75sp introduced in Sec.
Revised: Following Colored MNIST-75sp introduced in Sec.

Original: 3.1, we use the similar way to construct Colored
Revised: 3.1, we use the similar way to construct Colored

Original: Fashion-75sp and Colored Kuzushiji-75sp datasets based on the Fashion-MNIST [38] and KuzushijiMNIST [4] datasets.
Revised: Fashion-75sp and Colored Kuzushiji-75sp datasets based on the Fashion-MNIST [38] and KuzushijiMNIST [4] datasets.

Original: As the causal subgraphs of these two datasets are more complicated (fashionproduct and hiragana characters), they are more challenging.
Revised: As the causal subgraphs of these two datasets are more complicated (fashionproduct and hiragana characters), they are more challenging.

Original: Due to the page limits, here we set biasdegrees as { 0 . 8 , 0 . 9 , 0 . 95 } .
Revised: Due to the page limits, here we set biasdegrees as { 0 . 8 , 0 . 9 , 0 . 95 } .

Original: We report the main results on unbiased test sets.
Revised: We report the main results on unbiased test sets.

Original: Details are in App.
Revised: Details are in App.

Original: C.1.
Revised: C.1.

Original: Baselines and experimental setup.
Revised: Baselines and experimental setup.

Original: As DisC is a general framework which could be built on variousbase GNN models, we select three popular GNNs: GCN [16], GIN [39], and GCNII [3].
Revised: As DisC is a general framework which could be built on variousbase GNN models, we select three popular GNNs: GCN [15], GIN [39], and GCNII [3].

Original: Thecorresponding models are termed as DisC GCN , DisC GIN and DisC GCNII , respectively.
Revised: Thecorresponding models are termed as DisC GCN , DisC GIN and DisC GCNII , respectively.

Original: Hence, basemodels are the most straight baselines.
Revised: Hence, basemodels are the most straight baselines.

Original: Another baseline is the causal-inspired GNN method DIR [37].
Revised: Another baseline is the causal-inspired GNN method DIR [37].

Original: We also compare against a general debiasing method LDD [20] by replacing its encoder with GNNs.
Revised: We also compare against a general debiasing method LDD [19] by replacing its encoder with GNNs.

Revised: Graph Pooling method DiffPool [42] and graph disentangling method FactorGCN [40] are alsocompared.

Original: To keep fair comparison, our model uses the same GNN architecture and hyperparameters with thecorresponding base model.
Revised: To keep fair comparison, our model uses the same GNN architecture and hyperparameterswith the corresponding base model.

Original: All the experiments are run 4 times with different random seeds and wereport the accuracy and the standard error.
Revised: All the experiments are run 4 times with different random seedsand we report the accuracy and the standard error.

Original: More detailsabout hyperparameters are in App.
Revised: More details are in App.

Original: C.2.
Revised: C.2.

Original: 5.1 Quantitative Evaluation
Revised: 5.1 Quantitative Evaluation

Original: Main results.
Revised: Main results.

Original: The overall results are summarized in Table 1, and we have following observations:
Revised: The overall results are summarized in Table 1, and we have following observations:

Original: (1) DisC has much better generalization ability than base models.
Revised: (1) DisC has much better generalization ability than base models.

Original: DisC outperforms the corresponding base model consistently with a large margin.
Revised: DisC outperforms the correspond- ing base model consistently with a large margin.

Original: With heavier biases, our model achieves largerimprovements over base models.
Revised: With heavier biases, our model achieves larger improvements over base models.

Original: Specifically, for Colored MNIST-75sp, Colored Fashion-75spand Colored Kuzushiji-75sp with smaller bias degree ( i
Revised: Specifically, for Colored MNIST-75sp, Colored Fashion-75sp and Colored Kuzushiji-75sp with smaller bias degree ( i

Original: e., 0.8), our model achieves 40.02%,4.47% and 29.82% average improvements over corresponding base models, respectively.
Revised: e., 0.8), our model achieves 40.02%, 4.47% and 29.82% average improvements over corresponding base models, respectively.

Original: Surprisingly, with severer biases (0.9 and 0.95), DisC achieves 169.17%, 14.67% and 49.35% averageimprovements over base models on three datasets, respectively.
Revised: Surprisingly, with severer biases (0.9 and 0.95), DisC achieves 169.17%, 14.67% and 49.35% averageimprovements over base models on three datasets, respectively.

Original: It indicates that the proposed
Revised: It indicates that the proposed

Original: method is a general framework helping existing GNNs against the negative impact of bias.
Revised: method is a general framework helping existing GNNs against the negative impact of bias.

Original: (2) DisC significantly outperforms existing debiasing methods.
Revised: (2) DisC significantly outperforms existing debiasing methods.

Original: We notice that DIR could not achievesatisfying results.
Revised: We notice that DIR could not achieve satisfying results.

Original: The reason is that DIR utilizes CE loss to extract bias information, whichcould not fully capture the property of bias in severe bias scenarios.
Revised: The reason is that DIR utilizes CE loss to extract bias information, which could not fully capture the property of bias in severe bias scenarios.

Original: Moreover, LDD is a generaldebiasing method which is not designed for graph data.
Revised: Moreover, LDD is a general debiasing method which is not designed for graph data

Original: DisC outperforms corresponding LDDvariants with average 23.15%, indicating that the seamless joint of global-population-aware edgemasker with debiasing disentangle framework is very effective.
Revised: DisC outperforms corresponding LDDvariants with average 23.15%, indicating that the seamless joint of global-population-aware edgemasker with debiasing disentangle framework is very effective.

Original: Ablation studies.
Revised: Ablation studies.

Original: To validate the importance of each module in our method, in Fig.
Revised: To validate the importance of each module in our method, in Fig.

Original: 4, we conductablation studies on our variants (w.o. G means without the sample generation module) and therelated variants of LDD.
Revised: 4, we conductablation studies on our variants (w.o. G means without the sample generation module) and therelated variants of LDD.

Original: The major difference between DisC/w.o. G with LDD /w.o.
Revised: The major difference between DisC/w.o. G with LDD /w.o.

Original: G is the edgemask module.
Revised: G is the edge [Table] mask module.

Original: In most cases, DisC/w.o. G significantly outperforms LDD /w.o.
Revised: In most cases, DisC/w.o. G significantly outperforms LDD /w.o.

Original: G, indicating thenecessity of learning edge mask for graph data.
Revised: G, indicating thenecessity of learning edge mask for graph data.

Original: And DisC which has counterfactual sample generationmodule could further boost the performances based on the disentangled embeddings of DisC/w.o. G.
Revised: And DisC which has counterfactual sample generationmodule could further boost the performances based on the disentangled embeddings of DisC/w.o. G.

Original: However, LDD seldomly outperforms LDD /w.o.
Revised: However, LDD seldomly outperforms LDD /w.o.

Original: G or even achieves worse performances.
Revised: G or even achieves worse performances.

Original: That is,generating high-quality counterfactual samples needs well-disentangled causal and bias embeddings.
Revised: That is,generating high-quality counterfactual samples needs well-disentangled causal and bias embeddings.

Original: If embeddings are not well-disentangled, counterfactual samples may act as noisy samples, which [Table] [Figure] [Table]
Revised: If embeddings are not well-disentangled, counterfactual samples may act as noisy samples, whichwould prevent models from achieving further improvement.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: would prevent models from achieving further improvement.

Original: The edge masker could help the modelgenerate well-disentangled embeddings, which is crucial for overall performance.
Revised: The edge masker could help the modelgenerate well-disentangled embeddings, which is crucial for overall performance.

Original: [Figure]
Revised: [Figure]

Original: Robustness on unseen bias.
Revised: Robustness on unseen bias.

Original: Table 2 reports the results of DisC compared with its correspondingbase models on testing set with unseen bias, i.e., the pre-defined color (bias) sets of training set andtesting set are disjoint.
Revised: Table 2 reports the results of DisC compared with its correspondingbase models on testing set with unseen bias, i.e., the pre-defined color (bias) sets of training set and testing set are disjoint.

Original: The performances of base models further drop compared with the resultson seen bias scenario in Table 1.
Revised: The performances of base models further drop compared with the resultson seen bias scenario in Table 1.

Original: However, our model still achieves very stable performances, fullydemonstrating the generalization ability of our model on agnostic bias scenario.
Revised: However, our model still achieves very stable performances, fullydemonstrating the generalization ability of our model on agnostic bias scenario.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: 5.2 Qualitative Evaluation
Revised: 5.2 Qualitative Evaluation

Original: Visualization of edge mask.
Revised: Visualization of edge mask.

Original: To better illustrate the significant causal and bias subgraphs extractedby DisC GCN , we visualize the original images, original graph, and corresponding causal subgraphand bias subgraph of Colored MNIST-75sp with 0.9 bias degree in Fig. 
Revised: To better illustrate the significant causal and bias subgraphs extractedby DisC GCN , we visualize the original images, original graph [Table] and corresponding causal subgraphand bias subgraph of Colored MNIST-75sp with 0.9 bias degree in Fig. [Table]

Original: 5, where the width of edgerepresents the value of learned weight c ij or b ij .
Revised: where the width of edgerepresents the value of learned weight c ij or b ij .

Original: Fig.
Revised: Fig.

Original: 5(a) shows the visualization results of testinggraphs with the bias (color) that has been seen in the training set.
Revised: 5(a) shows the visualization results of testinggraphs with the bias (color) that has been seen in the training set.

Original: As we can see, our model coulddiscover the causal subgraphs where the most salient edges are in the digital subgraphs.
Revised: As we can see, our model coulddiscover the causal subgraphs where the most salient edges are in the digital subgraphs.

Original: With thesecausal subgraphs that highlight the structure information of digital, the GNNs will more easily extractthis causal information.
Revised: With thesecausal subgraphs that highlight the structure information of digital, the GNNs will more easily extractthis causal information.

Original: Fig.
Revised: Fig.

Original: 5(b) shows the visualization results of testing graphs with unseen bias.
Revised: 5(b) shows the visualization results of testing graphs with unseen bias.

Original: According to the visualization, our model could still discover the causal subgraph outline, indicatingour model could recognize causal subgraphs, whether the bias is seen or unseen.
Revised: According to the visualization, our model could still discover the causal subgraph outline, indicating our model could recognize causal subgraphs, whether the bias is seen or unseen.

Original: The visualizationresults of Colored Fashion-75sp and Colored Kuzushiji-75sp are shown in App.
Revised: The visualizationresults of Colored Fashion-75sp and Colored Kuzushiji-75sp are shown in App.

Original: D.
Revised: D.

Revised: [Figure]

Original: Projection of disentangled representation.
Revised: Projection of disentangled representation.

Original: Fig.
Revised: Fig.

Original: 6 shows the projection of latent vectors z c and z bextracted from the causal GNN g c and bias GNN g b of DisC GCN , respectively, using t-SNE [18] on
Revised: 6 shows the projection of latent vectors z c and z bextracted from the causal GNN g c and bias GNN g b of DisC GCN , respectively, using t-SNE [17] on

Original: Colored MNIST-75sp.
Revised: Colored MNIST-75sp.

Original: Fig.
Revised: Fig.

Original: 6 (a-b) are the projections of z c labeled by the target labels (digit) andbias labels (color), respectively.
Revised: 6 (a-b) are the projections of z c labeled by the target labels (digit) andbias labels (color), respectively.

Original: Fig.
Revised: Fig.

Original: 6 (c-d) are the projections of z b labeled by the target labels andbias labels, respectively.
Revised: 6 (c-d) are the projections of z b labeled by the target labels andbias labels, respectively.

Original: We observe that z c are clustered according to the target labels while z b are
Revised: We observe that z c are clustered according to the target labels while z b areclustered with the bias labels.

Original: [Figure]
Revised: And z c are mixed with bias labels and z b are mixed with target labels.

Original: The results indicate that DisC successfully learns the disentangled causal and bias representations.
Revised: The results indicate that DisC successfully learns the disentangled causal and bias representations.

Original: Transferability of the learned mask.
Revised: Transferability of the learned mask.

Original: As our model could extract
Revised: As our model could extract

Original: GNN-independent subgraphs, the learning edge weights can be usedto purify original biased graphs.
Revised: GNN-independent subgraphs, the learning edge weights can be usedto purify original biased graphs.

Original: These sparse subgraphs represent significant semantic information and can be universally transferred to any
Revised: These sparse subgraphs represent significant semantic information and can be universally transferred to any

Original: GNNs.
Revised: GNNs.

Original: To validate this point, we learn the edge mask by DisC GCNand prune the edges with least { 0% , 20% , 40% , 60% } weights whilekeeping the remaining edge weights.
Revised: To validate this point, we learn the edge mask by DisC GCNand prune the edges with least { 0% , 20% , 40% , 60% } weights whilekeeping the remaining edge weights.

Original: Then we train vanilla GIN and
Revised: Then we train vanilla GIN and

Original: GCNII on these weighted pruned datasets.
Revised: GCNII on these weighted pruned datasets.

Original: Fig.
Revised: Fig.

Original: 7 is the comparison ofthe results, where the dashed lines represent the results of base modelon original biased graphs and the solid lines represent the performance of GNNs on weighted pruned datasets.
Revised: 7 is the comparison ofthe results, where the dashed lines represent the results of base modelon original biased graphs and the solid lines represent the performance of GNNs on weighted pruned datasets.

Original: The results show thatthe GNNs trained on the pruned datasets achieve better performances,indicating our learned edge mask has considerable transferability.
Revised: The results show thatthe GNNs trained on the pruned datasets achieve better performances,indicating our learned edge mask has considerable transferability.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we are first to study the generalization problem of GNNs on severe bias datasets, whichis crucial to study the transparently knowledge learning mechanism of GNNs.
Revised: In this paper, we are first to study the generalization problem of GNNs on severe bias datasets, whichis crucial to study the transparently knowledge learning mechanism of GNNs.

Original: We analyze the problemin a causal view that the generalization of GNNs will be hindered by entangled representations aswell as the correlation between causal and bias variables.
Revised: We analyze the problemin a causal view that the generalization of GNNs will be hindered by entangled representations aswell as the correlation between causal and bias variables.

Original: To remove the impact from these twoaspects, we propose a general disentangling framework, DisC, which extracts causal substructureand bias substructure by two different functional GNNs, respectively.
Revised: To remove the impact from these twoaspects, we propose a general disentangling framework, DisC, which extracts causal substructureand bias substructure by two different functional GNNs, respectively.

Original: After the representations arewell-disentangled, we proliferate the counterfactual unbiased samples by randomly swapping thedisentangled vectors.
Revised: After the representations arewell-disentangled, we proliferate the counterfactual unbiased samples by randomly swapping thedisentangled vectors.

Original: With the new constructed benchmarks, we clearly validate the effectiveness,robustness, interpretability, and transferability of our method.
Revised: With the new constructed benchmarks, we clearly validate the effectiveness,robustness, interpretability, and transferability of our method.

Original: Limitations and societal impacts.

Original: Our method assumes that the graph consists of causal subgraphand bias subgraph.

Original: In reality, it may also exist non-informative subgraphs, which is neither causalnor biased for label.

Original: We would like to consider more fine-grained splitting of graphs in the future.

Original: When deploying GNNs to real-world applications, especially safety-critical fields, whether theresults of GNNs are stable is an important factor.

Original: The demands for a stable model are universal andextensive such as in the field of disease prediction [31], traffic states prediction [5], and financialapplications [35], where utilizing human-understandable causal knowledge for prediction is necessary.

Original: Checklist
Revised: Checklist

Original: For all authors
Revised: For all authors

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Sec. 6.
Revised: [Yes] See Sec. 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: No]
Revised: No]

Original: We could
Revised: We could

Original: not find any potential negative societal impacts of our work.
Revised: not forsee any potential negative societal impacts of our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sec. 3.2.
Revised: [Yes] See Sec. 3.2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide the
Revised: [Yes] We provide the

Original: URL of code and data for reproducing the main results.
Revised: URL of code and data for reproducing the main results.

Original: See Sec. 5.
Revised: See Sec. 5.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Sec.
Revised: [Yes] See Sec.

Original: 5 and App.
Revised: 5 and App.

Original: C.
Revised: C.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See Sec. 5.
Revised: [Yes] See Sec. 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Sec. 5.
Revised: [Yes] See Sec. 5.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We construct new
Revised: [Yes] We construct new

Original: assets based on existing assets.
Revised: assets based on existing assets.

Original: We have cited them in Sec. 5.
Revised: We have cited them in Sec. 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See App.
Revised: [Yes] See App.

Original: C.
Revised: C.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: We release new assets through a URL in Sec. 5.
Revised: We release new assets through a URL in Sec. 5.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [No] The source data for generating our data is publicly available.
Revised: [No] The source data for generating our data is publicly available.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifi-
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifi-

Original: able information or offensive content?
Revised: able information or offensive content?

Original: [No] We do not have personally identifiable
Revised: [No] We do not have personally identifiable

Original: information or offensive content.
Revised: information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]