Original: D EEP G RAPH N EURAL N ETWORKS WITH S HALLOW S UBGRAPH S AMPLERS
Revised: D EEP G RAPH N EURAL N ETWORKS WITH S HALLOW S UBGRAPH S AMPLERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While Graph Neural Networks (GNNs) are powerful models for learning representations on graphs, most state-of-the-art models do not have signiﬁcant accuracy gain beyond two to three layers.
Revised: While Graph Neural Networks (GNNs) are powerful models for learning representations on graphs, most state-of-the-art models do not have signiﬁcant accuracy gain beyond two to three layers.

Original: Deep GNNs fundamentally need to address: 1).
Revised: Deep GNNs fundamentally need to address: 1).

Original: expressivity challenge due to oversmoothing, and 2).
Revised: expressivity challenge due to oversmoothing, and 2).

Original: computation challenge due to neighborhood explosion.
Revised: computation challenge due to neighborhood explosion.

Original: We propose a simple “deep GNN, shallow sampler” design principle to improve both the GNN accuracy and efﬁciency — to generate representation of a target node, we use a deep GNN to pass messages only within a shallow, localized subgraph.
Revised: We propose a simple “deep GNN, shallow sampler” design principle to improve both the GNN accuracy and efﬁciency — to generate representation of a target node, we use a deep GNN to pass messages only within a shallow, localized subgraph.

Original: A properly sampled subgraph may exclude irrelevant or even noisy nodes, and still preserve the critical neighbor features and graph structures.
Revised: A properly sampled subgraph may exclude irrelevant or even noisy nodes, and still preserve the critical neighbor features and graph structures.

Original: The deep GNN then smooths the informative local signals to enhance feature learning, rather than oversmoothing the global graph signals into just “white noise”.
Revised: The deep GNN then smooths the informative local signals to enhance feature learning, rather than oversmoothing the global graph signals into just “white noise”.

Original: We theoretically justify why the combination of deep GNNs with shallow samplers yields the best learning performance.
Revised: We theoretically justify why the combination of deep GNNs with shallow samplers yields the best learning performance.

Original: We then propose various sampling algorithms and neural architecture extensions to achieve good empirical results.
Revised: We then propose various sampling algorithms and neural architecture extensions to achieve good empirical results.

Original: Experiments on ﬁve large graphs show that our models achieve signiﬁcantly higher accuracy and efﬁciency, compared with state-of-the-art.
Revised: Experiments on ﬁve large graphs show that our models achieve signiﬁcantly higher accuracy and efﬁciency, compared with state-of-the-art.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) have now become the state-of-the-art models for graph mining (Wu et al., 2020; Hamilton et al., 2017b; Zhang et al., 2019), facilitating applications such as social recommendation (Monti et al., 2017;
Revised: Graph Neural Networks (GNNs) have now become the state-of-the-art models for graph mining (Wu et al., 2020; Hamilton et al., 2017b; Zhang et al., 2019), facilitating applications such as social recommendation (Monti et al., 2017;

Original: Ying et al., 2018;
Revised: Ying et al., 2018;

Original: Pal et al., 2020), knowledge understanding (Schlichtkrull et al., 2018; Park et al., 2019; Zhang et al., 2020) and drug discovery (Stokes et al., 2020;
Revised: Pal et al., 2020), knowledge understanding (Schlichtkrull et al., 2018; Park et al., 2019; Zhang et al., 2020) and drug discovery (Stokes et al., 2020;

Original: Lo et al., 2018).
Revised: Lo et al., 2018).

Original: With the numerous architectures proposed (Kipf & Welling, 2016; Hamilton et al., 2017a; Veliˇckovi´c et al., 2018), it still remains an open question how to effectively design deep
Revised: With the numerous architectures proposed (Kipf & Welling, 2016; Hamilton et al., 2017a; Veliˇckovi´c et al., 2018), it still remains an open question how to effectively design deep

Original: GNNs.
Revised: GNNs.

Original: There are two fundamental obstacles that are intrinsic to the underlying graph structure:
Revised: There are two fundamental obstacles that are intrinsic to the underlying graph structure:

Original: • Expressivity challenge: deep GNNs tend to oversmooth (Li et al., 2018).
Revised: • Expressivity challenge: deep GNNs tend to oversmooth (Li et al., 2018).

Original: They collapse embeddings of different nodes into a ﬁxed low-dimensional subspace after repeated neighbor mixing.
Revised: They collapse embeddings of different nodes into a ﬁxed low-dimensional subspace after repeated neighbor mixing.

Original: • Computation challenge: deep GNNs recursively expand the adjacent nodes along message passing edges.
Revised: • Computation challenge: deep GNNs recursively expand the adjacent nodes along message passing edges.

Original: The neighborhood size may grow exponentially with model depth (Chen et al., 2017).
Revised: The neighborhood size may grow exponentially with model depth (Chen et al., 2017).

Original: Due to oversmoothing, one of the most popular GNN architectures, Graph Convolutional Network (GCN) (Kipf & Welling, 2016), has been theoretically proven as incapable of scaling to deep layers (Oono & Suzuki, 2020;
Revised: Due to oversmoothing, one of the most popular GNN architectures, Graph Convolutional Network (GCN) (Kipf & Welling, 2016), has been theoretically proven as incapable of scaling to deep layers (Oono & Suzuki, 2020;

Original: Rong et al., 2020; Huang et al., 2020).
Revised: Rong et al., 2020; Huang et al., 2020).

Original: Remedies to overcome the GCN limitations are two-folded.
Revised: Remedies to overcome the GCN limitations are two-folded.

Original: From the neural architecture perspective, researchers are actively seeking for more expressive neighbor aggregation operations (Veliˇckovi´c et al., 2018; Hamilton et al., 2017a;
Revised: From the neural architecture perspective, researchers are actively seeking for more expressive neighbor aggregation operations (Veliˇckovi´c et al., 2018; Hamilton et al., 2017a;

Original: Xu et al., 2018a), or transferring design components (such as residual connection) from deep CNNs to GNNs (Xu et al., 2018b; Li et al., 2019; Huang et al., 2018).
Revised: Xu et al., 2018a), or transferring design components (such as residual connection) from deep CNNs to GNNs (Xu et al., 2018b; Li et al., 2019; Huang et al., 2018).

Original: From the data perspective, various works (Klicpera et al., 2019a;b; Bojchevski et al., 2020) revisit classic graph analytic algorithms to reconstruct a graph with nicer topological property.
Revised: From the data perspective, various works (Klicpera et al., 2019a;b; Bojchevski et al., 2020) revisit classic graph analytic algorithms to reconstruct a graph with nicer topological property.

Original: The two kinds of works can also be combined to jointly improve the quality of message passing in deep GNNs.
Revised: The two kinds of works can also be combined to jointly improve the quality of message passing in deep GNNs.

Original: All the above GNN variants take a “global” view on the input graph G ( V , E ) — i.e. , all nodes are considered as belonging to the same G , whose size can often be massive.
Revised: All the above GNN variants take a “global” view on the input graph G ( V , E ) — i.e. , all nodes are considered as belonging to the same G , whose size can often be massive.

Original: To generate the node embedding, no matter how we modify the architecture and the graph structure, a deep enough GNN would always propagate the inﬂuence from the entire node set V into a single target node.
Revised: To generate the node embedding, no matter how we modify the architecture and the graph structure, a deep enough GNN would always propagate the inﬂuence from the entire node set V into a single target node.

Original: Intuitively, for a large graph, most nodes in V barely provide any useful information to the target nodes.
Revised: Intuitively, for a large graph, most nodes in V barely provide any useful information to the target nodes.

Original: We thus regard such “global view” on G as one of the root causes for both the expressivity and computation challenges discussed above.
Revised: We thus regard such “global view” on G as one of the root causes for both the expressivity and computation challenges discussed above.

Original: In this work, for the node embedding task, we take an alternative “local view” and interpret the GNN input as V =
Revised: In this work, for the node embedding task, we take an alternative “local view” and interpret the GNN input as V =

Original: (cid:83) v ∈V V [ v ] and E =
Revised: (cid:83) v ∈V V [ v ] and E =

Original: (cid:83) v ∈V E [ v ] .
Revised: (cid:83) v ∈V E [ v ] .

Original: In other words, each target node v belongs to some small graph G [ v ] capturing the characteristics of only the node v .
Revised: In other words, each target node v belongs to some small graph G [ v ] capturing the characteristics of only the node v .

Original: The entire input graph G is observed as the union of all such local yet latent G [ v ] .
Revised: The entire input graph G is observed as the union of all such local yet latent G [ v ] .

Original: Such simple global-tolocal switch of perspective enables us to address both the expressivity and computation challenges without resorting to alternative GNN architectures or reconstructing the graph.
Revised: Such simple global-tolocal switch of perspective enables us to address both the expressivity and computation challenges without resorting to alternative GNN architectures or reconstructing the graph.

Original: Present work: SHA D OW -GNN.
Revised: Present work: SHA D OW -GNN.

Original: We propose a “Deep GNN, shallow sampler” design principle that helps improve the expressive power and inference efﬁciency of various GNN architectures.
Revised: We propose a “Deep GNN, shallow sampler” design principle that helps improve the expressive power and inference efﬁciency of various GNN architectures.

Original: We break the conventional thinking that an L -layer (deep) GNN has to aggregate L -hop (faraway) neighbors.
Revised: We break the conventional thinking that an L -layer (deep) GNN has to aggregate L -hop (faraway) neighbors.

Original: We argue that the GNN receptive ﬁeld for a target node should be shallower than the GNN depth.
Revised: We argue that the GNN receptive ﬁeld for a target node should be shallower than the GNN depth.

Original: In other words, an L -layer GNN should only operate on a small subgraph G [ v ] surrounding the target node v , where G [ v ] consists of (part of) the L 0 -hop neighborhood.
Revised: In other words, an L -layer GNN should only operate on a small subgraph G [ v ] surrounding the target node v , where G [ v ] consists of (part of) the L 0 -hop neighborhood.

Original: The deep vs. shallow comparison is reﬂected by setting L 0 <
Revised: The deep vs. shallow comparison is reﬂected by setting L 0 <

Original: L .
Revised: L .

Original: We name such a GNN on G [ v ] as a SHA D OW -GNN.
Revised: We name such a GNN on G [ v ] as a SHA D OW -GNN.

Original: We justify our design principle from two aspects.
Revised: We justify our design principle from two aspects.

Original: Firstly, why do we need the neighborhood to be shallow ?
Revised: Firstly, why do we need the neighborhood to be shallow ?

Original: As a motivating example, the average number of 4-hop neighbors for the ogbn-products graph (Hu et al., 2020) is 0.6M, corresponding to 25% of the full graph size.
Revised: As a motivating example, the average number of 4-hop neighbors for the ogbn-products graph (Hu et al., 2020) is 0.6M, corresponding to 25% of the full graph size.

Original: Blindly encoding the 0.6M node features into a single embedding vector can create the “information bottleneck” (Alon & Yahav, 2020).
Revised: Blindly encoding the 0.6M node features into a single embedding vector can create the “information bottleneck” (Alon & Yahav, 2020).

Original: The irrelevant information from the majority of the 0.6M nodes may also “dilute” the truly useful signals from a small set of close neighbors.
Revised: The irrelevant information from the majority of the 0.6M nodes may also “dilute” the truly useful signals from a small set of close neighbors.

Original: A simple solution to the above issues is to manually create a shallow neighborhood by subgraph sampling.
Revised: A simple solution to the above issues is to manually create a shallow neighborhood by subgraph sampling.

Original: The second question regarding
Revised: The second question regarding

Original: SHA D OW -GNN is: why do we still need deep GNNs ?
Revised: SHA D OW -GNN is: why do we still need deep GNNs ?

Original: Using more number of layers than the number of hops means the same pair of nodes may exchange messages with each other multiple times.
Revised: Using more number of layers than the number of hops means the same pair of nodes may exchange messages with each other multiple times.

Original: Intuitively, this helps the GNN better absorb the subgraph information.
Revised: Intuitively, this helps the GNN better absorb the subgraph information.

Original: Theoretically, we prove that a
Revised: Theoretically, we prove that a

Original: GNN deeper than the hops of the subgraph can be more powerful than the 1-dimensional WeisfeilerLehman test (Shervashidze et al., 2011).
Revised: GNN deeper than the hops of the subgraph can be more powerful than the 1-dimensional WeisfeilerLehman test (Shervashidze et al., 2011).

Original: A shallow GNN, on the contrary, cannot accurately learn certain simple functions such as unweighted mean of the shallow neighborhood features.
Revised: A shallow GNN, on the contrary, cannot accurately learn certain simple functions such as unweighted mean of the shallow neighborhood features.

Original: Note that with GCN as the backbone, a SHA D OW -GCN still performs signal smoothing in each layer.
Revised: Note that with GCN as the backbone, a SHA D OW -GCN still performs signal smoothing in each layer.

Original: However, the important distinction is that a deep GCN smooths the full G regardless of the target node, while a SHA D OW -GCN constructs a customized smoothing domain G [ v ] for each target v .
Revised: However, the important distinction is that a deep GCN smooths the full G regardless of the target node, while a SHA D OW -GCN constructs a customized smoothing domain G [ v ] for each target v .

Original: The variance in those smoothing domains created by SHA D OW -GCN encourages variances in the node embedding vectors.
Revised: The variance in those smoothing domains created by SHA D OW -GCN encourages variances in the node embedding vectors.

Original: With such intuition, our analysis shows that SHA D OW -GNN does not oversmooth.
Revised: With such intuition, our analysis shows that SHA D OW -GNN does not oversmooth.

Original: Finally, since the sizes of the shallow neighborhoods are independent of the GNN depth, the computation challenge due to neighbor explosion is automatically addressed.
Revised: Finally, since the sizes of the shallow neighborhoods are independent of the GNN depth, the computation challenge due to neighbor explosion is automatically addressed.

Original: We propose various subgraph samplers for SHA D OW -GNN, including the simplest k -hop sampler and a sampler based on personalized PageRank, to improve the inference accuracy and computation efﬁciency.
Revised: We propose various subgraph samplers for SHA D OW -GNN, including the simplest k -hop sampler and a sampler based on personalized PageRank, to improve the inference accuracy and computation efﬁciency.

Original: By experiments on ﬁve standard benchmarks, our SHA D OW -SAGE and SHA D OW -GAT models achieve signiﬁcant accuracy gains compared with the original GraphSAGE and GAT models.
Revised: By experiments on ﬁve standard benchmarks, our SHA D OW -SAGE and SHA D OW -GAT models achieve signiﬁcant accuracy gains compared with the original GraphSAGE and GAT models.

Original: In the meantime, the inference cost is reduced by orders of magnitude.
Revised: In the meantime, the inference cost is reduced by orders of magnitude.

Original:  R ELATED W ORK AND P RELIMINARIES
Revised:  R ELATED W ORK AND P RELIMINARIES

Original: Deep GNNs.
Revised: Deep GNNs.

Original: Recently, numerous GNN models (Kipf & Welling, 2016; Defferrard et al., 2016;
Revised: Recently, numerous GNN models (Kipf & Welling, 2016; Defferrard et al., 2016;

Original: Hamilton et al., 2017a; Veliˇckovi´c et al., 2018; Xu et al., 2018b;a) have been proposed.
Revised: Hamilton et al., 2017a; Veliˇckovi´c et al., 2018; Xu et al., 2018b;a) have been proposed.

Original: In general, the input to a GNN is the graph G , and the outputs are representation vectors for each node, capturing both the feature and structural information of the neighborhood.
Revised: In general, the input to a GNN is the graph G , and the outputs are representation vectors for each node, capturing both the feature and structural information of the neighborhood.

Original: Most state-of-the-art GNNs use shallow models ( i.e. , 2 to 3 layers).
Revised: Most state-of-the-art GNNs use shallow models ( i.e. , 2 to 3 layers).

Original: As ﬁrst proposed by Li et al.
Revised: As ﬁrst proposed by Li et al.

Original: and further elaborated by Luan et al.
Revised: and further elaborated by Luan et al.

Original: Oono & Suzuki (2020); Zhao & Akoglu (2020); Huang et al.
Revised: Oono & Suzuki (2020); Zhao & Akoglu (2020); Huang et al.

Original: (2020), one of the major challenges to deepen GNNs is the “oversmoothing” of node features — each layer aggregation pushes the neighbor features towards similar values.
Revised: (2020), one of the major challenges to deepen GNNs is the “oversmoothing” of node features — each layer aggregation pushes the neighbor features towards similar values.

Original: Repeated aggregation over many layers results in node features being averaged over the full graph.
Revised: Repeated aggregation over many layers results in node features being averaged over the full graph.

Original: A deep GNN may thus generate indistinguishable embeddings for different nodes.
Revised: A deep GNN may thus generate indistinguishable embeddings for different nodes.

Original: Viewing oversmoothing as a limitation of the layer aggregation, researchers develop alternative architectures.
Revised: Viewing oversmoothing as a limitation of the layer aggregation, researchers develop alternative architectures.

Original: AS-GCN (Huang et al., 2018), DeepGCN (Li et al., 2019) and JK-net (Xu et al., 2018b) use skip-connection across layers.
Revised: AS-GCN (Huang et al., 2018), DeepGCN (Li et al., 2019) and JK-net (Xu et al., 2018b) use skip-connection across layers.

Original: MixHop (Abu-El-Haija et al., 2019), Snowball (Luan et al., 2019) and DAGNN (Liu et al., 2020) enable multi-hop message passing within a single layer.
Revised: MixHop (Abu-El-Haija et al., 2019), Snowball (Luan et al., 2019) and DAGNN (Liu et al., 2020) enable multi-hop message passing within a single layer.

Original: GraphSAGE (Hamilton et al., 2017a) and GCNII (Ming Chen et al., 2020) encourage self-to-self message passing which effectively form an implicit skip-connection.
Revised: GraphSAGE (Hamilton et al., 2017a) and GCNII (Ming Chen et al., 2020) encourage self-to-self message passing which effectively form an implicit skip-connection.

Original: GIN (Xu et al., 2018a) and DeeperGCN (Li et al., 2020a) propose more expressive neighbor aggregation operations.
Revised: GIN (Xu et al., 2018a) and DeeperGCN (Li et al., 2020a) propose more expressive neighbor aggregation operations.

Original: All the above focus on architectural exploration , which is a research direction orthogonal to ours.
Revised: All the above focus on architectural exploration , which is a research direction orthogonal to ours.

Original: We can construct the SHA D OW version of these GNNs in a plug-and-play fashion.
Revised: We can construct the SHA D OW version of these GNNs in a plug-and-play fashion.

Original: Lastly, DropEdge (Rong et al., 2020) and Bayesian-GDC (Hasanzadeh et al., 2020) propose regularization techniques by adapting dropout (Srivastava et al., 2014) to graphs.
Revised: Lastly, DropEdge (Rong et al., 2020) and Bayesian-GDC (Hasanzadeh et al., 2020) propose regularization techniques by adapting dropout (Srivastava et al., 2014) to graphs.

Original: Such techniques are only applied during training, and so oversmoothing during inference may not be alleviated.
Revised: Such techniques are only applied during training, and so oversmoothing during inference may not be alleviated.

Original: Learning from structural information.
Revised: Learning from structural information.

Original: Another line of research is to go beyond the layer-wise message passing and more explicitly utilize the graph structural information (Wu et al., 2019;
Revised: Another line of research is to go beyond the layer-wise message passing and more explicitly utilize the graph structural information (Wu et al., 2019;

Original: Klicpera et al., 2019a;
Revised: Klicpera et al., 2019a;

Original: Bojchevski et al., 2020; Liu et al., 2020; Frasca et al., 2020; You et al., 2019;
Revised: Bojchevski et al., 2020; Liu et al., 2020; Frasca et al., 2020; You et al., 2019;

Original: Li et al., 2020b).
Revised: Li et al., 2020b).

Original: In particular, APPNP (Klicpera et al., 2019a) and PPRGo (Bojchevski et al., 2020) utilize the personalized PageRank (Page et al., 1999) algorithm to re-deﬁne neighbor connections — instead of propagating features along the (noisy) graph edges, any nodes of structural signiﬁcance can directly propagate to the target node.
Revised: In particular, APPNP (Klicpera et al., 2019a) and PPRGo (Bojchevski et al., 2020) utilize the personalized PageRank (Page et al., 1999) algorithm to re-deﬁne neighbor connections — instead of propagating features along the (noisy) graph edges, any nodes of structural signiﬁcance can directly propagate to the target node.

Original: Other related methods such as GDC (Klicpera et al., 2019b) and AM-GCN (Wang et al., 2020) reconstructs the adjacency matrix in each GNN layer to short-cut important multi-hop neighbors.
Revised: Other related methods such as GDC (Klicpera et al., 2019b) and AM-GCN (Wang et al., 2020) reconstructs the adjacency matrix in each GNN layer to short-cut important multi-hop neighbors.

Original: Note that all the above methods takes a global view on
Revised: Note that all the above methods takes a global view on

Original: G and operate the neural networks on the full graph .
Revised: G and operate the neural networks on the full graph .

Original: On the other hand, the idea of using subgraph samples to improve the GNN efﬁciency has also been explored.
Revised: On the other hand, the idea of using subgraph samples to improve the GNN efﬁciency has also been explored.

Original: For example, SEAL (Zhang & Chen, 2018) extracts local k -hop enclosing subgraphsaround links to perform link prediction.
Revised: For example, SEAL (Zhang & Chen, 2018) extracts local k -hop enclosing subgraphs to perform link prediction.

Original: GraphSAINT (Zeng et al., 2020) propose random walk-based samplers to construct minibatches during training.
Revised: GraphSAINT (Zeng et al., 2020) propose random walk samplers to construct minibatches during training.

Original: Note that the training sampler of GraphSAINT aims at approximating the feature aggregation in the full L -hop neighborhood.

Original: So the sampling of GraphSAINT cannot be applied to the inference phase, leaving the issue of oversmoothing and neighbor exploration unaddressed.

Original: Notations.
Revised: Notations.

Original: We focus on the node classiﬁcation task, although our design principle can be naturally extended to other tasks.
Revised: We focus on the node classiﬁcation task, although our design principle can be naturally extended to other tasks.

Original: Let G ( V , E , X ) be an undirected graph, with node set V , edge set E ⊆ V×V and node feature matrix X ∈ R |V|× d .
Revised: Let G ( V , E , X ) be an undirected graph, with node set V , edge set E ⊆ V×V and node feature matrix X ∈ R |V|× d .

Original: The u -th row of X corresponds to the length-
Revised: The u -th row of X corresponds to the length-

Original: d feature of node u .
Revised: d feature of node u .

Original: Let A be the adjacency matrix of G where A u,v = 1 if edge ( u, v ) ∈ E and A u,v
Revised: Let A be the adjacency matrix of G where A u,v = 1 if edge ( u, v ) ∈ E and A u,v

Original: =otherwise.
Revised: =otherwise.

Original: Denote (cid:101)
Revised: Denote (cid:101)

Original: A as the adjacency matrix after symmetric normalization (used by GCN), and (cid:98)
Revised: A as the adjacency matrix after symmetric normalization (used by GCN), and (cid:98)

Original: A as the one after random walk normalization (used by GraphSAGE).
Revised: A as the one after random walk normalization (used by GraphSAGE).

Original: Let subscript “ [ u ] ” mark the quantities corresponding to a small subgraph surrounding node u .
Revised: Let subscript “ [ u ] ” mark the quantities corresponding to a small subgraph surrounding node u .

Original: For example, the subgraph itself is G [ u ] .
Revised: For example, the subgraph itself is G [ u ] .

Original: For an L -layer GNN, let superscript “ ( (cid:96) ) ” denote the layer- (cid:96) quantities ( 1 ≤ (cid:96) ≤ L ).
Revised: For an L -layer GNN, let superscript “ ( (cid:96) ) ” denote the layer- (cid:96) quantities ( 1 ≤ (cid:96) ≤ L ).

Original: Let d ( (cid:96) ) be the number of channels for layer (cid:96) ; H ( (cid:96) − 1) ∈
Revised: Let d ( (cid:96) ) be the number of channels for layer (cid:96) ; H ( (cid:96) − 1) ∈

Original: R |V|× d ( (cid:96) − 1) and H ( (cid:96) ) ∈
Revised: R |V|× d ( (cid:96) − 1) and H ( (cid:96) ) ∈

Original: R |V|× d ( (cid:96) ) be the input and output feature matrices.
Revised: R |V|× d ( (cid:96) ) be the input and output feature matrices.

Original: Thus, H (0)
Revised: Thus, H (0)

Original: =
Revised: =

Original: X and d (0) =
Revised: X and d (0) =

Original: d .
Revised: d .

Original: Further, let Y
Revised: Further, let Y

Original: = H ( L ) .
Revised: = H ( L ) .

Original: The operation of a layer can be abstracted as H ( (cid:96) )
Revised: The operation of a layer can be abstracted as H ( (cid:96) )

Original: =
Revised: =

Original: f (cid:0) H ( (cid:96) − 1) , A ; W ( (cid:96) ) (cid:1) , where W ( (cid:96) ) are the learnable weights.
Revised: f (cid:0) H ( (cid:96) − 1) , A ; W ( (cid:96) ) (cid:1) , where W ( (cid:96) ) are the learnable weights.

Original:  D EEP GNN, S HALLOW S AMPLER
Revised:  D EEP GNN, S HALLOW S AMPLER

Revised: Input: G (

Original: “Deep GNN, shallow sampler” is a design principle to improve the expressivity and efﬁciency of GNNs without modifying the layer architecture.
Revised: “Deep GNN, shallow sampler” is a design principle to improve the expressivity and efﬁciency of GNNs without modifying the layer architecture.

Original: Based on this principle, we construct SHA D OW -GNN by subgraph sampling.
Revised: Based on this principle, we construct SHA D OW -GNN by subgraph sampling.

Original: SHA D OW -GNN uses the same sampling procedure during both training and inference.
Revised: SHA D OW -GNN uses the same sampling procedure during both training and inference.

Original: The inference algorithm is shown in
Revised: The inference algorithm is shown in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: In line 4, we use f L to denote the composition of the function f for the L layers.
Revised: In line 4, we use f L to denote the composition of the function f for the L layers.

Original: Operation [ · ] v, : slices the v -th row of the matrix.
Revised: Operation [ · ] v, : slices the v -th row of the matrix.

Original: We discuss in Section 3.2 how to design SAMPLE to return informative shallow neighborhood.
Revised: We discuss in Section 3.2 how to design SAMPLE to return informative shallow neighborhood.

Original: Notice that the normal GNN is a special kind of SHA D OW -GNN.
Revised: Notice that the normal GNN is a special kind of SHA D OW -GNN.

Original: Under the normal setup, the
Revised: Under the normal setup, the

Original: GNN operates on the full graph G and the L layers propagate the inﬂuence from all the neighbors up to L hops away from the target node.
Revised: GNN operates on the full graph G and the L layers propagate the inﬂuence from all the neighbors up to L hops away from the target node.

Original: Such a GNN is equivalent to a SHA D OW -GNN when
Revised: Such a GNN is equivalent to a SHA D OW -GNN when

Original: SAMPLE returns the full L -hop subgraph.
Revised: SAMPLE returns the full L -hop subgraph.

Original: However, following our principle, a good SAMPLE should encourage most of V [ v ] to concentrate within the L 0 -hop neighborhood, where L 0 < L .
Revised: However, following our principle, a good SAMPLE should encourage most of V [ v ] to concentrate within the L 0 -hop neighborhood, where L 0 < L .

Original: Figures 5, in Appendix show the difference in the neighborhood composition for normal GNN and SHA D OWGNN.
Revised: Figures 6,in Appendix show the difference in the neighborhood composition for normal GNN and SHA D OWGNN.

Original: Section 3.1 discusses the beneﬁts of making the subgraph shallow and the GNN deep.
Revised: Section 3.1 discusses the beneﬁts of making the subgraph shallow and the GNN deep.

Original: 3. A NALYSIS ON E XPRESSIVITY
Revised: 3. A NALYSIS ON E XPRESSIVITY

Original: The SHA D OW -GNN design is motivated by the following:
Revised: The SHA D OW -GNN design is motivated by the following:

Original: A shallow neighborhood is sufﬁcient for the GNN to learn a good node representation; 2).
Revised: A shallow neighborhood is sufﬁcient for the GNN to learn a good node representation; 2).

Original: A shallow neighborhood is necessary to reduce the effect of noise on the GNN; 3).
Revised: A shallow neighborhood is necessary to reduce the effect of noise on the GNN; 3).

Original: A deep GNN is necessary to be expressive on the shallow neighborhood.
Revised: A deep GNN is necessary to be expressive on the shallow neighborhood.

Original: Point 1 is true in many real-world scenarios, and is also justiﬁed by the γ -decaying theorem (Zhang & Chen, 2018): to estimate various important graph metrics of a node from its L -hop neighborhood, the estimation error decays exponentially with the number of hops L .
Revised: Point 1 is true in many real-world scenarios, and is also justiﬁed by the γ -decaying theorem (Zhang & Chen, 2018): to estimate various important graph metrics of a node from its L -hop neighborhood, the estimation error decays exponentially with the number of hops L .

Original: The understanding on Point 2 is two-folded: real-world graphs are likely to include noisy nodes and edges due to erroneous graph construction (Klicpera et al., 2019b).
Revised: The understanding on Point 2 is two-folded: real-world graphs are likely to include noisy nodes and edges due to erroneous graph construction (Klicpera et al., 2019b).

Original: Additionally, even without errors, a node u may still be regarded as “white noise” to the target node v , simply because u is far away from v and u ’s information is irrelevant.
Revised: Additionally, even without errors, a node u may still be regarded as “white noise” to the target node v , simply because u is far away from v and u ’s information is irrelevant.

Original: The ﬁrst kind of noise may be handled by ﬁltering out the high frequency components of the node signals (Wu et al., 2019) or enforcing low rank constraints on the adjacency matrix (Grover et al., 2019; Jin et al., 2020).
Revised: The ﬁrst kind of noise may be handled by ﬁltering out the high frequency components of the node signals (Wu et al., 2019) or enforcing low rank constraints on the adjacency matrix (Grover et al., 2019; Jin et al., 2020).

Original: The second kind of noise cannot be ﬁltered as long as the GNN aggregates the full L -hop neighborhood.
Revised: The second kind of noise cannot be ﬁltered as long as the GNN aggregates the full L -hop neighborhood.

Original: In this regard, sampling provides an additional mechanism to deal with both kinds of noises.
Revised: In this regard, sampling provides an additional mechanism to deal with both kinds of noises.

Original: Also, since we customize different subgraphs for different target nodes, SAMPLE ﬁlters noise at the node level rather than the graph level.
Revised: Also, since we customize different subgraphs for different target nodes, SAMPLE ﬁlters noise at the node level rather than the graph level.

Original: For Point 3, while theoretical understanding on the beneﬁt of deep neural networks has been well established for Multi-Layer Perceptrons (MLPs) (Telgarsky, 2016), such conclusion does not directly transfer to GNNs.
Revised: For Point 3, while theoretical understanding on the beneﬁt of deep neural networks has been well established for Multi-Layer Perceptrons (MLPs) (Telgarsky, 2016), such conclusion does not directly transfer to GNNs.

Original: For SHA D OW -GNN, we prove a deeper model is more expressive than a shallower one.
Revised: For SHA D OW -GNN, we prove a deeper model is more expressive than a shallower one.

Original: Expressivity comparison with “deep GNN, deep sampler”.
Revised: Expressivity comparison with “deep GNN, deep sampler”.

Original: When increasing the depth of a normal GNN, the neighborhood around a target node (i.e., the full L -hop subgraph) also expands.
Revised: When increasing the depth of a normal GNN, the neighborhood around a target node (i.e., the full L -hop subgraph) also expands.

Original: Eventually, a deep normal GNN on a connected G will propagate inﬂuence of all V into a single target node.
Revised: Eventually, a deep normal GNN on a connected G will propagate inﬂuence of all V into a single target node.

Original: As discussed in Point 2 above, most V only act as “white noise” to a target node, thus causing difﬁculties in learning regardless of the layer propagation function.
Revised: As discussed in Point 2 above, most V only act as “white noise” to a target node, thus causing difﬁculties in learning regardless of the layer propagation function.

Original: We pick the GCN architecture for case study.
Revised: We pick the GCN architecture for case study.

Original: When the GCN depth grows, we analyze how a shallow sampler can ﬁlter out the “white noise” in the neighborhood and help preserve target node speciﬁc information.
Revised: When the GCN depth grows, we analyze how a shallow sampler can ﬁlter out the “white noise” in the neighborhood and help preserve target node speciﬁc information.

Original: Following the setup of Li et al.
Revised: Following the setup of Li et al.

Original: ; Luan et al.
Revised: ; Luan et al.

Original: ; Ming Chen et al.
Revised: ; Ming Chen et al.

Original: Zhao & Akoglu (2020), we study the asymptotic behavior of repeatedly applying the neighbor aggregation operation.
Revised: Zhao & Akoglu (2020), we study the asymptotic behavior of repeatedly applying the neighbor aggregation operation.

Original: i.e. , we analyze how much information is preserved in M =
Revised: i.e. , we analyze how much information is preserved in M =

Original: lim L →∞ (cid:101)
Revised: lim L →∞ (cid:101)

Original: A L X .
Revised: A L X .

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: An ∞ -layer SHA D OW -GCN generates the neighbor aggregation for v as:
Revised: An ∞ -layer SHA D OW -GCN generates the neighbor aggregation for v as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where e [ v ] is deﬁned by (cid:2)
Revised: where e [ v ] is deﬁned by (cid:2)

Original: e [ v ] (cid:3) u
Revised: e [ v ] (cid:3) u

Original: = (cid:114) 
Revised: = (cid:114) δ [

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: (cid:80)

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: [ v ] δ [ v ] ( w ) ; δ [ v ] ( u ) returns the degree of u in G [ v ] plus 1.
Revised: [ v ] δ [ v ] ( w ) ; δ [ v ] ( u ) returns the degree of u in G [ v ] plus 1.

Original: O VERSMOOTHING OF NORMAL GCN
Revised: O VERSMOOTHING OF NORMAL GCN

Original: With large enough L , the full L -hop neighborhood becomes V (assuming connected G ).
Revised: With large enough L , the full L -hop neighborhood becomes V (assuming connected G ).

Original: So ∀ u, v , we have G [ u ] =
Revised: So ∀ u, v , we have G [ u ] =

Original: G [ v ] =
Revised: G [ v ] =

Original: G , implying e [ u ] =
Revised: G , implying e [ u ] =

Original: e [ v ] and
Revised: e [ v ] and

Original: X [ u ] =
Revised: X [ u ] =

Original: X [ v ] =
Revised: X [ v ] =

Original: X .
Revised: X .

Original: From Proposition 3.1, the aggregation converges to a point where no feature and little structural information of the target is preserved.
Revised: From Proposition 3.1, the aggregation converges to a point where no feature and little structural information of the target is preserved.

Original: The only information in m [ v ] is v ’s degree.
Revised: The only information in m [ v ] is v ’s degree.

Original: L OCAL - SMOOTHING OF SHA D OW -GCN
Revised: L OCAL - SMOOTHING OF SHA D OW -GCN

Original: With a ﬁxed shallow subgraph, no matter how many times we aggregate using (cid:101)
Revised: With a ﬁxed shallow subgraph, no matter how many times we aggregate using (cid:101)

Original: A [ v ] , the many layers will not include the faraway irrelevant nodes.
Revised: A [ v ] , the many layers will not include the faraway irrelevant nodes.

Original: We can see m [ v ] as a linear combination of the neighbor node features X [ v ] .
Revised: We can see m [ v ] as a linear combination of the neighbor node features X [ v ] .

Original: Increasing the number of layers only pushes the coefﬁcients of each neighbor features to the stationary values.
Revised: Increasing the number of layers only pushes the coefﬁcients of each neighbor features to the stationary values.

Original: The domain X [ v ] of such linear transformation is solely determined by SAMPLE and is independent of the layer depth.
Revised: The domain X [ v ] of such linear transformation is solely determined by SAMPLE and is independent of the layer depth.

Original: Intuitively, if SAMPLE picks non-identical subgraphs for two nodes u and v , the aggregations should be different due to the different domains of the linear transformation.
Revised: Intuitively, if SAMPLE picks non-identical subgraphs for two nodes u and v , the aggregations should be different due to the different domains of the linear transformation.

Original: Therefore, SHA D OW -GCN preserves local feature information whereas normal GCN preserves none .
Revised: Therefore, SHA D OW -GCN preserves local feature information whereas normal GCN preserves none .

Original: For structural information in m [ v ] , note that e [ v ] is a normalized degree distribution of the subgraph around v , and (cid:2) e [ v ] (cid:3) v indicates the role of the target node in the subgraph.
Revised: For structural information in m [ v ] , note that e [ v ] is a normalized degree distribution of the subgraph around v , and (cid:2) e [ v ] (cid:3) v indicates the role of the target node in the subgraph.

Original: If we let SAMPLE return the 1 -hop subgraph, then (cid:2) e [ v ] (cid:3) v alone already contains all the information preserved by a normal GCN, which is v ’s degree in G .
Revised: If we let SAMPLE return the 1 -hop subgraph, then (cid:2) e [ v ] (cid:3) v alone already contains all the information preserved by a normal GCN, which is v ’s degree in G .

Original: Since e [ v ] additionally reﬂects the structure of v ’s ego-net, a deep SHA D OW -GCN with a na¨ıve 1-hop SAMPLE preserves more structural information than a deep GCN.
Revised: Since e [ v ] additionally reﬂects the structure of v ’s ego-net, a deep SHA D OW -GCN with a na¨ıve 1-hop SAMPLE preserves more structural information than a deep GCN.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Let m [ v ] =
Revised: Let m [ v ] =

Original: φ G ( v ) · m [ v ] where φ G is any non-zero function only depending on the structural property of v .
Revised: φ G ( v ) · m [ v ] where φ G is any non-zero function only depending on the structural property of v .

Original: Let M = { m [ v ] | v ∈ V} .
Revised: Let M = { m [ v ] | v ∈ V} .

Original: Given G , SAMPLE and some continuous probability distribution in R |V|× f to generate X , then m [ v ] (cid:54) = m
Revised: Given G , SAMPLE and some continuous probability distribution in R |V|× f to generate X , then m [ v ] (cid:54) = m

Original: [ u ] if V [ u ] (cid:54) =
Revised: [ u ] if V [ u ] (cid:54) =

Original: V [ v ] , almost surely.
Revised: V [ v ] , almost surely.

Original: Corollary 3.2.1.
Revised: Corollary 3.2.1.

Original: Consider SAMPLE 1 where ∀ v ∈ V , (cid:12)(cid:12) V [ v ] (cid:12)(cid:12) ≤ n .
Revised: Consider SAMPLE 1 where ∀ v ∈ V , (cid:12)(cid:12) V [ v ] (cid:12)(cid:12) ≤ n .

Original: Then |M| ≥ (cid:108) |V| n (cid:109) almost surely.
Revised: Then |M| ≥ (cid:108) |V| n (cid:109) almost surely.

Original: Corollary 3.2.2.
Revised: Corollary 3.2.2.

Original: Consider SAMPLE 2 where ∀ u, v ∈ V , V [ v ] (cid:54) =
Revised: Consider SAMPLE 2 where ∀ u, v ∈ V , V [ v ] (cid:54) =

Original: V [ u ] .
Revised: V [ u ] .

Original: Then |M| = |V| almost surely.
Revised: Then |M| = |V| almost surely.

Original: Theorem 3.2 proves SHA D OW -GCN does not oversmooth, as 1).
Revised: Theorem 3.2 proves SHA D OW -GCN does not oversmooth, as 1).

Original: a normal GCN pushes the aggregation of same-degree nodes to the same point, while SHA D OW -GCN with SAMPLE 2 ensures any two nodes (even with the same degree) can have different aggregation; 2).
Revised: a normal GCN pushes the aggregation of same-degree nodes to the same point, while SHA D OW -GCN with SAMPLE 2 ensures any two nodes (even with the same degree) can have different aggregation; 2).

Original: a normal GCN wipes out all the information in the initial node features after many times of aggregation, while SHA D OW -GCN always preserves node feature information.
Revised: a normal GCN wipes out all the information in the initial node features after many times of aggregation, while SHA D OW -GCN always preserves node feature information.

Original: In particular, if we set φ G ( v ) = (cid:0) δ [ v ] ( v ) (cid:1) − 1 / 2 , a normal
Revised: In particular, if we set φ G ( v ) = (cid:0) δ [ v ] ( v ) (cid:1) − 1 / 2 , a normal

Original: GCN generates only one unique value of m , while SHA D OW -GNN can still generate N for any φ G .
Revised: GCN generates only one unique value of m , while SHA D OW -GNN can still generate N for any φ G .

Original: Expressivity comparison with “shallow GNN, shallow sampler”.
Revised: Expressivity comparison with “shallow GNN, shallow sampler”.

Original: Most state-of-the-art GNNs belong to the “shallow GNN, shallow sampler” category.
Revised: Most state-of-the-art GNNs belong to the “shallow GNN, shallow sampler” category.

Original: However, we argue that a deep GNN is still desirable even though the neighborhood is shallow.
Revised: However, we argue that a deep GNN is still desirable even though the neighborhood is shallow.

Original: In the following, we ﬁrst give two examples showing when “shallow GNN, shallow sampler” fails to learn something that a SHA D OW -GNN can.
Revised: In the following, we ﬁrst give two examples showing when “shallow GNN, shallow sampler” fails to learn something that a SHA D OW -GNN can.

Original: Then we prove that SHA D OW -GNN is strictly more powerful than 1-dimensional WeisfeilerLehman test, thus more powerful than all standard “shallow GNN, shallow sampler” designs. 
Revised: Then we prove that SHA D OW -GNN is strictly more powerful than 1-dimensional WeisfeilerLehman test, thus more powerful than all standard “shallow GNN, shallow sampler” designs. [Figure]

Original: [Figure]
Revised: [Figure]

Original: In Figure 1, consider the 2-hop neighborhood (black dots) of some target node v (blue star).
Revised: In Figure 1, consider the 2-hop neighborhood (black dots) of some target node v (blue star).

Original: In the ﬁrst example, suppose another target node v (cid:48) has a slightly different 2-hop structure as reﬂected by an additional edge (red dashed line).
Revised: In the ﬁrst example, suppose another target node v (cid:48) has a slightly different 2-hop structure as reﬂected by an additional edge (red dashed line).

Original: Assume all nodes have identical features.
Revised: Assume all nodes have identical features.

Original: Then any 2-layer
Revised: Then any 2-layer

Original: GNN cannot distinguish v and v (cid:48) , while a 3-layer GNN can.
Revised: GNN cannot distinguish v and v (cid:48) , while a 3-layer GNN can.

Original: The red edge that distinguishes v and v (cid:48) is not having any inﬂuence on v (cid:48) until three times of message passing.
Revised: The red edge that distinguishes v and v (cid:48) is not having any inﬂuence on v (cid:48) until three times of message passing.

Original: In the second example, we consider a simple task of learning the unweighted mean of neighbor features, τ =
Revised: In the second example, we consider a simple task of learning the unweighted mean of neighbor features, τ =

Original: [Equation]
Revised: [Equation]

Original: Suppose G [ v ] consists of the star node, black nodes and black edges.
Revised: Suppose G [ v ] consists of the star node, black nodes and black edges.

Original: Surprisingly, even with such a “regular” subgraph structure, a normal L -layer GraphSAGE cannot learn τ accurately.
Revised: Surprisingly, even with such a “regular” subgraph structure, a normal L -layer GraphSAGE cannot learn τ accurately.

Original: The reasons are as follows.
Revised: The reasons are as follows.

Original: If L > 2 , then normal GraphSAGE will include nodes outside V [ v ] , and the only way to always exclude the impact of these nodes is by setting the weights for that layer to 0 — this effectively reduces the number of layers to 2 .
Revised: If L > 2 , then normal GraphSAGE will include nodes outside V [ v ] , and the only way to always exclude the impact of these nodes is by setting the weights for that layer to 0 — this effectively reduces the number of layers to 2 .

Original: Now since τ is linear, ReLU is not desirable, and should be bypassed by shifting X with the bias parameters.
Revised: Now since τ is linear, ReLU is not desirable, and should be bypassed by shifting X with the bias parameters.

Original: The output of the neural net then simpliﬁes to ζ = (cid:104)(cid:80) L (cid:96) =1 (cid:98) A (cid:96) [ v ] X
Revised: The output of the neural net then simpliﬁes to ζ = (cid:104)(cid:80) L(cid:96) =1 (cid:98) A (cid:96) [ v ] X

Original: [ v ] W (cid:96) (cid:105) v, : , where L = 2 for normal GraphSAGE.
Revised: [ v ] W (cid:96) (cid:105) v, : , where L = 2 for normal GraphSAGE.

Original: For the given (cid:98) A [ v ] , there does not exist a W (cid:96) for GraphSAGE to satisfy τ = ζ for any value of X [ v ] .
Revised: For the given (cid:98) A [ v ] , there does not exist a W (cid:96) for GraphSAGE to satisfy τ = ζ for any value of X [ v ] .

Original: On the other hand, with
Revised: On the other hand, with

Original: SAMPLE returning G [ v ] , SHA D OW -SAGE can learn τ with any precision if L is allowed to grow.
Revised: SAMPLE returning G [ v ] , SHA D OW -SAGE can learn τ with any precision if L is allowed to grow.

Original: The required L for SHA D OW -SAGE to reach the desired precision is determined by the mixing time of (cid:98) A [ v ] .
Revised: The required L for SHA D OW -SAGE to reach the desired precision is determined by the mixing time of (cid:98) A [ v ] .

Original: SHA D OW -SAGE can learn the unweighted mean because the eigenvector corresponding to the largest eigenvalue of (cid:98) A [ v ] is a uniform vector.
Revised: SHA D OW -SAGE can learn the unweighted mean because the eigenvector corresponding to the largest eigenvalue of (cid:98) A [ v ] is a uniform vector.

Original: In fact, a deep SHA D OW -SAGE can learn the unweighted feature mean on any neighborhood.
Revised: In fact, a deep SHA D OW -SAGE can learn the unweighted feature mean on any neighborhood.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Suppose we are given any GNN following the per-layer message passing design as:
Revised: Suppose we are given any GNN following the per-layer message passing design as:

Original: [Equation]
Revised: [Equation]

Original: where f ( (cid:96) )and f ( (cid:96) )are the layer- (cid:96) update and message functions, respectively, implemented as
Revised: where f ( (cid:96) )and f ( (cid:96) )are the layer- (cid:96) update and message functions, respectively, implemented as

Original: MLPs; u ∼ v denotes u is v ’s 1-hop neighbor.
Revised: MLPs; u ∼ v denotes u is v ’s 1-hop neighbor.

Original: Then, on the full L -hop subgraph, an L (cid:48) -layer
Revised: Then, on the full L -hop subgraph, an L (cid:48) -layer

Original: GNN is strictly more discriminative than an L -layer model, when L (cid:48) >
Revised: GNN is strictly more discriminative than an L -layer model, when L (cid:48) >

Original: L .
Revised: L .

Original: Moreover, with injective fand f 2 , the GNN model described above (using L (cid:48) >
Revised: Moreover, with injective fand f 2 , the GNN model described above (using L (cid:48) >

Original: L layers on the L -hop subgraph) is strictly more discriminative than the 1-dimensional Weisfeiler-Lehman test.
Revised: L layers on the L -hop subgraph) is strictly more discriminative than the 1-dimensional Weisfeiler-Lehman test.

Original: See Appendix A for the proof.
Revised: See Appendix A for the proof.

Original: By Theorem 3.3, given a shallow sampler, a deep GNN is strictly more powerful than a shallow one.
Revised: By Theorem 3.3, given a shallow sampler, a deep GNN is strictly more powerful than a shallow one.

Original: Even given a deep sampler, we can still use a “deeper” GNN to beat the “deep” GNN.
Revised: Even given a deep sampler, we can still use a “deeper” GNN to beat the “deep” GNN.

Original: Other practical concerns on deep GNNs include the difﬁculty in optimization and the computation cost.
Revised: Other practical concerns on deep GNNs include the difﬁculty in optimization and the computation cost.

Original: The optimal depth may be determined after considering such tradeoffs.
Revised: The optimal depth may be determined after considering such tradeoffs.

Original: 3. S AMPLER D ESIGN
Revised: 3. S AMPLER D ESIGN

Original: A fundamental question behind any sampler is: what is the right metric to reﬂect neighbor importance?

Original: SAMPLE should be based on such metric, while at the same time be lightweight and scalable.

Original: k -hop sampler.
Revised: k -hop sampler.

Original: Starting from the target node v , the sampler traverses up to k hops.
Revised: Starting from the target node v , the sampler traverses up to k hops.

Original: At a hop- (cid:96) node u , the sampler will add to its hop- ( (cid:96) + 1) node set either all neighbors of u , or b randomly selected neighbors of u .
Revised: At a hop- (cid:96) node u , the sampler will add to its hop- ( (cid:96) + 1) node set either all neighbors of u , or b randomly selected neighbors of u .

Original: The subgraph G [ v ] is induced from all the nodes selected by the sampler.
Revised: The subgraph G [ v ] is induced from all the nodes selected by the sampler.

Original: Here  k and  b are the sampling parameters.
Revised: Here depth k and budget b are the sampling parameters.

Original: Note that GraphSAGE also uses a k -hop sampler. However, the main difference is that SHA D OW -GNN runs on the induced subgraph of the k -hop neighbors. Compared with the normal GraphSAGE, a SHA D OW -SAGE is able to discover more complicated structural information from the subgraph constructed by our k -hop sampler (See Theorem 3.3).
Revised: See Appendix C for more analysis.

Original: PPR sampler.
Revised: PPR sampler.

Original: Personalized PageRank (PPR) has been recently combined with graph learning (Klicpera et al., 2019a;
Revised: Personalized PageRank (PPR) has been recently combined with graph learning (Klicpera et al., 2019a;

Original: Bojchevski et al., 2020; Li et al., 2020b).
Revised: Bojchevski et al., 2020; Li et al., 2020b).

Original: However, none of the existing works have explored PPR in the context of subgraph sampling.
Revised: However, none of the existing works have explored PPR in the context of subgraph sampling.

Original: Given a target v , our PPR sampler proceeds as follows: 1).
Revised: Given a target v , our PPR sampler proceeds as follows: 1).

Original: Compute the approximate PPR vector π v ∈ R |V| for v . 2).
Revised: Compute the approximate PPR vector π v ∈ R |V| for v . 2).

Original: Select the neighborhood V [ v ] such that for u ∈ V [ v ] , the PPR score [ π v ] u is large. 3).
Revised: Select the neighborhood V [ v ] such that for u ∈ V [ v ] , the PPR score [ π v ] u is large. 3).

Original: Construct the induced subgraph from V [ v ] .
Revised: Construct the induced subgraph from V [ v ] .

Original: For step 1), even though vector π v is of length- |V| , most of the PPR values are close to zero.
Revised: For step 1), even though vector π v is of length- |V| , most of the PPR values are close to zero.

Original: So we can use a fast algorithm (Andersen et al., 2006) to compute the approximate PPR score by traversing only the local region around v .
Revised: So we can use a fast algorithm (Andersen et al., 2006) to compute the approximate PPR score by traversing only the local region around v .

Original: Throughout the π v computation, most of its entries remain as the initial value of 0.
Revised: Throughout the π v computation, most of its entries remain as the initial value of 0.

Original: Therefore, the PPR sampler is scalable and efﬁcient w.r.t.
Revised: Therefore, the PPR sampler is scalable and efﬁcient w.r.t.

Original: both time and space complexity.
Revised: both time and space complexity.

Original: For step 2), we can either select V [ v ] based on top- p values in π v , or based on some threshold [ π v ] u > θ .
Revised: For step 2), we can either select V [ v ] based on top- p values in π v , or based on some threshold [ π v ] u > θ .

Original: Then p , θ are hyperparameters.
Revised: Then p , θ are hyperparameters.

Original: For step 3), notice that our PPR sampler only uses PPR scores as a node ﬁlter.
Revised: For step 3), notice that our PPR sampler only uses PPR scores as a node ﬁlter.

Original: The original graph structure is still preserved among V [ v ] , due to the induced subgraph step.
Revised: The original graph structure is still preserved among V [ v ] , due to the induced subgraph step.

Original: Such property differentiates our algorithm from existing ones such as
Revised: Such property differentiates our algorithm from existing ones such as

Original: APPNP (Klicpera et al., 2019a), GDC (Klicpera et al., 2019b) and PPRGo (Bojchevski et al., 2020).
Revised: APPNP (Klicpera et al., 2019a), GDC (Klicpera et al., 2019b) and PPRGo (Bojchevski et al., 2020).

Original: Speciﬁcally, the three related works all reconstruct the adjacency matrix by directly connecting the target v with nodes of top PPR scores.
Revised: Speciﬁcally, the three related works all reconstruct the adjacency matrix by directly connecting the target v with nodes of top PPR scores.

Original: Reconstruction promotes feature propagation from direct neighbors at the cost of losing structural information, since all V [ v ] would become 1-hop away from v .
Revised: Reconstruction promotes feature propagation from direct neighbors at the cost of losing structural information, since all V [ v ] would become 1-hop away from v .

Original: We show in Section 3.3 further connections between SHA D OW -GNN and the prior arts.
Revised: We show in Section 3.3 further connections between SHA D OW -GNN and the prior arts.

Original: Extensions.
Revised: Extensions.

Original: The k -hop and PPR samplers each assume shortest path distance and random walk landing probability as important metrics for measuring neighbor relevance.
Revised: The k -hop and PPR samplers each assume shortest path distance and random walk landing probability as important metrics for measuring neighbor relevance.

Original: Following such rationale, we can design many more sampling algorithms.
Revised: Following such rationale, we can design many more sampling algorithms.

Original: For example, instead of PPR scores, we can use Katz index (Katz, 1953) as our node ﬁltering condition.
Revised: For example, instead of PPR scores, we can use Katz index (Katz, 1953) as our node ﬁltering condition.

Original: This is another way to signify the importance of shortest path distance.
Revised: This is another way to signify the importance of shortest path distance.

Original: If the number of common neighbors is important, we can ﬁlter
Revised: If the number of common neighbors is important, we can ﬁlter

Original: V [ v ] by SimRank (Jeh & Widom, 2002).
Revised: V [ v ] by SimRank (Jeh & Widom, 2002).

Original: Apart from structural information, SAMPLE can also utilize node features.
Revised: Apart from structural information, SAMPLE can also utilize node features.

Original: For example, in the k -hop sampler, instead of randomly pick b neighbors, we can pick those with highest feature similarity (e.g., by cosine similarity or heat kernel (Wang et al., 2020)).
Revised: For example, in the k -hop sampler, instead of randomly pick b neighbors, we can pick those with highest feature similarity (e.g., by cosine similarity or heat kernel (Wang et al., 2020)).

Original: 3. A RCHITECTURAL E XTENSION
Revised: 3. A RCHITECTURAL E XTENSION

Original: Subgraph ensemble.
Revised: Subgraph ensemble.

Original: From Section 3.2, each reasonable SAMPLE reveals some perspectives of the graph, and they altogether present the full picture.
Revised: From Section 3.2, each reasonable SAMPLE reveals some perspectives of the graph, and they altogether present the full picture.

Original: We may then combine the information from various SAMPLE .
Revised: We may then combine the information from various SAMPLE .

Original: Assume a set of C candidates { SAMPLE i } , each returning (cid:0) G [ v ] (cid:1) i .
Revised: Assume a set of C candidates { SAMPLE i } , each returning (cid:0) G [ v ] (cid:1) i .

Original: We can either: 1).
Revised: We can either: 1).

Original: Construct a union-subgraph G (cid:63) [ v ] for SHA D OW -GNN to operate on, where V (cid:63) [ v ] = (cid:83) C i =1 (cid:0) V [ v ] (cid:1) i and E (cid:63) [ v ] = (cid:83) Ci =

Revised: Construct a union-subgraph G (cid:63) [ v ] for SHA D OW -GNN to operate on, where V (cid:63) [ v ] = (cid:83) Ci =1 (cid:0) V [ v ] (cid:1) i and E (cid:63) [ v ] = (cid:83) Ci =

Original: (cid:0) E
Revised: (cid:0) E

Original: [ v ] (cid:1) i , or 2).
Revised: [ v ] (cid:1) i , or 2).

Original: Use C parallel branches of L -layer GNN, each operating on ( G v ) i .
Revised: Use C parallel branches of L -layer GNN, each operating on ( G v ) i .

Original: So v ’s output embedding of each branch i (i.e., a vector z i ) can be aggregated by attention mechanism:
Revised: So v ’s output embedding of each branch i (i.e., a vector z i ) can be aggregated by attention mechanism:

Original: [Equation]
Revised: [Equation]

Original: where f Li is the function by the L -layer GNN at branch i ;
Revised: where f Li is the function by the L -layer GNN at branch i ;

Original: z i is a row vector of (cid:0) Z [ v ] (cid:1) i corresponding to v ; q is a learnable vector; (cid:101) w is normalized from w by softmax.
Revised: z i is a row vector of (cid:0) Z [ v ] (cid:1) i corresponding to v ; q is a learnable vector; (cid:101) w is normalized from w by softmax.

Original: y v is the ﬁnal embedding.
Revised: y v is the ﬁnal embedding.

Original: We can use identical GNN structure for all the C branches or even share weights among the branches.
Revised: We can use identical GNN structure for all the C branches or even share weights among the branches.

Original: { SAMPLE i } can also perform the same algorithm with different hyperparameters.
Revised: { SAMPLE i } can also perform the same algorithm with different hyperparameters.

Original: For example, we can let SAMPLE i be the PPR sampler with different threshold θ i .
Revised: For example, we can let SAMPLE i be the PPR sampler with different threshold θ i .

Original: A SHA D OW -SAGE-ensemble can imitate similar feature aggregation behavior as PPRGo (Bojchevski et al., 2020), while still having the additional advantage of learning rich subgraph structural information.
Revised: A SHA D OW -SAGE-ensemble can imitate similar feature aggregation behavior as PPRGo (Bojchevski et al., 2020), while still having the additional advantage of learning rich subgraph structural information.

Original: Note that PPRGo generates embedding by 1-hop aggregation on the reconstructed graph: τ v = (cid:80) u ∈V [ v ] π u h v , where π u =
Revised: Note that PPRGo generates embedding by 1-hop aggregation on the reconstructed graph: τ v = (cid:80) u ∈V [ v ] π u h v , where π u =

Original: [ π v ] u and h v =
Revised: [ π v ] u and h v =

Original: MLP ( x v ) .
Revised: MLP ( x v ) .

Original: We can partition V [ v ] = (cid:83) Ci =1 V i [ v ] such that nodes in V i [ v ] have similar PPR scores approximated by (cid:101)
Revised: We can partition V [ v ] = (cid:83) Ci =1 V i [ v ] such that nodes in V i [ v ] have similar PPR scores approximated by (cid:101)

Original: π i , and (cid:101) π i ≤ (cid:101) π
Revised: π i , and (cid:101) π i ≤ (cid:101) π

Original: i +1 .
Revised: i +1 .

Original: So τ v ≈ (cid:80) Ci =1 (cid:16) ρ i (cid:80) u ∈V (cid:48)
Revised: So τ v ≈ (cid:80) Ci =1 (cid:16) ρ i (cid:80) u ∈V (cid:48)

Original: i h u (cid:17) , where ρ i = (cid:101) π
Revised: i h u (cid:17) , where ρ i = (cid:101) π

Original: i − (cid:80) j<i (cid:101) π j and V (cid:48) i =
Revised: i − (cid:80) j<i (cid:101) π j and V (cid:48) i =

Original: (cid:83)
Revised: (cid:83)

Original: Ck
Revised: Ck

Original: = i V k
Revised: = i V k

Original: [ v ] .
Revised: [ v ] .

Original: Recall the discussion on learning unweighted feature mean of any V [ v ] (Section 3.1).
Revised: Recall the discussion on learning unweighted feature mean of any V [ v ] (Section 3.1).

Original: Setting θ i = (cid:101)
Revised: Setting θ i = (cid:101)

Original: π i and w i =
Revised: π i and w i =

Original: ρ i , SHA D OW -SAGE can approximate τ v .
Revised: ρ i , SHA D OW -SAGE can approximate τ v .

Original: As our PPR sampler also preserves local structure, our model can be more expressive than PPRGo.
Revised: As our PPR sampler also preserves local structure, our model can be more expressive than PPRGo.

Original: Sampling the reconstructed graph.
Revised: Sampling the reconstructed graph.

Original: Another extension is to build SHA D OW -GNN on the graph reconstructed by graph diffusion (Klicpera et al., 2019b; Frasca et al., 2020).
Revised: Another extension is to build SHA D OW -GNN on the graph reconstructed by graph diffusion (Klicpera et al., 2019b; Frasca et al., 2020).

Original: In the simplest case, we can apply our L -hop sampler on the reconstructed graph, and build a L (cid:48)
Revised: In the simplest case, we can apply our L -hop sampler on the reconstructed graph, and build a L (cid:48)

Original: -layer
Revised: -layer

Original: GNN on the subgraph.
Revised: GNN on the subgraph.

Original: When L = L (cid:48) , we recover GDC (Klicpera et al., 2019b).
Revised: When L = L (cid:48) , we recover GDC (Klicpera et al., 2019b).

Original: When L < L (cid:48) , we have SHA D OW -GDC.
Revised: When L < L (cid:48) , we have SHA D OW -GDC.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Setup.
Revised: Setup.

Original: We perform node classiﬁcationtasks on ﬁvelarge graphs: Flickr (Zeng et al., 2020), 
Revised: We perform node classiﬁcation on ﬁve graphs: Flickr (Zeng et al., 2020), Reddit (Hamilton et al., 2017a),

Original: Reddit (Hamilton et al., 2017a), Yelp (Zeng et al., 2020), ogbn-arxiv (Hu et al., 2020) and ogbn-products (Hu et al., 2020).
Revised: Yelp (Zeng et al., 2020), ogbn-arxiv (Hu et al., 2020) and ogbn-products (Hu et al., 2020).

Original: The Flickr , Reddit and Yelp graphs are under the inductive learning setting, while ogbn-arxiv and ogbn-products are transductive.

Original: The ogbn-arxiv and ogbn-products belong to the Open Graph Benchmark (Hu et al., 2020).

Original: The  sizes of the ﬁve graphs ranges from around 9K nodes ( Flickr ) toaround 2.5M nodes ( ogbn-products ).
Revised: The graph sizes range from around 9K nodes ( Flickr ) to 2.5M nodes ( ogbn-products ).

Original: Consistent with their original paper, we use the metric of “accuracy” for Flickr , Reddit , ogbn-arxiv and ogbn-products , and “F1-micro score” for Yelp .
Revised: Consistent with the original setup, we use the metric of “accuracy” for Flickr , Reddit , ogbn-arxiv and ogbn-products , and “F1-micro score” for Yelp .

Original: Seealso Appendix for further details.
Revised: See Appendix D.1.

Original: We consider six baselines and construct the corresponding SHA D OW models: GCN (Kipf & Welling, 2016), GraphSAGE (Hamilton et al., 2017a), GAT (Veli ˇ ckovi ´
Revised: We consider seven baselines and construct the corresponding SHA D OW models: GCN (Kipf & Welling, 2016), GraphSAGE (Hamilton et al., 2017a), GAT (Veliˇckovi´c et al., 2018), JK-Net

Original: c

Original: et al., 2018), JK-Net (Xu et al., 2018b), GIN (Xu et al., 2018a) and GraphSAINT (Zeng et al., 2020).
Revised: (Xu et al., 2018b), GIN (Xu et al., 2018a), SGC (Wu et al., 2019) and GraphSAINT (Zeng et al., 2020).

Original: The ﬁrst ﬁve baselines are representatives of the state-of-the-art GNN architectures.
Revised: The ﬁrst six are representatives of the state-of-the-art GNN architectures.

Original: They jointly cover various message aggregation functions as well as the skip connection design for facilitating deep GNN training.
Revised: They jointly cover various message aggregation functions as well as the skip connection design for facilitating deep GNN training.

Original: GraphSAINT is the state-of-the-art minibatch training algorithm for large graphs.
Revised: GraphSAINT is the state-of-the-art minibatch training algorithm for large graphs.

Revised: The

Original: The GraphSAINT sampling algorithm has been shown to work well with the GraphSAGE and GAT architectures.
Revised: GraphSAINT training has been shown to work well with the GraphSAGE and GAT architectures.

Original: In Table 1, for theﬁrst four methods ( i.e. , the rows of GCN,GIN, GraphSAGE and GAT), we train 3-layer and 5-layer models in the full batch fashion.
Revised: In Table 1, for the the rows of GCN, GraphSAGE and GAT, we train 3- and 5-layer models in the full batch fashion.

Original: However, for the large graphs ( e.g. , ogbn-products ), the GPU memory (15GB) is too small to support full batch training.
Revised: However, for the large graphs, the GPU memory (15GB) is too small to store the full batch.

Original: We therefore further run experiments using the  minibatch training algorithm of GraphSAINT. Note that “GraphSAINT-RW” means the random walk sampler as suggested by the GraphSAINT paper.
Revised: We therefore further run experiments using the GraphSAINT minibatch training algorithm, where “GraphSAINT-RW” denotes the random walk sampler recommended by Zeng et al.

Original: For all the experiments, we ﬁx the number of channels ( i.e. , the hidden dimension) for each layer as d ( (cid:96) )
Revised: For all the experiments, we ﬁx the number of channels ( i.e. , the hidden dimension) for each layer as d ( (cid:96) )

Original: = 256 .
Revised: = 256 .

Original: All accuracy results are measured by ﬁve runs without ﬁxing random seeds.
Revised: All accuracy results are measured by ﬁve runs without ﬁxing random seeds.

Original: Detailed hyperparameter tuning procedure and architecture conﬁguration are described in Appendix C.3.
Revised: Detailed hyperparameter tuning procedure and architecture conﬁguration are described in Appendix D.3.

Original: Comparison with State-of-the-Art.
Revised: Comparison with state-of-the-art.

Original: Table 1 shows the performance comparison of SHA D OWGNN with state-of-the-art GNNs.
Revised: Table 1 shows the performance comparison of SHA D OW -GNN with state-of-the-art GNNs.

Original: We use two metrics: test set accuracy  and inference cost .
Revised: We use the following metrics: test set accuracy / F1-micro score and inference cost.

Original: The inference cost – deﬁned as the average amount of computation to generate prediction for one test node – is purely a measure of computation complexity and is independent of hardware / implementation factors such as parallelization strategy, batch processing, distributed storage, etc .
Revised: The inference cost – deﬁned as the average amount of computation to generate prediction for one test node – is purely a measure of computation complexity and is independent of hardware / implementation factors such as parallelization strategy, batch processing, distributed storage, etc .

Original: Thus, the metric reﬂects the feasibility of deploying the particular GNN on real-world workloads, where the graph sizes and hardware speciﬁcations can be different from our setup.
Revised: Thus, the metric reﬂects the feasibility of deploying each GNN on real-world workloads, where the graph sizes and hardware speciﬁcations can be different from our setup.

Original: For SHA D OW -GNN, the inference cost does not consider the sampling overhead.
Revised: For SHA D OW -GNN, the inference cost does not consider the sampling overhead.

Original: See Figure 2 for the sampling time measurement and Appendix B for the equations for calculating the inference cost.
Revised: See Figure 2 for the sampling time measurement and Appendix B for the equations for calculating the inference cost.

Original: For the 3-layer and 5-layer models in Table 1, we simply stack the corresponding GNN layers.
Revised: For the 3-layer and 5-layer models in Table 1, we simply stack the corresponding GNN layers.

Original: During training, we further apply the DropEdge technique (Rong et al., 2020) to both the baseline and SHA D OW models.
Revised: During training, we further apply the DropEdge technique (Rong et al., 2020) to both the baseline and SHA D OW models.

Original: We observe that DropEdge helps improve the baseline accuracy by alleviating oversmoothing, and helps improves the SHA D OW accuracy due to its regularization effects.
Revised: We observe that DropEdge helps improve the baseline accuracy by alleviating oversmoothing, and helps improves the SHA D OW accuracy due to its regularization effects.

Original: We further compare the effectiveness of skip-connection ( i.e. , JK-Net) and shallow sampling in Table 2. .

Original: A CCURACY Comparing  GCN, GraphSAGE and GAT with their SHA D OW versions, SHA D OWGNN achieves signiﬁcantly higher accuracy across all the ﬁve datasets.
Revised: A CCURACY Comparing the normal GCN, GraphSAGE and GAT, SHA D OW -GNNs achieve signiﬁcantly higher accuracy across all the ﬁve datasets.

Original: Since such comparison is under the identical backbone GNN architectures, the accuracy gains validate the effectiveness of our design principle of “deep GNN, shallow sampler”.
Revised: Since we use the identical backbone GNN architectures, the accuracy gains validate the effectiveness of our design principle of “deep GNN, shallow sampler”.

Original: Note that that our subgraphs are truly shallow, since the the subgraphs generated by our 2 -hop and PPR samplers contain no more than 200 nodes.
Revised: Note that that our subgraphs are truly shallow, since the the subgraphs generated by our 2 -hop and PPR samplers contain no more than 200 nodes.

Original: Comparing the accuracy of the3-layer normal GNN and the3-layer SHA D OW -GNN, we can clearly see that a shallow neighborhood contains sufﬁcient information to generatea good node embedding.
Revised: For 3-layer, comparing the accuracy of the normal GNN with the SHA D OW -GNN, it is clear that a shallow neighborhood contains sufﬁcient information to generate good node embeddings.

Original: Comparing the accuracy of the 5-layer normal GNN and the 5-layer SHA D OW -GNN, we observe even higheraccuracy gains – for normal GNNs, increasing from 3 layers to 5 layers sometimes results in accuracy degradation (even with DropEdge).
Revised: For 5-layer accuracy comparison, we observe even higher gains – for normal GNNs, increasing from 3 layers to 5 layers sometimes results in accuracy degradation (even with DropEdge).

Original: For SHA D OW -GNN, making the GNN deeper is beneﬁcial most of the time.
Revised: For SHA D OW -GNN, making the GNN deeper is beneﬁcial most of the time.

Original: These observations validates our oversmoothing and expressivity analysis in Section 3.1.
Revised: These observations validates our oversmoothing and expressivity analysis in Section 3.1.

Original: Finally, the higher accuracy achieved by the PPR sampler compared with the 2 -hop sampler demonstrates the importance of a good sampling algorithm. 
Revised: Finally, the higher accuracy achieved by the PPR sampler compared with the 2 -hop sampler demonstrates the importance of a good sampling algorithm. FlickrReddit Yelp arxivproducts I n f e r e n ce ti m e p e r nod e ( m s ) PPR

Revised: SAGE

Revised: GAT

Original:  I NFERENCE COST From Table 1, the inference cost of SHA D OW -GNN is often orders of magnitude lower than that of the baselines.
Revised: [Figure] analysis (5- [Figure] I NFERENCE COST From Table 1, the inference cost of SHA D OW -GNN is often orders of magnitude lower than that of the baselines.

Original: The high cost of the baselines reﬂects the “neighbor explosion” challenge for deep GNNs.
Revised: The high cost of the baselines reﬂects the “neighbor explosion” challenge for deep GNNs.

Original: In short, for normal GNN, since we do not enforce a ﬁxed size subgraph, the inference computation complexity may grow exponentially with the depth of the GNN (Chen et al., 2017).
Revised: In short, for normal GNN, since we do not enforce a ﬁxed size subgraph, the inference computation complexity may grow exponentially with the depth of the GNN (Chen et al., 2017).

Original: For example, the number of (cid:96) -hop neighbors may be 10 × the number of ( (cid:96) − 1) -hop neighbors.
Revised: For example, the number of (cid:96) -hop neighbors may be 10 × the number of ( (cid:96) − 1) -hop neighbors.

Original: For SHA D OW -GNN, it is clear that the inference cost only grows linearly with the GNN depth.
Revised: For SHA D OW -GNN, it is clear that the inference cost only grows linearly with the GNN depth.

Original: From Table 1, we observe that even when the GNN depth is only 5, the high computation cost already makes the inference of normal GNNs practically infeasible.
Revised: From Table 1, we observe that even when the GNN depth is only 5, the high computation cost already makes the inference of normal GNNs practically infeasible.

Original: The results show the high efﬁciency of our design.
Revised: The results show the high efﬁciency of our design.

Original: Finally, note that for GraphSAINT, its sampling is only applied during training.
Revised: Finally, note that for GraphSAINT, its sampling is only applied during training.

Original: The inference computation of GraphSAINT falls back to be identical as its backbone architecture (which samples the full L -hop neighborhood).
Revised: The inference computation of GraphSAINT falls back to be identical as its backbone architecture (which samples the full L -hop neighborhood).

Original: On the other hand, forSHA D OW -GNN, we apply the same sampling and batching strategy, whether it is for training or for inference. 
Revised: On the other hand, for OW -GNN, we apply the same sampling and batching strategy, whether it is for training or for inference. [Figure] [Table]

Original: [Table]
Revised: [Figure]

Original: Evaluation on Sampler.
Revised: Evaluation on PPR sampler.

Original: Wefurther evaluate the  time overheadof sampling on real machines and how the sampling sizes affect the accuracy-execution time tradeoff.
Revised: We evaluate the PPR sampler in terms of its execution time overhead and accuracy-time tradeoff.

Original: In Figure [Table] we parallelize thePPR sampler(implemented by C++ and OpenMP ) on a 40-core Xeon CPU and the GNNinference computation(implemented by PyTorch ) on a NVIDIA P100 GPU.
Revised: In Figure 2, we parallelize the sampler on a 40-core Xeon CPU and the GNN computation on a NVIDIA P100 GPU (See Appendix D.2).

Original: Clearly, the sampler is lightweight: the sampling time is lower than the GNN computation time in most cases.
Revised: The sampler is lightweight: the sampling time is lower than the GNN computation time in most cases.

Original: Also, the sampling time per node does not necessarily grow with the full graph size.
Revised: Also, the sampling time per node does not necessarily grow with the full graph size.

Original: As described in Section 3.2, the approximate PPR computation(Andersen et al., 2006) achieves efﬁciency and scalability by only traversing a local region around each target node.Note that the CPU executing the sam- FlickrReddit
Revised: By Section 3.2, the approximate PPR computation achieves efﬁciency and scalability by only traversing a local region around each target node.

Original: [Figure] arxivproducts [Figure] I n f e r e n ce ti m e p e r nod e ( m s )
Revised: To evaluate the accuracy-time tradeoff, we take the 5 -layer models of Table 1 as the pretrained models.

Original:  PPR SAGE GAT [Figure]
Revised: Then for SHA D OW -GNN, we vary the PPR budget p from 50 to 200 with stride 50.

Original: FlickrRedditarxivproducts pler is much less powerful than the GPU. We can thus further reduce the sampling time by trivial parallelization on a more powerful machine. To understand how the sizes of the subgraphs affect the tradeoff between accuracy and inference time, we take the 5 -layer models of Table 1 as the pretrained models. Then for SHA D OW -GNN, we vary the PPR sampling size p from 50 to 200 with stride 50.
Revised: In

Original: In Figure 3 1 , the inference time of SHA D OW -GNN has already included the PPR sampling time.
Revised: Figure 3, the inference time of SHA D OW -GNN has already included the PPR sampling time.

Original: Firstly, consistent with Table 1, inference of SHA D OW -GNNs achieves higher accuracy than the normal GNNs, with orders of magnitude speedup in time.
Revised: Firstly, consistent with Table 1, inference of SHA D OW -GNNs achieves higher accuracy than the normal GNNs, with orders of magnitude speedup as well.

Original: In addition, based on the application requirements ( e.g. , latency constraint), SHA D OW -GNNs have the ﬂexibility of adjusting the sampling size without the need of retraining.
Revised: In addition, based on the application requirements ( e.g. , latency constraint), SHA D OW -GNNs have the ﬂexibility of adjusting the sampling size without the need of retraining.

Original: For  Reddit and ogbn-arxiv ,  reducing the subgraph size from 200 to 50 speeds up inference by 2 × to 4 × at the cost of less than 1% accuracy drop.[Figure] [Table] [Figure]
Revised: For example, on Reddit and ogbn-arxiv , directly reducing the subgraph size from 200 to 50 speeds up inference by 2 × to 4 × at the cost of less than 1% accuracy drop.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Evaluation on Other GNN Architectures.
Revised: Evaluation on other architectures.

Original: The idea of using shallow subgraphs on deep

Original: GNNs can potentially beneﬁt a wide range of GNN architectures.

Original: In addition to the GCN, GraphSAGE and GAT modelsevaluated in Table 1, we further compare JK-Net and GIN with their SHA D OW versions in Table 2.
Revised: In addition to the GCN, GraphSAGE and GAT models in Table 1, we further compare JK-Net and GIN with their SHA D OW versions in Table 2.

Original: Similar to DropEdge, the skip connection (or “jumping knowledge”) helps accuracy improvement of deeper models.
Revised: Similar to DropEdge, the skip connection (or “jumping knowledge”) helps accuracy improvement on deeper models.

Original: Compared with the normal JK-Net,we observe that increasing themodel depth beneﬁts SHA D OW -JK more.
Revised: Compared with the normal JK-Net, increasing the depth beneﬁts SHA D OW -JK more.

Original: The GIN architecture theoretically does not incur oversmoothing.
Revised: The GIN architecture theoretically does not oversmooth.

Original: However, we observe that thetraining of GIN  is very sensitive to hyperparameter settings.
Revised: However, we observe that the GIN training is very sensitive to hyperparameter settings.

Original: Since GIN aggregates neighbors by “sum” operation rather than “mean”, we hypothesize that it is more sensitive to  noisy neighbors 2 .
Revised: We hypothesize that such a challenge is due to the sensitivity of the sum aggregator on noisy neighbors ( e.g. , for GraphSAGE, a single noisy node can hardly cause a signiﬁcant perturbation on the aggregation, due to the averaging over the entire neighborhood).

Original: The accuracy improvement of SHA D OW -GIN compared with the normal GIN may thus  due to the exclusion of noises by shallow sampling (see Section 3.1).
Revised: The accuracy improvement of SHA D OW -GIN compared with the normal GIN may thus be due to noise ﬁltering by shallow sampling (see Section 3.1).

Original: The impact of noises  cansometimes be critical, as reﬂected by the 5-layer GIN accuracy on Reddit . 
Revised: The impact of noises / irrelevant neighbors can be critical, as reﬂected by the 5-layer GIN accuracy on Reddit . [Table] Effect of model depth.

Revised: To validate Theorem 3.2 that

Revised: SHA D OW -GCN does not oversmooth, we experiment on much larger GNN depth.

Original: C ONCLUSION
Revised: In the left plot of Figure 4, we run a normal SGC and a SHA D OW -SGC by varying the power on the adjacency matrix from 1 to 40 (See Appendix D.4 for detailed setup). While SGC gradually collapses local information into global “white noise”, accuracy of SHA D OW -SGC does not degrade. In the middle and right plots, we train the standard GCN architecture without the SGC simplications. Since ogbn-products is too large, we compare SHA D OW -GCN with GCN-SAINT.

Revised: For both methods, the deeper models converge slower.

Revised: Deeper models (whether it is GNN or other types of neural networks) are in general harder to optimize.

Revised: For SHA D OW -GCN, the validation set performance of the 15-layer model eventually catches up with the shallower models.

Revised: For GCN-SAINT, however, there consistently exists a large performance gap between the 15-layer model and the shallower ones.

Revised: Both experiments clearly show that shallow sampling prevents oversmoothing.

Original: We have presented a design principle, “deep GNNs, shallow samplers”, for accurate and efﬁcient
Revised: Finally, for ogbn-products in in Table 3, we note that even through > 98% of the subgraph nodes are with 2 hops (see Figure 6), increasing the GNN depth can beneﬁt accuracy for up to L (cid:48) = 7 layers. Evaluation on subgraph ensemble. Beneﬁt of ensemble is shown via the 5 -layer entries in Table 3.

Original: GNNs.
Revised: Different samplers, such as PPR and k -hop, preserve different kinds of local information. Rather than designing one “perfect” sampler, it may be more reasonable to ensemble a few simpler ones.

Original: We have enhanced the learning of various state-of-the-art GNNs by constructing their
Revised: Note that ensemble is only possible for subgraph based SHA D OW -GNN. In Figure 8 (Appendix E), we show that even the simplest 1 -hop sampler can beneﬁt training after ensemble.

Original:  SHA D OW versions.
Revised:  C ONCLUSION

Original: We have extensively analyzed the advantage of SHA D OW -GNN in terms of expressivity and proposed various sampling algorithms. We have also revealed the relationship between SHA D OW -GNN and several existing models by simple architectural extensions.
Revised: We have presented a design principle, “deep GNNs, shallow samplers”, enabling accurate and efﬁcient inference for a wide range of GNN architectures. We have theoretically justiﬁed the expressivity of SHA D OW -GNN and empirically demonstrates its beneﬁts under various setups and metrics.

Original: A P ROOFS
Revised: A P ROOFS

Original: Proof of Proposition 3.1.
Revised: Proof of Proposition 3.1.

Original: The GCN model performs symmetric normalization on the adjacency matrix.
Revised: The GCN model performs symmetric normalization on the adjacency matrix.

Original: SHA D OW -GCN follows the same way to normalize the subgraph adjacency matrix as:
Revised: SHA D OW -GCN follows the same way to normalize the subgraph adjacency matrix as:

Original: [Equation]
Revised: [Equation]

Original: where A [ v ] ∈ R N × N is the binary adjacency matrix for G [ v ] .
Revised: where A [ v ] ∈ R N × N is the binary adjacency matrix for G [ v ] .

Original: (cid:101)
Revised: (cid:101)

Original: A [ v ] is a real symmetric matrix and has the largest eigenvalue of 1 .
Revised: A [ v ] is a real symmetric matrix and has the largest eigenvalue of 1 .

Original: Since SAMPLE ensures the subgraph G [ v ] is connected, so the multiplicity of the largest eigenvalue is 1.
Revised: Since SAMPLE ensures the subgraph G [ v ] is connected, so the multiplicity of the largest eigenvalue is 1.

Original: By Theorem 1 of
Revised: By Theorem 1 of

Original: Huang et al.
Revised: Huang et al.

Original: (2020), we can bound the eigenvalues
Revised: (2020), we can bound the eigenvalues

Original: λ i by 1 = λ 1 > λ 2 > . . .
Revised: λ i by 1 = λ 1 > λ 2 > . . .

Original: > λ N > − 1 .
Revised: > λ N > − 1 .

Original: Performing eigen-decomposition on (cid:101)
Revised: Performing eigen-decomposition on (cid:101)

Original: A [ v ] , we have
Revised: A [ v ] , we have

Original: [Equation]
Revised: [Equation]

Original: where Λ is a diagonal matrix Λ i,i
Revised: where Λ is a diagonal matrix Λ i,i

Original: = λ
Revised: = λ

Original: i and matrix E [ v ] consists of the N normalized eigenvectors.
Revised: i and matrix E [ v ] consists of the N normalized eigenvectors.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Since | λ i | < 1 when i (cid:54) = 1 , we have lim L →∞ (cid:101)
Revised: Since | λ i | < 1 when i (cid:54) = 1 , we have lim L →∞ (cid:101)

Original: A L [ v ] =
Revised: A L [ v ] =

Original: e [ v ] e T [ v ] , where e v is the eigenvector corresponding to λ 1 .
Revised: e [ v ] e T [ v ] , where e v is the eigenvector corresponding to λ 1 .

Original: It is easy to see that (cid:2)
Revised: It is easy to see that (cid:2)

Original: e [ v ] (cid:3) u ∝ (cid:112) δ [ v ] ( u ) (Huang et al., 2020).
Revised: e [ v ] (cid:3) u ∝ (cid:112) δ [ v ] ( u ) (Huang et al., 2020).

Original: After normalization, (cid:2) e [ v ] (cid:3) u
Revised: After normalization, (cid:2) e [ v ] (cid:3) u

Original: = (cid:114)
Revised: = (cid:114)

Original: [Equation]
Revised: [Equation]

Original: It directly follows that m [ v ] = (cid:2) e [ v ] (cid:3) v · (cid:16) e T [ v ] X
Revised: It directly follows that m [ v ] = (cid:2) e [ v ] (cid:3) v · (cid:16) e T [ v ] X

Original: [ v ] (cid:17) , with value of e [ v ] deﬁned above.
Revised: [ v ] (cid:17) , with value of e [ v ] deﬁned above.

Original: Proof of Theorem 3.2.
Revised: Proof of Theorem 3.2.

Original: We ﬁrst prove the case of m [ v ] = m [ v ] .
Revised: We ﬁrst prove the case of m [ v ] = m [ v ] .

Original: i.e. , φ G ( v ) = 1 .
Revised: i.e. , φ G ( v ) = 1 .

Original: According to Proposition 3.1, the aggregation for each target node equals m [ v ] = (cid:2) e [ v ] (cid:3) v e T [ v ] X [ v ] .
Revised: According to Proposition 3.1, the aggregation for each target node equals m [ v ] = (cid:2) e [ v ] (cid:3) v e T [ v ] X [ v ] .

Original: Let N = |V| .
Revised: Let N = |V| .

Original: Let ¯¯
Revised: Let ¯¯

Original: e [ v ] ∈ R N × 1 be the vector expanded from e [ v ] , such that (cid:2) ¯¯
Revised: e [ v ] ∈ R N × 1 be the vector expanded from e [ v ] , such that (cid:2) ¯¯

Original: e [ v ] (cid:3) u = 0 if u / ∈ V [ v ] and the rest of the non-zero elements of ¯¯ e [ v ] are copied from e [ v ] accordingly.
Revised: e [ v ] (cid:3) u = 0 if u / ∈ V [ v ] and the rest of the non-zero elements of ¯¯ e [ v ] are copied from e [ v ] accordingly.

Original: We deﬁne:
Revised: We deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Suppose we only compute (cid:15) when two nodes u
Revised: Suppose we only compute (cid:15) when two nodes u

Original: and v have non-identical sampled neighborhood.
Revised: and v have non-identical sampled neighborhood.

Original: i.e.,
Revised: i.e.,

Original: V [ u ] (cid:54) =
Revised: V [ u ] (cid:54) =

Original: V [ v ] .
Revised: V [ v ] .

Original: Then given G , SAMPLE and some distribution to generate X , there are ﬁnite number of possible (cid:15) values.
Revised: Then given G , SAMPLE and some distribution to generate X , there are ﬁnite number of possible (cid:15) values.

Original: The reason is that G is ﬁnite and (cid:15) does not depend on X (where X itself can take inﬁnitely many values).
Revised: The reason is that G is ﬁnite and (cid:15) does not depend on X (where X itself can take inﬁnitely many values).

Original: Each of the possible (cid:15) values deﬁnes a hyperplane in R N − 1 by (cid:15) · x =(where x ∈ R N ).
Revised: Each of the possible (cid:15) values deﬁnes a hyperplane in R N − 1 by (cid:15) · x =(where x ∈ R N ).

Original: Let all such hyperplanes be H .
Revised: Let all such hyperplanes be H .

Original: Now suppose we build SHA D OW -GCN by performing SAMPLE on G and X .
Revised: Now suppose we build SHA D OW -GCN by performing SAMPLE on G and X .

Original: For any two v and u with V [ v ] (cid:54) =
Revised: For any two v and u with V [ v ] (cid:54) =

Original: V [ u ] , suppose their aggregations are the same: m [ v ] = m [ u ] .
Revised: V [ u ] , suppose their aggregations are the same: m [ v ] = m [ u ] .

Original: Then we must have (cid:15) · X = 0 .
Revised: Then we must have (cid:15) · X = 0 .

Original: In other words, ∀ i, X : ,i must fall on one of the hyperplanes in H .
Revised: In other words, ∀ i, X : ,i must fall on one of the hyperplanes in H .

Original: However, since X is generated from a continuous distribution in R N × f , X : ,i would not fall on all of the hyperplanes in H , almost surely.
Revised: However, since X is generated from a continuous distribution in R N × f , X : ,i would not fall on all of the hyperplanes in H , almost surely.

Original: Therefore, for any v and u such that V [ v ] (cid:54) =
Revised: Therefore, for any v and u such that V [ v ] (cid:54) =

Original: V [ u ] , m [ v ] (cid:54) = m
Revised: V [ u ] , m [ v ] (cid:54) = m

Original: [ u ] almost surely.
Revised: [ u ] almost surely.

Original: For a more general φ G ( v ) , since φ G does not depend on X , the proof follows exactly the same steps as above.
Revised: For a more general φ G ( v ) , since φ G does not depend on X , the proof follows exactly the same steps as above.

Original: Proof of Corollary 3.2.1.
Revised: Proof of Corollary 3.2.1.

Original: Note that ∀ u ∈ V , u ∈ V [ u ] .
Revised: Note that ∀ u ∈ V , u ∈ V [ u ] .

Original: For any node v , there are at most n − 1 other nodes in V with the same neighborhood as V [ v ] .
Revised: For any node v , there are at most n − 1 other nodes in V with the same neighborhood as V [ v ] .

Original: Such n − 1 possible nodes are exactly those in V [ v ] .
Revised: Such n − 1 possible nodes are exactly those in V [ v ] .

Original: By Theorem 3.2, ∀ v ∈ V , there are at most n − 1 other nodes in V having the same aggregation as m [ v ] .
Revised: By Theorem 3.2, ∀ v ∈ V , there are at most n − 1 other nodes in V having the same aggregation as m [ v ] .

Original: Equivalently, total number of possible aggregations is at least (cid:100)|V| /n (cid:101) .
Revised: Equivalently, total number of possible aggregations is at least (cid:100)|V| /n (cid:101) .

Original: Proof of Corollary 3.2.2.
Revised: Proof of Corollary 3.2.2.

Original: By deﬁnition of SAMPLE 2 , any pair of nodes have non-identical neighborhood.
Revised: By deﬁnition of SAMPLE 2 , any pair of nodes have non-identical neighborhood.

Original: By Theorem 3.2, any pair of nodes have non-identical aggregation.
Revised: By Theorem 3.2, any pair of nodes have non-identical aggregation.

Original: Equivalently, all nodes have different aggregation and |M| = |V| .
Revised: Equivalently, all nodes have different aggregation and |M| = |V| .

Original: Proof of Theorem 3.3.
Revised: Proof of Theorem 3.3.

Original: Deﬁne G L [ v ] as the subgraph induced from all the (cid:96) -hop neighbors of v , where 1 ≤ (cid:96) ≤ L .
Revised: Deﬁne G L [ v ] as the subgraph induced from all the (cid:96) -hop neighbors of v , where 1 ≤ (cid:96) ≤ L .

Original: We ﬁrst prove that an L (cid:48) -layer
Revised: We ﬁrst prove that an L (cid:48) -layer

Original: GNN is at least as expressive as an L -layer GNN on any L -hop subgraph.
Revised: GNN is at least as expressive as an L -layer GNN on any L -hop subgraph.

Original: We note that for the target node v , the only difference between these two architectures is that an L -layer GNN exactly performs L message passing iterations to propagate node information from at most L hops away, while an L (cid:48)
Revised: We note that for the target node v , the only difference between these two architectures is that an L -layer GNN exactly performs L message passing iterations to propagate node information from at most L hops away, while an L (cid:48)

Original: -layer
Revised: -layer

Original: GNN has L (cid:48) − L
Revised: GNN has L (cid:48) − L

Original: more message passing iterations before performing the L message passings.
Revised: more message passing iterations before performing the L message passings.

Original: Thanks to the universal approximation theorem (Hornik et al., 1989), we can let f ( (cid:96) ) [Equation]
Revised: Thanks to the universal approximation theorem (Hornik et al., 1989), we can let f ( (cid:96) ) 1 (cid:16) h

Original: h ( [Equation] − 1) v
Revised: ( (cid:96) − 1) v

Original: [Equation]
Revised: [Equation]

Original: since MLPs can model and learn such functions.
Revised: since MLPs can model and learn such functions.

Original: Then, the L (cid:48) -layer
Revised: Then, the L (cid:48) -layer

Original: GNN will have the same output as the L -layer GNN.
Revised: GNN will have the same output as the L -layer GNN.

Original: Then, we show that an L (cid:48) -layer
Revised: Then, we show that an L (cid:48) -layer

Original: GNN can learn something that an L -layer GNN cannot learn on some G L [ v ] .
Revised: GNN can learn something that an L -layer GNN cannot learn on some G L [ v ] .

Original: This can be proved exactly by the example given in Figure 1, where the red edge is included in the subgraph.
Revised: This can be proved exactly by the example given in Figure 1, where the red edge is included in the subgraph.

Original: In this G 2[ v (cid:48) ] around v (cid:48)
Revised: In this G 2[ v (cid:48) ] around v (cid:48)

Original: , a 2-layer GNN fails to capture the red edge, and will output the representation of v (cid:48) the same as when there is no red edge.
Revised: , a 2-layer GNN fails to capture the red edge, and will output the representation of v (cid:48) the same as when there is no red edge.

Original: In contrast, a GNN with more than 2 layers can capture the red edge immediately after the ﬁrst message passing round, where the two end nodes of the red edge receive messages from each other.
Revised: In contrast, a GNN with more than 2 layers can capture the red edge immediately after the ﬁrst message passing round, where the two end nodes of the red edge receive messages from each other.

Original: And after 3 message passing rounds, this captured additional information will be propagated to v (cid:48) , resulting in a representation different from that learned by the 2-layer GNN.
Revised: And after 3 message passing rounds, this captured additional information will be propagated to v (cid:48) , resulting in a representation different from that learned by the 2-layer GNN.

Original: Finally, we prove that with injective f 1 and f 2 , the GNN model described above (using L (cid:48) >
Revised: Finally, we prove that with injective f 1 and f 2 , the GNN model described above (using L (cid:48) >

Original: L layers on the L -hop subgraph) is strictly more discriminative than the 1-dimensional Weisfeiler-Lehman (1-WL) test.
Revised: L layers on the L -hop subgraph) is strictly more discriminative than the 1-dimensional Weisfeiler-Lehman (1-WL) test.

Original: We ﬁrst prove that no matter how many iterations L 1-WL runs, we can always run this GNN model on the L -hop subgraph with L (cid:48) layers to get at least the same discriminative power as 1-WL.
Revised: We ﬁrst prove that no matter how many iterations L 1-WL runs, we can always run this GNN model on the L -hop subgraph with L (cid:48) layers to get at least the same discriminative power as 1-WL.

Original: According to Xu et al.
Revised: According to Xu et al.

Original: (2018a), an L -layer GNN on G L [ v ] is theoretically as discriminative as 1-WL by being able to discriminate all nodes v that 1-WL with L iterations of color update can discriminate.
Revised: (2018a), an L -layer GNN on G L [ v ] is theoretically as discriminative as 1-WL by being able to discriminate all nodes v that 1-WL with L iterations of color update can discriminate.

Original: Suppose there are two node v, v (cid:48) that 1-WL gives different colors to after L iterations.
Revised: Suppose there are two node v, v (cid:48) that 1-WL gives different colors to after L iterations.

Original: Then applying the L -layer GNN on G L [ v ] and G L [ v (cid:48) ] will also return h ( L ) v (cid:54)
Revised: Then applying the L -layer GNN on G L [ v ] and G L [ v (cid:48) ] will also return h ( L ) v (cid:54)

Original: =
Revised: =

Original: h ( L ) v (cid:48) .
Revised: h ( L ) v (cid:48) .

Original: Since f 1 and f 2 are injective, we know h ( L + i ) v (cid:54)
Revised: Since f 1 and f 2 are injective, we know h ( L + i ) v (cid:54)

Original: =
Revised: =

Original: h ( L + i ) v (cid:48) for any i > 0 , which means an L (cid:48)
Revised: h ( L + i ) v (cid:48) for any i > 0 , which means an L (cid:48)

Original: -layer
Revised: -layer

Original: GNN ( L (cid:48) > L ) can also discriminate the two nodes.
Revised: GNN ( L (cid:48) > L ) can also discriminate the two nodes.

Original: [Figure]
Revised: [Figure]

Original: on which SHA D OW -GNN is more expressive than 1-WL
Revised: on which SHA D OW -GNN is more expressive than 1-WL

Original: Then it sufﬁces to give an example where 1-WL always fails to distinguish two nodes no matter how many iterations it runs, while the GNN model above can discriminate them with some L and L (cid:48) .
Revised: Then it sufﬁces to give an example where 1-WL always fails to distinguish two nodes no matter how many iterations it runs, while the GNN model above can discriminate them with some L and L (cid:48) .

Original: We construct a very simple example, as shown in Figure 4.
Revised: We construct a very simple example, as shown in Figure 5.

Original: Consider an undirected graph with two connected components (CC).
Revised: Consider an undirected graph with two connected components (CC).

Original: The ﬁrst
Revised: The ﬁrst

Original: CC is a hexagon.
Revised: CC is a hexagon.

Original: The second CC is a triangle.
Revised: The second CC is a triangle.

Original: Both CC are 2regular.
Revised: Both CC are 2regular.

Original: Suppose all nodes have identical attributes and edges are unweighted.
Revised: Suppose all nodes have identical attributes and edges are unweighted.

Original: Our goal is to discriminate a node u in the ﬁrst CC from a node v in the second CC.
Revised: Our goal is to discriminate a node u in the ﬁrst CC from a node v in the second CC.

Original: For 1-WL, no matter how many iterations L it runs, 1-WL always assigns the same label to u and v .
Revised: For 1-WL, no matter how many iterations L it runs, 1-WL always assigns the same label to u and v .

Original: For the GNN above, when we set L = 1 and L (cid:48) = 2 , we can already discriminate between the two nodes.
Revised: For the GNN above, when we set L = 1 and L (cid:48) = 2 , we can already discriminate between the two nodes.

Revised: Remark Note that even though we discuss the ( L + 1) -layer case in the above proof, increasing the layer L (cid:48) beyond L + 1 can still be beneﬁtial in many cases.

Revised: Consider the following:

Revised: [Equation]

Revised: We revisit the GraphSAGE example on Figure 1.

Original: B I NFERENCE C OMPLEXITY C ALCULATION
Revised: Suppose we use a L (cid:48) -layer GraphSAGE to approximate a function τ on the subgraph G L [ v ] . We consider an example τ as computing the unweighted mean of the subgraph node features. In this case, the error of approximating τ converges to zero when L (cid:48) goes to inﬁnity. The error can still be signiﬁcant if L (cid:48) is not sufﬁciently larger than

Revised: L .

Revised: The desired depth L (cid:48) is determined by the mixing time of the subgraph adjacency matrix.

Revised: C ASE

Revised: Consider a more “powerful” GNN when f 1 and f 2 of Equation 2 perform injective mapping.

Revised: Recall the following two facts:

Revised: • On the full graph, such a GNN of L 0 layers is as discriminative as the 1-WL test running L 0 iterations.

Revised: • 1-WL may take up to O ( N ) iterations to converge where N is the full graph size.

Revised: Now extend to the SHA D OW -GNN case.

Revised: Consider two target nodes u and v with their corresponding

Revised: L -hop subgraphs G L [ u ] and G L [ v ] .

Revised: An L (cid:48) -layer

Revised: GNN will output different embeddings for u and v , if u ’s color assigned by L (cid:48) -iteration 1-WL on G L [ u ] is different from v ’s color assigned by L (cid:48)

Revised: -iteration 1-WL on G L [ v ] .

Revised: The coloring of u and v can keep changing for O ( N ) iterations (here N is the size of G L [ u ] and G L [ v ] ).

Revised: Thus, increasing the GNN depth on L -hop subgraphs can improve discriminative power for up to O ( N ) layers.

Revised: B I NFERENCE C OMPLEXITY C ALCULATION

Original: Here we describe the equations to compute the “inference cost” of Table 1.
Revised: Here we describe the equations to compute the “inference cost” of Table 1.

Original: Recall that inference cost is a measure of computation complexity to generate node embeddings for a given GNN architecture.
Revised: Recall that inference cost is a measure of computation complexity to generate node embeddings for a given GNN architecture.

Original: The numbers in Table 1 shows on average, how many arithmetic operations is required to generate the embedding for each node.
Revised: The numbers in Table 1 shows on average, how many arithmetic operations is required to generate the embedding for each node.

Original: For a GNN layer (cid:96) , denote the number of input nodes as n ( (cid:96) − 1) and the number of output nodes as n ( (cid:96) ) .
Revised: For a GNN layer (cid:96) , denote the number of input nodes as n ( (cid:96) − 1) and the number of output nodes as n ( (cid:96) ) .

Original: Denote the number of edges connecting the input and output nodes as m ( (cid:96) ) .
Revised: Denote the number of edges connecting the input and output nodes as m ( (cid:96) ) .

Original: Recall that we use d ( (cid:96) ) to denote the number of channels, or, the hidden dimension.
Revised: Recall that we use d ( (cid:96) ) to denote the number of channels, or, the hidden dimension.

Original: In the following, we ignore the computation cost of non-linear activation, batch-normalization and applying bias, since their complexity is negligible compared with the other operations.
Revised: In the following, we ignore the computation cost of non-linear activation, batch-normalization and applying bias, since their complexity is negligible compared with the other operations.

Original: For the GCN architecture, each layer mainly performs two operations: aggregation of the neighbor features and linear transformation by the layer weights.
Revised: For the GCN architecture, each layer mainly performs two operations: aggregation of the neighbor features and linear transformation by the layer weights.

Original: So the number of multiplication-addition (MAC) operations of a layer equals:
Revised: So the number of multiplication-addition (MAC) operations of a layer equals:

Original: [Equation]
Revised: [Equation]

Original: Similarly, for GraphSAGE architecture, the number of MAC operations equals:
Revised: Similarly, for GraphSAGE architecture, the number of MAC operations equals:

Original: [Equation]
Revised: [Equation]

Original: 9) where the 2 factor is due to the weight transformation on the self-features.
Revised: (9) where the 2 factor is due to the weight transformation on the self-features.

Original: For GAT, suppose the number of attention heads is t .
Revised: For GAT, suppose the number of attention heads is t .

Original: Then the layer contains t weight matrices
Revised: Then the layer contains t weight matrices

Original: W i , each of shape d ( (cid:96) − 1)
Revised: W i , each of shape d ( (cid:96) − 1)

Original: × d ( (cid:96) ) t .
Revised: × d ( (cid:96) ) t .

Original: We ﬁrst transform each of the n ( (cid:96) − 1) nodes by W i .
Revised: We ﬁrst transform each of the n ( (cid:96) − 1) nodes by W i .

Original: Then for each edge ( u, v ) connecting the layer input u to the layer output v , we obtain its edge weight ( i.e. , a scalar) by computing the dot product between u ’s, v ’s transformed feature vectors and the model attention weight vector.
Revised: Then for each edge ( u, v ) connecting the layer input u to the layer output v , we obtain its edge weight ( i.e. , a scalar) by computing the dot product between u ’s, v ’s transformed feature vectors and the model attention weight vector.

Original: After obtaining the edge weight, the remaining computation is to aggregate the n ( (cid:96) − 1) features into the n ( (cid:96) ) nodes.
Revised: After obtaining the edge weight, the remaining computation is to aggregate the n ( (cid:96) − 1) features into the n ( (cid:96) ) nodes.

Original: The ﬁnal output is obtained by concatenating the features of different heads.
Revised: The ﬁnal output is obtained by concatenating the features of different heads.

Original: The number of MAC operations equals:
Revised: The number of MAC operations equals:

Original: [Equation]
Revised: [Equation]

Original: On the same graph, GCN is less expensive than GraphSAGE.
Revised: On the same graph, GCN is less expensive than GraphSAGE.

Original: GraphSAGE is in general less expensive than GAT, due to n ( (cid:96) − 1) > n ( (cid:96) ) .
Revised: GraphSAGE is in general less expensive than GAT, due to n ( (cid:96) − 1) > n ( (cid:96) ) .

Original: In addition, we note that for all architectures, C ( (cid:96) ) ∗ grows proportionally with n ( (cid:96) ) and m ( (cid:96) ) .
Revised: In addition, we note that for all architectures, C ( (cid:96) ) ∗ grows proportionally with n ( (cid:96) ) and m ( (cid:96) ) .

Original: For the normal GNN architecture, since we are using the full (cid:96) -hop neighborhood for each node, the value of n ( (cid:96) ) and m ( (cid:96) ) may grow exponentially with (cid:96) .
Revised: For the normal GNN architecture, since we are using the full (cid:96) -hop neighborhood for each node, the value of n ( (cid:96) ) and m ( (cid:96) ) may grow exponentially with (cid:96) .

Original: This is the “neighbor explosion” phenonemon and is the root cause of the high inference cost of the Table 1 baselines.
Revised: This is the “neighbor explosion” phenonemon and is the root cause of the high inference cost of the Table 1 baselines.

Original: For SHA D OW -GNN, suppose the subgraph contains n nodes and m edges.
Revised: For SHA D OW -GNN, suppose the subgraph contains n nodes and m edges.

Original: Then n ( (cid:96) ) = n and m ( (cid:96) )
Revised: Then n ( (cid:96) ) = n and m ( (cid:96) )

Original: = m .
Revised: = m .

Original: The inference cost of any SHA D OW -GNN is ensured to only grow linearly with the depth of the GNN.
Revised: The inference cost of any SHA D OW -GNN is ensured to only grow linearly with the depth of the GNN.

Original: C DETAILED E XPERIMENTAL S ETUP C. A DDITIONAL D ATASET D ETAILS 
Revised: C D ETAILS ON S AMPLERS

Revised: Here we present more detailed discussion on the proposed k -hop and PPR samplers. k -hop sampler. While previous works Hamilton et al. (2017a);

Revised: Ying et al.

Revised: (2018) have proposed to randomly sub-sample the multi-hop neighborhood to improve GNN computation efﬁciency, none of them explicitly set the GNN depth to be larger than k .

Revised: For example, GraphSAGE also uses a k -hop sampler.

Revised: In their case, the sampling improves the computation speed at the cost of accuracy drop.

Revised: In our case, such a sampling process can lead to both accuracy gain and computation efﬁciency improvement – This is partially due to the exclusion of noisy neighbors by the sampler, and partially due to the compensation of the information loss by the high expressivity of deep models.

Revised: More importantly, in the GraphSAGE design, the sampling depth k grows with the model depth.

Revised: Thus, “neighbor explosion” is inevitable unless the sampling budget b is set to 1 (in such a case, the sampler will likely return a path, which does not contain much information).

Revised: Another difference between our k -hop sampler and that of GraphSAGE is that SHA D OW -GNN runs on the induced subgraph of the k -hop neighbors.

Revised: Compared with the normal GraphSAGE, a SHA D OW -SAGE is able to discover more complicated structural information from the subgraph constructed by our k -hop sampler (See Theorem 3.3).

Revised: PPR sampler.

Revised: The approximate algorithm proposed in Andersen et al.

Revised: (2006) is both scalable and efﬁcient.

Revised: The number of nodes we need to visit to obtain the approximate PPR vector is much smaller than the graph size.

Revised: In addition, each visit only involves a scalar update, which is orders of magnitude cheaper than the cost of neighbor propagation in a GNN.

Revised: Quantitatively, for the three largest datasets, Reddit , Yelp and ogbn-products , for each target node on average, the number of nodes touched by the PPR computation is comparable to the full 2-hop neighborhood size, as shown in Table 4.

Revised: [Table]

Revised: D D ETAILED E XPERIMENTAL S ETUP D. A DDITIONAL D ATASET D ETAILS

Original: The statistics for the ﬁve benchmark graphs is listed in Table 3.
Revised: The statistics for the ﬁve benchmark graphs is listed in Table 5.

Original: Note that for Yelp , each node may have multiple labels, and thus we follow the original paper (Zeng et al., 2020) to report its
Revised: Note that for Yelp , each node may have multiple labels, and thus we follow the original paper (Zeng et al., 2020) to report its

Original: F1-micro score.
Revised: F1-micro score.

Original: For all the other graphs, a node is only associated with a single label, and so we report accuracy.
Revised: For all the other graphs, a node is only associated with a single label, and so we report accuracy.

Original: Note that for Reddit and Flickr , other papers (Hamilton et al., 2017a; Zeng et al., 2020) also report F1-micro score as the metric.
Revised: Note that for Reddit and Flickr , other papers (Hamilton et al., 2017a; Zeng et al., 2020) also report F1-micro score as the metric.

Original: However, since each node only has a single label, “F1-micro score” is exactly the same as “accuracy”.
Revised: However, since each node only has a single label, “F1-micro score” is exactly the same as “accuracy”.

Original: [Table]
Revised: [Table]

Original: C. H ARDWARE S PECIFICATION AND E NVIRONMENT
Revised: D. H ARDWARE S PECIFICATION AND E NVIRONMENT

Original: We run our experiments on a DGX1 machine with Dual 20-core Intel Xeon CPUs (E5-2698 v @ 2.2Ghz) and eight NVIDIA Tesla P100 GPUs (15GB of HBM2 memory).
Revised: We run our experiments on a DGX1 machine with Dual 20-core Intel Xeon CPUs (E5-2698 v @ 2.2Ghz) and eight NVIDIA Tesla P100 GPUs (15GB of HBM2 memory).

Original: The main memory is 512GB DDR4 memory.
Revised: The main memory is 512GB DDR4 memory.

Original: The code is written in Python 3.6.8 (where the sampling part is written with C++ parallelized by OpenMP , and the interface between C++ and Python is via PyBind11 ).
Revised: The code is written in Python 3.6.8 (where the sampling part is written with C++ parallelized by OpenMP , and the interface between C++ and Python is via PyBind11 ).

Original: We use PyTorch 1.4.0 on CUDA 10.2 with CUDNN 7.2.1 to train the model on GPU.
Revised: We use PyTorch 1.4.0 on CUDA 10.2 with CUDNN 7.2.1 to train the model on GPU.

Original: When measuring the execution time of Figure 2 and 3, we use the CPU (40-cores in total) to sample subgraphs in parallel and the GPU to execute the GNN layer operations.
Revised: When measuring the execution time of Figure 2 and 3, we use the CPU (40-cores in total) to sample subgraphs in parallel and the GPU to execute the GNN layer operations.

Original: C. H YPERPARAMETER S EARCHING P ROCEDURE
Revised: D. H YPERPARAMETER S EARCHING P ROCEDURE

Original: For all the experiments, as described in the “setup” paragraph of Section 4, we set the hidden dimension to be always d ( (cid:96) )
Revised: For all the experiments, as described in the “setup” paragraph of Section 4, we set the hidden dimension to be always d ( (cid:96) )

Original: = 256 .
Revised: = 256 .

Original: In addition, for all the GAT and SHA D OW -GAT experiments, we set the number of attention heads to be t = 4 .
Revised: In addition, for all the GAT and SHA D OW -GAT experiments, we set the number of attention heads to be t = 4 .

Original: For all the GIN and SHA D OW -GIN experiments, we use a 2-layer MLP (with hidden dimension 256 ) to perform the injective mapping in each GIN layer.
Revised: For all the GIN and SHA D OW -GIN experiments, we use a 2-layer MLP (with hidden dimension 256 ) to perform the injective mapping in each GIN layer.

Original: For all the JK-Net and SHA D OW -JK experiments, we use the concatenation operation to aggregate the hidden features of each layer in the JK layer.
Revised: For all the JK-Net and SHA D OW -JK experiments, we use the concatenation operation to aggregate the hidden features of each layer in the JK layer.

Original: For all the baseline and SHA D OW -GNN experiments, we use Adam optimizer (Kingma & Ba, 2014).
Revised: For all the baseline and SHA D OW -GNN experiments, we use Adam optimizer (Kingma & Ba, 2014).

Original: We perform grid search on the hyperparameter space deﬁned by:
Revised: We perform grid search on the hyperparameter space deﬁned by:

Original: • Activation function: { ReLU , ELU } • Dropout: { 0 . 00 , 0 .
Revised: • Activation function: { ReLU , ELU } • Dropout: { 0 . 00 , 0 .

Original: 50 } • Dropout: { 0 . 00 , 0 . 
Revised: 50 } • DropEdge: { 0 . 00 , 0 . 05 , 0 . 10 , 0 . 15 , 0 . 20 , 0 . 25 , 0 . 30 , 0 . 35 , 0 .

Original: 50 } • Learning rate: { 0 . 01 , 0 . 002 , 0 . 001 , 0 . 0002 , 0 . 0001 , 0 . 00002 }
Revised: 50 } • Learning rate: { 0 . 01 , 0 . 002 , 0 . 001 , 0 . 0002 , 0 . 0001 , 0 . 00002 }

Original: The sampling hyperparameters are tuned as follows.
Revised: The sampling hyperparameters are tuned as follows.

Original: For the PPR sampler, we consider two versions: one based on ﬁxed sampling budget p and the other based on PPR score thresholding θ .
Revised: For the PPR sampler, we consider two versions: one based on ﬁxed sampling budget p and the other based on PPR score thresholding θ .

Original: • If with ﬁxed budget, we then we disable the θ thresholding.
Revised: • If with ﬁxed budget, we then we disable the θ thresholding.

Original: We tune the budget by  { 100 , 125 , 150 , 175 , 200 } .
Revised: We tune the budget by θ ∈ { 100 , 125 , 150 , 175 , 200 } .

Original: • If with thresholding, we set θ ∈ { 0 . 01 , 0 . 05 } .
Revised: • If with thresholding, we set θ ∈ { 0 . 01 , 0 . 05 } .

Original: We still have an upper bound p on the subgraph size.
Revised: We still have an upper bound p on the subgraph size.

Original: So if there are q nodes in the neighborhood with PPR score larger than θ , the ﬁnal subgraph size would be max { p, q } .
Revised: So if there are q nodes in the neighborhood with PPR score larger than θ , the ﬁnal subgraph size would be max { p, q } .

Original: Such an upper bound eliminates the corner cases which may cause hardware inefﬁciency due to very large q .
Revised: Such an upper bound eliminates the corner cases which may cause hardware inefﬁciency due to very large q .

Original: In this case, we set pto be either 200 or 500 depending on the graph.
Revised: In this case, we set p

Original: For the k -hop sampler, we deﬁne the hyperparameter space as:
Revised: For the k -hop sampler, we deﬁne the hyperparameter space as:

Original: • Depth k =• Budget b ∈ { 5 , 10 , 15 , 20 }
Revised: • Depth k =• Budget b ∈ { 5 , 10 , 15 , 20 }

Original:  The hyperparameters to reproduce the Table 1 SHA D OW -GNN results are listed in Tables 4 and 5.
Revised: [Table] [Table] The hyperparameters to reproduce the Table 1 SHA D OW -GNN results are listed in Tables 6 and 7.

Original: The code will be shared with the reviewers at the end of the rebuttal period and released to public after paper acceptance.[Table] [Table]
Revised: The code will be shared with the reviewers at the end of the rebuttal period and released to public after paper acceptance.

Original: Flickr Reddit Yelp ogbnarxiv ogbnproducts 0 .0 .0 .0 .
Revised: Flickr Reddit Yelp ogbnarxiv ogbnproducts 0 .0 .0 .0 .

Original: R a ti o o f nod e
Revised: R a ti o o f nod e

Original: s i n V [ v ] k
Revised: s i n V [ v ] k

Original: =
Revised: =

Original: k
Revised: k

Original: =
Revised: =

Original: k
Revised: k

Original: =
Revised: =

Original: k ≥
Revised: k ≥

Original: Flickr Reddit Yelp ogbnarxiv ogbnproducts 0 .0 .0 .0 .
Revised: Flickr Reddit Yelp ogbnarxiv ogbnproducts 0 .0 .0 .0 .

Original: R a ti o o f nod e
Revised: R a ti o o f nod e

Original: s i n V [ v ] k
Revised: s i n V [ v ] k

Original: =
Revised: =

Original: k
Revised: k

Original: =k 
Revised: =k =k =

Original: =k
Revised: We compute the SGC model as Y = softmax (cid:16) (cid:101) A K XW (cid:17) where (cid:101) A = (cid:101) D − 12 (cid:101)

Original:  = 
Revised: A (cid:101) D − 12 and ˜ A = I + A . Matrix W is the only learnable parameter. K is the power on the adjacency matrix and we vary it from 1 to 40 in the experiments.

Revised: For SHA D OW -SGC, we compute the embedding for target v as y v = (cid:104) softmax (cid:16) (cid:101)

Revised: A K [ v ] X

Revised: [ v ] W (cid:17)(cid:105) v, : .

Revised: SGC and SHA D OW -SGC are trained with the same hyperparameters ( i.e. , learning rate, dropout).

Revised: SHA D OW -SGC uses the same sampler as the SHA D OW -GCN model in Table 1.

Revised: In the legend of

Revised: Figure 4, due to lack of space, we use S -SGC to denote SHA D OW -SGC.

Revised: “ F ”, “ R ” and “ A ” denote the datasets of Flickr , Reddit and ogbn-arxiv respectively.

Original: C. A DDITIONAL E XPERIMENTAL R ESULTS
Revised: E A DDITIONAL E XPERIMENTAL R ESULTS

Original: Is the PPR sampler really “shallow”?
Revised: Is the PPR sampler really “shallow”?

Original: The PPR sampler deﬁnes the neighborhood V [ v ] based on the magnitude of the PPR scores (cid:101) π v .
Revised: The PPR sampler deﬁnes the neighborhood V [ v ] based on the magnitude of the PPR scores (cid:101) π v .

Original: Although it is possible that a node far away from the target v may have a high PPR score and be selected by SAMPLE , such a case rarely happens in practice.
Revised: Although it is possible that a node far away from the target v may have a high PPR score and be selected by SAMPLE , such a case rarely happens in practice.

Original: Most of the nodes in V [ v ] are concentrated within the 2-hop neighborhood of v .
Revised: Most of the nodes in V [ v ] are concentrated within the 2-hop neighborhood of v .

Original: Let dist ( v, u ) denote the shortest path distance between nodes u and v .
Revised: Let dist ( v, u ) denote the shortest path distance between nodes u and v .

Original: Figure 5 shows the distribution of dist ( v, u ) for u ∈ V [ v ] .
Revised: Figure 6 shows the distribution of dist ( v, u ) for u ∈ V [ v ] .

Original: On the other hand, Figure 6 shows the composition of the neighborhood under the normal GNN.
Revised: On the other hand, Figure 7 shows the composition of the neighborhood under the normal GNN.

Original: It is clear that most of the nodes propagated by the normal GNN are far away from the target node.[Figure] [Figure]
Revised: It is clear that most of the nodes propagated by the normal GNN are far away from the target node.

Original: Epochs
Revised: Epochs

Original: V a li d a ti on
Revised: V a li d a ti on

Original: A cc u r ac y
Revised: A cc u r ac y

Original: PPR 1 -hop
Revised: PPR 1 -hop

Revised: PPR+ 1 -hop

Revised: Example on subgraph ensemble.

Original: PPR+
Revised: In addition to the Table 3 results on SHA D OW -GCN, Figure 8 further shows how subgraph ensemble improves convergence quality with the SHA D OW -SAGE model. We consider a PPR sampler with ﬁxed threshold θ = 0 . 01 , a 1 -hop sampler (which returns the full 1-hop neighbors) and the ensemble of these two. The 1 -hop sampler incurs signiﬁcant information loss since it preserves no multi-hop neighbors.

Original: [Figure]
Revised: As expected, its accuracy is low. Surprisingly, we observe accuracy gain when we ensemble the 1 -hop sampler and the PPR sampler by Equation 3. As discussed, different samplers preserve different kinds of local information. Rather than trying to design one “perfect” sampler, it may be more reasonable to ensemble a few simpler ones.