Original: G RAPH - LESS N EURAL N ETWORKS : T EACHING O LD MLP S N EW T RICKS VIA D ISTILLATION
Revised: G RAPH - LESS N EURAL N ETWORKS : T EACHING O LD MLP S N EW T RICKS VIA D ISTILLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Neural Networks (GNNs) have recently become popular for graph machine learning and have shown great results on wide node classiﬁcation tasks.
Revised: Graph Neural Networks GNNs) have recently become popular for graph machine learning and have shown great results on wide node classiﬁcation tasks.

Original: Yet,
Revised: Yet

Original: GNNs are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency.
Revised: GNNs are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency.

Original: Namely, GNN inference depends on neighbor nodes multiple hops away from the target, and fetching these nodes burdens latency-constrained applications.
Revised: Namely GNN inference depends on neighbor nodes multiple hops away from the target and fetching these nodes burdens latency-constrained applications.

Original: Existing inference acceleration methods like pruning and quantization can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation (MAC) operations.
Revised: Existing inference acceleration methods like pruning and quantization can speed up GNNs to some extent by reducing Multiplication -and- ACcumulation

Revised: MAC

Original:  However, their improvements are limited given the data dependency is not resolved.
Revised: ) operations. However, their improve - ments are limited given the data dependency is not resolved.

Original: Conversely, multi-layer perceptrons (MLPs) have no dependency on graph data and infer much faster than GNNs, even though they are less accurate than GNNs for node classiﬁcation in general.
Revised: Conversely, multi- layer perceptrons (MLPs) have no dependency on graph data and infer much faster than GNNs , even though they are less accurate than GNNs for node classiﬁcation in general

Original: Motivated by these complementary strengths and weaknesses, we bring GNNs and MLPs together via knowledge distillation (KD).
Revised: Motivated by these complementary strengths and weaknesses we bring GNNs and MLPs together via knowledge distillation ( KD)

Original: Our work shows that the performance of MLPs can be improved by large margins with GNN KD.
Revised: Our work shows that the performance of MLPs can be improved by large margins with GNN KD.

Original: We call the distilled MLPs Graph-less Neural Networks ( GLNN s) as they have no inference graph dependency.
Revised: We call the distilled MLPs Graph-less Neural Networks ( GLNN s) as they have no inference graph dependency.

Original: We show that GLNN s with competitive performance infer faster than GNNs by 146×-273× and faster than other acceleration methods by 14×-27×. Meanwhile, under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand-alone
Revised: We show that GLNN s with competitive performance infer faster than GNNs by 146×-273× and faster than other acceleration methods by 14×-27×. Meanwhile, under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand alone MLPs by

Original: MLPs by 12.36% on average and match GNNs on 6/7 datasets.
Revised: .36% on average and match GNNs on 6/7 datasets.

Original: A comprehensive analysis of GLNN shows when and why GLNN s can achieve competitive results to
Revised: A comprehensive analysis of GLNN shows when and why GLNN s can achieve competitive results to

Original: GNNs and suggests GLNN as a handy choice for latency-constrained applications.
Revised: GNNs and suggests GLNN as a handy choice for latency- constrained applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) have recently become very popular for graph machine learning (GML) research and have shown great results on node classiﬁcation tasks (Kipf & Welling, 2016;
Revised: Graph Neural Networks (GNNs) have recently become very popular for graph machine learning (GML) research and have shown great results on node classiﬁcation tasks (Kipf & Welling, 2016;

Original: Hamilton et al., 2017; Veliˇckovi´c et al., 2017) like product prediction on co-purchasing graphs and paper category prediction on citation graphs.
Revised: Hamilton et al., 2017; Veliˇckovi´c et al., 2017) like product prediction on co-purchasing graphs and paper category prediction on citation graphs.

Original: However, for large-scale industrial applications, MLPs remain the major workhorse, despite common (implicit) underlying graphs and suitability for GML formalisms.
Revised: However, for large-scale industrial applications, MLPs remain the major workhorse, despite common (implicit) underlying graphs and suitability for GML formalisms.

Original: One reason for this academic-industrial gap is the challenges in scalability and deployment brought by data dependency in GNNs (Zhang et al., 2020; Jia et al., 2020), which makes GNNs hard to deploy for latency-constraint applications that require fast inference.
Revised: One reason for this academic-industrial gap is the challenges in scalability and deployment brought by data dependency in GNNs (Zhang et al., 2020; Jia et al., 2020), which makes GNNs hard to deploy for latency-constraint applications that require fast inference.

Original: Neighborhood fetching caused by graph dependency is one of the major sources of GNN latency.
Revised: Neighborhood fetching caused by graph dependency is one of the major sources of GNN latency.

Original: Inference on a target node necessitates fetching topology and features of many neighbor nodes, especially on small-world graphs (detailed discussion in Section 4).
Revised: Inference on a target node necessitates fetching topology and features of many neighbor nodes, especially on small-world graphs (detailed discussion in Section 4).

Original: Common inference acceleration techniques like pruning (Zhou et al., 2021) and quantization (Tailor et al., 2021; Zhao et al., 2020) can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation (MAC) operations.
Revised: Common inference acceleration techniques like pruning (Zhou et al., 2021) and quantization (Tailor et al., 2021; Zhao et al., 2020) can speed up GNNs to some extent by reducing Multiplication-and-ACcumulation (MAC) operations.

Original: However, their improvements are limited given the graph dependency is not resolved.
Revised: However, their improvements are limited given the graph dependency is not resolved.

Original: Unlike GNNs,
Revised: Unlike GNNs,

Original: MLPs have no dependency on graph data and are easier to deploy than GNNs.
Revised: MLPs have no dependency on graph data and are easier to deploy than GNNs.

Original: They also enjoy the auxiliary beneﬁt of sidestepping the cold-start problem that often happens during the online prediction of relational data (Wei et al., 2020), meaning MLPs can infer reasonably even when neighbor information of a new encountered node is not immediately available.
Revised: They also enjoy the auxiliary beneﬁt of sidestepping the cold-start problem that often happens during the online prediction of relational data (Wei et al., 2020), meaning MLPs can infer reasonably even when neighbor information of a new encountered node is not immediately available.

Original: On the other hand, this lack of graph dependency typically hurts for relational learning tasks, limiting MLP performance on GML tasks compared to GNNs.
Revised: On the other hand, this lack of graph dependency typically hurts for relational learning tasks, limiting MLP performance on GML tasks compared to GNNs.

Original: We thus ask: can we bridge the two worlds, enjoy the low-latency and dependency-free nature of MLPs and the graph context awareness of GNNs at the same time?
Revised: We thus ask: can we bridge the two worlds, enjoy the low-latency and dependency-free nature of MLPs and the graph context awareness of GNNs at the same time?

Original: Present work.
Revised: Present work.

Original: Our key ﬁnding is that it is possible to distill knowledge from GNNs to MLPs without losing signiﬁcant performance, but reducing the inference time drastically for node classiﬁcation.
Revised: Our key ﬁnding is that it is possible to distill knowledge from GNNs to MLPs without losing signiﬁcant performance, but reducing the inference time drastically for node classiﬁcation.

Original: The knowledge distillation (KD) can be done ofﬂine, coupled with model training.
Revised: The knowledge distillation (KD) can be done ofﬂine, coupled with model training.

Original: In other words, we can shift considerable work from the latency-constrained inference step, where time reduction in milliseconds makes a huge difference, to the less time-sensitive training step, where time cost in hours or days is often tolerable.
Revised: In other words, we can shift considerable work from the latency-constrained inference step, where time reduction in milliseconds makes a huge difference, to the less time-sensitive training step, where time cost in hours or days is often tolerable.

Original: We call our approach Graph-less Neural Network ( GLNN ).
Revised: We call our approach Graph-less Neural Network ( GLNN ).

Original: Speciﬁcally, GLNN is a modeling paradigm involving KD from a GNN teacher to a student MLP; the resulting GLNN is an MLP optimized through KD, so it enjoys the beneﬁts of graph contextawareness in training but has no graph dependency in inference.
Revised: Speciﬁcally, GLNN is a modeling paradigm involving KD from a GNN teacher to a student MLP; the resulting GLNN is an MLP optimized through KD, so it enjoys the beneﬁts of graph context-awareness in training but has no graph dependency in inference.

Original: Regarding speed, GLNN s have superior efﬁciency and are 146×-273× faster than GNNs and 14×-27× faster than other inference acceleration methods.
Revised: Regarding speed, GLNN s have superior efﬁciency and are 146×-273× faster than GNNs and 14×-27× faster than other inference acceleration methods.

Original: Regarding performance, under a production setting involving both transductive and inductive predictions on 7 datasets, GLNN accuracies improve over MLPs by 12.36% on average and match GNNs on 6/7 datasets.
Revised: Regarding performance, under a production setting involving both transductive and inductive predictions ondatasets, GLNN accuracies improve over MLPs by 12.36% on average and match GNNs on 6/7 datasets.

Original: We comprehensively study when and why GLNN s can achieve competitive results as GNNs.
Revised: We comprehensively study when and why GLNN s can achieve competitive results as GNNs.

Original: Our analysis suggests the critical factors for such great performance are large MLP sizes and high mutual information between node features and labels.
Revised: Our analysis suggests the critical factors for such great performance are large MLP sizes and high mutual information between node features and labels.

Original: Our observations align with recent results in vision and language, which posit that large enough (or slightly modiﬁed) MLPs can achieve similar results as CNNs and Transformers (Liu et al., 2021;
Revised: Our observations align with recent results in vision and language, which posit that large enough (or slightly modiﬁed) MLPs can achieve similar results as CNNs and Transformers (Liu et al., 2021;

Original: Tolstikhin et al., 2021; Melas-Kyriazi, 2021; Touvron et al., 2021; Ding et al., 2021).
Revised: Tolstikhin et al., 2021; Melas-Kyriazi, 2021; Touvron et al., 2021; Ding et al., 2021).

Original: Our core contributions are as follows:
Revised: Our core contributions are as follows:

Original: • We propose GLNN s, which eliminate the neighborhood-fetching latency in GNN inference viacross-model KD to MLPs.
Revised: • We propose GLNN , which eliminates neighbor-fetching latency in GNN inference via KD to MLP.

Original: • We show GLNN s has competitive performance as GNNs, while enjoying 239×-758× faster inference than vanilla GNNs and 27×-32× faster inference than other inference acceleration methods.
Revised: • We show GLNN s has competitive performance as GNNs, while enjoying 239×-758× faster inference than vanilla GNNs and 27×-32× faster inference than other inference acceleration methods.

Original: • We study GLNN properties comprehensively by investigating their performance under different settings, how they work as regularizers, their inductive bias, expressiveness, and limitations.
Revised: • We study GLNN properties comprehensively by investigating their performance under different settings, how they work as regularizers, their inductive bias, expressiveness, and limitations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: The idea of GNNs started with generalizing convolution to graphs (Bruna et al., 2014; Defferrard et al., 2017) and later simpliﬁed the to a message-passing neural network (MPNN) framework by GCN (Kipf & Welling, 2016).
Revised: The idea of GNNs started with generalizing convolution to graphs (Bruna et al., 2014; Defferrard et al., 2017) and later simpliﬁed the to a message-passing neural network (MPNN) framework by GCN (Kipf & Welling, 2016).

Original: The majority of GNNs after are build on GCN and can be put under MPNN.
Revised: The majority of GNNs after are build on GCN and can be put under MPNN.

Original: For example, GAT employs attention (Veliˇckovi´c et al., 2017), PPNP employs personalized PageRank (Klicpera et al., 2019), GCNII and DeeperGCN employ residual connections and dense connections (Chen et al., 2020; Li et al., 2019).
Revised: For example, GAT employs attention (Veliˇckovi´c et al., 2017), PPNP employs personalized PageRank (Klicpera et al., 2019), GCNII and DeeperGCN employ residual connections and dense connections (Chen et al., 2020; Li et al., 2019).

Original: Inference Acceleration.
Revised: Inference Acceleration.

Original: Inference accelerationschemes have been proposed by hardware improvements (Chen et al., 2016;
Revised: Inference acceleration have been proposed by hardware improvements (Chen et al., 2016;

Original: Judd et al., 2016) and algorithmic improvements through pruning (Han et al., 2015), quantization (Gupta et al., 2015), and KD (Hinton et al., 2015).
Revised: Judd et al., 2016) and algorithmic improvements through pruning (Han et al., 2015), quantization (Gupta et al., 2015), and KD (Hinton et al., 2015).

Original: Speciﬁcally for GNNs, pruning (Zhou et al., 2021) and quantizing GNN parameters (Tailor et al., 2021; Zhao et al., 2020), or distilling to smaller GNNs (Yang et al., 2021b; Yan et al., 2020; Yang et al., 2021a) have been studied.
Revised: Speciﬁcally for GNNs, pruning (Zhou et al., 2021) and quantizing GNN parameters (Tailor et al., 2021; Zhao et al., 2020) have been studied.

Original: These approaches speed up GNN inference to a certain extent but do not eliminate the neighborhood-fetching latency.
Revised: These approaches speed up GNN inference to a certain extent but do not eliminate the neighborhood-fetching latency.

Original: In contrast, our cross-model KD approach solves this issue.
Revised: In contrast, our cross-model KD approach solves this issue.

Original: Another line of research focuses on fast GNN training via sampling (Hamilton et al., 2017; Zou et al., 2019;
Revised: Another line of research focuses on fast GNN training via sampling (Hamilton et al., 2017; Zou et al., 2019;

Original: Chen et al., 2018).
Revised: Chen et al., 2018).

Original: They are different and complementary to our goal on inference acceleration.
Revised: They are different and complementary to our goal on inference acceleration.

Original: P RELIMINARIES
Revised: GNN distillation. Several previous works tried to distill large teacher GNNs to smaller student GNNs. We introduce these works and clarify the differences.

Revised: LSP (Yang et al., 2021b) and TinyGNN (Yan et al., 2020) are similar, in the sense that they both do KD while preserving local neighbor information.

Revised: The student in LSP and TinyGNN was a GNN with fewer parameters, but not necessarily fewer layers than the teacher GNN, which makes both designs still depend on the graph data and require latency-inducing fetches.

Revised: KD in GFKD is done via generating fake graphs (Deng & Zhang, 2021).

Revised: GFKD focused on graph-level prediction tasks, for which data instances are independent graphs.

Revised: In our case, we consider node-level tasks and focus on removing dependency between data at inference time.

Revised: GraphSAIL (Xu et al., 2020) uses KD for GNN incremental learning.

Revised: Instead of efﬁciency, the goal is to train a student that works well on new data while preserving the performance on old data via

Revised: KD.

Revised: CPF (Yang et al., 2021a) proposed to distill GNNs to a student that includes label propagation (LP) to improve the GNN performance.

Revised: KD in CPF is no longer to a GNN, but it is still heavily graph-dependent as it uses LP.

Revised: In contrast, we remove the graph dependency by KD to MLPs.

Revised:  P RELIMINARIES

Original: Notations.
Revised: Notations.

Original: For GML tasks, the input is usually a graph and its node features, which we write as
Revised: For GML tasks, the input is usually a graph and its node features, which we write as

Original: G = ( V , E ) , with V stands for all nodes, and E stands for all edges.
Revised: G =( V , E ) , with V stands for all nodes, and E stands for all edges.

Original: Let N denote the total number of nodes.
Revised: Let N denote the total number of nodes.

Original: We use X ∈ R N × D to represent node features, with row x v being the D -dimensional feature of node v ∈ V .
Revised: We use X ∈ R N × D to represent node features, with row x v being the D -dimensional feature of node v ∈V .

Original: We represent edges with an adjacency matrix A , with A u,v = 1 if edge ( u, v ) ∈ E , and 0 otherwise.
Revised: We represent edges with an adjacency matrix A , with A u,v =1 if edge ( u,v ) ∈E , and 0 otherwise.

Original: For node classiﬁcation, one of the most important GML applications, the prediction targets are Y ∈ R N × K , where row y v is a K -dim one-hot vector for node v .
Revised: For node classiﬁcation, one of the most important GML applications, the prediction targets are Y ∈ R N × K , where row y v is a K -dim one-hot vector for node v .

Original: For a given G , usually a small portion of nodes will be labeled, which we mark using superscript L , i.e. V L , X L , and Y L .
Revised: For a given G , usually a small portion of nodes will be labeled, which we mark using superscript L , i.e. V L , X L , and Y L .

Original: The majority of nodes will be unlabeled, and we mark using the superscript U , i.e. V U , X U , and Y U .
Revised: The majority of nodes will be unlabeled, and we mark using the superscript U , i.e. V U , X U , and Y U .

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: Most GNNs ﬁt under the message-passing framework, where the representation h v of each node v is updated iteratively in each layer by collecting messages from its neighbors denoted as N ( v ) .
Revised: Most GNNs ﬁt under the message-passing framework, where the representation h v of each node v is updated iteratively in each layer by collecting messages from its neighbors denoted as N ( v ) .

Original: For the l -th layer, h ( l ) v is obtained from the previous layer representation [Equation]
Revised: For the l -th layer, h ( l ) v is obtained from the previous layer representation h ( l − 1) u

Original: [Equation]

Original: ( h (0) u
Revised: ( h (0) u

Original: = x u ) via an aggregation operation AGGR followed by an UPDATE operation as [Equation] ∈ N 
Revised: = x u ) via an aggregation operation AGGR followed by an UPDATE operation as h ( l ) N ( v ) =

Original: [Equation] v [Equation] and 
Revised: AGGR ( { h ( l − 1) u : u ∈N ( v ) } ) and h ( l ) v = UPDATE ( h ( l ) N ( v ) , h ( l − 1) v

Original: [Equation]
Revised: [Equation]

Original:  M OTIVATION
Revised:  M OTIVATION

Original: As mentioned in Sec 1, GNNs can have considerable inference latency due totheir graph dependency.
Revised: As mentioned in Sec 1, GNNs can have considerable inference latency due to graph dependency.

Original: An additional GNN layer involves fetching one more hop of neighbor nodes.
Revised: One more GNN layer means fetching one more hop of neighbor nodes.

Original: To inferfor a single node with a L -layer GNN on a graph with average degree R will require O ( R L ) fetchings.
Revised: To infer a single node with a L -layer GNNonagraphwithaveragedegree R willrequire O ( R L ) fetches.

Original: R can be large for real-world graphs, e.g. 208 for the Twittersocial graph (Ching et al., 2015).
Revised: R canbelargeforreal-worldgraphs, e.g. 208 for the Twitter (Ching et al., 2015).

Original: Also, as fetching for successive layers must be done sequentially, the total latency explodes quickly as L increases.
Revised: Also, as fetching for successive layers must be done sequentially, the total latency explodes quickly as L increases.

Original: Figure 1 illustrates the successive dependence added by each GNN layer, and the exponential explosion in the number of nodes and inference time induced by multiple GNN layers.
Revised: Figure 1 illustrates the dependency added by each GNN layer and the exponential explosion in the number of fetches and inference time.

Original: In contrast, the inference time of an MLP with the same number of layers is much smaller and only grows linearly.
Revised: In contrast, the inference time of an MLP with the same number of layers is much smaller and only grows linearly.

Original: This marked gap contributes greatly to the practicality of MLPs in industrial applications over GNNs.
Revised: This marked gap contributes greatly to the practicality of MLPs in industrial applications over GNNs.

Original: The node-fetching latency is exacerbated by two factors: ﬁrstly, newer GNN architectures are getting deeper, e.g. 64 layers (Chen et al., 2020), 112 layers (Li et al., 2019), and even 1001 layers (Li et al., 2021).
Revised: The node-fetching latency is exacerbated by two factors: ﬁrstly, newer GNN architectures are getting deeper, e.g. 64 layers (Chen et al., 2020), 112 layers (Li et al., 2019), and even 1001 layers (Li et al., 2021).

Original: Secondly, industrial-scale graphs are frequently too large to ﬁt into the memory of a single machine, necessitating sharding of the graph out of the main memory.
Revised: Secondly, industrial-scale graphs are frequently too large to ﬁt into the memory of a single machine, necessitating sharding of the graph out of the main memory.

Original: For example, Twitter has 288M monthly active users (nodes) and an estimated 60B followers (edges) as of 3/2015.
Revised: For example, Twitter has 288M monthly active users (nodes) and an estimated 60B followers (edges) as of 3/2015.

Original: Facebook has 1.39B active users with more than 400B edges as of 12/2014 (Ching et al., 2015).
Revised: Facebook has 1.39B active users with more than 400B edges as of 12/2014 (Ching et al., 2015).

Original: Even when stored in a sparse-matrix-friendly format (often COO or CSR), these graphs are on the order of TBs and are constantly growing.
Revised: Even when stored in a sparse-matrix-friendly format (often COO or CSR), these graphs are on the order of TBs and are constantly growing.

Original: Moving away from in-memory storage results in even slower neighbor-fetching.
Revised: Moving away from in-memory storage results in even slower neighbor-fetching.

Original: MLPs, on the other hand, lack the means to exploit graph topology, which hurts their performance for node classiﬁcation.
Revised: MLPs, on the other hand, lack the means to exploit graph topology, which hurts their performance for node classiﬁcation.

Original: For example, test accuracy on Products is 78.61 for GraphSAGE compared to 62.47 for an equal-sized MLP.
Revised: For example, test accuracy on Products is 78.61 for GraphSAGE compared to 62.47 for an equal-sized MLP.

Original: Nonetheless, recent results in vision and language posit that large  (or slightly modiﬁed) MLPs can achieve similar results as CNNs and Transformers (Liu et al., 2021).
Revised: Nonetheless, recent results in vision and language posit that large [Figure] (or slightly modiﬁed) MLPs can achieve similar results as CNNs and Transformers ( et al.

Original: We thus also ask: Can we bridge the best of GNNs and MLPs to get high-accuracy and low-latency models?
Revised: We thus also ask: Can we bridge the best of GNNs and MLPs to get high-accuracy and low-latency models?

Original: This motivates us to do cross-model KD from GNNs to MLPs.
Revised: This motivates us to do cross-model KD from GNNs to MLPs.

Original:  G RAPH - LESS N EURAL N ETWORKS
Revised:  G RAPH - LESS N EURAL N ETWORKS

Original: In this section, we introduce GLNN s in details, clarify experimental settings for our investigations, and answerthe following exploration questions regarding GLNN properties: 1) How do GLNN s compare to MLPs and GNNs? 2) Can GLNN s work well under both transductive and inductive settings?
Revised: In this section, we introduce GLNN s in details and answer exploration questions regarding GLNN properties: 1) How do GLNN s compare to MLPs and GNNs? 2) Can GLNN s work well under both transductive and inductive settings?

Original:  How do GLNN s compare to other inference acceleration methods? 4)
Revised: 3) How do GLNN s compare to other inference acceleration methods? 4)

Original: How do GLNN s beneﬁt from KD?
Revised: How do GLNN s beneﬁt from KD?

Original: 5) Do GLNN s have sufﬁcient model expressiveness? 6) When will GLNN s fail to work?
Revised: 5) Do GLNN s have sufﬁcient model expressiveness? 6) When will GLNN s fail to work?

Original: [Figure]

Original: 5. T HE GLNN F RAMEWORK
Revised: 5. T HE GLNN F RAMEWORK

Original: The idea of GLNN is straightforward, yet as we will see, extremely effective.
Revised: The idea of GLNN is straightforward, yet as we will see, extremely effective.

Original: In short, we train a “boosted” MLP via KD from a teacher GNN.
Revised: In short, we train a “boosted” MLP via KD from a teacher GNN.

Original: KD was introduced in Hinton et al.
Revised: KD was introduced in Hinton et al.

Original: (2015), with the main idea being transferring knowledge  from a cumbersome teacher to a simpler student.
Revised: (2015), where knowledge was transferred from a cumbersome teacher to a simpler student.

Original: In our case, we generate soft targets z v for each node v with a teacher GNN.
Revised: In our case, we generate soft targets z v for each node v with a teacher GNN.

Original: Then we train the student MLP with boththe true label y v and z v .
Revised: Then we train a student MLP with both true labels y v and z v .

Original: The objective is as Equation  with λ being a  being entropy divergence.
Revised: The objective is as Equation 1, with λ being a weight parameter, L label being the cross-entropy between y v and student predictions ˆ y v , L teacher being the KL-divergence.

Original: [Equation]
Revised: [Equation]

Original: The model after KD, i.e. GLNN , is essentially a MLP.
Revised: The model after KD, i.e. GLNN , is essentially a MLP.

Original: Therefore, GLNN s have no graph dependency during inference and are as fast as MLPs.
Revised: Therefore, GLNN s have no graph dependency during inference and are as fast as MLPs.

Original: On the other hand, through ofﬂine KD, GLNN parameters are optimized to predict and generalize as well as GNNs, with the added beneﬁt of faster inference and easier deployment.
Revised: On the other hand, through ofﬂine KD, GLNN parameters are optimized to predict and generalize as well as GNNs, with the added beneﬁt of faster inference and easier deployment.

Original: In Figure 2, we show the ofﬂine KD and online inference steps of GLNN s.
Revised: In Figure 2, we show the ofﬂine KD and online inference steps of GLNN s.

Original: 5. E XPERIMENT S ETTINGS
Revised: 5. E XPERIMENT S ETTINGS

Original: Datasets.
Revised: Datasets.

Original: We consider all ﬁve datasets used in the CPF paper (Yang et al., 2021a), i.e. Cora ,
Revised: We consider all ﬁve datasets used in the CPF paper (Yang et al., 2021a), i.e. Cora ,

Original: Citeseer , Pubmed , A-computer , and A-photo .
Revised: Citeseer , Pubmed , A-computer , and A-photo .

Original: To fully evaluate our method, we also include two more larger OGB datasets (Hu et al., 2020), i.e. Arxiv and Products .
Revised: To fully evaluate our method, we also include two more larger OGB datasets (Hu et al., 2020), i.e. Arxiv and Products .

Original: Model Architectures.
Revised: Model Architectures.

Original: For consistent results, we use GraphSAGE (Hamilton et al., 2017) with GCN aggregation as the teacher.
Revised: For consistent results, we use GraphSAGE (Hamilton et al., 2017) with GCN aggregation as the teacher.

Original: We conduct ablation studies of other GNN teachers like GCN (Kipf &
Revised: We conduct ablation studies of other GNN teachers like GCN (Kipf &

Original: Welling, 2016), GAT (Veliˇckovi´c et al., 2017) and, APPNP (Klicpera et al., 2019) in Section 6.
Revised: Welling, 2016), GAT (Veliˇckovi´c et al., 2017) and, APPNP (Klicpera et al., 2019) in Section 6.

Original: Evaluation Protocol.
Revised: Evaluation Protocol.

Original: For all experiments in this section, we report the average and standard deviation over ten runs with different random seeds.
Revised: For all experiments in this section, we report the average and standard deviation over ten runs with different random seeds.

Original: Model performance is measured as accuracy, and results are reported on test data with the best model selected using validation data. 
Revised: Model performance is measured as accuracy, and results are reported on test data with the best model selected using validation data. [Table] Datasets [Table]

Revised: Products

Revised: [Table]

Original: Transductive vs. Inductive.
Revised: Transductive vs. Inductive.

Original: Given G , X , and Y L , we divide the node classiﬁcation goal into two settings: transductive ( tran ) and inductive ( ind ).
Revised: Given G , X , and Y L , we consider node classiﬁcation under two settings: transductive ( tran ) and inductive ( ind ).

Original: For the inductive setting, we hold out part of the test data for inductive evaluation only.
Revised: For ind , we hold out some test data for inductive evaluation only.

Original: We ﬁrst select inductive nodes V Uind ⊂ V U to hold out, which partitions V U into the disjoint inductive subset and observed subset, i.e. V U
Revised: We ﬁrst select inductive nodes V Uind ⊂V U , which partitions V U into the disjoint inductive subset and observed subset, i.e. V U

Original: = V Uobs (cid:116) V Uind .
Revised: = V Uobs (cid:116)V Uind .

Original: Then we hold out  allthe edges connected to nodes in V Uind as well. Therefore, we end up with two disjoint graphs G =
Revised: Then we hold out v ∈V Uind and all edges connected to v ∈V Uind , which leads to two disjoint graphs G =

Original: G obs (cid:116) G ind with  shared nodes or edges.
Revised: G obs (cid:116)G ind with no shared nodes or edges.

Original: Node features and labels are partitioned into three disjoint sets, i.e. X =
Revised: Node features and labels are partitioned into three disjoint sets, i.e. X =

Original: X L
Revised: X L

Original: (cid:116) X Uobs (cid:116)
Revised: (cid:116) X Uobs (cid:116)

Original: X Uind , and Y =
Revised: X Uind , and Y =

Original: Y L (cid:116)
Revised: Y L (cid:116)

Original: Y Uobs (cid:116)
Revised: Y Uobs (cid:116)

Original: Y Uind .
Revised: Y Uind .

Original: Concretely, the input/output of both settings becomes
Revised: Concretely, the input/output of both settings become:

Original: • tran : train on G , X , and Y L ; evaluate on ( X U , Y U ) ; KD uses z v for v ∈ V .
Revised: • tran : train on G , X , and Y L ; evaluate on ( X U , Y U ) ; KD uses z v for v ∈V .

Original: • ind : train on G obs , X L , X Uobs , and Y L ; evaluate on ( X Uind , Y Uind ) ; KD uses z v for v ∈ V L (cid:116) V Uobs .
Revised: • ind : train on G obs , X L , X Uobs , and Y L ; evaluate on ( X Uind , Y Uind ) ; KD uses z v for v ∈V L (cid:116)V Uobs .

Revised: Note that for tran , all the nodes in the graph including the validation and test nodes are used for z generation.

Original: Further details of data split, hyperparameters, and model settings aregiven in Appendix A.[Table] [Table]
Revised: A discussion of this choice along with other experiment details are in Appendix A.

Original: 5. H OW DO GLNN S COMPARE TO MLP S AND GNN S?
Revised: 5. H OW DO GLNN S COMPARE TO MLP S AND GNN S

Original: We start by comparing GLNN s to equal-sized MLPs and GNNs.
Revised: We start by comparing GLNN s to equal-sized MLPs and GNNs.

Original: Speciﬁcally, the same number of layers and hidden dimensions.
Revised: Speciﬁcally, the same number of layers and hidden dimensions.

Original: We ﬁrst study the transductive setting, i.e. the standard setting for all used datasets in past works, so our experiment results in Table 1 can be directly compared to the results reported in previous literature like Yang et al.
Revised: We ﬁrst study the transductive setting, i.e. the standard setting for all used datasets in past works, so our experiment results in Table 1 can be directly compared to the results reported in previous literature like Yang et al.

Original: (2021a)
Revised: (2021a)

Original: and Hu et al.
Revised: and Hu et al.

Original: As shown in Table 1, the performance of all GLNN s improve over MLPs by large margins.
Revised: As shown in Table 1, the performance of all GLNN s improve over MLPs by large margins.

Original: On smaller datasets (ﬁrst 5 rows), GLNN s can even outperform the teacher GNNs.
Revised: On smaller datasets (ﬁrst 5 rows), GLNN s can even outperform the teacher GNNs.

Original: In other words, for each of these tasks, with the same parameter budget, there exists a set of MLP parameters that has GNN-competitive performance. We investigate the rationale in Sections 5.6 and 5.7.
Revised: In other words, for each task, with the same parameter budget, there exists a set of MLP parameters that has GNN-competitive performance (detailed discussion in Sections 5.6 and 5.7).

Original: For the larger OGB datasets (last 2 rows), the GLNN performance is improved over MLPs but still worse than the teacher GNNs.
Revised: For the larger OGB datasets (last 2 rows), the GLNN performance is improved over MLPs but still worse than the teacher GNNs.

Original: However, as we show in Table 2, this gap can be mitigated by increasing MLP size to MLPw i 1 .
Revised: However, as we show in Table 2, this gap can be mitigated by increasing MLP size to MLPw i 1 .

Original: In Figure 3 (right), we visualize the trade-off between prediction accuracy and model inference time with different model sizes.
Revised: In Figure 3 (right), we visualize the trade-off between prediction accuracy and model inference time with different model sizes.

Original: We show that gradually increasing GLNN size pushes its performance to be close to SAGE.
Revised: We show that gradually increasing GLNN size pushes its performance to be close to SAGE.

Original: On the other hand, when we reduce the number of layers of SAGE 2 , the accuracy quickly drops to be worse than GLNN s. A detailed discussion of the rationale for increasing MLP sizes is in Appendix B.
Revised: On the other hand, when we reduce the number of layers of SAGE 2 , the accuracy quickly drops to be worse than GLNN s. A detailed discussion of the rationale for increasing MLP sizes is in Appendix B.

Original: 5. C AN GLNN S WORK WELL UNDER BOTH TRANSDUCTIVE AND INDUCTIVE SETTINGS ?
Revised: 5. C AN GLNN S WORK WELL UNDER BOTH TRANSDUCTIVE AND INDUCTIVE SETTINGS ?

Original: Although transductive is the commonly studied setting for node classiﬁcation, it does not encompass prediction on unseen nodes.
Revised: Although transductive is the commonly studied setting for node classiﬁcation, it does not encompass prediction on unseen nodes.

Original: Therefore, it may not be the best way to evaluate a deployed model, which must often generate predictions for new data points as well as reliably maintain performance on old  ones.
Revised: Therefore, it may not be the best way to evaluate a deployed model, which must often generate predictions for new data points as well as reliably maintain performance on old [Figure] ones.

Original: Thus, to better understand the effectiveness of GLNN , we also consider their performance under a realistic production setting, which contains both transductive and inductive predictions.
Revised: Thus, to better understand the effectiveness of GLNN , we also consider their performance under a realistic production setting, which contains both transductive and inductive predictions.

Original: To evaluate a model inductively, we hold out some unlabeled test nodes from training to form an inductive subset, i.e. V U
Revised: To evaluate a model inductively, we hold out some unlabeled test nodes from training to form an inductive subset, i.e. V U

Original: = V Uobs (cid:116) V Uind .
Revised: = V Uobs (cid:116) V Uind .

Original: In production, a model might be re-trained periodically, e.g. weekly.
Revised: In production, a model might be re-trained periodically, e.g. weekly.

Original: The hold-out nodes in V Uind represent new nodes that entered the graph between two times of training.
Revised: The hold-out nodes in V Uind represent new nodes that entered the graph between two times of training.

Original: V Uind is usually small compared to the size of V Uobs – e.g. Graham (2012) estimates 5-7% for the fastest-growing tech startups.
Revised: V Uind is usually small compared to the size of V Uobs – e.g. Graham (2012) estimates 5-7% for the fastest-growing tech startups.

Original: In our case, to mitigate randomness and better evaluate generalizability, we consider a larger V U ind containing 20% of the test data.
Revised: In our case, to mitigate randomness and better evaluate generalizability, we consider a larger V Uind containing 20% of the test data.

Original: We also evaluate on V Uobs containing the other 80% of the test data, representing the standard transductive prediction on existing unlabeled nodes, since inference is commonly redone on existing nodes in real-world cases.[Figure] [Table]
Revised: We also evaluate on V Uobs containing the other 80% of the test data, representing the standard transductive prediction on existing unlabeled nodes, since inference is commonly redone on existing nodes in real-world cases.

Original: We report both results and a interpolated production ( prod ) results per dataset in Table 3.
Revised: We report both results and a interpolated production ( prod ) results per dataset in Table 3.

Original: The prod results paint a clearer picture of model generalization as well as accuracy in production settings.
Revised: The prod results paint a clearer picture of model generalization as well as accuracy in production settings.

Original: We also do an ablation study of different inductive split ratios other than 20-80 in Section 6.
Revised: We also do an ablation study of different inductive split ratios other than 20-80 in Section 6.

Original: In Table 3, we see that GLNN s can still improve over MLP by large margins for inductive predictions.
Revised: In Table 3, we see that GLNN s can still improve over MLP by large margins for inductive predictions.

Original: On 6/7 datasets, the GLNN prod performance are competitive to GNNs, which supports deploying
Revised: On 6/7 datasets, the GLNN prod performance are competitive to GNNs, which supports deploying

Original: GLNN as a much faster model with no or only slight performance loss.
Revised: GLNN as a much faster model with no or only slight performance loss.

Original: On the Arxiv dataset, the
Revised: On the Arxiv dataset, the

Original: GLNN performance is notably less than GNNs – we hypothesize this is due to Arxiv having a particularly challenging data split which causes distribution shift between test nodes and training nodes, which is hard for GLNN s to capture without utilizing neighbor information like GNNs.
Revised: GLNN performance is notably less than GNNs – we hypothesize this is due to Arxiv having a particularly challenging data split which causes distribution shift between test nodes and training nodes, which is hard for GLNN s to capture without utilizing neighbor information like GNNs.

Original: However, we note that GLNN performance is substantially improved over MLP.[Table]
Revised: However, we note that GLNN performance is substantially improved over MLP.

Original:  H OW DO GLNN S COMPARE TO OTHER INFERENCE ACCELERATION METHODS ?
Revised: 5. H OW DO GLNN S COMPARE TO OTHER INFERENCE ACCELERATION METHODS ?

Original: Common techniques of inference acceleration include pruning and quantization.
Revised: Common techniques of inference acceleration include pruning and quantization.

Original: These approaches can reduce model parameters and Multiplication-and-ACcumulation (MACs) operations.
Revised: These approaches can reduce model parameters and Multiplication-and-ACcumulation (MACs) operations.

Original: Still, they don’t eliminate neighborhood-fetching latency.
Revised: Still, they don’t eliminate neighborhood-fetching latency.

Original: Therefore, their speed gain on GNNs is less signiﬁcant than on NNs.
Revised: Therefore, their speed gain on GNNs is less signiﬁcant than on NNs.

Original: For GNNs speciﬁcally, we can also perform neighbor sampling to reduce the fetching latency.
Revised: For GNNs speciﬁcally, we can also perform neighbor sampling to reduce the fetching latency.

Original: We show an explicit speed comparison between vanilla SAGE, quantized SAGE from FP to INT8 (QSAGE), SAGE with 50% weights pruned (PSAGE), inference neighbor sampling with fan-out 15, and GLNN in Table 4.
Revised: We show an explicit speed comparison between vanilla SAGE, quantized SAGE from FP to INT8 (QSAGE), SAGE with 50% weights pruned (PSAGE), inference neighbor sampling with fan-out 15, and GLNN in Table 4.

Original: All numbers are on Products , with setting the same as Figure 1.
Revised: All numbers are on Products , with setting the same as Figure 1.

Original: We see that GLNN is considerably faster.
Revised: We see that GLNN is considerably faster.

Original: Two other kinds of methods considered as inference acceleration are GNN-to-GNN KD like TinyGNN (Yan et al., 2020) and Graph Augmented-MLPs (GA-MLPs) like SGC (Wu et al., 2019) or SIGN (Frasca et al., 2020).
Revised: Two other kinds of methods considered as inference acceleration are GNN-to-GNN KD like TinyGNN (Yan et al., 2020) and Graph Augmented-MLPs (GA-MLPs) like SGC (Wu et al., 2019) or SIGN (Frasca et al., 2020).

Original: Inference of GNN-to-GNN KD is likely to be slower than a GNN-L i with the same i as the student, since there will usually beoverheads introduced by some extra modules like the Peer-Aware Module (PAM) in TinyGNN.
Revised: Inference of GNN-to-GNN KD is likely to be slower than a GNN-L i with the same i as the student, since there will usually be some extra overheads like the Peer-Aware Module (PAM) in TinyGNN.

Original: GA-MLPs precompute augmented node features and apply MLPs to them.
Revised: GA-MLPs precompute augmented node features and apply MLPs to them.

Original: With precomputation, their inference time will be the same as MLPs for dimension-preserving augmentation (SGC) and the same as enlarged MLPw i for augmentation involves concatenation (SIGN).
Revised: With precomputation, their inference time will be the same as MLPs for dimension-preserving augmentation (SGC) and the same as enlarged MLPw i for augmentation involves concatenation (SIGN).

Original: Thus, for both kinds of approaches, it is sufﬁcient to compare GLNN with GNN-L i and MLPw i , which we have  already shown in Figure 3 (left).
Revised: Thus, for both kinds of approaches, it is sufﬁcient to compare GLNN with GNN-L i and MLPw i , which we have [Table] [Table] already shown in Figure 3 (left).

Original: We see that GNN-L i s are much slower than MLPs.
Revised: We see that GNN-L i s are much slower than MLPs.

Original: For GA-MLPs, since full pre-computation cannot be done for inductive nodes, GA-MLPs still need to fetch neighbor nodes.
Revised: For GA-MLPs, since full pre-computation cannot be done for inductive nodes, GA-MLPs still need to fetch neighbor nodes.

Original: This makes them much slower than MLPw i in the inductive setting, and even slower than pruned GNNs and TinyGNN as shown in Zhou et al.
Revised: This makes them much slower than MLPw i in the inductive setting [Table] and even slower than pruned GNNs and TinyGNN as shown in Zhou et al.

Original: 5. H OW DOES GLNN BENEFIT FROM DISTILLATION ?
Revised: H OW DOES GLNN BENEFIT FROM DISTILLATION ?

Original: We showed that GNNs are markedly better than MLPs on node classiﬁcation tasks.
Revised: We showed that GNNs are markedly better than MLPs on node classiﬁcation tasks.

Original: But, with 
Revised: But, with KD,

Original: KD, GLNN s can often become competitive to GNNs.
Revised: GLNN s can often become competitive to GNNs.

Original: This indicates that there exist suitable MLP parameters which can well approximate the ideal prediction function from node features to labels.
Revised: This indicates that there exist suitable MLP parameters which can well approximate the ideal prediction function from node features to labels.

Original: However, these parameters can be difﬁcult to learn through standard stochastic gradient descent.
Revised: However, these parameters can be difﬁcult to learn through standard stochastic gradient descent.

Original: We hypothesize that KD helps to ﬁnd them through regularization and transfer of inductive bias.
Revised: We hypothesize that KD helps to ﬁnd them through regularization and transfer of inductive bias.

Original: First, we show that KD can help to regularize the student model and prevent overﬁtting.
Revised: First, we show that KD can help to regularize the student model and prevent overﬁtting.

Original: By comparing the loss curves of a directly trained MLP and the GLNN in Figure 4, the gap between training and validation loss is visibly larger for MLPs than GLNNs, and MLPs show obvious overﬁtting trends.
Revised: By comparing the loss curves of a directly trained MLP and the GLNN in Figure 4, the gap between training and validation loss is visibly larger for MLPs than GLNNs, and MLPs show obvious overﬁtting trends.

Original: Second, we analyze the inductive bias that making GNNs powerful on node classiﬁcation, which suggests that node inferences should be inﬂuenced by the graph topology, especially by neighbor nodes.
Revised: Second, we analyze the inductive bias that making GNNs powerful on node classiﬁcation, which suggests that node inferences should be inﬂuenced by the graph topology, especially by neighbor nodes.

Original: In contrast, MLPs are more ﬂexible but have less inductive bias.
Revised: In contrast, MLPs are more ﬂexible but have less inductive bias.

Original: Similar difference exists between Transformers (Vaswani et al., 2017) and MLPs.
Revised: Similar difference exists between Transformers (Vaswani et al., 2017) and MLPs.

Original: In Liu et al.
Revised: In Liu et al.

Original: (2021), it is shown that the inductive bias in Transformers can be mitigated by a simple gate on large MLPs.
Revised: (2021), it is shown that the inductive bias in Transformers can be mitigated by a simple gate on large MLPs.

Original: For node classiﬁcation, we hypothesize that KD helps to mitigate the inductive bias, so GLNN s can perform competitively.
Revised: For node classiﬁcation, we hypothesize that KD helps to mitigate the inductive bias, so GLNN s can perform competitively.

Original: Soft labels from GNN teachers are heavily inﬂuenced by the graph topology due to inductive bias.
Revised: Soft labels from GNN teachers are heavily inﬂuenced by the graph topology due to inductive bias.

Original: They maintain nonzero probabilities on classes other than the ground truth provided by one-hot labels, which can be useful for the student to learn to complement the missing inductive bias in MLPs.
Revised: They maintain nonzero probabilities on classes other than the ground truth provided by one-hot labels, which can be useful for the student to learn to complement the missing inductive bias in MLPs.

Original: To evaluate this hypothesis quantitatively, we deﬁne the cut loss L cut ∈ [0 , 1] in Equation 2 to measure the consistency between model predictions and graph topology (details in Appendix C):
Revised: To evaluate this hypothesis quantitatively, we deﬁne the cut loss L cut ∈ [0 , 1] in Equation 2 to measure the consistency between model predictions and graph topology (details in Appendix C):

Original: [Equation]
Revised: [Equation]

Original: Here ˆ Y ∈ [0 , 1] N × K is the soft classiﬁcation probability output by the model, A and D are the adjacency and degree matrices.
Revised: Here ˆ Y ∈ [0 , 1] N × K is the soft classiﬁcation probability output by the model, A and D are the adjacency and degree matrices.

Original: When L cut is close to 1, it means the predictions and the graph topology are very consistent.
Revised: When L cut is close to 1, it means the predictions and the graph topology are very consistent.

Original: In our experiment, we observe that the average L cut for SAGE over ﬁve
Revised: In our experiment, we observe that the average L cut for SAGE over ﬁve

Original: CPF datasets is 0.9221, which means high consistency.
Revised: CPF datasets is 0.9221, which means high consistency.

Original: The same L cut for MLPs is only 0.7644, but for GLNN s it is 0.8986.
Revised: The same L cut for MLPs is only 0.7644, but for GLNN s it is 0.8986.

Original: This shows that the GLNN predictions indeed beneﬁt from the graph topology knowledge contained in the teacher outputs (the full table of L cut values in Appendix C).
Revised: This shows that the GLNN predictions indeed beneﬁt from the graph topology knowledge contained in the teacher outputs (the full table of L cut values in Appendix C).

Original: 5. D O GLNN S HAVE ENOUGH MODEL EXPRESSIVENESS ?
Revised: 5. D O GLNN S HAVE ENOUGH MODEL EXPRESSIVENESS ?

Original: Intuitively, the addition of neighbor information makes GNNs more powerful than MLPs when classifying nodes.
Revised: Intuitively, the addition of neighbor information makes GNNs more powerful than MLPs when classifying nodes.

Original: Thus, a natural question regarding KD from GNNs to MLPs is whether MLPs are expressive enough to represent graph data as well as GNNs.
Revised: Thus, a natural question regarding KD from GNNs to MLPs is whether MLPs are expressive enough to represent graph data as well as GNNs.

Original: Many recent works studied GNN model expressiveness (Xu et al., 2018;
Revised: Many recent works studied GNN model expressiveness (Xu et al., 2018;

Original: Chen et al., 2021).
Revised: Chen et al., 2021).

Original: The latter analyzed GNNs and GA-MLPs for node classiﬁcation and characterized expressiveness as the number of equivalence classes of rooted graphs induced by the model (formal deﬁnitions in Appendix D).
Revised: The latter analyzed GNNs and GA-MLPs for node classiﬁcation and characterized expressiveness as the number of equivalence classes of rooted graphs induced by the model (formal deﬁnitions in Appendix D).

Original: The conclusion is that GNNs are more powerful than GA-MLPs, but in most real-world cases their expressiveness is indistinguishable.
Revised: The conclusion is that GNNs are more powerful than GA-MLPs, but in most real-world cases their expressiveness is indistinguishable.

Original: We adopt the analysis framework from Chen et al.
Revised: We adopt the analysis framework from Chen et al.

Original: and show in Appendix D that the number of equivalence classes induced by GNNs and MLPs are (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L − 1 and |X| respectively.
Revised: and show in Appendix D that the number of equivalence classes induced by GNNs and MLPs are (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L − 1 and |X| respectively.

Original: Here m denotes the max node degree, L denotes the number of GNN layers, and X denotes the set of all possible node features.
Revised: Here m denotes the max node degree, L denotes the number of GNN layers, and X denotes the set of all possible node features.

Original: The former is apparently larger which concludes that GNNs are more expressive.
Revised: The former is apparently larger which concludes that GNNs are more expressive.

Original: Empirically, however, the gap makes little difference when |X| is large.
Revised: Empirically, however, the gap makes little difference when |X| is large.

Original: In real applications, node features can be high dimensional like bag-of-words, or even word embeddings, thus making |X| enormous.
Revised: In real applications, node features can be high dimensional like bag-of-words, or even word embeddings, thus making |X| enormous.

Original: Like for bag-of-words, |X| is in the order of O ( p D ) , where D is the vocabulary size, and p is the max word frequency.
Revised: Like for bag-of-words, |X| is in the order of O ( p D ) , where D is the vocabulary size, and p is the max word frequency.

Original: The expressiveness of a L-layer GNN is lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L − 

Revised: The expressiveness of a L-layer GNN is lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L −

Original: = O ( p D ( m − 1)(

Revised: = O ( p D ( m − 1)(

Original: L − 1) ) , but empirically, both MLPs and GNNs should have enough expressiveness given D is usually hundreds or bigger (see Table 5).
Revised: L − 1) ) , but empirically, both MLPs and GNNs should have enough expressiveness given D is usually hundreds or bigger (see Table 5).

Original: 5. W HEN WILL GLNN S FAIL TO WORK ?
Revised: 5. W HEN WILL GLNN S FAIL TO WORK ?

Original: As discussed in Section 5.7 and Appendix D, the goal of GML node classiﬁcation is to ﬁt a function f on the rooted graph G [ i ] and label y i .
Revised: As discussed in Section 5.7 and Appendix D, the goal of GML node classiﬁcation is to ﬁt a function f on the rooted graph G [ i ] and label y i .

Original: From the information theoretic perspective, ﬁtting f by minimizing the commonly used cross-entropy loss is equivalent to maximizing the mutual information (MI), I ( G [ i ] ; y i ) as shown in Qin et al.
Revised: From the information theoretic perspective, ﬁtting f by minimizing the commonly used cross-entropy loss is equivalent to maximizing the mutual information (MI), I ( G [ i ] ; y i ) as shown in Qin et al.

Original: If we consider G [ i ] as a joint distribution of two random variables X
Revised: If we consider G [ i ] as a joint distribution of two random variables X

Original: [ i ] and E [ i ] representing the node features and edges in G [ i ] respectively, we have
Revised: [ i ] and E [ i ] representing the node features and edges in G [ i ] respectively, we have

Original: [Equation]
Revised: [Equation]

Original:  I ( E [ i ] ; y i ) only depends on edges and labels, thus MLPs can only maximize I ( X
Revised: [Figure] I ( E [ i ] ; y i ) only depends on edges and labels, thus MLPs can only maximize I ( X

Original: [ i ] ; y i |E [ i ] ) .
Revised: [ i ] ; y i |E [ i ] ) .

Original: In the extreme case, I ( X [ i ] ; y i |E
Revised: In the extreme case, I ( X [ i ] ; y i |E

Original: [ i ] ) can be zero when y [ i ] is conditionally independent from X
Revised: [ i ] ) can be zero when y [ i ] is conditionally independent from X

Original: [ i ] given E [ i ] .
Revised: [ i ] given E [ i ] .

Original: For example, when every node is labeled by its degree or whether it forms a triangle.
Revised: For example, when every node is labeled by its degree or whether it forms a triangle.

Original: Then MLPs won’t be able to ﬁt meaningful functions, and neither will GLNN s. However, such cases are typically rare, and unexpected in practical settings our work is mainly concerned with.
Revised: Then MLPs won’t be able to ﬁt meaningful functions, and neither will GLNN s. However, such cases are typically rare, and unexpected in practical settings our work is mainly concerned with.

Original: For real GML tasks, node features and structural roles are often highly correlated (Lerique et al., 2020), hence MLPs can achieve reasonable results even only based on node features, and thus GLNN s can potentially achieve much better results.
Revised: For real GML tasks, node features and structural roles are often highly correlated (Lerique et al., 2020), hence MLPs can achieve reasonable results even only based on node features, and thus GLNN s can potentially achieve much better results.

Original: We study the failure case of GLNN s by creating a low MI scenario in Section 6.[Figure]
Revised: We study the failure case of GLNN s by creating a low MI scenario in Section 6.

Original:  A BLATION S TUDIES
Revised:  A BLATION S TUDIES

Original: In this section, we do ablation studies of GLNN s on node feature noise, inductive split rates, and teacher GNN architecture.
Revised: In this section, we do ablation studies of GLNN s on node feature noise, inductive split rates, and teacher GNN architecture.

Original: Reported results are test accuracies averaged over ﬁve datasets in CPF.
Revised: Reported results are test accuracies averaged over ﬁve datasets in CPF.

Original: Noisy node features.
Revised: Noisy node features.

Original: Following Section 5.8, we investigate failure cases of GLNN by adding different level of Gaussian noise to the node features to decrease their mutual information with labels.
Revised: Following Section 5.8, we investigate failure cases of GLNN by adding different level of Gaussian noise to the node features to decrease their mutual information with labels.

Original: Speciﬁcally, we replace X with ˜ X = (1 − α ) X + α(cid:15) .
Revised: Speciﬁcally, we replace X with ˜ X =(1 − α ) X + α(cid:15) .

Original: Here (cid:15) is isotropic Gaussian and independent from X , and α ∈ [0 , 1] denotes the noise level.
Revised: Here (cid:15) is isotropic Gaussian and independent from X , and α ∈ [0 , 1] denotes the noise level.

Original: We show the inductive performance of MLPs, GNNs, and GLNN under different noise levels in the left plot in Figure 5.
Revised: We show the inductive performance of MLPs, GNNs, and GLNN under different noise levels in the left plot in Figure 5.

Original: We see that as α increases, the accuracy of MLPs and GLNN s decrease faster than GNNs, while the performance of GLNN and GNNs are still comparable with small α s. When α reaches 1, ˜ X and Y will become independent, which corresponds to the extreme case discussed in Section 5.8.
Revised: We see that as α increases, the accuracy of MLPs and GLNN s decrease faster than GNNs, while the performance of GLNN and GNNs are still comparable with small α s. When α reaches 1, ˜ X and Y will become independent, which corresponds to the extreme case discussed in Section 5.8.

Original: Inductive split rate.
Revised: Inductive split rate.

Original: In Section 5.4, we use a 20-80 split of the test data for inductive evaluation.
Revised: In Section 5.4, we use a 20-80 split of the test data for inductive evaluation.

Original: In the middle plot in Figure 5, we show the results under different split rates.
Revised: In the middle plot in Figure 5, we show the results under different split rates (More detailed plots in Appendix

Original:  We see that as the inductive portion increase, GNN and MLP performance stays roughly the same, and the GLNN inductive performance drops slightly.
Revised: H). We see that as the inductive portion increase, GNN and MLP performance stays roughly the same, and the GLNN inductive performance drops slightly.

Original: We only consider rates up to 50-50 since having 50% or even more inductive nodes is highly atypical in practice.
Revised: We only consider rates up to 50-50 since having 50% or even more inductive nodes is highly atypical in practice.

Original: When a large amount of new data are encountered, practitioners can opt to retrain the model on all the data before deployment.
Revised: When a large amount of new data are encountered, practitioners can opt to retrain the model on all the data before deployment.

Original: Teacher GNN architecture.
Revised: Teacher GNN architecture.

Original: For experiments above, we used SAGE to represent GNNs.
Revised: For experiments above, we used SAGE to represent GNNs.

Original: In Figure 5 (right), we now results with other GNNs as teachers, e.g. GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´ c et al., 2017), and APPNP Klicpera et al.
Revised: In Figure 5 (right), we now results with other GNNs as teachers, e.g. GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´ c et al., 2017), and APPNP Klicpera et al.

Original: We see that GLNN s can learn from different GNN teachers and improve over MLPs.
Revised: We see that GLNN s can learn from different GNN teachers and improve over MLPs.

Original: Their performance with all four teachers is similar, with the GLNN distilled from APPNP slightly worse than others.
Revised: Their performance with all four teachers is similar, with the GLNN distilled from APPNP slightly worse than others.

Original: In fact, a similar phenomenon has been observed in Yang et al.
Revised: In fact, a similar phenomenon has been observed in Yang et al.

Original: (2021a)
Revised: (2021a)

Original: as well, i.e. APPNP beneﬁts the student the least.
Revised: as well, i.e. APPNP beneﬁts the student the least.

Original: One possible reason is that the ﬁrst step of APPNP is to utilize the node’s own feature for prediction (prior to propagating over the graph), which is very similar to what the student MLP is doing, and thus provides less extra information to MLPs than other teachers.
Revised: One possible reason is that the ﬁrst step of APPNP is to utilize the node’s own feature for prediction (prior to propagating over the graph), which is very similar to what the student MLP is doing, and thus provides less extra information to MLPs than other teachers.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we explored whether we can bridge the best of GNNs and MLPs to achieve accurate and fast GML models for deployment.
Revised: In this paper we explored whether we can bridge the best of GNNs and MLPs to achieve accurate and fast GML models for deployment.

Original: We found that KD from GNNs to MLPs helps to eliminate inference graph dependency, which results in GLNN s that are 146×-273× faster than GNNs while enjoying competitive performance.
Revised: We found that KD from GNNs to MLPs helps to eliminate inference graph dependency, which results in GLNN s that are 146×-273× faster than GNNs while enjoying competitive performance.

Original: We do a comprehensive study of GLNN properties.
Revised: We do a comprehensive study of GLNN properties.

Original: The promising results on 7 datasets across different domains show that GLNN s can be a handy choice for deploying latencyconstraint models.
Revised: The promising results on 7 datasets across different domains show that GLNN s can be a handy choice for deploying latency-constraint models.

Original: In our experiments, the current version of GLNN s on the Arxiv dataset doesn’t show competitive inductive performance.
Revised: In our experiments, the current version of GLNN s on the Arxiv dataset doesn’t show competitive inductive performance.

Original: More advanced distillation techniques can potentially improve the GLNN performance, and we leave this investigation as future work.
Revised: More advanced distillation techniques can potentially improve the GLNN performance, and we leave this investigation as future work

Original: A D ETAILED E XPERIMENT S ETTINGS A. D ATASETS
Revised: A D ETAILED E XPERIMENT S ETTINGS A. D ATASETS

Original: Here we provide a detailed description of the datasets we used to support our argument.
Revised: Here we provide a detailed description of the datasets we used to support our argument.

Original: Out of these datasets, 4 of them are citation graphs.
Revised: Out of these datasets, 4 of them are citation graphs.

Original: Cora, Citeseer, Pubmed, ogbn-arxiv with the node features being descriptions of the papers, either bag-of-word vector, TF-IDF vector, or word embedding vectors.
Revised: Cora, Citeseer, Pubmed, ogbn-arxiv with the node features being descriptions of the papers, either bag-of-word vector, TF-IDF vector, or word embedding vectors.

Original: In Table 5, we provided the basic statistics of these datasets.
Revised: In Table 5, we provided the basic statistics of these datasets.

Original: [Table]
Revised: [Table]

Original: For all datasets, we follow the setting in the original paper to split the data.
Revised: For all datasets, we follow the setting in the original paper to split the data.

Original: Speciﬁcally, for the ﬁve smaller datasets from the CPF paper, we use the CPF splitting strategy and each random seed corresponds to a different split.
Revised: Speciﬁcally, for the ﬁve smaller datasets from the CPF paper, we use the CPF splitting strategy and each random seed corresponds to a different split.

Original: For the OGB datasets, we follow the OGB ofﬁcial splits based on time and popularity for Arxiv and Products respectively.
Revised: For the OGB datasets, we follow the OGB ofﬁcial splits based on time and popularity for Arxiv and Products respectively.

Original: A. M ODEL H YPERPARAMETERS
Revised: A. M ODEL H YPERPARAMETERS

Original: The hyperparameters of GNN models on each dataset are taken from the best hyperparameters provided by the CPF paper and the OGB ofﬁcial examples.
Revised: The hyperparameters of GNN models on each dataset are taken from the best hyperparameters provided by the CPF paper and the OGB ofﬁcial examples.

Original: For the student MLPs and GLNN s, unless otherwise speciﬁed with -w i or -L i , we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN, so their total number of parameters stays the same as the teacher GNN. 
Revised: For the student MLPs and GLNN s, unless otherwise speciﬁed with -w i or -L i , we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN, so their total number of parameters stays the same as the teacher GNN. from the CPF paper.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For GLNN s we do a hyperparameter search of learning rate from [0.01, 0.005, 0.001], weight decay from [0, 0.001, 0.002, 0.005, 0.01], and dropout from 0.1 to 0.6.
Revised: For GLNN s we do a hyperparameter search of learning rate from [0.01, 0.005, 0.001], weight decay from [0, 0.001, 0.002, 0.005, 0.01] [Table] and dropout from 0.1 to 0.6.

Original: A. K NOWLEDGE D ISTILLATION
Revised: A. K NOWLEDGE D ISTILLATION

Original: We use the distillation method proposed in Hinton et al.
Revised: We use the distillation method proposed in Hinton et al.

Original: as in Equation 1, the hard labels are found to be helpful, so nonzero λ s was suggested.
Revised: as in Equation 1, the hard labels are found to be helpful, so nonzero λ s was suggested.

Original: In our case, we did a little tuning for λ but didn’t ﬁnd nonzero λ s to be very helpful.
Revised: In our case, we did a little tuning for λ but didn’t ﬁnd nonzero λ s to be very helpful.

Original: Therefore, we report all of our results with λ = 0 , i.e. only the second term involving soft labels is effective.
Revised: Therefore, we report all of our results with λ =0 , i.e. only the second term involving soft labels is effective.

Original: More careful tuning of λ should further improve the results since the searching space is strictly larger.
Revised: More careful tuning of λ should further improve the results since the searching space is strictly larger.

Original: We implemented a weighted version in our code, and we leave the choice of λ as future work.
Revised: We implemented a weighted version in our code, and we leave the choice of λ as future work.

Original: A. T HE T RANSDUCTIVE S ETTING AND T HE I NDUCTIVE S ETTING
Revised: A. T HE T RANSDUCTIVE S ETTING AND T HE I NDUCTIVE S ETTING

Original: Given G , X , and Y L , the goal of node classiﬁcation can be divided into two different settings, i.e. transductive and inductive.
Revised: Given G , X , and Y L , the goal of node classiﬁcation can be divided into two different settings, i.e. transductive and inductive.

Original: In real applications, the former can correspond to predict missing attributes of a user based on the user proﬁle and other existing users, and the latter can correspond to predict labels of some new nodes that are only seen during inference time.
Revised: In real applications, the former can correspond to predict missing attributes of a user based on the user proﬁle and other existing users, and the latter can correspond to predict labels of some new nodes that are only seen during inference time.

Original: To create the inductive setting on a given dataset, we hold out some nodes along with edges connected to these nodes during training and use them for inductive evaluation only.
Revised: To create the inductive setting on a given dataset, we hold out some nodes along with edges connected to these nodes during training and use them for inductive evaluation only.

Original: These nodes and edges are picked from the test data.
Revised: These nodes and edges are picked from the test data.

Original: Using notation deﬁned above, we pick the inductive nodes V Uind ⊂ V U , which partitions V U into the disjoint inductive subset and observed subset, i.e. V U
Revised: Using notation deﬁned above, we pick the inductive nodes V Uind ⊂V U , which partitions V U into the disjoint inductive subset and observed subset, i.e. V U

Original: =
Revised: =

Original: V Uobs (cid:116) V Uind .
Revised: V Uobs (cid:116)V Uind .

Original: Then we can take all the edges connected to nodes in V Uind to further partition the whole graph, so we end up with G =G [Equation]
Revised: Then we can take all the edges connected to nodes in V Uind to further partition the whole graph, so we end up with G =

Original: (cid:116) [Equation] [Equation] (cid:116)
Revised: G obs (cid:116)G ind , X = X L (cid:116)

Original: [Equation]

Original:  (cid:116)
Revised: X Uobs (cid:116)

Original: X Uind , and Y 
Revised: X Uind , and Y =

Original: = Y L (cid:116)
Revised: Y L (cid:116)

Original: Y Uobs (cid:116)
Revised: Y Uobs (cid:116)

Original: Y Uind .
Revised: Y Uind .

Original: We show the input and output of both settings using the notations below.
Revised: We show the input and output of both settings using the notations below.

Original: We visualize the difference between the inductive setting and the transductive setting in Figure 6.
Revised: We visualize the difference between the inductive setting and the transductive setting in Figure 8.

Original: [Figure]
Revised: [Figure]

Original: A. I MPLEMENTATION AND H ARDWARD D ETAILS
Revised: A. C HOOSING SOFT TARGETS UNDER THE TRANSDUCTIVE SETTING

Revised: For the transductive setting in Section 5.3, all the nodes in the graph, including the validation and test nodes, are used for the soft target generation.

Revised: We now discuss the rationale behind this choice.

Revised: Firstly, the transductive setting is the most common setting for graph data and it was used in most

Revised: GNN architecture works and GNN acceleration works we mentioned in the related work section, e.g. the CPF paper [1], where we take the dataset and model hyperparameters from.

Revised: Therefore, to avoid any confusion and for a fair comparison with numbers from previous literature, we start our experiments with exactly the same input and output as the standard transductive setting.

Revised: Under this setting, the inputs to GNNs include all the node features and the graph structure, so GLNNs are set to be able to access the same input.

Revised: As GLNN includes a teacher training step and a distillation step, the soft labels of all the nodes are intermediate outputs produced by the teacher training step, and thus used for the second distillation step for the best GLNN performance.

Revised: As you pointed out, this transductive setting can boil down to a sanity check when the student is sufﬁciently large.

Revised: That is the reason why we separate the setting to be GLNN and GLNN+ and report the results in Table 1 and Table 2 separately.

Revised: For GLNN, we are checking how well GLNNs can perform compared to GNNs under the equal-parameter constraint.

Revised: The point of Table 1 can be interpreted as given a ﬁxed parameter budget, whether there exists one set of parameters (one instantiation of the MLP) that can achieve competitive results as the GNN.

Revised: Only when this holds, should we further investigate the more interesting and challenging inductive case.

Revised: Secondly, the task we focus on is node classiﬁcation, which in many cases is considered as semi-supervised learning with very scarce labels.

Revised: For example, Pubmed only uses 60 labeled nodes

Revised: (20 per class) out of 20K nodes for training.

Revised: Rather than design an advanced model that can do few-shot learning, our goal here is to leverage as much data as possible to simplify the model for more efﬁcient inference.

Revised: We thus utilize the soft pseudo-labels on all the unlabelled nodes for the best GLNN performance.

Revised: In reality, when there is a large amount of separate unlabeled data, these unlabeled data can be used for GLNN distillation training and a different set of labeled data can be used for evaluation.

Revised: In our case, we mimic this scenario in the following inductive setting.

Original: The experiments on both baselines and our approach are implemented using PyTorch, the DGL (Wang et al., 2019) library for GNN algorithms, and Adam (Kingma & Ba, 2015) for optimization.
Revised: The experiments on both baselines and our approach are implemented using PyTorch, the DGL (Wang et al., 2019) library for GNN algorithms, and Adam (Kingma & Ba, 2015) for optimization.

Original: We run all experiments on a machine with 80 Intel(R)
Revised: We run all experiments on a machine with 80 Intel(R)

Original: Xeon(R) E5-2698 v4 @ 2.20GHz CPUs, and a single
Revised: Xeon(R) E5-2698 v4 @ 2.20GHz CPUs, and a single

Original: NVIDIA V100 GPU with 16GB RAM.
Revised: NVIDIA V100 GPU with 16GB RAM.

Original: B S PACE AND T IME C OMPLEXITY OF GNN S VS . MLP S
Revised: B S PACE AND T IME C OMPLEXITY OF GNN S VS . MLP S

Original: In Table 1, the model comparison was between equal-sized MLPs ( GLNN s) and GNNs.
Revised: In Table 1, the model comparison was between equal-sized MLPs ( GLNN s) and GNNs.

Original: This is how people used to compare models for the sake of fair space complexity, but it is not a completely fair comparison for cross-model comparison, especially  MLPs vs. GNNs.
Revised: This is how people used to compare models for the sake of fair space complexity, but it is not a completely fair comparison for cross-model comparison, especially for MLPs vs. GNNs.

Original: To do inference with GNNs, the graph needs to be loaded in the memory either entirely or batch by batch, which will take a much larger space than the model parameters.
Revised: To do inference with GNNs, the graph needs to be loaded in the memory either entirely or batch by batch, which will take a much larger space than the model parameters.

Original: Thus, the space complexity of GNNs is actually much higher than equal-sized MLPs.
Revised: Thus, the space complexity of GNNs is actually much higher than equal-sized MLPs.

Original: From the time complexity perspective, the major inference latency of GNNs comes from the data dependency as shown in Section 4.
Revised: From the time complexity perspective, the major inference latency of GNNs comes from the data dependency as shown in Section 4.

Original: Under the same setting as Figure 1, we show in the left part of Figure 3 that even a 5-layer MLP with 8 times wider hidden layers still runs much faster than a single-layer SAGE.
Revised: Under the same setting as Figure 1, we show in the left part of Figure 3 that even a 5-layer MLP with 8 times wider hidden layers still runs much faster than a single-layer SAGE.

Original: Another example of cross-model comparison is 
Revised: Another example of cross-model comparison is Transformers vs.

Original: Transformers vs. RNNs. Large Transformers can have more parameters than RNNs because of the attention mechanism, but they also run faster than RNNs in general.
Revised: RNNs. Large Transformers can have more parameters than RNNs because of the attention mechanism, but they also run faster than RNNs in general.

Original: Moreover, the comparison of GNNs and MLPs with the same number of parameters ignores the inductive bias difference.
Revised: Moreover, the comparison of GNNs and MLPs with the same number of parameters ignores the inductive bias difference.

Original: GNNs are advantaged since they get information from two sources when they do inference, i.e. the learned parameters and the graph data.
Revised: GNNs are advantaged since they get information from two sources when they do inference, i.e. the learned parameters and the graph data.

Original: For GNNs, there is an inductive bias of neighbor nodes inﬂuencing each other, so the knowledge missing from model parameters can be complemented.
Revised: For GNNs, there is an inductive bias of neighbor nodes inﬂuencing each other, so the knowledge missing from model parameters can be complemented.

Original: For MLPs, model parameters are the only source for them to remember the learned knowledge.
Revised: For MLPs, model parameters are the only source for them to remember the learned knowledge.

Original: They are being very ﬂexible with less inductive bias, which could require more parameters for the model to learn (see Section 5.6 for a more detailed discussion of inductive bias).
Revised: They are being very ﬂexible with less inductive bias, which could require more parameters for the model to learn (see Section 5.6 for a more detailed discussion of inductive bias).

Original: In Table 1, we saw that for equal-sized comparison, GLNN s are not as good as GNNs on the OGB datasets.
Revised: In Table 1, we saw that for equal-sized comparison, GLNN s are not as good as GNNs on the OGB datasets.

Original: Following the discussion above and given the GLNN s used in Table 1 are relatively small ( layers and 256 hidden dimensions) for millions of nodes in the OGB datasets, we ask whether this gap can be mitigated by increasing the MLP and thus GLNN sizes.
Revised: Following the discussion above and given the GLNN s used in Table 1 are relatively small (3 layers and 256 hidden dimensions) for millions of nodes in the OGB datasets, we ask whether this gap can be mitigated by increasing the MLP and thus GLNN sizes.

Original: Our answer is shown in Table 2.
Revised: Our answer is shown in Table 2.

Original: C C ONSISTENCY M EASURE OF M ODEL P REDICTIONS AND G RAPH T OPOLOGY B ASED ON M IN -C UT
Revised: C C ONSISTENCY M EASURE OF M ODEL P REDICTIONS AND G RAPH T OPOLOGY B ASED ON M IN -C UT

Original: We introduce a metric to measure the consistency between model predictions and graph topology based on the min-cut problem in Section 5.6.
Revised: We introduce a metric to measure the consistency between model predictions and graph topology based on the min-cut problem in Section 5.6.

Original: The K -way normalized min-cut problem, or simply min-cut, partitions N nodes in V into K disjoint subsets by removing the minimum volume of edges.
Revised: The K -way normalized min-cut problem, or simply min-cut, partitions N nodes in V into K disjoint subsets by removing the minimum volume of edges.

Original: According to Dhillon et al.
Revised: According to Dhillon et al.

Original: (2004), the min-cut problem can be expressed as
Revised: (2004), the min-cut problem can be expressed as

Original: [Equation]
Revised: [Equation]

Original: with C being the node assignment matrix that partitions V , i.e. C i,j = 1 if node i is assigned to class j .
Revised: with C being the node assignment matrix that partitions V , i.e. C i,j =1 if node i is assigned to class j .

Original: A being the adjacency matrix and D being the degree matrix.
Revised: A being the adjacency matrix and D being the degree matrix.

Original: This quantity we try to maximize here tells us whether the assignment is consistent with the graph topology.
Revised: This quantity we try to maximize here tells us whether the assignment is consistent with the graph topology.

Original: The bigger it is, the less edges need to be removed, and the assignment is more consistent with existing connections in the graph.
Revised: The bigger it is, the less edges need to be removed, and the assignment is more consistent with existing connections in the graph.

Original: In Bianchi et al.
Revised: In Bianchi et al.

Original: , the authors show that when replacing the hard assignments 
Revised: , the authors show that when replacing the hard assignments C ∈{ 0 , 1 } N × K probability ˆ Y ∈

Original: C ∈ { 0 , 1 } N × K with a soft classiﬁcation probability ˆ Y ∈ [0 , 1] N × K , a cut loss L cut in Equationcan become a good approximation of Equation 4 and beused as the measuringmetric.
Revised: 1] N × K , a cut loss L cut in Equation 2 can become a good approximation of Equation 4 and be measuring

Original: [Table]
Revised: [Table]

Original: D E XPRESSIVENESS OF GNN S VS . MLP S IN TERMS OF EQUIVALENCE CLASSES OF ROOTED GRAPHS
Revised: D E XPRESSIVENESS OF GNN S VS . MLP S IN TERMS OF EQUIVALENCE CLASSES OF ROOTED GRAPHS

Original: In Chen et al.
Revised: In Chen et al.

Original: (2021), the expressiveness of GNNs and GA-MLPs were theoretically quantiﬁed in terms of induced equivalence classes of rooted graphs.
Revised: (2021), the expressiveness of GNNs and GA-MLPs were theoretically quantiﬁed in terms of induced equivalence classes of rooted graphs.

Original: We adopt their framework and perform a similar analysis for GNNs vs. MLPs.
Revised: We adopt their framework and perform a similar analysis for GNNs vs. MLPs.

Original: We ﬁrst deﬁne rooted graphs.
Revised: We ﬁrst deﬁne rooted graphs.

Original: Deﬁnition 1 (Rooted Graph) .
Revised: Deﬁnition 1 (Rooted Graph) .

Original: A rooted graph, denoted as G [ i ] is a graph with one node i in G [ i ] designated as the root.
Revised: A rooted graph, denoted as G [ i ] is a graph with one node i in G [ i ] designated as the root.

Original: GNNs, GA-MLPs, and MLPs can all be considered as functions on rooted graphs.
Revised: GNNs, GA-MLPs, and MLPs can all be considered as functions on rooted graphs.

Original: The goal of a node-level task on node i with label y i is to ﬁt a function to the input-output pairs ( G [ i ] , y i ).
Revised: The goal of a node-level task on node i with label y i is to ﬁt a function to the input-output pairs ( G [ i ] , y i ).

Original: We denote the space of rooted graphs as E .
Revised: We denote the space of rooted graphs as E .

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2021), the expressive power of a model on graph data is evaluated by its ability to approximate functions on E .
Revised: (2021), the expressive power of a model on graph data is evaluated by its ability to approximate functions on E .

Original: This is further characterized as the number of induced equivalence classes of rooted graphs on E , with the equivalence relation deﬁned as the following.
Revised: This is further characterized as the number of induced equivalence classes of rooted graphs on E , with the equivalence relation deﬁned as the following.

Original: Given a family of functions F on E , we deﬁne an equivalence relation (cid:39) E , F among all rooted graphs such that ∀G [ i ] , G (cid:48)
Revised: Given a family of functions F on E , we deﬁne an equivalence relation (cid:39) E , F among all rooted graphs such that ∀G [ i ] , G (cid:48)

Original: [ j ] ∈ E , G [ i ] (cid:39) E , F G (cid:48)
Revised: [ j ] ∈E , G [ i ] (cid:39) E , F G (cid:48)

Original: [ j ] if and only if ∀ f ∈ F , f ( G [ i ] ) =
Revised: [ j ] if and only if ∀ f ∈F ,f ( G [ i ] )=

Original: f ( G (cid:48)
Revised: f ( G (cid:48)

Original: [ j ] ) .
Revised: [ j ] ) .

Original: We now give a proposition to characterize the GNN expressive power (proof in Appendix E).
Revised: We now give a proposition to characterize the GNN expressive power (proof in Appendix E).

Original: Proposition 1.
Revised: Proposition 1.

Original: With X denotes the set of all possible node features and assuming |X| ≥ 2 , with m denotes the maximum node degree and assuming m ≥ 3 , the total number of equivalence classes of rooted graphs induced by an L-layer GNN is lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L − 1 .
Revised: With X denotes the set of all possible node features and assuming |X| ≥ 2 , with m denotes the maximum node degree and assuming m ≥ 3 , the total number of equivalence classes of rooted graphs induced by an L-layer GNN is lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L − 1 .

Original: As shown in Proposition 1, the expressive power of GNNs grows doubly-exponentially in the number of layers L , which means it grows linearly in L after taking log(log( · )) .
Revised: As shown in Proposition 1, the expressive power of GNNs grows doubly-exponentially in the number of layers L , which means it grows linearly in L after taking log(log( · )) .

Original: The expressive power
Revised: The expressive power

Original: GA-MLPs only grows exponentially in L as shown in Chen et al.
Revised: GA-MLPs only grows exponentially in L as shown in Chen et al.

Original: Under this framework, the expressive power of MLPs, which corresponds to a 0-layer GA-MLP, is |X| .
Revised: Under this framework, the expressive power of MLPs, which corresponds to a 0-layer GA-MLP, is |X| .

Original: Since the former is much larger than the latter, the conclusion will be GNNs are much more expressive than MLPs.
Revised: Since the former is much larger than the latter, the conclusion will be GNNs are much more expressive than MLPs.

Original: The gap between these two numbers indeed exists, but empirically this gap will only make a difference when |X| is small.
Revised: The gap between these two numbers indeed exists, but empirically this gap will only make a difference when |X| is small.

Original: As in Chen et al.
Revised: As in Chen et al.

Original: (2021), both the lower bound proof and the constructed examples showing GNNs are more powerful than GA-MLPs assumed |X| =
Revised: (2021), both the lower bound proof and the constructed examples showing GNNs are more powerful than GA-MLPs assumed |X| =

Original: In real applications and datasets considered in this work, the node features can be high dimensional vectors like bag-of-words, which makes |X| enormous.
Revised: In real applications and datasets considered in this work, the node features can be high dimensional vectors like bag-of-words, which makes |X| enormous.

Original: Thus, this gap doesn’t matter much empirically.
Revised: Thus, this gap doesn’t matter much empirically.

Original: E P ROOF OF T HE P ROPOSITION
Revised: E P ROOF OF T HE P ROPOSITION

Original: To prove Proposition 1, we ﬁrst deﬁne rooted aggregation trees, which is similar to but different from rooted graphs.
Revised: To prove Proposition 1, we ﬁrst deﬁne rooted aggregation trees, which is similar to but different from rooted graphs.

Original: Deﬁnition 2 (Rooted Aggregation Tree) .
Revised: Deﬁnition 2 (Rooted Aggregation Tree) .

Original: The depth-K rooted aggregation tree of a rooted graph G [ i ] is a depth-K rooted tree with a (possibly many-to-one) mapping from every node in the tree to some node in G [ i ] , where (i) the root of the tree is mapped to node i , and (ii) the children of every node j in the tree are mapped to the neighbors of the node in G [ i ] to which j is mapped.
Revised: The depth-K rooted aggregation tree of a rooted graph G [ i ] is a depth-K rooted tree with a (possibly many-to-one) mapping from every node in the tree to some node in G [ i ] , where (i) the root of the tree is mapped to node i , and (ii) the children of every node j in the tree are mapped to the neighbors of the node in G [ i ] to which j is mapped.

Original: A rooted aggregation tree can be obtained by unrolling the neighborhood aggregation steps in the
Revised: A rooted aggregation tree can be obtained by unrolling the neighborhood aggregation steps in the

Original: GNNs.
Revised: GNNs.

Original: An illustration of rooted graphs and rooted aggregation trees can be found in Chen et al.
Revised: An illustration of rooted graphs and rooted aggregation trees can be found in Chen et al.

Original: (2021) Figure 4.
Revised: Figure 4.

Original: We denote the set of all rooted aggregation trees of depth L using T L .
Revised: We denote the set of all rooted aggregation trees of depth L using T L .

Original: Then we use T L, X ,m to denote a subset of T L , where the node features belong to X , and all the nodes have exactly degree m ( m children), and at least two nodes out of these m nodes have different features.
Revised: Then we use T L, X ,m to denote a subset of T L , where the node features belong to X , and all the nodes have exactly degree m ( m children), and at least two nodes out of these m nodes have different features.

Original: In other words, a node can’t have all identical children.
Revised: In other words, a node can’t have all identical children.

Original: With rooted aggregation trees deﬁned, we are ready to prove Proposition 1.
Revised: With rooted aggregation trees deﬁned, we are ready to prove Proposition 1.

Original: The proof is adapted from the proof of Lemma 3 in Chen et al.
Revised: The proof is adapted from the proof of Lemma 3 in Chen et al.

Original: Proof.
Revised: Proof.

Original: Since the number of equivalence classes on E induced by the family of all depth-L GNNs consists of all rooted graphs that share the same rooted aggregation tree of depth-L (Chen et al., 2021), the lower bound problem in Proposition 1 can be reduced to lower bound |T
Revised: Since the number of equivalence classes on E induced by the family of all depth-L GNNs consists of all rooted graphs that share the same rooted aggregation tree of depth-L (Chen et al., 2021), the lower bound problem in Proposition 1 can be reduced to lower bound |T

Original: L | , which can be further reduced to lower bound the subset |T L, X ,m | .
Revised: L | , which can be further reduced to lower bound the subset |T L, X ,m | .

Original: We now show |T L, X ,m | ≥ (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L −inductively.
Revised: We now show |T L, X ,m | [Equation] 1 inductively.

Original: When L = 1 , the root of the tree can have |X| different choices.
Revised: When L = 1 , the root of the tree can have |X| different choices.

Original: For the children nodes, we pick m features from |X| and repetitions are allowed.
Revised: For the children nodes, we pick m features from |X| and repetitions are allowed.

Original: This leads to (cid:0) |X| + m − 1 m (cid:1) cases.
Revised: This leads to (cid:0) |X| + m − 1 m (cid:1) cases.

Original: Therefore,
Revised: Therefore,

Original: T L +1 , X ,m = |

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original: |X| [Equation] Assuming the statement holds for L , we show it holds for L + 1 by constructing trees in T L +1 , X ,m from T, T (cid:48)
Revised: Assuming the statement holds for L , we show it holds for L +1 by constructing trees in T L +1 , X ,m from T,T (cid:48)

Original: ∈ T L, X ,m .
Revised: ∈T L, X ,m .

Original: We do this by assigning node features in X to the m children of each leaf node in T and T (cid:48) .
Revised: We do this by assigning node features in X to the m children of each leaf node in T and T (cid:48) .

Original: First note that when T and T (cid:48) are two non-isomorphic trees, two depth-L+1 trees constructed from T and T (cid:48) will be different no matter how the node features are assigned.
Revised: First note that when T and T (cid:48) are two non-isomorphic trees, two depth-L+1 trees constructed from T and T (cid:48) will be different no matter how the node features are assigned.

Original: Now we consider all the trees can be constructed from T by assign node features of children to leaf nodes.
Revised: Now we consider all the trees can be constructed from T by assign node features of children to leaf nodes.

Original: We ﬁrst consider all paths from the root to leaves in T .
Revised: We ﬁrst consider all paths from the root to leaves in T .

Original: Each path consists of a sequence of nodes where the node features form a one-to-one mapping to an L-tuple τ ∈ { ( x 1 , . . . , x L ) : x i ∈ X} .
Revised: Each path consists of a sequence of nodes where the node features form a one-to-one mapping to an L-tuple τ ∈{ ( x 1 ,...,x L ): x i ∈X} .

Original: Leaf nodes are called node under τ if the path from the root to it corresponds to τ .
Revised: Leaf nodes are called node under τ if the path from the root to it corresponds to τ .

Original: The children of nodes under different τ s are always distinguishable, and thus any assignments lead to distinct rooted aggregation trees of depth L + 1 .
Revised: The children of nodes under different τ sare alwaysdistinguishable, andthusanyassignments leadtodistinct rooted aggregation treesofdepth L +1 .

Original: The assignment of children of nodes under the same τ , on the other hand, could be overcounted.
Revised: The assignment of children of nodes under the same τ , on the other hand, could be overcounted.

Original: Therefore, to lower bound T L +1 , X ,m , we only consider a special way of assignments to avoid over counting, which is that children of all nodes under the same τ are assigned the same set of features.
Revised: Therefore, to lower bound T L +1 , X ,m , we only consider a special way of assignments to avoid over counting, which is that children of all nodes under the same τ are assigned the same set of features.

Original: Since we assumed that at least two nodes of T have different features, there are at least 2 L different τ s corresponding to the path from the root to leaves.
Revised: Since we assumed that at least two nodes of T have different features, there are at least 2 L different τ s corresponding to the path from the root to leaves.

Original: For a leaf node j under a ﬁxed τ , one of its children needs to have the same feature as j ’s parent node.
Revised: For a leaf node j under a ﬁxed τ , one of its children needs to have the same feature as j ’s parent node.

Original: This restriction is due to the deﬁnition of rooted aggregation trees.
Revised: This restriction is due to the deﬁnition of rooted aggregation trees.

Original: Therefore, we only pick features for the other m − 1 nodes, which will be (cid:0) |X| + m − 2 m − 1 (cid:1) cases for each j .
Revised: Therefore, we only pick features for the other m − 1 nodes, which will be (cid:0) |X| + m − 2 m − 1 (cid:1) cases for each j .

Original: Then through this construction, the total number of depth-L+1 trees from T can be lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L .
Revised: Then through this construction, the total number of depth-L+1 trees from T can be lower bounded by (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L .

Original: Finally, we have this lower bound holds for all T ∈ T L, X ,m , so we derive T L +1 , X ,m ≥ (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L T L, X ,m 
Revised: Finally, we have this lower bound holds for all T ∈T L, X ,m , so we derive T L +1 , X ,m ≥ (cid:0) |X| + m − 2 m − 1 (cid:1) 2 L T L, X ,m , and T L, X ,m ≥ (cid:0) |X| + m − 2 m −

Original: [Table]
Revised: [Table]

Revised: F A DVANCED GNN ARCHITECTURE AS TEACHER

Revised: In our main experiment, SAGE teacher is used throughout to avoid inﬂuence by model architecture.

Revised: To show GLNN has stronger performance given a stronger teacher, we take MLP+CS from the OGB leaderboard as a new teacher, which has reported accuracy 84.18% and ranks 8 as of Nov 2021.

Revised: We choose MLP+CS instead of the other top 7 because the others either rely on raw text (additional info to the given node feature), or require a GPU with >16GB memory, which we don’t have access to.

Revised: Also, their improvement is not super signiﬁcant compared to MLP+CS, i.e. 84% to 86%.

Revised: The result is below:

Revised: [Table]

Revised: With the new teacher, GLNN+ is even better than SAGE (78.61%), which shows GLNN can get stronger given a stronger teacher.

Revised: G GLNN WITH FEATURE AUGMENTATION FROM ONE - HOP NEIGHBORS

Revised: In our main experiment, the inductive performance of GLNN on the Arxiv dataset is less desirable than others.

Revised: We thus consider augment the node features with their one-hop neighbors to include more graph information.

Revised: This can be seen as a middle ground between pure GLNN s and GNNs.

Revised: For this new experiment, we follow the setting in 3 but with two new approaches.

Revised: We explain the setting of these two approaches below.

Revised: 1. 1-hop GA-MLP: ﬁrstly, for each node v , we collect its 1-hop neighbors u and use features of u to augment the raw feature of v , like in SGC[3].

Revised: Then we train an MLP on the graph with augmented node features.

Revised: Note if v is in the observed graph but u is in the inductive part, then v don’t collect feature from u . 2.

Revised: 1-hop GA-GLNN: Go through the same feature augmentation step as 1-hop GA-MLP.

Revised: Then train an MLP with distillation for teacher GNN.

Revised: We show in the table below, with 1-hop neighbor features, performance of GLNN improves a lot.

Revised: This is expected as we also observe signiﬁcant improvement from MLP to 1-hop GA-MLP, but we indeed see 1-hop GA-GLNN (68.83) can further improve from 1-hop GA-MLP (66.62) and nearly match the teacher (70.64).

Revised: [Table]

Revised: As we have shown in Figure 3, the 1-Layer GNN in our case is roughly 4 times slower than GLNN (29.31 vs. 7.56), which should be a good approximation for the speed comparison between 1-hop

Revised: GA-MLP/GA-GLNN and GLNN.

Revised: This result is practically beneﬁcial, as it gives practitioners more ﬂexibility about how much accuracy they want to trade for less inference time.

Revised: We have also included the table above in our appendix.

Revised: Thank you for helping us strengthen the experiments.

Revised: H M ODEL PERFORMANCE UNDER DIFFERENT INDUCTIVE SPLIT RATE

Revised: This section is a continuation of the ablation study of inductive split rate in Section 6.

Revised: It generalizes

Revised: Middle to

Revised: , and explicitly show transductive