Original: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classiﬁcation
Revised: NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classiﬁcation

Original: Abstract
Revised: Abstract

Original: Graph neural networks have been extensively studied for learning with interconnected data.
Revised: Graph neural networks have been extensively studied for learning with interconnected data.

Original: Despite this, recent evidence has revealed GNNs’ deﬁciencies related to over-squashing, heterophily, handling long-range dependencies, edge incompleteness and particularly, the absence of graphs altogether.
Revised: Despite this, recent evidence has revealed GNNs’ deﬁciencies related to over-squashing, heterophily, handling long-range dependencies, edge incompleteness and particularly, the absence of graphs altogether.

Original: While a plausible solution is to learn new adaptive topology for message passing, issues concerning quadratic complexity hinder simultaneous guarantees for scalability and precision in large networks.
Revised: While a plausible solution is to learn new adaptive topology for message passing, issues concerning quadratic complexity hinder simultaneous guarantees for scalability and precision in large networks.

Original: In this paper, we introduce a novel all-pair message passing scheme for efﬁciently propagating node signals between arbitrary nodes, as an important building block for a pioneering Transformer-style network for node classiﬁcation on large graphs, dubbed as N ODE F ORMER .
Revised: In this paper, we introduce a novel all-pair message passing scheme for efﬁciently propagating node signals between arbitrary nodes, as an important building block for a pioneering Transformer-style network for node classiﬁcation on large graphs, dubbed as N ODE F ORMER .

Original: Speciﬁcally, the efﬁcient computation is enabled by a kernerlized Gumbel-Softmax operator that reduces the algorithmic complexity to linearity w.r.t.
Revised: Speciﬁcally, the efﬁcient computation is enabled by a kernerlized Gumbel-Softmax operator that reduces the algorithmic complexity to linearity w.r.t.

Original: node numbers for learning latent graph structures from large, potentially fully-connected graphs in a differentiable manner.
Revised: node numbers for learning latent graph structures from large, potentially fully-connected graphs in a differentiable manner.

Original: We also provide accompanying theory as justiﬁcation for our design.
Revised: We also provide accompanying theory as justiﬁcation for our design.

Original: Extensive experiments demonstrate the promising efﬁcacy of the method in various tasks including node classiﬁcation on graphs (with up to 2M nodes) and graph-enhanced applications (e.g., image classiﬁcation) where input graphs are missing.
Revised: Extensive experiments demonstrate the promising efﬁcacy of the method in various tasks including node classiﬁcation on graphs (with up to 2M nodes) and graph-enhanced applications (e.g., image classiﬁcation) where input graphs are missing.

Original: The codes are available at https://github.com/qitianwu/NodeFormer .
Revised: The codes are available at https://github.com/qitianwu/NodeFormer .

Original:  Introduction
Revised:  Introduction

Original: Relational structure inter-connecting instance nodes as a graph is ubiquitous from social domains (e.g., citation networks) to natural science (protein-protein interaction), where graph neural networks (GNNs)
Revised: Relational structure inter-connecting instance nodes as a graph is ubiquitous from social domains (e.g., citation networks) to natural science (protein-protein interaction), where graph neural networks (GNNs)

Original: [32, 19, 14, 36] have shown promising power for leveraging such data dependence as geometric priors.
Revised: [32, 19, 14, 36] have shown promising power for leveraging such data dependence as geometric priors.

Original: However, there arises increasing evidence challenging the core GNN hypothesis that propagating information along observed graph structures will necessarily produce better nodelevel representations for prediction on each individual instance node.
Revised: However, there arises increasing evidence challenging the core GNN hypothesis that propagating information along observed graph structures will necessarily produce better nodelevel representations for prediction on each individual instance node.

Original: Conﬂicts with this premise lead to commonly identiﬁed deﬁciencies with GNN message-passing rules w.r.t.
Revised: Conﬂicts with this premise lead to commonly identiﬁed deﬁciencies with GNN message-passing rules w.r.t.

Original: heterophily [53], over-squashing [2], long-range dependencies [8], and graph incompleteness [11], etc.
Revised: heterophily [53], over-squashing [2], long-range dependencies [8], and graph incompleteness [11], etc.

Original: Moreover, in graph-enhanced applications, e.g., text classiﬁcation [46], vision navigation [12], physics simulation [30], etc., graph structures are often unavailable though individual instances are strongly inter-correlated.
Revised: Moreover, in graph-enhanced applications, e.g., text classiﬁcation [46], vision navigation [12], physics simulation [30], etc., graph structures are often unavailable though individual instances are strongly inter-correlated.

Original: A common practice is to artiﬁcially construct a graph via some predeﬁned rules (e.g., k -NN), which is agnostic to downstream tasks and may presumably cause the misspeciﬁcation of
Revised: A common practice is to artiﬁcially construct a graph via some predeﬁned rules (e.g., k -NN), which is agnostic to downstream tasks and may presumably cause the misspeciﬁcation of

Original: GNNs’ inductive bias on input geometry (induced by the local feature propagation design).
Revised: GNNs’ inductive bias on input geometry (induced by the local feature propagation design).

Original: Natural solutions resort to organically combining learning optimal graph topology with message passing.
Revised: Natural solutions resort to organically combining learning optimal graph topology with message passing.

Original: However, one critical difﬁculty is the scalability issue with O ( N 2 ) (where N denotes
Revised: However, one critical difﬁculty is the scalability issue with O ( N 2 ) (where N denotes

Original: #nodes) computational complexity, which is prohibitive for large networks (with 10 K ⇠ 1 M nodes).
Revised: #nodes) computational complexity, which is prohibitive for large networks (with 10 K ⇠ 1 M nodes).

Original: Some existing approaches harness neighbor sampling [51], anchor-based adjacency surrogates [4] and hashing schemes [43] to reduce the overhead; however, these strategies may sacriﬁce model precision and still struggle to handle graphs with million-level nodes.
Revised: Some existing approaches harness neighbor sampling [51], anchor-based adjacency surrogates [4] and hashing schemes [43] to reduce the overhead; however, these strategies may sacriﬁce model precision and still struggle to handle graphs with million-level nodes.

Original: Another obstacle lies in the increased degrees of freedom due to at least an N ⇥ N all-pair similarity matrix, which may result in large combinatorial search space and vulnerability to over-ﬁtting.
Revised: Another obstacle lies in the increased degrees of freedom due to at least an N ⇥ N all-pair similarity matrix, which may result in large combinatorial search space and vulnerability to over-ﬁtting.

Original: In this work, we introduce a novel all-pair message passing scheme that can scale to large systems without compromising performance.
Revised: In this work, we introduce a novel all-pair message passing scheme that can scale to large systems without compromising performance.

Original: We develop a kernelized Gumbel-Softmax operator that seamlessly synthesizes random feature map [27] and approximated sampling strategy [16], for distilling latent structures among all the instance nodes and yielding moderate gradients through differentiable optimization.
Revised: We develop a kernelized Gumbel-Softmax operator that seamlessly synthesizes random feature map [27] and approximated sampling strategy [16], for distilling latent structures among all the instance nodes and yielding moderate gradients through differentiable optimization.

Original: Though such a combination of two operations involving randomness could potentially result in mutual distortion, we theoretically prove that the new operator can still guarantee a wellposed approximation for concrete variables (discrete structures) with the error bounded by feature dimensions.
Revised: Though such a combination of two operations involving randomness could potentially result in mutual distortion, we theoretically prove that the new operator can still guarantee a wellposed approximation for concrete variables (discrete structures) with the error bounded by feature dimensions.

Original: Furthermore, such a design can reduce the algorithmic complexity of learning new topology per layer to O ( N ) by avoiding explicit computation for the cumbersome all-pair similarity.
Revised: Furthermore, such a design can reduce the algorithmic complexity of learning new topology per layer to O ( N ) by avoiding explicit computation for the cumbersome all-pair similarity.

Original: The proposed module opens the door to a new class of graph networks, i.e., N ODE F ORMER ( Scalable
Revised: The proposed module opens the door to a new class of graph networks, i.e., N ODE F ORMER ( Scalable

Original: Transformers for Node Classiﬁcation ), that is capable of efﬁciently propagating messages between arbitrary node pairs in ﬂexible layer-speciﬁc latent graphs.
Revised: Transformers for Node Classiﬁcation ), that is capable of efﬁciently propagating messages between arbitrary node pairs in ﬂexible layer-speciﬁc latent graphs.

Original: And to accommodate input graphs (if any), we devise two simple techniques: a relational bias and an edge-level regularization loss, as guidance for properly learning adaptive structures.
Revised: And to accommodate input graphs (if any), we devise two simple techniques: a relational bias and an edge-level regularization loss, as guidance for properly learning adaptive structures.

Original: We evaluate our approach on diverse node classiﬁcation tasks ranging from citation networks to images/texts.
Revised: We evaluate our approach on diverse node classiﬁcation tasks ranging from citation networks to images/texts.

Original: The results show its promising power for tackling heterophily, long-range dependencies, large-scale graphs, graph incompleteness and the absence of input graphs.
Revised: The results show its promising power for tackling heterophily, long-range dependencies, large-scale graphs, graph incompleteness and the absence of input graphs.

Original: The contributions of this paper are summarized as follows:
Revised: The contributions of this paper are summarized as follows:

Original: • We develop a kernelized Gumbel-Softmax operator which is proven to serve as a well-posed approximation for concrete variables, particularly the discrete latent structure among data points.
Revised: • We develop a kernelized Gumbel-Softmax operator which is proven to serve as a well-posed approximation for concrete variables, particularly the discrete latent structure among data points.

Original: The new module can reduce the algorithmic complexity for learning new message-passing topology from quadratic to linear w.r.t.
Revised: The new module can reduce the algorithmic complexity for learning new message-passing topology from quadratic to linear w.r.t.

Original: node numbers, without sacriﬁcing the precision.
Revised: node numbers, without sacriﬁcing the precision.

Original: This serves as a pioneering model that successfully scales graph structure learning to large graphs with million-level nodes.
Revised: This serves as a pioneering model that successfully scales graph structure learning to large graphs with million-level nodes.

Original: • We further propose N ODE F ORMER , a new class of graph networks with layer-wise message passing as operated over latent graphs potentially connecting all nodes.
Revised: • We further propose N ODE F ORMER , a new class of graph networks with layer-wise message passing as operated over latent graphs potentially connecting all nodes.

Original: The latter are optimized in an end-to-end differentiable fashion through a new objective that essentially pursues sampling optimal topology from a posterior conditioned on node features and labels.
Revised: The latter are optimized in an end-to-end differentiable fashion through a new objective that essentially pursues sampling optimal topology from a posterior conditioned on node features and labels.

Original: To our knowledge, N ODE F ORMER is the ﬁrst Transformer model that scales all-pair message passing to large node classiﬁcation graphs.
Revised: To our knowledge, N ODE F ORMER is the ﬁrst Transformer model that scales all-pair message passing to large node classiﬁcation graphs.

Original: • We demonstrate the model’s efﬁcacy by extensive experiments over a diverse set of datasets, including node classiﬁcation benchmarks and image/text classiﬁcation, where signiﬁcant improvement over strong GNN models and SOTA structure learning methods is shown.
Revised: • We demonstrate the model’s efﬁcacy by extensive experiments over a diverse set of datasets, including node classiﬁcation benchmarks and image/text classiﬁcation, where signiﬁcant improvement over strong GNN models and SOTA structure learning methods is shown.

Original: Besides, it successfully scales to large graph datasets with up to 2M nodes where prior arts failed, and reduces the time/space consumption of the competitors by up to 93.1%/80.6% on moderate sized datasets.
Revised: Besides, it successfully scales to large graph datasets with up to 2M nodes where prior arts failed, and reduces the time/space consumption of the competitors by up to 93.1%/80.6% on moderate sized datasets.

Original:  Related Works
Revised:  Related Works

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: Building expressive GNNs is a fundamental problem in learning over graph data.
Revised: Building expressive GNNs is a fundamental problem in learning over graph data.

Original: With Graph Attention Networks (GAT) [36] as an early attempt, there are many follow-up works, e.g., [22, 42], considering weighting the edges in input graph for enhancing the expressiveness.
Revised: With Graph Attention Networks (GAT) [36] as an early attempt, there are many follow-up works, e.g., [22, 42], considering weighting the edges in input graph for enhancing the expressiveness.

Original: Other studies, e.g., [28, 52] focus on sparsifying input structures to promote robust representations.
Revised: Other studies, e.g., [28, 52] focus on sparsifying input structures to promote robust representations.

Original: There are also quite a few approaches that propose scalable GNNs through, e.g., subgraph sampling [48], linear feature mapping [39], and channel-wise transformation [49], etc.
Revised: There are also quite a few approaches that propose scalable GNNs through, e.g., subgraph sampling [48], linear feature mapping [39], and channel-wise transformation [49], etc.

Original: However, these works cannot learn new edges out of the scope of input geometry, which may limit the model’s receptive ﬁelds within local neighbors and neglect global information.
Revised: However, these works cannot learn new edges out of the scope of input geometry, which may limit the model’s receptive ﬁelds within local neighbors and neglect global information.

Original: Graph Structure Learning.
Revised: Graph Structure Learning.

Original: Going beyond observed topology, graph structure learning targets learning a new graph for message passing among all the instances [54].
Revised: Going beyond observed topology, graph structure learning targets learning a new graph for message passing among all the instances [54].

Original: One line of work is similaritydriven where the conﬁdence of edges are reﬂected by some similarity functions between node pairs, e.g., Gaussian kernels [43], cosine similarity [4], attention networks [17], non-linear MLP [7] etc.
Revised: One line of work is similaritydriven where the conﬁdence of edges are reﬂected by some similarity functions between node pairs, e.g., Gaussian kernels [43], cosine similarity [4], attention networks [17], non-linear MLP [7] etc.

Original: Another line of work optimizes the adjacency matrix.
Revised: Another line of work optimizes the adjacency matrix.

Original: Due to the increased optimization difﬁculties, some sophisticated training methods are introduced, such as bi-level optimization [11], variational
Revised: Due to the increased optimization difﬁculties, some sophisticated training methods are introduced, such as bi-level optimization [11], variational

Original: [Table] [Table]
Revised: [Table] [Table]

Original: approaches [10, 20], Bayesian inference [51] and projected gradient descent [18].
Revised: approaches [10, 20], Bayesian inference [51] and projected gradient descent [18].

Original: To push further the limits of structure learning
Revised: To push further the limits of structure learning

Original: [Table]
Revised: [Table]

Original: this paper proposes a new model N ODE F ORMER (for enabling scalable node-level Transformers) whose merits are highlighted via a high-level comparison in Table 1.
Revised: this paper proposes a new model N ODE F ORMER (for enabling scalable node-level Transformers) whose merits are highlighted via a high-level comparison in Table 1.

Original: In particular, N ODE F ORMER enables efﬁcient structure learning in each layer, does not require input graphs and successfully scales to graphs with 2M nodes.
Revised: In particular, N ODE F ORMER enables efﬁcient structure learning in each layer, does not require input graphs and successfully scales to graphs with 2M nodes.

Original: Node-Level v.s. Graph-Level Prediction.
Revised: Node-Level v.s. Graph-Level Prediction.

Original: We emphasize upfront that our focus is on node-level prediction tasks involving a single large graph such that scalability is paramount, especially if we are to consider arbitrary relationships across all nodes (each node is an instance with label and one can treat all the nodes non-i.i.d.
Revised: We emphasize upfront that our focus is on node-level prediction tasks involving a single large graph such that scalability is paramount, especially if we are to consider arbitrary relationships across all nodes (each node is an instance with label and one can treat all the nodes non-i.i.d.

Original: generated due to the inter-dependence) for structure-learning purposes.
Revised: generated due to the inter-dependence) for structure-learning purposes.

Original: Critically though, this scenario is quite distinct from graph-level classiﬁcation tasks whereby each i.i.d.
Revised: Critically though, this scenario is quite distinct from graph-level classiﬁcation tasks whereby each i.i.d.

Original: instance is itself a small graph and fully connecting nodes within each graph is computationally inexpensive.
Revised: instance is itself a small graph and fully connecting nodes within each graph is computationally inexpensive.

Original: While this latter scenario has been explored in the context of graph structure learning [38] and all-pair message passing design, e.g., graph Transformers [9], existing efforts do not scale to the large graphs endemic to node-level prediction.
Revised: While this latter scenario has been explored in the context of graph structure learning [38] and all-pair message passing design, e.g., graph Transformers [9], existing efforts do not scale to the large graphs endemic to node-level prediction.

Original:  N ODE F ORMER : A Transformer Graph Network at Scale
Revised:  N ODE F ORMER : A Transformer Graph Network at Scale

Original: Let G =
Revised: Let G =

Original: ( N , E ) denote a graph with N a node set ( |N| = N ) and E ✓ N ⇥ N an edge set ( |E| = E ).
Revised: ( N , E ) denote a graph with N a node set ( |N| = N ) and E ✓ N ⇥ N an edge set ( |E| = E ).

Original: Each node u 2 N is assigned with node features x u 2 R D and a label y u .
Revised: Each node u 2 N is assigned with node features x u 2 R D and a label y u .

Original: We deﬁne an adjacency matrix A
Revised: We deﬁne an adjacency matrix A

Original: = { a uv } 2 { 0 , 1 } N ⇥ N where a uv = 1 if edge ( u, v ) 2 E and a uv = 0 otherwise.
Revised: = { a uv } 2 { 0 , 1 } N ⇥ N where a uv = 1 if edge ( u, v ) 2 E and a uv = 0 otherwise.

Original: Without loss of generality,
Revised: Without loss of generality,

Original: E could be an empty set in case of no input structure.
Revised: E could be an empty set in case of no input structure.

Original: There are two common settings: transductive learning, where testing nodes are within the graph used for training, and inductive learning which handles new unseen nodes out of the training graph.
Revised: There are two common settings: transductive learning, where testing nodes are within the graph used for training, and inductive learning which handles new unseen nodes out of the training graph.

Original: The target is to learn a function for node-level prediction, i.e., estimate labels for unlabeled or new nodes in the graph.
Revised: The target is to learn a function for node-level prediction, i.e., estimate labels for unlabeled or new nodes in the graph.

Original: General Model and Key Challenges.
Revised: General Model and Key Challenges.

Original: We start with the observation that the input structures may not be the ideal one for propagating signals among nodes and instead there exist certain latent structures that could facilitate learning better node representations.
Revised: We start with the observation that the input structures may not be the ideal one for propagating signals among nodes and instead there exist certain latent structures that could facilitate learning better node representations.

Original: We thus consider the updating rule
Revised: We thus consider the updating rule

Original: [Equation]
Revised: [Equation]

Original: where Z ( l ) = { z ( l ) u } u 2 N
Revised: where Z ( l ) = { z ( l ) u } u 2 N

Original: and
Revised: and

Original: ˜ A ( l ) =
Revised: ˜ A ( l ) =

Original: { ˜ a ( l ) uv } u,v 2 N denotes the node representations and the estimated latent graph of the l -th layer, respectively, and g , h are both differentiable functions aiming at 1) structure estimation for a layer-speciﬁc latent graph ˜ A ( l ) based on node representations and 2) feature propagation for updating node representations, respectively.
Revised: { ˜ a ( l ) uv } u,v 2 N denotes the node representations and the estimated latent graph of the l -th layer, respectively, and g , h are both differentiable functions aiming at 1) structure estimation for a layer-speciﬁc latent graph ˜ A ( l ) based on node representations and 2) feature propagation for updating node representations, respectively.

Original: The model deﬁned by Eqn.
Revised: The model deﬁned by Eqn.

Original: 1 follows the spirit of Transformers [35] (where in particular ˜
Revised: 1 follows the spirit of Transformers [35] (where in particular ˜

Original: A ( l ) can be seen as an attentive graph) that potentially enables message passing between any node pair in each layer, which, however, poses two challenges :
Revised: A ( l ) can be seen as an attentive graph) that potentially enables message passing between any node pair in each layer, which, however, poses two challenges :

Original: • (Scalability) : How to reduce the prohibitive quadratic complexity for
Revised: • (Scalability) : How to reduce the prohibitive quadratic complexity for

Original: learning new graphs?
Revised: learning new graphs?

Original: • (Differentiability) : How to enable end-to-end differentiable optimization for discrete structures?
Revised: • (Differentiability) : How to enable end-to-end differentiable optimization for discrete structures?

Original: Notice that the ﬁrst challenge is non-trivial in node-level prediction tasks (the focus of our paper), since the latent graphs could potentially connect all the instance nodes (e.g., from thousands to millions, depending on dataset sizes), which is fairly hard to guarantee both precision and scalability.
Revised: Notice that the ﬁrst challenge is non-trivial in node-level prediction tasks (the focus of our paper), since the latent graphs could potentially connect all the instance nodes (e.g., from thousands to millions, depending on dataset sizes), which is fairly hard to guarantee both precision and scalability.

Original: 3.1 Efﬁcient Learning Discrete Structures
Revised: 3.1 Efﬁcient Learning Discrete Structures

Original: We describe our new message-passing scheme with an efﬁcient kernelized Gumbel-Softmax operator to resolve the aforementioned challenges.
Revised: We describe our new message-passing scheme with an efﬁcient kernelized Gumbel-Softmax operator to resolve the aforementioned challenges.

Original: We assume z (0) u
Revised: We assume z (0) u

Original: = x u as the initial node representation.
Revised: = x u as the initial node representation.

Original: Kernelized Message Passing.
Revised: Kernelized Message Passing.

Original: We deﬁne a full-graph attentive network that estimates latent interactions among instance nodes and enables corresponding densely-connected message passing:
Revised: We deﬁne a full-graph attentive network that estimates latent interactions among instance nodes and enables corresponding densely-connected message passing:

Original: [Equation]
Revised: [Equation]

Original: where W ( l ) Q , W ( l ) K and W ( l ) V are learnable parameters in l -th layer.
Revised: where W ( l ) Q , W ( l ) K and W ( l ) V are learnable parameters in l -th layer.

Original: We omit non-linearity activation (after aggregation) for brevity.
Revised: We omit non-linearity activation (after aggregation) for brevity.

Original: The updating for N nodes in one layer using Eqn. 2 requires prohibitive
Revised: The updating for N nodes in one layer using Eqn. 2 requires prohibitive

Original: O ( N 2 ) complexity.
Revised: O ( N 2 ) complexity.

Original: Also, given large N , the normalization in the denominator would shrink attention weights to zero and lead to gradient vanishing.
Revised: Also, given large N , the normalization in the denominator would shrink attention weights to zero and lead to gradient vanishing.

Original: We call this problem as over-normalizing .
Revised: We call this problem as over-normalizing .

Original: To accelerate the full-graph model, we observe that the dot-then-exponentiate operation in Eqn.
Revised: To accelerate the full-graph model, we observe that the dot-then-exponentiate operation in Eqn.

Original: 2 can be converted into a pairwise similarity function:
Revised: 2 can be converted into a pairwise similarity function:

Original: [Equation]
Revised: [Equation]

Original: where  ( · , · ) : R d ⇥ R d !
Revised: where  ( · , · ) : R d ⇥ R d !

Original: R is a positive-deﬁnite kernel measuring the pairwise similarity.
Revised: R is a positive-deﬁnite kernel measuring the pairwise similarity.

Original: The kernel function can be further approximated by random features (RF) [27]which serves as an unbiased estimation via  ( a , b ) =
Revised: The kernel function can be further approximated by random features (RF) [27]which serves as an unbiased estimation via  ( a , b ) =

Original: h  ( a ) ,  ( b ) i V ⇡  ( a ) >  ( b ) , where the ﬁrst equation is by Mercer’s theorem with  : R d !
Revised: h  ( a ) ,  ( b ) i V ⇡  ( a ) >  ( b ) , where the ﬁrst equation is by Mercer’s theorem with  : R d !

Original: V a basis function and V a high-dimensional vector space, and  ( · ) : R d !
Revised: V a basis function and V a high-dimensional vector space, and  ( · ) : R d !

Original: R m is a low-dimensional feature map with random transformation.
Revised: R m is a low-dimensional feature map with random transformation.

Original: There are many potential choices for  , e.g., Positive Random Features (PRF) [6]
Revised: There are many potential choices for  , e.g., Positive Random Features (PRF) [6]

Original: [Equation]
Revised: [Equation]

Original: where w k ⇠ N (0 , I d ) is i.i.d.
Revised: where w k ⇠ N (0 , I d ) is i.i.d.

Original: sampled random transformation.
Revised: sampled random transformation.

Original: The RF converts dot-thenexponentiate operation into inner-product in vector space, which enables us to re-write Eqn.(assuming q u = W ( l ) Q z ( l ) u , k u = W ( l ) K z ( l ) u and v u = W ( l ) V z ( l ) u for simplicity):
Revised: The RF converts dot-thenexponentiate operation into inner-product in vector space, which enables us to re-write Eqn.(assuming q u = W ( l ) Q z ( l ) u , k u = W ( l ) K z ( l ) u and v u = W ( l ) V z ( l ) u for simplicity):

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: The key advantage of Eqn.
Revised: The key advantage of Eqn.

Original: 5 is that the two summations are shared by each u , so that one only needs to compute them once and re-used for others.
Revised: 5 is that the two summations are shared by each u , so that one only needs to compute them once and re-used for others.

Original: Such a property enables O ( N ) computational complexity for full-graph message passing, which paves the way for learning graph structures among large-scale instances.
Revised: Such a property enables O ( N ) computational complexity for full-graph message passing, which paves the way for learning graph structures among large-scale instances.

Original: Moreover, one can notice that Eqn.
Revised: Moreover, one can notice that Eqn.

Original: 5 avoids computing the N ⇥ N similarity matrix, i.e., { ˜ a ( l ) uv } N ⇥ N , required by Eqn. 2, thus also reducing the learning difﬁculties.
Revised: 5 avoids computing the N ⇥ N similarity matrix, i.e., { ˜ a ( l ) uv } N ⇥ N , required by Eqn. 2, thus also reducing the learning difﬁculties.

Original: Nevertheless, Eqn. 5 still suffers what we mentioned the over-normalizing issue.
Revised: Nevertheless, Eqn. 5 still suffers what we mentioned the over-normalizing issue.

Original: The crux is that the message passing is operated on a weighted fully-connected graph where, in fact, only partial edges are important.
Revised: The crux is that the message passing is operated on a weighted fully-connected graph where, in fact, only partial edges are important.

Original: Also, such a deterministic way of feature aggregation over all the instances may increase the risk for over-ﬁtting, especially when N is large.
Revised: Also, such a deterministic way of feature aggregation over all the instances may increase the risk for over-ﬁtting, especially when N is large.

Original: We next resolve the issues by distilling a sparse structure from the fully-connected graph.
Revised: We next resolve the issues by distilling a sparse structure from the fully-connected graph.

Original: Differentiable Stochastic Structure Learning.
Revised: Differentiable Stochastic Structure Learning.

Original: The difﬁcultly lies in how to enable differentiable optimization for discrete graph structures.
Revised: The difﬁcultly lies in how to enable differentiable optimization for discrete graph structures.

Original: The weight ˜ a ( l ) uv given by Eqn. 2 could be used to deﬁne a categorical distribution for generating latent edges from distribution Cat ( ⇡ ( l ) u ) where ⇡ ( l ) u
Revised: The weight ˜ a ( l ) uv given by Eqn. 2 could be used to deﬁne a categorical distribution for generating latent edges from distribution Cat ( ⇡ ( l ) u ) where ⇡ ( l ) u

Original: = { ⇡ ( l ) uv } Nv =1 and ⇡ ( l ) uv =
Revised: = { ⇡ ( l ) uv } Nv =1 and ⇡ ( l ) uv =

Original: p ( v | u ) =
Revised: p ( v | u ) =

Original: ˜ a ( l ) uv .
Revised: ˜ a ( l ) uv .

Original: Then in principle, we can sample over the categorical distribution multiple times for each node to obtain its neighbors.
Revised: Then in principle, we can sample over the categorical distribution multiple times for each node to obtain its neighbors.

Original: However, the sampling process would introduce discontinuity and hinders back-propagation.
Revised: However, the sampling process would introduce discontinuity and hinders back-propagation.

Original: Fortunately, we notice that the Eqn.can be modiﬁed to incorporate the reparametrization trick [16] to allow differentiable learning:
Revised: Fortunately, we notice that the Eqn.can be modiﬁed to incorporate the reparametrization trick [16] to allow differentiable learning:

Original: [Equation]
Revised: [Equation]

Original: where g u is i.i.d.
Revised: where g u is i.i.d.

Original: sampled from Gumbel distribution and ⌧ is a temperature coefﬁcient.
Revised: sampled from Gumbel distribution and ⌧ is a temperature coefﬁcient.

Original: Eqn. 6 is a continuous relaxation of sampling one neighbored node for u over Cat ( ⇡ ( l ) u ) and ⌧ controls the closeness to hard discrete samples [23].
Revised: Eqn. 6 is a continuous relaxation of sampling one neighbored node for u over Cat ( ⇡ ( l ) u ) and ⌧ controls the closeness to hard discrete samples [23].

Original: Following similar reasoning as Eqn. 3 and 5, we can yield z ( l +1) u
Revised: Following similar reasoning as Eqn. 3 and 5, we can yield z ( l +1) u

Original: [Equation]
Revised: [Equation]

Original: Eqn. 7 achieves message passing over a sampled latent graph (where we only sample once for each node) and still guarantees linear complexity as Eqn. 5.
Revised: Eqn. 7 achieves message passing over a sampled latent graph (where we only sample once for each node) and still guarantees linear complexity as Eqn. 5.

Original: In practice, we can sample K times (e.g.,
Revised: In practice, we can sample K times (e.g.,

Original: K = 5 ) for each node and take an average of the aggregated results.
Revised: K = 5 ) for each node and take an average of the aggregated results.

Original: Due to space limit, we defer more details concerning the differentiable sampling-based message passing to Appendix A. Besides, in Fig.
Revised: Due to space limit, we defer more details concerning the differentiable sampling-based message passing to Appendix A. Besides, in Fig.

Original: 5 and Alg.
Revised: 5 and Alg.

Original: 1 of Appendix A, we present an illustration for node embedding updating in each layer, from a matrix view that is practically used for implementation.
Revised: 1 of Appendix A, we present an illustration for node embedding updating in each layer, from a matrix view that is practically used for implementation.

Original: 3.2 Well-posedness of the Kernelized Gumbel-Softmax Operator
Revised: 3.2 Well-posedness of the Kernelized Gumbel-Softmax Operator

Original: One reasonable concern for Eqn.
Revised: One reasonable concern for Eqn.

Original: 7 is whether the RF approximation for kernel functions maintains the well-posedness of Gumbel approximation for the target discrete variables.
Revised: 7 is whether the RF approximation for kernel functions maintains the well-posedness of Gumbel approximation for the target discrete variables.

Original: As a justiﬁcation for the new message-passing function, we next answer two theoretical questions: 1) How is the approximation capability of RF for the original dot-then-exponentiate operation with Gumbel variables in Eqn.
Revised: As a justiﬁcation for the new message-passing function, we next answer two theoretical questions: 1) How is the approximation capability of RF for the original dot-then-exponentiate operation with Gumbel variables in Eqn.

Original: Does Eqn.
Revised: Does Eqn.

Original: 7 still guarantee a continuous relaxation of the categorical distributions?
Revised: 7 still guarantee a continuous relaxation of the categorical distributions?

Original: We formulate the results as follows and defer proofs to Appendix B.
Revised: We formulate the results as follows and defer proofs to Appendix B.

Original: Theorem 1 (Approximation Error for Softmax-Kernel) .
Revised: Theorem 1 (Approximation Error for Softmax-Kernel) .

Original: Assume k
Revised: Assume k

Original: q u k 

Revised: q u k

Original: and k k v k 2 are bounded by r , then with probability at least 1  ✏ , the gap  = 
Revised: and k k v k 2 are bounded by r , then with probability at least 1  ✏ , the gap  =

Original:  ( q u / p ⌧ ) >  ( k
Revised:  ( q u / p ⌧ ) >  ( k

Original: [Equation]
Revised: [Equation]

Original: / p ⌧
Revised: / p ⌧

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: p ⌧
Revised: p ⌧

Original: [Equation]
Revised: [Equation]

Original: p ⌧ )  )
Revised: p ⌧ )  )

Original: [Equation]
Revised: [Equation]

Original: where  is deﬁned by Eqn. 4, will be bounded by O
Revised: where  is deﬁned by Eqn. 4, will be bounded by O

Original: [Equation]
Revised: [Equation]

Original: We can see that the error bound of RF for approximating original softmax-kernel function depends on both the dimension of feature map  and temperature ⌧ .
Revised: We can see that the error bound of RF for approximating original softmax-kernel function depends on both the dimension of feature map  and temperature ⌧ .

Original: Notably, the error bound is independent of node number N , which implies that the approximation ability is insensitive to dataset sizes.
Revised: Notably, the error bound is independent of node number N , which implies that the approximation ability is insensitive to dataset sizes.

Original: The second question is non-trivial since Eqn.
Revised: The second question is non-trivial since Eqn.

Original: 7 involves randomness of Gumbel variables and random transformation in  , which cannot be decoupled apart.
Revised: 7 involves randomness of Gumbel variables and random transformation in  , which cannot be decoupled apart.

Original: We deﬁne c uv = 
Revised: We deﬁne c uv =

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: Nw =1  ( q u / p ⌧ ) >  ( k w / p ⌧ ) e gw/ ⌧ as the result from the kernelized Gumbel-Softmax and c u = { c uv } Nv =1 denotes the sampled edge vector for node u .
Revised: Nw =1  ( q u / p ⌧ ) >  ( k w / p ⌧ ) e gw/ ⌧ as the result from the kernelized Gumbel-Softmax and c u = { c uv } Nv =1 denotes the sampled edge vector for node u .

Original: We can arrive at the result as follows.
Revised: We can arrive at the result as follows.

Original: Theorem 2 (Property of Kernelized Gumbel-Softmax Random Variables) .
Revised: Theorem 2 (Property of Kernelized Gumbel-Softmax Random Variables) .

Original: Suppose m is sufﬁciently large, we have the convergence property for the kernelized Gumbel-Softmax operator
Revised: Suppose m is sufﬁciently large, we have the convergence property for the kernelized Gumbel-Softmax operator

Original: [Equation]
Revised: [Equation]

Original: It shows that when i) the dimension of feature map is large enough and ii) the temperature goes to zero, the distribution from which latent structures are sampled would converge to the original categorical distribution.
Revised: It shows that when i) the dimension of feature map is large enough and ii) the temperature goes to zero, the distribution from which latent structures are sampled would converge to the original categorical distribution.

Original: Remark.
Revised: Remark.

Original: The two theorems imply a trade-off between RF approximation and Gumbel-Softmax approximation w.r.t.
Revised: The two theorems imply a trade-off between RF approximation and Gumbel-Softmax approximation w.r.t.

Original: the choice of ⌧ .
Revised: the choice of ⌧ .

Original: A large ⌧ would help to reduce the burden on kernel dimension m , and namely, small ⌧ would require a very large m to guarantee enough RF approximation precision.
Revised: A large ⌧ would help to reduce the burden on kernel dimension m , and namely, small ⌧ would require a very large m to guarantee enough RF approximation precision.

Original: On the other hand, if ⌧ is too large, the weight on each edge will converge to 1 N , i.e., the model nearly degrades to mean pooling, while a small ⌧ would endow the kernelized Gumbel-Softmax with better approximation to the categorical distribution.
Revised: On the other hand, if ⌧ is too large, the weight on each edge will converge to 1 N , i.e., the model nearly degrades to mean pooling, while a small ⌧ would endow the kernelized Gumbel-Softmax with better approximation to the categorical distribution.

Original: Empirical studies on this are presented in Appendix E.
Revised: Empirical studies on this are presented in Appendix E.

Original: 3.3 Input Structures as Relational Bias
Revised: 3.3 Input Structures as Relational Bias

Original: Eqn.
Revised: Eqn.

Original: 7 does not leverage any information from observed geometry which, however, is often recognized important for modeling physically-structured data [3].
Revised: 7 does not leverage any information from observed geometry which, however, is often recognized important for modeling physically-structured data [3].

Original: We therefore accommodate input topology (if any) as relational bias via modifying the attention weight as ˜ a ( l ) uv
Revised: We therefore accommodate input topology (if any) as relational bias via modifying the attention weight as ˜ a ( l ) uv

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where b ( l ) is a learnable scalar as relational bias for any adjacent node pairs ( u, v ) and  is a certain (bounded) activation function like sigmoid.
Revised: where b ( l ) is a learnable scalar as relational bias for any adjacent node pairs ( u, v ) and  is a certain (bounded) activation function like sigmoid.

Original: The relational bias aims at assigning adjacent nodes in G with proper weights, and the node representations could be accordingly updated by
Revised: The relational bias aims at assigning adjacent nodes in G with proper weights, and the node representations could be accordingly updated by

Original: [Equation]
Revised: [Equation]

Original: Eqn. 8 increases the algorithmic complexity for message passing to O ( N + E ) , albeit within the same order-of-magnitude as common GNNs operating on input graphs.
Revised: Eqn. 8 increases the algorithmic complexity for message passing to O ( N + E ) , albeit within the same order-of-magnitude as common GNNs operating on input graphs.

Original: Also, one can consider higher-order adjacency as relational bias for better expressiveness at some expense of efﬁciency, as similarly done by [1].
Revised: Also, one can consider higher-order adjacency as relational bias for better expressiveness at some expense of efﬁciency, as similarly done by [1].

Original: We summarize the feed-forward computation of N ODE F ORMER in Alg. 1.
Revised: We summarize the feed-forward computation of N ODE F ORMER in Alg. 1.

Original: 3.4 Learning Objective
Revised: 3.4 Learning Objective

Original: Given training labels Y tr = { y u } u 2 N tr , where N tr denotes the set of labeled nodes, the common practice is to maximize the observed data log-likelihood which yields a supervised loss (with C classes)
Revised: Given training labels Y tr = { y u } u 2 N tr , where N tr denotes the set of labeled nodes, the common practice is to maximize the observed data log-likelihood which yields a supervised loss (with C classes)

Original: [Equation]
Revised: [Equation]

Original: where I [ · ] is an indicator function.
Revised: where I [ · ] is an indicator function.

Original: However, it may not sufﬁce to generalize well due to that the graph topology learning increases the degrees of freedom and the number of training labels is not comparable to that.
Revised: However, it may not sufﬁce to generalize well due to that the graph topology learning increases the degrees of freedom and the number of training labels is not comparable to that.

Original: Therefore, we additionally introduce an edge-level regularization:
Revised: Therefore, we additionally introduce an edge-level regularization:

Original: [Equation]
Revised: [Equation]

Original: where d u denotes the in-degree of node u and ⇡ ( l ) uv is the predicted probability for edge ( u, v ) at the l -th layer.
Revised: where d u denotes the in-degree of node u and ⇡ ( l ) uv is the predicted probability for edge ( u, v ) at the l -th layer.

Original: Eqn.
Revised: Eqn.

Original: 10 is a maximum likelihood estimation for edges in E , with data distribution deﬁned
Revised: 10 is a maximum likelihood estimation for edges in E , with data distribution deﬁned

Original: [Equation]
Revised: [Equation]

Original: We next show how to efﬁciently obtain ⇡ ( l ) uv .
Revised: We next show how to efﬁciently obtain ⇡ ( l ) uv .

Original: Although the feed-forward N ODE F ORMER computation deﬁned by Eqn.
Revised: Although the feed-forward N ODE F ORMER computation deﬁned by Eqn.

Original: 7 does not explicitly produce the value for each ⇡ ( l ) uv , we can query their values by
Revised: 7 does not explicitly produce the value for each ⇡ ( l ) uv , we can query their values by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A cc u r a cy

Original: Cora

Original: A cc u r a cy CiteSeer 60. A cc u r a cy Deezer 25.
Revised: [Table]

Original: A cc u r a cy

Original: Actor GCN GAT MixHop
Revised: [Table]

Original: JKNet

Original: DropEdge LDS IDGL NodeFormer is caused by out-
Revised: where the summation term can be re-used from once computation

Original: [Table]

Original: [Table]
Revised: [Table]

Original: where the summation term can be re-used from once computation, as is done by Eqn. 5 and Eqn. 7.
Revised: as is done by Eqn. 5 and Eqn. 7.

Original: Therefore, after once computation for the summation that requires O ( N ) , the computation for each ⇡ ( l ) uv requires O (1) complexity, yielding the total complexity controlled within O ( E ) (since we only need to query the observed edges).
Revised: Therefore, after once computation for the summation that requires O ( N ) , the computation for each ⇡ ( l ) uv requires O (1) complexity, yielding the total complexity controlled within O ( E ) (since we only need to query the observed edges).

Original: The ﬁnal objective can be the combination of two: L =
Revised: The ﬁnal objective can be the combination of two: L =

Original: L s +  L e , where  controls how much emphasis is put on input topology.
Revised: L s +  L e , where  controls how much emphasis is put on input topology.

Original: We depict the whole data ﬂow of
Revised: We depict the whole data ﬂow of

Original: N ODE F ORMER ’s training in Fig.
Revised: N ODE F ORMER ’s training in Fig.

Original:  Evaluation
Revised:  Evaluation

Original: We consider a diverse set of datasets for experiments and present detailed dataset information in
Revised: We consider a diverse set of datasets for experiments and present detailed dataset information in

Original: Appendix D. For implementation, we set  as sigmoid function and ⌧ as 0.25 for all datasets.
Revised: Appendix D. For implementation, we set  as sigmoid function and ⌧ as 0.25 for all datasets.

Original: The output prediction layer is a one-layer MLP.
Revised: The output prediction layer is a one-layer MLP.

Original: More implementation details are presented in Appendix C.
Revised: More implementation details are presented in Appendix C.

Original: All experiments are conducted on a NVIDIA V100 with 16 GB memory.
Revised: All experiments are conducted on a NVIDIA V100 with 16 GB memory.

Original: As baseline models, we basically consider GCN [19] and GAT [36].
Revised: As baseline models, we basically consider GCN [19] and GAT [36].

Original: Besides, we compare with some advanced GNN models, including JKNet [44] and MixHop [1].
Revised: Besides, we compare with some advanced GNN models, including JKNet [44] and MixHop [1].

Original: These GNN models all rely on input graphs.
Revised: These GNN models all rely on input graphs.

Original: We further consider DropEdge [28] and two SOTA graph structure learning methods,
Revised: We further consider DropEdge [28] and two SOTA graph structure learning methods,

Original: LDS-GNN [11] and IDGL [4] for comparison.
Revised: LDS-GNN [11] and IDGL [4] for comparison.

Original: For large-scale datasets, we additionally compare with two scalable GNNs, a linear model SGC [39] and a graph-sampling model GraphSAINT [48].
Revised: For large-scale datasets, we additionally compare with two scalable GNNs, a linear model SGC [39] and a graph-sampling model GraphSAINT [48].

Original: More detailed information about these models are presented in Appendix C. All the experiments are repeated ﬁve times with different initializations.
Revised: More detailed information about these models are presented in Appendix C. All the experiments are repeated ﬁve times with different initializations.

Original: 4.1 Experiments on Transductive Node Classiﬁcation
Revised: 4.1 Experiments on Transductive Node Classiﬁcation

Original: We study supervised node classiﬁcation in transductive setting on common graph datasets: Cora ,
Revised: We study supervised node classiﬁcation in transductive setting on common graph datasets: Cora ,

Original: Citeseer , Deezer and Actor .
Revised: Citeseer , Deezer and Actor .

Original: The ﬁrst two have high homophily ratios and the last two are identiﬁed as heterophilic graphs [53, 21].
Revised: The ﬁrst two have high homophily ratios and the last two are identiﬁed as heterophilic graphs [53, 21].

Original: These datasets are of small or medium sizes (with 2K ⇠ 20K nodes).
Revised: These datasets are of small or medium sizes (with 2K ⇠ 20K nodes).

Original: We use random splits with train/valid/test ratios as 50%/25%/25% and accuracy on test nodes for evaluation.
Revised: We use random splits with train/valid/test ratios as 50%/25%/25%.

Revised: For evaluation metrics, we use ROC-AUC for binary classiﬁcation on Deezer and Accuracy for other datasets with more than 2 classes.

Original: Results are plotted in Fig.
Revised: Results are plotted in Fig.

Original: 2 and N ODE F ORMER achieves the best mean accuracy across four datasets and in particular, outperforms other models by a large margin on two heterophilic graphs.
Revised: 2 and N ODE F ORMER achieves the best mean Accuracy/ROCAUC across four datasets and in particular, outperforms other models by a large margin on two heterophilic graphs.

Original: The results indicate that N ODE F ORMER can handle both homophilious and nonhomophilious graphs.
Revised: The results indicate that N ODE F ORMER can handle both homophilious and nonhomophilious graphs.

Original: Compared with two structure learning models LDS and IDGL, N ODE F ORMER yields signiﬁcantly better performance, which shows its superiority.
Revised: Compared with two structure learning models LDS and IDGL, N ODE F ORMER yields signiﬁcantly better performance, which shows its superiority.

Original: Also, for deezer , LDS and
Revised: Also, for Deezer , LDS and

Original: IDGL suffers from out-of-memory (OOM).
Revised: IDGL suffers from out-of-memory (OOM).

Original: In fact, the major difﬁculty for deezer is the large
Revised: In fact, the major difﬁculty for Deezer is the large

Original: [Table]
Revised: [Table]

Original: (nearly 30K), which causes OOM for IDGL even with the anchor approximation.
Revised: (nearly 30K), which causes OOM for IDGL even with the anchor approximation.

Original: In contrast, N ODE F ORMER manages to scale and produce desirable accuracy.
Revised: In contrast, N ODE F ORMER manages to scale and produce desirable accuracy.

Original: 4.2 Experiments on Larger Graph Datasets
Revised: 4.2 Experiments on Larger Graph Datasets

Original: To further test the scalability, we consider two large-sized networks, OGB-Proteins and Amazon2M , with over 0.1 million and 2 million of nodes, respectively.
Revised: To further test the scalability, we consider two large-sized networks, OGB-Proteins and Amazon2M , with over 0.1 million and 2 million of nodes, respectively.

Original: OGB-Proteins is a multi-task dataset with 112 output dimensions, while Amazon2M is extracted from the Amazon Co-Purchasing network that entails long-range dependence [13].
Revised: OGB-Proteins is a multi-task dataset with 112 output dimensions, while Amazon2M is extracted from the Amazon Co-Purchasing network that entails long-range dependence [13].

Original: For OGB-Proteins , we use the protocol of [15] and ROC-AUC for evaluation.
Revised: For OGB-Proteins , we use the protocol of [15] and ROC-AUC for evaluation.

Original: For Amazon2M , we adopt random splitting with 50%/25%/25% nodes for training, validation and testing, respectively.
Revised: For Amazon2M , we adopt random splitting with 50%/25%/25% nodes for training, validation and testing, respectively.

Original: Due to the large dataset size, we adopt mini-batch partition for training, in which case, for N ODE F ORMER we only consider structure learning among nodes in a random mini-batch.
Revised: Due to the large dataset size, we adopt mini-batch partition for training, in which case, for N ODE F ORMER we only consider structure learning among nodes in a random mini-batch.

Original: We use batch size 10000 and 100000 for Proteins and Amazon2M , respectively.
Revised: We use batch size 10000 and 100000 for Proteins and Amazon2M , respectively.

Original: While the mini-batch partition may sacriﬁce the exposure to all instances, we found using large batch size can yield decent performance, which is also allowable thanks to the O ( N ) complexity of our model.
Revised: While the mini-batch partition may sacriﬁce the exposure to all instances, we found using large batch size can yield decent performance, which is also allowable thanks to the O ( N ) complexity of our model.

Original: For example, even setting the batch size as 100000, we found N ODE F ORMER costs only 4GB GPU memory for training on Amazon2M .
Revised: For example, even setting the batch size as 100000, we found N ODE F ORMER costs only 4GB GPU memory for training on Amazon2M .

Original: Table 2 presents the results on OGB-Proteins where for fair comparison mini-batch training is also used for other models except GraphSAINT.
Revised: Table 2 presents the results on OGB-Proteins where for fair comparison mini-batch training is also used for other models except GraphSAINT.

Original: We found that N ODE F ORMER yields much better ROC-AUC and only requires comparable memory as simple GNN models.
Revised: We found that N ODE F ORMER yields much better ROC-AUC and only requires comparable memory as simple GNN models.

Original: Table 3 reports the results on Amazon2M which shows that N ODE F ORMER outperforms baselines by a large margin and the memory cost is even fewer than GCN.
Revised: Table 3 reports the results on Amazon2M which shows that N ODE F ORMER outperforms baselines by a large margin and the memory cost is even fewer than GCN.

Original: This shows its practical efﬁcacy and scalability on large-scale datasets and also the capability for addressing long-range dependence with shallow layers (we use L = 3 ).
Revised: This shows its practical efﬁcacy and scalability on large-scale datasets and also the capability for addressing long-range dependence with shallow layers (we use L = 3 ).

Original: 4.3 Experiments on Graph-Enhanced Applications
Revised: 4.3 Experiments on Graph-Enhanced Applications

Original: We apply our model to semi-supervised image and text classiﬁcation on Mini-ImageNet and 20News-Groups datasets, without input graphs.
Revised: We apply our model to semi-supervised image and text classiﬁcation on Mini-ImageNet and 20News-Groups datasets, without input graphs.

Original: The instances of Mini-ImageNet [37] are 84 ×
Revised: The instances of Mini-ImageNet [37] are 84 ×

Original: RGB images and we randomly choose 30 classes each of which contains 600 samples for experiments.
Revised: RGB images and we randomly choose 30 classes each of which contains 600 samples for experiments.

Original: 20News-Groups [25] consists of nearly 10K texts whose features are extracted by TF-IDF.
Revised: 20News-Groups [25] consists of nearly 10K texts whose features are extracted by TF-IDF.

Original: More details for preprocessing are presented in Appendix D. Also, for each dataset, we randomly split instances into 50%/25%/25% for train/valid/test.
Revised: More details for preprocessing are presented in Appendix D. Also, for each dataset, we randomly split instances into 50%/25%/25% for train/valid/test.

Original: Since there is no input graph, we use k -NN (over input node features) for artiﬁcially constructing a graph for enabling GNN’s message passing and the graph-based components (edge regularization and relational bias) of N ODE F ORMER .
Revised: Since there is no input graph, we use k -NN (over input node features) for artiﬁcially constructing a graph for enabling GNN’s message passing and the graph-based components (edge regularization and relational bias) of N ODE F ORMER .

Original: Tablepresents the comparison results under different k ’s. We can see that N ODE F ORMER achieves the best performance in seven cases out of eight.
Revised: Tablepresents the comparison results under different k ’s. We can see that N ODE F ORMER achieves the best performance in seven cases out of eight.

Original: The performance of GNN competitors varies signiﬁcantly with different k values, and N ODE F ORMER is much less sensitive.
Revised: The performance of GNN competitors varies signiﬁcantly with different k values, and N ODE F ORMER is much less sensitive.

Original: Intriguingly, when we do not use the input graph, i.e., removing both the edge regularization and relational bias, N ODE F ORMER can still yield competitive even superior results on Mini-ImageNet .
Revised: Intriguingly, when we do not use the input graph, i.e., removing both the edge regularization and relational bias, N ODE F ORMER can still yield competitive even superior results on Mini-ImageNet .

Original: This suggests that the k -NN graphs are not necessarily informative and besides, our model learns useful latent graph structures from data.
Revised: This suggests that the k -NN graphs are not necessarily informative and besides, our model learns useful latent graph structures from data.

Original: 4.4 Further Discussions
Revised: 4.4 Further Discussions

Original: Comparison of Time/Space Consumption.
Revised: Comparison of Time/Space Consumption.

Original: Fig.
Revised: Fig.

Original: 3 plots training/inference time and GPU memory costs of N ODE F ORMER and two SOTA structure learning models.
Revised: 3 plots training/inference time and GPU memory costs of N ODE F ORMER and two SOTA structure learning models.

Original: Compared with LDS, N ODEF ORMER reduces the training time, inference time, memory cost by up to 93.1%, 97.9%, 75.6%, respectively; compared with IDGL (using anchor-based approximation for speedup), N ODE F ORMER reduces the training time, inference time, memory cost by up to 61.8%, 80.8%, 80.6%, respectively.
Revised: Compared with LDS, N ODEF ORMER reduces the training time, inference time, memory cost by up to 93.1%, 97.9%, 75.6%, respectively; compared with IDGL (using anchor-based approximation for speedup), N ODE F ORMER reduces the training time, inference time, memory cost by up to 61.8%, 80.8%, 80.6%, respectively.

Original: Ablation on Stochastic Components.
Revised: Ablation on Stochastic Components.

Original: Table 2 and 3 also include two variants of N ODE F ORMER for ablation study.
Revised: Table 2 and 3 also include two variants of N ODE F ORMER for ablation study.

Original: 1) N ODE F ORMER -dt: replace Gumbel-Softmax by original Softmax (with temperature 1.0) for deterministic propagation; 2) N ODE F ORMER -tp: use original Softmax with temperature set as 0.25 (the same as N ODE F ORMER ).
Revised: 1) N ODE F ORMER -dt: replace Gumbel-Softmax by original Softmax (with temperature 1.0) for deterministic propagation; 2) N ODE F ORMER -tp: use original Softmax with temperature set as 0.25 (the same as N ODE F ORMER ).

Original: There is performance drop when removing the Gumbel components, which may be due to over-normalizing or over-ﬁtting that are ampliﬁed in large datasets, as we discussed in Section 3.1 and the kernelized Gumbel-Softmax operator shows its effectiveness.
Revised: There is performance drop when removing the Gumbel components, which may be due to over-normalizing or over-ﬁtting that are ampliﬁed in large datasets, as we discussed in Section 3.1 and the kernelized Gumbel-Softmax operator shows its effectiveness.

Original: Ablation on Edge Loss and Relational Bias.
Revised: Ablation on Edge Loss and Relational Bias.

Original: We study the effects of edge-level regularization and relation bias as ablation study shown in Table 6 located in Appendix E, where the results consistently show that both components contribute to some positive effects and suggest that our edge-level loss and relation bias can both help to leverage useful information from input graphs.
Revised: We study the effects of edge-level regularization and relation bias as ablation study shown in Table 6 located in Appendix E, where the results consistently show that both components contribute to some positive effects and suggest that our edge-level loss and relation bias can both help to leverage useful information from input graphs.

Original: Impact of Temperature and Feature Map Dimension.
Revised: Impact of Temperature and Feature Map Dimension.

Original: We study the effects of ⌧ and m in Fig.located in Appendix E and the variation trend accords with our theoretical analysis in Section 3.2.
Revised: We study the effects of ⌧ and m in Fig.located in Appendix E and the variation trend accords with our theoretical analysis in Section 3.2.

Original: Speciﬁcally, the result shows that the test accuracy increases and then falls with the temperature changing from low to high values (usually achieves the peak accuracy with a temperature of 0.4).
Revised: Speciﬁcally, the result shows that the test accuracy increases and then falls with the temperature changing from low to high values (usually achieves the peak accuracy with a temperature of 0.4).

Original: Besides, we can see that when the temperature is relatively small, the test accuracy goes high with the dimension of random features increasing.
Revised: Besides, we can see that when the temperature is relatively small, the test accuracy goes high with the dimension of random features increasing.

Original: However, when the temperature is large, the accuracy would drop even with large feature dimension m .
Revised: However, when the temperature is large, the accuracy would drop even with large feature dimension m .

Original: Such a phenomenon accords with the theoretical result presented in Section 3.2.
Revised: Such a phenomenon accords with the theoretical result presented in Section 3.2.

Original: For low temperature which enables desirable approximation performance for
Revised: For low temperature which enables desirable approximation performance for

Original: Gumbel-Softmax, then larger random feature dimension would help to produce better approximation to the original exponentiate-then-dot operator.
Revised: Gumbel-Softmax, then larger random feature dimension would help to produce better approximation to the original exponentiate-then-dot operator.

Original: In contrast, high temperature could not guarantee precise approximation for the original categorical distribution, which deteriorates the performance.
Revised: In contrast, high temperature could not guarantee precise approximation for the original categorical distribution, which deteriorates the performance.

Original: Visualization and Implications.
Revised: Visualization and Implications.

Original: Fig.
Revised: Fig.

Original: 4 visualizes node embeddings and edge connections (ﬁlter out the edges with weights larger than a threshold) on 20News-Groups and Mini-Imagenet , which show that N ODE F ORMER tends to assign more weights for nodes with the same class and sparse edges for nodes with different classes.
Revised: 4 visualizes node embeddings and edge connections (ﬁlter out the edges with weights larger than a threshold) on 20News-Groups and Mini-Imagenet , which show that N ODE F ORMER tends to assign more weights for nodes with the same class and sparse edges for nodes with different classes.

Original: This helps to interpret why N ODE F ORMER improves the performance on downstream node-level prediction: the latent structures can propagate useful information to help the model learn better node representations that can be easily distinguished by the classiﬁer.
Revised: This helps to interpret why N ODE F ORMER improves the performance on downstream node-level prediction: the latent structures can propagate useful information to help the model learn better node representations that can be easily distinguished by the classiﬁer.

Original: We also compare the learned structures with original graphs in Fig.
Revised: We also compare the learned structures with original graphs in Fig.

Original: 7 located in
Revised: 7 located in

Original: Appendix E.
Revised: Appendix E.

Original: We can see that the latent structures learned by N ODE F ORMER show different patterns from the observed ones, especially for heterophilic graphs.
Revised: We can see that the latent structures learned by N ODE F ORMER show different patterns from the observed ones, especially for heterophilic graphs.

Original: Another interesting phenomenon is that there exist some dominant nodes which are assigned large weights by other nodes, forming some vertical ‘lines’ in the heatmap.
Revised: Another interesting phenomenon is that there exist some dominant nodes which are assigned large weights by other nodes, forming some vertical ‘lines’ in the heatmap.

Original: This suggests that these nodes could contain critical information for the learning tasks and play as pivots that could improve the connectivity of the whole system.
Revised: This suggests that these nodes could contain critical information for the learning tasks and play as pivots that could improve the connectivity of the whole system.

Original:  Why N ODE F ORMER Improves Downstream Prediction?
Revised:  Why N ODE F ORMER Improves Downstream Prediction?

Original: There remains a natural question concerning our learning process: how effective can the learned latent topology be for downstream tasks?
Revised: There remains a natural question concerning our learning process: how effective can the learned latent topology be for downstream tasks?

Original: We next dissect the rationale from a Bayesian perspective.
Revised: We next dissect the rationale from a Bayesian perspective.

Original: In fact, our model induces a predictive distribution p ( Y , ˜
Revised: In fact, our model induces a predictive distribution p ( Y , ˜

Original: A | X , A ) =
Revised: A | X , A ) =

Original: p ( ˜ A
Revised: p ( ˜ A

Original: | X , A ) p ( Y | ˜
Revised: | X , A ) p ( Y | ˜

Original: A , X , A ) where we can treat the estimated graph ˜ A as a latent variable.
Revised: A , X , A ) where we can treat the estimated graph ˜ A as a latent variable.

Original: Speciﬁcally, p ( ˜ A | X , A ) is instantiated with the structure estimation module and p ( Y | ˜
Revised: Speciﬁcally, p ( ˜ A | X , A ) is instantiated with the structure estimation module and p ( Y | ˜

Original: A , X , A ) is instantiated with the feature propagation module.
Revised: A , X , A ) is instantiated with the feature propagation module.

Original: In principle, ideal latent graphs should account for downstream tasks and maximize the potentials
Revised: In principle, ideal latent graphs should account for downstream tasks and maximize the potentials

Original: [Figure]
Revised: [Figure]

Original: of message passing for producing informative node representations
Revised: of message passing for producing informative node representations

Original: Thus, optimal latent graphs presumably come from the posterior p ( ˜ A | Y , X , A ) =
Revised: Thus, optimal latent graphs presumably come from the posterior p ( ˜ A | Y , X , A ) =

Original: p ( Y | X , A , ˜ A ) p
Revised: p ( Y | X , A , ˜ A ) p

Original: (˜ A | X , A )
Revised: (˜ A | X , A )

Original: R
Revised: R

Original: Y p
Revised: Y p

Original: ( Y | X , A , ˜ A )
Revised: ( Y | X , A , ˜ A )

Original: p (˜ A | X , A ) d Y which is given by
Revised: p (˜ A | X , A ) d Y which is given by

Original: Bayes theorem.
Revised: Bayes theorem.

Original: Unfortunately, such a posterior is unknown and intractable for the integration.
Revised: Unfortunately, such a posterior is unknown and intractable for the integration.

Original: A Variational Perspective.
Revised: A Variational Perspective.

Original: An intriguing conclusion stems from another view into the learning process: we can treat the structure estimation as a variational distribution q ( ˜ A | X , A ) and our learning objective in Section 3.4 can be viewed as the embodiment of a minimization problem over the predictive and variational distributions via
Revised: An intriguing conclusion stems from another view into the learning process: we can treat the structure estimation as a variational distribution q ( ˜ A | X , A ) and our learning objective in Section 3.4 can be viewed as the embodiment of a minimization problem over the predictive and variational distributions via

Original: [Equation]
Revised: [Equation]

Original: where D denotes the Kullback-Leibler divergence.
Revised: where D denotes the Kullback-Leibler divergence.

Original: Speciﬁcally, the predictive term is equivalent to minimizing the supervised loss (with Gumbel-Softmax as a surrogate for sampling-based estimates over q ( ˜ A | X , A ) ), and the KL regularization term is embodied with the edge-level MLE loss (Eqn. 10) (if we deﬁne the prior distribution p 

Revised: Speciﬁcally, the predictive term is equivalent to minimizing the supervised loss (with Gumbel-Softmax as a surrogate for sampling-based estimates over q ( ˜ A | X , A ) ), and the KL regularization term is embodied with the edge-level MLE loss (Eqn. 10) (if we deﬁne the prior distribution p

Original: ( ˜ A | X , A ) following Eqn. 11).
Revised: ( ˜ A | X , A ) following Eqn. 11).

Original: One may notice that Eqn. 13 is essentially the Evidence Lower Bound (ELBO) for the log-likelihood log p ( Y | X , A ) .
Revised: One may notice that Eqn. 13 is essentially the Evidence Lower Bound (ELBO) for the log-likelihood log p ( Y | X , A ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume q can exploit arbitrary distributions over ˜ A .
Revised: Assume q can exploit arbitrary distributions over ˜ A .

Original: When Eqn. 13 achieves the optimum, we have 1) D ( q ( ˜ A
Revised: When Eqn. 13 achieves the optimum, we have 1) D ( q ( ˜ A

Original: | X , A ) k p
Revised: | X , A ) k p

Original: ˜ A | Y , X , A ))
Revised: ˜ A | Y , X , A ))

Original: = 0 and 2)
Revised: = 0 and 2)

Original: log p
Revised: log p

Original: ( Y | X , A ) is maximized.
Revised: ( Y | X , A ) is maximized.

Original: The proposition indicates that our adopted learning objective intrinsically minimizes the divergence between latent graphs generated by the model and the samples from the posterior p ( ˜ A | Y , X , A ) that ideally helps to propagate useful adjacent information w.r.t.
Revised: The proposition indicates that our adopted learning objective intrinsically minimizes the divergence between latent graphs generated by the model and the samples from the posterior p ( ˜ A | Y , X , A ) that ideally helps to propagate useful adjacent information w.r.t.

Original: downstream tasks.
Revised: downstream tasks.

Original: Therefore, a well-trained network of N ODE F ORMER on labeled data could produce effective latent topology that contributes to boosting the downstream performance.
Revised: Therefore, a well-trained network of N ODE F ORMER on labeled data could produce effective latent topology that contributes to boosting the downstream performance.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes a scalable and efﬁcient graph Transformer (especially for node level) that can propagate layer-wise node signals between arbitrary pairs beyond input topology.
Revised: This paper proposes a scalable and efﬁcient graph Transformer (especially for node level) that can propagate layer-wise node signals between arbitrary pairs beyond input topology.

Original: The key module, a kernelized Gumbel-Softmax operator, enables us to learn layer-speciﬁc latent graphs with linear algorithmic complexity without compromising the precision.
Revised: The key module, a kernelized Gumbel-Softmax operator, enables us to learn layer-speciﬁc latent graphs with linear algorithmic complexity without compromising the precision.

Original: The results on diverse graph datasets and situations verify the effectiveness, scalability, and stability.
Revised: The results on diverse graph datasets and situations verify the effectiveness, scalability, and stability.

Original: We provide more discussions on the limitations and potential impacts in Appendix F.
Revised: We provide more discussions on the limitations and potential impacts in Appendix F.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Revised: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and

Original: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).
Revised: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).