Original: Towards Scalable (All-Pair) Message Passing for Node Classification beyond Explicit Topology
Revised: Towards Scalable (All-Pair) Message Passing for Node Classification beyond Explicit Topology

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Graph neural networks have been extensively studied for learning with interconnected data.
Revised: Graph neural networks have been extensively studied for learning with interconnected data.

Original: Despite this, recent evidence has revealed GNNs’ deficienciesrelated to over-squashing, heterophily, handling long-range dependencies, edgeincompleteness and particularly, the absence of graphs altogether.
Revised: Despite this, recent evidence has revealed GNNs’ deficienciesrelated to over-squashing, heterophily, handling long-range dependencies, edgeincompleteness and particularly, the absence of graphs altogether.

Original: While a plausiblesolution is to learn new topology for message passing, issues concerning quadraticcomplexity hinder simultaneous guarantees for scalability and precision in largenetworks.
Revised: While a plausiblesolution is to learn new topology for message passing, issues concerning quadraticcomplexity hinder simultaneous guarantees for scalability and precision in largenetworks.

Original: In this paper, we introduce a novel all-pair message passing scheme forefficiently propagating layer-wise signals between arbitrary nodes.
Revised: In this paper, we introduce a novel all-pair message passing scheme forefficiently propagating layer-wise signals between arbitrary nodes.

Original: Specifically,the efficient computation per layer is enabled by a kernerlized Gumbel-Softmaxoperator that reduces the algorithmic complexity to linearity w.r.t.
Revised: Specifically,the efficient computation per layer is enabled by a kernerlized Gumbel-Softmaxoperator that reduces the algorithmic complexity to linearity w.r.t.

Original: node numbersfor learning latent structures from large, potentially fully-connected graphs in adifferentiable manner.
Revised: node numbersfor learning latent structures from large, potentially fully-connected graphs in adifferentiable manner.

Original: We also provide accompanying theory as justification for ourdesign.
Revised: We also provide accompanying theory as justification for ourdesign.

Original: Extensive experiments demonstrate the promising efficacy of the methodin various tasks including node classification on different sizes of graphs (1K ∼ 1M)and graph-enhanced applications where input topology is missing.
Revised: Extensive experiments demonstrate the promising efficacy of the methodin various tasks including node classification on different sizes of graphs (1K ∼ 1M)and graph-enhanced applications where input topology is missing.

Original: The full versionof codes will be made public upon publication.
Revised: The full versionof codes will be made public upon publication.

Original:  Introduction
Revised:  Introduction

Original: Relational structure inter-connecting instance nodes as a graph is ubiquitous from social domains(e.g.
Revised: Relational structure inter-connecting instance nodes as a graph is ubiquitous from social domains(e.g.

Original: , citation networks) to natural science (protein-protein interaction), where graph neural networks(GNNs)
Revised: , citation networks) to natural science (protein-protein interaction), where graph neural networks(GNNs)

Original: [32, 19, 14, 36] have shown promising power for leveraging such data dependence asgeometric priors.
Revised: [32, 19, 14, 36] have shown promising power for leveraging such data dependence asgeometric priors.

Original: However, there arises increasing evidence challenging the core GNN hypothesisthat propagating information along observed graph structures will necessarily produce better nodelevel representations for prediction on each individual instance node.
Revised: However, there arises increasing evidence challenging the core GNN hypothesisthat propagating information along observed graph structures will necessarily produce better nodelevel representations for prediction on each individual instance node.

Original: Conflicts with this premiselead to commonly identified deficiencies with GNN message-passing rules w.r.t.
Revised: Conflicts with this premiselead to commonly identified deficiencies with GNN message-passing rules w.r.t.

Original: heterophily [45],over-squashing [2], long-range dependencies [8] and graph incompleteness [11], etc.
Revised: heterophily [45],over-squashing [2], long-range dependencies [8] and graph incompleteness [11], etc.

Original: Moreover, in graph-enhanced applications, e.g., text classification [42], vision navigation [12], physicssimulation [30], etc., graph structures are often unavailable though individual instances are stronglyinter-correlated.
Revised: Moreover, in graph-enhanced applications, e.g., text classification [42], vision navigation [12], physicssimulation [30], etc., graph structures are often unavailable though individual instances are stronglyinter-correlated.

Original: A common practice is to artificially construct a graph via some predefined rules(e.g.
Revised: A common practice is to artificially construct a graph via some predefined rules(e.g.

Original: , k -NN), which is agnostic to downstream tasks, presumably causing misspecification of GNNs’inductive bias on input geometry.
Revised: , k -NN), which is agnostic to downstream tasks, presumably causing misspecification of GNNs’inductive bias on input geometry.

Original: Natural solutions resort to organically combining learning optimal graph topology with messagepassing.
Revised: Natural solutions resort to organically combining learning optimal graph topology with messagepassing.

Original: However, one critical difficulty is the scalability issue with O ( N 2 ) (where N denotes#nodes) computational complexity, which is prohibitive for large networks (with 10 K ∼ 1 M nodes).
Revised: However, one critical difficulty is the scalability issue with O ( N 2 ) (where N denotes#nodes) computational complexity, which is prohibitive for large networks (with 10 K ∼ 1 M nodes).

Original: Some existing approaches harness neighbor sampling [43], anchor-based adjacency surrogates [4]and hashing schemes [40] to reduce the overhead; however, these strategies may sacrifice modelprecision and still struggle to handle graphs with million-level nodes.
Revised: Some existing approaches harness neighbor sampling [43], anchor-based adjacency surrogates [4]and hashing schemes [40] to reduce the overhead; however, these strategies may sacrifice modelprecision and still struggle to handle graphs with million-level nodes.

Original: Another obstacle lies in the
Revised: Another obstacle lies in the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: increased degrees of freedom due to at least an N × N all-pair similarity matrix, which may result inlarge combinatorial search space and vulnerability to over-fitting.
Revised: increased degrees of freedom due to at least an N × N all-pair similarity matrix, which may result inlarge combinatorial search space and vulnerability to over-fitting.

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we introduce a novel all-pair message passing scheme that can scale to large systemswithout compromising performance.
Revised: we introduce a novel all-pair message passing scheme that can scale to large systemswithout compromising performance.

Original: We develop a kernelized Gumbel-Softmax operator that seamlessly synthesizes random feature map [26] and approximated sampling strategy [16], for distillinglatent structures among all the instance nodes and yielding moderate gradients through differentiableoptimization.
Revised: We develop a kernelized Gumbel-Softmax operator that seamlessly synthesizes random feature map [26] and approximated sampling strategy [16], for distillinglatent structures among all the instance nodes and yielding moderate gradients through differentiableoptimization.

Original: Though such a combination of two operations involving randomness could potentiallyresult in mutual distortion, we theoretically prove that the new operator can still guarantee a wellposed approximation for concrete variables (discrete structures) with the error bounded by featuredimensions.
Revised: Though such a combination of two operations involving randomness could potentiallyresult in mutual distortion, we theoretically prove that the new operator can still guarantee a wellposed approximation for concrete variables (discrete structures) with the error bounded by featuredimensions.

Original: Furthermore, such a design can reduce the algorithmic complexity of learning newtopology per layer to O ( N ) by avoiding explicit computation for the cumbersome all-pair similarity.
Revised: Furthermore, such a design can reduce the algorithmic complexity of learning newtopology per layer to O ( N ) by avoiding explicit computation for the cumbersome all-pair similarity.

Original: The proposed module opens the door for a new class of GNNs capable of efficiently propagatingmessages between arbitrary node pairs in flexible layer-specific latent graphs.
Revised: The proposed module opens the door for a new class of GNNs capable of efficiently propagatingmessages between arbitrary node pairs in flexible layer-specific latent graphs.

Original: And to accommodateinput graphs (if any), we devise two simple techniques: a relational bias and a edge-level regularizationloss, as guidance for learning new structures.
Revised: And to accommodateinput graphs (if any), we devise two simple techniques: a relational bias and a edge-level regularizationloss, as guidance for learning new structures.

Original: We evaluate our approach on diverse node classificationtasks ranging from citation networks to images/texts.
Revised: We evaluate our approach on diverse node classificationtasks ranging from citation networks to images/texts.

Original: The results show its promising power fortackling heterophily, long-range dependencies, large-scale graphs, graph incompleteness and learningin the absence of input graphs.
Revised: The results show its promising power fortackling heterophily, long-range dependencies, large-scale graphs, graph incompleteness and learningin the absence of input graphs.

Original: The contributions of this paper are summarized as follows:
Revised: The contributions of this paper are summarized as follows:

Original: 1) We develop a kernelized Gumbel-Softmax operator which is proven to serve as a well-posed
Revised: 1) We develop a kernelized Gumbel-Softmax operator which is proven to serve as a well-posed

Original: approximation for concrete variables, particularly the discrete latent structure among data points.
Revised: approximation for concrete variables, particularly the discrete latent structure among data points.

Original: Thenew module can reduce the algorithmic complexity for learning new message-passing topology fromquadratic to linear w.r.t.
Revised: Thenew module can reduce the algorithmic complexity for learning new message-passing topology fromquadratic to linear w.r.t.

Original: instance node numbers (e
Revised: instance node numbers (e

Original: g., 1K ∼ 1M) of a dataset.2)
Revised: g., 1K ∼ 1M) of a dataset.2)

Original: We further propose a novel class of GNNs with layer-wise message passing as operated over latentgraphs potentially connecting all nodes.
Revised: We further propose a novel class of GNNs with layer-wise message passing as operated over latentgraphs potentially connecting all nodes.

Original: The latter are optimized with the network parameters in anend-to-end differentiable fashion through a new objective that essentially pursues sampling optimaltopology from a posterior conditioned on input features and downstream labels.3)
Revised: The latter are optimized with the network parameters in anend-to-end differentiable fashion through a new objective that essentially pursues sampling optimaltopology from a posterior conditioned on input features and downstream labels.3)

Original: We demonstrate the model’s efficacy by extensive experiments over 10 datasets where significantimprovement over strong GNN models and SOTA structure learning methods is shown.
Revised: We demonstrate the model’s efficacy by extensive experiments over 10 datasets where significantimprovement over strong GNN models and SOTA structure learning methods is shown.

Original: Besides, itsuccessfully scales to large graph datasets with up to 2M nodes where prior arts failed, and reducesthe time/space consumption of the competitors by up to 93.1%/80.6% on moderate sized datasets.
Revised: Besides, itsuccessfully scales to large graph datasets with up to 2M nodes where prior arts failed, and reducesthe time/space consumption of the competitors by up to 93.1%/80.6% on moderate sized datasets.

Original:  Related Works and Comparison
Revised:  Related Works and Comparison

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: Building expressive GNNs is a fundamental problem in learning overgraph data.
Revised: Building expressive GNNs is a fundamental problem in learning overgraph data.

Original: With Graph Attention Networks (GAT) [36] as an early attempt, there are many follow-upworks, e.g., [21, 39], considering weighting the edges in input graph for enhancing the expressiveness.
Revised: With Graph Attention Networks (GAT) [36] as an early attempt, there are many follow-upworks, e.g., [21, 39], considering weighting the edges in input graph for enhancing the expressiveness.

Original: Other studies, e.g., [27, 44] focus on sparsifying input structures to promote robust representations.
Revised: Other studies, e.g., [27, 44] focus on sparsifying input structures to promote robust representations.

Original: However, these works cannot learn new edges out of the scope of input geometry, which may limitthe model’s receptive fields within local neighbors and neglect global information.
Revised: However, these works cannot learn new edges out of the scope of input geometry, which may limitthe model’s receptive fields within local neighbors and neglect global information.

Original: Graph Structure Learning.
Revised: Graph Structure Learning.

Original: Going beyond observed topology, graph structure learning targetslearning a new graph for message passing among all the instances [47].
Revised: Going beyond observed topology, graph structure learning targetslearning a new graph for message passing among all the instances [47].

Original: One line of work is similarity- driven where the confidence of edges are reflected by some similarity functions between node pairs,e.g., Gaussian kernels [40], cosine similarity [4], attention networks [17], non-linear MLP [7] etc.
Revised: One line of work is similarity- driven where the confidence of edges are reflected by some similarity functions between node pairs,e.g., Gaussian kernels [40], cosine similarity [4], attention networks [17], non-linear MLP [7] etc.

Original: Another line of work optimizes the adjacency matrix.
Revised: Another line of work optimizes the adjacency matrix.

Original: Due to the increased optimization difficulties,some sophisticated training methods are introduced for learning, such as bi-level optimization [11],variational approaches [10, 34], Bayesian inference [43] and projected gradient descent [18].
Revised: Due to the increased optimization difficulties,some sophisticated training methods are introduced for learning, such as bi-level optimization [11],variational approaches [10, 34], Bayesian inference [43] and projected gradient descent [18].

Original: To pushfurther the limits of structure learning, this paper proposes a new model ALIGNS (for A LImber
Revised: To pushfurther the limits of structure learning, this paper proposes a new model ALIGNS (for A LImber

Original: Graph Network at Scale) whose merits are highlighted through a high-level comparison in Table 1.
Revised: Graph Network at Scale) whose merits are highlighted through a high-level comparison in Table 1.

Original: Node-Level v.s. Graph-Level Prediction.
Revised: Node-Level v.s. Graph-Level Prediction.

Original: We emphasize upfront that our focus is on node-levelprediction tasks involving a single large graph such that scalability is paramount, especially if we areto consider arbitrary relationships across all nodes (each node is an instance with label and one cantreat all the nodes non-i.i.d.
Revised: We emphasize upfront that our focus is on node-levelprediction tasks involving a single large graph such that scalability is paramount, especially if we areto consider arbitrary relationships across all nodes (each node is an instance with label and one cantreat all the nodes non-i.i.d.

Original: generated due to the inter-dependence) for structure-learning purposes.
Revised: generated due to the inter-dependence) for structure-learning purposes.

Original: Critically though, this scenario is quite distinct from graph-level classification tasks whereby eachi.i.d.
Revised: Critically though, this scenario is quite distinct from graph-level classification tasks whereby eachi.i.d.

Original: instance is itself a small graph and fully connecting nodes within each graph is computationallyinexpensive.
Revised: instance is itself a small graph and fully connecting nodes within each graph is computationallyinexpensive.

Original: While this latter scenario has been explored in the context of graph structure learning[38, 9], existing efforts do not scale to the large graphs endemic to node-level prediction.
Revised: While this latter scenario has been explored in the context of graph structure learning[38, 9], existing efforts do not scale to the large graphs endemic to node-level prediction.

Original:  ALIGNS: A Limber Graph Network at Scale
Revised:  ALIGNS: A Limber Graph Network at Scale

Original: Let G =
Revised: Let G =

Original: ( N , E ) denote a graph with N a node set ( |N| = N ) and E ⊆
Revised: ( N , E ) denote a graph with N a node set ( |N| = N ) and E ⊆

Original: N ×N an edge set ( |E| = E ).
Revised: N ×N an edge set ( |E| = E ).

Original: Each node u ∈ N is assigned with node features x u ∈ R D and a label y u .
Revised: Each node u ∈ N is assigned with node features x u ∈ R D and a label y u .

Original: We define an adjacencymatrix A
Revised: We define an adjacencymatrix A

Original: = { a uv } ∈ { 0 , 1 } N × N where a uv = 1 if edge ( u, v ) ∈ E and a uv = 0 otherwise.
Revised: = { a uv } ∈ { 0 , 1 } N × N where a uv = 1 if edge ( u, v ) ∈ E and a uv = 0 otherwise.

Original: Withoutloss of generality, E could be an empty set in case of no input structure.
Revised: Withoutloss of generality, E could be an empty set in case of no input structure.

Original: There are two commonsettings: transductive learning, where testing nodes are within the graph used for training, andinductive learning which handles new unseen nodes out of the training graph.
Revised: There are two commonsettings: transductive learning, where testing nodes are within the graph used for training, andinductive learning which handles new unseen nodes out of the training graph.

Original: The target is to learn afunction for node-level prediction, i.e., estimate labels for unlabeled or new nodes in the graph.
Revised: The target is to learn afunction for node-level prediction, i.e., estimate labels for unlabeled or new nodes in the graph.

Original: General Model and Key Challenges.
Revised: General Model and Key Challenges.

Original: We start with the observation that the input structures may notbe the ideal one for propagating signals among nodes and instead there exist certain latent structuresthat could facilitate learning better node representations.
Revised: We start with the observation that the input structures may notbe the ideal one for propagating signals among nodes and instead there exist certain latent structuresthat could facilitate learning better node representations.

Original: We thus consider the updating rule
Revised: We thus consider the updating rule

Original: [Equation]
Revised: [Equation]

Original: where Z ( l ) = { z ( l ) u } u ∈N and ˜ A ( l ) =
Revised: where Z ( l ) = { z ( l ) u } u ∈N and ˜ A ( l ) =

Original: { ˜ a ( l ) uv } u,v ∈N denotes the node representations and the estimatedlatent graph of the l -th layer, respectively, and g , h are both differentiable functions aiming at 1)structure estimation for a layer-specific latent graph
Revised: { ˜ a ( l ) uv } u,v ∈N denotes the node representations and the estimatedlatent graph of the l -th layer, respectively, and g , h are both differentiable functions aiming at 1)structure estimation for a layer-specific latent graph

Original: ˜ A ( l ) based on node representations and 2) featurepropagation for updating node representations, respectively.
Revised: ˜ A ( l ) based on node representations and 2) featurepropagation for updating node representations, respectively.

Original: The model defined by Eqn.
Revised: The model defined by Eqn.

Original: 1 potentiallyenables message passing between any node pair in each layer, which, however, poses two challenges :1) how to reduce the prohibitive algorithmic complexity for learning new graphs in each layer; 2)how to enable end-to-end differentiable optimization for learning discrete structures.
Revised: 1 potentiallyenables message passing between any node pair in each layer, which, however, poses two challenges :1) how to reduce the prohibitive algorithmic complexity for learning new graphs in each layer; 2)how to enable end-to-end differentiable optimization for learning discrete structures.

Original: Notice that thefirst technical challenge is particularly non-trivial in our case, i.e., node-level prediction tasks, sincethe latent graphs could potentially connect all the instance nodes (e.g., from thousands to millions,depending on dataset sizes), which is fairly hard to simultaneously guarantee precision and scalability.
Revised: Notice that thefirst technical challenge is particularly non-trivial in our case, i.e., node-level prediction tasks, sincethe latent graphs could potentially connect all the instance nodes (e.g., from thousands to millions,depending on dataset sizes), which is fairly hard to simultaneously guarantee precision and scalability.

Original: 3.1 Efficient Learning Discrete Structures
Revised: 3.1 Efficient Learning Discrete Structures

Original: We describe our new message-passing scheme with an efficient kernelized Gumbel-Softmax operatorto resolve the aforementioned challenges.
Revised: We describe our new message-passing scheme with an efficient kernelized Gumbel-Softmax operatorto resolve the aforementioned challenges.

Original: We assume z (0) u
Revised: We assume z (0) u

Original: = x u as the initial node representation.
Revised: = x u as the initial node representation.

Original: Kernelized Message Passing.
Revised: Kernelized Message Passing.

Original: We define a full-graph attentive network that estimates latent interactions among instance nodes and enables corresponding densely-connected message passing:
Revised: We define a full-graph attentive network that estimates latent interactions among instance nodes and enables corresponding densely-connected message passing:

Original: [Equation]
Revised: [Equation]

Original: where W ( l ) Q , W ( l ) K and W ( l ) V are learnable parameters in l -th layer.
Revised: where W ( l ) Q , W ( l ) K and W ( l ) V are learnable parameters in l -th layer.

Original: We omit non-linearity activation(after aggregation) for brevity.
Revised: We omit non-linearity activation(after aggregation) for brevity.

Original: Eqn. 2 requires prohibitive O ( N 2 ) complexity.
Revised: Eqn. 2 requires prohibitive O ( N 2 ) complexity.

Original: Also, given large N ,the normalization in the denominator would shrink attention weights to zero and lead to gradientvanishing.
Revised: Also, given large N ,the normalization in the denominator would shrink attention weights to zero and lead to gradientvanishing.

Original: We call this problem as over-normalizing .
Revised: We call this problem as over-normalizing .

Original: To accelerate the full-graph model, we observe that the dot-then-exponentiate operation in Eqn.
Revised: To accelerate the full-graph model, we observe that the dot-then-exponentiate operation in Eqn.

Original: 2 canbe converted into a pairwise similarity function:
Revised: 2 canbe converted into a pairwise similarity function:

Original: [Equation]
Revised: [Equation]

Original: where κ ( · , · ) : R d × R d → R is a positive-definite kernel measuring the pairwise similarity.
Revised: where κ ( · , · ) : R d × R d → R is a positive-definite kernel measuring the pairwise similarity.

Original: The kernelfunction can be further approximated by random features (RF) [26]which serves as an unbiasedestimation via κ ( a , b ) = ⟨ Φ( a ) , Φ( b ) ⟩ V ≈ ϕ ( a ) ⊤ ϕ ( b ) , where the first equation is by Mercer’stheorem with Φ : R d → V a basis function and V a high-dimensional vector space, and ϕ ( · ) : R d → 127 R m is a low-dimensional feature map with random transformation.
Revised: The kernelfunction can be further approximated by random features (RF) [26]which serves as an unbiasedestimation via κ ( a , b ) = ⟨ Φ( a ) , Φ( b ) ⟩ V ≈ ϕ ( a ) ⊤ ϕ ( b ) , where the first equation is by Mercer’stheorem with Φ : R d → V a basis function and V a high-dimensional vector space, and ϕ ( · ) : R d → 127 R m is a low-dimensional feature map with random transformation.

Original: There are many potential choicesfor ϕ , e.g., Positive Random Features (PRF) [6]
Revised: There are many potential choicesfor ϕ , e.g., Positive Random Features (PRF) [6]

Original: [Equation]
Revised: [Equation]

Original: where w k ∼ N (0 , I d ) is i.i.d.
Revised: where w k ∼ N (0 , I d ) is i.i.d.

Original: sampled random transformation.
Revised: sampled random transformation.

Original: The RF converts dot-thenexponentiate operation into inner-product in vector space, which enables us to re-write Eqn.
Revised: The RF converts dot-thenexponentiate operation into inner-product in vector space, which enables us to re-write Eqn.

Original: 3(assuming q u = W ( l ) Q z ( l ) u , k u = W ( l )
Revised: 3(assuming q u = W ( l ) Q z ( l ) u , k u = W ( l )

Original: K z
Revised: K z

Original: ( l ) u and v u = W ( l ) V z ( l ) u for simplicity):
Revised: ( l ) u and v u = W ( l ) V z ( l ) u for simplicity):

Original: [Equation]
Revised: [Equation]

Original: The key advantage of Eqn.
Revised: The key advantage of Eqn.

Original: 5 is that the two summations are shared by each u , so that one onlyneeds to compute them once and re-used for others.
Revised: 5 is that the two summations are shared by each u , so that one onlyneeds to compute them once and re-used for others.

Original: Such a property enables O ( N ) computationalcomplexity for full-graph message passing, which paves the way for learning graph structures amonglarge-scale instances.
Revised: Such a property enables O ( N ) computationalcomplexity for full-graph message passing, which paves the way for learning graph structures amonglarge-scale instances.

Original: Moreover, one can notice that Eqn.
Revised: Moreover, one can notice that Eqn.

Original: 5 avoids computing the N × N similaritymatrix, i.e., { ˜ a ( l ) uv } N × N , required by Eqn. 2, thus also reducing the learning difficulties.
Revised: 5 avoids computing the N × N similaritymatrix, i.e., { ˜ a ( l ) uv } N × N , required by Eqn. 2, thus also reducing the learning difficulties.

Original: Nevertheless, Eqn. 5 still suffers what we mentioned the over-normalizing issue.
Revised: Nevertheless, Eqn. 5 still suffers what we mentioned the over-normalizing issue.

Original: The crux is thatthe message passing is operated on a weighted fully-connected graph where, in fact, only partialedges are important.
Revised: The crux is thatthe message passing is operated on a weighted fully-connected graph where, in fact, only partialedges are important.

Original: Also, such a deterministic way of feature aggregation over all the instances mayincrease the risk for over-fitting, especially when N is large.
Revised: Also, such a deterministic way of feature aggregation over all the instances mayincrease the risk for over-fitting, especially when N is large.

Original: We next resolve the issues by distillinga sparse structure from the fully-connected graph.
Revised: We next resolve the issues by distillinga sparse structure from the fully-connected graph.

Original: Differentiable Stochastic Structure Learning.
Revised: Differentiable Stochastic Structure Learning.

Original: The difficultly lies in how to enable differentiableoptimization for discrete graph structures.
Revised: The difficultly lies in how to enable differentiableoptimization for discrete graph structures.

Original: The weight ˜ a ( l ) uv given by Eqn. 2 could be used to definea categorical distribution for generating latent edge e uv ∼ Cat ( π ( l ) u )
Revised: The weight ˜ a ( l ) uv given by Eqn. 2 could be used to definea categorical distribution for generating latent edge e uv ∼ Cat ( π ( l ) u )

Original: where π ( l ) u
Revised: where π ( l ) u

Original: = { π ( l ) uv } Nv =1and π ( l ) uv = p ( v | u ) =
Revised: = { π ( l ) uv } Nv =1and π ( l ) uv = p ( v | u ) =

Original: ˜ a ( l ) uv .
Revised: ˜ a ( l ) uv .

Original: Then in principle, we can sample over the categorical distributionmultiple times for each node to obtain its neighbors.
Revised: Then in principle, we can sample over the categorical distributionmultiple times for each node to obtain its neighbors.

Original: However, the sampling process would introducediscontinuity and hinders back-propagation.
Revised: However, the sampling process would introducediscontinuity and hinders back-propagation.

Original: Fortunately, we notice that the Eqn.
Revised: Fortunately, we notice that the Eqn.

Original: 3 can be modified toincorporate the reparametrization trick [16] to allow differentiable learning:
Revised: 3 can be modified toincorporate the reparametrization trick [16] to allow differentiable learning:

Original: [Equation]
Revised: [Equation]

Original: where g u is i.i.d.
Revised: where g u is i.i.d.

Original: sampled from Gumbel distribution and τ is a temperature coefficient.
Revised: sampled from Gumbel distribution and τ is a temperature coefficient.

Original: Eqn.
Revised: Eqn.

Original: 6 isa continuous relaxation of sampling over Cat ( π ( l ) u ) and τ controls the closeness to hard discretesamples [22].
Revised: 6 isa continuous relaxation of sampling over Cat ( π ( l ) u ) and τ controls the closeness to hard discretesamples [22].

Original: Following similar reasoning as Eqn. 3 and 5, we can yieldz ( l
Revised: Following similar reasoning as Eqn. 3 and 5, we can yieldz ( l

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: Eqn. 7 achieves message passing over a sampled latent graph (where we only sample once for eachnode) and still guarantees linear complexity as Eqn. 5.
Revised: Eqn. 7 achieves message passing over a sampled latent graph (where we only sample once for eachnode) and still guarantees linear complexity as Eqn. 5.

Original: In practice, we can sample K times (e.g.,
Revised: In practice, we can sample K times (e.g.,

Original: K = 5 ) for each node and take an average of the aggregated results (see Alg. 1).
Revised: K = 5 ) for each node and take an average of the aggregated results (see Alg. 1).

Original: 3.2 Well-posedness of the Kernelized Gumbel-Softmax Operator
Revised: 3.2 Well-posedness of the Kernelized Gumbel-Softmax Operator

Original: One reasonable concern for Eqn.
Revised: One reasonable concern for Eqn.

Original: 7 is whether the RF approximation for kernel functions maintains thewell-posedness of Gumbel approximation for the target discrete variables.
Revised: 7 is whether the RF approximation for kernel functions maintains thewell-posedness of Gumbel approximation for the target discrete variables.

Original: As a justification for thenew message-passing function, we next answer two theoretical questions: 1) How is the approximationcapability of RF for the original dot-then-exponentiate operation with Gumbel variables in Eqn. 6? 2)
Revised: As a justification for thenew message-passing function, we next answer two theoretical questions: 1) How is the approximationcapability of RF for the original dot-then-exponentiate operation with Gumbel variables in Eqn. 6? 2)

Original: Does Eqn.
Revised: Does Eqn.

Original: 7 still guarantee a continuous relaxation of the categorical distributions?
Revised: 7 still guarantee a continuous relaxation of the categorical distributions?

Original: We formulate theresults as follows and defer proofs to Appendix A.
Revised: We formulate theresults as follows and defer proofs to Appendix A.

Original: Theorem 1 (Approximation Error for Softmax-Kernel) .
Revised: Theorem 1 (Approximation Error for Softmax-Kernel) .

Original: Assume ∥
Revised: Assume ∥

Original: q u ∥ 2 and ∥ k v ∥ 2 are bounded by r ,then with probability at least 1 − ϵ , the gap ∆ = (cid:12)(cid:12) ϕ
Revised: q u ∥ 2 and ∥ k v ∥ 2 are bounded by r ,then with probability at least 1 − ϵ , the gap ∆ = (cid:12)(cid:12) ϕ

Original: ( q u / √ τ )
Revised: ( q u / √ τ )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ( k v / √ τ ) − κ ( q
Revised: ( k v / √ τ ) − κ ( q

Original: [Equation]
Revised: [Equation]

Original: / √ τ, k
Revised: / √ τ, k

Original: [Equation]
Revised: [Equation]

Original: / √ τ ) (cid:12)(cid:12) ) ,where ϕ is defined by Eqn. 4, will be bounded by O (cid:18)(cid:113) exp(6 r/τ ) mϵ (cid:19) .
Revised: / √ τ ) (cid:12)(cid:12) ) ,where ϕ is defined by Eqn. 4, will be bounded by O (cid:18)(cid:113) exp(6 r/τ ) mϵ (cid:19) .

Original: We can see that the error bound of RF for approximating original softmax-kernel function dependson both the dimension of feature map ϕ and temperature τ .
Revised: We can see that the error bound of RF for approximating original softmax-kernel function dependson both the dimension of feature map ϕ and temperature τ .

Original: Notably, the error bound is independentof node number N , which implies that the approximation ability is insensitive to dataset sizes.
Revised: Notably, the error bound is independentof node number N , which implies that the approximation ability is insensitive to dataset sizes.

Original: The second question is non-trivial since Eqn.
Revised: The second question is non-trivial since Eqn.

Original: 7 involves randomness of Gumbel variablesand random transformation in ϕ , which cannot be decoupled apart.
Revised: 7 involves randomness of Gumbel variablesand random transformation in ϕ , which cannot be decoupled apart.

Original: We define c
Revised: We define c

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: as the result from the kernelized Gumbel-Softmax and c u ={ c uv } Nv =1 denotes the sampled edge vector for node u .
Revised: as the result from the kernelized Gumbel-Softmax and c u ={ c uv } Nv =1 denotes the sampled edge vector for node u .

Original: We can arrive at the result as follows.
Revised: We can arrive at the result as follows.

Original: Theorem 2 (Property of Kernelized Gumbel-Softmax Random Variables) .
Revised: Theorem 2 (Property of Kernelized Gumbel-Softmax Random Variables) .

Original: Suppose m is sufficientlylarge, we have the convergence property for the kernelized Gumbel-Softmax operator
Revised: Suppose m is sufficientlylarge, we have the convergence property for the kernelized Gumbel-Softmax operator

Original: [Equation]
Revised: [Equation]

Original: The theorem shows that when i) the dimension of feature map is large enough and ii) the temperaturegoes to zero, the distribution from which latent structures are sampled would converge to the originalcategorical distribution.
Revised: The theorem shows that when i) the dimension of feature map is large enough and ii) the temperaturegoes to zero, the distribution from which latent structures are sampled would converge to the originalcategorical distribution.

Original: Remark.
Revised: Remark.

Original: The two theorems imply a trade-off between RF approximation and Gumbel-Softmaxapproximation w.r.t.
Revised: The two theorems imply a trade-off between RF approximation and Gumbel-Softmaxapproximation w.r.t.

Original: the choice of τ .
Revised: the choice of τ .

Original: A large τ would help to reduce the burden on kernel dimensionm , and namely, small τ would require a very large m to guarantee enough RF approximation precision.
Revised: A large τ would help to reduce the burden on kernel dimensionm , and namely, small τ would require a very large m to guarantee enough RF approximation precision.

Original: On the other hand, if τ is too large, the weight on each edge will converge to 1 N , i.e., the model nearlydegrades to mean pooling, while a small τ would endow the kernelized Gumbel-Softmax with betterapproximation to the categorical distribution.
Revised: On the other hand, if τ is too large, the weight on each edge will converge to 1 N , i.e., the model nearlydegrades to mean pooling, while a small τ would endow the kernelized Gumbel-Softmax with betterapproximation to the categorical distribution.

Original: Empirical studies are presented in Appendix D.2.
Revised: Empirical studies are presented in Appendix D.2.

Original: 3.3 Input Structures as Relational Bias
Revised: 3.3 Input Structures as Relational Bias

Original: Input: Node features X , observed adjacency A .
Revised: Input: Node features X , observed adjacency A .

Original: Z (0) ← X ; for l = 0 . . .
Revised: Z (0) ← X ; for l = 0 . . .

Original: , L − 1 do for k = 1 , 2 , . . .
Revised: , L − 1 do for k = 1 , 2 , . . .

Original: K do
Revised: K do

Original: [Equation]
Revised: [Equation]

Original: Eqn.
Revised: Eqn.

Original: 7 does not leverage any informationfrom observed geometry which, however,is often recognized important for modelingphysically-structured data [3].
Revised: 7 does not leverage any informationfrom observed geometry which, however,is often recognized important for modelingphysically-structured data [3].

Original: We therefore accommodate input topology (if any) asrelational bias via modifying the attentionweight as ˜ a ( l ) uv ← ˜ a ( l ) uv
Revised: We therefore accommodate input topology (if any) asrelational bias via modifying the attentionweight as ˜ a ( l ) uv ← ˜ a ( l ) uv

Original: + I
Revised: + I

Original: [ a uv = 1] σ ( b ( l ) ) ,where b ( l ) is a learnable scalar as relationalbias for any adjacent node pairs ( u, v ) andσ is a certain (bounded) activation functionlike sigmoid.
Revised: [ a uv = 1] σ ( b ( l ) ) ,where b ( l ) is a learnable scalar as relationalbias for any adjacent node pairs ( u, v ) andσ is a certain (bounded) activation functionlike sigmoid.

Original: The relational bias aims atassigning adjacent nodes in G with properweights, and the node representations couldbe accordingly updated by
Revised: The relational bias aims atassigning adjacent nodes in G with properweights, and the node representations couldbe accordingly updated by

Original: [Equation]
Revised: [Equation]

Original: We summarize the feed-forward computation of ALIGNS in Alg. 1.
Revised: We summarize the feed-forward computation of ALIGNS in Alg. 1.

Original: Eqn.
Revised: Eqn.

Original: 8 increases the algorithmiccomplexity for message passing to O ( N + E ) , albeit within the same order-of-magnitude as common
Revised: 8 increases the algorithmiccomplexity for message passing to O ( N + E ) , albeit within the same order-of-magnitude as common

Original: GNNs operating on input graphs.
Revised: GNNs operating on input graphs.

Original: Also, one can consider higher-order adjacency as relational bias forbetter expressiveness at some expense of efficiency, as similarly done by [1].
Revised: Also, one can consider higher-order adjacency as relational bias forbetter expressiveness at some expense of efficiency, as similarly done by [1].

Original: The missing results on Deezer is caused by out-of-memory (OOM).
Revised: The missing results on Deezer is caused by out-of-memory (OOM).

Original: 3.4 Learning Objective
Revised: 3.4 Learning Objective

Revised: [Figure]

Original: Given node features X = { x u } and training labels Y =
Revised: Given node features X = { x u } and training labels Y =

Original: { y u } , we are to maximize the datalog-likelihood which yields a supervised loss (with C classes)
Revised: { y u } , we are to maximize the datalog-likelihood which yields a supervised loss (with C classes)

Original: [Equation]
Revised: [Equation]

Original: where I [ · ] is an indicator function.
Revised: where I [ · ] is an indicator function.

Original: However, it may not suffice to generalize well.
Revised: However, it may not suffice to generalize well.

Original: The reason is thatthe graph topology learning increases the degrees of freedom and the number of training labels is notcomparable to that.
Revised: The reason is thatthe graph topology learning increases the degrees of freedom and the number of training labels is notcomparable to that.

Original: Therefore, we additionally introduce an edge-level regularization:
Revised: Therefore, we additionally introduce an edge-level regularization:

Original: [Equation]
Revised: [Equation]

Original: where d u denotes the in-degree of node u and π ( l ) uv is the predicted probability for edge ( u, v ) at l -thlayer.
Revised: where d u denotes the in-degree of node u and π ( l ) uv is the predicted probability for edge ( u, v ) at l -thlayer.

Original: Eqn.
Revised: Eqn.

Original: 10 is essentially a maximum likelihood estimation (MLE) for edges in E , whose datadistribution is defined as
Revised: 10 is essentially a maximum likelihood estimation (MLE) for edges in E , whose datadistribution is defined as

Original: [Equation]
Revised: [Equation]

Original: otherwise.
Revised: otherwise

Revised: [Equation]

Original: We next show how to efficiently obtain π ( l ) uv .
Revised: We next show how to efficiently obtain π ( l ) uv .

Original: Although the feed-forward ALIGNS computationdefined by Eqn.
Revised: Although the feed-forward ALIGNS computationdefined by Eqn.

Original: 7 does not explicitly produce the value for each π ( l ) uv , we can query their values by
Revised: 7 does not explicitly produce the value for each π ( l ) uv , we can query their values by

Original: [Equation]
Revised: [Equation]

Original: (12) where the summation term can be re-used from previous computation in Eqn.
Revised: (12) where the summation term can be re-used from once computation, as is done by Eqn.

Original:  Therefore, thecomputation for each π ( l ) uv requires O (1) complexity, yielding the total complexity controlled within O ( E ) (since we only need to query the observed edges).
Revised: 5 and Eqn. 7. Therefore, after once computation for the summation that requires O ( N ) , the computation for eachπ ( l ) uv requires O (1) complexity, yielding the total complexity controlled within O ( E ) (since we onlyneed to query the observed edges).

Original: The final objective can be the combinationof two: 
Revised: The final objective can be the combination of two: L =

Original: L= L s + λ L e , where λ controls how much emphasis is put on input topology.
Revised: L s + λ L e ,where λ controls how much emphasis is put on input topology.

Original:  Evaluation
Revised:  Evaluation

Original: We consider a diverse set of datasets for experiments and present detailed dataset information in
Revised: We consider a diverse set of datasets for experiments and present detailed dataset information in

Original: Appendix C. For implementation, we set σ as sigmoid function and τ as 0.25 for all datasets.
Revised: Appendix C. For implementation, we set σ as sigmoid function and τ as 0.25 for all datasets.

Original: Theoutput prediction layer is a one-layer MLP.
Revised: Theoutput prediction layer is a one-layer MLP.

Original: More implementation details are presented in Appendix B.
Revised: More implementation details are presented in Appendix B.

Original: All experiments are conducted on a NVIDIA V100 with 16 GB memory.
Revised: All experiments are conducted on a NVIDIA V100 with 16 GB memory.

Original: As baseline models, we basically consider GCN [19] and GAT [36].
Revised: As baseline models, we basically consider GCN [19] and GAT [36].

Original: Besides, we compare withsome advanced GNN models, including JKNet [41] and MixHop [1].
Revised: Besides, we compare withsome advanced GNN models, including JKNet [41] and MixHop [1].

Original: These GNN models all rely oninput graphs.
Revised: These GNN models all rely oninput graphs.

Original: We further consider DropEdge [27] and two SOTA graph structure learning methods,
Revised: We further consider DropEdge [27] and two SOTA graph structure learning methods,

Original: LDS-GNN [11] and IDGL [4] for comparison.
Revised: LDS-GNN [11] and IDGL [4] for comparison.

Original: More detailed information about these models arepresented in Appendix B.
Revised: More detailed information about these models arepresented in Appendix B.

Original: All the experiments are repeated five times with different initializations.
Revised: All the experiments are repeated five times with different initializations.

Original: 4.1 Experiments on Transductive Node Classification
Revised: 4.1 Experiments on Transductive Node Classification

Original: We study supervised node classification in transductive setting on several common graph datasets:
Revised: We study supervised node classification in transductive setting on several common graph datasets:

Original: Cora , Citeseer , Deezer and Actor .
Revised: Cora , Citeseer , Deezer and Actor .

Original: The first two have high homophily ratios and the last two
Revised: The first two have high homophily ratios and the last two

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: best mean accuracyacross four datasets and in particular, outperforms other models by a large margin on two heterophilicgraphs.
Revised: best mean accuracyacross four datasets and in particular, outperforms other models by a large margin on two heterophilicgraphs.

Original: The results indicate that ALIGNS can handle both homophilious and non-homophiliousgraphs.
Revised: The results indicate that ALIGNS can handle both homophilious and non-homophiliousgraphs.

Original: Compared with two structure learning models LDS and IDGL, ALIGNS yields significantlybetter performance, which demonstrates the superiority of ALIGNS.
Revised: Compared with two structure learning models LDS and IDGL, ALIGNS yields significantlybetter performance, which demonstrates the superiority of ALIGNS.

Original: Also, for deezer , LDS and
Revised: Also, for deezer , LDS and

Original: IDGL suffers from out-of-memory (OOM).
Revised: IDGL suffers from out-of-memory (OOM).

Original: In fact, the major difficulty for deezer is the largedimensions of input node features (nearly 30K), which causes OOM for IDGL even with the anchorapproximation.
Revised: In fact, the major difficulty for deezer is the largedimensions of input node features (nearly 30K), which causes OOM for IDGL even with the anchorapproximation.

Original: In contrast, ALIGNS manages to scale and produce desirable accuracy.
Revised: In contrast, ALIGNS manages to scale and produce desirable accuracy.

Original: 4.2 Experiments on Larger Graph Datasets
Revised: 4.2 Experiments on Larger Graph Datasets

Original: To further test the scalability of ALIGNS, we consider two large-sized networks, OGB-Proteins and
Revised: To further test the scalability of ALIGNS, we consider two large-sized networks, OGB-Proteins and

Original: Amazon2M , with over 0.1 million and 2 million of nodes, respectively.
Revised: Amazon2M , with over 0.1 million and 2 million of nodes, respectively.

Original: OGB-Proteins is a multi-taskdataset with 112 output dimensions, while Amazon2M is extracted from the Amazon Co-Purchasingnetwork that entails long-range dependence [13].
Revised: OGB-Proteins is a multi-taskdataset with 112 output dimensions, while Amazon2M is extracted from the Amazon Co-Purchasingnetwork that entails long-range dependence [13].

Original: For OGB-Proteins , we use the protocol of [15]and ROC-AUC for evaluation.
Revised: For OGB-Proteins , we use the protocol of [15]and ROC-AUC for evaluation.

Original: For Amazon2M , unlike [5] using random splits, we consider a morechallenging setting: we rank the nodes according to their degrees, and use the top 10% for trainingand the rest for testing.
Revised: For Amazon2M , unlike [5] using random splits, we consider a morechallenging setting: we rank the nodes according to their degrees, and use the top 10% for trainingand the rest for testing.

Original: This introduces distribution shifts and is more alike to real situations wherefrequent items are often used for training a model that needs to handle infrequent testing products.
Revised: This introduces distribution shifts and is more alike to real situations wherefrequent items are often used for training a model that needs to handle infrequent testing products.

Original: For validation set, we hold out 20% data from the training set.
Revised: For validation set, we hold out 20% data from the training set.

Original: In Table 2, we present the resultson OGB-Proteins where we only compare our model with MLP and GCN since other competitorswould suffer due to OOM.
Revised: In Table 2, we present the resultson OGB-Proteins where we only compare our model with MLP and GCN since other competitorswould suffer due to OOM.

Original: ALIGNS yields much better ROC-AUC than the baselines.
Revised: ALIGNS yields much better ROC-AUC than the baselines.

Original: Table 3 reportsthe results on Amazon2M .
Revised: Table 3 reportsthe results on Amazon2M .

Original: Due to the large dataset size that is even prohibitive for GCN, we adoptmini-batch partition for training in two different ways.
Revised: Due to the large dataset size that is even prohibitive for GCN, we adoptmini-batch partition for training in two different ways.

Original: 1) Batch : randomly partition nodes into minibatch sized 100000; 2) Cluster : use the METIS algorithm [5] for graph partition.
Revised: 1) Batch : randomly partition nodes into minibatch sized 100000; 2) Cluster : use the METIS algorithm [5] for graph partition.

Original: For ALIGNS, weonly consider structure learning among nodes in a mini-batch, which is helpful for saving space andcosts only 5GB GPU memory for training on Amazon2M .
Revised: For ALIGNS, weonly consider structure learning among nodes in a mini-batch, which is helpful for saving space andcosts only 5GB GPU memory for training on Amazon2M .

Original: Table 3 shows that ALIGNS outperformsbaselines by a large margin, which demonstrates its practical efficacy on large-scale datasets and alsothe capability for addressing long-range dependence with shallow layers (we set L = 3 ).
Revised: Table 3 shows that ALIGNS outperformsbaselines by a large margin, which demonstrates its practical efficacy on large-scale datasets and alsothe capability for addressing long-range dependence with shallow layers (we set L = 3 ).

Original: 4.3 Experiments on Inductive Node Classification
Revised: 4.3 Experiments on Inductive Node Classification

Original: We next study inductive learning, where the model needs to handle unseen nodes with new edgesduring inference, over two social networks Twitch-DE and Facebook-Baylor93 .
Revised: We next study inductive learning, where the model needs to handle unseen nodes with new edgesduring inference, over two social networks Twitch-DE and Facebook-Baylor93 .

Original: For each dataset,we use the clustering algorithm to split all the nodes into five subgraphs and adopt 3/1/1 subgraphsfor train/valid/test.
Revised: For each dataset,we use the clustering algorithm to split all the nodes into five subgraphs and adopt 3/1/1 subgraphsfor train/valid/test.

Original: Following [20] we use ROC-AUC and accuracy as evaluation metric for Twitchand Facebook , respectively.
Revised: Following [20] we use ROC-AUC and accuracy as evaluation metric for Twitchand Facebook , respectively.

Original: Fig.
Revised: Fig.

Original: 2 presents the results where we study models’ performance onoriginal and modified testing graphs, including: 1) randomly drop 90% and 97% edges; 2) randomlyadding 5% and 10% noised edges; 3) randomly connecting nodes with 1% and 5% probabilities; 4)
Revised: 2 presents the results where we study models’ performance onoriginal and modified testing graphs, including: 1) randomly drop 90% and 97% edges; 2) randomlyadding 5% and 10% noised edges; 3) randomly connecting nodes with 1% and 5% probabilities; 4)

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: constructing a k -NN graph with input node features ( k = 5 or 10 ).
Revised: constructing a k -NN graph with input node features ( k = 5 or 10 ).

Original: The figures show that ALIGNSoutperforms all the competitors in nearly all the cases.
Revised: The figures show that ALIGNSoutperforms all the competitors in nearly all the cases.

Original: Notably, ALIGNS yields much more stableperformance than other models w.r.t.
Revised: Notably, ALIGNS yields much more stableperformance than other models w.r.t.

Original: different testing structures.
Revised: different testing structures.

Original: In particular, GCN and GAT arequite vulnerable for perturbation on structures which essentially introduce distribution shift betweentraining and testing graphs.
Revised: In particular, GCN and GAT arequite vulnerable for perturbation on structures which essentially introduce distribution shift betweentraining and testing graphs.

Original: JKNet and MixHop could alleviate the sensitivity to a certain degree, yetstill suffer when replacing the input graph as randomly connected ones.
Revised: JKNet and MixHop could alleviate the sensitivity to a certain degree, yetstill suffer when replacing the input graph as randomly connected ones.

Original: With new structures learnedfrom data, ALIGNS can perform much more stable than standard GNNs relying on input graphs.
Revised: With new structures learnedfrom data, ALIGNS can perform much more stable than standard GNNs relying on input graphs.

Original: 4.4 Experiments on Graph-Enhanced Applications
Revised: 4.4 Experiments on Graph-Enhanced Applications

Original: We proceed to apply our model to semi-supervised image and text classification on
Revised: We proceed to apply our model to semi-supervised image and text classification on

Original: Mini-ImageNet and 20News-Groups datasets, without any given input graphs.
Revised: Mini-ImageNet and 20News-Groups datasets, without any given input graphs.

Original: The instancesof Mini-ImageNet [37] are 84×84 RGB images and we randomly choose 30 classes each of whichcontains 600 samples for experiments.
Revised: The instancesof Mini-ImageNet [37] are 84×84 RGB images and we randomly choose 30 classes each of whichcontains 600 samples for experiments.

Original: 20News-Groups [24] consists of nearly 10K texts whosefeatures are extracted by TF-IDF.
Revised: 20News-Groups [24] consists of nearly 10K texts whosefeatures are extracted by TF-IDF.

Original: More details for preprocessing are presented in Appendix C. Also,for each dataset, we randomly split instances into 50%/25%/25% for train/valid/test.
Revised: More details for preprocessing are presented in Appendix C. Also,for each dataset, we randomly split instances into 50%/25%/25% for train/valid/test.

Original: To enable message passing (especially for baseline models), we use k -NN (over input node features) for artificiallyconstructing a graph.
Revised: To enable message passing (especially for baseline models), we use k -NN (over input node features) for artificiallyconstructing a graph.

Original: Table 4 presents the comparison results under different k ’s. We can see that
Revised: Table 4 presents the comparison results under different k ’s. We can see that

Original: ALIGNS achieves the best performance in seven cases out of eight in total.
Revised: ALIGNS achieves the best performance in seven cases out of eight in total.

Original: The performance ofbaseline models varies significantly with different k values, and ALIGNS is much less sensitive.
Revised: The performance ofbaseline models varies significantly with different k values, and ALIGNS is much less sensitive.

Original: 4.5 Further Discussions
Revised: 4.5 Further Discussions

Original: Comparison of Time/Space Consumption.
Revised: Comparison of Time/Space Consumption.

Original: Fig.
Revised: Fig.

Original: 3 plots training/inference time and GPU memorycosts of ALIGNS and two SOTA structure learning models.
Revised: 3 plots training/inference time and GPU memorycosts of ALIGNS and two SOTA structure learning models.

Original: Compared with LDS, ALIGNS reducesthe training time, inference time, memory cost by up to 93.1%, 97.9%, 75.6%, respectively; comparedwith IDGL (using anchor-based approximation for acceleration), ALIGNS reduces the training time,inference time, memory cost by up to 61.8%, 80.8%, 80.6%, respectively.
Revised: Compared with LDS, ALIGNS reducesthe training time, inference time, memory cost by up to 93.1%, 97.9%, 75.6%, respectively; comparedwith IDGL (using anchor-based approximation for acceleration), ALIGNS reduces the training time,inference time, memory cost by up to 61.8%, 80.8%, 80.6%, respectively.

Original: Ablation on Edge Loss and Relational Bias.
Revised: Ablation on Edge Loss and Relational Bias.

Original: We study the effects of edge-level regularization andrelation bias as ablation study shown in Table 4.
Revised: We study the effects of edge-level regularization andrelation bias as ablation study shown in Table 4.

Original: More results are deferred to Appendix D.1 andthey consistently show that both components contribute to some positive effects and suggest that ouredge-level loss and relation bias can both help to leverage useful information from input graphs.
Revised: More results are deferred to Appendix D.1 andthey consistently show that both components contribute to some positive effects and suggest that ouredge-level loss and relation bias can both help to leverage useful information from input graphs.

Original: Ablation on Stochastic Components.
Revised: Ablation on Stochastic Components.

Original: Table 2 and 3 also include two variants of ALIGNS forablation study.
Revised: Table 2 and 3 also include two variants of ALIGNS forablation study.

Original: 1) ALIGNS-dt: replace Gumbel-Softmax by original Softmax (with temperature 1.0)for deterministic propagation; 2) ALIGNS-tp: use original Softmax with temperature set as 0.25 (thesame as ALIGNS).
Revised: 1) ALIGNS-dt: replace Gumbel-Softmax by original Softmax (with temperature 1.0)for deterministic propagation; 2) ALIGNS-tp: use original Softmax with temperature set as 0.25 (thesame as ALIGNS).

Original: There is more or less performance drop when removing the Gumbel components.
Revised: There is more or less performance drop when removing the Gumbel components.

Original: The possible cause stems from over-normalizing or over-fitting that are amplified in large datasets, aswe discussed in Section 3.1 and the kernelized Gumbel-Softmax operator shows its effectiveness.
Revised: The possible cause stems from over-normalizing or over-fitting that are amplified in large datasets, aswe discussed in Section 3.1 and the kernelized Gumbel-Softmax operator shows its effectiveness.

Original: Impact of Temperature and Feature Map Dimension.
Revised: Impact of Temperature and Feature Map Dimension.

Original: We study the effects of τ and m in
Revised: We study the effects of τ and m in

Original: Appendix D.2 and the variation trend accords with our theoretical analysis in Section 3.2.
Revised: Appendix D.2 and the variation trend accords with our theoretical analysis in Section 3.2.

Original: Visualization and Implications.
Revised: Visualization and Implications.

Original: We visualize the produced structures by ALIGNS on differentdatasets in Appendix D.3.
Revised: We visualize the produced structures by ALIGNS on differentdatasets in Appendix D.3.

Original: There are some common and special patterns, like dense connectionbetween inter-class nodes and sparse connection between inter-class nodes.
Revised: There are some common and special patterns, like dense connectionbetween inter-class nodes and sparse connection between inter-class nodes.

Original: This suggests that thelatent graphs can propagate useful information for better distinguishing nodes with distinct classes,which helps to interpret why ALIGNS can improves prediction.
Revised: This suggests that thelatent graphs can propagate useful information for better distinguishing nodes with distinct classes,which helps to interpret why ALIGNS can improves prediction.

Original: We also compare the learnedstructures with original graphs and the results turn out to be quite different across homophilous andheterophilic graph datasets.
Revised: We also compare the learnedstructures with original graphs and the results turn out to be quite different across homophilous andheterophilic graph datasets.

Original: Also we found ALIGNS tends to assign large edge weights for a smallportion of dominant nodes.
Revised: Also we found ALIGNS tends to assign large edge weights for a smallportion of dominant nodes.

Original: This can be an interesting discovery that implies that some nodes may beimportant anchors or serve as pivots for enhancing the whole connectivity of the dataset.
Revised: This can be an interesting discovery that implies that some nodes may beimportant anchors or serve as pivots for enhancing the whole connectivity of the dataset.

Original:  Why ALIGNS Improves Downstream Prediction?
Revised:  Why ALIGNS Improves Downstream Prediction?

Original: There remains a natural question concerning our learning process: how effective can the learned latenttopology be for downstream tasks?
Revised: There remains a natural question concerning our learning process: how effective can the learned latenttopology be for downstream tasks?

Original: We next dissect the rationale from a Bayesian perspective.
Revised: We next dissect the rationale from a Bayesian perspective.

Original: In fact,our model induces a predictive distribution p ( Y , ˜
Revised: In fact,our model induces a predictive distribution p ( Y , ˜

Original: A | X , A ) =
Revised: A | X , A ) =

Original: p ( ˜ A
Revised: p ( ˜ A

Original: | X , A ) p ( Y | ˜
Revised: | X , A ) p ( Y | ˜

Original: A , X , A ) where wecan treat the estimated graph ˜ A as a latent variable.
Revised: A , X , A ) where wecan treat the estimated graph ˜ A as a latent variable.

Original: Specifically, p ( ˜ A | X , A ) is instantiated with thestructure estimation module and p ( Y | ˜
Revised: Specifically, p ( ˜ A | X , A ) is instantiated with thestructure estimation module and p ( Y | ˜

Original: A , X , A ) is instantiated with the feature propagation module.
Revised: A , X , A ) is instantiated with the feature propagation module.

Original: In principle, ideal latent graphs should account for downstream tasks and maximize the potentialsof message passing for producing informative node representations.
Revised: In principle, ideal latent graphs should account for downstream tasks and maximize the potentialsof message passing for producing informative node representations.

Original: Thus, optimal latent graphspresumably come from the posterior p ( ˜ A | Y , X , A ) =
Revised: Thus, optimal latent graphspresumably come from the posterior p ( ˜ A | Y , X , A ) =

Original: p ( Y | X , A , ˜ A ) p
Revised: p ( Y | X , A , ˜ A ) p

Original: (˜ A | X , A ) (cid:82)
Revised: (˜ A | X , A ) (cid:82)

Original: Y p
Revised: Y p

Original: ( Y | X , A , ˜ A )
Revised: ( Y | X , A , ˜ A )

Original: p (˜ A | X , A ) d Y which is given by
Revised: p (˜ A | X , A ) d Y which is given by

Original: Bayes theorem.
Revised: Bayes theorem.

Original: Unfortunately, such a posterior is unknown and intractable for the integration.
Revised: Unfortunately, such a posterior is unknown and intractable for the integration.

Original: A Variational Perspective.
Revised: A Variational Perspective.

Original: An intriguing conclusion stems from another view into the learningprocess: we can treat the structure estimation as a variational distribution q ( ˜ A | X , A ) and ourlearning objective in Section 3.4 can be viewed as the embodiment of a minimization problem overthe predictive and variational distributions via
Revised: An intriguing conclusion stems from another view into the learningprocess: we can treat the structure estimation as a variational distribution q ( ˜ A | X , A ) and ourlearning objective in Section 3.4 can be viewed as the embodiment of a minimization problem overthe predictive and variational distributions via

Original: [Equation]
Revised: [Equation]

Original: where D denotes the Kullback-Leibler divergence.
Revised: where D denotes the Kullback-Leibler divergence.

Original: Specifically, the predictive term is equivalent tominimizing the supervised loss (with Gumbel-Softmax as a surrogate for sampling-based estimatesover q ( ˜ A | X , A ) ), and the KL regularization term is embodied with the edge-level MLE loss (Eqn.
Revised: Specifically, the predictive term is equivalent tominimizing the supervised loss (with Gumbel-Softmax as a surrogate for sampling-based estimatesover q ( ˜ A | X , A ) ), and the KL regularization term is embodied with the edge-level MLE loss (Eqn.

Original: 10)(if we define the prior distribution p 

Revised: 10)(if we define the prior distribution p

Original: ( ˜ A | X , A ) following Eqn. 11).
Revised: ( ˜ A | X , A ) following Eqn. 11).

Original: One may notice that Eqn. 

Revised: One may notice that Eqn.

Original: isessentially the Evidence Lower Bound (ELBO) for the log-likelihood log p ( Y | X , A ) .
Revised: isessentially the Evidence Lower Bound (ELBO) for the log-likelihood log p ( Y | X , A ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume q can exploit arbitrary distributions over ˜ A .
Revised: Assume q can exploit arbitrary distributions over ˜ A .

Original: When Eqn. 13 achieves theoptimum, we have 1) D ( q ( ˜ A | X , A )
Revised: When Eqn. 13 achieves theoptimum, we have 1) D ( q ( ˜ A | X , A )

Original: ∥ p ( ˜ A | Y , X , A ))
Revised: ∥ p ( ˜ A | Y , X , A ))

Original: = 0 and 2)
Revised: = 0 and 2)

Original: log p
Revised: log p

Original: ( Y | X , A ) is maximized.
Revised: ( Y | X , A ) is maximized.

Original: The proposition indicates that our adopted learning objective intrinsically minimizes the divergencebetween latent graphs generated by the model and the samples from the posterior p ( ˜ A | Y , X , A )that ideally helps to propagate useful adjacent information w.r.t.
Revised: The proposition indicates that our adopted learning objective intrinsically minimizes the divergencebetween latent graphs generated by the model and the samples from the posterior p ( ˜ A | Y , X , A )that ideally helps to propagate useful adjacent information w.r.t.

Original: downstream tasks.
Revised: downstream tasks.

Original: Therefore,a well-trained network of ALIGNS on labeled data could produce effective latent topology thatcontributes to boosting the downstream performance.
Revised: Therefore,a well-trained network of ALIGNS on labeled data could produce effective latent topology thatcontributes to boosting the downstream performance.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes a new scalable and efficient message-passing scheme that can propagate layerwise node signals between arbitrary pairs beyond explicit topology.
Revised: This paper proposes a new scalable and efficient message-passing scheme that can propagate layerwise node signals between arbitrary pairs beyond explicit topology.

Original: The key module, a kernelized
Revised: The key module, a kernelized

Original: Gumbel-Softmax operator, enables us to learn layer-specific latent topology with linear algorithmiccomplexity without compromising the precision in practice.
Revised: Gumbel-Softmax operator, enables us to learn layer-specific latent topology with linear algorithmiccomplexity without compromising the precision in practice.

Original: We also provide accompanying theory asjustification for the model design and interpretations that may help to understand its principles.
Revised: We also provide accompanying theory asjustification for the model design and interpretations that may help to understand its principles.

Original: Thepromising results on diverse datasets and situations verify the effectiveness, scalability, and stabilityfor graph-structured data with distinct properties and also the absence of input graphs.
Revised: Thepromising results on diverse datasets and situations verify the effectiveness, scalability, and stabilityfor graph-structured data with distinct properties and also the absence of input graphs.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A
Revised: [Yes] See Appendix A

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See Appendix C
Revised: [Yes] See Appendix C

Original: for dataset information.
Revised: for dataset information.

Original: See the supplementary for the codes.
Revised: See the supplementary for the codes.

Original: They will be released
Revised: They will be released

Original: public upon publication
Revised: public upon publication

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Appendix B
Revised: [Yes] See Appendix B

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See the experiment section
Revised: [Yes] See the experiment section

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B
Revised: [Yes] See Appendix B

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix C
Revised: [Yes] See Appendix C

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Proof for Technical ResultsA.1 Proof for Theorem 1
Revised: A Proof for Technical ResultsA.1 Proof for Theorem 1

Original: To prove our theorem, we first introduce the following lemma given by the Lemma 2 in [6].
Revised: To prove our theorem, we first introduce the following lemma given by the Lemma 2 in [6].

Original: Proposition 1.
Revised: Proposition 1.

Original: Denote a softmax kernel as SM ( x , y ) =
Revised: Denote a softmax kernel as SM ( x , y ) =

Original: exp( x ⊤ y ) .
Revised: exp( x ⊤ y ) .

Original: The Positive Random Features defined by Eqn. 4 for softmax-kernel estimation, i.e., (cid:99) SM m ( x , y )
Revised: The Positive Random Features defined by Eqn. 4 for softmax-kernel estimation, i.e., (cid:99) SM m ( x , y )

Original: = 1 m (cid:80) mi =1 [exp( w ⊤ i x −∥ x ∥ 2 2 ) exp( w ⊤ i y − ∥
Revised: = 1 m (cid:80) mi =1 [exp( w ⊤ i x −∥ x ∥ 2 2 ) exp( w ⊤ i y − ∥

Original: y ∥ 2 2 )] , has the mean and variance over w ∼ N (0 , I d ) as
Revised: y ∥ 2 2 )] , has the mean and variance over w ∼ N (0 , I d ) as

Original: [Equation]
Revised: [Equation]

Original: The lemma shows that the Positive Random Features can achieve unbiased approximation for thesoftmax kernel with a quantified variance.
Revised: The lemma shows that the Positive Random Features can achieve unbiased approximation for thesoftmax kernel with a quantified variance.

Original: Back to our main theorem, suppose the L2-norms of q u and k v are bounded by r , we can derive theprobability using the Chebyshev’s inequality:
Revised: Back to our main theorem, suppose the L2-norms of q u and k v are bounded by r , we can derive theprobability using the Chebyshev’s inequality:

Original: [Equation]
Revised: [Equation]

Original: where ∆ = (cid:12)(cid:12)(cid:12)(cid:100) SM m ( q u / √ τ, k v / √ τ ) − SM ( q u / √ τ, k v / √ τ ) (cid:12)(cid:12)(cid:12) denotes the deviation of the kernelapproximation.
Revised: where ∆ = (cid:12)(cid:12)(cid:12)(cid:100) SM m ( q u / √ τ, k v / √ τ ) − SM ( q u / √ τ, k v / √ τ ) (cid:12)(cid:12)(cid:12) denotes the deviation of the kernelapproximation.

Original: Using the result in Lemma 1, we can further obtain that the RHS of Eqn.
Revised: Using the result in Lemma 1, we can further obtain that the RHS of Eqn.

Original: 15 is nogreater than
Revised: 15 is nogreater than

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since ∥ q u + k v √ τ ∥ 2 ≤ 4 rτ and 2 q ⊤ u k v τ ≤ 2 rτ , we can achieve the stated result:P
Revised: Since ∥ q u + k v √ τ ∥ 2 ≤ 4 rτ and 2 q ⊤ u k v τ ≤ 2 rτ , we can achieve the stated result:

Original: [Equation]
Revised: [Equation]

Original: A.2 Proof for Theorem 2
Revised: A.2 Proof for Theorem 2

Original: Before entering the proof for the theorem, we first introduce two basic technical lemmas.
Revised: Before entering the proof for the theorem, we first introduce two basic technical lemmas.

Original: Whilesuch results are already mentioned in previous studies [16, 22], their proofs will be useful forthe subsequent reasoning.
Revised: Whilesuch results are already mentioned in previous studies [16, 22], their proofs will be useful forthe subsequent reasoning.

Original: Therefore, we restate the proofs as building blocks for the followingpresentation.
Revised: Therefore, we restate the proofs as building blocks for the followingpresentation.

Original: Proposition 2.
Revised: Proposition 2.

Original: Given real numbers x i , x j ∈ R and u i , u j i.i.d.
Revised: Given real numbers x i , x j ∈ R and u i , u j i.i.d.

Original: sampled from uniform distribution within (0 , 1) .
Revised: sampled from uniform distribution within (0 , 1) .

Original: With Gumbel perturbation defined as g ( u ) = − log( − log( u )) , we have the probability 
Revised: With Gumbel perturbation defined as g ( u ) = − log( − log( u )) , we have the probability P ( x

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Due to g ( u ) = − log( − log( u )) , the inequality of interests x i + g ( u i ) >
Revised: Due to g ( u ) = − log( − log( u )) , the inequality of interests x i + g ( u i ) >

Original: x j + g ( u j ) can berearranged as
Revised: x j + g ( u j ) can berearranged as

Original: [Equation]
Revised: [Equation]

Original: Since log( u j ) < 0 , Eqn. 18 can be written asu [Equation] xi i
Revised: Since log( u j ) < 0 , Eqn. 18 can be written as

Original: [Equation]
Revised: [Equation]

Original: As u i , u j are i.i.d.
Revised: As u i , u j are i.i.d.

Original: sampled from a uniform distribution, the probability when the above formula canbe calculated via:
Revised: sampled from a uniform distribution, the probability when the above formula canbe calculated via:

Original: [Equation]
Revised: [Equation]

Original: Thus, we conclude the proof with
Revised: Thus, we conclude the proof with

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.
Revised: Proposition 3.

Original: Let X ∼ Gumbel
Revised: Let X ∼ Gumbel

Original: ( α, τ ) (i.e. X k =
Revised: ( α, τ ) (i.e. X k =

Original: [Equation]
Revised: [Equation]

Revised: log α

Original: [Equation]
Revised: [Equation]

Original: log α k + g k ) /τ )
Revised: + g k ) /τ )

Original: (cid:80) ni
Revised: [Equation]

Original:  =1 exp(
Revised: ni =1 exp

Original: [Equation]
Revised: [Equation]

Original: logα
Revised: log

Original:  i + g i ) /τ ) ) with location parametersα ∈ (0 , ∞ ) n and temperature τ ∈ (0 , ∞ ) , then:
Revised: [Equation] i [Equation] g i ) /τ ) ) with location parametersα ∈ (0 , ∞ ) n and temperature τ ∈ (0 , ∞ ) , then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This result can be similarly proved as Lemma 2.
Revised: This result can be similarly proved as Lemma 2.

Original: The event of interests X k >
Revised: The event of interests X k >

Original: X
Revised: X

Original: i , ∀ i ̸ = k isequivalent to
Revised: i , ∀ i ̸ = k isequivalent to

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since all the above inequalities are independent given u k , we can rearrange the first inequality as
Revised: Since all the above inequalities are independent given u k , we can rearrange the first inequality as

Original: [Equation]
Revised: [Equation]

Original: Since u 1 ∼ U [0 , 1] , the probability for the first inequality in Eqn.
Revised: Since u 1 ∼ U [0 , 1] , the probability for the first inequality in Eqn.

Original: 22 being true would be u α 1 /α k k .
Revised: 22 being true would be u α 1 /α k k .

Original: Thus, the probability for Eqn.
Revised: Thus, the probability for Eqn.

Original: 22 being true can be calculated via
Revised: 22 being true can be calculated via

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we assume (cid:80) ni
Revised: For simplicity, we assume (cid:80) ni

Original: =

Revised: =

Original: α i = 1 .
Revised: α i = 1 .

Original: Then for any g k ∈ [0 , 1] , we obtain
Revised: Then for any g k ∈ [0 , 1] , we obtain

Original: [Equation]
Revised: [Equation]

Original: and arrive at the result for the first bullet point.
Revised: and arrive at the result for the first bullet point.

Original: For the second bullet point, when τ → 0 , we have
Revised: For the second bullet point, when τ → 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Such a fact indicates that the output of a Concrete distribution with τ → 0 will be a one-hot vector( X argmax i
Revised: Such a fact indicates that the output of a Concrete distribution with τ → 0 will be a one-hot vector( X argmax i

Original: α i
Revised: α i

Original: = 1 ).
Revised: = 1 ).

Original: This yields the conclusion that
Revised: This yields the conclusion that

Original: [Equation]
Revised: [Equation]

Original: Now we turn to the proof of our theorem.
Revised: Now we turn to the proof of our theorem.

Original: We are to prove that the kernelized form in Eqn.
Revised: We are to prove that the kernelized form in Eqn.

Original: 7 has thesame property as the original Gumbel-Softmax in the limit sense (when τ goes to zero).
Revised: 7 has thesame property as the original Gumbel-Softmax in the limit sense (when τ goes to zero).

Original: We recallthat we have defined q u = W ( l ) Q z ( l ) u , k u = W ( l ) K z ( l ) u and v u = W ( l ) V z ( l ) u for simplicity.
Revised: We recallthat we have defined q u = W ( l ) Q z ( l ) u , k u = W ( l ) K z ( l ) u and v u = W ( l ) V z ( l ) u for simplicity.

Original: First, by definition we have
Revised: First, by definition we have

Original: [Equation]
Revised: [Equation]

Original: The property holds that for ∀ w ̸ = v , we have lim τ → 0 ϕ
Revised: The property holds that for ∀ w ̸ = v , we have lim τ [Equation] 0 ϕ

Original: [Equation]
Revised: [Equation]

Original: τ ) e gwτ equals to ∞ or 0, i.e. theoutput of the kernelized Gumbel-Softmax is still a one-hot vector when τ → 0 .
Revised: τ ) e gwτ equals to ∞ or 0, i.e. theoutput of the kernelized Gumbel-Softmax is still a one-hot vector when τ → 0 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Here Y v is defined in the same way as c uv in Section 3.2.
Revised: Here Y v is defined in the same way as c uv in Section 3.2.

Original: We thus have P (lim τ → 0 Y v = 1) =
Revised: We thus have P (lim τ → 0 Y v = 1) =

Original: P ( Y v > Y v ′ , ∀ v ′ ̸
Revised: P ( Y v > Y v ′ , ∀ v ′ ̸

Original: = v ) .
Revised: = v ) .

Original: To compute P ( Y v > Y v ′ , ∀ v ′ ̸ = v ) , for simplicity, let us consider the probability P
Revised: To compute P ( Y v > Y v ′ , ∀ v ′ ̸ = v ) , for simplicity, let us consider the probability P

Original: ( Y v > Y v ′ )
Revised: ( Y v > Y v ′ )

Original: =
Revised: =

Original: P ( ϕ ( q u √ τ )
Revised: P ( ϕ ( q u √ τ )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ( k v √ τ ) e gvτ > ϕ ( q u √ τ )
Revised: ( k v √ τ ) e gvτ > ϕ ( q u √ τ )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ( k v ′ √ τ ) e gv ′ τ ) .
Revised: ( k v ′ √ τ ) e gv ′ τ ) .

Original: To keep notation clean, we define
Revised: To keep notation clean, we define

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then the above-mentioned probability can be rewritten as P (log β v + g v τ > log β v ′
Revised: Then the above-mentioned probability can be rewritten as P (log β v + g v τ > log β v ′

Original: + g v ′ τ ) , whereβ v and β v ′ are
Revised: + g v ′ τ ) , whereβ v and β v ′ are

Original: two i.i.d.
Revised: two i.i.d.

Original: random variables.
Revised: random variables.

Original: From Lemma 1, we have E ( β v ) =
Revised: From Lemma 1, we have E ( β v ) =

Original: exp( q ⊤ u k v /τ )
Revised: exp( q ⊤ u k v /τ )

Original: =
Revised: =

Original: α 1 τ v , E ( β v ′ )
Revised: α 1 τ v , E ( β v ′ )

Original: =
Revised: =

Original: exp( q ⊤ u k v ′ /τ )
Revised: exp( q ⊤ u k v ′ /τ )

Original: =
Revised: =

Original: α 1 τ v ′ , whereα v and α v ′ are two constant values.
Revised: α 1 τ v ′ , whereα v and α v ′ are two constant values.

Original: Then using Lemma 2, we have
Revised: Then using Lemma 2, we have

Original: [Equation]
Revised: [Equation]

Original: According to the Chebyshev’s inequality, we have P ( | β v − α 1 τ v | ≤ ϵ v ) ≥ 1 − σ 2 v ϵ 2 v .
Revised: According to the Chebyshev’s inequality, we have P ( | β v − α 1 τ v | ≤ ϵ v ) ≥ 1 − σ 2 v ϵ 2 v .

Original: Here σ 2 v =
Revised: Here σ 2 v =

Original: V w ( (cid:100) SM m ( q u √ τ , k j √ τ )) , which can given by Lemma 1.
Revised: V w ( (cid:100) SM m ( q u √ τ , k j √ τ )) , which can given by Lemma 1.

Original: Due to the convexity of logarithmic function, we have
Revised: Due to the convexity of logarithmic function, we have

Original: [Equation]
Revised: [Equation]

Original: (32) and subsequently,
Revised: (32) and subsequently,

Original: [Equation]
Revised: [Equation]

Original: Therefore we have P ( | log β v − 1 τ log α v | ≤
Revised: Therefore we have P ( | log β v − 1 τ log α v | ≤

Original: [Equation]
Revised: [Equation]

Original: [Equation] P ( |
Revised: ) ≥ P ( |

Original: [Equation]
Revised: [Equation]

Original:  1 τ v | ≤ ϵ v ) .
Revised: α 1 τ v | ≤ ϵ v ) .

Original: Based on this, we canderive the result:
Revised: Based on this, we canderive the result:

Original: [Equation]
Revised: [Equation]

Original: Since β v and β v ′ are two i.i.d.
Revised: Since β v and β v ′ are two i.i.d.

Original: random variables, we have
Revised: random variables, we have

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we denote ϵ =
Revised: For simplicity, we denote ϵ =

Original: [Equation]
Revised: [Equation]

Original: and P ϵ =
Revised: and P ϵ =

Original: (1 − σ
Revised: (1 − σ

Original: [Equation]
Revised: [Equation]

Original: (1 − σ 2 v ′ ϵ 2 v ′ ) .
Revised: 1 − σ 2 v ′ ϵ 2 v ′ ) .

Original: We therefore have
Revised: We therefore have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the triangular inequality, we can yield
Revised: Using the triangular inequality, we can yield

Original: [Equation]
Revised: [Equation]

Original: Combining Eqn. 36 and 37, we have
Revised: Combining Eqn. 36 and 37, we have

Original: [Equation]
Revised: [Equation]

Original: Let c = log β v − log β v ′ , so that E ( c ) = 1 τ (log α v − log α v ′ ) .
Revised: Let c = log β v − log β v ′ , so that E ( c ) = 1 τ (log α v − log α v ′ ) .

Original: From Eqn.
Revised: From Eqn.

Original: 38, we can obtain
Revised: 38, we can obtain

Original: P ( c ≥ E ( c ) − ϵ ) ≥ P ϵ .
Revised: P ( c ≥ E ( c ) − ϵ ) ≥ P ϵ .

Original: According to Lemma 2, the probability P ( E ( c ) − ϵ ≥ g v ′ − g v τ ) can be written as
Revised: According to Lemma 2, the probability P ( E ( c ) − ϵ ≥ g v ′ − g v τ ) can be written as

Original: [Equation]
Revised: [Equation]

Original: Since c, g v , g v ′ are generated independently, combining Eqn. 39 and 40, we can yield
Revised: Since c, g v , g v ′ are generated independently, combining Eqn. 39 and 40, we can yield

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, from Eq.
Revised: Similarly, from Eq.

Original: 38 we have P ( c ≤ E ( c ) + ϵ ) ≥ P ϵ and subsequently,
Revised: 38 we have P ( c ≤ E ( c ) + ϵ ) ≥ P ϵ and subsequently,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus we have P ( c ≤ g v ′ − g v τ ) ≥ P ϵ (1 − 1 1+ αv ′ αv
Revised: Thus we have P ( c ≤ g v ′ − g v τ ) ≥ P ϵ (1 − 1 1+ αv ′ αv

Original: e − τϵ ) and also
Revised: e − τϵ ) and also

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining Eqn.
Revised: Combining Eqn.

Original: 41 and 43, we conclude that
Revised: 41 and 43, we conclude that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Based on this we consider the limitation for two sides and thus obtain
Revised: Based on this we consider the limitation for two sides and thus obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then with similar reasoning as Lemma 3, we have
Revised: Then with similar reasoning as Lemma 3, we have

Original: [Equation]
Revised: [Equation]

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: where x = q u √ τ , y =
Revised: where x = q u √ τ , y =

Original: k v,v ′ √ τ .
Revised: k v,v ′ √ τ .

Original: Therefore, P ϵ is dependent of the precision ϵ v , ϵ v ′ , the random featuredimension m , and the temperature τ .
Revised: Therefore, P ϵ is dependent of the precision ϵ v , ϵ v ′ , the random featuredimension m , and the temperature τ .

Original: If m is sufficiently large, σ would converge to zero and P ϵ goesto 1.
Revised: If m is sufficiently large, σ would converge to zero and P ϵ goesto 1.

Original: In such a case, Eqn. 46 holds once τ tends to zero.
Revised: In such a case, Eqn. 46 holds once τ tends to zero.

Original: We thus conclude the proof.
Revised: We thus conclude the proof.

Original: A.3 Proof for Proposition 1
Revised: A.3 Proof for Proposition 1

Original: According to our definitions in Section 5, we have
Revised: According to our definitions in Section 5, we have

Original: [Equation]
Revised: [Equation]

Original: Since we assume q ϕ can exploit arbitrary distributions over ˜ A , we know that when the ELBO isoptimized to the optimum, D KL ( q ϕ (
Revised: Since we assume q ϕ can exploit arbitrary distributions over ˜ A , we know that when the ELBO isoptimized to the optimum, D KL ( q ϕ (

Original: ˜ A
Revised: ˜ A

Original: | X , A ) ∥ p ( ˜ A | Y , X , A ))
Revised: | X , A ) ∥ p ( ˜ A | Y , X , A ))

Original: = 0 holds.
Revised: = 0 holds.

Original: Otherwise, there existsϕ ∗ ̸ =
Revised: Otherwise, there existsϕ ∗ ̸ =

Original: ϕ such that ELBO ( θ, ϕ ∗ ) >
Revised: ϕ such that ELBO ( θ, ϕ ∗ ) >

Original: ELBO ( θ, ϕ ) .
Revised: ELBO ( θ, ϕ ) .

Original: Pushing further, when the optimum is achieved,log p θ ( Y | X , A ) would equal to ELBO and namely is maximized.
Revised: Pushing further, when the optimum is achieved,log p θ ( Y | X , A ) would equal to ELBO and namely is maximized.

Original: B Implementation Details
Revised: B Implementation Details

Original: We present implementation details in our experiments for reproducibility.
Revised: We present implementation details in our experiments for reproducibility.

Original: We implement our modelas well as the baselines with Python 3.8, Pytorch 1.8 and Pytorch Geometric 2.0.
Revised: We implement our modelas well as the baselines with Python 3.8, Pytorch 1.8 and Pytorch Geometric 2.0.

Original: For baseline models MLP, GCN, GAT, MixHop and JKnet, we use the implementation providedby [20] 2 .
Revised: For baseline models MLP, GCN, GAT, MixHop and JKnet, we use the implementation providedby [20] 2 .

Original: For DropEdge and two structure learning baseline models (LDS and IDGL), we also referto their implementation provided by the original papers [27, 11, 4].
Revised: For DropEdge and two structure learning baseline models (LDS and IDGL), we also referto their implementation provided by the original papers [27, 11, 4].

Original: Concretely, we use GCN as thebackbone for them.
Revised: Concretely, we use GCN as thebackbone for them.

Original: B.1 Details for Transductive Experiments
Revised: B.1 Details for Transductive Experiments

Original: Architectures.
Revised: Architectures.

Original: For experiments on the datasets Cora , Citeseer , Deezer and Actor , the baselinemodels (GCN, GAT, MixHop, JKNet) are implemented with the following settings:
Revised: For experiments on the datasets Cora , Citeseer , Deezer and Actor , the baselinemodels (GCN, GAT, MixHop, JKNet) are implemented with the following settings:

Original: https:// /CUAI Non-Homophily-
Revised: 2 https://github.com/CUAI/Non-Homophily-Benchmarks.

Original: • Two GNN layers with hidden size 32 by default (unless otherwise mentioned).
Revised: • Two GNN layers with hidden size 32 by default (unless otherwise mentioned).

Original: GAT uses 8 attentionheads followed by its original setting.•
Revised: GAT uses 8 attentionheads followed by its original setting.•

Original: The activation function is ReLU (except GAT using ELU).
Revised: The activation function is ReLU (except GAT using ELU).

Original: The architecture for our ALIGNS is:
Revised: The architecture for our ALIGNS is:

Original: • Two message-passing layers with hidden size 32, and we use 4 attention heads for each layer.• The activation function is ELU, and we specify σ as sigmoid function to bound the relational bias.For experiments on the two large
Revised: • Two message-passing layers with hidden size 32, and we use 4 attention heads for each layer.• The activation function is ELU, and we specify σ as sigmoid function to bound the relational bias.

Original: datasets (OGB-Proteins

Original:  and Amazon2M , we implement ALIGNS with 3 layers with hidden size 64 for better model capacity
Revised: For experiments on the two large datasets (OGB-Proteins and Amazon2M), we implement ALIGNSwith 3 layers with hidden size 64 for better model capacity.

Original: We consider the same layer number andhidden size for competitors MLP and GCN.
Revised: We consider the same layer number andhidden size for competitors MLP and GCN.

Original: Training Details.
Revised: Training Details.

Original: In each epoch, we feed the whole data into the model (except on Amazon2M forwhich we use mini-batch partition as we mentioned in Section 4.1), calculate the loss and conductgradient descent accordingly.
Revised: In each epoch, we feed the whole data into the model (except on Amazon2M forwhich we use mini-batch partition as we mentioned in Section 4.1), calculate the loss and conductgradient descent accordingly.

Original: Concretely, we use BCE-Loss for two-class classification and NLLLoss for multi-class classification, the Adam optimizer is used for gradient-based optimization.
Revised: Concretely, we use BCE-Loss for two-class classification and NLLLoss for multi-class classification, the Adam optimizer is used for gradient-based optimization.

Original: Thetraining procedure will repeat the above process until a given budget of 500 epochs.
Revised: Thetraining procedure will repeat the above process until a given budget of 500 epochs.

Original: Finally, wereport the test accuracy achieved by the epoch that gives the highest accuracy on validation dataset.
Revised: Finally, wereport the test accuracy achieved by the epoch that gives the highest accuracy on validation dataset.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: For each model, we use grid search on validation set for hyper-parametersetting.
Revised: For each model, we use grid search on validation set for hyper-parametersetting.

Original: The learning rate is searched within { 0 . 01 , 0 . 001 , 0 . 0001 , 0 . 00001 } , weight decay within{ 0 . 05 , 0 . 005 , 0 . 0005 , 0 . 00005 } , and dropout probability within { 0 . 0 , 0 .
Revised: The learning rate is searched within { 0 . 01 , 0 . 001 , 0 . 0001 , 0 . 00001 } , weight decay within{ 0 . 05 , 0 . 005 , 0 . 0005 , 0 . 00005 } , and dropout probability within { 0 . 0 , 0 .

Original: 5 } The hyperparameters forthe models are set as follows (we use the same notation as the original papers).
Revised: 5 } The hyperparameters forthe models are set as follows (we use the same notation as the original papers).

Original: • For GCN and GAT, the learning rate is 0.01, and weight decay is set to 0.05.
Revised: • For GCN and GAT, the learning rate is 0.01, and weight decay is set to 0.05.

Original: No dropout is used.• For MixHop, the hyperparameters are the same as above, except that we further use grid search forhidden channels within {8, 16, 32, 64, 128}.
Revised: No dropout is used. • For MixHop, the hyperparameters are the same as above, except that we further use grid search forhidden channels within {8, 16, 32, 64, 128}.

Original: We adopt 2 hops for all the four datasets.• For JKNet, GCN is used as the backbone.
Revised: We adopt 2 hops for all the four datasets.• For JKNet, GCN is used as the backbone.

Original: Learning rate is set to 0.01 for Deezer and 0.001 for allthe other three datasets.
Revised: Learning rate is set to 0.01 for Deezer and 0.001 for allthe other three datasets.

Original: We concatenate the features in the final stage for Deezer, while we usemax-pooling for the three other datasets.
Revised: We concatenate the features in the final stage for Deezer, while we usemax-pooling for the three other datasets.

Original: The hidden size is set as default, except for Deezer as 64.•
Revised: The hidden size is set as default, except for Deezer as 64.•

Original: For DropEdge, the hidden size is chosen from {32, 64, 96, 128, 160}, the learning rate is within{0.01, 0.001, 0.0001}, and the dropedge rate is chosen from {0.3, 0.4, 0.5}.•
Revised: For DropEdge, the hidden size is chosen from {32, 64, 96, 128, 160}, the learning rate is within{0.01, 0.001, 0.0001}, and the dropedge rate is chosen from {0.3, 0.4, 0.5}.•

Original: For LDS, the sampling time S = 16 , the patience window size ρ = 6 , the hidden size ∈ {8, 16,32, 64}, the inner learning rate γ ∈ {1e-4, 1e-3, 1e-2, 1e-1}, and the number of updates used tocompute the truncated hypergradient τ ∈ {5, 10, 15}.• For IDGL, we use its original version without anchor approximation on Cora , Citeseer andActor .
Revised: For LDS, the sampling time S = 16 , the patience window size ρ = 6 , the hidden size ∈ {8, 16,32, 64}, the inner learning rate γ ∈ {1e-4, 1e-3, 1e-2, 1e-1}, and the number of updates used tocompute the truncated hypergradient τ ∈ {5, 10, 15}.• For IDGL, we use its original version without anchor approximation on Cora , Citeseer andActor .

Original: For Deezer , even using anchor approximation, it would also suffer from out-of-memory.
Revised: For Deezer , even using anchor approximation, it would also suffer from out-of-memory.

Original: Besides, we set: ϵ = 0 .
Revised: Besides, we set: ϵ = 0 .

Original: 01 , hidden size ∈ {16, 64, 96, 128}, λ ∈ {0.5, 0.6, 0.7, 0.8}, η ∈ {0, 0.1,0.2}, α ∈ {0, 0.1, 0.2}, β ∈ {0, 0.1}, γ ∈ {0.1, 0.2}, m ∈ {6, 9, 12}.•
Revised: 01 , hidden size ∈ {16, 64, 96, 128}, λ ∈ {0.5, 0.6, 0.7, 0.8}, η ∈ {0, 0.1,0.2}, α ∈ {0, 0.1, 0.2}, β ∈ {0, 0.1}, γ ∈ {0.1, 0.2}, m ∈ {6, 9, 12}.•

Original: For our ALIGNS model, on Cora and Citeseer , the learning rate is 0.001 and weight decay is0.005.
Revised: For our ALIGNS model, on Cora and Citeseer , the learning rate is 0.001 and weight decay is0.005.

Original: The learning rate is 1e-5 for Deezer and 1e-4 for Actor , and the weight decay are set as0.05.
Revised: The learning rate is 1e-5 for Deezer and 1e-4 for Actor , and the weight decay are set as0.05.

Original: For OGB-Proteins and Amazon2M , the
Revised: For OGB-Proteins and Amazon2M , the

Original: learning rate is 0.01 and no weight decaying is used.
Revised: learning rate is 0.01 and no weight decaying is used.

Original: No dropout is used for all datasets.
Revised: No dropout is used for all datasets.

Original: The temperature τ is set to 0.25.
Revised: The temperature τ is set to 0.25.

Original: The weight λ is set to 1 for
Revised: The weight λ is set to 1 for

Original: Cora , Citeseer , 0.01 for Actor , Deezer , Amazon2m and 0.1 for OGB-Proteins .
Revised: Cora , Citeseer , 0.01 for Actor , Deezer , Amazon2m and 0.1 for OGB-Proteins .

Original: The samplingnumber K is set as 5 for two large datasets and 10 for others.
Revised: The samplingnumber K is set as 5 for two large datasets and 10 for others.

Original: The dimension of feature map m is30.
Revised: The dimension of feature map m is30.

Original: B.2 Details for Inductive Experiments
Revised: B.2 Details for Inductive Experiments

Original: Architectures.
Revised: Architectures.

Original: For experiments on inductive settings, baseline models are implemented in thesame manner as the transductive setting.
Revised: For experiments on inductive settings, baseline models are implemented in thesame manner as the transductive setting.

Original: The architecture for our ALIGNS is also the same as thetransductive setting.
Revised: The architecture for our ALIGNS is also the same as thetransductive setting.

Original: Training Details.
Revised: Training Details.

Original: The validation and testing graphs are unseen during training in the inductive setting,which means that the train/valid/test adjacent matrix are disjoint to each other.
Revised: The validation and testing graphs are unseen during training in the inductive setting,which means that the train/valid/test adjacent matrix are disjoint to each other.

Original: Concretely, in eachepoch, we feed the training data matrix into the model, conduct gradient descent with BCE-Loss and
Revised: Concretely, in eachepoch, we feed the training data matrix into the model, conduct gradient descent with BCE-Loss and

Original: Adam optimizer, and then statistic accuracy on the validation and test graphs.
Revised: Adam optimizer, and then statistic accuracy on the validation and test graphs.

Original: Typically, we modifythe valid/test graph structures (drop edge, add noise, random initialization and k -NN) to evaluateour model’s robustness against distribution shift in this setting.
Revised: Typically, we modifythe valid/test graph structures (drop edge, add noise, random initialization and k -NN) to evaluateour model’s robustness against distribution shift in this setting.

Original: We finally report the test accuracyachieved by the epoch that gives the highest accuracy on the validation graph.
Revised: We finally report the test accuracyachieved by the epoch that gives the highest accuracy on the validation graph.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: We also use validation set for hyper-parameter tuning.
Revised: We also use validation set for hyper-parameter tuning.

Original: The optimal hyperparameters vary with different models and datasets in the inductive setting.
Revised: The optimal hyperparameters vary with different models and datasets in the inductive setting.

Original: We list them in detail asfollows:
Revised: We list them in detail asfollows:

Original: • For MLP, the learning rate is 1e-3 on FB100 and 1e-5 on Twitch .
Revised: • For MLP, the learning rate is 1e-3 on FB100 and 1e-5 on Twitch .

Original: Two datasets share the sameweight decay of 0.1 and the same dropout of 0.5.• For GCN, the learning rate is 1e-3 on FB100 and 1e-4 on Twitch .
Revised: Two datasets share the sameweight decay of 0.1 and the same dropout of 0.5.• For GCN, the learning rate is 1e-3 on FB100 and 1e-4 on Twitch .

Original: Two datasets share the sameweight decay of 0.1 and the same dropout of 0.5.•
Revised: Two datasets share the sameweight decay of 0.1 and the same dropout of 0.5.•

Original: For GAT, the learning rate is 1e-4 on FB100 and 1e-4 on Twitch .
Revised: For GAT, the learning rate is 1e-4 on FB100 and 1e-4 on Twitch .

Original: Two datasets share the same
Revised: Two datasets share the same

Original: weight decay of 0.1.
Revised: weight decay of 0.1.

Original: We apply a dropout of 0.5 on Twitch , and remove it on FB100 .
Revised: We apply a dropout of 0.5 on Twitch , and remove it on FB100 .

Original: • For MixHop, the learning rate is 1e-4 on FB100 and 1e-5 on Twitch .
Revised: • For MixHop, the learning rate is 1e-4 on FB100 and 1e-5 on Twitch .

Original: Two datasets share the samedropout of 0.5.
Revised: Two datasets share the samedropout of 0.5.

Original: The weight decay on FB100 is 0.05 , other than 0.005 on Twitch .
Revised: The weight decay on FB100 is 0.05 , other than 0.005 on Twitch .

Original: We adopt 2 hopsfor both datasets.•
Revised: We adopt 2 hopsfor both datasets.•

Original: For JKNet, we use GAT as our backbone and use max-pooling in the final stage.
Revised: For JKNet, we use GAT as our backbone and use max-pooling in the final stage.

Original: The learning rateis 1e-3 on FB100 and 1e-4 on Twitch .
Revised: The learning rateis 1e-3 on FB100 and 1e-4 on Twitch .

Original: The weight decay on FB100 is 5e-3, and 5e-2 on Twitch .We apply a dropout of 0.4 on FB100 , and turn it off on Twitch .
Revised: The weight decay on FB100 is 5e-3, and 5e-2 on Twitch .We apply a dropout of 0.4 on FB100 , and turn it off on Twitch .•

Original: • For our ALIGNS model, the learning rate is 1e-5 on Twitch and 1e-4 on FB100 .
Revised: For our ALIGNS model, the learning rate is 1e-5 on Twitch and 1e-4 on FB100 .

Original: Two datasetsshare the same weight decay of 0.1.
Revised: Two datasetsshare the same weight decay of 0.1.

Original: Since FB100 and Twitch are non-homophilous datasets, weabandon edge-level regularization in this setting.
Revised: Since FB100 and Twitch are non-homophilous datasets, weabandon edge-level regularization in this setting.

Original: We apply dropout of 0.5 on FB100 , and turn it offon Twitch .
Revised: We apply dropout of 0.5 on FB100 , and turn it offon Twitch .

Original: The temperature τ is set to 0.25 on
Revised: The temperature τ is set to 0.25 on

Original: both datasets.
Revised: both datasets.

Original: The dimension of feature map m is60 on Twitch , and 80 on FB100 .
Revised: The dimension of feature map m is60 on Twitch , and 80 on FB100 .

Original: In both datasets, the number of samples K is set to 5.
Revised: In both datasets, the number of samples K is set to 5.

Original: B.3 Details for Graph-Enhanced Experiments
Revised: B.3 Details for Graph-Enhanced Experiments

Original: Architectures.
Revised: Architectures.

Original: The architectures of baselines (GCN, GAT, LDS and IDGL) and ALIGNS model arethe same as the transductive setting, except that we use grid search to adaptively tune the hidden size.
Revised: The architectures of baselines (GCN, GAT, LDS and IDGL) and ALIGNS model arethe same as the transductive setting, except that we use grid search to adaptively tune the hidden size.

Original: Besides, we also adopt BatchNorm for baseline models.
Revised: Besides, we also adopt BatchNorm for baseline models.

Original: Training Details.
Revised: Training Details.

Original: The input data have no graph structures in this setting.
Revised: The input data have no graph structures in this setting.

Original: As mentioned in Section 4.4,we use k -NN for artificially constructing a graph to enable message passing.
Revised: As mentioned in Section 4.4,we use k -NN for artificially constructing a graph to enable message passing.

Original: The training procedureis the same as the transductive setting.
Revised: The training procedureis the same as the transductive setting.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The hyperparameters for the models are listed as follows.
Revised: The hyperparameters for the models are listed as follows.

Original: • For GCN, the learning rate is 0.01 and the weight decay is 5e-4 on both datasets.
Revised: • For GCN, the learning rate is 0.01 and the weight decay is 5e-4 on both datasets.

Original: The size of hidden
Revised: The size of hidden

Original: channel is set to 64.
Revised: channel is set to 64.

Original: Dropout is not used during training.
Revised: Dropout is not used during training.

Original: • For GAT, the learning rate is 0.001 and the weight decay is 5e-3 on both datasets.
Revised: • For GAT, the learning rate is 0.001 and the weight decay is 5e-3 on both datasets.

Original: The size ofhidden channel is 32 on Mini-ImageNet and 64 on 20News .
Revised: The size ofhidden channel is 32 on Mini-ImageNet and 64 on 20News .

Original: No dropout is used during training.
Revised: No dropout is used during training.

Original: The number of attention heads is 8.
Revised: The number of attention heads is 8.

Original: • For LDS, its hyperparameters are determined by grid search in the same manner as in the transductive setting. • For IDGL, we use its anchor-based version that can scale to these two datasets.
Revised: • For LDS, its hyperparameters are determined by grid search in the same manner as in the transductive setting.• For IDGL, we use its anchor-based version that can scale to these two datasets.

Original: Besides, on 20News ,we set: hidden size=64, learning rate=0.01 λ = 0 .
Revised: Besides, on 20News ,we set: hidden size=64, learning rate=0.01 λ = 0 .

Original: 7 , η = 0 .
Revised: 7 , η = 0 .

Original: 1 , α = 0 .
Revised: 1 , α = 0 .

Original: 1 , β = 0 , γ = 0 .
Revised: 1 , β = 0 , γ = 0 .

Original: 1 , ϵ = 0 .
Revised: 1 , ϵ = 0 .

Original: 01 ,m = 12 .
Revised: 01 ,m = 12 .

Original: On Mini-ImageNet , we set: hidden size=96, learning rate=0.01 λ = 0 .
Revised: On Mini-ImageNet , we set: hidden size=96, learning rate=0.01 λ = 0 .

Original: 8 , η = 0 .
Revised: 8 , η = 0 .

Original: 2 ,α = 0 , β = 0 , γ = 0 .
Revised: 2 ,α = 0 , β = 0 , γ = 0 .

Original: 1 , ϵ = 0 .
Revised: 1 , ϵ = 0 .

Original: 01 , m = 12 .
Revised: 01 , m = 12 .

Original: • For our ALIGNS model, the size of hidden channels is 64 on 20News , and 128 on Mini-ImageNet.
Revised: • For our ALIGNS model, the size of hidden channels is 64 on 20News , and 128 on Mini-ImageNet

Original: The number of attention heads is set to 6 on Mini-ImageNet and 4 on 20News .
Revised: The number of attention heads is set to 6 on Mini-ImageNet and 4 on 20News .

Original: For both datasets, the learning rate is 
Revised: For both datasets,the learning rate is 0.001, and the weight decay is 0.005.

Original: and the weight

Original: decay

Original: is 0.

Original: The weight λ is set to 1.
Revised: The weight λ is set to 1.

Original: The temperature τ is set to 0.25 for both datasets.
Revised: The temperatureτ is set to 0.25 for both datasets.

Original: The number of samples K is  and the dimension of feature
Revised: The number of samples K is 10, and the dimension of featuremap m is 30.

Original: map

Original: m

Original: is

Original: No dropout is used during training
Revised: No dropout is used during training.

Original: C Dataset Information
Revised: C Dataset Information

Original: We present detailed information for our used datasets concerning the data collection, preprocessingand statistic information.
Revised: We present detailed information for our used datasets concerning the data collection, preprocessingand statistic information.

Original: Table 5 provides an overview of the datasets we used in the experiment.
Revised: Table 5 provides an overview of the datasets we used in the experiment.

Original: [Table]
Revised: [Table]

Original: C.1 Dataset Information
Revised: C.1 Dataset Information

Original: Transductive learning datasets.
Revised: Transductive learning datasets.

Original: For experiments on transductive node classification, we evaluateour model on two homophilous datasets Cora and Citeseer [33], and other two non-homophilousdatasets Actor [25] and Deezer [29].
Revised: For experiments on transductive node classification, we evaluateour model on two homophilous datasets Cora and Citeseer [33], and other two non-homophilousdatasets Actor [25] and Deezer [29].

Original: The first two are citation network datasets that contain sparsebag-of-words feature vectors for each document and a list of citation links between documents.
Revised: The first two are citation network datasets that contain sparsebag-of-words feature vectors for each document and a list of citation links between documents.

Original: The citation links are treated as (undirected) edges and each document has a class label.
Revised: The citation links are treated as (undirected) edges and each document has a class label.

Original: Deezeris a social network of users on Deezer from European countries, where edges represent mutualfollower relationships.
Revised: Deezeris a social network of users on Deezer from European countries, where edges represent mutualfollower relationships.

Original: The node features are based on artists liked by each user and nodes are labeledwith reported gender.
Revised: The node features are based on artists liked by each user and nodes are labeledwith reported gender.

Original: Actor is a graph representing actor co-occurrence in Wikipedia pages.
Revised: Actor is a graph representing actor co-occurrence in Wikipedia pages.

Original: Eachnode corresponds to an actor, and the edge between two nodes denotes co-occurrence on the same
Revised: Eachnode corresponds to an actor, and the edge between two nodes denotes co-occurrence on the same

Original: Wikipedia page.
Revised: Wikipedia page.

Original: Node features correspond to some keywords in the Wikipedia pages and the nodesare classified into five categories w.r.t.
Revised: Node features correspond to some keywords in the Wikipedia pages and the nodesare classified into five categories w.r.t.

Original: words of actor’s Wikipedia.
Revised: words of actor’s Wikipedia.

Original: These four datasets are relativelysmall with thousands of instances and edges ( Deezer is the largest one with nearly 20K nodes).
Revised: These four datasets are relativelysmall with thousands of instances and edges ( Deezer is the largest one with nearly 20K nodes).

Original: To evaluate ALIGNS’s scalability, we further consider two large datasets: OGB-Proteins [15] and
Revised: To evaluate ALIGNS’s scalability, we further consider two large datasets: OGB-Proteins [15] and

Original: Amazon2M [5].
Revised: Amazon2M [5].

Original: These two datasets have million-level nodes and edges and require the model forscalable training.
Revised: These two datasets have million-level nodes and edges and require the model forscalable training.

Original: The OGB-Proteins dataset is an undirected, and typed (according to species)graph in which nodes represent proteins and edges indicate different types of biologically meaningfulassociations between proteins.
Revised: The OGB-Proteins dataset is an undirected, and typed (according to species)graph in which nodes represent proteins and edges indicate different types of biologically meaningfulassociations between proteins.

Original: All edges come with 8-dimensional features, where each dimensionrepresents the approximate confidence of a single association type and takes on values between 0and 1.
Revised: All edges come with 8-dimensional features, where each dimensionrepresents the approximate confidence of a single association type and takes on values between 0and 1.

Original: The proteins come from 8 species and our task is to predict the presence of 112 proteinfunctions in a multi-label binary classification setup respectively.
Revised: The proteins come from 8 species and our task is to predict the presence of 112 proteinfunctions in a multi-label binary classification setup respectively.

Original: Amazon2M is extracted from
Revised: Amazon2M is extracted from

Original: Amazon Co-Purchasing network [23], where each node represents a product, and the graph linkrepresents whether two products are purchased together, the node features are generated by extractingbag-of-word features from the product descriptions.
Revised: Amazon Co-Purchasing network [23], where each node represents a product, and the graph linkrepresents whether two products are purchased together, the node features are generated by extractingbag-of-word features from the product descriptions.

Original: The top-level categories are used as labels forthe products.
Revised: The top-level categories are used as labels forthe products.

Original: Inductive learning datasets.
Revised: Inductive learning datasets.

Original: For experimens on inductive node classification, we evaluate our modelon two non-homophilous datasets: Twitch-DE [28] and Facebook100-Baylor93 [35].
Revised: For experimens on inductive node classification, we evaluate our modelon two non-homophilous datasets: Twitch-DE [28] and Facebook100-Baylor93 [35].

Original: Twitchcontains seven networks where users are nodes and mutual friendships between them are edges.
Revised: Twitchcontains seven networks where users are nodes and mutual friendships between them are edges.

Original: Nodefeatures are games liked, location and streaming habits.
Revised: Nodefeatures are games liked, location and streaming habits.

Original: The class labels denote whether a streameruses explicit language.
Revised: The class labels denote whether a streameruses explicit language.

Original: Facebook100 consists of 100 Facebook friendship network snapshots from2005, each of which has as nodes the Facebook users from a given American university.
Revised: Facebook100 consists of 100 Facebook friendship network snapshots from2005, each of which has as nodes the Facebook users from a given American university.

Original: Each nodeis labeled with the reported gender of the user.
Revised: Each nodeis labeled with the reported gender of the user.

Original: The node features are major, year, high school, etc.
Revised: The node features are major, year, high school, etc.

Original: Specifically, we use the subgraph DE from Twitch and the subgraph Baylor93 from Facebook100for experiments.
Revised: Specifically, we use the subgraph DE from Twitch and the subgraph Baylor93 from Facebook100for experiments.

Original: Both of them have roughly 10k nodes.
Revised: Both of them have roughly 10k nodes.

Original: Graph-enhanced application datasets.
Revised: Graph-enhanced application datasets.

Original: We evaluate our model on two datasets without graphstructure: 20News-Groups [24] and Mini-ImageNet [37].
Revised: We evaluate our model on two datasets without graphstructure: 20News-Groups [24] and Mini-ImageNet [37].

Original: The 20News dataset is a collection ofapproximately 20,000 newsgroup documents (nodes), partitioned (nearly) evenly across 20 differentnewsgroups.
Revised: The 20News dataset is a collection ofapproximately 20,000 newsgroup documents (nodes), partitioned (nearly) evenly across 20 differentnewsgroups.

Original: We take 10 classes from 20 newsgroups and use words (TFIDF) with a frequency ofmore than 5% as features.
Revised: We take 10 classes from 20 newsgroups and use words (TFIDF) with a frequency ofmore than 5% as features.

Original: The Mini-ImageNet dataset consists of 84 × 84 RGB images from 100different classes with 600 samples per class.
Revised: The Mini-ImageNet dataset consists of 84 × 84 RGB images from 100different classes with 600 samples per class.

Original: For our experiment use, we choose 30 classes from thedataset, each with 600 images (nodes) that have 128 features extracted by CNN.
Revised: For our experiment use, we choose 30 classes from thedataset, each with 600 images (nodes) that have 128 features extracted by CNN.

Original: C.2 Dataset Preprocessing
Revised: C.2 Dataset Preprocessing

Original: All the datasets we used in the experiment are directly collected from the source, except
Revised: All the datasets we used in the experiment are directly collected from the source, except

Original: Mini-ImageNet , whose features are extracted by ourselves.
Revised: Mini-ImageNet , whose features are extracted by ourselves.

Original: Following the setting of [31], we com- pute node embeddings via a CNN model with 4 convolutional layers followed by a fully-connectedlayer resulting in a 128 dimensional embedding.
Revised: Following the setting of [31], we com- pute node embeddings via a CNN model with 4 convolutional layers followed by a fully-connectedlayer resulting in a 128 dimensional embedding.

Original: Finally, the 128 dimensional outputs are treated asthe features of the nodes (images) for subsequent GNN-based downstream task.
Revised: Finally, the 128 dimensional outputs are treated asthe features of the nodes (images) for subsequent GNN-based downstream task.

Original: D More Experiment Results
Revised: D More Experiment Results

Revised: [Figure]

Original: D.1 Extra Ablation Studies
Revised: D.1 Extra Ablation Studies

Revised: [Figure]

Original: Some ablation study results have been reported in Section 4.5 and here we supplement extra resultson more datasets.
Revised: Some ablation study results have been reported in Section 4.5 and here we supplement extra resultson more datasets.

Original: In Table 6, we summarize the experiment results on the four transductive datasetsfor ALIGNS w/ and w/o relational bias and edge-level regularization.
Revised: In Table 6, we summarize the experiment results on the four transductive datasetsfor ALIGNS w/ and w/o relational bias and edge-level regularization.

Original: The results show that bothcomponents contribute to positive effects, especially the edge-level regularization, without whichwe can see a significant drop for the test accuracy on Cora and Citeseer .
Revised: The results show that bothcomponents contribute to positive effects, especially the edge-level regularization, without whichwe can see a significant drop for the test accuracy on Cora and Citeseer .

Original: The reason could bethat these two datasets are homophilous graphs where observed edges are reliable for providingbeneficial information and in contrast, the other datasets are non-homophilous graphs where thetraining signal from observed edges might not help a lot and could even be harmful.
Revised: The reason could bethat these two datasets are homophilous graphs where observed edges are reliable for providingbeneficial information and in contrast, the other datasets are non-homophilous graphs where thetraining signal from observed edges might not help a lot and could even be harmful.

Original: Similar resultsand interpretations are applicable for relational bias.
Revised: Similar resultsand interpretations are applicable for relational bias.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: D.2 Impact of Temperature and Random Feature Dimension
Revised: D.2 Impact of Temperature and Random Feature Dimension

Original: Fig.
Revised: Fig.

Original: 4 shows the impact of the temperature τ and the dimension of feature map m on Cora .
Revised: 4 shows the impact of the temperature τ and the dimension of feature map m on Cora .

Original: Thefigure illustrates that the test accuracy increases and then falls with the temperature changing fromlow to high values (usually achieves the peak accuracy with a temperature of 0.4).
Revised: Thefigure illustrates that the test accuracy increases and then falls with the temperature changing fromlow to high values (usually achieves the peak accuracy with a temperature of 0.4).

Original: Besides, we cansee that when the temperature is relatively small, the test accuracy goes high with the dimension ofrandom features increasing.
Revised: Besides, we cansee that when the temperature is relatively small, the test accuracy goes high with the dimension ofrandom features increasing.

Original: However, when the temperature is large, the accuracy would drop evenwith large feature dimension m .
Revised: However, when the temperature is large, the accuracy would drop evenwith large feature dimension m .

Original: Such a phenomenon accords with the theoretical result presented in
Revised: Such a phenomenon accords with the theoretical result presented in

Original: Section 3.2.
Revised: Section 3.2.

Original: For low temperature which enables desirable approximation performance for GumbelSoftmax, then larger random feature dimension would help to produce better approximation to theoriginal exponentiate-then-dot operator.
Revised: For low temperature which enables desirable approximation performance for GumbelSoftmax, then larger random feature dimension would help to produce better approximation to theoriginal exponentiate-then-dot operator.

Original: In contrast, high temperature could not guarantee preciseapproximation for the original categorical distribution, which deteriorates the performance.
Revised: In contrast, high temperature could not guarantee preciseapproximation for the original categorical distribution, which deteriorates the performance.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D.3 Visualization and Implications
Revised: D.3 Visualization and Implications

Original: [Figure]
Revised: [Figure]

Original: I n p u t
Revised: I n p u t

Original: G r a p h
Revised: G r a p h

Original: L a y e r L a y e r
Revised: L a y e r L a y e r

Original: We provide the visualization results of node embeddings and edge connections (filter out the edgeswith weights larger than a threshold) on 20News-Groups and Mini-Imagenet in Fig.
Revised: We provide the visualization results of node embeddings and edge connections (filter out the edgeswith weights larger than a threshold) on 20News-Groups and Mini-Imagenet in Fig.

Original: The figuresshow that ALIGNS tends to assign more weights for nodes with the same class and sparse edgesfor nodes with different classes.
Revised: The figuresshow that ALIGNS tends to assign more weights for nodes with the same class and sparse edgesfor nodes with different classes.

Original: This helps to interpret why ALIGNS improves the performance ondownstream node-level prediction: the latent structures can propagate useful information to help themodel learn better node representations that can be easily distinguished by the classifier.
Revised: This helps to interpret why ALIGNS improves the performance ondownstream node-level prediction: the latent structures can propagate useful information to help themodel learn better node representations that can be easily distinguished by the classifier.

Original: Besides,we plot the attention maps of two model layers in Fig.
Revised: Besides,we plot the attention maps of two model layers in Fig.

Original: 6 and also the input graph structures on
Revised: 6 and also the input graph structures on

Original: Cora , Citeseer , Deezer and Actor .
Revised: Cora , Citeseer , Deezer and Actor .

Original: We can see that the latent structures learned by ALIGNS havedifferent patterns than the observed ones, especially for heterophilic graphs.
Revised: We can see that the latent structures learned by ALIGNS havedifferent patterns than the observed ones, especially for heterophilic graphs.

Original: Another interesting phenomenon is that there exist some dominant nodes which are assigned large weights by othernodes, forming some vertical ‘lines’ in the heatmap.
Revised: Another interesting phenomenon is that there exist some dominant nodes which are assigned large weights by othernodes, forming some vertical ‘lines’ in the heatmap.

Original: This suggests that these nodes could containcritical information for the learning tasks and play as pivots that could improve the connectivity ofthe whole system.
Revised: This suggests that these nodes could containcritical information for the learning tasks and play as pivots that could improve the connectivity ofthe whole system.

Original: E Current Limitations, Outlooks and Potential Impacts
Revised: E Current Limitations, Outlooks and Potential Impacts

Original: The proposed model in this paper mainly focuses on node-level tasks for graph-structured data,though it could be extended to other graph-related tasks including link prediction, node regression,sub-graph recognition by using the ALIGNS network as a backbone encoder for node representationlearning.
Revised: The proposed model in this paper mainly focuses on node-level tasks for graph-structured data,though it could be extended to other graph-related tasks including link prediction, node regression,sub-graph recognition by using the ALIGNS network as a backbone encoder for node representationlearning.

Original: For practitioners who would like to apply our model into more industry applications, likerecommendation, advertisement, autonomous driving, etc.
Revised: For practitioners who would like to apply our model into more industry applications, likerecommendation, advertisement, autonomous driving, etc.

Original: that involve graph data processing, onecan also treat our kernelized Gumbel-Softmax operator as a plug-in component for enabling efficientand scalable learning over data points’ inter-dependence and latent interactions, which may help toenhance the representational power and capacity for existing encoder architectures.
Revised: that involve graph data processing, onecan also treat our kernelized Gumbel-Softmax operator as a plug-in component for enabling efficientand scalable learning over data points’ inter-dependence and latent interactions, which may help toenhance the representational power and capacity for existing encoder architectures.

Original: To stay focusedin this paper, we leave these interesting and promising directions as future investigations.
Revised: To stay focusedin this paper, we leave these interesting and promising directions as future investigations.

Original: And, to ourknowledge, we have not identified any negative social impact of this work.
Revised: And, to ourknowledge, we have not identified any negative social impact of this work.