Original: O N THE B OTTLENECK OF G RAPH N EURAL N ETWORKS AND ITS P RACTICAL I MPLICATIONS
Revised: O N THE B OTTLENECK OF G RAPH N EURAL N ETWORKS AND ITS P RACTICAL I MPLICATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNNs) were shown to effectively learn from highly structured data containing elements (nodes) with relationships (edges) between them.
Revised: Graph neural networks (GNNs) were shown to effectively learn from highly structured data containing elements (nodes) with relationships (edges) between them.

Original: GNN variants differ in how each node in the graph absorbs the information ﬂowing from its neighbor nodes.
Revised: GNN variants differ in how each node in the graph absorbs the information ﬂowing from its neighbor nodes.

Original: In this paper, we highlight an inherent problem in GNNs: the mechanism of propagating information between neighbors creates a bottleneck when every node aggregates messages from its neighbors.
Revised: In this paper, we highlight an inherent problem in GNNs: the mechanism of propagating information between neighbors creates a bottleneck when every node aggregates messages from its neighbors.

Original: This bottleneck causes the over-squashing of exponentially-growing information into ﬁxed-size vectors.
Revised: This bottleneck causes the over-squashing of exponentially-growing information into ﬁxed-size vectors.

Original: As a result, the graph fails to propagate messages ﬂowing from distant nodes and performs poorly when the prediction task depends on long-range information.
Revised: As a result, the graph fails to propagate messages ﬂowing from distant nodes and performs poorly when the prediction task depends on long-range information.

Original: We demonstrate that the bottleneck hinders popular GNNs from ﬁtting long-range signals in the training data.
Revised: We demonstrate that the bottleneck hinders popular GNNs from ﬁtting long-range signals in the training data.

Original: We show that GNNs that absorb incoming edges equally, like GCN and GIN, are more susceptible to over-squashing than other GNN types.
Revised: We show that GNNs that absorb incoming edges equally, like GCN and GIN, are more susceptible to over-squashing than other GNN types.

Original: We further show that existing, extensively-tuned, GNN-based models of long-range problems suffer from over-squashing, and that breaking the bottleneck improves state-of-the-art results without any hyperparameter tuning or additional weights.
Revised: We further show that existing, extensively-tuned, GNN-based models of long-range problems suffer from over-squashing, and that breaking the bottleneck improves state-of-the-art results without any hyperparameter tuning or additional weights.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural networks (GNNs) (Scarselli et al., 2008; Micheli, 2009) have seen growing popularity over the last few years (Duvenaud et al., 2015; Hamilton et al., 2017; Xu et al., 2019).
Revised: Graph neural networks (GNNs) (Scarselli et al., 2008; Micheli, 2009) have seen growing popularity over the last few years (Duvenaud et al., 2015; Hamilton et al., 2017; Xu et al., 2019).

Original: Many domains can be naturally represented as graphs.
Revised: Many domains can be naturally represented as graphs.

Original: Therefore, GNNs provide a convenient and general framework to model a variety of real-world complex structural data such as social networks, knowledge graphs, computer programs, and chemical and biological systems.
Revised: Therefore, GNNs provide a convenient and general framework to model a variety of real-world complex structural data such as social networks, knowledge graphs, computer programs, and chemical and biological systems.

Original: A GNN layer can be viewed as a message-passing step (Gilmer et al., 2017), where each node updates its state by aggregating messages ﬂowing from its neighbors.
Revised: A GNN layer can be viewed as a message-passing step (Gilmer et al., 2017), where each node updates its state by aggregating messages ﬂowing from its neighbors.

Original: GNN variants (Li et al., 2016;
Revised: GNN variants (Li et al., 2016;

Original: Veli ˇ ckovi ´ c et al., 2018; Kipf and Welling, 2017) mostly differ in how each node aggregates the representations of its neighbors and combines them with its own representation.
Revised: Veli ˇ ckovi ´ c et al., 2018; Kipf and Welling, 2017) mostly differ in how each node aggregates the representations of its neighbors and combines them with its own representation.

Original: In this paper, we show that this message-passing mechanism creates a numerical information bottleneck when computing neighbor aggregation.
Revised: In this paper, we show that this message-passing mechanism creates a numerical information bottleneck when computing neighbor aggregation.

Original: Problems that depend on long-range interaction between nodes must use as many GNN layers as the desired radius of a node’s receptive ﬁeld.
Revised: Problems that depend on long-range interaction between nodes must use as many GNN layers as the desired radius of a node’s receptive ﬁeld.

Original: Unfortunately, the number of nodes in the receptive ﬁeld grows exponentially with the number of layers.
Revised: Unfortunately, the number of nodes in the receptive ﬁeld grows exponentially with the number of layers.

Original: This causes over-squashing : information from the exponentially-growing receptive ﬁeld is compressed into ﬁxed-length vectors.
Revised: This causes over-squashing : information from the exponentially-growing receptive ﬁeld is compressed into ﬁxed-length vectors.

Original: Consequently, the graph fails to propagate messages ﬂowing from distant nodes; the model learns only short-range signals from the training data, and overall, generalizes poorly.
Revised: Consequently, the graph fails to propagate messages ﬂowing from distant nodes; the model learns only short-range signals from the training data, and overall, generalizes poorly.

Original: In fact, the GNN bottleneck is analogous to the bottleneck of sequential recurrent models.
Revised: In fact, the GNN bottleneck is analogous to the bottleneck of sequential recurrent models.

Original: Traditional seq2seq models (Sutskever et al., 2014; Cho et al., 2014a;b) suffered from a bottleneck at every decoder state – the model had to encapsulate the entire input sequence into a ﬁxed-size vector.
Revised: Traditional seq2seq models (Sutskever et al., 2014; Cho et al., 2014a;b) suffered from a bottleneck at every decoder state – the model had to encapsulate the entire input sequence into a ﬁxed-size vector.

Original: In
Revised: In

Original: GNNs, the bottleneck is even more harmful, because the receptive ﬁeld of a node grows exponentially with the number of message propagation steps, rather than linearly as in recurrent models.
Revised: GNNs, the bottleneck is even more harmful, because the receptive ﬁeld of a node grows exponentially with the number of message propagation steps, rather than linearly as in recurrent models.

Original: This difference is illustrated in Figure 1.
Revised: This difference is illustrated in Figure 1.

Original: This work does not aim to propose a new GNN variant.
Revised: This work does not aim to propose a new GNN variant.

Original: Rather, the main contribution of this work is highlighting the inherent bottleneck problem of GNNs, and this is the ﬁrst work to discuss the over-squashing problem.
Revised: Rather, the main contribution of this work is highlighting the inherent bottleneck problem of GNNs, and this is the ﬁrst work to discuss the over-squashing problem.

Original: We use a controlled synthetic problem to demonstrate the existence of a bottleneck and to provide combinatorial upper bounds for the graph size given the network’s hidden dimension (Section 5).
Revised: We use a controlled synthetic problem to demonstrate the existence of a bottleneck and to provide combinatorial upper bounds for the graph size given the network’s hidden dimension (Section 5).

Original: We show, analytically and empirically, that GCN (Kipf and Welling, 2017)
Revised: We show, analytically and empirically, that GCN (Kipf and Welling, 2017)

Original: [Figure]
Revised: [Figure]

Original: and GIN (Xu et al., 2019) suffer from over-squashing more than other types of GNNs, and even in small graphs.
Revised: and GIN (Xu et al., 2019) suffer from over-squashing more than other types of GNNs, and even in small graphs.

Original: We further show that existing models of real-world datasets suffer from over-squashing: breaking the bottleneck relatively reduces the error rate by 42% in the QM9 dataset, by 12% in
Revised: We further show that existing models of real-world datasets suffer from over-squashing: breaking the bottleneck relatively reduces the error rate by 42% in the QM9 dataset, by 12% in

Original: ENZYMES, and by 4.8% in NCI1, without any hyperparameter tuning nor additional weights.
Revised: ENZYMES, and by 4.8% in NCI1, without any hyperparameter tuning nor additional weights.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A directed graph G =
Revised: A directed graph G =

Original: ( V , E ) contains nodes V and edges E ,
Revised: ( V , E ) contains nodes V and edges E ,

Original: where ( u, v ) ∈
Revised: where ( u, v ) ∈

Original: E denotes an edge from a node u to a node v .
Revised: E denotes an edge from a node u to a node v .

Original: For brevity, in the following deﬁnitions we treat all edges as having the same type ; in general, every edge can have a type and features (Schlichtkrull et al., 2018).
Revised: For brevity, in the following deﬁnitions we treat all edges as having the same type ; in general, every edge can have a type and features (Schlichtkrull et al., 2018).

Original: Graph neural networks Graph neural networks operate by propagating neural messages between neighboring nodes.
Revised: Graph neural networks Graph neural networks operate by propagating neural messages between neighboring nodes.

Original: At every propagation step (a graph layer): the network computes each node’s sent message; every node aggregates its received messages; and each node updates its representation by combining the aggregated incoming messages with its own previous representation.
Revised: At every propagation step (a graph layer): the network computes each node’s sent message; every node aggregates its received messages; and each node updates its representation by combining the aggregated incoming messages with its own previous representation.

Original: Formally, each node is associated with an initial representation h (0) v ∈ R d 0 .
Revised: Formally, each node is associated with an initial representation h (0) v ∈ R d 0 .

Original: This representation is usually derived from the node’s label or its given features.
Revised: This representation is usually derived from the node’s label or its given features.

Original: Then, a GNN layer updates each node’s representation given its neighbors, yielding h (1) v ∈ R d .
Revised: Then, a GNN layer updates each node’s representation given its neighbors, yielding h (1) v ∈ R d .

Original: In general, the k -th layer of a GNN is a parametric function f k that is applied to each node by considering its neighbors:
Revised: In general, the k -th layer of a GNN is a parametric function f k that is applied to each node by considering its neighbors:

Original: [Equation]
Revised: [Equation]

Original: where N v is the set of nodes that have edges to v : N v = { u ∈ V | ( u, v ) ∈ E} .
Revised: where N v is the set of nodes that have edges to v : N v = { u ∈ V | ( u, v ) ∈ E} .

Original: The total number of layers K is usually determined empirically as a hyperparameter.
Revised: The total number of layers K is usually determined empirically as a hyperparameter.

Original: The design of the function f is what mostly distinguishes one type of GNN from the other.
Revised: The design of the function f is what mostly distinguishes one type of GNN from the other.

Original: For example, graph convolutional networks (GCN) (Kipf and Welling, 2017) deﬁne f as:
Revised: For example, graph convolutional networks (GCN) (Kipf and Welling, 2017) deﬁne f as:

Original: [Equation]
Revised: [Equation]

Original: where σ is a nonlinearity such as ReLU , and c u,v is a normalization factor often set to (cid:112)
Revised: where σ is a nonlinearity such as ReLU , and c u,v is a normalization factor often set to (cid:112)

Original: |N v | · |N u | or |N v | (Hamilton et al., 2017).
Revised: |N v | · |N u | or |N v | (Hamilton et al., 2017).

Original: As another example, graph isomorphism networks (GIN) (Xu et al., 2019) update a node’s representation using the following deﬁnition:
Revised: As another example, graph isomorphism networks (GIN) (Xu et al., 2019) update a node’s representation using the following deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: Usually, the last ( K -th) layer’s output is used for prediction: in node-prediction tasks, h ( K ) v is used to predict a label for v ; in graph-prediction tasks, a permutation-invariant “readout” function aggregates the nodes of the ﬁnal layer using summation, averaging, or a weighted sum (Li et al., 2016).
Revised: Usually, the last ( K -th) layer’s output is used for prediction: in node-prediction tasks, h ( K ) v is used to predict a label for v ; in graph-prediction tasks, a permutation-invariant “readout” function aggregates the nodes of the ﬁnal layer using summation, averaging, or a weighted sum (Li et al., 2016).

Original: [Figure]
Revised: [Figure]

Original:  T HE GNN B OTTLENECK
Revised:  T HE GNN B OTTLENECK

Original: When a prediction problem relies on long-range interaction between nodes, the GNN is required to have as many layers as the estimated length of these interactions.
Revised: When a prediction problem relies on long-range interaction between nodes, the GNN is required to have as many layers as the estimated length of these interactions.

Original: However, the number of nodes in the receptive ﬁeld of a node grows exponentially with the number of layers.
Revised: However, the number of nodes in the receptive ﬁeld of a node grows exponentially with the number of layers.

Original: As a result, an exponentially-growing amount of information is squashed into a ﬁxed-length vector (i.e., the result of the (cid:80) in Equations (2) and (3)), and crucial messages fail to reach their distant destinations.
Revised: As a result, an exponentially-growing amount of information is squashed into a ﬁxed-length vector (i.e., the result of the (cid:80) in Equations (2) and (3)), and crucial messages fail to reach their distant destinations.

Original: Instead, the model may learn only short-ranged signals and consequently generalize poorly at test time.
Revised: Instead, the model may learn only short-ranged signals and consequently generalize poorly at test time.

Original: For example, consider the N EIGHBORS M ATCH problem of Figure 2.
Revised: For example, consider the N EIGHBORS M ATCH problem of Figure 2.

Original: Green nodes ( A , B , C ) have a varying number of blue neighbors ( A ) and an alphabetical label.
Revised: Green nodes ( A , B , C ) have a varying number of blue neighbors ( A ) and an alphabetical label.

Original: Every example in the dataset has a different mapping from numbers of neighbors to labels.
Revised: Every example in the dataset has a different mapping from numbers of neighbors to labels.

Original: The rest of the graph (marked as ) represents a general, unknown, graph structure.
Revised: The rest of the graph (marked as ) represents a general, unknown, graph structure.

Original: The goal is to predict a label for the target node, which is marked with a question mark ( ? ), according to its number of blue neighbors.
Revised: The goal is to predict a label for the target node, which is marked with a question mark ( ? ), according to its number of blue neighbors.

Original: The correct answer is C in this case, because the target node has two blue neighbors, like the node marked with C in the same graph.
Revised: The correct answer is C in this case, because the target node has two blue neighbors, like the node marked with C in the same graph.

Revised: Since every example in the dataset has a different mapping from numbers of neighbors to labels, this long-range message propagation and matching between the target node and all the green nodes be performed for every example.

Original: Since the model must propagate information from all green nodes before predicting the label, a bottleneck at the target node is inevitable.
Revised: Since the model must propagate information from all green nodes before predicting the label, a bottleneck at the target node is inevitable.

Original: This bottleneck causes over-squashing , which can prevent the model from ﬁtting the training data perfectly, even though the desired prediction is obvious in a global view.
Revised: This bottleneck causes over-squashing , which can prevent the model from ﬁtting the training data perfectly, even though the desired prediction is obvious in a global view.

Original: We demonstrate the bottleneck empirically in an instance of this problem in Section 4; in Section 5, we provide upper bounds for the learnable graph size.
Revised: We demonstrate the bottleneck empirically in an instance of this problem in Section 4; in Section 5, we provide upper bounds for the learnable graph size.

Original: Although this is a contrived problem, it resembles real-world problems that are often modeled as graphs.
Revised: Although this is a contrived problem, it resembles real-world problems that are often modeled as graphs.

Original: For example, a computer program in a language such as Python may declare multiple variables (i.e., the green nodes in Figure 2) along with their types and values (their numbers of blue neighbors in Figure 2); later in the program, predicting which variable should be used in a speciﬁc location (predict the alphabetical label in Figure 2) must use one of the available variables based on the required type and the required value at that point.
Revised: For example, a computer program in a language such as Python may declare multiple variables (i.e., the green nodes in Figure 2) along with their types and values (their numbers of blue neighbors in Figure 2); later in the program, predicting which variable should be used in a speciﬁc location (predict the alphabetical label in Figure 2) must use one of the available variables based on the required type and the required value at that point.

Original: We experiment with this problem in Section 4.4.
Revised: We experiment with this problem in Section 4.4.

Original: Short- vs. long-range problems Much of prior GNN work has focused on problems that were local in nature, where the underlying inductive bias was that a node’s most relevant context is its local neighborhood, and long-range interaction was not necessarily needed.
Revised: Short- vs. long-range problems Much of prior GNN work has focused on problems that were local in nature, where the underlying inductive bias was that a node’s most relevant context is its local neighborhood, and long-range interaction was not necessarily needed.

Original: With the growing popularity of GNNs, their adoption expanded to domains that required longer-range information propagation as well, without addressing the inherent bottleneck.
Revised: With the growing popularity of GNNs, their adoption expanded to domains that required longer-range information propagation as well, without addressing the inherent bottleneck.

Original: In this paper, we focus on problems that require long-range information.
Revised: In this paper, we focus on problems that require long-range information.

Original: That is, a correct prediction requires considering the local environment of a node and interactions beyond the close neighborhood.
Revised: That is, a correct prediction requires considering the local environment of a node and interactions beyond the close neighborhood.

Original: For example, a chemical property of a molecule (Ramakrishnan et al., 2014; Gilmer et al., 2017) can depend on the combination of atoms that reside in the molecule’s opposite sides .
Revised: For example, a chemical property of a molecule (Ramakrishnan et al., 2014; Gilmer et al., 2017) can depend on the combination of atoms that reside in the molecule’s opposite sides .

Original: Problems of this kind require long-range interaction, and  thus, a large number of GNN layers.
Revised: Problems of this kind require long-range interaction, and [Figure] thus, a large number of GNN layers.

Original: Since the receptive ﬁeld of each node grows exponentially with the number of layers, the more layers – the more harmful the effect of the bottleneck.
Revised: Since the receptive ﬁeld of each node grows exponentially with the number of layers, the more layers – the more harmful the effect of the bottleneck.

Original: In problems that are local in nature – the bottleneck is less troublesome, because a GNN can perform well with only few layers (e.g., 2 layers in Kipf and Welling (2017)), and the receptive ﬁeld of a node can be exponentially smaller.
Revised: In problems that are local in nature – the bottleneck is less troublesome, because a GNN can perform well with only few layers (e.g., 2 layers in Kipf and Welling (2017)), and the receptive ﬁeld of a node can be exponentially smaller.

Original: Domains such as citation networks (Sen et al., 2008), social networks (Leskovec and Mcauley, 2012), and product recommendations (Shchur et al., 2018) usually raise short-range problems and are thus not the focus of this paper.[Figure]
Revised: Domains such as citation networks (Sen et al., 2008), social networks (Leskovec and Mcauley, 2012), and product recommendations (Shchur et al., 2018) usually raise short-range problems and are thus not the focus of this paper.

Original:  E VALUATION
Revised:  E VALUATION

Original: First, we wish to empirically show that the GNN bottleneck exists, even in small graphs.
Revised: First, we wish to empirically show that the GNN bottleneck exists, even in small graphs.

Original: We generated a synthetic benchmark that is theoretically solvable; however, in practice, all GNNs fail to reach 100% training accuracy because of the bottleneck (Section 4.1).
Revised: We generated a synthetic benchmark that is theoretically solvable; however, in practice, all GNNs fail to reach 100% training accuracy because of the bottleneck (Section 4.1).

Original: Second, we examine whether the bottleneck exists in prior models, which addressed real-world problems (Sections 4.2 to 4.4).
Revised: Second, we examine whether the bottleneck exists in prior models, which addressed real-world problems (Sections 4.2 to 4.4).

Original: To that end, we wish to measure over-squashing in existing models.
Revised: To that end, we wish to measure over-squashing in existing models.

Original: But, how can we measure over-squashing?
Revised: But, how can we measure over-squashing?

Original: Instead, we measure whether breaking the bottleneck improves the results of long-range problems.
Revised: Instead, we measure whether breaking the bottleneck improves the results of long-range problems.

Original: Adding a fully-adjacent layer (FA)
Revised: Adding a fully-adjacent layer (FA)

Original: We took off-the-shelf, extensively tuned, models, and modiﬁed adjacency in the last layer by modifying the authors’ original code: given a GNN with K layers, we modiﬁed the K -th layer to be a fully-adjacent layer .
Revised: We took off-the-shelf, extensively tuned, models, and modiﬁed adjacency in the last layer by modifying the authors’ original code: given a GNN with K layers, we modiﬁed the K -th layer to be a fully-adjacent layer .

Original: A fully-adjacent layer is a GNN layer in which every pair of nodes is connected by an edge.
Revised: A fully-adjacent layer is a GNN layer in which every pair of nodes is connected by an edge.

Original: In terms of Equations (1) to (3), converting an existing layer to be fully-adjacent means that N v :=
Revised: In terms of Equations (1) to (3), converting an existing layer to be fully-adjacent means that N v :=

Original: V for every node v ∈ V , only in that layer.
Revised: V for every node v ∈ V , only in that layer.

Original: This does not change the type of layer nor add weights, but only changes the notion of adjacency of a data sample in a single layer.
Revised: This does not change the type of layer nor add weights, but only changes the notion of adjacency of a data sample in a single layer.

Original: Thus, the K − 1 graph layers exploit the graph structure using their original sparse topology, and only the K -th layer is an FA layer that allows the topology-aware node-representations to interact directly and consider nodes beyond their original neighbors.
Revised: Thus, the K − 1 graph layers exploit the graph structure using their original sparse topology, and only the K -th layer is an FA layer that allows the topology-aware node-representations to interact directly and consider nodes beyond their original neighbors.

Original: Hopefully, this would ease information ﬂow, prevent over-squashing, and reduce the effect of the previously-existed bottleneck.
Revised: Hopefully, this would ease information ﬂow, prevent over-squashing, and reduce the effect of the previously-existed bottleneck.

Original: We then re-trained the models without performing any additional tuning, to rule out hyperparameter tuning as the source of improvement.
Revised: We then re-trained the models without performing any additional tuning, to rule out hyperparameter tuning as the source of improvement.

Original: Statistics of all datasets can be found in Appendix D.
Revised: Statistics of all datasets can be found in Appendix D.

Original: We emphasize that an FA layer is a simple solution.
Revised: We emphasize that an FA layer is a simple solution.

Original: Its purpose is to demonstrate that over-squashing in GNNs is so prevalent and untreated that even the simplest solution helps .
Revised: Its purpose is to demonstrate that over-squashing in GNNs is so prevalent and untreated that even the simplest solution helps .

Original: This simple solution opens the path for a variety of follow-up improvements and better solutions for over-squashing.
Revised: This simple solution opens the path for a variety of follow-up improvements and better solutions for over-squashing.

Original: 4. S YNTHETIC B ENCHMARK : N EIGHBORS M ATCH
Revised: 4. S YNTHETIC B ENCHMARK : N EIGHBORS M ATCH

Original: The N EIGHBORS M ATCH problem (Figure 2) is a simple contrived problem that we designed to provide an intuition to the extend of the effect of over-squashing, while allowing us to control the intensity of over-squashing.
Revised: The N EIGHBORS M ATCH problem (Figure 2) is a simple contrived problem that we designed to provide an intuition to the extend of the effect of over-squashing, while allowing us to control the intensity of over-squashing.

Original: We focus on the training accuracy of a model.
Revised: We focus on the training accuracy of a model.

Original: T REE -N EIGHBORS M ATCH
Revised: T REE -N EIGHBORS M ATCH

Original: We created an instance of the general N EIGHBORS M ATCH problem that we described in Section 3 and portrayed in Figure 2.
Revised: We created an instance of the general N EIGHBORS M ATCH problem that we described in Section 3 and portrayed in Figure 2.

Original: As observed before (Micheli, 2009; Chen et al., 2018), the receptive ﬁeld of a node in a graph grows exponentially with the number of layers.
Revised: As observed before (Micheli, 2009; Chen et al., 2018), the receptive ﬁeld of a node in a graph grows exponentially with the number of layers.

Original: Thus, from the perspective of a single node v , the rest of the graph may look like a tree, rooted at v (Garg et al., 2020).
Revised: Thus, from the perspective of a single node v , the rest of the graph may look like a tree, rooted at v (Garg et al., 2020).

Original: To simulate this exponentially-growing receptive ﬁeld, we instantiated the subgraph in the middle of the graph (marked as in Figure 2) as a binary tree of depth depth where the green nodes are its leaves, and the target node is the tree’s root.
Revised: To simulate this exponentially-growing receptive ﬁeld, we instantiated the subgraph in the middle of the graph (marked as in Figure 2) as a binary tree of depth depth where the green nodes are its leaves, and the target node is the tree’s root.

Original: All edges are directed toward the root, such that information is propagated from all nodes toward the target node.
Revised: All edges are directed toward the root, such that information is propagated from all nodes toward the target node.

Original: The goal, as in Section 3, is to predict a label for the target node, where the correct answer is the label of the green node that has the same number of blue neighbors as the target node.
Revised: The goal, as in Section 3, is to predict a label for the target node, where the correct answer is the label of the green node that has the same number of blue neighbors as the target node.

Original: An illustration is shown in Figure 5 in the appendix.
Revised: An illustration is shown in Figure 5 in the appendix.

Original: In this section we observe the bottleneck empirically; in Section 5 we provide a combinatorial upper bound for the learnable graph size in this problem.
Revised: In this section we observe the bottleneck empirically; in Section 5 we provide a combinatorial upper bound for the learnable graph size in this problem.

Original: Model We implemented a network with depth +1 graph layers to allow an additional nonlinear layer after the information from the leaves reaches the target node.
Revised: Model We implemented a network with depth +1 graph layers to allow an additional nonlinear layer after the information from the leaves reaches the target node.

Original: Our PyTorch Geometric (Fey and
Revised: Our PyTorch Geometric (Fey and

Original: Lenssen, 2019) implementation is included in the supplementary material and we make it publicly available.
Revised: Lenssen, 2019) implementation is included in the supplementary material and we make it publicly available.

Original: Our training conﬁguration and hyperparameter ranges are detailed in Appendix A.
Revised: Our training conﬁguration and hyperparameter ranges are detailed in Appendix A.

Original: Results Figure 3 shows the following surprising results: GNNs fail to ﬁt the dataset starting from depth =4 .
Revised: Results Figure 3 shows the following surprising results: GNNs fail to ﬁt the dataset starting from depth =4 .

Original: For example, the training accuracy of GCN  at depth =4 is 70%.
Revised: For example, the training accuracy of GCN (Kipf and Welling, 2017) at depth =4 is 70%.

Original: At depth =5 , all GNNs fail to perfectly ﬁt the data.
Revised: At depth =5 , all GNNs fail to perfectly ﬁt the data.

Original: Starting from depth =4 , the models suffered from over-squashing that resulted in underﬁtting : the bottleneck prevented the models from distinguishing between different training examples, even after they were observed tens of thousands of times.
Revised: Starting from depth =4 , the models suffered from over-squashing that resulted in underﬁtting : the bottleneck prevented the models from distinguishing between different training examples, even after they were observed tens of thousands of times.

Original: These results clearly show the existence of the bottleneck and its negative effect, even in small graphs.
Revised: These results clearly show the existence of the bottleneck and its negative effect, even in small graphs.

Original: Discussion GCN and GIN  managed to perfectly ﬁt depth =3 at most, while GGNN  and GAT also reached 100% accuracy at depth
Revised: Discussion GCN and GIN (Xu et al., 2019) managed to perfectly ﬁt depth =3 at most, while GGNN (Li et al., 2016) and GAT (Veli ˇ ckovi ´

Original:  =4 .
Revised: c et al., 2018) also reached 100% accuracy at depth =4 .

Original: This difference can be explained by their neighbor aggregation computation: consider the target node that receives messages in the depth ’th step.
Revised: This difference can be explained by their neighbor aggregation computation: consider the target node that receives messages in the depth ’th step.

Original: GCN and GIN aggregate all neighbors before combining them with the target node’s representation, and thus they must compress the information ﬂowing from all leaves into a single vector, and only afterward interact with the target node’s own representation (Equations (2) and (3)).
Revised: GCN and GIN aggregate all neighbors before combining them with the target node’s representation, and thus they must compress the information ﬂowing from all leaves into a single vector, and only afterward interact with the target node’s own representation (Equations (2) and (3)).

Original: In contrast, a GAT layer uses the target’s own representation to weight incoming messages.
Revised: In contrast, a GAT layer uses the target’s own representation to weight incoming messages.

Original: The target node can thus ignore the irrelevant incoming edge  and absorb only the relevant incoming edge, which contains information ﬂowing from half of the leaves.
Revised: The target node can thus ignore the irrelevant incoming edge (by giving them very small attention weight) and absorb mostly the relevant incoming edge, which contains information ﬂowing from half of the leaves.

Original: Following Levy et al.
Revised: Following Levy et al.

Original: (2018), we hypothesize that the GRU cell in
Revised: (2018), we hypothesize that the GRU cell in

Original: GGNNs ﬁlters incoming edges as GAT, but perform this ﬁltering as element-wise attention.
Revised: GGNNs ﬁlters incoming edges as GAT, but perform this ﬁltering as element-wise attention.

Original: Since the number of leaves grows exponentially with depth , it is expected that GNNs that need to compress only half of the information (GGNN and GAT) will succeed at a depth that is larger by 1.
Revised: Since the number of leaves grows exponentially with depth , it is expected that GNNs that need to compress only half of the information (GGNN and GAT) will succeed at a depth that is larger by 1.

Original: If all GNNs have reached low training accuracy, how do GNN-based models usually do ﬁt the training data in public datasets of long-range problems?
Revised: If all GNNs have reached low training accuracy, how do GNN-based models usually do ﬁt the training data in public datasets of long-range problems?

Original: We hypothesize that they learn only short-range signals from the training set, rather than learning the information that was squashed in the bottleneck.
Revised: We hypothesize that they learn only short-range signals from the training set, rather than learning the information that was squashed in the bottleneck.

Original: 4. Q UANTUM C HEMISTRY : QM
Revised: Over-squashing or just long-range? To rule out the possibility that the long-range itself is preventing the GNNs from ﬁtting the data, we repeated the experiment of Figure 3 for depth s 4 to 8, where the distance between the leaves and the target node remained the same (4 to 8), but the amount of information was as in depth =3 . In other words, the graph looks like a tree of depth =3 , where the root is connected to a “chain” of length of up to 5, and the target node is at the other side of the “chain”.

Original:  Data
Revised: This setting maintains the long-range as in the original problem, but reduces the amount of information that needs to be squashed. In other words, this setting disentangles of the effect of the long-range itself from the effect of the growing amount of information (i.e., from over-squashing). In this setting, all GNN types managed to easily ﬁt the data to close to 100% across all distances, showing that the problem is not the long-range itself, but rather, the growing amount of information (i.e., over-squashing). 4. Q UANTUM C HEMISTRY : QM Data

Original: The QM9 dataset (Ramakrishnan et al., 2014; Gilmer et al., 2017; Wu et al., 2018) contains ~130,000 graphs with ~18 nodes.
Revised: The QM9 dataset (Ramakrishnan et al., 2014; Gilmer et al., 2017; Wu et al., 2018) contains ~130,000 graphs with ~18 nodes.

Original: Each graph is a molecule where nodes are atoms, and undirected, typed edges are different types of bonds between the atoms.
Revised: Each graph is a molecule where nodes are atoms, and undirected, typed edges are different types of bonds between the atoms.

Original: The goal is to regress each graph toreal-valued quantum chemical properties such as dipole moment and isotropic polarizability .
Revised: The goal is to regress each graph toreal-valued quantum chemical properties such as dipole moment and isotropic polarizability .

Original:  Models We used the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning for multiple GNNs, by searching over 500 conﬁgurations; we took the same splits and their best-found conﬁgurations.
Revised: [Table] Models We used the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning for multiple GNNs, by searching over 500 conﬁgurations; we took the same splits and their best-found conﬁgurations.

Original: For most GNNs, Brockschmidt found that the best results are achieved using eight propagation steps.
Revised: For most GNNs, Brockschmidt found that the best results are achieved using eight propagation steps.

Original: This hints that this problem depends on long-range information and relies on both graph structure and distant nodes.
Revised: This hints that this problem depends on long-range information and relies on both graph structure and distant nodes.

Original: We modiﬁed the last layer to be an FA layer by extending their implementation.
Revised: We modiﬁed the last layer to be an FA layer by extending their implementation.

Original: We re-trained each modiﬁed model for each target property using the same code, conﬁguration, and training scheme as Brockschmidt (2020), training each model ﬁve times (using different random seeds) for each target property task.
Revised: We re-trained each modiﬁed model for each target property using the same code, conﬁguration [Table] and training scheme as Brockschmidt (2020), training each model ﬁve times (using different random seeds) for each target property task.

Original: We compare the “base” models, reported by Brockschmidt, with our modiﬁed and re-trained “ + FA” models.
Revised: We compare the “base” models, reported by Brockschmidt, with our modiﬁed and re-trained “ + FA” models.

Original: Results Results for the top GNNs are shown in Table 1; results for the other GNNs are shown in
Revised: Results Results for the top GNNs are shown in Table 1; results for the other GNNs are shown in

Original: Appendix B due to space limitation.
Revised: Appendix B due to space limitation.

Original: The main results are that breaking the bottleneck by modifying a single layer to be an FA layer signiﬁcantly reduces the error rate, by 42% on average, across six
Revised: The main results are that breaking the bottleneck by modifying a single layer to be an FA layer signiﬁcantly reduces the error rate, by 42% on average, across six

Original: GNN types.
Revised: GNN types.

Original: These experiments clearly show evidence for a bottleneck in the original GNN models.
Revised: These experiments clearly show evidence for a bottleneck in the original GNN models.

Original: Can larger hidden dimension achieve a similar improvement?
Revised: Can larger hidden dimension achieve a similar improvement?

Original: We trained another set of models with doubled dimension sizes (Appendix B).
Revised: We trained another set of models with doubled dimension sizes (Appendix B).

Original: These models achieved only 5.5% improvement over the base model, while adding the FA layer achieved 42% improvement using the original dimensions.
Revised: These models achieved only 5.5% improvement over the base model, while adding the FA layer achieved 42% improvement using the original dimensions.

Original: If all GNNs beneﬁt from direct interaction between all nodes, maybe the graph structure is not even needed?
Revised: If all GNNs beneﬁt from direct interaction between all nodes, maybe the graph structure is not even needed?

Original: We trained another set of models (Appendix B) where all layers are FA layers, thus ignoring the original graph structure; these models produced 1500% worse error.[Table]
Revised: We trained another set of models (Appendix B) where all layers are FA layers, thus ignoring the original graph structure; these models produced 1500% worse error.

Original: Over-squashing or under-reaching?
Revised: Over-squashing or under-reaching?

Original: Barceló et al.
Revised: Barceló et al.

Original: (2020) discuss the inability of a GNN node to observe nodes that are farther away than the number of layers K .
Revised: (2020) discuss the inability of a GNN node to observe nodes that are farther away than the number of layers K .

Original: We denote this limitation as underreaching : for every ﬁxed number of layers K , local information cannot travel farther than distance K along edges.
Revised: We denote this limitation as underreaching : for every ﬁxed number of layers K , local information cannot travel farther than distance K along edges.

Original: So, was the improvement of the FA layer in Table 1 achieved thanks to the reduction in over-squashing, or did the FA layer only extend the nodes’ reachability and prevent under-reaching?
Revised: So, was the improvement of the FA layer in Table 1 achieved thanks to the reduction in over-squashing, or did the FA layer only extend the nodes’ reachability and prevent under-reaching?

Original: To answer this [Table] we measured the graphs’ diameter in the QM9 dataset – the maximum shortest path between two nodes in a graph.
Revised: To answer this, we measured the graphs’ diameter in the QM9 dataset – the maximum shortest path between two nodes in a graph.

Original: We found that the average diameter is 6 .
Revised: We found that the average diameter is 6 .

Original: 35 ± 0 .
Revised: 35 ± 0 .

Original: 91 , the maximum diameter is [Table] and the 90th percentile is 8, while most models were trained with 8 layers.
Revised: 91 , the maximum diameter is 10 , and the 90th percentile is 8, while most models were trained with 8 layers.

Original: That is, at least 90% of the examples in the dataset certainly did not suffer from under-reaching, because the number of layers was greater or equal than their diameter.
Revised: That is, at least 90% of the examples in the dataset certainly did not suffer from under-reaching, because the number of layers was greater or equal than their diameter.

Original: We trained another set of models withlayers, which did not show an improvement over the base models.
Revised: We trained another set of models withlayers, which did not show an improvement over the base models.

Original: We conclude that the source of improvement was clearly not the increased reachability, but instead, the reduction in over-squashing. 
Revised: We conclude that the source of improvement was clearly not the increased reachability, but instead, the reduction in over-squashing. [Table] [Table] [Figure]

Original: 4. B IOLOGICAL B ENCHMARKS
Revised: B IOLOGICAL B ENCHMARKS

Original: Data We experimented with two popular biological datasets.
Revised: Data We experimented with two popular biological datasets.

Original: NCI1 (Wale et al., 2008) contains ~ nodes and its task is to predict whether a biochemical compound contains anti-lung-cancer activity.
Revised: NCI1 (Wale et al., 2008) contains ~ nodes and its task is to predict whether a biochemical compound contains anti-lung-cancer activity.

Original: ENZYMES (Borgwardt et al., 2005) contains ~36 nodes and its task is to classify an enzyme to one out of six classes.
Revised: ENZYMES (Borgwardt et al., 2005) contains ~36 nodes and its task is to classify an enzyme to one out of six classes.

Original: We used the same 10-folds and training/validation/test split as Errica et al.
Revised: We used the same 10-folds and training/validation/test split as Errica et al.

Original: Models We used the implementation of Errica et al.
Revised: Models We used the implementation of Errica et al.

Original: (2020) who performed a fair and thorough comparison between GNNs.
Revised: (2020) who performed a fair and thorough comparison between GNNs.

Original: The ﬁnal reported result is the average of 30 test runs (10-folds × 3).
Revised: The ﬁnal reported result is the average of 30 test runs (10-folds × 3).

Original: Additional training details are provided in Appendix C.
Revised: Additional training details are provided in Appendix C.

Original: In ENZYMES, Errica et al.
Revised: In ENZYMES, Errica et al.

Original: found that a baseline that does not use the graph topology at all (“ No
Revised: found that a baseline that does not use the graph topology at all (“ No

Original: Struct ”) performs better than all GNNs.
Revised: Struct ”) performs better than all GNNs.

Original: In NCI1, GIN performed best.
Revised: In NCI1, GIN performed best.

Original: We converted the last layer into an FA layer by modifying the implementation of Errica et al., and repeated the same training procedure.
Revised: We converted the last layer into an FA layer by modifying the implementation of Errica et al., and repeated the same training procedure.

Original: We compare the “base” models from Errica et al.
Revised: We compare the “base” models from Errica et al.

Original: with our re-trained “+FA” models.
Revised: with our re-trained “+FA” models.

Original: Results Results are shown in Table 2.
Revised: Results Results are shown in Table 2.

Original: The main results are as follows: (a) in NCI1, GIN+FA improves by 1.5% over GIN-base, which was previously the best performing model; (b) in ENZYMES, where
Revised: The main results are as follows: (a) in NCI1, GIN+FA improves by 1.5% over GIN-base, which was previously the best performing model; (b) in ENZYMES, where

Original: Errica et al.
Revised: Errica et al.

Original: (2020) found that none of the GNNs exploit the topology of the graph, we ﬁnd that
Revised: (2020) found that none of the GNNs exploit the topology of the graph, we ﬁnd that

Original: GIN+FA does exploit the structure and improves by 8.1% over GIN-base and by 2.5% over No Struct .
Revised: GIN+FA does exploit the structure and improves by 8.1% over GIN-base and by 2.5% over No Struct .

Original: On average, models with FA layers relatively reduce the error rate by 12% in ENZYMES and by 4.8% in NCI1.
Revised: On average, models with FA layers relatively reduce the error rate by 12% in ENZYMES and by 4.8% in NCI1.

Original: These experiments clearly show evidence for a bottleneck in the original GNN models.[Table] [Table] [Table]
Revised: These experiments clearly show evidence for a bottleneck in the original GNN models.

Original: 4. P ROGRAMS : V AR M ISUSE
Revised: 4. P ROGRAMS : V AR M ISUSE

Original: Data V AR M ISUSE (Allamanis et al., 2018) is a node-prediction problem that depends on long-range information in computer programs.
Revised: Data V AR M ISUSE (Allamanis et al., 2018) is a node-prediction problem that depends on long-range information in computer programs.

Original: We used the same training/validation/SeenProjTest/UnseenProjTest splits as Allamanis et al.
Revised: We used the same training/validation/SeenProjTest/UnseenProjTest splits as Allamanis et al.

Original: (2018) and Brockschmidt (2020).
Revised: (2018) and Brockschmidt (2020).

Original: Models We use the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning by searching over 30 conﬁgurations for each GNN type.
Revised: Models We use the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning by searching over 30 conﬁgurations for each GNN type.

Original: The best results were found using six to ten propagation steps, which hints that this problem requires long-range information.
Revised: The best results were found using six to ten propagation steps, which hints that this problem requires long-range information.

Original: We added aGAT FA layer on top of the ﬁnal node representations, and used the resulting representations for node classiﬁcation.
Revised: We added a FA layer on top of the ﬁnal node representations, and used the resulting representations for node classiﬁcation.

Original: We used the same best found conﬁgurations as Brockschmidt (2020) add re-trained the modiﬁed models ﬁve times using different random seeds.
Revised: We used the same best found conﬁgurations as Brockschmidt (2020) add re-trained the modiﬁed models ﬁve times using different random seeds.

Original: Results Results are shown in Table 3.
Revised: Results Results are shown in Table 3.

Original: The main result is that adding an FA layer to most GNNs improves their SeenProjTest accuracy, obtaining a new state-of-the-art of 88.8%.
Revised: The main result is that adding an FA layer to all GNNs improves their SeenProjTest accuracy, obtaining a new state-of-the-art of 88.4%.

Original: in the Unseen ProjTest set, adding an an FA layer improves the results of some of  the GNNs. These improvements are signiﬁcant, since they were achieved on extensively-tuned models, without further tuning. 
Revised: In the Unseen ProjTest set, adding an an FA layer improves the results of some of most of the GNNs, obtaining a new state-ofthe-art of 83.8% These improvements are signiﬁcant, since they were achieved on extensively-tuned models, without further tuning. [Table]

Original:  C OMBINATORIAL A NALYSIS
Revised:  C OMBINATORIAL A NALYSIS

Original: In this section, we analyze the bottleneck combinatorially in the T REE -N EIGHBORS M ATCH problem.
Revised: In this section, we analyze the bottleneck combinatorially in the T REE -N EIGHBORS M ATCH problem.

Original: We provide a combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .
Revised: We provide a combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .

Original: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN h ( k ) v , as d .
Revised: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN h ( k ) v , as d .

Original: To have enough bits at the location of the bottleneck and perfectly ﬁt the training data, we must satisfy:
Revised: To have enough bits at the location of the bottleneck and perfectly ﬁt the training data, we must satisfy:

Original: [Equation]
Revised: [Equation]

Original: We provide the complete analysis in Appendix E.
Revised: We provide the complete analysis in Appendix E.

Original: Since factorial grows faster than an exponent with a constant base, we can see that a small increase of depth requires a much larger increase in d .
Revised: Since factorial grows faster than an exponent with a constant base, we can see that a small increase of depth requires a much larger increase in d .

Original: Speciﬁcally, it means that for d =32 as in the experiments in Section 4.1, the combinatorial upper bound of the model is as low as depth =7 .
Revised: Speciﬁcally, it means that for d =32 as in the experiments in Section 4.1, the combinatorial upper bound of the model is as low as depth =7 .

Original: That is, a model with d =32 cannot obtain 100% accuracy for depth ≥ 8 .
Revised: That is, a model with d =32 cannot obtain 100% accuracy for depth ≥ 8 .

Original: In practice, the problem is worse; i.e., the empirical upper bound is lower, because even if a solution to storing some information in a vector of a certain size exists, it is not guaranteed that a gradient descent-based algorithm will ﬁnd it.
Revised: In practice, the problem is worse; i.e., the empirical upper bound is lower, because even if a solution to storing some information in a vector of a certain size exists, it is not guaranteed that a gradient descent-based algorithm will ﬁnd it.

Original: Figure 4 shows the combinatorial upper bounds for depth given d ∈ { 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } .
Revised: Figure 4 shows the combinatorial upper bounds for depth given d ∈ { 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } .

Original: We repeated the experiments from Section 4.1 and report the max empirical depth for each value of d .
Revised: We repeated the experiments from Section 4.1 and report the max empirical depth for each value of d .

Original: As shown in Figure 4, even with d =512 , the combinatorial upper bound is as low as depth =10 .
Revised: As shown in Figure 4, even with d =512 , the combinatorial upper bound is as low as depth =10 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Under-reaching Although GNNs can be as powerful as the Weisfeiler-Lehman graph isomorphism test (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019), their expressiveness captures only a small fragment of ﬁrst-order logic (Barceló et al., 2020); the main limitation arises from the inability of a node to be aware of nodes that are farther away than the number of layers K , while the existence of such nodes can be easily described using logic.
Revised: Under-reaching Although GNNs can be as powerful as the Weisfeiler-Lehman graph isomorphism test (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019), their expressiveness captures only a small fragment of ﬁrst-order logic (Barceló et al., 2020); the main limitation arises from the inability of a node to be aware of nodes that are farther away than the number of layers K , while the existence of such nodes can be easily described using logic.

Original: We denote this limitation as under-reaching .
Revised: We denote this limitation as under-reaching .

Original: The over-squashing limitation described in this paper is tighter : even when information is reachable within
Revised: The over-squashing limitation described in this paper is tighter : even when information is reachable within

Original: K edges, this information might fail to ﬂow in the bottleneck (as we demonstrate in Section 4.2).
Revised: K edges, this information might fail to ﬂow in the bottleneck (as we demonstrate in Section 4.2).

Original: Over-smoothing The over-smoothing phenomenon is related, but not identical, to over-squashing: node representations become indistinguishable and prediction performance severely degrades when the number of layers increases (Li et al., 2018; Klicpera et al., 2019;
Revised: Over-smoothing The over-smoothing phenomenon is related, but not identical, to over-squashing: node representations become indistinguishable and prediction performance severely degrades when the number of layers increases (Li et al., 2018; Klicpera et al., 2019;

Original: Wu et al., 2020; Oono and
Revised: Wu et al., 2020; Oono and

Original: Suzuki, 2020).
Revised: Suzuki, 2020).

Original: Several approaches were proposed to mitigate over-smoothing, such as normalization (Zhao and Akoglu, 2020), edge-based dropout (Rong et al., 2020), and noise-reduction regularization (Chen et al., 2020).
Revised: Several approaches were proposed to mitigate over-smoothing, such as normalization (Zhao and Akoglu, 2020), edge-based dropout (Rong et al., 2020), and noise-reduction regularization (Chen et al., 2020).

Original: This might explain the empirical optimality of few layers (e.g., only two layers in
Revised: This might explain the empirical optimality of few layers (e.g., only two layers in

Original: Kipf and Welling (2017)); however, some problems depend on longer-range information propagation and thus require more layers, to avoid under-reaching .
Revised: Kipf and Welling (2017)); however, some problems depend on longer-range information propagation and thus require more layers, to avoid under-reaching .

Revised: Although over-smoothing and over-squashing are related, they are different problems, and there are cases where one occurs without the other.

Revised: For example, imagine a triangular graph containing only three nodes: A, B, and C, where every node has an edge to each of the other nodes.

Original: Both over-smoothing and  over-squashing are caused by using deep GNN for long-range problems, but their effect is different.
Revised: As we increase the number of layers, the representations of A, B and C might become indistinguishable, and thus suffer from over-smoothing . However, there will be no over-squashing in this case, because there is no growing amount of information that is squashed into ﬁxed-sized vectors while passing long-range messages. In the other direction, in the case of the T REE -N EIGHBORS M ATCH problem, there is no reason for over-smoothing to occur, because there are no two nodes that can converge to the same representation. A node in a “higher” level (Figure 5) contains twice the information than the node in the “lower” level.

Revised: Thus, this is a case where over-squashing can occur without over-smoothing .

Original: Avoiding over-squashing Gilmer et al.
Revised: Avoiding over-squashing Gilmer et al.

Original: (2017) add “virtual edges” to shorten long distances, and
Revised: (2017) add “virtual edges” to shorten long distances, and

Original: Scarselli et al.
Revised: Scarselli et al.

Original: (2008) add “supersource nodes”; however, these were mostly ad hoc solutions.
Revised: (2008) add “supersource nodes”; however, these were mostly ad hoc solutions.

Original: Nikolentzos et al.
Revised: Nikolentzos et al.

Original: (2019) update a node’s representation by recursively aggregating information from “k-hop” neighbors away at every layer in a computationally expensive approach, which forces k to be 2 or 3 at the most.
Revised: (2019) update a node’s representation by recursively aggregating information from “k-hop” neighbors away at every layer in a computationally expensive approach, which forces k to be 2 or 3 at the most.

Original: Allamanis et al.
Revised: Allamanis et al.

Original: (2018) designed program analyses that serve as 16 “shortcut” edge types; however, these analyses are speciﬁc to their problem and require a human domain expert.
Revised: (2018) designed program analyses that serve as 16 “shortcut” edge types; however, these analyses are speciﬁc to their problem and require a human domain expert.

Original: Although some previous work avoided over-squashing by various proﬁtable means, none of these works explicitly identiﬁed the bottleneck and its negative cross-domain implications.
Revised: Although some previous work avoided over-squashing by various proﬁtable means, none of these works explicitly identiﬁed the bottleneck and its negative cross-domain implications.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We highlight a new inherent problem that limits graph neural networks: a bottleneck that causes over-squashing.
Revised: We highlight a new inherent problem that limits graph neural networks: a bottleneck that causes over-squashing.

Original: Problems that depend on long-range interaction require as many GNN layers as the desired radius of each node’s receptive ﬁeld.
Revised: Problems that depend on long-range interaction require as many GNN layers as the desired radius of each node’s receptive ﬁeld.

Original: This causes an exponentially-growing amount of information to be squashed into a ﬁxed-length vector.
Revised: This causes an exponentially-growing amount of information to be squashed into a ﬁxed-length vector.

Original: As a result, the GNN fails to propagate long-range information, learns only short-range signals from the training data instead, and generalizes poorly when the prediction task depends on long-range interaction.
Revised: As a result, the GNN fails to propagate long-range information, learns only short-range signals from the training data instead, and generalizes poorly when the prediction task depends on long-range interaction.

Original: We demonstrate the existence of the bottleneck in a synthetic problem and show that GCN and GIN are more susceptible to over-squashing than other GNNs.
Revised: We demonstrate the existence of the bottleneck in a synthetic problem and show that GCN and GIN are more susceptible to over-squashing than other GNNs.

Original: We analyze this problem combinatorially and provide upper bounds for the tree depth.
Revised: We analyze this problem combinatorially and provide upper bounds for the tree depth.

Original: We further show that models of chemical, biological and programmatical benchmarks suffer from the over-squashing by showing that they can be dramatically improved using a simple FA layer.
Revised: We further show that models of chemical, biological and programmatical benchmarks suffer from the over-squashing by showing that they can be dramatically improved using a simple FA layer.

Original: We conclude that over-squashing in GNNs is so prevalent and untreated that in some benchmarks – that even the simplest solution helps.
Revised: We conclude that over-squashing in GNNs is so prevalent and untreated that in some benchmarks – that even the simplest solution helps.

Original: This simple solution opens the path for a variety of follow-up improvements and even better solutions for over-squashing.
Revised: This simple solution opens the path for a variety of follow-up improvements and even better solutions for over-squashing.

Original: A T REE -N EIGHBORS M ATCH – T RAINING D ETAILS
Revised: A T REE -N EIGHBORS M ATCH – T RAINING D ETAILS

Original: Data We created a separate dataset for every depth and sampled up to 32,000 examples per dataset.
Revised: Data We created a separate dataset for every depth and sampled up to 32,000 examples per dataset.

Original: The label of each leaf (“A”, “B”, “C” in Figure 2) is represented as a one-hot vector.
Revised: The label of each leaf (“A”, “B”, “C” in Figure 2) is represented as a one-hot vector.

Original: To tease the effect of the bottleneck from the ability of a GNN to count neighbors, we concatenated each leaf node’s initial representation with a 1-hot vector representing the number of blue neighbors, instead of creating the blue nodes.
Revised: To tease the effect of the bottleneck from the ability of a GNN to count neighbors, we concatenated each leaf node’s initial representation with a 1-hot vector representing the number of blue neighbors, instead of creating the blue nodes.

Original: The target node is initialized with an all-zeros vector as its (missing) label, concatenated with a 1-hot vector representing its number of blue neighbors.
Revised: The target node is initialized with an all-zeros vector as its (missing) label, concatenated with a 1-hot vector representing its number of blue neighbors.

Original: Intermediate nodes are initialized with a vector of zeros.
Revised: Intermediate nodes are initialized with a vector of zeros.

Original: Model The network has an initial linear layer, followed by depth + 1 GNN layers.
Revised: Model The network has an initial linear layer, followed by depth + 1 GNN layers.

Original: Afterward, the ﬁnal target node representation goes through a linear layer and a softmax to predict its label.
Revised: Afterward, the ﬁnal target node representation goes through a linear layer and a softmax to predict its label.

Original: We experimented with GCN (Kipf and Welling, 2017), GGNN (Li et al., 2016), GIN (Xu et al., 2019) and GAT (Veli ˇ ckovi ´ c et al., 2018) as the graph layers.[Table]
Revised: We experimented with GCN (Kipf and Welling, 2017), GGNN (Li et al., 2016), GIN (Xu et al., 2019) and GAT (Veliˇckovi´c et al., 2018) as the graph layers.

Original: In Section 4. [Table] we used model dimensions of d =32 .
Revised: In Section 4.1, we used model dimensions of d =32 .

Original: Larger values lead to the exact same trend.
Revised: Larger values lead to the exact same trend.

Original: We further discuss the theoretical and empirical aspects of the dimension in Section 5.
Revised: We further discuss the theoretical and empirical aspects of the dimension in Section 5.

Original: We added residual connections, summing every node with its own representation in the previous layer to increase expressivity [Table] and layer normalization which eased convergence.
Revised: We added residual connections, summing every node with its own representation in the previous layer to increase expressivity, and layer normalization which eased convergence.

Original: We used the Adam optimizer with a learning rate of 10 − 3 , decayed by 0 .
Revised: We used the Adam optimizer with a learning rate of 10 − 3 , decayed by 0 .

Original: 5 after every 1000 epochs without an increase in training accuracy, and stopped training after 2000 epochs of no training accuracy improvement.
Revised: 5 after every 1000 epochs without an increase in training accuracy, and stopped training after 2000 epochs of no training accuracy improvement.

Original: This usually lead to tens of thousands of training epochs, sometimes reaching 100,000 epochs.
Revised: This usually lead to tens of thousands of training epochs, sometimes reaching 100,000 epochs.

Original: To rule out hyperparameter tuning as the source of degraded performance
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  we experimented with changing activations (ReLu, tanh, MLP, none), using layer normalization and batch normalization, residual connections, various batch sizes [Table] and whether or not the same GNN weights should be “unrolled” over time steps.
Revised: To rule out hyperparameter tuning as the source of degraded performance, we experimented with changing activations (ReLu, tanh, MLP, none), using layer normalization and batch normalization, residual connections, various batch sizes, and whether or not the same GNN weights should be “unrolled” over time steps.

Original: The presented results were obtained using the conﬁgurations that achieved the best results. 
Revised: The presented results were obtained using the conﬁgurations that achieved the best results. [Table]

Original: B QM9 – A DDITIONAL R ESULTS 
Revised: B QM9 – A DDITIONAL R ESULTS B. A DDITIONAL GNN T YPES

Original: Because of space limitations, in Section 5.3 of the paper we presented results on the QM9 dataset only for GNN-MLP0, R-GAT and GNN-FiLM.
Revised: Because of space limitations, in Section 5.3 of the paper we presented results on the QM9 dataset only for GNN-MLP0, R-GAT and GNN-FiLM.

Original: In this section, we show that additional GNN architectures beneﬁt from breaking the bottleneck using a fully-adjacent layer: GGNN (Li et al., 2016), R-GCN (Schlichtkrull et al., 2018) and R-GIN (Xu et al., 2019).
Revised: In this section, we show that additional GNN architectures beneﬁt from breaking the bottleneck using a fully-adjacent layer: GGNN (Li et al., 2016), R-GCN (Schlichtkrull et al., 2018) and R-GIN (Xu et al., 2019).

Original: All experiments were performed using the extensively-tuned implementation of Brockschmidt (2020) who tuned a wide variety of hyperparameters.
Revised: All experiments were performed using the extensively-tuned implementation of Brockschmidt (2020) who tuned a wide variety of hyperparameters.

Original: For examples, they experimented with ReLU, GELU, leaky-ReLU and tanh as activations.
Revised: For examples, they experimented with ReLU, GELU, leaky-ReLU and tanh as activations.

Original: Table 4 contains additional results for GGNN, R-GCN and R-GIN.
Revised: Table 4 contains additional results for GGNN, R-GCN and R-GIN.

Original: As shown in Table 4, adding an
Revised: As shown in Table 4, adding an

Original: FA layer signiﬁcantly improves results across all GNN architectures, for all properties except for “mu” in R-GAT, where adding an FA layer results in a slightly higher error rate. 
Revised: FA layer signiﬁcantly improves results across all GNN architectures, for all properties except for “mu” in R-GAT, where adding an FA layer results in a slightly higher error rate. B. A LTERNATIVE S OLUTIONS

Original: Table 5 shows additional experiments, all performed using GCN.
Revised: Table 5 shows additional experiments, all performed using GCN.

Original: base † is the original model of
Revised: base † is the original model of

Original: Brockschmidt (2020) as in Table 4.
Revised: Brockschmidt (2020) as in Table 4.

Original: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.
Revised: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.

Original: 2 × d is a model that was trained with a doubled hidden dimension size, d
Revised: 2 × d is a model that was trained with a doubled hidden dimension size, d

Original: = 256 instead of d =as in the base model.
Revised: = 256 instead of d =as in the base model.

Original: As shown, doubling the hidden dimension size leads to a small improvement of only 5.5% reduction in error.
Revised: As shown, doubling the hidden dimension size leads to a small improvement of only 5.5% reduction in error.

Original: In comparison, the + FA model used the original dimension sizes and achieves a much larger improvement of 43.40%.
Revised: In comparison, the + FA model used the original dimension sizes and achieves a much larger improvement of 43.40%.

Original: [Table]

Original: All FA is a model that was trained with all GNN layers converted into FA layers, practically ignoring the graph topology.
Revised: All FA is a model that was trained with all GNN layers converted into FA layers, practically ignoring the graph topology.

Original: This lead to much worse results of more than 1500% higher error.
Revised: This lead to much worse results of more than 1500% higher error.

Original: This shows that the graph topology is important in this benchmark, and that a direct interaction between nodes (as in a single FA layer) must be performed in addition to considering the topology.
Revised: This shows that the graph topology is important in this benchmark, and that a direct interaction between nodes (as in a single FA layer) must be performed in addition to considering the topology.

Original: 2 × FA is a model where the last layer was modiﬁed into an FA layer, and an additional FA layer was stacked on top of it.
Revised: 2 × FA is a model where the last layer was modiﬁed into an FA layer, and an additional FA layer was stacked on top of it.

Original: This lead to results that are very similar to + FA.
Revised: This lead to results that are very similar to + FA.

Revised: Penultimate FA is a model where the FA layer is the penultimate layer (the K − 1 -th), followed by a standard GNN layer as the K -th layer.

Revised: This leads to results that are even slightly better than + FA.

Original: C B IOLOGICAL B ENCHMARKS – T RAINING D ETAILS
Revised: [Table] B. P ARTIAL -FA L AYERS

Revised: We also examined whether instead of adding a “full fully-adjacent layer”, we can randomly sample only a fraction of these edges.

Revised: We randomly sampled only { 0 . 25 , 0 . 5 , 0 .

Revised: 75 } of the edges in the full FA layer in every example, and trained the model for each target property 5 times.

Revised: Tableshows the results of these experiments using GCN.

Revised: base † is the original model of Brockschmidt (2020) as in Table 4.

Revised: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.

Revised: 75 }× FA are the models were only a fraction of the edges in the FA layer were used.

Revised: As shown in Table 6, the full FA layer achieves the largest reduction in error (-43.4%), but even a fraction of the edges improves the results over the base model.

Revised: For example, using only half of the edges ( 0.5 × FA ) reduces the error by 31.5%.

Revised: Overall, the percentage of used edges in the partial-FA layer is correlated with its reduction in error.

Revised: C B IOLOGICAL B ENCHMARKS – T RAINING D ETAILS

Original: We used the implementation of Errica et al.
Revised: We used the implementation of Errica et al.

Original: (2020) who performed a fair and thorough comparison between GNNs, by splitting each dataset to 10-folds; then, for each GNN type they select a conﬁguration among a grid of 72 conﬁgurations according to the validation set; ﬁnally, the best conﬁguration for each fold is trained three additional times, early stopped using the validation set, and evaluated on the test set.
Revised: (2020) who performed a fair and thorough comparison between GNNs, by splitting each dataset to 10-folds; then, for each GNN type they select a conﬁguration among a grid of 72 conﬁgurations according to the validation set; ﬁnally, the best conﬁguration for each fold is trained three additional times, early stopped using the validation set, and evaluated on the test set.

Original: The ﬁnal reported result is the average of all 30 test runs (10-folds × 3).
Revised: The ﬁnal reported result is the average of all 30 test runs (10-folds × 3).

Original: The ﬁnal standard deviation is computed among the average results of each of the ten folds.
Revised: The ﬁnal standard deviation is computed among the average results of each of the ten folds.

Original: D D ATA S TATISTICS 
Revised: D D ATA S TATISTICS D. S YNTHETIC D ATASET : T REE -N EIGHBORS M ATCH

Original: [Equation] [Table] [Equation]

Original: [Table] [Equation] Statistics of the synthetic T REE -N EIGHBORS M ATCH dataset are shown in Table 6.
Revised: Statistics of the synthetic T REE -N EIGHBORS M ATCH dataset are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: [-32, > 10 32, > 
-]
Revised: [-32, > 10 32, > 
-]

Original: D. Q UANTUM C HEMISTRY : QM 
Revised: D. Q UANTUM C HEMISTRY : QM

Original: Statistics of the quantum chemistry QM9 dataset, as used in Brockschmidt (2020) are shown in
Revised: Statistics of the quantum chemistry QM9 dataset, as used in Brockschmidt (2020) are shown in

Original: Table 7.
Revised: Table 8.

Original: [Table]

Original:  B IOLOGICAL B ENCHMARKS
Revised: D. B IOLOGICAL B ENCHMARKS

Original: Statistics of the biological datasets, as used in Errica et al.
Revised: Statistics of the biological datasets, as used in Errica et al.

Original: (2020), are shown in Table 8.
Revised: (2020), are shown in Table 9. [Table]

Original: [Table]
Revised: [Table]

Original:  ISUSE
Revised: D. V AR M ISUSE

Original: Statistics of the V AR M ISUSE dataset, as used in Allamanis et al.
Revised: Statistics of the V AR M ISUSE dataset, as used in Allamanis et al.

Original: (2018) and Brockschmidt (2020),
Revised: (2018) and Brockschmidt (2020),

Original: are shown in Table 9.
Revised: are shown in Table 10.

Original: [Table]
Revised: [Table]

Original: E C OMBINATORIAL A NALYSIS
Revised: E C OMBINATORIAL A NALYSIS

Original: We analyize the combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .
Revised: We analyize the combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .

Original: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN, i.e., the size of a node vector h ( k ) v , as d .
Revised: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN, i.e., the size of a node vector h ( k ) v , as d .

Original: A full tree of arity m has m depth leaves.
Revised: A full tree of arity m has m depth leaves.

Original: As described in Section 4.1, given an arrangement of blue neighbors, all possible permutations of the labels {A, B, C, ...} are valid.
Revised: As described in Section 4.1, given an arrangement of blue neighbors, all possible permutations of the labels {A, B, C, ...} are valid.

Original: Thus, the number of leaf label assignments is (cid:0) m depth (cid:1) ! .
Revised: Thus, the number of leaf label assignments is (cid:0) m depth (cid:1) ! .

Original: Right before interacting with the target node and predicting the label, a single vector of size d must encapsulate the information ﬂowing from all leaves (Equations (2) and (3)).
Revised: Right before interacting with the target node and predicting the label, a single vector of size d must encapsulate the information ﬂowing from all leaves (Equations (2) and (3)).

Original: 1 Such a vector contains d ﬂoating-point elements, each of them is stored as f bits.
Revised: 1 Such a vector contains d ﬂoating-point elements, each of them is stored as f bits.

Original: Overall, the number of possible cases that this vector can distinguish between is b f · d .
Revised: Overall, the number of possible cases that this vector can distinguish between is b f · d .

Original: The number of possible cases that the vector can distinguish between must be greater than the number of different examples this vector may encounter in the training data.
Revised: The number of possible cases that the vector can distinguish between must be greater than the number of different examples this vector may encounter in the training data.

Original: Thus, to have enough bits to perfectly ﬁt the training data at the location of the bottleneck, depth and d must satisfy Equation (6); considering binary trees ( m =2 ), and ﬂoating-point values of f = binary ( b =2 ) bits, we get Equation (7):
Revised: Thus, to have enough bits to perfectly ﬁt the training data at the location of the bottleneck, depth and d must satisfy Equation (6); considering binary trees ( m =2 ), and ﬂoating-point values of f = binary ( b =2 ) bits, we get Equation (7):

Original: [Equation]
Revised: [Equation]

Original: Now, we can ﬁx either d or depth and solve for the other.
Revised: Now, we can ﬁx either d or depth and solve for the other.

Original: For example, in Figure 4 we ﬁxed different values of d and found the maximal depth ∈ N that satisﬁes Equation (7), to get the combinatorial max depth for each value of d .
Revised: For example, in Figure 4 we ﬁxed different values of d and found the maximal depth ∈ N that satisﬁes Equation (7), to get the combinatorial max depth for each value of d .