Original: O N THE B OTTLENECK OF G RAPH N EURAL N ETWORKS AND ITS P RACTICAL I MPLICATIONS
Revised: O N THE B OTTLENECK OF G RAPH N EURAL N ETWORKS AND ITS P RACTICAL I MPLICATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNNs) were shown to effectively learn from highly structured data containing elements (nodes) with relationships (edges) between them.
Revised: Since the proposal of the graph neural network (GNN) by Gori et al. and

Original: GNN variants differ in how each node in the graph absorbs the information ﬂowing from its neighbor nodes.
Revised: Scarselli et al.

Original: In this paper, we highlight an inherent problem in GNNs: the mechanism of propagating information between neighbors creates a bottleneck when every node aggregates messages from its neighbors.
Revised: (2008) one of the major problems in training GNNs was their struggle to propagate information between distant nodes in the graph. We propose a new explanation for this problem: GNNs are susceptible to a bottleneck when aggregating messages across a long path.

Original: This bottleneck causes the over-squashing of exponentially-growing information into ﬁxed-size vectors.
Revised: This bottleneck causes the over-squashing of exponentially growing information into ﬁxed-size vectors.

Original: As a result, the graph fails to propagate messages ﬂowing from distant nodes and performs poorly when the prediction task depends on long-range information.
Revised: As a result, GNNs fail to propagate messages originating from distant nodes and perform poorly when the prediction task depends on long-range interaction.

Revised: In this paper, we highlight the inherent problem of over-squashing in GNNs.

Original: We demonstrate that the bottleneck hinders popular GNNs from ﬁtting long-range signals in the training data.
Revised: We demonstrate that the bottleneck hinders popular GNNs from ﬁtting long-range signals in the training data.

Original: We  show that GNNs that absorb incoming edges equally, like GCN and GIN, are more susceptible to over-squashing than other GNN types.
Revised: We further show that GNNs that absorb incoming edges equally, such as GCN and GIN, are more susceptible to over-squashing than GAT and GGNN.

Original: We further show that existing, extensively-tuned, GNN-based models of long-range problems suffer from over-squashing, and that breaking the bottleneck improves  state-of-the-art results without anyhyperparameter tuning or additional weights.
Revised: Finally, we show that prior work, which extensively tuned GNN models of long-range problems, suffer from over-squashing and that breaking the bottleneck improves their state-of-the-art results without any tuning or additional weights

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural networks (GNNs) (Scarselli et al., 2008; Micheli, 2009) have seen  growing popularity over the last few years (Duvenaud et al., 2015; Hamilton et al., 2017; Xu et al., 2019).
Revised: Graph neural networks (GNNs) (Gori et al., 2005; Scarselli et al., 2008; Micheli, 2009) have seen sharply growing popularity over the last few years (Duvenaud et al., 2015; Hamilton et al., 2017; Xu et al., 2019).

Original: Many domains can be naturally represented as graphs. Therefore, GNNs provide aconvenient and general  framework to modela variety of real-world complex structural data  such as social networks, knowledge graphs, computer programs,and chemical and biological systems.
Revised: GNNs provide a general and efﬁcient framework to model complex structural data containing elements (nodes) with relationships (edges) between them. A variety of real-world domains such as social networks, knowledge graphs, computer programs, chemical and biological systems can be naturally represented as graphs. Therefore, many graph-structured domains are commonly modeled using GNNs.

Original: A GNN layer can be viewed as a message-passing step (Gilmer et al., 2017), where each node updates its state by aggregating messages ﬂowing from its  neighbors.
Revised: A GNN layer can be viewed as a message-passing step (Gilmer et al., 2017), where each node updates its state by aggregating messages ﬂowing from its direct neighbors.

Original: GNN variants (Li et al., 2016;
Revised: GNN variants (Li et al., 2016;

Original: Veli ˇ ckovi ´ c et al., 2018; Kipf and Welling, 2017) mostly differ in how each node aggregates the representations of its neighborsand combines them with its own representation. 
Revised: Veliˇckovi´c et al., 2018; Kipf and Welling, 2017) mostly differ in how each node aggregates the representations of its neighbors with its own representation. However, most problems also require the interaction between nodes that are not directly connected, and they achieve this by stacking multiple GNN layers.

Original: In this paper, we show that this message-passing mechanism creates a numerical information bottleneck when computing neighbor aggregation.
Revised: To be solved, different learning problems require different ranges of interaction between nodes in the graph. We call this required range of interaction between nodes – the problem radius . In practice, GNNs were observed not to beneﬁt from more than few layers. The accepted explanation for this phenomenon is over-smoothing : node representations become indistinguishable when the number of layers increases (Wu et al., 2020).

Revised: Nonetheless, over-smoothing was mostly demonstrated in short-range tasks (Li et al., 2018; Klicpera et al., 2018;

Revised: Chen et al., 2020a; Oono and Suzuki, 2020; Zhao and Akoglu, 2020;

Revised: Rong et al., 2020; Chen et al., 2020b) – tasks that have small problem radii , where a node’s correct prediction mostly depends on its local neighborhood.

Revised: Such tasks include paper subject classiﬁcation (Sen et al., 2008) and product category classiﬁcation (Shchur et al., 2018).

Revised: Since the learning problems depend mostly on short-range information in these datasets, it makes sense why more layers than the problem radius might be extraneous.

Revised: In contrast, in tasks that also depend on long-range information (and thus have larger problem radii ), we hypothesize that the explanation for limited performance is over-squashing .

Revised: We further discuss the differences between over-squashing and over-smoothing in Section 6.

Revised: [Figure]

Revised: To allow a node to receive information from other nodes at a radius of K , the GNN needs to have at least K layers, or otherwise, it will suffer from under-reaching – these distant nodes will simply not be aware of each other.

Original: Problems that depend on long-range interaction between nodes must use as many GNN layers as the desired radius of a node’s receptive ﬁeld.
Revised: Clearly, to avoid under-reaching, problems that depend on long-range interaction between nodes require as many GNN layers as the range of the interaction.

Original: Unfortunately, the number of nodes in the receptive ﬁeld grows exponentially with the number of layers.
Revised: However, as the number of layers increases, the number of nodes in each node’s receptive ﬁeld grows exponentially .

Original: This causes over-squashing : information from the exponentially-growing receptive ﬁeld is compressed into ﬁxed-length  vectors.
Revised: This causes over-squashing : information from the exponentially-growing receptive ﬁeld is compressed into ﬁxed-length node vectors.

Original: Consequently, the graph fails to propagate messages ﬂowing from distant nodes; the model learns only short-range signals from the training data, and overall, generalizes poorly.
Revised: Consequently, the graph fails to propagate messages ﬂowing from distant nodes; the model learns only short-range signals from the training data; and overall, generalizes poorly at test time.

Original: In fact, the GNN bottleneck is analogous to the bottleneck of sequential recurrent models.
Revised: In fact, the GNN bottleneck is analogous to the bottleneck of sequential RNN models.

Original: Traditional seq2seq models (Sutskever et al., 2014; Cho et al., 2014a;b) suffered from a bottleneck at every decoder state – the model had to encapsulate the entire input sequence into a ﬁxed-size vector.In
Revised: Traditional seq2seq models (Sutskever et al., 2014; Cho et al., 2014a;b) suffered from a bottleneck at every decoder state – the model had to encapsulate the entire input sequence into a ﬁxed-size vector.

Original:  GNNs, the bottleneck is even more harmful, because the receptive ﬁeld of a node grows exponentially with the number of message propagation steps, rather than linearly as in recurrent models.
Revised: Thus in RNNs, the receptive ﬁeld of a node grows linearly with the number of recursive applications. However in GNNs, the bottleneck is asymptotically more harmful, because the receptive ﬁeld of a node grows exponentially .

Original: This difference is illustrated in Figure 1.
Revised: This difference is illustrated in Figure 1.

Original: This work does not aim to propose a new GNN variant.
Revised: This work does not aim to propose a new GNN variant.

Original: Rather, the main contribution of this work is highlighting the inherent bottleneck problem of  GNNs,  and this is the ﬁrst work to discuss the over-squashing problem.
Revised: Rather, the main contribution of this work is explaining the widely known problem of long-range information ﬂow in GNNs, by highlighting the inherent bottleneck and showing its harmful practical implications. This is the ﬁrst work to introduce the over-squashing phenomenon, which is a novel explanation for one of the major and well-known issues in training GNNs.

Original: We use a controlledsynthetic problem to demonstrate  the existence of a bottleneck and to provide combinatorial upper bounds for the graph size given the network’s hidden dimension (Section 5).
Revised: We use a controlled problem to demonstrate how over-squashing prevents GNNs from ﬁtting long-range patterns in the data, and to provide theoretical lower bounds for the required hidden size given the problem radius (Section 5).

Original: We show, analytically and empirically, thatGCN (Kipf and Welling, 2017) [Figure]
Revised: We show, analytically and empirically, that

Original:  and GIN (Xu et al., 2019) suffer from over-squashing more than other types of GNNs, and even in small graphs.
Revised: GCN (Kipf and Welling, 2017) and GIN (Xu et al., 2019) are susceptible to over-squashing more than other types of GNNs such as GAT (Veliˇckovi´c et al., 2018) and GGNN (Li et al., 2016).

Original: We further show that existing models of real-world datasets suffer from over-squashing: breaking the bottleneckrelatively reduces the error rate by 42% in the QM9 dataset, by 12% in ENZYMES,and by 4.8% in NCI1,  without any hyperparameter tuning nor additional weights.
Revised: We further show that prior work that extensively tuned GNNs to real-world datasets suffer from over-squashing: breaking the bottleneck reduces the error rate by 42% in the QM9 dataset, by 12% in ENZYMES, by 4.8% in NCI1, and improves accuracy in Variable Misuse, without any hyperparameter tuning nor additional weights.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A directed graph G =
Revised: A directed graph G =

Original: ( V , E ) contains nodes V and edges E ,
Revised: ( V , E ) contains nodes V and edges E ,

Original: where ( u, v ) ∈
Revised: where ( u, v ) ∈

Original: E denotes an edge from a node u to a node v .
Revised: E denotes an edge from a node u to a node v .

Original: For brevity, in the following deﬁnitions we treat all edges as having the same type ; in general, every edge can have a type and features (Schlichtkrull et al., 2018).
Revised: For brevity, in the following deﬁnitions we treat all edges as having the same type ; in general, every edge can have a type and features (Schlichtkrull et al., 2018).

Original: Graph neural networks Graph neural networks operate by propagating neural messages between neighboring nodes.
Revised: Graph neural networks Graph neural networks operate by propagating neural messages between neighboring nodes.

Original: At every propagation step (a graph layer): the network computes each node’s sent message; every node aggregates its received messages; and each node updates its representation by combining the aggregated incoming messages with its own previous representation. 
Revised: At every propagation step (a graph layer): the network computes each node’s sent message; every node aggregates its received messages; and each node updates its representation by combining the aggregated incoming messages with its own previous representation. [Figure]

Original:  Formally, each node is associated with an initial representation h (0) v ∈ R d 0 .
Revised: [Equation] Formally, each node is associated with an initial representation h (0) v ∈ R d 0 .

Original: This representation is usually derived from the node’s label or its given features.
Revised: This representation is usually derived from the node’s label or its given features.

Original: Then, a GNN layer updates each node’s representation given its neighbors, yielding h (1) v ∈ R d .
Revised: Then, a GNN layer updates each node’s representation given its neighbors, yielding h (1) v ∈ R d .

Original: In general, the k -th layer of a GNN is a parametric function f k that is applied to each node by considering its neighbors:
Revised: In general, the k -th layer of a GNN is a parametric function f k that is applied to each node by considering its neighbors:

Original: [Equation]
Revised: [Equation]

Original: where N v is the set of nodes that have edges to v : N v = { u ∈ V | ( u, v ) ∈ E} .
Revised: where N v is the set of nodes that have edges to v : N v = { u ∈ V | ( u, v ) ∈ E} .

Original: The total number of layers K is usually determined empirically as a hyperparameter.
Revised: The total number of layers K is usually determined empirically as a hyperparameter.

Original: The design of the function f is what mostly distinguishes one type of GNN from the other.
Revised: The design of the function f is what mostly distinguishes one type of GNN from the other.

Original: For example, graph convolutional networks (GCN)(Kipf and Welling, 2017) deﬁne f as:
Revised: For example, graph convolutional networks (GCN) deﬁne f as:

Original: [Equation]
Revised: [Equation]

Original: where σ is a nonlinearity such as ReLU , and c u,v is a normalization factor often set to (cid:112)
Revised: where σ is a nonlinearity such as ReLU , and c u,v is a normalization factor often set to (cid:112)

Original: |N v | · |N u | or |N v | (Hamilton et al., 2017).
Revised: |N v | · |N u | or |N v | (Hamilton et al., 2017).

Original: As another example, graph isomorphism networks (GIN) (Xu et al., 2019) update a node’s representation using the following deﬁnition:
Revised: As another example, graph isomorphism networks (GIN) (Xu et al., 2019) update a node’s representation using the following deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: Usually, the last ( K -th) layer’s output is used for prediction: in node-prediction tasks, h ( K ) v is used to predict a label for v ; in graph-prediction tasks, a permutation-invariant “readout” function aggregates the nodes of the ﬁnal layer using summation, averaging, or a weighted sum (Li et al., 2016).[Figure]
Revised: Usually, the last ( K -th) layer’s output is used for prediction: in node-prediction, h ( K ) v is used to predict a label for v ; in graph-prediction, a permutation-invariant “readout” function aggregates the nodes of the ﬁnal layer using summation, averaging, or a weighted sum (Li et al., 2016).

Original:  T HE GNN B OTTLENECK
Revised:  T HE GNN B OTTLENECK

Revised: Given a graph G =

Revised: ( V , E ) and a given node v , we denote the problem’s required range of interaction, the problem radius , by r .

Revised: r is generally unknown in advance, and usually approximated empirically by tuning the number of layers K .

Revised: We denote the set of nodes in the receptive ﬁeld of v by N Kv , which is deﬁned recursively as N 1 v :=

Revised: N v and N Kv :=

Original:  When a prediction problem relies on long-range interaction between nodes, the GNN is required to have as many layers  as the estimated length of these interactions.
Revised: N K − 1 v [Equation] [Equation] When a prediction problem relies on long-range interaction between nodes, the GNN must have as many layers K as the estimated range of these interactions, or otherwise, these distant nodes would not be able to interact.

Revised: It is thus required that K ≥ r .

Original: However, the number of nodes in the receptive ﬁeldof a node grows exponentially with the number of layers.
Revised: However, the number of nodes in each node’s receptive ﬁeld grows exponentially with the number of layers: (cid:12)(cid:12) N Kv (cid:12)(cid:12) = O (exp ( K ))

Original:  As a result, an exponentially-growing amount of information is squashed into a ﬁxed-length vector (i.e., the result of the (cid:80) in Equations (2) and (3)), and crucial messages fail to reach their distant destinations.
Revised: (Chen et al., 2018). As a result, an exponentially-growing amount of information is squashed into a ﬁxed-length vector (the vector resulting from the (cid:80) in Equations (2) and (3)), and crucial messages fail to reach their distant destinations.

Original: Instead, the model may learn only short-ranged signals and consequently generalize poorly at test time.
Revised: Instead, the model might learn only short-ranged signals and consequently generalize poorly at test time.

Original: For example, consider the N EIGHBORS M ATCH problem of Figure 2.
Revised: Example Consider the N EIGHBORS M ATCH problem of Figure 2.

Original: Green nodes ( A , B , C ) have a varying number of blue neighbors ( A ) and an alphabetical label.
Revised: Green nodes ( A , B , C ) have a varying number of blue neighbors ( A ) and an alphabetical label.

Original: Every example in the dataset  has a different mapping from numbers of neighbors to labels.
Revised: Each example in the dataset is a different graph that has a different mapping from numbers of neighbors to labels.

Original: The rest of the graph (marked as ) represents a general, unknown, graph structure.
Revised: The rest of the graph (marked as ) represents a general, unknown, graph structure.

Original: The goal is to predict a label for the target node, which is marked with a question mark ( ? ), according to its number of blue neighbors.
Revised: The goal is to predict a label for the target node, which is marked with a question mark ( ? ), according to its number of blue neighbors.

Original: The correct answer is C in this case, because the target node has two blue neighbors, like the node marked with C in the same graph.
Revised: The correct answer is C in this case, because the target node has two blue neighbors, like the node marked with C in the same graph.

Original: Since every example in the dataset has a different mapping from numbers of neighbors to labels, this long-range message propagation and matching between the target node and all the green nodes  be performed for every example.
Revised: Every example in the dataset has a different mapping from numbers of neighbors to labels, and thus the message propagation and matching between the target node and all the green nodes must be performed for every graph in the dataset.

Original: Since the model must propagate information from all green nodes before predicting the label, a bottleneck at the target node is inevitable.
Revised: Since the model must propagate information from all green nodes before predicting the label, a bottleneck at the target node is inevitable.

Original: This bottleneck causes over-squashing , which can prevent the model from ﬁtting the training data perfectly, even though the desired prediction is obvious in a global view.
Revised: This bottleneck causes over-squashing , which can prevent the model from ﬁtting the training data perfectly, even though the desired prediction is obvious in a global view.

Original: We demonstrate the bottleneck empirically in an instance of this problem in Section 4; in Section 5, we provide upper bounds for the learnable graph size.
Revised: We demonstrate the bottleneck empirically in an instance of this problem in Section 4; in Section 5, we provide theoretical lower bounds for the GNN’s hidden size.

Original: Although this is a contrived problem, it resembles real-world problems that are often modeled as graphs.
Revised: Although this is a contrived problem, it resembles real-world problems that are often modeled as graphs.

Original: For example, a computer program in a language such as Python may declare multiple variables (i.e., the green nodes in Figure 2) along with their types and values (their numbers of blue neighbors in Figure 2); later in the program, predicting which variable should be used in a speciﬁc location (predict the alphabetical label in Figure 2) must use one of theavailable variables  based on the required type and the required value at that point.
Revised: For example, a computer program in a language such as Python may declare multiple variables (i.e., the green nodes in Figure 2) along with their types and values (their numbers of blue neighbors in Figure 2); later in the program, predicting which variable should be used in a speciﬁc location (predict the alphabetical label in Figure 2) must use one of the variables that are available in scope based on the required type and the required value at that point.

Original: We experiment with this  problem in Section 4.4.
Revised: We experiment with this VarMisuse problem in Section 4.4.

Original: Short- vs. long-range problems Much of prior GNN work has focused on problems that were local in nature,  where the underlying inductive bias was that a node’s most relevant context is its local neighborhood, and long-range interaction was not necessarily needed.
Revised: Short- vs. long-range problems Much of prior GNN work has focused on problems that were local in nature, with small problem radii, where the underlying inductive bias was that a node’s most relevant context is its local neighborhood, and long-range interaction was not necessarily needed.

Original: With the growing popularity of GNNs, their adoption expanded to domains that required longer-range information propagation as well, without addressing the inherent bottleneck.
Revised: With the growing popularity of GNNs, their adoption expanded to domains that required longer-range information propagation as well, without addressing the inherent bottleneck.

Original: In this paper, we focus on problems that require long-range information.
Revised: In this paper, we focus on problems that require long-range information.

Original: That is, a correct prediction requires considering the local environment of a node and interactions beyond the close neighborhood.
Revised: That is, a correct prediction requires considering the local environment of a node and interactions beyond the close neighborhood.

Original: For example, a chemical property of a molecule (Ramakrishnan et al., 2014; Gilmer et al., 2017) can depend on the combination of atoms that reside in the molecule’s opposite sides .
Revised: For example, a chemical property of a molecule (Ramakrishnan et al., 2014; Gilmer et al., 2017) can depend on the combination of atoms that reside in the molecule’s opposite sides .

Original: Problems of this kind require long-range interaction, and[Figure] thus, a large number of GNN layers.
Revised: Problems of this kind require long-range interaction, and thus, a large number of GNN layers.

Original: Since the receptive ﬁeld of each node grows exponentially with the number of layers, the more layers – the more harmful the effect of the bottleneck.
Revised: Since the receptive ﬁeld of each node grows exponentially with the number of layers, the more layers – over-squashing is more harmful.

Original: In problems that are local in nature  – the bottleneck is less troublesome, because a GNN can perform well with only few layers (e.g.,  2 layers in Kipf and Welling (2017)), and the receptive ﬁeld of a node can be exponentially smaller.
Revised: In problems that are local in nature (small r ) – the bottleneck is less troublesome, because a GNN can perform well with only few layers (e.g., K = 2 layers in Kipf and Welling (2017)), and the receptive ﬁeld of a node can be exponentially smaller.

Original: Domains such as citation networks (Sen et al., 2008), social networks (Leskovec and Mcauley, 2012),  and product recommendations (Shchur et al., 2018) usually raise short-range problems and are thus not the focus of this paper. 
Revised: Domains such as citation networks (Sen et al., 2008), social networks (Leskovec and Mcauley, 2012), academic co-authorship, and product recommendations (Shchur et al., 2018) usually raise short-range problems and are thus not the focus of this paper. So, how long is long-range? We discuss and analyze this question in Section 5.

Original:  E VALUATION
Revised:  E VALUATION

Original: First, we wish to empirically show that the GNN bottleneck exists, even in small graphs.
Revised: First, we wish to empirically show that the GNN bottleneck exists, and ﬁnd the smallest value of r that causes over-squashing.

Original: We generated a synthetic benchmark that is theoretically solvable; however, in practice, all GNNs fail to reach 100% training accuracy because of the bottleneck (Section 4.1).
Revised: We generated a synthetic benchmark that is theoretically solvable; however, in practice, all GNNs fail to reach 100% training accuracy because of the bottleneck (Section 4.1).

Original: Second, we examine whether the bottleneck exists in prior models, which addressed real-world problems (Sections 4.2 to 4.4).
Revised: Second, we examine whether the bottleneck exists in prior work, which addressed real-world problems (Sections 4.2 to 4.4).

Original: To that end, we wish to measure over-squashing in existing models.
Revised: To that end, we wish to measure over-squashing in existing models.

Original: But, how can we measure over-squashing?
Revised: But, how can we measure over-squashing?

Original: Instead, we measure whether breaking the bottleneck improves the results of long-range problems.
Revised: Instead, we measure whether breaking the bottleneck improves the results of long-range problems.

Original: Adding a fully-adjacent layer (FA)
Revised: Adding a fully-adjacent layer (FA)

Original: We took off-the-shelf, extensively tuned, models, and modiﬁed adjacency in the last layer by modifying the authors’ original code: given a GNN with K layers, we modiﬁed the K -th layer to be a fully-adjacent layer .
Revised: In Sections 4.2 to 4.4, we took off-the-shelf, extensively tuned, models, and modiﬁed adjacency in the last layer: given a GNN with K layers, we modiﬁed the K -th layer to be a fully-adjacent layer (FA).

Original: A fully-adjacent layer is a GNN layer in which every pair of nodes is connected by an edge.
Revised: A fully-adjacent layer is a GNN layer in which every pair of nodes is connected by an edge.

Original: In terms of Equations (1) to (3), converting an existing layer to be fully-adjacent means that N v :=
Revised: In terms of Equations (1) to (3), converting an existing layer to be fully-adjacent means that N v :=

Original: V for every node v ∈ V , only in that layer.
Revised: V for every node v ∈ V , only in that layer.

Original: This does not change the type of layer nor add weights, but only changes the notion of adjacency of a data sample in a single layer.
Revised: This does not change the type of layer nor add weights, but only changes the notion of adjacency of a data sample in a single layer.

Original: Thus, the K − 1 graph layers exploit the graph structure using their original sparse topology, and only the K -th layer is an FA layer that allows the topology-aware node-representations to interact directly and consider nodes beyond their original neighbors.
Revised: Thus, the K − 1 graph layers exploit the graph structure using their original sparse topology, and only the K -th layer is an FA layer that allows the topology-aware node-representations to interact directly and consider nodes beyond their original neighbors.

Original: Hopefully, this would ease information ﬂow, prevent over-squashing, and reduce the effect of the previously-existed bottleneck.
Revised: Hopefully, this would ease information ﬂow, prevent over-squashing, and reduce the effect of the previously-existed bottleneck.

Original: We then re-trained the models  without performing any additional tuning, to rule out hyperparameter tuning as the source of improvement.
Revised: We then re-trained the models using the authors’ original code, without performing any additional tuning, to rule out hyperparameter tuning as the source of improvement.

Original: Statistics of all datasets can be found in Appendix D.
Revised: Statistics of all datasets can be found in Appendix D.

Original: We emphasize that an FA layer is a simple solution.
Revised: An FA layer is a simple solution.

Original: Its purpose is  to demonstrate that over-squashing in GNNs is so prevalent and untreated that even the simplest solution helps .
Revised: Its purpose is merely to demonstrate that over-squashing in GNNs is so prevalent and untreated that even the simplest solution helps .

Revised: The main contribution of our paper is not the solution, but rather, highlighting and explaining the over-squashing problem .

Original: This simple solution opens the path for a variety of follow-up improvements and better solutions for over-squashing.
Revised: This simple solution opens the path for a variety of follow-up improvements and better solutions for over-squashing.

Original: 4. S YNTHETIC B ENCHMARK : N EIGHBORS M ATCH
Revised: 4. S YNTHETIC B ENCHMARK : N EIGHBORS M ATCH

Original:  The N EIGHBORS M ATCH problem (Figure 2) is a simple contrived problem that we designed to provide an intuition to the extend of the effect of over-squashing, while allowing us to control the  intensity of over-squashing.
Revised: [Figure] The N EIGHBORS M ATCH problem (Figure 2) is a simple contrived problem that we designed to provide an intuition to the extent of the effect of over-squashing, while allowing us to control the problem radius r , and thus control the intensity of over-squashing.

Original: We focus on the training accuracy of a model.
Revised: We focus on the training accuracy of a model, and show that the bottleneck can prevent models from ﬁtting the training set.

Original: T REE -N EIGHBORS M ATCH
Revised: T REE -N EIGHBORS M ATCH

Original: We created an instance of the general N EIGHBORS M ATCH problem that we described in Section 3 and portrayed in Figure 2. As observed before (Micheli, 2009; Chen et al., 2018), the receptive ﬁeld of a node in a graph grows exponentially with the number of layers. Thus, from the perspective of a single node v , the rest of the graph may look like a tree, rooted at v (Garg et al., 2020). 
Revised: From the perspective of a single node v , the rest of the graph may look like a tree of height K , rooted at v (Xu et al., 2018; Garg et al., 2020). To simulate this exponentially-growing receptive ﬁeld, we created an instance of the general N EIGHBORS M ATCH problem that we described in

Original: To simulate this exponentially-growing receptive ﬁeld, we instantiated the subgraph in the middle of the graph (marked as in Figure 2) as a binary tree of depth depth where the green nodes are its leaves, and the target node is the tree’s root.
Revised: Section 3 and portrayed in Figure 2. We instantiated the subgraph in the middle of the graph (marked as in Figure 2) as a binary tree of depth depth where the green nodes are its leaves, and the target node is the tree’s root.

Original: All edges are directed toward the root, such that information is propagated from all nodes toward the target node.
Revised: All edges are directed toward the root, such that information is propagated from all nodes toward the target node.

Original: The goal, as in Section 3, is to predict a label for the target node, where the correct answer is the label of the green node that has the same number of blue neighbors as the target node.
Revised: The goal, as in Section 3, is to predict a label for the target node, where the correct answer is the label of the green node that has the same number of blue neighbors as the target node.

Original: An illustration is shown in Figure 5 in the appendix.
Revised: An illustration is shown in Figure 5 in the appendix.

Original:  In this section we observe the bottleneck empirically; in Section 5 we provide a combinatorial upper bound for the learnable graph size in this problem.
Revised: This allows us to control the problem radius, i.e., r = depth . In this section we observe the bottleneck empirically; in Section 5 we provide lower bound for the GNN’s hidden size given r .

Original: Model We implemented a network with depth +1 graph layers to allow an additional nonlinear layer after the information from the leaves reaches the target node.
Revised: Model We implemented a network with r +1 graph layers to allow an additional nonlinearity after the information from the leaves reaches the target node.

Original: Our PyTorch Geometric (Fey and Lenssen, 2019) implementation is included in the supplementary material and we make it publicly available.
Revised: Our PyTorch Geometric (Fey and Lenssen, 2019) implementation is available at https://github.com/tech-srl/bottleneck/ .

Original: Our training conﬁguration and hyperparameter ranges are detailed in Appendix A.
Revised: Our training conﬁguration and hyperparameter ranges are detailed in Appendix A.

Original: Results Figure 3 shows the following surprising results: GNNs fail to ﬁt the dataset starting from depth =4 .
Revised: Results Figure 3 shows the following surprising results: GNNs fail to ﬁt the dataset starting from r =4 .

Original: For example, the training accuracy of GCN (Kipf and Welling, 2017) at depth =4 is 70%.
Revised: For example, the training accuracy of GCN (Kipf and Welling, 2017) at r =4 is 70%.

Original: At depth =5 , all GNNs fail to perfectly ﬁt the data.
Revised: At r =5 , all GNNs fail to perfectly ﬁt the data.

Original: Starting from depth =4 , the models suffered from over-squashing that resulted in underﬁtting : the bottleneck prevented the models from distinguishing between different training examples, even after they were observed tens of thousands of times.
Revised: Starting from r =4 , the models suffered from over-squashing that resulted in underﬁtting : the bottleneck prevented the models from distinguishing between different training examples, even after they were observed tens of thousands of times.

Original: These results clearly show the existence of the bottleneck and its negative effect, even in small graphs.
Revised: These results clearly show the existence of over-squashing, starting from r =4 .

Original: Discussion GCN and GIN(Xu et al., 2019) managed to perfectly ﬁt depth =3 at most, while GGNN(Li et al., 2016) and GAT(Veli ˇ ckovi ´ c et al., 2018) also reached 100% accuracy at depth =4 .
Revised: Why did some GNNs perform better than others? GCN and GIN managed to perfectly ﬁt r =3 at most, while GGNN and GAT also reached 100% accuracy at r =4 .

Original: This difference can be explained by their neighbor aggregation computation: consider the target node that receives messages in the depth ’th step.
Revised: This difference can be explained by their neighbor aggregation computation: consider the target node that receives messages in the r ’th step.

Original: GCN and GIN aggregate all neighbors before combining them with the target node’s representation, and thus they  must compress the information ﬂowing from all leaves into a single vector, and only afterward interact with the target node’s own representation (Equations (2) and (3)).
Revised: GCN and GIN aggregate all neighbors before combining them with the target node’s [Table] representation; they thus must compress the information ﬂowing from all leaves into a single vector; and only afterward interact with the target node’s own representation (Equations (2) and (3)).

Original: In contrast,a GATlayer uses the target’s own representation to weight incoming messages.
Revised: In contrast, GAT uses attention to weight incoming messages given the target’s representation.

Original: The target node can thus  ignore the irrelevant incoming edge(by giving them very small attention weight) and absorb mostly the relevant incoming edge, which contains information ﬂowing from half of the leaves.
Revised: The target node can thus (softly) ignore the irrelevant incoming edge and absorb mostly the relevant incoming edge, which contains information ﬂowing from half of the leaves.

Original: Following Levy et al.
Revised: Following Levy et al.

Original: (2018), we hypothesize that the GRU cell in GGNNs ﬁlters incoming edges as GAT, but perform this ﬁltering as element-wise attention.
Revised: (2018), we hypothesize that the GRU cell in GGNNs ﬁlters incoming edges as GAT, but perform this ﬁltering as element-wise attention.

Original: Since the number of leaves grows exponentially with depth , it is expected that GNNs that need to compress only half of the information (GGNN and GAT) will succeed at a depth that is larger by 1.
Revised: Since the number of leaves grows exponentially with r , it is expected that GNNs that need to compress only half of the information (GGNN and GAT) will succeed at an r that is larger by 1.

Original: If all GNNs have reached low training accuracy, how do GNN-based models usually do ﬁt the training data in public datasets of long-range problems?
Revised: If all GNNs have reached low training accuracy, how do GNN-based models usually do ﬁt the training data in public datasets of long-range problems?

Original: We hypothesize that they learn only short-range signals from the training set, rather than learning the information that was squashed in the bottleneck.
Revised: We hypothesize that they learn only short-range signals from the training set, rather than learning the information that was squashed in the bottleneck, and thus generalize poorly at test time.

Original: Over-squashing or just long-range?
Revised: Over-squashing or just long-range?

Original: To rule out the possibility that the long-range itself is preventing the GNNs from ﬁtting the data, we repeated the experiment of Figure 3 for depth s 4 to 8, where the distance between the leaves and the target node remained the same (4 to 8), but the amount of information was as in depth =3 .
Revised: To rule out the possibility that the long-range itself is preventing the GNNs from ﬁtting the data, we repeated the experiment of Figure 3 for depths 4 to 8, where the distance between the leaves and the target node remained the same (4 to 8), but the amount of over-squashing was as in r =2 .

Original: In other words, the graph looks like a tree of depth =3 , where the root is connected to a “chain” of length of up to 5, and the target node is at the other side of the “chain”.
Revised: That is, the graph looks like a tree of depth =2 , where the root is connected to a “chain” of length of up to 6, and the target node is at the other side of the chain.

Original: This setting maintains the long-range as in the original problem, but reduces the amount of information that needs to be squashed.
Revised: This setting maintains the long-range as in the original problem, but reduces the amount of information that needs to be squashed.

Original: In other words, this setting disentangles of the effect of the long-range itself from the effect of the growing amount of information (i.e., from over-squashing).
Revised: In other words, this setting disentangles of the effect of the long-range itself from the effect of the growing amount of information (i.e., from over-squashing).

Original: In this setting, all GNN types managed to easily ﬁt the data to close to 100% across all distances, showing that the problem is not the long-range itself, but rather, thegrowing amount of information (i.e., over-squashing).
Revised: In this setting, all GNN types managed to easily ﬁt the data to close to 100% across all distances, showing that the problem is not the long-range itself, but rather, the amount of over-squashing.

Original: 4. Q UANTUM C HEMISTRY : QM Data
Revised: 4. Q UANTUM C HEMISTRY : QM Data

Original: The QM9 dataset (Ramakrishnan et al., 2014; Gilmer et al., 2017; Wu et al., 2018) contains ~130,000 graphs with ~18 nodes.
Revised: The QM9 dataset (Ramakrishnan et al., 2014; Gilmer et al., 2017; Wu et al., 2018) contains ~130,000 graphs with ~18 nodes.

Original: Each graph is a molecule where nodes are atoms, and undirected, typed edges are different types of bonds between the atoms.
Revised: Each graph is a molecule where nodes are atoms, and undirected, typed edges are different types of bonds between the atoms.

Original: The goal is to regress each graph toreal-valued quantum chemical properties such as dipole moment and isotropic polarizability .
Revised: The goal is to regress each graph toreal-valued quantum chemical properties such as dipole moment and isotropic polarizability .

Original: [Table] Models We used the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning for multiple GNNs, by searching over 500 conﬁgurations; we took the same splits and their best-found conﬁgurations.
Revised: Models We modiﬁed the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning for multiple GNNs, by searching over 500 conﬁgurations; we took the same splits and their best-found conﬁgurations.

Original: For most GNNs, Brockschmidt found that the best results are achieved using eight propagation steps.
Revised: For most GNNs, Brockschmidt found that the best results are achieved using K =8 layers.

Original: This hints that this problem depends on long-range information and relies on both graph structure and distant nodes.
Revised: This hints that this problem depends on long-range information and relies on both graph structure and distant nodes.

Original: We modiﬁed the last layer to be an FA layer by extending their implementation.

Original: We re-trained each modiﬁed model for each target property using the same code, conﬁguration [Table] and training scheme as Brockschmidt (2020), training each model ﬁve times (using different random seeds) for each target property task.
Revised: We re-trained each modiﬁed model for each target property using the same code, conﬁguration, and training scheme as Brockschmidt (2020), training each model ﬁve times (using different random seeds) for each target property task.

Original: We compare the “base” models, reported by Brockschmidt, with our modiﬁed and re-trained “ + FA” models.
Revised: We compare the “base” models, reported by Brockschmidt, with our modiﬁed and re-trained “ + FA” models.

Original: Results Results for the top GNNs are shown in Table 1; results for the other GNNs are shown in
Revised: Results Results for the top GNNs are shown in Table 1; results for the other GNNs are shown in

Original: Appendix B due to space limitation.
Revised: Appendix B due to space limitation.

Original: The main results are that breaking the bottleneck by modifying a single layer to be an FA layer signiﬁcantly reduces the error rate, by 42% on average, across six
Revised: The main results are that breaking the bottleneck by modifying a single layer to be an FA layer signiﬁcantly reduces the error rate, by 42% on average, across six

Original: GNN types.
Revised: GNN types.

Original: These experiments clearly show evidence for a bottleneck in the original GNN models.
Revised: These experiments clearly show evidence for a bottleneck in the original GNN models.

Original: Can larger hidden dimension achieve a similar improvement?

Original: We trained another set of models with doubled dimension sizes (Appendix B).

Original: These models achieved only 5.5% improvement over the base model, while adding the FA layer achieved 42% improvement using the original dimensions.

Original: If all GNNs beneﬁt from direct interaction between all nodes, maybe the graph structure is not even needed?

Original: We trained another set of models (Appendix B) where all layers are FA layers, thus ignoring the original graph structure; these models produced 1500% worse error.

Original: Over-squashing or under-reaching?
Revised: Over-squashing or under-reaching?

Original: Barceló et al.
Revised: Barceló et al.

Original: (2020) discuss the inability of a GNN node to observe nodes that are farther away than the number of layers K .
Revised: (2020) discuss the inability of a GNN node to observe nodes that are farther away than the number of layers K .

Original: We denote this limitation as underreaching : for every ﬁxed number of layers K , local information cannot travel farther than distance K along edges.
Revised: We denote this limitation as underreaching : for every ﬁxed number of layers K , local information cannot travel farther than distance K along edges.

Original: So, was the improvement of the FA layer in Table 1 achieved thanks to the reduction in over-squashing, or did the FA layer only extend the nodes’ reachability and prevent under-reaching?
Revised: So, was the improvement of the FA layer in Table 1 achieved thanks to the reduction in over-squashing, or did the FA layer only extend the nodes’ reachability and prevent under-reaching?

Original: To answer this, we measured the graphs’ diameter in the QM9 dataset – the maximum shortest path between two nodes in a graph.
Revised: To answer this, we measured the graphs’ diameter in the QM9 dataset – the maximum shortest path between two nodes in a graph.

Original: We found that the average diameter is 6 .
Revised: We found that the average diameter is 6 .

Original: 35 ± 0 .
Revised: 35 ± 0 .

Original: 91 , the maximum diameter is 10 , and the 90th percentile is 8, while most models were trained with 8 layers.
Revised: 91 , the maximum diameter is 10 , and the 90th percentile is 8 , while most models were trained with K =8 layers.

Original: That is, at least 90% of the examples in the dataset certainly did not suffer from under-reaching, because the number of layers was greater or equal than their diameter.
Revised: That is, at least 90% of the examples in the dataset certainly did not suffer from under-reaching, because the number of layers was greater or equal than their diameter.

Original: We trained another set of models withlayers, which did not show an improvement over the base models.
Revised: We trained another set of models with 10 layers, which did not show an improvement over the base models.

Original: We conclude that the source of improvement was clearly not the increased reachability, but instead, the reduction in over-squashing.
Revised: We conclude that the source of improvement was clearly not the increased reachability, but instead, the reduction in over-squashing.

Original: [Table]
Revised: Can larger hidden sizes achieve a similar improvement? We trained another set of models with doubled dimensions. These models achieved only 5.5% improvement over the base model (Appendix B.2), while adding the FA layer achieved 42% improvement using the original dimensions. Consistently, in Section 5 we present an analysis that shows how dimensions increase is ineffective in preventing over-squashing.

Original: [Table]
Revised: Is the entire FA layer needed? We experimented with using only a sampled fraction of edges in the FA layer. As Appendix B.3 shows, the fraction of added edges in the last layer correlates with the decrease in error. For example, using only half of the possible edges in the last layer (a “semi-adjacent” layer) still reduces the error rate by 31.5% on average compared to “base”.

Original: [Figure]
Revised: If all GNNs beneﬁtted from direct interaction between all nodes, maybe the graph structure is not even needed? We trained another set of models (Appendix B.2) where all layers are FA layers, thus ignoring the original graph topology; these models produced 1500% higher (worse) error.

Original:  B IOLOGICAL B ENCHMARKS
Revised: 4. B IOLOGICAL B ENCHMARKS

Original: Data We experimented with two popular biological datasets. NCI1  (Wale et al., 2008) contains ~ nodes  and its task is to predict whether a biochemical compound contains anti-lung-cancer activity.
Revised: Data The NCI1 dataset (Wale et al., 2008) contains ~30 nodes on average and its task is to predict whether a biochemical compound contains anti-lung-cancer activity.

Original: ENZYMES (Borgwardt et al., 2005) contains ~36 nodes and its task is to classify an enzyme to one out of six classes.
Revised: ENZYMES (Borgwardt et al., 2005) contains ~36 nodes and its task is to classify an enzyme to one out of six classes.

Original: We used the same 10-folds andtraining/validation/test split as Errica et al.
Revised: We used the same 10-folds and split as Errica et al.

Original: Models We used the implementation of Errica et al.
Revised: Models We used the implementation of Errica et al.

Original: (2020) who performed a fair and thorough comparison between GNNs.
Revised: (2020) who performed a fair and thorough comparison between GNNs.

Original: The ﬁnal reported result is the average of 30 test runs (10-folds × 3).
Revised: The ﬁnal reported result is the average of 30 test runs (10-folds × 3).

Original: Additional training details are provided in Appendix C.
Revised: Additional training details are provided in Appendix C.

Original: In ENZYMES, Errica et al.
Revised: In ENZYMES, Errica et al.

Original: found that a baseline that does not use the graph topology at all (“ No
Revised: found that a baseline that does not use the graph topology at all (“ No

Original: Struct ”) performs better than all GNNs.
Revised: Struct ”) performs better than all GNNs.

Original: In NCI1, GIN performed best.
Revised: In NCI1, GIN performed best.

Original: We converted the last layer into an FA layer by modifying the implementation of Errica et al., and repeated the same training procedure.
Revised: We converted the last layer into an FA layer by modifying the implementation of Errica et al., and repeated the same training procedure.

Original: We compare the “base” models from Errica et al.
Revised: We compare the “base” models from Errica et al.

Original: with our re-trained “+FA” models.
Revised: with our re-trained “+FA” models.

Original: Results Results are shown in Table 2.
Revised: Results Results are shown in Table 2.

Original: The main results are as follows: (a) in NCI1, GIN+FA improves by 1.5% over GIN-base, which was previously the best performing model; (b) in ENZYMES, where
Revised: The main results are as follows: (a) in NCI1, GIN+FA improves by 1.5% over GIN-base, which was previously the best performing model; (b) in ENZYMES, where

Original: Errica et al.
Revised: Errica et al.

Original: (2020) found that none of the GNNs exploit the topology of the graph, we ﬁnd that
Revised: (2020) found that none of the GNNs exploit the topology of the graph, we ﬁnd that

Original: GIN+FA does exploit the structure and improves by 8.1% over GIN-base and by 2.5% over No Struct .
Revised: GIN+FA does exploit the structure and improves by 8.1% over GIN-base and by 2.5% over No Struct .

Original: On average, models with FA layers relatively reduce the error rate by 12% in ENZYMES and by 4.8% in NCI1.
Revised: On average, models with FA layers relatively reduce the error rate by 12% in ENZYMES and by 4.8% in NCI1.

Original: These experiments clearly show evidence for a bottleneck in the original GNN models. 
Revised: These experiments clearly show evidence for a bottleneck in the original GNN models. [Table] [Table] The

Revised: [Table]

Original:  4. P ROGRAMS : V AR M ISUSE
Revised: r [Table] [Table] 4. P ROGRAMS : V AR M ISUSE

Original: Data V AR M ISUSE (Allamanis et al., 2018) is a node-prediction problem that depends on long-range information in computer programs.
Revised: Data V AR M ISUSE (Allamanis et al., 2018) is a node-prediction problem that depends on long-range information in computer programs.

Original: We used the sametraining/validation/SeenProjTest/UnseenProjTest splits as Allamanis et al.
Revised: We used the same splits as Allamanis et al.

Original: (2018) and Brockschmidt (2020). Models We use the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning by searching over 30 conﬁgurations for each GNN type.
Revised: Models We use the implementation of Brockschmidt (2020) who performed an extensive hyperparameter tuning by searching over 30 conﬁgurations for each GNN type.

Original: The best results were found using six to ten propagation steps, which hints that this problem requires long-range information.
Revised: The best results were found using 6-10 layers, which hints that this problem requires long-range information.

Original: We added a FA layer on top of the ﬁnal node representations, and used the resulting representations for node classiﬁcation.
Revised: We modiﬁed the last layer to be an FA layer, and used the resulting representations for node classiﬁcation.

Original: We used the same best found conﬁgurations as Brockschmidt (2020) add re-trained the modiﬁed models ﬁve times using different random seeds.
Revised: We used the same best found conﬁgurations as Brockschmidt (2020) add re-trained each model ﬁve times using different random seeds.

Original: Results Results are shown in Table 3.
Revised: Results Results are shown in Table 3.

Original: The main result is that adding an FA layer to all GNNs improves their SeenProjTest accuracy, obtaining a new state-of-the-art of 88.4%.
Revised: The main result is that adding an FA layer to all GNNs improves their SeenProjTest accuracy, obtaining a new state-of-the-art of 88.4%.

Original: In the Unseen ProjTest set, adding an an FA layer improves the results of some of most of the GNNs, obtaining a new state-ofthe-art of 83.8% These improvements are signiﬁcant,  since they were achieved on extensively-tuned models, without further tuning.[Table]
Revised: In the Unseen ProjTest set, adding an an FA layer improves the results of some of most of the GNNs, obtaining a new stateof-the-art of 83.8% These improvements are signiﬁcant, especially since they were achieved on extensively-tuned models, without further tuning.

Original:  C OMBINATORIAL A NALYSIS
Revised:  H OW L ONG IS L ONG -R ANGE ?

Original: In this section, we analyze the bottleneck combinatorially in the T REE -N EIGHBORS M ATCH problem.
Revised: In this section, we analyze over-squashing combinatorially in the T REE -N EIGHBORS M ATCH problem.

Original: We provide a combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt  (learn to 100% training accuracy) given its hidden vector size d .
Revised: We provide a combinatorial lower bound for the minimal hidden size that a GNN requires to perfectly ﬁt the data (learn to 100% training accuracy) given its problem radius r .

Original: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN h ( k ) v , as d .
Revised: We denote the arity of such a tree by m ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN, i.e., the size of a node vector h ( k ) v , as d .

Revised: A full tree of arity m and problem radius r = depth has m r leaves.

Revised: Given an arrangement of blue neighbors, all possible permutations of the labels {A, B, C, ...} are valid, modulo replacing sibling nodes.

Revised: Thus, the number of leaf label assignments is ( m r )!

Original: To have enough bits at the location of the bottleneck and perfectly ﬁt the training data, we must satisfy:
Revised: /m r − 1 . Right before interacting with the target node and predicting the label, a single vector of size d must encapsulate the information ﬂowing from all leaves (Equations (2) and (3)). 1 Such a vector contains d ﬂoating-point elements, each of them is stored as f bits.

Revised: Overall, the number of possible cases that this vector can distinguish between is b f · d .

Revised: The number of possible cases that the vector can distinguish between must be greater than the number of different examples that this vector may encounter in the training data.

Revised: This requirement is expressed in Equation (6).

Revised: Considering binary trees ( m =2 ), and ﬂoating-point values of f = binary ( b =2 ) bits, we get Equation (7):

Original: [Equation]
Revised: [Equation]

Original: We provide the complete analysis in Appendix E.

Original: Since factorial grows faster than an exponent with a constant base, we can see that a small increase of depth requires a much larger increase in d .
Revised: Since factorial grows faster than an exponent with a constant base, we can see that a small increase in the problem radius r requires a much larger increase in d .

Original: Speciﬁcally, it means that for d =32 as in the experiments in Section 4.1, the combinatorial upper bound of the model is as low as depth =7  .
Revised: Speciﬁcally, it means that for d = as in the experiments in Section 4.1, the maximal problem radius is as low as r =7 , because the minimal d for r =8 is greater than 32 .

Original: That is, a model with d =32 cannot obtain 100% accuracy for depth ≥ 8 .
Revised: That is, a model with d =32 cannot obtain 100% accuracy for r> 7 .

Original: In practice, the problem is worse; i.e., the empirical upper bound is lower, because even if a solution to storing some information in a vector of a certain size exists,it is not guaranteed that a gradient descent-based algorithm will ﬁnd it.
Revised: In practice, the problem is worse; i.e., the empirical minimal d is higher than the combinatorial, because even if a solution to storing some information in a vector of a certain size exists, a gradient descent-based algorithm is not guaranteed to ﬁnd it.

Original: Figure 4 shows the combinatorial upper bounds for depth given d ∈ { 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } .
Revised: Figure 4 shows the combinatorial lower bound of d given r .

Original: We  repeated the experiments from Section 4.1 and report the max empirical depth for each value of d .
Revised: We also repeated the experiments from Section 4.1 and report the minimal empirical d for each value of r .

Original: As shown in Figure 4, even with  d =512 , the combinatorial upper bound is as low as depth =10 .
Revised: As shown in Figure 4, the empirical and the theoretical minimal d grow exponentially with r ; for example, even d =512 can empirically ﬁt r =7 at most.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Under-reaching Although GNNs can be as powerful as the Weisfeiler-Lehman graph isomorphism test (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019), their expressiveness captures only a small fragment of ﬁrst-order logic (Barceló et al., 2020); the main limitation arises from the inability of a node to be aware of nodes that are farther away than the number of layers K , while the existence of such nodes can be easily described using logic.
Revised: Under-reaching Although GNNs can be as powerful as the Weisfeiler-Lehman graph isomorphism test (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019), their expressiveness captures only a small fragment of ﬁrst-order logic (Barceló et al., 2020); the main limitation arises from the inability of a node to be aware of nodes that are farther away than the number of layers K , while the existence of such nodes can be easily described using logic.

Original: We denote this limitation as under-reaching .
Revised: We denote this limitation as under-reaching .

Original: The over-squashing limitation described in this paper is tighter : even when information is reachable within K edges, this information might fail to ﬂow in the bottleneck (as we demonstrate in Section 4.2).
Revised: Nevertheless, even when information is reachable within K edges, this information might be oversquashed along the way (as we demonstrate in Section 4.2), and thus the over-squashing limitation described in this paper is tighter than under-reaching .

Original: Over-smoothing The over-smoothing phenomenon is related, but not identical, to over-squashing: node representations become indistinguishable and prediction performance severely degrades when the number of layers increases (Li et al., 2018; Klicpera et al., 2019;
Revised: Over-smoothing As observed before, node representations become indistinguishable and prediction performance severely degrades when the number of layers increases.

Original:  Wu et al., 2020; Oono and Suzuki, 2020).Several approaches were proposed to mitigate over-smoothing, such as normalization (Zhao and Akoglu, 2020), edge-based dropout (Rong et al., 2020), and noise-reduction regularization (Chen et al., 2020).
Revised: The accepted explanation to this phenomenon is over-smoothing (Li et al., 2018; Wu et al., 2020; Oono and Suzuki, 2020).

Original: This might explain the empirical optimality of few layers  (e.g., only two layers in 
Revised: This might explain the empirical optimality of few layers in short-range tasks (e.g., only K =2 layers in Kipf and Welling (2017)).

Original: Kipf and Welling (2017)); however, some problems depend on longer-range information propagation and thus require more layers, to avoid under-reaching .
Revised: Nonetheless, some problems depend on longer-range information propagation and thus require more layers, to avoid under-reaching .

Revised: In this paper, we hypothesize that in long-range problems, the explanation for the degraded performance is over-squashing rather than over-smoothing .

Original: Although over-smoothing and over-squashing are related, they are  different problems, and there are cases where one occurs without the other.
Revised: Although over-smoothing and over-squashing are related, they are disparate phenomena that occur in different types of problems.

Original: For example, imagine a triangular graph containing only three nodes: A, B, and C, where every node has  an edge to each of the other nodes.
Revised: For example, imagine a triangular graph containing only three nodes, where every node has a scalar value, an edge to each of the other nodes, and needs to compute a function of its own value and the other nodes’ values. The problem radius r in this case is r =1 .

Original: As we increase the number of layers, the representations of A, B and C might become indistinguishable, and thus suffer from over-smoothing .
Revised: As we increase the number of layers, the representations of the nodes might become indistinguishable, and thus suffer from over-smoothing .

Original: However, there will be no over-squashing in this case, because there is no growing amount of information that is squashed into ﬁxed-sized vectors while passing long-range messages.
Revised: However, there will be no over-squashing in this case, because there is no growing amount of information that is squashed into ﬁxed-sized vectors while passing long-range messages.

Original: In the other direction, in thecase of the T REE -N EIGHBORS M ATCH problem, there is no reason for over-smoothing to occur, because there are no two nodes that can converge to the same representation.
Revised: Contrarily, in the T REE -N EIGHBORS M ATCH problem, there is no reason for over-smoothing to occur, because there are no two nodes that can converge to the same representation.

Original: A node in a “higher” level (Figure 5) contains twice the information than the node in the “lower” level.
Revised: A node in a “higher” level in the tree contains twice the information than a node in a “lower” level.

Original: Thus, this is a case where over-squashing can occur without over-smoothing .
Revised: Thus, this is a case where over-squashing can occur without over-smoothing .

Revised: Avoiding over-squashing Although some previous work avoided over-squashing by various profitable means, none of these explicitly explained these means using over-squashing, and did not identify the bottleneck and its negative cross-domain implications.

Original: Avoiding over-squashing Gilmer et al.
Revised: Gilmer et al.

Original: (2017) add “virtual edges” to shorten long distances, and Scarselli et al.
Revised: (2017) add “virtual edges” to shorten long distances, and Scarselli et al.

Original: (2008) add “supersource nodes”; however, these were mostly ad hoc solutions.
Revised: (2008) add “supersource nodes”; however, these were mostly ad hoc solutions.

Original: Nikolentzos et al.
Revised: Nikolentzos et al.

Original: (2019) update a node’s representation by recursively aggregating information from “k-hop” neighbors away at every layer in a computationally expensive approach, which forces k to be 2 or 3 at the most.
Revised: (2019) update a node’s representation by recursively aggregating information from “k-hop” neighbors away at every layer in a computationally expensive approach, which forces k to be 2 or 3 at the most.

Original: Allamanis et al.
Revised: Allamanis et al.

Original: (2018) designed program analyses that serve as 16 “shortcut” edge types; however, these analyses are speciﬁc to their problem and require a human domain expert.
Revised: (2018) designed program analyses that serve as 16 “shortcut” edge types that were speciﬁc to their problem.

Original: Although some previous work avoided over-squashing by various proﬁtable means, none of these works explicitly identiﬁed the bottleneck and its negative cross-domain implications.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We highlight a new inherent problem that limits graph neural networks: a bottleneck that causes over-squashing.
Revised: We propose a novel explanation to a well known limitation in training graph neural networks: a bottleneck that causes over-squashing.

Original: Problems that depend on long-range interaction require as many GNN layers as the desired radius of each node’s receptive ﬁeld.
Revised: Problems that depend on long-range interaction require as many GNN layers as the desired radius of each node’s receptive ﬁeld.

Original: This causes an exponentially-growing amount of information to be squashed into a ﬁxed-length vector.
Revised: This causes an exponentiallygrowing amount of information to be squashed into a ﬁxed-length vector.

Original: As a result, the GNN fails to propagate long-range information, learns only short-range signals from the training data instead, and generalizes poorly when the prediction task depends on long-range interaction.
Revised: As a result, the GNN fails to propagate long-range information, learns only short-range signals from the training data instead, and generalizes poorly when the prediction task depends on long-range interaction.

Original: We demonstrate the existence of the bottleneck in a synthetic problem and show that GCN and GIN are more susceptible to over-squashing than other GNNs.
Revised: We demonstrate the existence of the bottleneck in a controlled problem and show that GCN and GIN are more susceptible to over-squashing than GAT and GGNN.

Original: We analyze this problem combinatorially and provide upper bounds for the tree depth.
Revised: We analyze this problem combinatorially and provide lower bounds for the hidden size given the problem radius.

Original: We further show that  models of chemical, biological and programmatical benchmarks suffer fromthe over-squashing by showing that they can be dramatically improved using a simple FA layer.
Revised: We further show that prior models of chemical, biological and programmatical benchmarks suffer from over-squashing by showing that they can be dramatically improved using a simple FA layer.

Original: We conclude that over-squashing in GNNs is so prevalent and untreated that in some benchmarks –that even the simplest solution helps.
Revised: We conclude that over-squashing in GNNs is so prevalent and untreated that in some benchmarks – even the simplest solution helps.

Original: This simple solution opens the path for a variety of follow-up improvements and even better solutions for over-squashing. 
Revised: Our observations open the path for a variety of follow-up improvements and even better solutions for over-squashing. A CKNOWLEDGMENTS

Revised: We would like to thank Federico Errica for his help in using his framework; Petar Veli ˇ ckovi ´ c for helpful discussions about GAT; and Jorge Perez for helpful discussions about the expressiveness of

Revised: GNNs.

Revised: We are also grateful to (alphabetically): Chen Zarfati, Elad Nachmias, Gail Weiss, Horace

Revised: He, Lotem Fridman, Pavol Bielik, Roy Sadaka, Shaked Brody, Yoav Goldberg, and the anonymous reviewers for their useful comments and suggestions.

Original: A T REE -N EIGHBORS M ATCH – T RAINING D ETAILS
Revised: A T REE -N EIGHBORS M ATCH – T RAINING D ETAILS

Original: Data We created a separate dataset for every  depth  and sampled up to 32,000 examples per dataset.
Revised: Data We created a separate dataset for every tree depth (which is equal to r , the problem radius) and sampled up to 32,000 examples per dataset.

Original: The label of each leaf (“A”, “B”, “C” in Figure 2) is represented as a one-hot vector.
Revised: The label of each leaf (“A”, “B”, “C” in Figure 2 in the paper) is represented as a one-hot vector.

Original: To tease the effect of the bottleneck from the ability of a GNN to count neighbors, we concatenated each leaf node’s initial representation with a 1-hot vector representing the number of blue neighbors, instead of creating the blue nodes.
Revised: To tease the effect of the bottleneck from the ability of a GNN to count neighbors, we concatenated each leaf node’s initial representation with a 1-hot vector representing the number of blue neighbors, instead of creating the blue nodes.

Original: The target node is initialized with an all-zeros vector as its (missing) label, concatenated with a 1-hot vector representing its number of blue neighbors.
Revised: The target node is initialized with an all-zeros vector as its (missing) label, concatenated with a 1-hot vector representing its number of blue neighbors.

Original: Intermediate nodes are initialized with a vector of zeros.
Revised: Intermediate nodes are initialized with a vector of zeros.

Original: Model The network has an initial linear layer, followed by depth + 1 GNN layers.
Revised: Model The network has an initial linear layer, followed by r + 1 GNN layers.

Original: Afterward, the ﬁnal target node representation goes through a linear layer and a softmax to predict its label.
Revised: Afterward, the ﬁnal target node representation goes through a linear layer and a softmax to predict its label.

Original: We experimented with GCN (Kipf and Welling, 2017), GGNN (Li et al., 2016), GIN (Xu et al., 2019) and GAT (Veliˇckovi´c et al., 2018) as the graph layers.
Revised: We experimented with GCN (Kipf and Welling, 2017), GGNN (Li et al., 2016), GIN (Xu et al., 2019) and GAT (Veliˇckovi´c et al., 2018) as the graph layers.

Original: In Section 4.1, we used model dimensions of d =32 .
Revised: In Section 4.1 of the paper, we used model dimensions of d =32 .

Original: Larger values lead to the exact same trend.
Revised: Larger values lead to the exact same trend.

Original: We further discuss the theoretical and empirical aspects of the dimension in Section 5.

Original: We added residual connections, summing every node with its own representation in the previous layer to increase expressivity, and layer normalization which eased convergence.
Revised: We added residual connections, summing every node with its own representation in the previous layer to increase expressivity, and layer normalization which eased convergence.

Original: We used the Adam optimizer with a learning rate of 10 − 3 , decayed by 0 .
Revised: We used the Adam optimizer with a learning rate of 10 − 3 , decayed by 0 .

Original: 5 after every 1000 epochs without an increase in training accuracy, and stopped training after 2000 epochs of no training accuracy improvement.
Revised: 5 after every 1000 epochs without an increase in training accuracy, and stopped training after 2000 epochs of no training accuracy improvement.

Original: This usually lead to tens of thousands of training epochs, sometimes reaching 100,000 epochs.
Revised: This usually lead to tens of thousands of training epochs, sometimes reaching 100, epochs.

Original: [Figure] [Table] To rule out hyperparameter tuning as the source of degraded performance, we experimented with changing activations (ReLu, tanh, MLP, none), using layer normalization and batch normalization, residual connections, various batch sizes, and whether or not the same GNN weights should be “unrolled” over time steps.
Revised: To rule out hyperparameter tuning as the source of degraded performance, we experimented with changing activations (ReLu, tanh, MLP, none), using layer normalization and batch normalization, residual connections, various batch sizes, and whether or not the same GNN weights should be “unrolled” over time steps.

Original: The presented results were obtained using the conﬁgurations that achieved the best results.
Revised: The presented results were obtained using the conﬁgurations that achieved the best results.

Original: [Table]
Revised: [Table]

Original: B QM9 – A DDITIONAL R ESULTS B. A DDITIONAL GNN T YPES
Revised: B QM9 – A DDITIONAL R ESULTS B. A DDITIONAL GNN T YPES

Original: Because of space limitations, in Section 5.3 of the paper we presented results on the QM9 dataset only for GNN-MLP0, R-GAT and GNN-FiLM.
Revised: Because of space limitations, in Section 5.3 of the paper we presented results on the QM9 dataset only for R-GIN, R-GAT and GGNN. In this section

Original: In this section, we show that additional GNN architectures beneﬁt from breaking the bottleneck using a fully-adjacent layer: GGNN (Li et al., 2016), R-GCN (Schlichtkrull et al., 2018) and R-GIN (Xu et al., 2019).
Revised: [Table] we show that additional GNN architectures beneﬁt from breaking the bottleneck using a fully-adjacent layer: GNN-MLP , R-GCN (Schlichtkrull et al., 2018) and GNN-FiLM (Brockschmidt, 2020).

Original: All experiments were performed using the extensively-tuned implementation of Brockschmidt (2020) who tuned a wide variety of hyperparameters.
Revised: All experiments were performed using the extensively-tuned implementation of Brockschmidt (2020) who experimented with over 500 hyperparameter conﬁgurations.

Original: For examples, they experimented with ReLU, GELU, leaky-ReLU and tanh as activations.

Original: Table 4 contains additional results for GGNN, R-GCN and R-GIN.
Revised: Table 4 contains additional results for GGNN, R-GCN and R-GIN.

Original: As shown in Table 4, adding an
Revised: As shown in Table 4, adding an

Original: FA layer signiﬁcantly improves results across all GNN architectures, for all propertiesexcept for “mu” in R-GAT, where adding an FA layer results in a slightly higher error rate.
Revised: FA layer signiﬁcantly improves results across all GNN architectures, for all properties

Original:  B. A LTERNATIVE S OLUTIONS
Revised: . B. A LTERNATIVE S OLUTIONS

Original: Table 5 shows additional experiments, all performed using GCN.
Revised: Table 5 shows additional experiments, all performed using GCN.

Original: base † is the original model of
Revised: base † is the original model of

Original: Brockschmidt (2020) as in Table 4.
Revised: Brockschmidt (2020) as in Table 4.

Original: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.
Revised: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.

Original: 2 × d is a model that was trained with a doubled hidden dimension size, d
Revised: 2 × d is a model that was trained with a doubled hidden dimension size, d

Original: = 256 instead of d =as in the base model.
Revised: = 256 instead of d =as in the base model.

Original: As shown, doubling the hidden dimension size leads to a small improvement of only 5.5% reduction in error.
Revised: As shown, doubling the hidden dimension size leads to a small improvement of only 5.5% reduction in error.

Original: In comparison, the + FA model used the original dimension sizes and achieves a much larger improvement of 43.40%.
Revised: In comparison, the + FA model used the original dimension sizes and achieves a much larger improvement of 43.40%.

Original: All FA is a model that was trained with all GNN layers converted into FA layers, practically ignoring the graph topology.
Revised: All FA is a model that was trained with all GNN layers converted into FA layers, practically ignoring the graph topology.

Original: This lead to much worse results of more than 1500% higher error.
Revised: This lead to much worse results of more than 1500% higher error.

Original: This shows that the graph topology is important in this benchmark, and that a direct interaction between nodes (as in a single FA layer) must be performed in addition to considering the topology.
Revised: This shows that the graph topology is important in this benchmark, and that a direct interaction between nodes (as in a single FA layer) must be performed in addition to considering the topology.

Original: 2 × FA is a model where the last layer was modiﬁed into an FA layer, and an additional FA layer was stacked on top of it.
Revised: 2 × FA is a model where the last layer was modiﬁed into an FA layer, and an additional FA layer was stacked on top of it.

Original: This lead to results that are very similar to + FA.
Revised: This lead to results that are very similar to + FA.

Original: Penultimate FA is a model where the FA layer is the penultimate layer (the K − 1 -th), followed by a standard GNN layer as the K -th layer.
Revised: Penultimate FA is a model where the FA layer is the penultimate layer (the K − 1 -th), followed by a standard GNN layer as the K -th layer.

Original: This leads to results that are even slightly better than + FA. 
Revised: This leads to results that are even slightly better than + FA. [Table]

Original: [Table]
Revised: [Table]

Original: B. P ARTIAL -FA L AYERS
Revised: B. P ARTIAL -FA L AYERS

Original: We also examined whether instead of adding a “full fully-adjacent layer”, we can randomly sample only a fraction of these edges.
Revised: We also examined whether instead of adding a “full fully-adjacent layer”, we can randomly sample only a fraction of these edges.

Original: We randomly sampled only { 0 . 25 , 0 . 5 , 0 .
Revised: We randomly sampled only { 0 . 25 , 0 . 5 , 0 .

Original: 75 } of the edges in the full FA layer in every example, and trained the model for each target property 5 times.
Revised: 75 } of the edges in the full FA layer in every example [Table] and trained the model for each target property 5 times.

Original: Tableshows the results of these experiments using GCN.
Revised: Tableshows the results of these experiments using GCN.

Original: base † is the original model of Brockschmidt (2020) as in Table 4.
Revised: base † is the original model of Brockschmidt (2020) as in Table 4.

Original: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.
Revised: + FA is the model that we re-trained with the last layer modiﬁed to an FA layer.

Original: 75 }× FA are the models were only a fraction of the edges in the FA layer were used.
Revised: 75 }× FA are the models were only a fraction of the edges in the FA layer were used.

Original: As shown in Table 6, the full FA layer achieves the largest reduction in error (-43.4%), but even a fraction of the edges improves the results over the base model.
Revised: As shown in Table [Table] the full FA layer achieves the largest reduction in error (-43.4%), but even a fraction of the edges improves the results over the base model.

Original: For example, using only half of the edges ( 0.5 × FA ) reduces the error by 31.5%.
Revised: For example, using only half of the edges ( 0.5 × FA ) reduces the error by 31.5%.

Original: Overall, the percentage of used edges in the partial-FA layer is correlated with its reduction in error.
Revised: Overall, the percentage of used edges in the partial-FA layer is correlated with its reduction in error.

Original: C B IOLOGICAL B ENCHMARKS – T RAINING D ETAILS
Revised: C B IOLOGICAL B ENCHMARKS – T RAINING D ETAILS

Original: We used the implementation of Errica et al.
Revised: We used the implementation of Errica et al.

Original: (2020) who performed a fair and thorough comparison between GNNs, by splitting each dataset to 10-folds; then, for each GNN type they select a conﬁguration among a grid of 72 conﬁgurations according to the validation set; ﬁnally, the best conﬁguration for each fold is trained three additional times, early stopped using the validation set, and evaluated on the test set.
Revised: (2020) who performed a fair and thorough comparison between GNNs, by splitting each dataset to 10-folds; then, for each GNN type they select a conﬁguration among a grid of 72 conﬁgurations according to the validation set; ﬁnally, the best conﬁguration for each fold is trained three additional times, early stopped using the validation set, and evaluated on the test set.

Original: The ﬁnal reported result is the average of all 30 test runs (10-folds × 3).
Revised: The ﬁnal reported result is the average of all 30 test runs (10-folds × 3).

Original: The ﬁnal standard deviation is computed among the average results of each of the ten folds.
Revised: The ﬁnal standard deviation is computed among the average results of each of the ten folds.

Original: D D ATA S TATISTICS D. S YNTHETIC D ATASET : T REE -N EIGHBORS M ATCH
Revised: D D ATA S TATISTICS D. S YNTHETIC D ATASET : T REE -N EIGHBORS M ATCH

Original: Statistics of the synthetic T REE -N EIGHBORS M ATCH dataset are shown in Table 7.
Revised: Statistics of the synthetic T REE -N EIGHBORS M ATCH dataset are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: D. Q UANTUM C HEMISTRY : QM 
Revised: D. Q UANTUM C HEMISTRY : QM

Original: Statistics of the quantum chemistry QM9 dataset, as used in Brockschmidt (2020) are shown in
Revised: Statistics of the quantum chemistry QM9 dataset, as used in Brockschmidt (2020) are shown in

Original: Table 8.
Revised: Table 8.

Revised: [Table]

Original: D. B IOLOGICAL B ENCHMARKS
Revised: D. B IOLOGICAL B ENCHMARKS

Original: Statistics of the biological datasets, as used in Errica et al.
Revised: Statistics of the biological datasets, as used in Errica et al.

Original: (2020), are shown in Table 9.
Revised: (2020), are shown in Table 9.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D. V AR M ISUSE
Revised: D. V AR M ISUSE

Original: Statistics of the V AR M ISUSE dataset, as used in Allamanis et al.
Revised: Statistics of the V AR M ISUSE dataset, as used in Allamanis et al.

Original: (2018) and Brockschmidt (2020),
Revised: (2018) and Brockschmidt (2020),

Original: are shown in Table 10.
Revised: are shown in Table 10.

Original: [Table]

Original: E C OMBINATORIAL A NALYSIS
Revised: E C OMBINATORIAL A NALYSIS

Original: We analyize the combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .
Revised: We analyize the combinatorial upper bound for the maximal depth that a GNN can perfectly ﬁt (learn to 100% training accuracy) given its hidden vector size d .

Original: We denote the arity of such a tree by m  ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN, i.e., the size of a node vector h ( k ) v , as d .
Revised: We denote the arity of such a tree by m [Table] ( =2 in our experiments); the counting base as b =2 ; the number of bits in a ﬂoating-point variable as f =32 ; and the hidden dimension of the GNN, i.e., the size of a node vector h ( k ) v , as d .

Original: A full tree of arity m has m depth leaves.
Revised: A full tree of arity m has m depth leaves.

Original: As described in Section 4.1, given an arrangement of blue neighbors, all possible permutations of the labels {A, B, C, ...} are valid.
Revised: As described in Section 4.1, given an arrangement of blue neighbors, all possible permutations of the labels {A, B, C, ...} are valid.

Original: Thus, the number of leaf label assignments is (cid:0) m depth (cid:1) ! .
Revised: Thus, the number of leaf label assignments is (cid:0) m depth (cid:1) ! .

Original: Right before interacting with the target node and predicting the label, a single vector of size d must encapsulate the information ﬂowing from all leaves (Equations (2) and (3)).
Revised: Right before interacting with the target node and predicting the label, a single vector of size d must encapsulate the information ﬂowing from all leaves (Equations (2) and (3)).

Original: 1 Such a vector contains d ﬂoating-point elements, each of them is stored as f bits.
Revised: 2 Such a vector contains d ﬂoating-point elements, each of them is stored as f bits.

Original: Overall, the number of possible cases that this vector can distinguish between is b f · d .
Revised: Overall, the number of possible cases that this vector can distinguish between is b f · d .

Original: The number of possible cases that the vector can distinguish between must be greater than the number of different examples this vector may encounter in the training data.
Revised: The number of possible cases that the vector can distinguish between must be greater than the number of different examples this vector may encounter in the training data.

Original: Thus, to have enough bits to perfectly ﬁt the training data at the location of the bottleneck, depth and d must satisfy Equation (6); considering binary trees ( m =2 ), and ﬂoating-point values of f = binary ( b =2 ) bits, we get Equation (7):
Revised: Equation (6); considering binary trees ( m =2 ), and ﬂoating-point values of f =32 binary ( b =2 ) bits, we get Equation (7):

Original: [Equation]
Revised: [Equation]

Original: Now, we can ﬁx either d or depth and solve for the other.
Revised: Now, we can ﬁx either d or depth and solve for the other.

Original: For example, in Figure 4 we ﬁxed different values of d and found the maximal depth ∈ N that satisﬁes Equation (7), to get the combinatorial max depth for each value of d .
Revised: For example, in Figure 4 we ﬁxed different values of d and found the maximal depth ∈ N that satisﬁes Equation (7), to get the combinatorial max depth for each value of d .