Original: B REAKING THE E XPRESSIVE B OTTLENECKS OF G RAPH N EURAL N ETWORKS
Revised: B REAKING THE E XPRESSIVE B OTTLENECKS OF G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressiveness of graph neural networks (GNNs), showing that the neighborhood aggregation GNNs were at most as powerful as 1-WL test in distinguishing graph structures.
Revised: Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressiveness of graph neural networks (GNNs), showing that the neighborhood aggregation GNNs were at most as powerful as 1-WL test in distinguishing graph structures.

Original: There were also improvements proposed in analogy to k -WL test ( k > 1 ).
Revised: There were also improvements proposed in analogy to k -WL test ( k ą 1 ).

Original: However, the aggregators in these GNNs are far from injective as required by the WL test, and suffer from weak distinguishing strength, making it become expressive bottlenecks.
Revised: However, the aggregators in these GNNs are far from injective as required by the WL test, and suffer from weak distinguishing strength, making it become expressive bottlenecks.

Original: In this paper, we improve the expressiveness by exploring powerful aggregators.
Revised: In this paper, we improve the expressiveness by exploring powerful aggregators.

Original: We reformulate aggregation with the corresponding aggregation coefﬁcient matrix, and then systematically analyze the requirements of the aggregation coefﬁcient matrix for building more powerful aggregators and even injective aggregators.
Revised: We reformulate aggregation with the corresponding aggregation coefﬁcient matrix, and then systematically analyze the requirements of the aggregation coefﬁcient matrix for building more powerful aggregators and even injective aggregators.

Original: It can also be viewed as the strategy for preserving the rank of hidden features, and implies that basic aggregators correspond to a special case of low-rank transformations.
Revised: It can also be viewed as the strategy for preserving the rank of hidden features, and implies that basic aggregators correspond to a special case of low-rank transformations.

Original: We also show the necessity of applying nonlinear units ahead of aggregation, which is different from most aggregation-based
Revised: We also show the necessity of applying nonlinear units ahead of aggregation, which is different from most aggregation-based

Original: GNNs.
Revised: GNNs.

Original: Based on our theoretical analysis, we develop two GNN layers, ExpandingConv and CombConv.
Revised: Based on our theoretical analysis, we develop two GNN layers, ExpandingConv and CombConv.

Original: Experimental results show that our models signiﬁcantly boost performance, especially for large and densely connected graphs.
Revised: Experimental results show that our models signiﬁcantly boost performance, especially for large and densely connected graphs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs are ubiquitous in the real world.
Revised: Graphs are ubiquitous in the real world.

Original: Social networks, trafﬁc networks, knowledge graphs, and molecular structures are typical graph-structured data.
Revised: Social networks, trafﬁc networks, knowledge graphs, and molecular structures are typical graph-structured data.

Original: Graph Neural Networks (GNNs) Scarselli et al. ; Gori et al. (2005), leveraging the power of neural networks to graph-structured data, have a rapid development recently Kipf & Welling (2016); Hamilton et al. ; Bronstein et al.
Revised: Graph Neural Networks (GNNs) (Scarselli et al., 2008; Gori et al., 2005), leveraging the power of neural networks to graph-structured data, have a rapid development recently (Kipf & Welling, 2016; Hamilton et al., 2017; Bronstein et al., 2017; Gilmer et al., 2017; Duvenaud et al., 2015).

Original: (2017); Gilmer et al.

Original: ; Duvenaud et al.

Original: Expressive power of GNNs measures their abilities to represent different graph structures Sato (2020).
Revised: Expressive power of GNNs measures their abilities to represent different graph structures (Sato, 2020).

Original: It decides the performance of GNNs where the awareness of graph structures is required, especially on large graphs with complex topologies.
Revised: It decides the performance of GNNs where the awareness of graph structures is required, especially on large graphs with complex topologies.

Original: The neighborhood aggregation scheme (or message passing) follows the same pattern with weisﬁler-lehman (WL) graph isomorphism test Weisfeiler & Leman (1968) to encode graph structures, where node representations are computed iteratively by aggregating transformed representations of its neighbors with structural information learned implicitly.
Revised: The neighborhood aggregation scheme (or message passing) follows the same pattern with weisﬁler-lehman (WL) graph isomorphism test (Weisfeiler & Leman, 1968) to encode graph structures, where node representations are computed iteratively by aggregating transformed representations of its neighbors with structural information learned implicitly.

Original: Therefore, the WL test is used to measure the expressiveness of GNNs.
Revised: Therefore, the WL test is used to measure the expressiveness of GNNs.

Original: Unfortunately, general GNNs are at most as powerful as 1-order WL test Morris et al. Xu et al.
Revised: Unfortunately, general GNNs are at most as powerful as 1-order WL test (Morris et al., 2019; Xu et al., 2019).

Original: There is also work trying to improve the expressiveness that are beyond 1-order WL test Maron et al. ; Morris et al.
Revised: There is also work trying to improve the expressiveness that are beyond 1-order WL test (Maron et al., 2019; Morris et al., 2019;

Original: ; Chen et al. ; Li et al. (2020b); Vignac et al.
Revised: Chen et al., 2019; Li et al., 2020b; Vignac et al., 2020).

Original: However, the weak distinguishing strength of aggregators is the fundamental limitation.
Revised: However, the weak distinguishing strength of aggregators is the fundamental limitation.

Original: The expressiveness analysis measured by the WL test assumes that aggregators are injective, which is usually unreachable.
Revised: The expressiveness analysis measured by the WL test assumes that aggregators are injective, which is usually unreachable.

Original: Therefore, this motivates us to investigate the following questions: What are the key factors to limit the expressiveness of GNN?
Revised: Therefore, this motivates us to investigate the following questions: What are the key factors to limit the expressiveness of GNN?

Original: and how to break these limitations?
Revised: and how to break these limitations?

Original: Aggregators are permutation invariant functions that operate on sets while preserving permutation invariance.
Revised: Aggregators are permutation invariant functions that operate on sets while preserving permutation invariance.

Original: Zaheer et al.
Revised: (Zaheer et al., 2017)

Original: ﬁrst theoretically studied permutation invariant functions and provided a family of functions to which any permutation invariant function must belong.
Revised: ﬁrst theoretically studied permutation invariant functions and provided a family of functions to which any permutation invariant function must belong.

Original: Xu et al. (2019) extended it on multisets but only for countable space.
Revised: (Xu et al., 2019) extended it on multisets but only for countable space.

Original: Corso et al. (2020) further extended it to uncountable space.
Revised: (Corso et al., 2020) further extended it to uncountable space.

Original: Murphy et al. (2018) and Murphy et al. (2019) expressed a permutation invariant function by approximating an average over permutation-sensitive functions with tractability strategies.
Revised: (Murphy et al., 2018) and (Murphy et al., 2019) expressed a permutation invariant function by approximating an average over permutation-sensitive functions with tractability strategies.

Original: Dehmamy et al. (2019) showed that a single propagation rule applied in general GNNs is rather restrictive in learning graph moments Lin & Skiena (1995).
Revised: (Dehmamy et al., 2019) showed that a single propagation rule applied in general GNNs is rather restrictive in learning graph moments (Lin & Skiena, 1995).

Original: They and Corso et al. (2020) improved the distinguishing strength of aggregation by leveraging multiple basic aggregators (SUM,
Revised: They and (Corso et al., 2020) improved the distinguishing strength of aggregation by leveraging multiple basic aggregators (SUM,

Original: MEAN, NORMALIZED MEAN, MAX/MIN, and STD).
Revised: MEAN, NORMALIZED MEAN, MAX/MIN, and STD).

Original: This strategy showed its effectiveness on tasks taken from classical graph theory.
Revised: This strategy showed its effectiveness on tasks taken from classical graph theory.

Original: In contrast to existing studies towards aggregators in GNNs, we provide a new GNN formulation, where the aggregation is represented as the multiplication of the corresponding hidden feature matrix of neighbors and the aggregation coefﬁcient matrix.
Revised: In contrast to existing studies towards aggregators in GNNs, we provide a new GNN formulation, where the aggregation is represented as the multiplication of the corresponding hidden feature matrix of neighbors and the aggregation coefﬁcient matrix.

Original: This new formulation enables us to answer the following questions: (i) when a GNN will lose its expressive power; (ii) How to build aggregators with higher distinguishing strength, even injective aggregators.
Revised: This new formulation enables us to answer the following questions: (i) when a GNN will lose its expressive power; (ii) How to build aggregators with higher distinguishing strength, even injective aggregators.

Original: Based on our theoretical analysis, we propose two GNN layers: ExpandingConv and CombConv, and evaluate them on general graph classiﬁcation and graph regression tasks.
Revised: Based on our theoretical analysis, we propose two GNN layers: ExpandingConv and CombConv, and evaluate them on general graph classiﬁcation and graph regression tasks.

Original: Our key contributions are summarized as follows:
Revised: Our key contributions are summarized as follows:

Original: • We formalize the distinguishing strength of aggregators as a partial order, and theoretically show that the choice of aggregators can be bottlenecks of expressiveness.
Revised: • We formalize the distinguishing strength of aggregators as a partial order, and theoretically show that the choice of aggregators can be bottlenecks of expressiveness.

Original: We also propose to apply nonlinear units ahead of aggregations to break the distinguishing strength limitations of aggregators as well as to achieve an implicit sampling mechanism.
Revised: We also propose to apply nonlinear units ahead of aggregations to break the distinguishing strength limitations of aggregators as well as to achieve an implicit sampling mechanism.

Original: • We reformulate the neighborhood aggregation with the aggregation coefﬁcient matrix and then provide a theoretical point of view on building powerful aggregators and even injective aggregators.
Revised: • We reformulate the neighborhood aggregation with the aggregation coefﬁcient matrix and then provide a theoretical point of view on building powerful aggregators and even injective aggregators.

Original: • We propose ExpandingConv and CombConv layers which achieve state-of-the-art performance on a variety of graph tasks.
Revised: • We propose ExpandingConv and CombConv layers which achieve state-of-the-art performance on a variety of graph tasks.

Original: We also show that multi-head GAT is one of the ExpandingConv implementations, which brings a theoretical explanation for its effectiveness.
Revised: We also show that multi-head GAT is one of the ExpandingConv implementations, which brings a theoretical explanation for its effectiveness.

Original:  P RELIMINARIES 2. N OTATIONS
Revised:  P RELIMINARIES 2. N OTATIONS

Original: For a graph G ( V, E ) , we denote the set of edges, nodes and node feature vectors respectively by
Revised: For a graph G p V, E q , we denote the set of edges, nodes and node feature vectors respectively by

Original: E G , V G and X G .
Revised: E G , V G and X G .

Original: N ( v ) represents the set of neighbors of v including itself, i.e., N ( v ) =
Revised: N p v q represents the set of neighbors of v including itself, i.e., N p v q “ t u P

Original: { u ∈ V G | ( u, v ) ∈ E G } ∪ { v } .
Revised: V G |p u, v q P E G u Y t v u .

Original: We use [ n ] to denote the set { 1 , 2 , ..., n } .
Revised: We use r n s to denote the set t 1 , 2 , ..., n u .

Original: {{ ... }} represents a multi-set, i.e., a set with possibly repeating elements.
Revised: tt ... uu represents a multi-set, i.e., a set with possibly repeating elements.

Original: Π n represents the set of all permutations of the integers 1 to n .
Revised: Π n represents the set of all permutations of the integersto n .

Original: h π , where π ∈ Π | h | , is a reordering of the elements of a sequence h according to π .
Revised: h π , where π P Π | h | , is a reordering of the elements of a sequence h according to π .

Original: Given a matrix X ∈ R a × b , X T represents the transpose of X , and vec ( X ) ∈ R ab × 1 represents the column stack of X .
Revised: Given a matrix X P R a ˆ b , X T represents the transpose of X , and vec p X q P R ab ˆ 1 represents the column stack of X .

Original: 2. G RAPH N EURAL N ETWORKS
Revised: 2. G RAPH N EURAL N ETWORKS

Original: Most GNNs adopt the neighborhood aggregation scheme Gilmer et al. (2017) to learn the node representations, which utilizes both node features and graph structures.
Revised: Most GNNs adopt the neighborhood aggregation scheme (Gilmer et al., 2017) to learn the node representations, which utilizes both node features and graph structures.

Original: In the k -th layer, the representation of node v
Revised: In the k -th layer, the representation of node v

Original: [Equation]
Revised: [Equation]

Original: Aggregators in GNNs.
Revised: Aggregators in GNNs.

Original: An aggregator is a permutation invariant function Zaheer et al. (2017) with bounded size inputs.
Revised: An aggregator is a permutation invariant function (Zaheer et al., 2017) with bounded size inputs.

Original: It satisﬁes: (i) size insensitive: an aggregator can take an arbitrary but ﬁnite size of inputs; (ii) permutation invariant: an aggregator is invariant to the permutation of input.
Revised: It satisﬁes: (i) size insensitive: an aggregator can take an arbitrary but ﬁnite size of inputs; (ii) permutation invariant: an aggregator is invariant to the permutation of input.

Original: There are a limited number of basic aggregators such as SUM, MEAN, NORMALIZED MEAN, MAX/MIN,
Revised: There are a limited number of basic aggregators such as SUM, MEAN, NORMALIZED MEAN, MAX/MIN,

Original: STD, etc.
Revised: STD, etc.

Original: Most proposed GNNs apply one of these aggregators.
Revised: Most proposed GNNs apply one of these aggregators.

Original: Sum-of-power mapping Zaheer et al. (2017) and normalized moments Corso et al. (2020) can also be used as aggregators and they allow for a variable number of aggregators.
Revised: Sum-of-power mapping (Zaheer et al., 2017) and normalized moments (Corso et al., 2020) can also be used as aggregators and they allow for a variable number of aggregators.

Original:  P ROPOSED M ODEL
Revised:  P ROPOSED M ODEL

Original: In this section, we ﬁrst formalize the distinguishing strength of aggregators as a partial order, and show why basic aggregators used in popular GNNs become bottlenecks of expressiveness.
Revised: In this section, we ﬁrst formalize the distinguishing strength of aggregators as a partial order, and show why basic aggregators used in popular GNNs become bottlenecks of expressiveness.

Original: Then, we analyze the requirements for building powerful aggregators and even injective aggregators.
Revised: Then, we analyze the requirements for building powerful aggregators and even injective aggregators.

Original: Finally, we introduce two GNN layers based on our theoretical analysis.
Revised: Finally, we introduce two GNN layers based on our theoretical analysis.

Original: 3. D ISTINGUISHING S TRENGTH OF A GGREGATORS
Revised: 3. D ISTINGUISHING S TRENGTH OF A GGREGATORS

Original: To ensure generality, our analysis of aggregators is always considered in multisets and uncountable case, where the inputs are continuous and with possibly repeating elements.
Revised: To ensure generality, our analysis of aggregators is always considered in multisets and uncountable case, where the inputs are continuous and with possibly repeating elements.

Revised: We ﬁrst introduce distinguishing strength under the concept of partial order (Schmidt, 2011).

Original: Distinguishing strength.
Revised: Distinguishing strength.

Original: The distinguishing strength of aggregator f aggr 1 is stronger than f aggr 2 , denoted by f aggr 1 (cid:23) f aggr 2 , if and only if for any two multisets x 1 and x 2 where the number of elements can be different, f aggr 2 ( x 1 ) (cid:54) = f aggr 2 ( x 2 ) ⇒ f aggr 1 ( x 1 ) (cid:54) =
Revised: The distinguishing strength of aggregator f aggr 1 is stronger than f aggr 2 , denoted by f aggr 1 ľ f aggr 2 , if and only if for any two multisets x 1 and x 2 where the number of elements can be different, f aggr 2 p x 1 q ‰ f aggr 2 p x 2 q ñ f aggr 1 p x 1 q ‰ f aggr 1 p x 2 q .

Original: f aggr 1 ( x 2 ) . Meanwhile, if there exist x (cid:48) 1 and x (cid:48) 2 such that f aggr 1 ( x(cid:48) 1 ) (cid:54) = f aggr 1 ( x (cid:48) 2 ) but f aggr 2 ( x(cid:48) 1 ) = f aggr 2 ( x (cid:48) 2 ) , f aggr 1 is strictly stronger than f aggr 2 , denoted by f aggr 1 (cid:31)
Revised: Meanwhile, if there exist x 1 1 and x 1 2 such that f aggr 1 p x 1 1 q ‰ f aggr 1 p x 1 2 q but f aggr 2 p x 1 1 q “ f aggr 2 p x 1 2 q , f aggr 1 is strictly stronger than f aggr 2 , denoted by f aggr 1 ą f aggr 2 .

Original:  f aggr 2 .
Revised: If f aggr 1 ľ f aggr 2 ľ f aggr 1 , we say these two aggregators have the same distinguishing strength, denoted by f aggr 1 “ f aggr 2 .

Original: If f aggr 1 (cid:23) f aggr 2 (cid:23) f aggr 1  ,we say these two aggregators have the same distinguishing strength, denoted by f aggr  1 = f aggr 2 . If there does not exist  f aggr 1 (cid:23) f aggr  2 or f aggr 2 (cid:23) f aggr  1  , we say f aggr 1 and f aggr 2 are incomparable.
Revised: If there exist multisets x 1 and x 2 such that f aggr 1 p x 1 q ‰ f aggr 1 p x 2 q , f aggr 2 p x 1 q “ f aggr 2 p x 2 q , and there also exist x 1 1 and x 1 2 such that f aggr 1 p x 1 1 q “ f aggr 1 p x 1 2 q , f aggr 2 p x 1 1 q ‰ f aggr 2 p x 1 2 q , we say f aggrand f aggr 2 are incomparable.

Original: Distinguishing strength is a partial order, and  all aggregators form a poset.
Revised: Distinguishing strength is a partial order, and the set of all aggregators form a poset.

Original: In this poset, the aggregators with the greatest distinguishing strength should be injective.
Revised: In this poset, the aggregators with the greatest distinguishing strength should be injective.

Original: With the deﬁnition of distinguishing strength, we can compare any two aggregators.
Revised: With the deﬁnition of distinguishing strength, we can compare any two aggregators.

Original: The distinguishing strength of widely used aggregators SUM, MEAN, MAX/MIN is incomparable.
Revised: The distinguishing strength of widely used aggregators SUM, MEAN, MAX/MIN is incomparable.

Original: One can easily give two multisets of elements that are distinguished by one aggregator but are not distinguished by the others.
Revised: One can easily give two multisets of elements that are distinguished by one aggregator but are not distinguished by the others as showed in (Corso et al., 2020).

Original: Equivariant aggregator.
Revised: Equivariant aggregator.

Original: f aggr : {{ R d }} → R d is an equivariant aggregator if and only if f aggr ( {{ T · x 1 , T · x 2 , · · · , T · x n }} ) = T · f aggr ( {{ x 1 , x 2 , · · · , x n }} ) for any T ∈ R m × d and {{ x i ∈ R d | i ∈ [ n ] }} .
Revised: f aggr : tt R d uu Ñ R d is an equivariant aggregator if and only if f aggr ptt T ¨ x 1 , T ¨ x 2 , ¨ ¨ ¨ , T ¨ x n uuq “ T ¨ f aggr ptt x 1 , x 2 , ¨ ¨ ¨ , x n uuq for any T P R m ˆ d and tt x i P R d | i P r n suu .

Original: Widely used SUM and MEAN are equivariant aggregators but MAX/MIN is not.
Revised: Widely used SUM and MEAN are equivariant aggregators but MAX/MIN is not.

Original: We denote f aggr 1 ⊗ f aggr 2 a new aggregator by combing f aggr 1 and f aggr 2 with f aggr 1 ⊗ f aggr2 ( X ) =
Revised: We denote f aggr 1 b f aggr 2 a new aggregator by combing f aggr 1 and f aggr 2 with f aggr 1 b f aggr

Original: [ f aggr 1 ( X ) || f aggr 2 ( X )] , where || denotes concatenation.
Revised: 2 p X q “ r f aggr 1 p X q|| f aggr 2 p X qs , where || denotes concatenation.

Original: Lemma 1.
Revised: Lemma 1.

Original: (i)
Revised: (i)

Original: For any continuous function g , we have g ◦ f aggr (cid:22) f aggr , and when g is injective, f aggr =
Revised: For any continuous function g , we have g ˝ f aggr ĺ f aggr , and when g is injective, f aggr “ g ˝ f aggr ; (ii) f aggr 1 b f aggr 2 ľ f aggr 1 and f aggr 1 b f aggr 2 ľ f aggr 2 .

Original: g ◦ f aggr ; (ii) f aggr 1 ⊗ f aggr 2 (cid:23) f aggr 1 or f aggr 2 , and if f aggr 1 and faggrare incomparable, f aggr 1 ⊗ f aggr 2 (cid:31) f aggr 1 or f aggr 2 ; (iii) If f aggr is an equivariant aggregator, then f aggr ( T · x 1 , T · x 2 , · · · , T · x n ) (cid:22) f aggr ( x 1 , x 2 , · · · , x n ) for any T ∈ R m × d and {{ x i ∈ R d | i ∈ [ n ] }} .
Revised: If f aggr 1 and f aggr 2 are incomparable, f aggr 1 b f aggr 2 ą f aggr 1 and f aggr 1 b f aggr 2 ą f aggr 2 ; (iii) If f aggr is an equivariant aggregator, then f aggr p T ¨ x 1 , T ¨ x 2 , ¨ ¨ ¨ , T ¨ x n q ĺ f aggr p x 1 , x 2 , ¨ ¨ ¨ , x n q for any T P R m ˆ d and tt x i P R d | i P r n suu .

Original: We prove Lemma 1 in Appendix B. Lemma 1 indicates that aggregators become bottlenecks of distinguishing strength.
Revised: We prove Lemma 1 in Appendix B. Lemma 1 indicates that aggregators become bottlenecks of distinguishing strength.

Original: For the equivariant aggregator, any linear transformation before aggregating and any transformation after aggregating have no contribution to the distinguishing strength.
Revised: For the equivariant aggregator, any linear transformation before aggregation and any transformation after aggregation have no contribution to the distinguishing strength.

Original: For
Revised: For

Original: SUM and MEAN, we have g ( SUM ( T · x 1 , T · x 2 , · · · , T · x n )) (cid:22) SUM ( x 1 , x 2 , · · · , x n ) and g ( MEAN ( T · x 1 , T · x 2 , · · · , T · x n )) (cid:22) MEAN ( x 1 , x 2 , · · · , x n ) , where T ∈ R m × d , and g can be any continuous function.
Revised: SUM and MEAN, we have g p SUM p T ¨ x 1 , T ¨ x 2 , ¨ ¨ ¨ , T ¨ x n qq ĺ SUM p x 1 , x 2 , ¨ ¨ ¨ , x n q and g p MEAN p T ¨ x 1 , T ¨ x 2 , ¨ ¨ ¨ , T ¨ x n qq ĺ MEAN p x 1 , x 2 , ¨ ¨ ¨ , x n q , where T P R m ˆ d , and g can be any continuous function.

Original: Based on Lemma 1, we can now compare the distinguishing strength of aggregation in some popular GNNs.
Revised: Based on Lemma 1, we can now compare the distinguishing strength of aggregations in some popular GNNs.

Original: GIN-0 sums all hidden features of neighbors at ﬁrst, and then pass them to a 2-layer MLP.
Revised: GIN-0 sums all hidden features of neighbors at ﬁrst, and then pass them to a 2-layer MLP.

Original: Therefore, when considering in a continuous input features space, the distinguishing strength of GIN-0 is at most as powerful as the SUM aggregator.
Revised: Therefore, when considering in a continuous input features space, the distinguishing strength of GIN-0 is at most as powerful as the SUM aggregator.

Original: GCN uses a NORMALIZED MEAN (denoted by n MEAN) aggregator.
Revised: GCN uses a NORMALIZED MEAN (denoted by n MEAN) aggregator.

Original: Given a node v and its neighbors, n MEAN ( v, u 1 , · · · , u n ) = 1 √ |N ( v ) | · ( h v √ |N ( v ) | + h u 1 √ |N
Revised: Given a node v and its neighbors, n MEAN p v, u 1 , ¨ ¨ ¨ , u n q “ [Equation] | N [Equation]

Original: ( u 1 ) | + · · · + h u n √
Revised: h [Equation]

Original: |N
Revised: | N

Revised: [Equation]

Original: u n ) | ) .
Revised: q| [Equation] 1 q .

Original: n MEAN is also an equivariant aggregator, and the distinguishing strength of aggregation in GCN is at most as powerful as n MEAN.
Revised: n MEAN is also an equivariant aggregator, and the distinguishing strength of aggregation in GCN is at most as powerful as n MEAN.

Original: GAT corresponds to the weighted SUM aggregation, where the weight coefﬁcients are the functions of hidden features.
Revised: GAT corresponds to the weighted SUM aggregation, where the weight coefﬁcients are the functions of hidden features.

Original: This makes the distinguishing strength of GAT and SUM incomparable.
Revised: This makes the distinguishing strength of GAT and SUM incomparable.

Original: Based on these observations, a potential approach to breaking the distinguishing strength limitation is to apply a nonlinear processing on inputs before aggregation.
Revised: Based on these observations, a potential approach to breaking the distinguishing strength limitation is to apply a nonlinear processing on inputs before aggregation.

Original: 3. B UILDING P OWERFUL A GGREGATORS
Revised: 3. B UILDING P OWERFUL A GGREGATORS

Original: In this section, we analyze the requirements for building more powerful aggregators and further injective aggregators.
Revised: In this section, we analyze the requirements for building more powerful aggregators and further injective aggregators.

Original: We ﬁrst introduce a new representation of GNN layers which uniﬁes several popular GNN layers.
Revised: We ﬁrst introduce a new representation of GNN layers which uniﬁes several popular GNN layers.

Original: Given a node v and its neighbors N ( v ) , our new formulation represents the
Revised: Given a node v and its neighbors N p v q , our new formulation represents the

Original: GNN operation as follows: 
Revised: GNN operation as follows: $

Original: [Equation]
Revised: [Equation]

Original: Here, m v ∈ R|N ( v ) |  is the aggregation coefﬁcients vector of node v .
Revised: Here, m v P R | N p v q| is the aggregation coefﬁcients vector of node v .

Original: Note that m v should be the mapping of local structures such as local topologies, node or edge features  to ensure the same encoding of isomorphic graphs.
Revised: Note that m v should be the mapping of local structures such as node degrees, node or edge features of the k -hop neighbors assigned on node v to ensure the same encoding of isomorphic graphs.

Original: ¯ h ( t − 1) T vπ = ( h ( t − 1) v , h ( t − 1) u, · · · , h ( t − 1) u | N ( v ) | ) T ∈ R 
Revised: ¯ h p t ´ 1 q T vπ “ p h p t ´ 1 q v , h p t ´ 1 q u, ¨ ¨ ¨ , h p t ´ 1 q u | N p v q| q T P R | N p v q|ˆ

Original: |N ( v ) |× d is the matrix representation of v ’s neighbors according to a permutation π .
Revised: d is the matrix representation of v ’s neighbors according to a permutation π .

Original: f NN : R d → R d  is a neural network that updates hidden feature h ( t ) v of node vwith the aggregation result r ( t ) v .
Revised: f NN : R d Ñ R d 1 is a neural network that extracts task-relevant information from the aggregated representation r p t q v , and is used to update hidden feature h p t q v of node v .

Revised: According to Equation 2, the aggregation should be with high distinguishing strength to avoid in- distinguishability among neighbors.

Revised: Meanwhile, the extraction should be powerful enough to efﬁciently extract task-relevant structural patterns from the aggregated representation of neighbors.

Original: We give this three-stage representation of GCN, GIN and GAT respectively as follows:GCN :
Revised: Based on these observations, we reformulate GCN, GIN0 and GAT with their corresponding threestage representations as follows:

Original: [Equation]
Revised: [Equation]

Original: GIN0 :
Revised: N p v q| r p t q v “ m vπ ¯ h p t ´ 1 q T vπ h p t q v “ MLP p

Original: [Equation]
Revised: r p t q v q ,

Original: [Equation]
Revised: GAT : $’& ’% m p t q v “ p att p

Original: t
Revised: h

Original: [Equation] 1) T
Revised: p t ´ 1 q v , h p t ´ 1 q v q , att p

Original: [Equation]
Revised: h p t ´ 1 q v , h p t ´ 1 q uq , ¨ ¨ ¨ , att p h p t ´ 1 q v , h p t ´ 1 q u | N p v q´ 1 | qq r p t q v “ m p t q vπ ¯ h p t ´ 1 q T vπ h p t q v “ σ p Wr p t q v ` b q .

Original: Their default formulations are given in Appendix A. In GCN, m v is the function ofnode degrees. In
Revised: Their default formulations are given in Appendix A. In the aggregation step, GCN’s m v is the mapping of neighbors’ degrees; GIN0’s m v is the mapping of

Original: GAT, m v is the function of hidden features.Both of which ensure the same encoding of isomorphic graphs.
Revised: node v ’s degree which is equivalent to SUM aggregator; GAT’s m v is the mapping of neighbors’ features.

Original: From the above representations, the only difference between GCN and GAT is aggregation coefﬁcients. Meanwhile, GCN and GAT use a 1-layer MLP in the structural/feature extraction step while GIN uses a 2-layer MLP.
Revised: All of them are the mappings of local structures as given in Equation 2.

Original: In this three-stage representation, the aggregation is reformulated as the multiplication of the aggregation coefﬁcients vector and the feature matrix of neighbors.
Revised: In this three-stage representation, the aggregation is reformulated as the multiplication of the aggregation coefﬁcients vector and the feature matrix of neighbors.

Revised: It provides insights on improving the distinguishing strength of aggregations.

Original: Next, we explain how to characterize the permutation invariance in this formulation.
Revised: First, we show how to characterize the permutation invariance in this formulation.

Original: Let M ∈ R s × n denote an aggregation coefﬁcient matrix where s ≥ 1 .
Revised: Let M P R s ˆ n denote an aggregation coefﬁcient matrix where s ě 1 .

Original: Note that in GCN, GIN and GAT, s is restricted to be 1.
Revised: Note that in GCN, GIN and GAT, s is restricted to be 1.

Original: h π ∈ R n × d is the matrix representation of n input elements according to π .
Revised: h π P R n ˆ d is the matrix representation of n input elements according to π .

Original: The aggregation computation in the second step of Equation 2 is f aggr ( M , h π ) = MP π h π=
Revised: The aggregation computation in the second step of Equation 2 is f aggr p M , h π q “ MP π h π

Original:  M π h π , where P π is the permutation matrix according to π .
Revised: “ M π h π , where P π is the permutation matrix according to π .

Original: P
Revised: P

Original: π h π ensures the same output for all h π , π ∈ Π | h | .
Revised: π h π ensures the same output for all h π , π P Π | h | .

Original: M π = MP π is the reordering of columns of M according to π .
Revised: M π “ MP π is the reordering of columns of M according to π .

Original: For any π 1 , π 2 ∈ Π n , f aggr ( M , h π 1 ) = f aggr ( M , h π 2 ) , thus permutation invariance holds.
Revised: For any π 1 , π 2 P Π n , f aggr p M , h π 1 q “ f aggr p M , h π 2 q , thus permutation invariance holds.

Original: Once M is decided, we obtain a unique aggregator denoted by f M .
Revised: Once M is decided, we obtain a unique aggregator denoted by f M .

Original: For any sequence of input elements h , f M ( h ) =
Revised: For any sequence of input elements h , f M p

Original:  f aggr ( M , h π ) , where π ∈ Π n can be any ordering of neighbors.
Revised: h q “ f aggr p M , h π q , where π P Π n can be any ordering of neighbors.

Original: Next, we analyze the distinguishing strength of f M .
Revised: Next, we analyze the distinguishing strength of f M .

Original: Proposition 1.
Revised: Proposition 1.

Original: For any two matrices M ∈ R s × n and M (cid:48) ∈ R s (cid:48) × n with s, s (cid:48) ≤ n , we have (i) f (cid:0) MM (cid:48) (cid:1) (cid:23) f M , where (cid:0) MM (cid:48) (cid:1) means stacking these two matrices; (ii) f (cid:0) MM (cid:48) (cid:1) (cid:31) f M if and only if rank ( (cid:0) MM (cid:48) (cid:1) )
Revised: For any two matrices M P R s ˆ n and M 1 P R s 1 ˆ n with s, s 1 ď n , we have (i) f ` MM 1 ˘ ľ f M , where ` MM˘ means stacking these two matrices; (ii) f ` MM 1 ˘ ą f M if and only if rank p `

Original: > rank ( M ) ; (iii) Any multiset of size n is distinguishable with f M if and only if rank ( M ) = n .
Revised: MM˘ q ą rank p M q ; (iii) Any multiset of size n is distinguishable with f M if and only if rank p M q “ n .

Original: We prove Proposition 1 in Appendix C. Proposition 1 shows that the distinguishing strength of an aggregator is decided by the rank of the corresponding M .
Revised: We prove Proposition 1 in Appendix C. Proposition 1 shows that the distinguishing strength of f M is decided by the rank of the corresponding M .

Original: Yet,  Proposition 1 is only suiable for multisets aggregated with shared f M .
Revised: Yet, the distinguishing strength analysis in Proposition 1 is only suitable for multisets aggregated with shared f M .

Original: The aggregation conducted on different nodes may use different M , also known as different aggregators, such as in GCN, GIN and GAT.
Revised: Next, we extend the analysis for the case of different aggregators. Let Res p

Original: To maximally distinguish different structures, the aggregation results using different aggregators should be different no matter what the inputs of aggregators are. This is because the aggregation results are the mapping of  local structures.
Revised: f M q denote the set of all outputs of f M . Our proposed three-stage representation also provides useful insight on the constraints among different aggregators. That is, in order to fully distinguish different local structures, for any two different f M 1 and f M 2 , Res p f M 1 qX Res p f M 2 q “

Original: For example, the aggregation results of a k -layer GNN are the mapping of k -hop neighbors assigned on each node.
Revised: H .

Original: To maximally distinguish different local structures, the aggregation operation should map them to different results.
Revised: This is because to fully distinguish different local structures, we should ensure their aggregated representations are different.

Original: Note that the aggregation coefﬁcients matrix M is limited to be the mapping of local structures  as given in Equation 2, which means different M corresponds to different local structures.
Revised: Since M is restricted to be the mapping of local structures (such as khop neighbors), different M means that the corresponding local structures are different.

Original: Therefore, the aggregation results of different aggregators must be different.
Revised: Therefore, the aggregation results of different f M must be different.

Original: However, it is not satisﬁed by existing GNNs, and there are few studies on distinguishing multisets aggregated by different aggregators.
Revised: However, it is not satisﬁed by existing GNNs, and there are few studies on distinguishing multisets aggregated by different aggregators.

Original: In Proposition 2, we present a detailed analysis of it.
Revised: In Proposition 2, we present a detailed analysis of it.

Original: Let Res ( f M ) denote the set of results aggregated by f M .

Original: Proposition 2.
Revised: Proposition 2.

Original: For any M 1 , M 2 ∈ R s × n 1 and M (cid:48) 1 , M (cid:48) 2 ∈ R s (cid:48) × n 2 , (i) Res ( f(cid:0) M
Revised: For any M 1 , M 2 P R s ˆ n 1 and M 1 1 , M 1 2 P R s 1 ˆ n 2 , (i) Res p f

Original:  M (cid:48) 1 (cid:1) ) ∩{+M 
 [Equation]+}
Revised: ` M {+M 
 [Equation]+}

Original: Res ( f (cid:0) M2 M (cid:48) 2 (cid:1) ) ⊆
Revised: Res p [Equation] M

Original:  Res ( f M 1 ) ∩ Res ( f M 2 ) ; (ii)
Revised: [Equation] Res p f M 2 q , then rank p

Original: If Res ( f (cid:0) M 1 M (cid:48) (cid:1) ) ∩ Res ( f (cid:0) M 2 M (cid:48) (cid:1) )
Revised: [Equation]

Original: ⊂ Res ( f M 1 ) ∩ Res ( f M 2 ) , then rank ( (cid:0) M 1 M 2 M (cid:48) 1 M (cid:48) 2 (cid:1) ) > rank ( (cid:0) M 1 M 2 (cid:1) ) ; We prove Proposition 2 in Appendix D.  Next, we provide a sufﬁcient condition for building desired multiple injective aggregators with the outputs having no intersections.
Revised: We prove Proposition 2 in Appendix D. Proposition 2 shows the necessity of preserving the rank of aggregation coefﬁcient matrix when considering the distinguishing strength among different aggregators. Next, we provide a sufﬁcient condition for building desired multiple injective aggregators with the outputs having no intersections.

Original: Proposition 3.
Revised: Proposition 3.

Original: For any two aggregators f M 1 and f M 2 with M 1 ∈ R s × n 1 and M 2 ∈ R s × n 2 , if rank ( (cid:0) M 1 M2 (cid:1) )
Revised: For any two aggregators f M 1 and f M 2 with M 1 P R s ˆ n 1 and M 2 P R s ˆ n 2 , if rank p ` M 1 M

Original: = n 1 + n 2 , then f M 1 and f M 2 are injective and Res ( f M 1 ) ∩{+
+}
Revised: [Equation] ˘ q “ n 1 ` n 2 , then f M 1 and f M 2 are injective and Res p f M {+
+}

Original:  Res ( f M 2 ) = ∅ .
Revised: q X Res p f M 2 q “ H .

Original: We prove Proposition 3 in Appendix E. Proposition 1, 2 and 3 provide a new perspective for building powerful aggregators and even injective aggregators.
Revised: We prove Proposition 3 in Appendix E. Proposition 1, 2 and 3 provide a new perspective for building powerful aggregators and even injective aggregators.

Original: Compared with the distinguishing strength studies in Xu et al.

Original: (2019) and Corso et al. (2020), as well as existing strategies for building injective aggregators, e.g., sum-of-power mapping Zaheer et al. (2017) and normalized moments Corso et al. (2020), we reformulate the aggregation with aggregation coefﬁcients matrix and show the relations of the distinguishing strength of aggregators and the rank of the corresponding aggregation coefﬁcients matrices.
Revised: Compared with the distinguishing strength studies in (Xu et al., 2019) and (Corso et al., 2020), as well as existing strategies for building injective aggregators, e.g., sum-of-power mapping (Zaheer et al., 2017) and normalized moments (Corso et al., 2020), we reformulate the aggregation with aggregation coefﬁcients matrix and show the relations of the distinguishing strength of aggregators and the rank of the corresponding aggregation coefﬁcients matrices.

Original: Besides, the aggregation of this method is controlled by aggregation coefﬁcients which can be learned from graph data to better leverage structural information.
Revised: Besides, the aggregation of this method is controlled by aggregation coefﬁcients which can be learned from graph data to better leverage structural information.

Original: In this paper, to simplify the analysis, we only consider the aggregations within one-hop neighbors.
Revised: In this paper, to simplify the analysis, we only consider the aggregations within one-hop neighbors.

Original: The results can be easily extended to more sophisticated aggregators with the overall framework unchanged
Revised: The results can be easily extended to more sophisticated aggregators with the overall framework unchanged

Original: In the perspective of preserving the rank of hidden features among neighbors, r = Mh indicates that rank ( r ) < min ( rank ( M ) , rank ( h )) .
Revised: In the perspective of preserving the rank of hidden features among neighbors, r “ Mh indicates that rank p r q ď min p rank p M q , rank p h qq .

Original: To preserve the rank of hidden features in aggregations such that rank ( r ) = rank ( h ) , we need rank ( M ) ≥ rank ( h ) .
Revised: To preserve the rank of hidden features in aggregations such that rank p r q “ rank p h q , we need rank p M q ě rank p h q .

Original: This builds a connection between improving the distinguishing strength of aggregators and preserving the rank of hidden features among neighbors, both of which have the requirements on the rank of M .
Revised: This builds a connection between improving the distinguishing strength of aggregators and preserving the rank of hidden features among neighbors, both of which have the requirements on the rank of M .

Original: General aggregators such as ones in GCN, GIN-0 and GAT have rank ( M ) = 1 .
Revised: General aggregators such as ones in GCN, GIN-0 and GAT have rank p M q “ 1 .

Original: Thus, rank ( r ) is always ﬁxed to 1 no matter what the rank of the input features is.
Revised: Thus, rank p r q is always ﬁxed to 1 no matter what the rank of the input features is.

Original: Correspondingly, they have a weak distinguishing strength.
Revised: Correspondingly, they have a weak distinguishing strength.

Original: Equation 2 splits the aggregation and feature/structure extraction into two independent steps, which helps to ﬁgure out that the expressive power loss happens in the aggregation step, and then the model extracts feature/structure information on the distorted encodings of neighbors.
Revised: Equation 2 splits the aggregation and feature/structure extraction into two independent steps, which helps to ﬁgure out that the expressive power loss happens in the aggregation step, and then the model extracts feature/structure information on the distorted encodings of neighbors.

Original: From Equation 2, the aggregation can be considered as a representation regularization step, which uniﬁes different multisets of neighbors into the same representation style while holding permutation invariance.
Revised: From Equation 2, the aggregation can be considered as a representation regularization step, which uniﬁes different multisets of neighbors into the same representation style while holding permutation invariance.

Original: Then, the model can extract structural information on this regulated data with a shared trainable matrix as the third step in Equation 2.
Revised: Then, the model can extract structural information on this regulated data with a shared trainable matrix as the third step in Equation 2.

Original: Based on this observation, we propose two novel GNN layers:
Revised: Based on this observation, we propose two novel GNN layers:

Original: ExpandingConv and CombConv.
Revised: and CombConv.

Original: 3. E XPANDING C ONV
Revised: 3. E XPANDING C ONV

Original: We ﬁrst give ExpandingConv framework.
Revised: In this section, we ﬁrst present ExpandingConv framework.

Original: Then we provide one of its implementations and analyze how ExpandingConv achieves more powerful aggregations.
Revised: Then we provide one of its implementations and analyze how ExpandingConv achieves more powerful aggregations.

Original: The ExpandingConv framework is
Revised: The ExpandingConv framework is

Original: [Equation]
Revised: [Equation]

Original: where m ( t ) uv ∈ R s × 1 with s > 1 and f local ( u, v ) is the mapping of local structures between nodes u and v .
Revised: where m p t q uv P R s ˆ 1 with s ą 1 and f local p u, v q is the mapping of local structures between nodes u and v .

Original: The implementation of f local ( u, v ) is very ﬂexible with the only restriction of ensuring the same encoding of isomorphic graphs.
Revised: The implementation of f local p u, v q is very ﬂexible with the only restriction of ensuring the same encoding of isomorphic graphs.

Original: vec ( m ( t) uv h ( t − 1) T u ) ∈
Revised: vec p m p t

Original:  R sd × 1 is the expanded representation of hidden features h ( t − 1) u∈ R d
Revised: q uv h p t ´ 1 q T u q P R sd ˆ 1 is the expanded representation of hidden features h p t ´ 1 q u

Original: ×
Revised: P R d ˆ 1 .

Original: Then a GNN layer f aggr learns structural information on this expanded representations.
Revised: Then a GNN layer f aggr learns structural information on this expanded representations.

Original: We introduce an implelentation as follows:
Revised: We introduce an implelentation as follows:

Original: [Equation]
Revised: [Equation]

Original: In Equation 3, we implement f local ( u, v ) as the function of hidden features of nodes u and v .
Revised: In Equation [Equation] we implement f local p u, v q as the function of hidden features of nodes u and v .

Original: There can be other implementations, and we leave them for future work.
Revised: There can be other implementations, and we leave them for future work.

Original: W ∈ R s × 2 d and b ∈ R s × 1 are trainable matrices.
Revised: W P R s ˆ 2 d and b P R s ˆare trainable matrices.

Revised: (Luan et al., 2019) empirically showed that different nonlinear activatoin functions have different contributions in preserving the rank of matrices.

Original: We use  Tanh as the activation function in the computation of m ( t ) uv to  preserve the rank of aggregation coefﬁcient matrices Luan et al.
Revised: We use the recommended Tanh as the activation function in the computation of m p t q uv to better preserve the rank of aggregation coefﬁcient matrices.

Original: MLP denotes a 2-layer perceptron.
Revised: MLP denotes a 2-layer perceptron.

Original: Next, we represent Equation 3 with the corresponding three-stage representation as given in Section 3.2 to obtain its aggregation coefﬁcient matrix and analyze its distinguishing strength.
Revised: Next, we represent Equation 3 with the corresponding three-stage representation as given in Section 3.2 to obtain its aggregation coefﬁcient matrix and analyze its distinguishing strength.

Original: To simplify this process, we only consider 1-layer MLP with W (cid:48)
Revised: To simplify this process, we only consider 1-layer MLP with W 1 P R d ˆ sd and b 1 P R d ˆ 1 . [Equation]

Original: [-∈ R d × sd and b (cid:
-]
Revised: [-∈ R d × sd and b (cid:
-][Table] ř [Table]

Original: ) ∈ R d × 1 .
Revised: [Table] ř [Table]

Original: h
Revised: m

Original: [Equation]
Revised: [Table]

Revised: ˛ ‹‹‹‹‹‹‚ “ ¨ ˚˚˚˚˚˚˝

Original: t
Revised: W

Revised: [Table]

Revised: W 1r

Original: [Equation]
Revised: [Table]

Original: = (cid:88)
Revised: W 1r

Original: [Equation]
Revised: [Table]

Original:  
Revised: ˛ ‹‹‹‹‹‹‚ ¨ ˚˚˚˚˚˚˝

Original: [Equation]
Revised: [Table]

Revised: q

Original: [Equation]
Revised: [Table]

Revised: q

Original: [Equation]
Revised: [Table]

Original: 
Revised: q ˛ ‹‹‹‹‹‹‚

Original: [Equation]
Revised: [Table]

Original:   =  
Revised: ¨ ˚˚˚˚˚˚˝

Original: [Equation]
Revised: [Table] | [Table]

Original:    
Revised: ˛ ‹‹‹‹‹‹‚ “ ¨ ˚˚˚˚˚˚˝ W 1r

Original: [Equation]
Revised: [Table] [Table]

Original: where N i ( v ) ⊆
Revised: W 1r 2 , [Table] W 1r d, [Table]

Revised: ˛ ‹‹‹‹‹‹‚ ¨ ˚˚˚˚˚˚˝

Revised: [Table]

Revised: M

Revised: [Table]

Revised: [Table]

Revised: ¨ ˚˚˚˚˚˚˝

Revised: [Table]

Original: N ( v ) | i ∈
Revised: | [Table] [Equation] | N

Revised: [Equation]

Revised: [Equation]

Revised: (4) where N i p v q Ď

Revised: N p v q| i

Original: [ d ] are sampled subsets of neighbors in each dimension. 
Revised: P r d s are sampled subsets of neighbors in each dimension. M

Original: [Equation]
Revised: [Equation]

Original: h ( t − 1) u
Revised: [Equation] , m [Equation]

Original: [Equation]
Revised: [Equation]

Original: h
Revised: m

Original: [Equation]
Revised: [Equation]

Original: t
Revised: q

Revised: P

Revised: R s ˆ| N i p v q| and ¯ h p t

Original: [Equation]
Revised: [Equation]

Original: ) u, · · · , h ( t − 1) u |N i ( v ) | ) ∈ R d ×|N i ( v ) | are aggregation coefﬁcients matrix and hidden feature matrix corresponding to the subset of neighbors
Revised: 1 q v i “ p h p t ´ 1 q u, h p t ´ 1 q u, ¨ ¨ ¨ , h p t ´ 1 q u | N i p v q| q P R d ˆ| N i p v q| are aggregation coefﬁcients matrix and hidden feature matrix corresponding to the subset of neighbors

Original: N i ( v ) ⊆ N ( v ) according to π i .
Revised: N i p v q Ď N p v q according to π i .

Original: We denote [ h ( t − 1) v ||
Revised: We denote r h p t ´ 1 q v ||

Original: ¯ h ( t − 1) v i ] =
Revised: ¯ h p t ´ 1 q

Original: [Equation]
Revised: v

Original: [Equation]
Revised: i s “ p h p t ´ 1 q v || h p t ´ 1 q u, h p t ´ 1 q v

Original: ||
Revised: ||

Original: [Equation]

Original: t
Revised: h

Original: [Equation] 1) u [Equation]
Revised: p t ´ 1 q u, ¨ ¨ ¨ , h p t ´ 1 q

Original: [Equation]
Revised: v

Original: |
Revised: ||

Original: [Equation] [Equation] ( t [Equation]
Revised: h

Original: [Equation]

Original: ( t [Equation] 1) u |N i
Revised: p t ´ 1 q u |

Original: [Equation] v ) | ) ∈ R 2 d ×|N i ( v ) | , then
Revised: N i p v q| q P R 2 d ˆ| N i p v q| , then

Original: M ( [Equation] [Equation] =
Revised: M p t q v i “

Original: Tanh ( W[
Revised: Tanh p W

Original: [Equation]
Revised: r h p t ´ 1 q

Original: [Equation]
Revised: v

Original: || ¯ h ( t 
Revised: || ¯ h p t ´ 1 q

Original: [Equation] [Equation]
Revised: v

Original: + b ) ∈ R s ×|N i ( v ) | .
Revised: i s ` b q P R s ˆ| N i p v q| .

Original: According to Equation 4, we ﬁnally obtain the three-stage representation equivalent to Equation 3. 
Revised: According to Equation 4, we ﬁnally obtain the three-stage representation equivalent to Equation 3. $

Original: [Equation]
Revised: [Equation]

Original: According to the computation of ¯ r ( t ) v i , rank (¯ r ( t ) v i ) ≤ min ( s, d, |N i ( v ) | ) .
Revised: According to the computation of ¯ r p t q v i , rank p ¯ r p t q v i q ď min p s, d, | N i p v q|q .

Original: By conﬁguring a larger s , we have rank (¯ r ( t ) v i ) > 1 with a high probability, which is different from general GNNs with a rank of 1.
Revised: By conﬁguring a larger s , we have rank p ¯ r p t q v i q ą 1 with a high probability, which is different from general GNNs with a rank of 1.

Original: As analyzed in Section 3.2, this achieves more powerful aggregators as well as preserves the rank of hidden features among neighbors.
Revised: As analyzed in Section 3.2, this achieves more powerful aggregators as well as preserves the rank of hidden features among neighbors.

Original: The obtained ¯ r ( t ) v i after aggregation is the uniﬁed representations of neighbors.
Revised: The obtained ¯ r p t q v i after aggregation is the uniﬁed representations of neighbors.

Original: We then use the trainable matrix W (cid:48) ∈ R sd × d to extracts feature/structure information.
Revised: We then use the trainable matrix W 1 P R sd ˆ d to extracts feature/structure information.

Original: Unlike the aggregation step, the dimensions reduction here (from sd to d ) would not cause information loss.
Revised: Unlike the aggregation step, the dimensions reduction here (from sd to d ) would not cause information loss.

Original: This can be explained  that only task-relevant structural information needs to be preserved and passed to the next layer, and it can be embedded in lower dimensions.
Revised: This can be explained by the fact that only task-relevant structural information needs to be preserved and passed to the next layer, and it can be embedded in lower dimensions.

Original: Comparisons with multi-head GAT.
Revised: Comparisons with multi-head GAT.

Original: Proposition 4.
Revised: Proposition 4.

Original: Multi-head GAT is an implementation of ExpandingConv as follows:
Revised: Multi-head GAT is an implementation of ExpandingConv as follows:

Original: [Equation]
Revised: [Equation]

Original: where W = || Kk =1 W k ∈ R kd × d is the concatenation of the trainable matrix in all K heads.
Revised: where W “ || Kk “ 1 W k P R kd ˆ d is the concatenation of the trainable matrix in all K heads.

Original: We prove Proposition 4 in Appendix F.
Revised: We prove Proposition 4 in Appendix F.

Original: Although multi-head GAT is based on attention mechanism, ExpandingConv provides a new perspective to explain its effectiveness.
Revised: Although multi-head GAT is based on attention mechanism, ExpandingConv provides a new perspective to explain its effectiveness.

Original: Applying multi-head attention mechanism helps to preserve the rank of hidden features as well as achieve more powerful aggregators, leading to less distortion among neighbors.
Revised: Applying multi-head attention mechanism helps to preserve the rank of hidden features as well as achieve more powerful aggregators.

Revised: However, the usage of LeakyReLU may be harmful to preserving the rank of the aggregation coefﬁcient matrix (Luan et al., 2019).

Original: GAT as well as most other GNNs (such as GCN, GIN, etc) follows the same pattern that applies nonlinear units after aggregation.
Revised: GAT as well as most other GNNs (such as GCN, GIN, etc) follows the same pattern that applies nonlinear units after aggregation.

Original: According to the analysis in Section 3.1, Equation 3 applies MLP on vec ( m ( t) uv h
Revised: According to the analysis in Section 3.1, Equation 3 applies MLP on vec p m p t

Original: ( t − 1) T u ) before SUM to break the distinguishing strength limitation of SUM.
Revised: q uv h p t ´ 1 q T u q before SUM to break the distinguishing strength limitation of SUM.

Original: It also produces other interesting results.
Revised: It also produces other interesting results.

Original: By reformulating Equation 3 with its three-stage representation as Equation 5, each dimension of hidden features aggregates on a subset of neighbors independently, which corresponds to a kind of dimension-wise neighbor sampling mechanism.
Revised: By reformulating Equation 3 with its three-stage representation as Equation 5, each dimension of hidden features aggregates on a subset of neighbors independently, which corresponds to a kind of dimension-wise neighbor sampling mechanism.

Original: We call the modiﬁcation of applying ReLU ahead of SUM aggregator as Re -SUM mechanism.
Revised: We call the modiﬁcation of applying ReLU ahead of SUM aggregator as Re -SUM mechanism.

Original: Mishra et al. (2020) and Rong et al. (2019) studied dropedge and node masking mechanism on node-level predictions, both of which can be considered as neighbor sampling strategies that have shown their effectiveness in improving the generalization ability of aggregation-based GNNs and are also used as unbiased data augmentation technique for training.
Revised: (Mishra et al., 2020) and (Rong et al., 2019) studied dropedge and node masking mechanism on node-level predictions, both of which can be considered as neighbor sampling strategies that have shown their effectiveness in improving the generalization ability of aggregation-based GNNs and are also used as unbiased data augmentation technique for training.

Original: Compared with dropedge and node masking, Re -SUM realizes a dimension-wise neighbor sampling. and it does not need to manually set the sampling ratio since this mechanism takes effects implicitly.
Revised: Compared with dropedge and node masking, Re -SUM realizes a dimension-wise neighbor sampling, and it does not need to manually set the sampling ratio since this mechanism takes effects implicitly.

Original: Re -SUM shows that the neural network itself can perform sampling by properly combining nonlinear units and aggregators, without explicitly modifying the network architecture.
Revised: Re -SUM shows that the neural network itself can perform sampling by properly combining nonlinear units and aggregators, without explicitly modifying the network architecture.

Original: Our experimental results veriﬁed the effectiveness of the Re -SUM on a variety of graph tasks.
Revised: Our experimental results veriﬁed the effectiveness of the Re -SUM on a variety of graph tasks.

Original: 3. C OMB C ONV
Revised: 3. C OMB C ONV

Original: The CombConv framework is 
Revised: The CombConv framework is #

Original: [Equation]
Revised: [Equation]

Original: where m ( t ) uv ∈ R d × 1 and (cid:12) denotes element-wise product.
Revised: where m p t q uv P R d ˆ 1 and d denotes element-wise product.

Original: An implementation of CombConv is given as follows:
Revised: An implementation of CombConv is given as follows:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R d × 2 d and b ∈ R d × 1 .
Revised: where W P R d ˆ 2 d and b P R d ˆ 1 .

Original: Similar to ExpandingConv, CombConv also applies Re -SUM aggregation.
Revised: Similar to ExpandingConv, CombConv also applies Re -SUM aggregation.

Original: The difference is that each dimension of hidden features is aggregated with an independent weighted aggregator, which works like a comb.
Revised: The difference is that each dimension of hidden features is aggregated with an independent weighted aggregator.

Original: ExpandingConv with s = 1 corresponds to a special case of CombConv where all dimensions share the same aggregator.
Revised: ExpandingConv with s “ 1 corresponds to a special case of CombConv where all dimensions share the same aggregator.

Original: Therefore, the distinguishing strength of CombConv is stronger than ExpandingConv with s = 1 .
Revised: Therefore, the distinguishing strength of CombConv is stronger than ExpandingConv with s “ 1 .

Original: Meanwhile, CombConv does not expand the hidden features of nodes in aggregation.
Revised: Meanwhile, CombConv does not expand the hidden features of nodes in aggregation.

Original: Hence, it requires fewer parameters.
Revised: Hence, it requires fewer parameters.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate ExpandingConv and CombConv on graph-level prediction tasks onOGB
Revised: In this section, we evaluate ExpandingConv and CombConv on graph-level prediction tasks on

Original: Weihua Hu (2020) and QM9 Ramakrishnan et al.
Revised: OGB (Weihua Hu, 2020), TU (Kersting et al., 2016; Yanardag & Vishwanathan, 2015) and QM

Original: ; Wu et al.
Revised: Conﬁgurations. We use the default dataset splits for OGB. The QM9 dataset is randomly split into 80% train, 10% validation and 10% test as given in (Morris et al., 2019; Maron et al., 2019). For TU dataset, we follow the standard 10-fold cross validation protocol and splits from (Zhang et al., 2018) and report our results following the protocol described in (Xu et al., 2019;

Original: Ruddigkeit et al.
Revised: Ying et al., 2018). We use the concatenation of hidden features from all layers to compute the entire graph representations (Xu et al., 2018). In our tests, all models are equipped with batch normalization (Ioffe & Szegedy, 2015) on each hidden layer when evaluating on OGB and TU, and are not when evaluating on QM9.

Original: All datasets’ descriptions and detailed experimental setup are given in Appendix H.
Revised: All datasets’ descriptions and detailed hyperparameter settings are given in Appendix H.

Original: The code is available at https://github.com/qslim/epcb-gnns . We ﬁrst conduct comprehensive ablation studies to evaluate the effectiveness of ExpandingConv, CombConv and Re -SUM mechanism  as given in Table 1.
Revised: We ﬁrst conduct comprehensive ablation studies to evaluate the effectiveness of powerful aggregators and Re -SUM mechanism on OGB and QM9 as given in Table 1 and Table 2. Then, we compare the performance of ExpandingConv and CombConv with competitive baselines on all three datasets as given in Table 3 and Table 4 to show their improvements.

Original: ExpC- s denotes ExpandingConv with
Revised: ExpC- s denotes ExpandingConv with

Original: W ∈ R s × (cid:63) .
Revised: W P R s ˆ‹ .

Original: We use ExpC* and CombC* to denote the ExpandingConv and CombConv without
Revised: We use ExpC* and CombC* to denote the ExpandingConv and CombConv without

Original: Re -SUM.
Revised: Re -SUM.

Revised: [Table]

Revised: [Table]

Revised: U

Revised: [Table]

Revised: 4. A BLATION S TUDIES

Original: Effect of powerful aggregators.
Revised: Effect of powerful aggregators.

Original: For complex graphs structures with dense connections or with abundant node/edge features, they would beneﬁt from a higher expressive model to maximumly distinguish different structures and extract relevant structural patterns as the model goes deeper to leverage large receptive ﬁelds.
Revised: For complex graph structures with dense connections or with abundant node/edge features, they would beneﬁt from a higher expressive model to maximumly distinguish different structures and extract relevant structural patterns as the model goes deeper to leverage large receptive ﬁelds.

Original: This is validated on both QM9 and OGB.
Revised: This is validated on both QM9 and OGB.

Original: We conﬁg s = 1 , 4 , 8 , 16 ,of ExpC- s for all 12 targets of QM9.
Revised: We conﬁgure s “ 1 , 4 , 8 , 16 , 32 of ExpC- s for all 12 targets of QM9.

Original: As we apply  larger s , the model continuously achieves better performance on most targets.
Revised: As we apply a larger s , the model continuously achieves better performance on most targets.

Original: We randomly select s = 1 , 4 for ogbg-ppa, ogbg-molhiv and s = 1 ,for ogbg-code.
Revised: We randomly select s “ 1 , 4 for ogbg-ppa, ogbgmolhiv and s “ 1 , 5 for ogbg-code.

Original: The results show that applying larger s gains performance improvements, especially on ogbg-ppa which involves large graphs with dense connections.
Revised: The results show that applying larger s gains performance improvements, especially on ogbg-ppa which involves large graphs with dense connections.

Original: Effect of Re -SUM mechanism.
Revised: Effect of Re -SUM mechanism.

Original: In Table 1, the performance differences between ExpC*- (CombC*) and ExpC-1 (CombC) show the effectiveness of Re -SUM.
Revised: In Table 1 and Table 2, the performance differences between ExpC*-1 (CombC*) and ExpC-1 (CombC) show the effectiveness of Re -SUM.

Original: In our tests, theRe -SUM
Revised: In our tests, the

Original: [Table] can be extremely powerful on graphs with dense connections such as ogbg-ppa [Table] which is validated on both ExpandingConv (with 6.85% improvements) and CombConv (with 4% improvements).
Revised: Re -SUM can be extremely powerful on graphs with dense connections such as ogbg-ppa, which is validated on both ExpandingConv (with 6.85% improvements) and CombConv (with 4% improvements).

Original: On most targets of QM
Revised: On most targets of QM9, this mechanism also gains improvements.

Original: [Table]
Revised: For small graphs with sparse connections such as ogbg-hiv and ogbg-molpcba, the improvements are not very signiﬁcant.

Original: this mechanism also gains improvements
Revised: Table 3 and Table 4 show the performance comparisons of our models with baselines on QM9, TU and OGB respectively. All datasets in QM9 and OGB graph-level predictions are used for evaluations. For TU, we use 3 widely used datasets: COLLAB includes graphs with dense connections;

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparisons with baselines.
Revised: REDDIT-BINARY (RDT-B) and REDDIT-MULTI-12K (RDT-M12) include large and sparse graphs with one center node having dense connections with other nodes.

Original: Table 2 shows the performance comparisons of our models with baselines evaluated on QM9 and OGB respectively. All results of baselines are taken from the original papers except for GIN0 on QM9.
Revised: All results of baselines are taken from the original papers except for the results of GraphSAGE on TU, multi-head GAT on OGB and GIN0* on QM9 which were not reported by the original papers. We report the results of GraphSAGE provided by (Ying et al., 2018) and evaluate multi-head GAT and GIN0* by ourselves.

Original: To ensure a fair comparison,we implement GIN0 without batch normalization denoted by GIN0*. For ExpandingConv, we set s = 4 for ogbg-ppa andogbg-molhiv; s = 5 for ogbg-code; s ∈ { 4 , 8 , 16 , 32 } for all 12 targets in QM9.
Revised: To ensure a fair comparison, for OGB and TU, we conﬁgure the number of heads in multi-head GAT and s in ExpC- s to be the same which is selected in t 3 , 4 , 5 u . For QM9, the number of heads isand s P t 4 , 8 , 16 , 32 u . GIN0* in QM9 denotes GIN0 without batch normalization.

Original: Note that our models achieve the lowest error on 7 out of the 12 targets ofQM9.
Revised: Compared with baselines, our models achieve the best performance on 7 out of all 12 targets of

Original:  On ogbg-ppa, ExpandingConv and CombConv achieve 2.6%and 0.7% higher classiﬁcation accuraciesrespectively compared with SOTA models.
Revised: QM9, 3 out of all 4 graph-level prediction datasets of OGB and all 3 selected TU datasets. Our models get 1.9% improvements on COLLAB and 2.83% improvements on REDDIT-MULTI-12K compared with SOTA baselines. On ogbg-ppa, our models achieve 2.6% higher classiﬁcation accuracies compared with SOTA baselines.

Original: On ogbg-code, they achieve 1.5%and 1.1% improvements. 
Revised: On ogbg-code, they achieve 1.5% improvements. Multi-head

Revised: GAT can also be considered as an implementation of ExpandingConv.

Revised: However, its performance on graph-level predictions is not competitive.

Revised: According to its three-stage representation, the usage of LeakyReLU in the aggregation step is harmful to preserving the rank, and the usage of softmax makes it harder to analyze the rank.

Revised: In the extraction step, the 1-layer MLP may have a limited representation power to represent the desired extraction functions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We show how basic aggregators used in general GNNs become expressive bottlenecks.
Revised: We show how basic aggregators used in general GNNs become expressive bottlenecks.

Original: To address this limitation, we develop theoretical foundations of building powerful aggregators.
Revised: To address this limitation, we develop theoretical foundations of building powerful aggregators.

Original: We also propose the Re -SUM mechanism which achieves dimension-wise sampling.
Revised: We also propose the Re -SUM mechanism which achieves dimension-wise sampling.

Original: To evaluate their effectiveness, we develop two novel GNN layers, and conduct extensive experiments on public graph benchmarks.
Revised: To evaluate their effectiveness, we develop two novel GNN layers, and conduct extensive experiments on public graph benchmarks.

Original: The results are consistent with our analysis, and our proposed models achieve SOTA performance on a variety of graph-level prediction benchmarks.
Revised: The results are consistent with our analysis, and our proposed models achieve SOTA performance on a variety of graph-level prediction benchmarks.

Original: A GCN, GAT AND GIN
Revised: A GCN, GAT AND GIN

Original: Here, we present implementations of GCN, GAT and GIN for the usage of our analysis.
Revised: Here, we present implementations of GCN, GAT and GIN for the usage of our analysis.

Original: Graph Convolution Networks (GCN) Kipf & Welling (2016).
Revised: Graph Convolution Networks (GCN) (Kipf & Welling, 2016).

Original: [Equation]
Revised: [Equation]

Original: Graph Attention Networks (GAT) 
Revised: Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017).

Original: [Equation]
Revised: [Equation]

Original: Graph Isomorphism Networks (GIN-0) Xu et al.
Revised: Graph Isomorphism Networks (GIN-0) (Xu et al., 2019).

Original: [Equation]
Revised: [Equation]

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: (i)
Revised: (i)

Original: For any two multisets x 1 and x 2 , if g ( f aggr ( x 1 )) (cid:54) = g ( f aggr ( x 2 )) , then f aggr ( x 1 ) (cid:54) = f aggr ( x 2 ) .
Revised: For any two multisets x 1 and x 2 , if g p f aggr p x 1 qq ‰ g p f aggr p x 2 qq , then f aggr p x 1 q ‰ f aggr p x 2 q .

Original: Therefore, we have f aggr (cid:23) g ◦ f aggr .
Revised: Therefore, we have f aggr ľ g ˝ f aggr .

Original: If g is injective, then f aggr ( x 1 ) (cid:54) = f aggr ( x 2 ) ⇒ g ( f aggr ( x 1 )) (cid:54) =
Revised: If g is injective, then f aggr p x 1 q ‰ f aggr p x 2 q ñ g p f aggr p x 1 qq ‰ g p f aggr p x 2 qq .

Original: g ( f aggr ( x 2 )) . We have f aggr (cid:23) g ◦ f aggr (cid:23) f aggr , therefore f aggr = g ◦ f aggr .
Revised: We have f aggr ľ g ˝ f aggr ľ f aggr , therefore f aggr “ g ˝ f aggr .

Original: (ii)
Revised: (ii)

Original: For any two multisets x 1 and x 2 , f aggr 1 ( x 1 ) (cid:54) = f aggr 1 ( x 2 ) ⇒ [ f aggr 1 ( x 1 ) ||
Revised: For any two multisets x 1 and x 2 , f aggr 1 p x 1 q ‰ f aggr 1 p x 2 q ñ r f aggr 1 p x 1 q|| f aggr 2 p x 1 qs ‰

Original:  f aggr2 ( x 1 )] (cid:54) = [ f aggr 1 ( x 2 ) || f aggr 2 ( x 2 )] .
Revised: r f aggr 1 p x 2 q|| f aggr 2 p x 2 qs .

Original: Therefore, f aggr 1 ⊗ f aggr 2 (cid:23) f aggr 1 .
Revised: Therefore, f aggr 1 b f aggr 2 ľ f aggr 1 .

Original: If f aggr 1 and f aggr 2 are incomparable, there exist x 3 and x 4 such that f aggr 2 ( x 3 ) (cid:54) = f aggr 2 ( x 4 ) but f aggr 1 ( x 3 ) = f aggr 1 ( x 4 ) .
Revised: If f aggr 1 and f aggr 2 are incomparable, there exist x 3 and x 4 such that f aggr 2 p x 3 q ‰ f aggr 2 p x 4 q but f aggr 1 p x 3 q “ f aggr 1 p x 4 q .

Original: Therefore, there exist x 3 and x 4 such that [ f aggr 1 ( x 3 ) || f aggr 2 ( x 3 )] (cid:54) = [ f aggr 1 ( x 4 ) ||
Revised: Therefore, there exist x 3 and x 4 such that r f aggr 1 p x 3 q|| f aggr 2 p x 3 qs ‰ r f aggr 1 p x 4 q|| f aggr 2 p x 4 qs but f aggr 1 p x 3 q “ f aggr 1 p x 4 q .

Original: f aggr2 ( x 4 )] but f aggr 1 ( x 3 ) = f aggr 1 ( x 4 ) . f aggr 1 ⊗ f aggr 2 (cid:31) f aggr 1 .
Revised: f aggr 1 b f aggr 2 ą f aggr 1 .

Original: (iii) Since f aggr is an equivariant aggregator, then f aggr ( T · x 1 , T · x 2 , · · · , T · x n ) = T · f aggr ( x 1 , x 2 , · · · , x n ) (cid:22) f aggr ( x 1 , x 2 , · · · , x n ) .
Revised: (iii) Since f aggr is an equivariant aggregator, then f aggr p T ¨ x 1 , T ¨ x 2 , ¨ ¨ ¨ , T ¨ x n q “ T ¨ f aggr p x 1 , x 2 , ¨ ¨ ¨ , x n q ĺ f aggr p x 1 , x 2 , ¨ ¨ ¨ , x n q .

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: (i)
Revised: (i)

Original: [Equation]

Original: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: then for any x 1 and x 2 , we have
Revised: then for any x 1 and x 2 , we have

Original: [Equation]
Revised: [Equation]

Original: and therefore we conclude that f (cid:0) MM (cid:48) (cid:1) (cid:23) f M .(ii) “ f (cid:0) MM
Revised: and therefore we conclude that f ` MM 1 ˘ ľ f M .

Original: [Table]
Revised: [Equation]

Original: (cid:1)
Revised: We prove the claim by contradiction.

Original: [Table] f M ⇐
Revised: Assume that rank p `

Original: [Table]
Revised: [Equation]

Original: ( (cid:0) MM (cid:48) [Table] > rank ( M
Revised: ˘ q ą rank p M q and f ` MM 1 ˘ “ f M

Original: [Table]
Revised: [Equation]

Original: We prove the claim by contradiction.

Original: Assume that rank ( (cid:0) MM (cid:48) (cid:1) ) > rank ( M ) and f (cid:0) MM (cid:48) (cid:1) =
Revised: f ` MM 1 ˘ “ f M means that for any x 1 and x 2 , m π x 1 π “ m π x 2 π ô ` [Equation]

Original: f M .
Revised: 1 π

Original: f (cid:0) MM (cid:48) (cid:1) =
Revised: [Equation] π x 2 π [Equation]

Original: f M means that for any x 1 and x 2 , m π x 1 π = m π x 2 π ⇔ (cid:0) MM (cid:48) (cid:1) π x 1 π = (cid:0) MM (cid:48) (cid:1) π x 2 π , where π is the ordering of input elements.Let s =
Revised: where π is the ordering of input elements.

Original:  x 1 π − x 2 π .
Revised: Let s “ x 1 π ´ x 2 π .

Original: For any i ∈ [ n ] , s [ i ] = x 1 π
Revised: For any i P r n s , s r i s “ x 1 π

Original: [ i ] − x 2 π [ i ] ∈ R .
Revised: r i s ´ x 2 π r i s P R .

Original: Then for any s ∈ R n , m π s = 0 ⇔ (cid:0) MM (cid:48) (cid:1) π s = 0 .
Revised: Then for any s P R n , m π s “ 0 ô ` MM˘ π s “ 0 .

Original: The system of linear equations m π x = 0 and (cid:0) M M (cid:48) (cid:1) π x = 0 share the same solution space.
Revised: The system of linear equations m π x “ 0 and ` MM˘ π x “ 0 share the same solution space.

Original: Let R S denote the rank of this solution space, then rank ( m π ) + R S = rank ( (cid:0) MM (cid:48) (cid:1) π ) + R S = n .
Revised: Let R S denote the rank of this solution space, then rank p m π q ` R S “ rank p ` MM˘ π q ` R S “ n .

Original: Therefore, rank ( m π ) = rank ( (cid:0) MM (cid:48) (cid:1) π ) , then we have rank ( M ) =
Revised: Therefore, rank p m π q “ rank p ` MM˘ π q , then we have rank p M q “ rank p `

Original: rank ( (cid:0) MM (cid:48) (cid:1) ) .
Revised: MM˘ q .

Original: Since we assumed that rank ( (cid:0) MM (cid:48) (cid:1) ) > rank ( M ) , we reach a contradiction. 
Revised: Since we assumed that rank p ` MM˘ q ą rank p M q , we reach a contradiction. [Equation] M

Original: [Equation]
Revised: [Equation]

Revised: We prove an equivalent proposition “rank p `

Original: ⇒
Revised: MM˘ q ď rank p M q ñ f ` MM 1 ˘ ĺ f M ”. Note that rank p ` MM˘ q ě rank p M q and f ` MM 1 ˘ ľ f M as given in Proposition 1(i). We only need to prove “rank p

Original: [Equation]
Revised: [Equation]

Original: > rank 
Revised: rank p

Original: [Equation]
Revised: [Equation]

Original: M [Equation] We prove an equivalent proposition “rank ( (cid:0) MM (cid:48) (cid:1) ) ≤ rank ( M ) ⇒ f (cid:0) MM (cid:48) (cid:1) (cid:22) f M ”.
Revised: q “ rank p M q means that any row in M 1 is linearly dependent to rows in M .

Original: Note that rank ( (cid:0) MM (cid:48) (cid:1) ) ≥ rank ( M ) and f (cid:0) MM (cid:48) (cid:1) (cid:23) f M as given in Proposition 1(i). We only need to prove “rank ( (cid:0) MM (cid:48) (cid:1) )
Revised: Therefore, there exists L P R s 1 ˆ s so that ` MM 1 ˘ “ ` IL ˘ M . For any x 1 and x 2 with MP π x 1 π “ MP π x 2 π , `

Original: =
Revised: IL ˘

Original: rank ( M ) ⇒ f (cid:0) MM (cid:48) (cid:1) = f M ”.
Revised: MP π x 1 π “ `

Original: rank ( (cid:0) MM (cid:48) (cid:1) ) = rank ( M ) means that any row in M (cid:48) is linearly dependent to rows in M .
Revised: IL ˘

Original: Therefore, there exists L ∈ R s (cid:48)
Revised: MP π x 2 π , and therefore `

Original: ×
Revised: MM˘

Original: s so that (cid:0) MM (cid:48) (cid:1) =
Revised: P

Original: (cid:0) IL (cid:1) M .
Revised: [Equation] MM [Equation]

Original: For any x 1 and x 2 with MP π x 1 π = MP π x 2 π , (cid:0) IL (cid:1) MP π x 1 π = (cid:0) IL (cid:1) MP π x 2 π , and therefore (cid:0)
Revised: ˘

Original: MM (cid:48) (cid:1) P π x 1 π = (cid:0) MM (cid:48) (cid:1) P π x 2 π , where π is the ordering of input elements.
Revised: P π x 2 π , where π is the ordering of input elements.

Original: That is, for any x 1 and x 2 , f M ( x 1 ) =
Revised: That is, for any x 1 and x 2 , f M p x 1 q “ [Equation]

Original: f M ( x2 ) ⇒ f
Revised: M [Equation] x

Original: (cid:0) MM(cid:48) (cid:1) ( x 1 ) =
Revised: [Equation] q ñ [Equation] MM

Original:  f (cid:0) MM (cid:48) (cid:1) ( x 2 ) , thus f (cid:0) MM (cid:48) (cid:1) (cid:22) f M .
Revised: [Equation] ˘ p x 1 q “ f [Equation] MM 1 ˘ p x 2 q , thus f ` MM 1 ˘ ĺ f M .

Original: Finally, we have f (cid:0) MM (cid:48) (cid:1) =
Revised: Finally, we have [Equation]

Original: f M . iii) “Any multiset of size n is distinguishable with f M ⇒ rank ( M ) = n ”
Revised: (iii) “Any multiset of size n is distinguishable with f M ñ rank p M q “ n ”

Original: Since rank ( M ) ≤ n , we prove an equivalent proposition “rank ( M ) < n ⇒ there exists at least two multisets which are indistinguishable”.
Revised: Since rank p M q ď n , we prove an equivalent proposition “rank p M q ă n ñ there exists at least two multisets which are indistinguishable”.

Original: Considering the system of linear equations y = Mx where x ∈ R n , if rank ( M ) < n , then there exists y (cid:48) such that rank ( M ) = rank ( M , y (cid:48) )
Revised: Considering the system of linear equations y “ Mx where x P R n , if rank p M q ă n , then there exists y 1 such that rank p M q “ rank p M , y 1 q ă n .

Original: < n .

Original: According to the Rouch´e–Capelli theorem, there are inﬁnite solutions x (cid:48) i such that y (cid:48) = Mx [-(cid:48) 
 =-]
Revised: According to the Rouch´e–Capelli theorem, there are inﬁnite solutions x 1 i such that y 1 “ Mx [-(cid:48) 
 =-]1 1 “ Mx 1 2 “ ¨ ¨ ¨ “ Mx 18 , Each x 1 i comes from a multiset with a particular order.

Original: Mx (cid:48)

Original: 2 = · · · = Mx (cid:48) ∞ , Each x (cid:48)

Original: i comes from a multiset with a particular order.

Original: Next, we need to prove that all these x (cid:48) i come from more than one multiset.
Revised: Next, we need to prove that all these x 1 i come from more than one multiset.

Original: As a multiset with bounded size n constitutes at most n !
Revised: As a multiset with bounded size n constitutes at most n !

Original: different orders, the inﬁnite number of x (cid:48) i corresponds to y (cid:48) must come from more than one multisets, making these multisets indistinguishable.
Revised: different orders, the inﬁnite number of x 1 i corresponds to y 1 must come from more than one multisets, making these multisets indistinguishable.

Original: “Any multiset of size n is distinguishable with f M ⇐ rank ( M ) = n ”
Revised: “Any multiset of size n is distinguishable with f M ð rank p M q “ n ”

Original: Since rank ( M ) = n and s = n , for any x ∈ R n , y = Mx ∈ R n is unique.
Revised: Since rank p M q “ n and s “ n , for any x P R n , y “ Mx P R n is unique.

Original: Correspondingly, for any P π x π , M ( P π x π ) is unique.
Revised: Correspondingly, for any P π x π , M p P π x π q is unique.

Original: D P ROOF OF P ROPOSITION
Revised: D P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: (i)
Revised: (i)

Original: According to the proof of Proposition 1(i), f (cid:0) MM (cid:48) (cid:1) ( x ) = (cid:18)
Revised: According to the proof of Proposition 1(i), f ` MM 1 ˘ p x q “ ˆ f M p x q f M 1 p x q ˙ .

Original: f M ( x ) f M (cid:48) ( x ) (cid:19) . For any x 1 and x 2 with f (cid:0) M 1 M (cid:48) 1 (cid:1) ( x 1 ) =
Revised: For any x 1 and x 2 with f ` M 1 M ˘ p x 1 q “ f ` M

Original: f (cid:0) M 2 M (cid:48) 2 (cid:1) ( x 2 ) , f M 1 ( x 1 ) = f M 2 ( x 2 ) holds.
Revised: M 1˘ p x 2 q , f M 1 p x 1 q “ f M 2 p x 2 q holds.

Original: Meanwhile, f (cid:0) M  1  M(cid:48)
Revised: Meanwhile, f ` M 1 M ˘ p x 1 q “ f ` M

Original: (cid:1) ( x 1 ) = f (cid:0) M 2 M (cid:48) 2 (cid:1) ( x [-
-]
Revised: M 1˘ p x [-
-]2 q P Res p f

Original: ) ∈
Revised: ` M

Original:  Res ( f (cid:0) M 1 M(cid:48) 1 (cid:1) )
Revised: M 1˘ q X Res p f ` M

Original: ∩ Res ( f (cid:0) M 2 M (cid:48) 2 (cid:1) ) , and f M 1 ( x 1 ) = f M 2 ( x 2 ) ∈ Res ( f M 1 ) ∩ Res ( f M 2 ) .
Revised: M 1˘ q , and f M 1 p x 1 q “ f M 2 p x 2 q P Res p f M 1 q X Res p f M 2 q .

Original: Therefore, for any e ∈ Res ( f(cid:0)
Revised: Therefore, for any e P Res p f

Original:  M1 M (cid:48) (cid:1) ) ∩ Res ( f (cid:0) M 2 M(cid:48) (cid:1)
Revised: ` M M 1˘ q X Res p f ` M

Original: ) , we have e ∈ Res ( f M 1 ){+
+}
Revised: M 1˘ q , we have e P Res p f M {+
+}

Original: ∩ Res ( f M 2 ) .
Revised: q X Res p f M 2 q .

Original: That is
Revised: That is

Original: Res ( f (cid:0) M1 M (cid:48) 1 (cid:1) )
Revised: Res p f ` M M 1˘ q X Res p f ` M

Original: ∩ Res ( f (cid:0) M 2 M (cid:48) 2 (cid:1) ) ⊆

Original:  Res ( f M 1 ) ∩ Res ( f M 2 ) .
Revised: M 1˘ q Ď Res p f M 1 q X Res p f M 2 q .

Original: (ii)
Revised: (ii)

Original: We prove an equivalent proposition “Res ( f (cid:0) M1 M (cid:48) (cid:1) )
Revised: We prove an equivalent proposition “Res p f ` M M 1˘ qX Res p f

Original: ∩ Res ( f (cid:0) M 2 M(cid:48) (cid:1) ) =
Revised: ` M

Original:  Res ( f M 1 ) ∩ Res ( f M 2 ) ⇐ rank ( (cid:0) M 1 M 2 M (cid:48) 1 M (cid:48)
Revised: M 1˘ q “ Res p f M 1 qX Res p f M 2 q ð rank p ` M 1 M 2 M 1 1 M 1˘ q “ rank p `

Original: (cid:1) ) =
Revised: M 1 M˘ q ”.

Original: rank 
Revised: rank p

Original: ( (cid:0)
Revised: ` M 1 M 2 M 1 1 M 1˘ q “ rank p ` M 1 M˘ q means that any row in ` M 1 1 M 1˘ is linearly dependent to `

Original: M M
Revised: M 1 M˘ .

Original: (cid:1) ) ”. rank ( (cid:0) M 1 M 2 M (cid:48) 1 M 
Revised: Therefore, there exists L P R s 1 ˆ s so that ` M 1 M 2 M 1 1 M 1 2 ˘ “ `

Original: (cid:48)
Revised: IL ˘`

Original: (cid:1) )
Revised: M 1 M˘ .

Original: = rank ( (cid:0) M 1 M 2 (cid:1) ) means that any row in (cid:0) M(cid:48) 1  M(cid:48) 2(cid:1) is linearly dependent to (cid:0) M 1  M 2(cid:1) .Therefore, there exists L ∈ R s (cid:48)
Revised: Correspondingly, ` M 1 M 1 1 ˘ “ ` IL ˘ M 1 and ` M 2 M 1 2 ˘ “ ` IL ˘ M 2 .

Original: × s so that (cid:0) M 1  M2 M (cid:48) 1  M(cid:48) 2 (cid:1) =
Revised: For any x 1 and xwith M 1 P π x 1 π “ M 2 P π x 2 π , `

Original: (cid:0) IL (cid:1)(cid:0) M 1 M 2 (cid:1) . Correspondingly, (cid:0) M 1 M (cid:48) 1 (cid:1) = (cid:0) IL (cid:1) M 1 and (cid:0) M 2 M (cid:48) 2 (cid:1) = (cid:0) IL (cid:1) M 2 . For any x 1 and x 2 with M 1 P π x 1 π = M 2 P π x 2 π , (cid:0) IL (cid:1)
Revised: IL ˘

Original: M 1 P π x 1 π = (cid:0) IL (cid:1)
Revised: M 1 P π x 1 π “ ` IL ˘

Original: M 2 P π x 2 π , and therefore (cid:0) M 1 M(cid:48) 1 (cid:1) P π x 1 π = (cid:0) M 2 M (cid:48) 2 (cid:1) P π x 2 π , where π is the ordering of input elements.
Revised: M 2 P π x 2 π , and therefore ` M 1 M 1 1 ˘ P π x 1 π “ ` M 2 M 1˘ P π x 2 π , where π is the ordering of input elements.

Original: Thus for any xand x 2 , f M 1 ( x 1 ) = f M 2 ( x 2 ) ∈ Res ( f M 1 ) ∩ Res ( f M 2 ) ⇒ f 
Revised: Thus for any x 1 and x 2 , f M 1 p x 1 q “ f M 2 p x 2 q P Res p f M 1 qX Res p f M 2 q ñ f ` M 1 M ˘ p x

Original: (cid:0) M 1 M (cid:48) (cid:1) ( x 1 ) = f (cid:0) M 2 M (cid:48) (cid:1) ( x 2 ) ∈
Revised: [Equation]

Original:  Res ( f (cid:0) M1 M (cid:48) 1 (cid:1) )
Revised: [Equation] P Res p f ` M M 1˘ qX Res p f

Original: ∩ Res ( f (cid:0) M 2 M(cid:48) 2 (cid:1) ) .
Revised: ` M

Original:  Hence, Res ( f M 1 ) ∩ Res ( f M 2 ) ⊆
Revised: M 1˘ q . Hence, Res p f M 1 q X Res p

Original:  Res ( f (cid:0) M 1 M (cid:48) (cid:1) ) ∩ Res ( f (cid:0) M2 M (cid:48) (cid:1) ) .
Revised: f M 2 q Ď Res p f ` M 1 M ˘ q X Res p f ` M M 1˘ q .

Original: According to Proposition 2(i), Res ( f M 1 ) ∩ Res ( f M 2 ) = Res ( f (cid:0)
Revised: According to Proposition 2(i), Res p f M 1 q X Res p f M 2 q “ Res p f ` M 1 M ˘ q X Res p f ` M

Original: [Equation]

Original: [Equation]
Revised: M 1˘ q .

Original: E P ROOF OF P ROPOSITION
Revised: E P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Since M 1 ∈ R s × n 1 , M 2 ∈ R s × n 2 and rank ( (cid:0) M 1 M 2 (cid:1) ) = n 1 + n 2 , we have rank ( M 1 ) = n 1 and rank ( M 2 ) = n 2 .
Revised: Since M 1 P R s ˆ n 1 , M 2 P R s ˆ n 2 and rank p ` M 1 M 2 ˘ q “ n 1 ` n 2 , we have rank p M 1 q “ nand rank p M 2 q “ n 2 .

Original: According to Proposition 1, f M 1 and f M 2 are injective.
Revised: According to Proposition 1, f M 1 and f M 2 are injective.

Original: We build the system of linear equations y
Revised: We build the system of linear equations y

Original: = Ax , where x ∈ R n 1 + n 2 , and A = (cid:0)
Revised: “ Ax , where x P R n 1 ` n 2 , and A “ `

Original: M 1 M 2 (cid:1)(cid:0) I n0 0 − I n 2 (cid:1) ∈
Revised: M 1 M˘` I n0 0 ´ I n˘

Original:  R s × ( n 1 + n 2 ) .
Revised: P R s ˆp n 1 ` n 2 q .

Original: Then, rank ( A ) = rank ( (cid:0) M 1 M 2 (cid:1)(cid:0) I n0 0 − I n 2 (cid:1) ) =
Revised: Then, rank p A q “ rank p `

Original: rank ( (cid:0) M 1 M 2 (cid:1) )
Revised: M 1 M˘` I n0 0 ´ I n˘ q “ rank p `

Original: = n 1 + n 2 , which means Ax = 0 has no non-zero solutions.Let x (cid:48)
Revised: M 1 M˘ q “ n 1 ` n 2 , which means Ax “ 0 has no non-zero solutions.

Original: = ( x [1] , x [2] , · · · , x [ n 1 ]) and x (cid:48)(cid:48) = ( x [ n 1 + 1] , x [ n 1 + 2] , · · · , x [ n 1 + n 2 ]) such that x = (cid:0) x (cid:48) x (cid:48)(cid:48) (cid:1) .
Revised: Let x 1 “ p x r 1 s , x r 2 s , ¨ ¨ ¨ , x r n 1 sq and x 2 “ p x r n 1 ` 1 s , x r n 1 ` 2 s , ¨ ¨ ¨ , x r n 1 ` n 2 sq such that x “ ` xx˘ .

Original: For any x (cid:54) = 0 ,
Revised: For any x ‰ 0 ,

Original: Ax = ( M
Revised: Ax “ p M

Original: M 2 ) (cid:18) I n  − I n(cid:19) (cid:18) x (cid:48)
Revised: M 2 q ˆ I n  ´ I n˙

Original: x (cid:48)(cid:48) (cid:19) = M 1 x (cid:48) − M 2 x (cid:48)(cid:48) (cid:54) = 0 .
Revised: ˆ xx˙ “ M 1 x 1 ´ M 2 x 2 ‰ 0 .

Original: Therefore, for any x (cid:48) ∈ R n 1 , x (cid:48)(cid:48) ∈ R n 2 and x (cid:48) , x (cid:48)(cid:48) (cid:54) = 0 , M 1 x (cid:48) (cid:54) = M 2 x (cid:48)(cid:48) , hence M 1 ( P π x (cid:48) π ) (cid:54) =
Revised: Therefore, for any x 1 P R n 1 , x 2 P R n 2 and x 1 , x 2 ‰ 0 , M 1 x 1 ‰ M 2 x 2 , hence M 1 p P π x 1 π q ‰

Original: M 2 ( P π x (cid:48)(cid:48) π ) for any P π x (cid:48) π and P π x (cid:48)(cid:48) π .
Revised: M 2 p P π x 2 π q for any P π x 1 π and P π x 2 π .

Original: As a result, Res ( f M 1 ) ∩ Res ( f M 2 ) = ∅ .
Revised: As a result, Res p f M 1 q X Res p f M 2 q “ H .

Original: F P ROOF OF P ROPOSITION
Revised: F P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: For Multi-head GAT, there are two types of implementations on aggregating each head, concatenation and average .
Revised: For Multi-head GAT, there are two types of implementations on aggregating each head, concatenation and average .

Original: Here, we only consider the average aggregation implementation.
Revised: Here, we only consider the average aggregation implementation.

Original: [Equation]
Revised: [Equation]

Original: where W k ∈ R d × d is the trainable matrix for the k -th head, and W = || 
Revised: where W k P R d ˆ d is the trainable matrix for the k -th head, and W “ || Kk

Original: Kk =1 W k ∈ R kd × d is the concatenation of the trainable matrix in all K heads;
Revised: “ 1 W k P R kd ˆ d is the concatenation of the trainable matrix in all K heads;

Original: [Equation]
Revised: [Equation]

Original: t − 1) u ] a 2 T [ W 2 h ( t − 1) v || W 2 h ( t − 1) u ] ... a KT [ W 1 h ( t − 1) v || W K h ( t − 1) u ]   ) .

Original: Let ˜ a (cid:63) = ( a (cid:63) 1 , a (cid:63) 2 , · · · , a (cid:63)K ) and ˜ a (cid:48) (cid:63) = ( a (cid:63)K +1 , a (cid:63)K +2 , · · · , a (cid:63) 2 K ) .
Revised: Let ˜ a ‹ “ p a ‹ 1 , a ‹ 2 , ¨ ¨ ¨ , a ‹ K q and ˜ a 1‹ “ p a ‹ K ` 1 , a ‹ K ` 2 , ¨ ¨ ¨ , a ‹ 2 K q .

Original: Then,   a 1 T [ W 1 h ( t − 1) v || W 1 h ( t − 1) u ] a 2 T [ W 2 h ( t − 1) v || W 2 h ( t − 1) u ] ... a KT [ W 1 h ( t − 1) v|| W K h ( t − 1) u ]
Revised: Then, ¨ ˚˚˚˚˝ a 1 T r W 1 h p t ´ 1 q v || W 1 h p t ´ 1 q u s a 2 T r W 2 h p t ´ 1 q v || W 2 h p t ´ 1 q u s ... a KT r W 1 h p t ´ 1 q v

Original:   =   [˜ a 1 || ˜ a(cid:48) 1 ] T [ W 1 h ( t − 1) v || W 1 h ( t − 1) u ] [˜ a 2 || ˜ a (cid:48) 2 ] T [ W 2 h ( t − 1) v || W 2 h ( t − 1) u ] ...
Revised: || W K h p t ´ 1 q u s ˛ ‹‹‹‹‚ “ ¨ ˚˚˚˚˝ r ˜ a 1 || ˜ a 1 1 s T r W 1 h p t ´ 1 q v || W 1 h p t ´ 1 q u s r ˜ a 2 || ˜ a 1 2 s T r W 2 h p t ´ 1 q v || W 2 h p t ´ 1 q u s ...

Original: [˜ a K || ˜ a (cid:48) K ] T [ W 1 h ( t − 1) v ||W K h ( t − 1) u ]
Revised: r ˜ a K || ˜ a 1 K s T r W 1 h p t ´ 1 q v ||

Original:   =   ˜ a 1 T W 1 h ( t − 1) v + ˜ a (cid:48) 1 T W 1 h( t − 1)
Revised: W K h p t ´ 1 q u s ˛ ‹‹‹‹‚ “ ¨ ˚˚˚˚˝ ˜ a 1 T W 1 h p t ´ 1 q v ` ˜ a 1 1 T W 1 h

Original:  u ˜ a 2 T W 2 h ( t − 1) v + ˜ a (cid:48) 2 T W 2 h ( t − 1) u ...
Revised: p t ´ 1 q u ˜ a 2 T W 2 h p t ´ 1 q v ` ˜ a 1 2 T W 2 h p t ´ 1 q u ...

Original: ˜ a KT W K h ( t − 1) v + ˜ a (cid:48) KT W K h 
Revised: ˜ a KT W K h p t ´ 1 q v ` ˜ a 1 KT W K h p t ´ 1 q u ˛ ‹‹‹‹‚ “ ¨ ˚˚˚˚˝ ˜ a 1 T W 1 h p t ´ 1 q v ˜ a 2 T W 2 h

Original: ( t − 1) u
Revised: p t ´ 1 q v ...

Original:   =   ˜ a 1  T W 1 h ( t − 1) v ˜ a  2 T W 2 h ( t − 1) v ...
Revised: ˜ a KT W K h p t ´ 1 q v ˛ ‹‹‹‹ ‚ ` ¨ ˚˚˚˚˝ ˜ a 1 1 T W 1 h p t ´ 1 q u ˜ a 1 2 T W 2 h p t ´ 1 q u ...

Original: ˜ a  KT W K h ( t − 1) v   +  
Revised: ˜ a 1 KT W K h p t ´ 1 q u ˛ ‹‹‹‹‚ [Equation] ˚˚˚˝ [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ˛ ‹‹‹‚ ¨ ˚˚˝

Revised: W

Revised: W...

Revised: [Equation]

Revised: ˛ ‹‹‚ h p t

Revised: [Equation]

Revised: [Equation]

Revised: v ` ¨ ˚˚˚˝ ˜ a 1 1 T ˜ a 1 2 T . . .

Original: 
Revised: ˜ a 1 KT ˛ ‹‹‹‚ ¨ ˚˚˝ W W... W K ˛ ‹‹‚ h p t ´ 1 q

Original: [Equation]
Revised: [Equation]

Original:  
Revised: |

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: multi-head GAT is an implementation of ExpandingConv as follows:
Revised: multi-head GAT is an implementation of ExpandingConv as follows:

Original: [Equation]
Revised: [Equation]

Original: G C OMPARISONS WITH M ULTI - AGGREGATOR I MPLEMENTATIONS
Revised: G C OMPARISONS WITH M ULTI - AGGREGATOR I MPLEMENTATIONS

Original: ExpandingConv can also be considered as a kind of multi-aggregator scheme.
Revised: ExpandingConv can also be considered as a kind of multi-aggregator scheme.

Original: In Equation 5, each row of M v i can be viewed as a weighted aggregator where the weight coefﬁcients are learned from data.
Revised: In Equation 5, each row of M v i can be viewed as a weighted aggregator where the weight coefﬁcients are learned from data.

Original: Proposition 1 shows that to obtain higher distinguishing strength by utilizing more aggregators, the weight coefﬁcients of newly added aggregators should be linearly independent to all existing aggregators.
Revised: Proposition 1 shows that to obtain higher distinguishing strength by utilizing more aggregators, the weight coefﬁcients of newly added aggregators should be linearly independent to all existing aggregators.

Original: The distinguishing strength of weighted aggregators is incomparable with basic aggregators.
Revised: The distinguishing strength of weighted aggregators is incomparable with basic aggregators.

Original: However, since each row of M v i is equivalent to an independent aggregator, one can simply modify the implementation of f local ( u, v ) to obtain the variant whose distinguishing strength is strict stronger than basic aggregators as follows:
Revised: However, since each row of M v i is equivalent to an independent aggregator, one can simply modify the implementation of f local p u, v q to obtain the variant whose distinguishing strength is strict stronger than basic aggregators as follows:

Original: [Equation]
Revised: [Equation]

Original: Compared with lerveraging multiple basic aggregators in Corso et al. and Dehmamy et al. (2019), lerveraging weighted aggregator allows for variable numbers of aggregators.
Revised: Compared with lerveraging multiple basic aggregators in (Corso et al., 2020) and (Dehmamy et al., 2019), lerveraging weighted aggregator allows for variable numbers of aggregators.

Original: Meanwhile, the weighted coefﬁcients are learned from data, which can better capture relevant structural patterns.
Revised: Meanwhile, the weighted coefﬁcients are learned from data, which can better capture relevant structural patterns.

Original: H D ETAILS OF E XPERIMENTAL S ETUP
Revised: H D ETAILS OF E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: Benchmark datasets for graph kernels provided by TU Kersting et al. (2016) suffer from their small scales of data, making them not sufﬁcient to evaluate the performance of models Dwivedi et al.
Revised: Benchmark datasets for graph kernels provided by TU (Kersting et al., 2016) suffer from their small scales of data, making them not sufﬁcient to evaluate the performance of models (Dwivedi et al., 2020).

Original: Our evaluations are conducted on graph property predictions datasets ogbg-ppa, ogbgcode, ogbg-molhiv in OGB Weihua Hu (2020) and QM9 Ramakrishnan et al. ; Wu et al. Ruddigkeit et al. (2012) which are large-scale graph datasets including graph classiﬁcation and graph regression tasks.
Revised: Our evaluations are conducted on graph property predictions datasets ogbg-ppa, ogbg-code, ogbg-molhiv in OGB (Weihua Hu, 2020) and QM9 (Ramakrishnan et al., 2014; Wu et al., 2018; Ruddigkeit et al., 2012) which are large-scale graph datasets including graph classiﬁcation and graph regression tasks.

Original: ogbg-ppa is extracted from the protein-protein association networks with large and densely connected graphs.
Revised: ogbg-ppa is extracted from the protein-protein association networks with large and densely connected graphs.

Original: ogbg-code is a collection of Abstract Syntax Trees (ASTs) obtained from Python method deﬁnitions with large and sparse graphs.
Revised: ogbg-code is a collection of Abstract Syntax Trees (ASTs) obtained from Python method deﬁnitions with large and sparse graphs.

Original: ogbg-molhiv is molecular property prediction datasets with relative small graphs.
Revised: ogbg-molhiv is molecular property prediction datasets with relative small graphs.

Original: QM9 consists 134K small organic molecules with the task to predict 12 targets for each molecule.
Revised: QM9 consists 134K small organic molecules with the task to predict 12 targets for each molecule.

Original: All data is obtained from pytorch-geometric library Fey & Lenssen (2019).
Revised: All data is obtained from pytorch-geometric library (Fey & Lenssen, 2019).

Original: Conﬁgurations.
Revised: [Table]

Original: We use the default dataset splits for OGB.
Revised: [Figure]

Original: The QM9 dataset is randomly split into 80% train, 10% validation and 10% test as given in Morris et al.
Revised: [Table] [Figure] [Table] [Table]

Original: ; Maron et al.
Revised: [Figure]

Original: We use the concatenation of hidden features from all layers to compute the entire graph representations Xu et al.
Revised: [Table] [Figure] [Table] [Figure]

Original: In our tests, all models are equipped with batch normalization Ioffe & Szegedy (2015) on each hidden layer when evaluating on OGB, and are not when evaluating on QM9.
Revised: [Table] [Figure] [Table]

Original: [Table]
Revised: [Figure]

Original: The shared hyperparameter settings of ExpC*-1, ExpC- s , CombC* and CombC on all 12 targets of QM9: batch sizes = 64; lr = 0.0001; step size = 30; lr decay = 0.85; readout = SUM.
Revised: The shared hyperparameter settings of ExpC*-1, ExpC- s , CombC* and CombC on all 12 targets of QM9: batch sizes = 64; lr = 0.0001; step size = 30; lr decay = 0.85; readout = SUM.

Original: hidden = 256 for ExpC*-1 and ExpC- s ; hidden = 512 for CombC* and CombC. Table 4 gives the individual hyperparameter settings of each model on each target, including the number of layers.
Revised: hidden = 256 for ExpC*-1 and ExpC- s ; hidden = 512 for CombC* and CombC. Table 7 gives the individual hyperparameter settings of each model on each target, including the number of layers.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: CombC*,CombC

Original: I M ORE E XPERIMENTAL R ESULTS
Revised: I M ORE E XPERIMENTAL R ESULTS

Original: We present more results of ablation studies on OGB and QM9, which demonstrate the effectiveness of ExpandingConv, CombConv and Re -SUM.
Revised: We present more results of ablation studies on OGB and QM9, which demonstrate the effectiveness of ExpandingConv, CombConv and Re -SUM.

Original: [Figure]
Revised: [Figure]

Original: T r a i n i n g P
Revised: T r a i n i n g P

Original: e
Revised: e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m a n c e
Revised: o r m a n c e

Original: [Figure]
Revised: [Figure]

Revised: T r a i n i n g P

Revised: e

Revised: r

Revised: f

Revised: o r m a n c e

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: e a n A b s o l u t e E rr o r
Revised: e a n A b s o l u t e E rr o r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]