Original: F ROM S TARS TO S UBGRAPHS : U PLIFTING A NY GNN WITH L OCAL S TRUCTURE A WARENESS
Revised: F ROM S TARS TO S UBGRAPHS : U PLIFTING A NY GNN WITH L OCAL S TRUCTURE A WARENESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Message Passing Neural Networks (MPNNs) are a common type of Graph Neural
Revised: Message Passing Neural Networks (MPNNs) are a common type of Graph Neural

Original: Network (GNN), in which each node’s representation is computed recursively by aggregating representations (“messages”) from its immediate neighbors akin to a star-shaped pattern.
Revised: Network (GNN), in which each node’s representation is computed recursively by aggregating representations (“messages”) from its immediate neighbors akin to a star-shaped pattern.

Original: MPNNs are appealing for being efﬁcient and scalable, however their expressiveness is upper-bounded by the 1st-order Weisfeiler-Leman isomorphism test (1-WL).
Revised: MPNNs are appealing for being efﬁcient and scalable, however their expressiveness is upper-bounded by the 1st-order Weisfeiler-Leman isomorphism test (1-WL).

Original: In response, prior works propose highly expressive models at the cost of scalability and sometimes generalization performance.
Revised: In response, prior works propose highly expressive models at the cost of scalability and sometimes generalization performance.

Original: Our work stands between these two regimes: we introduce a general framework to uplift any MPNN to be more expressive, with limited scalability overhead and greatly improved practical performance.
Revised: Our work stands between these two regimes: we introduce a general framework to uplift any MPNN to be more expressive, with limited scalability overhead and greatly improved practical performance.

Original: We achieve this by extending local aggregation in MPNNs from star patterns to general subgraph patterns (e g., k -egonets): in our framework, each node representation is computed as the encoding of a surrounding induced subgraph rather than encoding of immediate neighbors only (i e. a star).
Revised: We achieve this by extending local aggregation in MPNNs from star patterns to general subgraph patterns (e g., k -egonets): in our framework, each node representation is computed as the encoding of a surrounding induced subgraph rather than encoding of immediate neighbors only (i e. a star).

Original: We choose the subgraph encoder to be a GNN (mainly MPNNs, considering scalability) to design a general framework that serves as a wrapper to uplift any GNN.
Revised: We choose the subgraph encoder to be a GNN (mainly MPNNs, considering scalability) to design a general framework that serves as a wrapper to uplift any GNN.

Original: We call our proposed method GNN-AK (GNN As Kernel), as the framework resembles a convolutional neural network by replacing the kernel with GNNs.
Revised: We call our proposed method GNN-AK (GNN As Kernel), as the framework resembles a convolutional neural network by replacing the kernel with GNNs.

Original: Theoretically, we show that our framework is strictly more powerful than 1&2-WL and is not less powerful than 3-WL.
Revised: Theoretically, we show that our framework is strictly more powerful than 1&2-WL and is not less powerful than 3-WL.

Original: We also design subgraph sampling strategies which greatly reduce memory footprint and improve speed while maintaining performance.
Revised: We also design subgraph sampling strategies which greatly reduce memory footprint and improve speed while maintaining performance.

Original: Our method sets new state-of-the-art performance by large margins for several well-known graph ML tasks; speciﬁcally, 0 .
Revised: Our method sets new state-of-the-art performance by large margins for several well-known graph ML tasks; speciﬁcally, 0 .

Original: 08 MAE on
Revised: 08 MAE on

Original: ZINC, 74.79% and 86.887% accuracy on CIFAR10 and PATTERN respectively
Revised: ZINC, 74.79% and 86.887% accuracy on CIFAR10 and PATTERN respectively

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs are permutation invariant, combinatorial structures used to represent relational data, with wide applications ranging from drug discovery, social network analysis, image analysis to bioinformatics (Duvenaud et al., 2015; Fan et al., 2019; Shi et al., 2019; Wu et al., 2020).
Revised: Graphs are permutation invariant, combinatorial structures used to represent relational data, with wide applications ranging from drug discovery, social network analysis, image analysis to bioinformatics (Duvenaud et al., 2015; Fan et al., 2019; Shi et al., 2019; Wu et al., 2020).

Original: In recent years, Graph Neural Networks (GNNs) have rapidly surpassed traditional methods like heuristically deﬁned features and graph kernels to become the dominant approach for graph ML tasks.
Revised: In recent years, Graph Neural Networks (GNNs) have rapidly surpassed traditional methods like heuristically deﬁned features and graph kernels to become the dominant approach for graph ML tasks.

Original: Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017) are the most common type of
Revised: Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017) are the most common type of

Original: GNNs owing to their intuitiveness, effectiveness and efﬁciency.
Revised: GNNs owing to their intuitiveness, effectiveness and efﬁciency.

Original: They follow a recursive aggregation mechanism where each node aggregates information from its immediate neighbors repeatedly.
Revised: They follow a recursive aggregation mechanism where each node aggregates information from its immediate neighbors repeatedly.

Original: However, unlike simple multi-layer feedforward networks (MLPs) which are universal approximators of continuous functions (Hornik et al., 1989), MPNNs cannot approximate all permutation-invariant graph functions (Maron et al., 2019b).
Revised: However, unlike simple multi-layer feedforward networks (MLPs) which are universal approximators of continuous functions (Hornik et al., 1989), MPNNs cannot approximate all permutation-invariant graph functions (Maron et al., 2019b).

Original: In fact, their expressiveness is upper bounded by the ﬁrst order Weisfeiler-Leman (1-WL) isomorphism test (Xu et al., 2018).
Revised: In fact, their expressiveness is upper bounded by the ﬁrst order Weisfeiler-Leman (1-WL) isomorphism test (Xu et al., 2018).

Original: Importantly, researchers have shown that such 1-WL equivalent GNNs are not expressive, or powerful, enough to capture basic structural concepts, i.e., counting motifs such as cycles or triangles (Zhengdao et al., 2020; Arvind et al., 2020) that are shown to be informative for bio- and chemo-informatics (Elton et al., 2019).
Revised: Importantly, researchers have shown that such 1-WL equivalent GNNs are not expressive, or powerful, enough to capture basic structural concepts, i.e., counting motifs such as cycles or triangles (Zhengdao et al., 2020; Arvind et al., 2020) that are shown to be informative for bio- and chemo-informatics (Elton et al., 2019).

Original: The weakness of MPNNs urges researchers to design more expressive GNNs, which are able to discriminate graphs from an isomorphism test perspective; Chen et al.
Revised: The weakness of MPNNs urges researchers to design more expressive GNNs, which are able to discriminate graphs from an isomorphism test perspective; Chen et al.

Original: (2019) prove the equivalence between such tests and universal permutation invariant function approximation, which theoretically justiﬁes this line of work.
Revised: (2019) prove the equivalence between such tests and universal permutation invariant function approximation, which theoretically justiﬁes it.

Original: As k -WL is strictly more expressive than 1 -WL, many works (Morris et al., 2019; 2020b) try to incorporate k -WL in the design of more powerful GNNs, while others 
Revised: As k -WL is strictly more expressive than 1 -WL, many works (Morris et al., 2019; 2020b) try to incorporate k -WL in the design of more powerful GNNs, while others approach k -WL

Original: [Figure]
Revised: [Figure]

Original:  ’s embedding when applying a GNN kernel on subgraph j ).
Revised: rooted subgraphs, and convolves all subgraphs with a base GNN as kernel, producing multiple rich subgraph-node embeddings of the form Emb ( i | Sub[ j ]) (node i ’s embedding when applying a GNN kernel on subgraph j ).

Original: From these, we extract and concatenate three encodings for a given node j :
Revised: From these, we extract and concatenate three encodings for a given node j :

Original: (i) centroid Emb
Revised: (i) centroid Emb

Original: j | Sub[ j ]) , (ii) subgraph (cid:80) i
Revised: j | Sub[ j ]) , (ii) subgraph (cid:80) i

Original: Emb ( i | Sub[ j ]) (the output of GNN( Sub[ j ] ), and (iii) context (cid:80) i
Revised: Emb ( i | Sub[ j ]) , and (iii) context (cid:80) i

Original: Emb ( j |
Revised: Emb ( j |

Original: Sub
Revised: Sub

Original: [ i ]) .
Revised: [ i ]) .

Original: GNN-AK  repeats the process for L layers, and sumsover all resulting node embeddings to compute the ﬁnal graph embedding.
Revised: GNN-AK + repeats the process for L layers, then sums all resulting node embeddings to compute the ﬁnal graph embedding.

Revised: As a weaker version, GNN-AK only contains (i) and (ii).

Original: approach k -WL expressiveness indirectly from matrix invariant operations (Maron et al., 2019a;b; Keriven & Peyr ´ e, 2019) and matrix language perspectives (Balcilar et al., 2021).
Revised: expressiveness indirectly from matrix invariant operations (Maron et al., 2019a;b; Keriven & Peyr ´ e, 2019) and matrix language perspectives (Balcilar et al., 2021).

Original: However, they require O( k )-order
Revised: However, they require O( k )-order

Original: tensors to achieve k -WL expressiveness, and thus are not scalable or feasible for application on large, practical graphs.
Revised: tensors to achieve k -WL expressiveness, and thus are not scalable or feasible for application on large, practical graphs.

Original: Besides, the bias-variance tradeoff between complexity and generalization (Neal et al., 2018) and the fact that almost all graphs (i.e. O( 2( n 2 ) ) graphs on n vertices, Babai et al.
Revised: Besides, the bias-variance tradeoff between complexity and generalization (Neal et al., 2018) and the fact that almost all graphs (i.e. O( 2( n 2 ) ) graphs on n vertices, Babai et al.

Original: (1980)) can be distinguished by 1-WL challenge the necessity of developing such extremely expressive models.
Revised: (1980)) can be distinguished by 1-WL challenge the necessity of developing such extremely expressive models.

Original: In a complementary line of work, Loukas (2020a) sheds light on developing more powerful GNNs while maintaining linear scalability, ﬁnding that MPNNs can be universal approximators provided that nodes are sufﬁciently distinguishable.
Revised: In a complementary line of work, Loukas (2020a) sheds light on developing more powerful GNNs while maintaining linear scalability, ﬁnding that MPNNs can be universal approximators provided that nodes are sufﬁciently distinguishable.

Original: Relatedly, several works propose to add features to make nodes more distinguishable, such as identiﬁers (Loukas, 2020a), subgraph counts (Bouritsas et al., 2020), distance encoding (Li et al., 2020), and random features (Sato et al.,  2021).
Revised: Relatedly, several works propose to add features to make nodes more distinguishable, such as identiﬁers (Loukas, 2020a), subgraph counts (Bouritsas et al., 2020), distance encoding (Li et al., 2020), and random features (Sato et al., 2021; Abboud et al., 2021).

Original: However, these methods either focus on handcrafted features which lose the premise of automatic learning, or create permutation sensitive features that hurt generalization.
Revised: However, these methods either focus on handcrafted features which lose the premise of automatic learning, or create permutation sensitive features that hurt generalization.

Original: Present Work.
Revised: Present Work.

Original: Our work stands between the two regimes of extreme expressiveness at the cost of slow and unscalable k -order GNNs, and the limited expressiveness yet high scalability of MPNNs.
Revised: Our work stands between the two regimes of extremely expressive but unscalable korder GNNs, and the limited expressiveness yet high scalability of MPNNs.

Original: Speciﬁcally, we propose a general framework that serves as a “wrapper” to uplift any GNN (mainly
Revised: Speciﬁcally, we propose a general framework that serves as a “wrapper” to uplift any GNN.

Original: MPNNs in practice). We observe that MPNNs’ local neighbor aggregation follows a star pattern, where the representation of a node is characterized by applying an injective aggregator function as an encoder to the star subgraph (comprised of the central node and edges to neighbors).
Revised: We observe that MPNNs’ local neighbor aggregation follows a star pattern, where the representation of a node is characterized by applying an injective aggregator function as an encoder to the star subgraph (comprised of the central node and edges to neighbors).

Original: We propose a design which naturally generalizes from encoding the star to encoding a more ﬂexibly deﬁned subgraph, and we replace the standard injective aggregator with a GNN: in short, we characterize the new representation of a node by using a GNN to encode a locally induced encompassing subgraph, as shown in Fig.1.
Revised: We propose a design which naturally generalizes from encoding the star to encoding a more ﬂexibly deﬁned subgraph, and we replace the standard injective aggregator with a GNN: in short, we characterize the new representation of a node by using a GNN to encode a locally induced encompassing subgraph, as shown in Fig.1.

Original: This uplifts GNN as a base model in effect by applying it on each subgraph instead of the whole input graph.
Revised: This uplifts GNN as a base model in effect by applying it on each subgraph instead of the whole input graph.

Original: Interestingly, this generalization has a closeconnection to Convolutional Neural Networks (CNN) in computer vision: like the CNN that convolves image patches with a kernel to compute new pixel embeddings, our designed wrapper convolves subgraphs with a GNN to generate new node embeddings.
Revised: This generalization is close to Convolutional Neural Networks (CNN) in computer vision: like the CNN that convolves image patches with a kernel to compute new pixel embeddings, our designed wrapper convolves subgraphs with a GNN to generate new node embeddings.

Original: Hence, we name our approach GNN-AK (GNN As Kernel).
Revised: Hence, we name our approach GNN-AK (GNN As Kernel).

Original: We show theoretically that GNN-AK is strictly more powerful than 1&2-WL with any MPNN as base model, and is not less powerful than 3-WL with PPGN (Maron et al., 2019a) used.
Revised: We show theoretically that GNN-AK is strictly more powerful than 1&2-WL with any MPNN as base model, and is not less powerful than 3-WL with PPGN (Maron et al., 2019a) used.

Original: We also give sufﬁcient conditions under which GNN-AK can successfully distinguish two non-isomorphic graphs.
Revised: We also give sufﬁcient conditions under which GNN-AK can successfully distinguish two non-isomorphic graphs.

Original: Given this increase in expressive power, we discuss careful implementation strategies for GNN-AK , which allow us to carefully leverage multiple modalities of information from subgraph encoding.
Revised: Given this increase in expressive power, we discuss careful implementation strategies for GNN-AK , which allow us to carefully leverage multiple modalities of information from subgraph encoding, and resulting in an empirically more expressive version GNN-AK + .

Original: As a result, GNN-AK induces a constant factor overhead in memory.
Revised: As a result, GNN-AK and GNN-AK + induce a constant factor overhead in memory.

Original: To amplify our method’s practicality, we further develop a subgraph sampling strategy inspired by Dropout (Srivastava et al., 2014) to drastically reduce this overhead ( 1 - 3 × in practice) without hurting performance.
Revised: To amplify our method’s practicality, we further develop a subgraph sampling strategy inspired by Dropout (Srivastava et al., 2014) to drastically reduce this overhead ( 1 - 3 × in practice) without hurting performance.

Original: We conduct extensive experiments on 4 simulation datasets and 5 well-known real-world graph classiﬁcation & regression benchmarks (Dwivedi et al., 2020; Hu et al., 2020), to show signiﬁcant and consistent practical beneﬁts of our approach across different MPNNs and datasets.
Revised: We conduct extensive experiments on 4 simulation datasets and 5 well-known real-world graph classiﬁcation & regression benchmarks (Dwivedi et al., 2020; Hu et al., 2020), to show signiﬁcant and consistent practical beneﬁts of our approach across different MPNNs and datasets.

Original: Speciﬁcally, GNN-AK  sets new state-of-the-art performance on ZINC, CIFAR10, and PATTERN – for example, on ZINC we see a relative error reduction of 60.3%, 50.5%, and 39.4% for base model beingGCN (Kipf &
Revised: Speciﬁcally, GNN-AK + sets new state-of-the-art performance on ZINC, CIFAR10, and PATTERN – for example, on ZINC we see a relative error reduction of 60.3%, 50.5%, and 39.4% for base model being

Original:  Welling, 2017), GIN (Xu et al., 2018), and (a variant of) PNA (Corso et al., 2020) respectively.
Revised: GCN (Kipf & Welling, 2017), GIN (Xu et al., 2018), and (a variant of) PNA (Corso et al., 2020) respectively.

Original: To summarize, our contributions are listed as follows:
Revised: To summarize, our contributions are listed as follows:

Original: • A General GNN-AK Framework.
Revised: • A General GNN-AK Framework.

Original: We propose GNN-AK (GNN As Kernel), a general framework which uplifts any GNN by encoding local subgraph structure with a GNN.
Revised: We propose GNN-AK (and enhanced GNN-AK + ), a general framework which uplifts any GNN by encoding local subgraph structure with a GNN.

Original: • Theoretical Findings.
Revised: • Theoretical Findings.

Original: We show that GNN-AK ’s expressiveness is strictly better than 1&2-WL, and is not less powerful than 3-WL.
Revised: We show that GNN-AK ’s expressiveness is strictly better than 1&2-WL, and is not less powerful than 3-WL.

Original: We analyze sufﬁcient conditions for successful discrimination.
Revised: We analyze sufﬁcient conditions for successful discrimination.

Original: • Effective and Efﬁcient Realization.
Revised: • Effective and Efﬁcient Realization.

Original: We presentan effective  GNN-AK implementation to fully exploit all node embeddings within a subgraph.
Revised: We present effective implementations for GNN-AK and GNN-AK + to fully exploit all node embeddings within a subgraph.

Original: We design efﬁcient online subgraph samplingand extraction methods to mitigate memory and runtime overhead while maintaining performance.
Revised: We design efﬁcient online subgraph sampling to mitigate memory and runtime overhead while maintaining performance.

Original: • Experimental Results.
Revised: • Experimental Results.

Original: We show strong empirical results, demonstrating both expressivity improvements as well as practical performance gains where we achieve new state-of-the-art performance on several graph-level benchmarks.
Revised: We show strong empirical results, demonstrating both expressivity improvements as well as practical performance gains where we achieve new state-of-the-art performance on several graph-level benchmarks.

Original: Our implementation is easy-to-use, and directly accepts any GNN from PyG (Fey & Lenssen, 2019) for plug-and-play use.
Revised: Our implementation is easy-to-use, and directly accepts any GNN from PyG (Fey & Lenssen, 2019) for plug-and-play use.

Original: See code at https://github.com/GNNAsKernel/GNNAsKernel .
Revised: See code at https://github.com/GNNAsKernel/GNNAsKernel .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Improving Expressiveness of GNNs:
Revised: Improving Expressiveness of GNNs:

Original: Several works other than those mentioned in Sec.1 tackle expressive GNNs.
Revised: Several works other than those mentioned in Sec.1 tackle expressive GNNs.

Original: Murphy et al.
Revised: Murphy et al.

Original: (2019) achieve universality by summing permutation-sensitive functions across a combinatorial number of permutations, limiting feasibility.
Revised: (2019) achieve universality by summing permutation-sensitive functions across a combinatorial number of permutations, limiting feasibility.

Original: Dasoulas et al.
Revised: Dasoulas et al.

Original: (2020) adds node indicators to make them distinguishable, but at the cost of an invariant model, while Vignac et al.
Revised: (2020) adds node indicators to make them distinguishable, but at the cost of an invariant model, while Vignac et al.

Original: (2020) further addresses the invariance problem, but at the cost of quadratic time complexity.
Revised: (2020) further addresses the invariance problem, but at the cost of quadratic time complexity.

Original: Corso et al.
Revised: Corso et al.

Original: (2020) generalizes MPNN’s default sum aggregator, but is still limited by 1 -WL.
Revised: (2020) generalizes MPNN’s default sum aggregator, but is still limited by 1 -WL.

Original: Beani et al.
Revised: Beani et al.

Original: (2021) generalizes spatial and spectral aggregation with > 1 -WL expressiveness, but using expensive eigendecomposition.
Revised: (2021) generalizes spatial and spectral aggregation with > 1 -WL expressiveness, but using expensive eigendecomposition.

Original: Recently, Bodnar et al.
Revised: Recently, Bodnar et al.

Original: (2021b) introduce MPNNs over simplicial complexes that shares similar expressiveness as GNN-AK .
Revised: (2021b) introduce MPNNs over simplicial complexes that shares similar expressiveness as GNN-AK .

Original: Ying et al.
Revised: Ying et al.

Original: (2021) studies transformer with above 1-WL expressiveness.
Revised: (2021) studies transformer with above 1-WL expressiveness.

Original: Azizian & Lelarge (2021) surveys GNN expressiveness work.
Revised: Azizian & Lelarge (2021) surveys GNN expressiveness work.

Original: Leveraging Substructures in Learning: Exploiting subgraph information in GNNs is not new; in fact, k -WL considers all k node subgraphs.
Revised: Leveraging Substructures in Learning: Exploiting subgraph information in GNNs is not new; in fact, k -WL considers all k node subgraphs.

Original: Monti et al.
Revised: Monti et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: (2019) exploit motif information within aggregation, and others (Bouritsas et al., 2020; Barcel ´ o et al., 2021) augment MPNN features with handcrafted subgraph based features.
Revised: (2019) exploit motif information within aggregation, and others (Bouritsas et al., 2020; Barcel ´ o et al., 2021) augment MPNN features with handcrafted subgraph based features.

Original: MixHop (Abu-El-Haija et al., 2019) directly aggregates k -hop information by using adjacency matrix powers, ignoring neighbor connections.
Revised: MixHop (Abu-El-Haija et al., 2019) directly aggregates k -hop information by using adjacency matrix powers, ignoring neighbor connections.

Original: Towards a meta-learning goal, G-meta (Huang & Zitnik, 2020) applies GNNs on rooted subgraphs around each node to help transferring ability.
Revised: Towards a meta-learning goal, G-meta (Huang & Zitnik, 2020) applies GNNs on rooted subgraphs around each node to help transferring ability.

Original: Tahmasebi & Jegelka (2020) only theoretically justiﬁes subgraph convolution with GNN by showing its ability in counting substructures.
Revised: Tahmasebi & Jegelka (2020) only theoretically justiﬁes subgraph convolution with GNN by showing its ability in counting substructures.

Original: Zhengdao et al.
Revised: Zhengdao et al.

Original: (2020) also represent a node by encoding its local subgraph, however using non-scalable relational pooling.
Revised: (2020) also represent a node by encoding its local subgraph, however using non-scalable relational pooling.

Original: k -hop GNN (Nikolentzos et al., 2020) uses k -egonet in a specially designed way: it encodes a rooted subgraph via sequentially passing messages from k -th hops in the subgraph to k -1 hops, until it reaches the root node, and use the root node as encoding of the subgraph.
Revised: k -hop GNN (Nikolentzos et al., 2020) uses k -egonet in a specially designed way: it encodes a rooted subgraph via sequentially passing messages from k -th hops in the subgraph to k −hops, until it reaches the root node, and use the root node as encoding of the subgraph.

Original: EgoGNNs (Sandfelder et al., 2021)only computes a context encoding with SGC (Wu et al., 2019) as the subgraph encoder, and only be studied on node-level tasks.
Revised: Ego-GNNs (Sandfelder et al., 2021) computes a context encoding with SGC (Wu et al., 2019) as the subgraph encoder, and only be studied on node-level tasks.

Original: Both k -hop GNN and Ego-GNNs can be viewed as a special case of GNN-AK .
Revised: Both k -hop GNN and Ego-GNNs can be viewed as a special case of GNN-AK .

Revised: You et al.

Revised: (2021) designs ID-GNNs which inject node identity during message passing by coloring the k -egonet, which is the receptive ﬁeld of each node in k -layer GNN (Hamilton et al., 2017) – this differs from GNN-AK , which encodes each node by encoding its rooted subgraph.

Original: Unlike GNN-AK using rooted subgraphs, Fey et al.
Revised: Unlike GNN-AK which uses rooted subgraphs, Fey et al.

Original: (2020); Thiede et al.
Revised: Thiede et al.

Original: ; Bodnar et al.
Revised: Bodnar et al.

Original: (2021a)
Revised: (2021a)

Original: design GNNs to use certain subgraph patterns (like cycles and paths) inthe message passing phase, however their preprocessing requires solving the subgraph isomorphism problem.
Revised: design GNNs to use certain subgraph patterns (like cycles and paths) in message passing, however their preprocessing requires solving the subgraph isomorphism problem.

Original: To summarize, 1) no existing general  framework to uplift any existing GNNs; 2) no existing subgraph-based works address scalability issue involved with using subgraphs, which is a huge issue forall subgraph-based methods in practice.
Revised: To summarize, our work differs by (i) proposing a general subgraph encoding framework for uplifting GNNs, and (ii) addressing scalability issues involved with using subgraphs, which poses signiﬁcant challenges for subgraph-based methods in practice.

Original: T HE  G ENERAL F RAMEWORK AND T HEORY
Revised:  G ENERAL F RAMEWORK AND T HEORY

Original: We ﬁrst introduce our setting and formalisms.
Revised: We ﬁrst introduce our setting and formalisms.

Original: Let G =
Revised: Let G =

Original: ( V , E ) be a graph with node features x i ∈ R d , ∀ i ∈ V .
Revised: ( V , E ) be a graph with node features x i ∈ R d , ∀ i ∈ V .

Original: We consider graph-level problems where the goal is to classify/regress a target y G by learning a graph-level representation h G .
Revised: We consider graph-level problems where the goal is to classify/regress a target y G by learning a graph-level representation h G .

Original: Let N k ( v ) be the set of nodes in the k -hop egonet rooted at node v .
Revised: Let N k ( v ) be the set of nodes in the k -hop egonet rooted at node v .

Original: N ( v ) =
Revised: N ( v ) =

Original: N 1 ( v ) \ v denotes the immediate neighbors of node v .
Revised: N 1 ( v ) \ v denotes the immediate neighbors of node v .

Original: For S ⊆ V , let
Revised: For S ⊆ V , let

Original: G [ S ] be the induced subgraph: G [ S ] = ( S , { ( i, j
Revised: G [ S ] be the induced subgraph: G [ S ] = ( S , { ( i, j

Original: ) ∈ E| i ∈ S , j ∈ S} ) .
Revised: ) ∈ E| i ∈ S , j ∈ S} ) .

Original: Then G [ N k ( v )] denotes the k -hop egonet rooted at node v .
Revised: Then G [ N k ( v )] denotes the k -hop egonet rooted at node v .

Original: We also deﬁne Star ( v ) =
Revised: We also deﬁne Star ( v ) =

Original: ( N 1 ( v ) , { ( v,
Revised: ( N 1 ( v ) , { ( v,

Original: j ) ∈
Revised: j ) ∈

Original: E| j ∈ N ( v ) } ) be the induced star-like subgraph around v .
Revised: E| j ∈ N ( v ) } ) be the induced star-like subgraph around v .

Original: We use {·} denotes multiset, i.e. set that allows repetition.
Revised: We use {·} denotes multiset, i.e. set that allows repetition.

Original: Before presenting GNN-AK , we highlight the insights in designing GNN-AK and driving the expressiveness boost.
Revised: Before presenting GNN-AK , we highlight the insights in designing GNN-AK and driving the expressiveness boost.

Original: Insight 1: Generalizing star to subgraph.
Revised: Insight 1: Generalizing star to subgraph.

Original: In MPNNs, every node aggregates information from its immediate neighbors following a star pattern.
Revised: In MPNNs, every node aggregates information from its immediate neighbors following a star pattern.

Original: Consequently, MPNNs fail to distinguish any non-isomorphic regular graphs where all stars are the same, since all nodes have the same degree.
Revised: Consequently, MPNNs fail to distinguish any non-isomorphic regular graphs where all stars are the same, since all nodes have the same degree.

Original: Even simply generalizing star to the induced, 1-hop egonet considers connections among neighbors, enabling distinguishing regular graphs.
Revised: Even simply generalizing star to the induced, 1-hop egonet considers connections among neighbors, enabling distinguishing regular graphs.

Original: Insight 2: Divide and conquer.
Revised: Insight 2: Divide and conquer.

Original: When two graphs are non-isomorphic, there exists a subgraph where this difference is captured (see
Revised: When two graphs are non-isomorphic, there exists a subgraph where this difference is captured (see

Original: Figure 2).
Revised: Figure 2).

Original: Although a ﬁxed-expressiveness GNN may not distinguish the two original graphs, it may distinguish the two smaller subgraphs, given that the required expressiveness for successful discrimination is proportional to graph size (Loukas, 2020b).
Revised: Although a ﬁxed-expressiveness GNN may not distinguish the two original graphs, it may distinguish the two smaller subgraphs, given that the required expressiveness for successful discrimination is proportional to graph size (Loukas, 2020b).

Original: As such, GNN-AK divides the harder problem of encoding the whole graph to smaller and easier problems of encoding its subgraphs, and “conquers” the encoding with the base GNN.
Revised: As such, GNN-AK divides the harder problem of encoding the whole graph to smaller and easier problems of encoding its subgraphs, and “conquers” the encoding with the base GNN.

Original: 3. F ROM S TARS TO S UBGRAPHS
Revised: 3. F ROM S TARS TO S UBGRAPHS

Original: We ﬁrst take a close look at MPNNs, identifying their limitations and expressiveness bottleneck.
Revised: We ﬁrst take a close look at MPNNs, identifying their limitations and expressiveness bottleneck.

Original: MPNNs repeatedly update each node’s embedding by aggregating embeddings from their neighbors a ﬁxed number of times (layers) and computing a graph-level embedding h G by global pooling.
Revised: MPNNs repeatedly update each node’s embedding by aggregating embeddings from their neighbors a ﬁxed number of times (layers) and computing a graph-level embedding h G by global pooling.

Original: Let h ( l ) v denote the l -th layer embedding of node v .
Revised: Let h ( l ) v denote the l -th layer embedding of node v .

Original: Then, MPNNs compute h G by 
Revised: Then, MPNNs compute h G by h

Original: [Equation]
Revised: [Equation]

Revised: ) v =

Revised: [Equation]

Original: where h (0) i
Revised: where h (0) i

Original: = x i is the original features, L is the number of layers, and φ ( l ) and f ( l ) are the l -th layer update and aggregation functions.
Revised: = x i is the original features, L is the number of layers, and φ ( l ) and f ( l ) are the l -th layer update and aggregation functions.

Original: φ ( l ) , f ( l ) and POOL vary among different MPNNs and inﬂuence their expressiveness and performance.
Revised: φ ( l ) , f ( l ) and POOL vary among different MPNNs and inﬂuence their expressiveness and performance.

Original: MPNNs achieve maximum expressiveness ( 1 -WL) when all three functions are injective (Xu et al., 2018).
Revised: MPNNs achieve maximum expressiveness ( 1 -WL) when all three functions are injective (Xu et al., 2018).

Original: MPNNs’ expressiveness upper bound follows from its close relation to the 1-WL isomorphism test (Morris et al., 2019).
Revised: MPNNs’ expressiveness upper bound follows from its close relation to the 1-WL isomorphism test (Morris et al., 2019).

Original: Similar to MPNNs which repeatedly aggregate self and neighbor representations, at t -th iteration, for each node v , 1-WL test aggregates the node’s own label (or color) c
Revised: Similar to MPNNs which repeatedly aggregate self and neighbor representations, at t -th iteration, for each node v , 1-WL test aggregates the node’s own label (or color) c

Original: ( t ) v and its neighbors’ labels { c ( t ) u | u ∈ N ( v ) } , and hashes this multi-set of labels (cid:110)
Revised: ( t ) v and its neighbors’ labels { c ( t ) u | u ∈ N ( v ) } , and hashes this multi-set of labels (cid:110)

Original: c ( t ) v , { c ( t ) u | u ∈ N ( v ) } (cid:111) into a new, compressed label c ( t +1) v .
Revised: c ( t ) v , { c ( t ) u | u ∈ N ( v ) } (cid:111) into a new, compressed label c ( t +1) v .

Original: 1-WL outputs the set of all node labels (cid:8) c ( T ) v | v ∈ V (cid:9) as G ’s ﬁngerprint, and decides two graphs to be non-isomorphic as soon as their ﬁngerprints differ.
Revised: 1-WL outputs the set of all node labels (cid:8) c ( T ) v | v ∈ V (cid:9) as G ’s ﬁngerprint, and decides two graphs to be non-isomorphic as soon as their ﬁngerprints differ.

Original: The hash process in 1-WL outputs a new label c ( t +1) v that uniquely characterizes the star graph
Revised: The hash process in 1-WL outputs a new label c ( t +1) v that uniquely characterizes the star graph

Original: Star ( v ) around v , i.e. two nodes u, v are assigned different compressed labels only if Star ( u ) and
Revised: Star ( v ) around v , i.e. two nodes u, v are assigned different compressed labels only if Star ( u ) and

Original: Star ( v ) differ.
Revised: Star ( v ) differ.

Original: Hence, it is easy to see that when two non-isomorphic unlabeled (i.e., all nodes have the same label)
Revised: Hence, it is easy to see that when two non-isomorphic unlabeled (i.e., all nodes have the same label)

Original: d -regular graphs have the same number of nodes, 1 -WL cannot distinguish them.
Revised: d -regular graphs have the same number of nodes, 1 -WL cannot distinguish them.

Original: This failure limits the expressiveness of 1-WL, but also identiﬁes its bottleneck: the star is not distinguishing enough.
Revised: This failure limits the expressiveness of 1-WL, but also identiﬁes its bottleneck: the star is not distinguishing enough.

Original: Instead, we propose to generalize the star Star ( v ) to subgraphs, such as the egonet G [ N 1 ( v )] and more generally k -hop egonet G [ N k ( v )] .
Revised: Instead, we propose to generalize the star Star ( v ) to subgraphs, such as the egonet G [ N 1 ( v )] and more generally k -hop egonet G [ N k ( v )] .

Original: This results in an improved version of 1-WL which we call Subgraph-1-WL .
Revised: This results in an improved version of 1-WL which we call Subgraph-1-WL .

Original: Formally,
Revised: Formally,

Original: Deﬁnition 3.1 ( Subgraph-1-WL ) .
Revised: Deﬁnition 3.1 ( Subgraph-1-WL ) .

Original: Subgraph-1-WL generalizes the 1-WL graph isomorphism test algorithm by replacing color reﬁnement (at iteration t ) by c ( t +1) v = HASH
Revised: Subgraph-1-WL generalizes the 1-WL graph isomorphism test algorithm by replacing color reﬁnement (at iteration t ) by c ( t +1) v = HASH

Original: ( Star ( t ) ( v )) with c ( t +1) v = HASH
Revised: ( Star ( t ) ( v )) with c ( t +1) v = HASH

Original: ( G ( t ) [ N k ( v )]) , ∀ v ∈ V where HASH ( · ) is an injective function on graphs.
Revised: ( G ( t ) [ N k ( v )]) , ∀ v ∈ V where HASH ( · ) is an injective function on graphs.

Original: Note that an injective hash function for star graphs is equivalent to that for multi-sets, which is easy to derive (Zaheer et al., 2017).
Revised: Note that an injective hash function for star graphs is equivalent to that for multi-sets, which is easy to derive (Zaheer et al., 2017).

Original: In contrast, Subgraph-1-WL must hash a general subgraph , where an injective hash function for graphs is non-trivial (as hard as graph isomorphism).
Revised: In contrast, Subgraph-1-WL must hash a general subgraph , where an injective hash function for graphs is non-trivial (as hard as graph isomorphism).

Original: Thus, we derive a variant called Subgraph-1-WL ∗ by using a weaker choice for HASH ( · ) – speciﬁcally, 1 -WL.
Revised: Thus, we derive a variant called Subgraph-1-WL ∗ by using a weaker choice for HASH ( · ) – speciﬁcally, 1 -WL.

Original: Effectively, we nest 1 -WL inside Subgraph-1-WL.
Revised: Effectively, we nest 1 -WL inside Subgraph-1-WL.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 3.2 ( Subgraph-1-WL ∗ ) .
Revised: Deﬁnition 3.2 ( Subgraph-1-WL ∗ ) .

Original: Subgraph-1-WL ∗ is a less expressive variant of Subgraph-1WL where c ( t +1) v = 1-WL ( G ( t ) [ N k ( v )]) .
Revised: Subgraph-1-WL ∗ is a less expressive variant of Subgraph-1WL where c ( t +1) v = 1-WL ( G ( t ) [ N k ( v )]) .

Original: We further transfer Subgraph-1-WL to neural networks, resulting in GNN-AK whose expressiveness is upper bounded by Subgraph-1-WL.
Revised: We further transfer Subgraph-1-WL to neural networks, resulting in GNN-AK whose expressiveness is upper bounded by Subgraph-1-WL.

Original: The natural transformation with maximum expressiveness is to replace the hash function with a universal subgraph encoder of G [ N k ( v )] , which is non-trivial as it implies solving the challenging graph isomorphism problem in the worst case.
Revised: The natural transformation with maximum expressiveness is to replace the hash function with a universal subgraph encoder of G [ N k ( v )] , which is non-trivial as it implies solving the challenging graph isomorphism problem in the worst case.

Original: Analogous to using 1 -WL as a weaker choice for HASH ( · ) inside Subgraph-1-WL ∗ , we can use use any GNN (most practically, MPNN) as an encoder for subgraph G [ N k ( v )] .
Revised: Analogous to using 1 -WL as a weaker choice for HASH ( · ) inside Subgraph-1-WL ∗ , we can use use any GNN (most practically, MPNN) as an encoder for subgraph G [ N k ( v )] .

Original: Let G ( l ) [ N k ( v )]
Revised: Let G ( l ) [ N k ( v )]

Original: =
Revised: =

Original: G [ N k ( v ) | H ( l ) ] be the attributed subgraph with hidden features H ( l ) at the l -th layer.
Revised: G [ N k ( v ) | H ( l ) ] be the attributed subgraph with hidden features H ( l ) at the l -th layer.

Original: Then, GNN-AK computes h G by h ( l ) v = GNN ( l ) (cid:16) G ( l ) [ N k ( v )] (cid:17)
Revised: Then, GNN-AK computes h G by h [Equation] l [Equation]

Revised: [Equation]

Revised: v

Original: [Equation]
Revised: [Equation]

Original: Notice that GNN-AK acts as a “wrapper” for any base GNN (mainly MPNN).
Revised: Notice that GNN-AK acts as a “wrapper” for any base GNN (mainly MPNN).

Original: This uplifts its expressiveness as well as practical performance as we demonstrate in the following sections.
Revised: This uplifts its expressiveness as well as practical performance as we demonstrate in the following sections.

Original: 3. T HEORY : E XPRESSIVENESS A NALYSIS
Revised: 3. T HEORY : E XPRESSIVENESS A NALYSIS

Original: We next theoretically study the expressiveness of GNN-AK , by investigating the expressiveness of
Revised: We next theoretically study the expressiveness of GNN-AK , by investigating the expressiveness of

Original: Subgraph-1-WL.
Revised: Subgraph-1-WL.

Original: We ﬁrst establish that GNN-AK and Subgraph-1-WL have the same expressiveness under certain conditions.
Revised: We ﬁrst establish that GNN-AK and Subgraph-1-WL have the same expressiveness under certain conditions.

Original: A GNN is able to distinguish two graphs if its embeddings for two graphs are not identical.
Revised: A GNN is able to distinguish two graphs if its embeddings for two graphs are not identical.

Original: A GNN is said to have the same expressiveness as a graph isomorphism test when for any two graphs the GNN outputs different embeddings if and only if (iff) the isomorphism test deems them non-isomorphic.
Revised: A GNN is said to have the same expressiveness as a graph isomorphism test when for any two graphs the GNN outputs different embeddings if and only if (iff) the isomorphism test deems them non-isomorphic.

Original: We give following theorems with proof in Appendix.
Revised: We give following theorems with proof in Appendix.

Original: Theorem 1.
Revised: Theorem 1.

Original: When the base model is an MPNN with sufﬁcient number of layers and injective φ, f and POOL functions shown in Eq.
Revised: When the base model is an MPNN with sufﬁcient number of layers and injective φ, f and POOL functions shown in Eq.

Original: (1), and MPNN-AK has an injective POOL function shown in Eq.
Revised: (1), and MPNN-AK has an injective POOL function shown in Eq.

Original: (2), then MPNN-AK is as powerful as Subgraph-1-WL ∗ .
Revised: (2), then MPNN-AK is as powerful as Subgraph-1-WL ∗ .

Original: See Appendix.
Revised: See Appendix.

Original: A.2 for proof.
Revised: A.2 for proof.

Original: A more general version of Theorem 1 is that GNN-AK is as powerful as Subgraph-1-WL iff base GNN of GNN-AK is as powerful as the HASH function of Subgraph-1WL in distinguishing subgraphs, following the same proof logic.
Revised: A more general version of Theorem 1 is that GNN-AK is as powerful as Subgraph-1-WL iff base GNN of GNN-AK is as powerful as the HASH function of Subgraph-1WL in distinguishing subgraphs, following the same proof logic.

Original: The Theorem implies that we can characterize expressiveness of GNN-AK through studying Subgraph-1-WL and Subgraph-1-WL ∗ .
Revised: The Theorem implies that we can characterize expressiveness of GNN-AK through studying Subgraph-1-WL and Subgraph-1-WL ∗ .

Original: Theorem 2.
Revised: Theorem 2.

Original: Subgraph-1-WL ∗ is strictly more powerful than 1&2-WL 1 .
Revised: Subgraph-1-WL ∗ is strictly more powerful than 1&2-WL 1 .

Original: A direct corollary from Theorem 1&2 is as follows, which is empirically veriﬁed in Table 1.
Revised: A direct corollary from Theorem 1&2 is as follows, which is empirically veriﬁed in Table 1.

Original: Corollary 2.1.
Revised: Corollary 2.1.

Original: When MPNN is 1-WL expressive, MPNN-AK is strictly more powerful than 1&2-WL.
Revised: When MPNN is 1-WL expressive, MPNN-AK is strictly more powerful than 1&2-WL.

Original: Theorem 3.
Revised: Theorem 3.

Original: When HASH ( · ) is 3 -WL expressive, Subgraph-1-WL is no less powerful than 3 -WL, that is, it can discriminate some graphs for which 3 -WL fails.
Revised: When HASH ( · ) is 3 -WL expressive, Subgraph-1-WL is no less powerful than 3 -WL, that is, it can discriminate some graphs for which 3 -WL fails.

Original: A direct corollary from Theorem 1&3 is as follows, which is empirically veriﬁed in Table 1.
Revised: A direct corollary from Theorem 1&3 is as follows, which is empirically veriﬁed in Table 1.

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: PPGN-AK can distinguish some 3-WL-failed non-isomorphic graphs.
Revised: PPGN-AK can distinguish some 3-WL-failed non-isomorphic graphs.

Original: Theorem 4.
Revised: Theorem 4.

Original: For any k ≥ 3 , there exists a pair of k -WL-failed graphs that cannot be distinguished by Subgraph-1-WL even with injective HASH ( · ) when t -hop egonets are used with t ≤ 4 .
Revised: For any k ≥ 3 , there exists a pair of k -WL-failed graphs that cannot be distinguished by Subgraph-1-WL even with injective HASH ( · ) when t -hop egonets are used with t ≤ 4 .

Original: Theorem 4 is proven (Appendix.A.5) by observing that with limited t all rooted subgraphs of two non-isomorphic graphs from CFI ( k ) family (Cai et al., 1992) are isomorphic, i.e. local rooted subgraph is not enough to capture the “global” difference.
Revised: Theorem 4 is proven (Appendix.A.5) by observing that with limited t all rooted subgraphs of two non-isomorphic graphs from CFI ( k ) family (Cai et al., 1992) are isomorphic, i.e. local rooted subgraph is not enough to capture the “global” difference.

Original: This opens a future direction of generalizing rooted subgraph to general subgraph (as in k -WL) while keeping number of subgraphs in O( |V| ).
Revised: This opens a future direction of generalizing rooted subgraph to general subgraph (as in k -WL) while keeping number of subgraphs in O( |V| ).

Original: Proposition 1 (sufﬁcient conditions) .
Revised: Proposition 1 (sufﬁcient conditions) .

Original: For two non-isomorphic graphs G, H , Subgraph-1-WL with k -egonet can successfully distinguish them if: 1) for any node reordering v G 1 , ..., v G |V G | and v H 1 , ..., v H |V H | , ∃ i ∈ [1 , max( |V G | , |V H | )] that G [ N k ( v Gi )] and H [ N k ( v Hi )] are non-isomorphic 2 ; and 2) HASH ( · ) is discriminative enough that HASH ( G [ N k ( v Gi )]) (cid:54) =
Revised: For two non-isomorphic graphs G, H , Subgraph-1-WL with k -egonet can successfully distinguish them if: 1) for any node reordering v G 1 , ..., v G |V G | and v H 1 , ..., v H |V H | , ∃ i ∈ [1 , max( |V G | , |V H | )] that G [ N k ( v Gi )] and H [ N k ( v Hi )] are non-isomorphic 2 ; and 2) HASH ( · ) is discriminative enough that HASH ( G [ N k ( v Gi )]) (cid:54) =

Original: HASH ( H [ N k
Revised: HASH ( H [ N k

Original: v Hi )]) .
Revised: v Hi )]) .

Original: This implies subgraph size should be large enough to capture difference, but not larger which requires more expressive base model (Loukas, 2020a).
Revised: This implies subgraph size should be large enough to capture difference, but not larger which requires more expressive base model (Loukas, 2020a).

Original: We empirically verify Prop. 1 in Table 2.
Revised: We empirically verify Prop. 1 in Table 2.

Original:  C ONCRETE R EALIZATION
Revised:  C ONCRETE R EALIZATION

Original: Next, we present concrete designs to effectively and efﬁciently realize GNN-AK in practice. Specifically, we present (1) three types of encodings that fully exploit all intermediate node embeddings across rooted subgraphs; (2) a subgraph pooling design to incorporate distance-to-centroid, readily computed during subgraph extraction; and (3) a random-walk based rooted subgraph extraction for graphs with small diameter to reduce memory footprint of k -hop egonets that can reach almost entire graph even with a small k .
Revised: We ﬁrst realize GNN-AK with two type of encodings, and then present an empirically more expressive version, GNN-AK + , with (i) an additional context encoding , and (ii) a subgraph pooling design to incorporate distance-to-centroid, readily computed during subgraph extraction. Next, we discuss a random-walk based rooted subgraph extraction for graphs with small diameter to reduce memory footprint of k -hop egonets.

Original: We conclude this section with time and space complexity analysis.
Revised: We conclude this section with time and space complexity analysis.

Original: When |V G | < |V H | , ∀ i ∈ {|V G | , |V G | + 1 , ..., |V H |} , let G [ N k ( v G i )] denote an empty subgraph. Three Types of Encodings.
Revised: Notation.

Original: Let G =
Revised: Let G =

Original: ( V , E ) be theinput graph with N = |V| , G ( l ) [ N k ( v )] be the k -hop egonet rooted at node v ∈ V in which h ( l ) u denotes node u ’s hidden representation for u ∈ N k
Revised: ( V , E ) be the graph with N = |V| , G ( l ) [ N k ( v )] be the k -hop egonet rooted at node v ∈ V in which h ( l ) u denotes node u ’s hidden representation for u ∈ N k

Original: ( v ) at the l -th layer of GNN-AK .
Revised: ( v ) at the l -th layer of GNN-AK .

Original: To simplify notation, we use Sub ( l ) [ v ] instead of G ( l ) [ N k ( v )] togenerally indicate the the attribute-enriched induced subgraph for v .
Revised: To simplify notation, we use Sub ( l ) [ v ] instead of G ( l ) [ N k ( v )] to indicate the the attributeenriched induced subgraph for v .

Original: We consider all intermediate node embeddings across rooted subgraphs.
Revised: We consider all intermediate node embeddings across rooted subgraphs.

Original: Speciﬁcally, let Emb ( i | Sub ( l ) [ j ]) denote node
Revised: Speciﬁcally, let Emb ( i | Sub ( l ) [ j ]) denote node

Original: i ’s embedding when applying base GNN ( l ) on Sub ( l ) [ j ] ; we consider node embeddings for every j ∈ V and every i ∈ Sub ( l ) [ j ] .
Revised: i ’s embedding when applying base GNN ( l ) on Sub ( l ) [ j ] ; we consider node embeddings for every j ∈ V and every i ∈ Sub ( l ) [ j ] .

Original: Note that the base GNN can have multiplegraph convolutional layers, and Emb refers to the node embeddings at the last layer before global pooling POOL GNN that generates subgraph-level encoding. 
Revised: Note that the base GNN can have multiple convolutional layers, and Emb refers to the node embeddings at the last layer before global pooling POOL GNN that generates subgraph-level encoding. Realization of GNN-AK .

Original: Accordingly we can formally rewrite Eq.
Revised: We can formally rewrite Eq.

Original: (2) as
Revised: (2) as

Original: [Equation]
Revised: [Equation]

Original: The encoding in Eq. (3) is the default and referred to as subgraph encoding, as it encodes the rooted subgraph Sub ( l ) [ v ]  .
Revised: We refer to the encoding of the rooted subgraph Sub ( l ) [ v ] in Eq. (3) as the subgraph encoding .

Original: Typical choices of POOL GNN ( l ) are SUM and MEAN.
Revised: Typical choices of POOL GNN ( l ) are SUM and MEAN.

Original: We observe that subgraph encoding alone does not fully exploit all information side (cid:80) v ∈V |N k ( v ) | intermediate embeddings, and propose two additional encodings, namely centroid and context.
Revised: As each rooted subgraph has a root node, POOL GNN ( l ) can additionally be realized to differentiate the root node by self-concatenating its own representation, resulting in the following realization as each layer of GNN-AK :

Original: [Equation]
Revised: [Equation]

Original: We remark that these three different encodings contain non-redundant information, which is empir- ically veriﬁed in Table 7 : for node v , subgraph encoding captures the default encoding of rooted subgraph around v ; centroid encoding only takes the root of the subgraph, which is exactly the rep- resentation of applying base GNN on original graph when base GNN only has 1 layer; and context encoding captures views of node v from different subgraph points-of-view, i.e. representation in “context” of others.
Revised: (cid:17)

Original: Overall, we realize Eq.
Revised: where h ( l +1) | Centroid v :=

Original: (2) in practice by fusing all three encodings as [Equation] where FUSE is concatenation or sum.
Revised: Emb ( v | Sub ( l +1) [ v ]) (4) where FUSE is concatenation or sum, and h ( l ) | Centroid v is referred to as the centroid encoding .

Original: We illustrate in Figure 1 the l -th layer of GNN-AK .
Revised: The realization of GNN-AK in Eq.4 closely follows the theory in Sec.3.

Original: Subgraph Extraction and Distance-to-centroid (D2C) Feature.

Original:  GNN-AK extracts the rooted subgraph for every node and records the node and edge mapping between original graph and |V| rooted subgraphs. The subgraph extraction can be preprocessed ofﬂine to maximize speed, while in practice we implement it online without much overhead, thanks to fast k -hop propagation with complexity
Revised: Realization of GNN-AK + . We further develop GNN-AK + , which is more expressive than GNN-AK , based on two observations. First , we observe that Eq.4 does not fully exploit all information inside the rich intermediate embeddings generated for Eq.4, and propose an additional context encoding .

Original: O ( k |E| ) .
Revised: [Equation]

Original: Notably, along with  k -hop propagation for extracting k -hop egonet, the D2C within each subgraph is readily recorded at no additional cost and can be used to augment node features, which has been shown to help expressiveness (Li et al., 2020) theoretically. Therefore, GNN-AK uses the
Revised: Different from subgraph and centroid encodings, the context encoding captures views of node v from different subgraph contexts, or points-of-view. Second , GNN-AK extracts the rooted subgraph for every node with efﬁcient k -hop propagation (complexity O ( k |E| ) ), along which the distance-tocentroid (D2C) within each subgraph is readily recorded at no additional cost and can be used to augment node features; (Li et al., 2020) shows this theoretically improves expressiveness.

Original:  D2C by default in two ways: (1) augmenting hidden representation h ( l ) vin Eq. (6) by concatenating it with the encoding of D2C; (2) using it to gate the subgraph and context encodings before POOL Subgraph and POOL Context ,  the latter inspired by the hypothesis that embeddings  far from v contribute differently than those close to v .
Revised: Therefore, we propose to uses the D2C by default in two ways in GNN-AK + : (i) augmenting hidden representation h ( l ) v by concatenating it with the encoding of D2C; (ii) using it to gate the subgraph and context encodings before POOL Subgraph and POOL Context , with the intuition that embeddings of nodes far from v contribute differently than those close to v .

Original: To formalize the gate mechanism guided by D2C, let d ( l ) i | j be the encoding of distance from node i to j at l -th layer 3 .
Revised: To formalize the gate mechanism guided by D2C, let d ( l ) i | j be the encoding of distance from node i to j at l -th layer 3 .

Original: Applying gate mechanism changes Eq.
Revised: Applying gating changes Eq.

Original: (5) to
Revised: (5) to

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes element-wise multiplication.
Revised: where (cid:12) denotes element-wise multiplication.

Original: Similar changes apply to Eq.
Revised: Similar changes apply to Eq.

Revised: (3) to get h ( l ) | Subgraph gated, v .

Revised: Formally, each layer of GNN-AK + is deﬁned as

Original: We provide ablation study for the proposed node encodings and D2C feature in Appendix.
Revised: [Equation] Proposition 2. GNN-AK + is more powerful than GNN-AK . See Proof in Appendix.

Original: A.10.
Revised: A.A.7.

Original: Beyond k -egonet Subgraphs.
Revised: Beyond k -egonet Subgraphs.

Original: The k -hop egonet (or k -egonet) is a natural choice in our framework, but can be too large when the input graph’s diameter is small, as in social networks (Kleinberg, 2000), or when the graph is dense.
Revised: The k -hop egonet (or k -egonet) is a natural choice in our framework, but can be too large when the input graph’s diameter is small, as in social networks (Kleinberg, 2000), or when the graph is dense.

Original: To limit subgraph size, we also design a random-walk based subgraph extractor.
Revised: To limit subgraph size, we also design a random-walk based subgraph extractor.

Original: Speciﬁcally, to extract a subgraph rooted at node v , we perform a ﬁxed-length random walk starting at v , resulting in visited nodes N rw ( v ) and their induced subgraph G [ N rw ( v )] .
Revised: Speciﬁcally, to extract a subgraph rooted at node v , we perform a ﬁxed-length random walk starting at v , resulting in visited nodes N rw ( v ) and their induced subgraph G [ N rw ( v )] .

Original: In practice, we use adaptive random walks as in Grover & Leskovec (2016).
Revised: In practice, we use adaptive random walks as in Grover & Leskovec (2016).

Original: To reduce randomness, we use multiple truncated random walks and union the visited nodes as N rw ( v ) .
Revised: To reduce randomness, we use multiple truncated random walks and union the visited nodes as N rw ( v ) .

Original: Moreover, we employ online subgraph extraction during training that re-extracts subgraphs at every epoch, which further alleviates the effect of randomness via regularization.
Revised: Moreover, we employ online subgraph extraction during training that re-extracts subgraphs at every epoch, which further alleviates the effect of randomness via regularization.

Original: Complexity Analysis.
Revised: Complexity Analysis.

Original: Let us assume k -egonets as rooted subgraphs, and an MPNN as base model.
Revised: Assuming k -egonets as rooted subgraphs, and an MPNN as base model.

Original: For each graph G =
Revised: For each graph G =

Original: ( V , E ) , the subgraph extraction takes O ( k |E| ) runtime complexity, and outputs |V| subgraphs, which collectively can be represented as a union graph G ∪ =
Revised: ( V , E ) , the subgraph extraction takes O ( k |E| ) runtime complexity, and outputs |V| subgraphs, which collectively can be represented as a union graph G ∪ =

Original: ( V ∪ , E ∪ ) with |V| disconnected components, where |V ∪ | = (cid:80) v ∈V |N
Revised: ( V ∪ , E ∪ ) with |V| disconnected components, where |V ∪ | = (cid:80) v ∈V |N

Original: k ( v ) | and |E ∪ | = (cid:80) v ∈V |E G [ N k ( v )] | .
Revised: k ( v ) | and |E ∪ | = (cid:80) v ∈V |E G [ N k ( v )] | .

Original: GNN-AK  can be viewed as applying base GNN on the union graph.
Revised: GNN-AK ( + ) can be viewed as applying base GNN on the union graph.

Original: Assumingthe base GNN has O ( |V| + |E| ) runtime and memory complexity at forward pass, GNN-AK  has O ( |V ∪ | + |E ∪ | ) runtime and memory cost.
Revised: Assuming base GNN has O ( |V| + |E| ) runtime and memory complexity, GNN-AK ( + ) has O ( |V ∪ | + |E ∪ | ) runtime and memory cost.

Original: For rooted subgraphs of size s , GNN-AK  induces an O ( s ) factor overhead over the base model.
Revised: For rooted subgraphs of size s , GNN-AK ( + ) induces an O ( s ) factor overhead over the base model.

Original:  I MPROVING S CALABILITY : S UBGRAPH D ROP
Revised:  I MPROVING S CALABILITY : S UBGRAPH D ROP

Original: The complexity analysis reveals that GNN-AK introduces a constant factor overhead (subgraph size) in runtime and memory over the base GNN.
Revised: The complexity analysis reveals that GNN-AK ( + ) introduce a constant factor overhead (subgraph size) in runtime and memory over the base GNN.

Original: Subgraph size can be naturally reduced by choosing smaller k for k -egonet, or by ranking and truncating visited nodes in a random walk setting.
Revised: Subgraph size can be naturally reduced by choosing smaller k for k -egonet, or by ranking and truncating visited nodes in a random walk setting.

Original: However limiting to very small subgraphs tends to hurt performance as we empirically show in Table 6.
Revised: However limiting to very small subgraphs tends to hurt performance as we empirically show in Table 6.

Original: Here, we present a different subsampling-based approach that carefully selects only a subset of the |V| rooted subgraphs.
Revised: Here, we present a different subsampling-based approach that carefully selects only a subset of the |V| rooted subgraphs.

Original: Further more, inspired from Dropout (Srivastava et al., 2014), we only drop subgraphs during training while still use all subgraphs when evaluation.
Revised: Further more, inspired from Dropout (Srivastava et al., 2014), we only drop subgraphs during training while still use all subgraphs when evaluation.

Original: Novel strategies are designed speciﬁcally for three type of encodings to eliminate the estimation bias between training and evaluation.
Revised: Novel strategies are designed speciﬁcally for three type of encodings to eliminate the estimation bias between training and evaluation.

Original: We name it SubgraphDrop for dropping subgraphs during training.
Revised: We name it SubgraphDrop for dropping subgraphs during training.

Original: SubgraphDrop signiﬁcantly reduces memory overhead while keeping performance nearly the same as training with all subgraphs.
Revised: SubgraphDrop signiﬁcantly reduces memory overhead while keeping performance nearly the same as training with all subgraphs.

Original: We ﬁrst present subgraph sampling strategies, then introduce the designs of aligning training and evaluation.
Revised: We ﬁrst present subgraph sampling strategies, then introduce the designs of aligning training and evaluation.

Original: Fig.
Revised: Fig.

Original: 5 in Appendix.
Revised: 5 in Appendix.

Original: A.7 provides a pictorial illustration.
Revised: A.8 provides a pictorial illustration.

Original: Subgraph Sampling.
Revised: Subgraph Sampling.

Original: Intuitively, if u, v are directly connected in G , subgraphs G [ N k ( u )] and
Revised: Intuitively, if u, v are directly connected in G , subgraphs G [ N k ( u )] and

Original: G [ N k ( v )] share a large overlap and may contain redundancy.
Revised: G [ N k ( v )] share a large overlap and may contain redundancy.

Original: With this intuition, we aim to sample only m (cid:28)|V| minimally redundant subgraphs to reduce memory overhead.
Revised: With this intuition, we aim to sample only m (cid:28)|V| minimally redundant subgraphs to reduce memory overhead.

Original: We study three fast sampling strategies (See Appendix. A.7) that select subgraphs to evenly cover the whole graph, where each node is covered by ≈ R (redundancy factor) selected subgraphs; R is a hyperparameter used as a sampling stopping condition.
Revised: We study three fast sampling strategies (See Appendix.A.8) that select subgraphs to evenly cover the whole graph, where each node is covered by ≈ R (redundancy factor) selected subgraphs; R is a hyperparameter used as a sampling stopping condition.

Original: Then, GNN-AK  -S (with Sampling) has roughly R times the overhead of base model ( R ≤ 3 in practice).
Revised: Then, GNN-AK ( + ) -S (with Sampling) has roughly R times the overhead of base model ( R ≤ 3 in practice).

Original: We remark that our subsampling strategies are randomized and fast, which are both desired characteristics for an online Dropout-like sampling during training.
Revised: We remark that our subsampling strategies are randomized and fast, which are both desired characteristics for an online Dropout-like sampling in training.

Original: Training with SubgraphDrop.
Revised: Training with SubgraphDrop.

Original: Although dropping redundant subgraphs greatly reduces overhead, it still loses information.
Revised: Although dropping redundant subgraphs greatly reduces overhead, it still loses information.

Original: Thus, as in Dropout (Srivastava et al., 2014), we only “drop” subgraphs during training while still using all of them during evaluation.
Revised: Thus, as in Dropout (Srivastava et al., 2014), we only “drop” subgraphs during training while still using all of them during evaluation.

Original: Randomness in sampling strategies enforces that selected subgraphs differ across training epochs, preserving most information due to amortization.
Revised: Randomness in sampling strategies enforces that selected subgraphs differ across training epochs, preserving most information due to amortization.

Original: On the other hand, it makes it difﬁcult for the three encodings during training to align with full-mode evaluation.
Revised: On the other hand, it makes it difﬁcult for the three encodings during training to align with full-mode evaluation.

Original: Next, we propose an alignment procedure for each type of encoding.
Revised: Next, we propose an alignment procedure for each type of encoding.

Original: Subgraph and Centroid Encoding in Training : Let S ⊆
Revised: Subgraph and Centroid Encoding in Training: Let S ⊆

Original: V be the set of root nodes of the selected subgraphs.
Revised: V be the set of root nodes of the selected subgraphs.

Original: When sampling during training, subgraph and centroid encoding can only be computed for nodes v ∈ S , following Eq.
Revised: When sampling during training, subgraph and centroid encoding can only be computed for nodes v ∈ S , following Eq.

Original: (3) and Eq.
Revised: (3) and Eq.

Original: (4), resulting incomplete subgraph encodings { h ( l ) | Subgraph v | v ∈ S} and centroid encodings { h ( l ) | Centroid v | v ∈ S} .
Revised: (4), resulting incomplete subgraph encodings { h ( l ) | Subgraph v | v ∈ S} and centroid encodings { h ( l ) | Centroid v | v ∈ S} .

Original: To estimate uncomputed encodings of u ∈ V \ S , we propose to propagate encodings from S to V \ S .
Revised: To estimate uncomputed encodings of u ∈ V \ S , we propose to propagate encodings from S to V \ S .

Original: Formally, let k max = max u ∈V\S Dist ( u, S ) where Dist ( u, S )
Revised: Formally, let k max = max u ∈V\S Dist ( u, S ) where Dist ( u, S )

Original: =
Revised: =

Original: min v ∈S ShortestPathDistance ( u, v ) .
Revised: min v ∈S ShortestPathDistance ( u, v ) .

Original: Then we partition U =
Revised: Then we partition U =

Original: V \S into {U 1 , ..., U k max } with U d = { u ∈ U| Dist ( u, S ) =
Revised: V \S into {U 1 , ..., U k max } with U d = { u ∈ U| Dist ( u, S ) =

Original: d } .
Revised: d } .

Original: We propose to spread vector encodings of S out iteratively, i.e. compute vectors of U d from U d − 1 .
Revised: We propose to spread vector encodings of S out iteratively, i.e. compute vectors of U d from U d − 1 .

Original: Formally, we haveh
Revised: Formally, we have

Original: [Equation]
Revised: [Equation]

Original: Context Encoding in Training: Following Eq.
Revised: Context Encoding in Training: Following Eq.

Original: (5), context encodings can be computed for every node v ∈ V as each node is covered at least R times during training with SubgraphDrop.
Revised: (5), context encodings can be computed for every node v ∈ V as each node is covered at least R times during training with SubgraphDrop.

Original: However when POOL Context is SUM ( · ) , the scale of h ( l ) | Context v is smaller than the one in full-mode evaluation.
Revised: However when POOL Context is SUM ( · ) , the scale of h ( l ) | Context v is smaller than the one in full-mode evaluation.

Original: Thus, we scale the context encodings up to align with full-mode evaluation.
Revised: Thus, we scale the context encodings up to align with full-mode evaluation.

Original: Formally,h
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: | Context [Equation] When POOL Context is MEAN ( · ) , the context encoding is computed without any modiﬁcation.
Revised: When POOL Context is MEAN ( · ) , the context encoding is computed without any modiﬁcation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we (1) empirically verify the expressiveness beneﬁt of GNN-AK  on 4 simulation datasets; (2) show GNN-AK  boosts practical performance signiﬁcantly on 5 real-world datasets; (3) demonstrate the effectiveness of SubgraphDrop ; (4) conduct ablation studies of concrete designs.
Revised: In this section we (1) empirically verify the expressiveness beneﬁt of GNN-AK ( + ) on 4 simulation datasets; (2) show GNN-AK ( + ) boosts practical performance signiﬁcantly on 5 real-world datasets; (3) demonstrate the effectiveness of SubgraphDrop ; (4) conduct ablation studies of concrete designs.

Original: Simulation Datasets: 1) EXP (Abboud et al., 2020) contains 600 pairs of 1&2 -WL failed graphs that are splited into two classes where each graph of a pair is assigned to two different classes.
Revised: Simulation Datasets: 1) EXP (Abboud et al., 2021) contains 600 pairs of 1&2 -WL failed graphs that are splited into two classes where each graph of a pair is assigned to two different classes.

Original: 2) SR25 (Balcilar et al., 2021) has 15 strongly regular graphs ( 3 -WL failed) with 25 nodes each.
Revised: SR25 (Balcilar et al., 2021) has 15 strongly regular graphs ( 3 -WL failed) with 25 nodes each.

Original: SR25 is translated to a 15 way classiﬁcation problem with the goal of mapping each graph into a different class.
Revised: SR is translated to a 15 way classiﬁcation problem with the goal of mapping each graph into a different class.

Original: 3) Substructure counting (i.e. triangle, tailed triangle, star and 4-cycle) problems on random graph dataset (Zhengdao et al., 2020).
Revised: 3) Substructure counting (i.e. triangle, tailed triangle, star and 4-cycle) problems on random graph dataset (Zhengdao et al., 2020).

Original: 4) Graph property regression (i.e. connectedness, diameter, radius) tasks on random graph dataset (Corso et al., 2020).
Revised: 4) Graph property regression (i.e. connectedness, diameter, radius) tasks on random graph dataset (Corso et al., 2020).

Original: All simulation datasets are used to empirically verify the expressiveness of GNN-AK  .
Revised: All simulation datasets are used to empirically verify the expressiveness of GNN-AK ( + ) .

Original: Real-world Datasets: ZINC-12K, CIFAR10,
Revised: Real-world Datasets: ZINC-12K, CIFAR10,

Original: PATTER from Benchmarking GNNs (Dwivedi et al., 2020) and MolHIV, and MolPCBA from Open
Revised: PATTER from Benchmarking GNNs (Dwivedi et al., 2020) and MolHIV, and MolPCBA from Open

Original: Graph Benchmark (Hu et al., 2020).
Revised: Graph Benchmark (Hu et al., 2020).

Original: See Table 5 for all dataset statistics.
Revised: See Table 5 for all dataset statistics.

Original: Baselines.
Revised: Baselines.

Original: We use GCN (Kipf & Welling, 2017), GIN (Xu et al., 2018), PNA ∗ 4 (Corso et al., 2020), and 3-WL powerful PPGN (Maron et al., 2019a) directly, which also server as base model of GNNAK to see its general uplift effect.
Revised: We use GCN (Kipf & Welling, 2017), GIN (Xu et al., 2018), PNA ∗ 4 (Corso et al., 2020), and 3-WL powerful PPGN (Maron et al., 2019a) directly, which also server as base model of GNNAK to see its general uplift effect.

Original: GatedGCN (Dwivedi et al., 2020), DGN (Beani et al., 2021), PNA (Corso et al., 2020), GSN(Bouritsas et al., 2020), HIMP (Fey et al., 2020), and CIN (Bodnar et al., 2021a) are referenced directly from literature for real-world datasets comparison.
Revised: GatedGCN (Dwivedi et al., 2020), DGN (Beani et al., 2021), PNA (Corso et al., 2020), GSN(Bouritsas et al., 2020), HIMP (Fey et al., 2020), and CIN (Bodnar et al., 2021a) are referenced directly from literature for real-world datasets comparison.

Original: Hyperparameter and model conﬁguration are described in Appendix.
Revised: Hyperparameter and model conﬁguration are described in Appendix.

Original: A. [Table]
Revised: A.10.

Original: 6. E MPIRICAL V ERIFICATION OF E XPRESSIVENESS
Revised: 6. E MPIRICAL V ERIFICATION OF E XPRESSIVENESS

Original: [Table]
Revised: [Table]

Original: Table 1 presents the results on simulation datasets.
Revised: Table 1 presents the results on simulation datasets.

Revised: To save space we present GNN-AK + with different base models but only one one version of GNN-AK : GIN-AK .

Original: All GNN-AK  variants perform perfectly on EXP, while only PPGN alonecould do so previously.
Revised: All GNN-AK ( + ) variants perform perfectly on EXP, while only PPGN alone do so previously.

Original: Moreover, PPGN-AK  reaches perfect accuracy on SR25, while PPGN fails.
Revised: Moreover, PPGN-AK + reaches perfect accuracy on SR25, while PPGN fails.

Original: Similarly, GNN-AK  consistently boosts all MPNNs for substructure and graph property predictiontasks ( PPGN-AK  is OOM as it is quadratic in input size)
Revised: Similarly, GNN-AK ( + ) consistently boosts all MPNNs for substructure and graph property prediction ( PPGN-AK + is OOM as it is quadratic in input size).

Original: [Table]
Revised: [Table]

Original: [Table] In Table 2 we look into PPGN-AK ’s performance on SR25 as a function of kegonets ( k ∈ [1 , 2] ), as well as the number of PPGN (inner) layers and (outer) iterations for PPGN-AK .
Revised: In Table 2 we look into PPGN-AK ’s performance on SR25 as a function of kegonets ( k ∈ [1 , 2] ), as well as the number of PPGN (inner) layers and (outer) iterations for PPGN-AK .

Original: We ﬁnd that at least 2 inner layers is needed with 1egonet subgraphs to achieve top performance.
Revised: We ﬁnd that at least 2 inner layers is needed with 1egonet subgraphs to achieve top performance.

Original: With 2-egonets more inner layers helps, although performance is sub-par, attributed to PPGN’s disability to distinguish larger (sub)graphs, aligning with Conjecture 1 (Sec. 3.2).
Revised: With 2-egonets more inner layers helps, although performance is sub-par, attributed to PPGN’s disability to distinguish larger (sub)graphs, aligning with Conjecture 1 (Sec. 3.2).

Original: 6. C OMPARING WITH SOTA AND G ENERALITY
Revised: 6. C OMPARING WITH SOTA AND G ENERALITY

Original: Having studied expressiveness tasks [Table] we turn to performance on real-world datasets, as shown in
Revised: Having studied expressiveness tasks, we turn to performance on real-world datasets, as shown in

Original: Table 3.
Revised: Table 3.

Original: We observe similar performance lifts across all datasets and base GNNs (we omit PPGN due to scalability), demonstrating our framework’s generality.
Revised: We observe similar performance lifts across all datasets and base GNNs (we omit PPGN due to scalability), demonstrating our framework’s generality.

Original: Remarkably, GNN-AK  sets new SOTA performance for several benchmarks – ZINC, CIFAR10, and PATTERN – with a relative error re-
Revised: Remarkably, GNN-AK + sets new SOTA performance for several benchmarks – ZINC, CIFAR10, and PATTERN – with a relative error reduction of 60.

Original: duction of 60.3%, 50.5%, and 39.4% for base model being GCN, GIN, and PNA ∗ respectively.

Original: [Table]
Revised: [Table]

Original: 6. S CALING UP BY S UBSAMPLING
Revised: 6. S CALING UP BY S UBSAMPLING

Original: In some cases
Revised: In some cases

Original: [Table]
Revised: [Table]

Original: GNN-AK ’s overhead leads to OOM, especially for complex models like PNA ∗ that are resource-demanding.
Revised: GNN-AK ( + )’s overhead leads to OOM, especially for complex models like PNA ∗ that are resource-demanding.

Original: Sampling with SubgraphDrop enables training using practical resources.
Revised: Sampling with SubgraphDrop enables training using practical resources.

Original: Notably
Revised: Notably

Original: [Table]
Revised: [Table]

Original: GNN-AK  -S models, shown at the end of Table [Table] do not compromise and can even improve performance as compared to their non-sampled counterpart, in alignment with Dropout’s beneﬁts Srivastava et al.
Revised: GNN-AK + -S models, shown at the end of Table 3, do not compromise and can even improve performance as compared to their non-sampled counterpart, in alignment with Dropout’s beneﬁts Srivastava et al.

Original: Next we evaluate resource-savings, speciﬁcally on ZINC-12K and CIFAR10.
Revised: Next we evaluate resource-savings, speciﬁcally on ZINC-12K and CIFAR10.

Original: Table 4 shows that GIN-AK-S with varying R provides an effective handle to trade off resources with performance.
Revised: Table 4 shows that GIN-AK + -S with varying R provides an effective handle to trade off resources with performance.

Original: Importantly, the rate in which performance decays with smaller R is much lower than the rates at which runtime and memory decrease.
Revised: Importantly, the rate in which performance decays with smaller R is much lower than the rates at which runtime and memory decrease.

Original: [Table]
Revised: [Table]

Original: Ablation Study.
Revised: Ablation Study.

Original: We present ablation results on various structural components of GNN-AK  in Appendix A.10.
Revised: We present ablation results on various structural components of GNN-AK ( + ) in Appendix A.11.

Original: Table 6 shows the performance of GIN-AK  for varying size egonets with k .
Revised: Table 6 shows the performance of GIN-AK + for varying size egonets with k .

Original: Tableillustrates the added beneﬁt of various encodings and D2C feature.
Revised: Table 7 illustrates the added beneﬁt of various encodings and D2C feature.

Original: Table 8 extensively studies the effect of context encoding and D2C
Revised: Table 8 extensively studies the effect of context encoding and D2C

Original: [Table]
Revised: [Table]

Original: as they are not explained by Subgraph-1-WL ∗ .
Revised: as they are not explained by Subgraph-1-WL ∗ .

Original: Table 9 studies the effect of base model
Revised: Table 9 studies the effect of base model

Original: ’s depth (or expressiveness) for GNN-AK  with and without D2C.
Revised: ’s depth (or expressiveness) for GNN-AK + with and without D2C.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work introduces a new, general-purpose framework called GNN-As-Kernel ( GNN-AK ) to uplift the expressiveness of any GNN, with the key idea of employing a base GNN as a kernel on induced subgraphs of the input graph, generalizing from the star-pattern aggregation of classical MPNNs.
Revised: Our work introduces a new, general-purpose framework called GNN-As-Kernel ( GNN-AK ) to uplift the expressiveness of any GNN, with the key idea of employing a base GNN as a kernel on induced subgraphs of the input graph, generalizing from the star-pattern aggregation of classical MPNNs.

Original: Our approach provides an expressiveness and performance boost, while retaining practical scalability of MPNNs—a highly sought-after middle ground between the two regimes of scalable yet lessexpressive MPNNs and high-expressive yet practically infeasible and poorly-generalizing k -order designs.
Revised: Our approach provides an expressiveness and performance boost, while retaining practical scalability of MPNNs—a highly sought-after middle ground between the two regimes of scalable yet less-expressive MPNNs and high-expressive yet practically infeasible and poorly-generalizing korder designs.

Original: We theoretically studied the expressiveness of GNN-AK ,as well as provided a concrete design, introducing SubgraphDropduring training for shrinking runtime and memory footprint.
Revised: We theoretically studied the expressiveness of GNN-AK , provided a concrete design and the more powerful GNN-AK + , introducing SubgraphDrop for shrinking runtime and memory footprint.

Original: Extensive experiments on both simulated and real-world benchmark datasets empirically justiﬁed that GNN-AK  (i) uplifts base GNN expressiveness for multiple base GNN choices (e.g. over 1 / 2{+1&
+} -WLfor
Revised: Extensive experiments on both simulated and real-world benchmark datasets empirically justiﬁed that GNN-AK ( + ) (i) uplifts base GNN expressiveness for multiple base GNN choices (e.g. over {+1&
+} -WL

Original:  MPNNs, and over 3 -WL for PPGN), (ii) which translates to performance gains with SOTA results onseveral graph-level benchmarks, (iii) while retaining scalability to practical graphs.
Revised: for MPNNs, and over 3 -WL for PPGN), (ii) which translates to performance gains with SOTA results on graph-level benchmarks, (iii) while retaining scalability to practical graphs.

Original: A A PPENDIX A. S URVEY OF C ONNECTIONS TO CNN AND k -WL
Revised: A A PPENDIX A. S URVEY OF C ONNECTIONS TO CNN AND k -WL

Original: GNN-AK has a similar convolutional structure as CNN, and in fact historically many spatial GNNs are inspired by CNN; see Wu et al.
Revised: GNN-AK has a similar convolutional structure as CNN, and in fact historically many spatial GNNs are inspired by CNN; see Wu et al.

Original: (2020) for a detailed survey.
Revised: (2020) for a detailed survey.

Original: The non-Euclidean nature of graphs makes such generalizations non-trivial.
Revised: The non-Euclidean nature of graphs makes such generalizations non-trivial.

Original: MoNet (Monti et al., 2017) introduces pseudo-coordinates for nodes, while PatchySAN (Niepert et al., 2016) learns to order and truncate neighboring nodes for convolution purposes.
Revised: MoNet (Monti et al., 2017) introduces pseudo-coordinates for nodes, while PatchySAN (Niepert et al., 2016) learns to order and truncate neighboring nodes for convolution purposes.

Original: However, both methods aim to mimic the formulation of the CNN without admitting the inherent difference between graphs and images.
Revised: However, both methods aim to mimic the formulation of the CNN without admitting the inherent difference between graphs and images.

Original: In contrast, GNN-AK , generalizes
Revised: In contrast, GNN-AK , generalizes

Original: CNN to graphs with a base GNN kernel, similar to how a CNN kernel encodes image patches.
Revised: CNN to graphs with a base GNN kernel, similar to how a CNN kernel encodes image patches.

Original: GNN-AK also shares connections with two variants of k -WL test algorithms:
Revised: GNN-AK also shares connections with two variants of k -WL test algorithms:

Original: depth- k 1 -dim WL
Revised: depth- k 1 -dim WL

Original: (Cai et al., 1992; Weisfeiler, 1976) and deep WL (Arvind et al., 2020).
Revised: (Cai et al., 1992; Weisfeiler, 1976) and deep WL (Arvind et al., 2020).

Original: The former recursively applies 1-WL to all size- k subgraphs, with slightly weaker expressiveness than k -WL, and the latter reduces the number of such subgraphs for the k -WL test.
Revised: The former recursively applies 1-WL to all size- k subgraphs, with slightly weaker expressiveness than k -WL, and the latter reduces the number of such subgraphs for the k -WL test.

Original: Instead of working on all O ( n k ) size-
Revised: Instead of working on all O ( n k ) size-

Original: k subgraphs, we keep linear scalability by only applying 1-WL-equivalent MPNNs to O( n ) rooted subgraphs.
Revised: k subgraphs, we keep linear scalability by only applying 1-WL-equivalent MPNNs to O( n ) rooted subgraphs.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: (Xu et al., 2018) proved that with sufﬁcient number of layers and all injective functions,
Revised: (Xu et al., 2018) proved that with sufﬁcient number of layers and all injective functions,

Original: MPNN is as powerful as 1-WL.
Revised: MPNN is as powerful as 1-WL.

Original: Then Eq.2 outputs different vectors for two graphs
Revised: Then Eq.2 outputs different vectors for two graphs

Original: iff Subgraph1-WL ∗ encodes different labels with c ( t +1) v = 1-WL ( G ( t ) [ N k ( v )]) .
Revised: iff Subgraph1-WL ∗ encodes different labels with c ( t +1) v = 1-WL ( G ( t ) [ N k ( v )]) .

Original: With POOL in Eq.2 also to be injective, MPNN-AK outputs different vectors iff Subgraph-1-WL ∗ outputs different ﬁngerprints for two graphs.
Revised: With POOL in Eq.2 also to be injective, MPNN-AK outputs different vectors iff Subgraph-1-WL ∗ outputs different ﬁngerprints for two graphs.

Original: Then MPNN-AK is as powerful as Subgraph-1-WL ∗ .
Revised: Then MPNN-AK is as powerful as Subgraph-1-WL ∗ .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We ﬁrst prove that if two graphs are identiﬁed as isomorphic by Subgraph-1-WL ∗ , they are also determined as isomorphic by 1-WL.
Revised: We ﬁrst prove that if two graphs are identiﬁed as isomorphic by Subgraph-1-WL ∗ , they are also determined as isomorphic by 1-WL.

Original: Then we present a pair of non-isomorphic graphs that can be distinguished by Subgraph-1-WL ∗ but not by 1-WL.
Revised: Then we present a pair of non-isomorphic graphs that can be distinguished by Subgraph-1-WL ∗ but not by 1-WL.

Original: Together these two imply that Subgraph-1WL ∗ is strictly more powerful than 1-WL.
Revised: Together these two imply that Subgraph-1WL ∗ is strictly more powerful than 1-WL.

Original: Comparing with 2-WL can be concluded from the fact that 1-WL and 2-WL are equivalent in expressiveness (Maron et al., 2019a).
Revised: Comparing with 2-WL can be concluded from the fact that 1-WL and 2-WL are equivalent in expressiveness (Maron et al., 2019a).

Original: In the proof we use 1-hop egonet subgraphs for Subgraph-1-WL ∗ .
Revised: In the proof we use 1-hop egonet subgraphs for Subgraph-1-WL ∗ .

Original: Assume graphs G and H have the same number of nodes (otherwise easily determined as nonisomorphic) and are two non-isomorphic graphs but Subgraph-1-WL ∗ determines them as isomorphic.
Revised: Assume graphs G and H have the same number of nodes (otherwise easily determined as nonisomorphic) and are two non-isomorphic graphs but Subgraph-1-WL ∗ determines them as isomorphic.

Original: Then for any iteration t , set (cid:110) 1-WL ( G ( t )
Revised: Then for any iteration t , set (cid:110) 1-WL ( G ( t )

Original: [ N 1 ( v )]) | v ∈ V G (cid:111) is the same as set (cid:110)
Revised: [ N 1 ( v )]) | v ∈ V G (cid:111) is the same as set (cid:110)

Original: 1-WL ( H ( t )
Revised: 1-WL ( H ( t )

Original: [ N 1 ( v )]) | v ∈ V H (cid:111) .
Revised: [ N 1 ( v )]) | v ∈ V H (cid:111) .

Original: Then there existing an ordering of nodes v G 1 , ..., v Gn and v H 1 , ..., v HN with N =
Revised: Then there existing an ordering of nodes v G 1 , ..., v Gn and v H 1 , ..., v HN with N =

Original: |V G | =
Revised: |V G | =

Original: |V H | , such that for any node order i = 1 , ..., N , 1-WL ( G ( t ) [ N 1 ( v Gi )])
Revised: |V H | , such that for any node order i = 1 , ..., N , 1-WL ( G ( t ) [ N 1 ( v Gi )])

Original: = 1-WL
Revised: = 1-WL

Original: ( H ( t )
Revised: ( H ( t )

Original: [ N 1 ( v Hi )]) .
Revised: [ N 1 ( v Hi )]) .

Original: This implies that structure G [ N 1 ( v Gi )] and H [ N 1 ( v Hi )] are not distinguishable by 1-WL.
Revised: This implies that structure G [ N 1 ( v Gi )] and H [ N 1 ( v Hi )] are not distinguishable by 1-WL.

Original: Hence Star ( v G i ) and Star ( v H i ) are hashed to the same label otherwise the 1-WL that includes the hashed result of Star ( v Gi ) and
Revised: Hence Star ( v G i ) and Star ( v H i ) are hashed to the same label otherwise the 1-WL that includes the hashed result of Star ( v Gi ) and

Original: Star ( v Hi ) can also distinguish G [ N 1 ( v Gi )] and H [ N 1 ( v Hi )] .
Revised: Star ( v Hi ) can also distinguish G [ N 1 ( v Gi )] and H [ N 1 ( v Hi )] .

Original: Then for any iteration t and any node with order i ,
Revised: Then for any iteration t and any node with order i ,

Original: HASH (cid:16) Star
Revised: HASH (cid:16) Star

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ) (cid:17) implies that 1-WL fails in distinguishing G and
Revised: ) (cid:17) implies that 1-WL fails in distinguishing G and

Original: H .
Revised: H .

Original: In fact if we replace the 1-WL hashing function in Subgraph-1-WL ∗ to a stronger version
Revised: In fact if we replace the 1-WL hashing function in Subgraph-1-WL ∗ to a stronger version

Original: HASH (cid:16)(cid:8) HASH ( Star ( v G ( t ) )) , 1-WL ( G ( t ) [ N 1 ( v )]) (cid:9)(cid:17) , this directly implies the above statement.
Revised: HASH (cid:16)(cid:8) HASH ( Star ( v G ( t ) )) , 1-WL ( G ( t ) [ N 1 ( v )]) (cid:9)(cid:17) , this directly implies the above statement.

Original: In Figure 2, two 4-regular graphs are presented that cannot be distinguished by 1-WL but can be distinguished by Subgraph-1-WL ∗ .
Revised: In Figure 2, two 4-regular graphs are presented that cannot be distinguished by 1-WL but can be distinguished by Subgraph-1-WL ∗ .

Original: We visualize the 1-hop egonets that are structurally different among graphs A and B .
Revised: We visualize the 1-hop egonets that are structurally different among graphs A and B .

Original: It’s easy to see that
Revised: It’s easy to see that

Original: A ’s egonet
Revised: A ’s egonet

Original: A [ N 1 (1)] and B ’s egonet B [ N 1 (1)] can be distinguished by 1-WL, as degree distribution is not the same.
Revised: A [ N 1 (1)] and B ’s egonet B [ N 1 (1)] can be distinguished by 1-WL, as degree distribution is not the same.

Original: Hence, A and B can be distinguished by Subgraph-1-WL ∗ .
Revised: Hence, A and B can be distinguished by Subgraph-1-WL ∗ .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We prove by showing a pair of 3-WL failed non-isomorphic graphs can be distinguished by
Revised: We prove by showing a pair of 3-WL failed non-isomorphic graphs can be distinguished by

Original: Subgraph-1-WL (see deﬁnition of “no less powerful” in Zhengdao et al.
Revised: Subgraph-1-WL (see deﬁnition of “no less powerful” in Zhengdao et al.

Original: (2020): we call A is no/not
Revised: (2020): we call A is no/not

Original: A 

Revised: A

Original: B A's Subgraph
Revised: B A's Subgraph

Original: A's Subgraph
Revised: A's Subgraph

Original: B's Subgraph 8 B's Subgraph
Revised: B's Subgraph 8 B's Subgraph

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: [Figure]
Revised: [Figure]

Original: B's subgraph less powerful than B if there exists a pair of non-isomorphic graphs that cannot be distinguished by B but can be distinguished by A.), assuming HASH ( · ) is 3-WL discriminative.
Revised: B's subgraph less powerful than B if there exists a pair of non-isomorphic graphs that cannot be distinguished by B but can be distinguished by A.), assuming HASH ( · ) is 3-WL discriminative.

Original: Figure 3 shows two strongly regular graphs that can not be distinguished by 3-WL (any strongly regular graphs are not distinguishable by 3-WL (Arvind et al., 2020)), along with their 1-hop egonet rooted subgraphs.
Revised: Figure 3 shows two strongly regular graphs that can not be distinguished by 3-WL (any strongly regular graphs are not distinguishable by 3-WL (Arvind et al., 2020)), along with their 1-hop egonet rooted subgraphs.

Original: Notice that all 1-hop egonet rooted subgraphs in A (also in B ) are the same, resulting that Subgraph1-WL can successfully distinguish A and B if HASH can distinguish the showed two 1-hop egonets.
Revised: Notice that all 1-hop egonet rooted subgraphs in A (also in B ) are the same, resulting that Subgraph1-WL can successfully distinguish A and B if HASH can distinguish the showed two 1-hop egonets.

Original: Now we prove that 3-WL can distinguish these two subgraphs .
Revised: Now we prove that 3-WL can distinguish these two subgraphs .

Original: 3-WL constructs a coloring of 3tuples of all vertices in a graph, and uses the histogram of colors of all k -tuples as ﬁngerprint of the graph.
Revised: 3-WL constructs a coloring of 3tuples of all vertices in a graph, and uses the histogram of colors of all k -tuples as ﬁngerprint of the graph.

Original: Then, different 3-tuples correspond to different colors or bins in the histogram.
Revised: Then, different 3-tuples correspond to different colors or bins in the histogram.

Original: As a triangle is a unique type of 3-tuple, at iteration 0 of 3-WL, the histogram of all 3-tuples counts the number of triangles in the graph.
Revised: As a triangle is a unique type of 3-tuple, at iteration 0 of 3-WL, the histogram of all 3-tuples counts the number of triangles in the graph.

Original: Notice that A ’s subgraph has 2 × (cid:0) 43 (cid:1) = 8 triangles, and B ’s subgraph contains 6 triangles.
Revised: Notice that A ’s subgraph has 2 × (cid:0) 43 (cid:1) = 8 triangles, and B ’s subgraph contains 6 triangles.

Original: This implies that even at iteration 0, 3-WL can distinguish these two subgraphs.
Revised: This implies that even at iteration 0, 3-WL can distinguish these two subgraphs.

Original: Hence, when HASH ( · ) is 3-WL expressive, Subgraph-1-WL can distinguish A and B .
Revised: Hence, when HASH ( · ) is 3-WL expressive, Subgraph-1-WL can distinguish A and B .

Original: Therefore there exists a pair of 3-WL failed non-isomorphic graphs that can be distinguished by
Revised: Therefore there exists a pair of 3-WL failed non-isomorphic graphs that can be distinguished by

Original: Subgraph-1-WL.
Revised: Subgraph-1-WL.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: For any k ≥ 3 , Cai et al.
Revised: For any k ≥ 3 , Cai et al.

Original: (1992) provided a scheme to construct a pair of k -WL-failed nonisomorphic graphs based on a base graph G with separator size k + 1 (more speciﬁcally, G can be chosen as a degree-3 regular graph with separator size k + 1 , see Corollary 6.5 in Cai et al.
Revised: (1992) provided a scheme to construct a pair of k -WL-failed nonisomorphic graphs based on a base graph G with separator size k + 1 (more speciﬁcally, G can be chosen as a degree-3 regular graph with separator size k + 1 , see Corollary 6.5 in Cai et al.

Original: We describe the proposed scheme of constructing the two graphs A and B brieﬂy, for details see
Revised: We describe the proposed scheme of constructing the two graphs A and B brieﬂy, for details see

Original: Section 6 in Cai et al.
Revised: Section 6 in Cai et al.

Original: At high level, this scheme ﬁrst constructs A by replacing every node (with degree d ) in G by a carefully designed graph X d , and then rewires two edges in A to its non-isomorphic counter graph B .
Revised: At high level, this scheme ﬁrst constructs A by replacing every node (with degree d ) in G by a carefully designed graph X d , and then rewires two edges in A to its non-isomorphic counter graph B .

Original: Speciﬁcally, the small graph X d =
Revised: Speciﬁcally, the small graph X d =

Original: ( V d , E d ) is deﬁned as follows.
Revised: ( V d , E d ) is deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: B d [Equation] M d , where A d = { a i | 1 ≤ i ≤ d } , B d = { b i | 1 ≤ i ≤ d } , and M d = { m S | S ⊆ { 1 , ..., d } , | S | is even }
Revised: M d , where A d = { a i | 1 ≤ i ≤ d } , B d = { b i | 1 ≤ i ≤ d } , and M d = { m S | S ⊆ { 1 , ..., d } , | S | is even }

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence each vertex v with degree d in G is replaced by a graph
Revised: Hence each vertex v with degree d in G is replaced by a graph

Original: X
Revised: X

Original: ( v ) =
Revised: ( v ) =

Original: X d of size 2 d − 1 + 2 d , including a “middle” section M d of size 2 d − 1 and d pairs of vertices ( a i , b i ) representing “endpoints” of each edge incident with v in the original graph G .
Revised: X d of size 2 d − 1 + 2 d , including a “middle” section M d of size 2 d − 1 and d pairs of vertices ( a i , b i ) representing “endpoints” of each edge incident with v in the original graph G .

Original: Let ( u, v ) ∈
Revised: Let ( u, v ) ∈

Original: E ( G ) , then two small graphs X ( v ) and X ( u ) in A are connected by the following.
Revised: E ( G ) , then two small graphs X ( v ) and X ( u ) in A are connected by the following.

Original: Let ( a u,v , b u,v ) , ( a v,u , b v,u ) be the pair of vertices associated with ( u, v ) in X ( u ) and X ( v ) respectively, then a u,v is connected to a v,u , and b u,v is connected to b v,u .
Revised: Let ( a u,v , b u,v ) , ( a v,u , b v,u ) be the pair of vertices associated with ( u, v ) in X ( u ) and X ( v ) respectively, then a u,v is connected to a v,u , and b u,v is connected to b v,u .

Original: After constructing A , B is modiﬁed from A by rewiring a single edge ( u, v ) in the original graph G .
Revised: After constructing A , B is modiﬁed from A by rewiring a single edge ( u, v ) in the original graph G .

Original: Speciﬁcally, now in B , a u,v is connected to b v,u and b u,v is connected to a v,u .
Revised: Speciﬁcally, now in B , a u,v is connected to b v,u and b u,v is connected to a v,u .

Original: Cai et al.
Revised: Cai et al.

Original: (1992) proved that A and B are non-isomorphic, and when base graph
Revised: (1992) proved that A and B are non-isomorphic, and when base graph

Original: G is a degree-3 regular graph with separator size k + 1 , A and B are not distinguishable by k -WL.
Revised: G is a degree-3 regular graph with separator size k + 1 , A and B are not distinguishable by k -WL.

Original: See a visualization in Figure 4.
Revised: See a visualization in Figure 4.

Original: 3 regular graph with separator size k + 1 for k -WL-failed case.
Revised: 3 regular graph with separator size k + 1 for k -WL-failed case.

Original: We provide several useful lemmas related to graph X d in the following, which are then used in proving Theorem 4.
Revised: We provide several useful lemmas related to graph X d in the following, which are then used in proving Theorem 4.

Original: Lemma 5 (Cai et al.
Revised: Lemma 5 (Cai et al.

Original: X d has exactly 2 d − 1 automorphisms, and each automorphism is determined by interchanging a i and b i for each i in some subset S ⊆ { 1 , ..., d } of even cardinality.
Revised: X d has exactly 2 d − 1 automorphisms, and each automorphism is determined by interchanging a i and b i for each i in some subset S ⊆ { 1 , ..., d } of even cardinality.

Original: Lemma 5 is directly taken from Lemma 6.1 in Cai et al.
Revised: Lemma 5 is directly taken from Lemma 6.1 in Cai et al.

Original: Lemma 6.
Revised: Lemma 6.

Original: The shortest path distance between a i and b i in X d for any i is exactly 4, and diameter of X d is 4.
Revised: The shortest path distance between a i and b i in X d for any i is exactly 4, and diameter of X d is 4.

Original: Proof.
Revised: Proof.

Original: Based on the construction shown in Eq.11, for any i , a i and b i are not directly connected, and any middle nodes share no connection.
Revised: Based on the construction shown in Eq.11, for any i , a i and b i are not directly connected, and any middle nodes share no connection.

Original: This implies the shortest path between a i and b i is larger than 3.
Revised: This implies the shortest path between a i and b i is larger than 3.

Original: Let m S be one middle node connected with a i .
Revised: Let m S be one middle node connected with a i .

Original: We construct a new set S (cid:48)
Revised: We construct a new set S (cid:48)

Original: = { p, q } , with p (cid:54) =
Revised: = { p, q } , with p (cid:54) =

Original: i, p ∈ S and q / ∈ S , then m S (cid:48) connects with b i based on Eq.11.
Revised: i, p ∈ S and q / ∈ S , then m S (cid:48) connects with b i based on Eq.11.

Original: Additionally, a p connects to both m S and m S (cid:48) , which implies the shortest path distance between a i and b i ≤ 4 .
Revised: Additionally, a p connects to both m S and m S (cid:48) , which implies the shortest path distance between a i and b i ≤ 4 .

Original: Hence, the shortest path distance between a i and b i is exactly 4.
Revised: Hence, the shortest path distance between a i and b i is exactly 4.

Original: To show the diameter of X d is 4, it’s easy to observe that distance( a i , b i ) only decreases when replacing a i with other nodes in X d (or reverse, replacing b i with other nodes in X d ).
Revised: To show the diameter of X d is 4, it’s easy to observe that distance( a i , b i ) only decreases when replacing a i with other nodes in X d (or reverse, replacing b i with other nodes in X d ).

Original: Lemma 7.
Revised: Lemma 7.

Original: For any k ≥ 1 , k -egonets of a i and b i in X d are isomorphic.
Revised: For any k ≥ 1 , k -egonets of a i and b i in X d are isomorphic.

Original: Proof.
Revised: Proof.

Original: When k ≥ 4 , X d [ N k ( a i )]
Revised: When k ≥ 4 , X d [ N k ( a i )]

Original: =
Revised: =

Original: X d [ N k ( b i )]
Revised: X d [ N k ( b i )]

Original: =
Revised: =

Original: X d based on Lemma 6.
Revised: X d based on Lemma 6.

Original: Now for 1 ≤ k ≤ 3 , we analyze the k -hop egonets of a i and b i sequentially.
Revised: Now for 1 ≤ k ≤ 3 , we analyze the k -hop egonets of a i and b i sequentially.

Original: First, X d [ N 1 ( a i )] and X d [ N 1 ( b i )] are two depth-1 trees with equal number of leaves (middle nodes in X d ), and of course are isomorphic.
Revised: First, X d [ N 1 ( a i )] and X d [ N 1 ( b i )] are two depth-1 trees with equal number of leaves (middle nodes in X d ), and of course are isomorphic.

Original: Similarly, X d [ N 2 ( a i )] and X d [ N 2 ( b i )] are two depth-2 trees with equal number of depth-1 nodes and equal number of leaves, which are also isomorphic.
Revised: Similarly, X d [ N 2 ( a i )] and X d [ N 2 ( b i )] are two depth-2 trees with equal number of depth-1 nodes and equal number of leaves, which are also isomorphic.

Original: Last, X d [ N 3 ( a i )] and X d [ N 3 ( b i )] are the graphs by removing b i from X d and a i from X d , respectively.
Revised: Last, X d [ N 3 ( a i )] and X d [ N 3 ( b i )] are the graphs by removing b i from X d and a i from X d , respectively.

Original: It’s trivial to observe that they are also isomorphic.
Revised: It’s trivial to observe that they are also isomorphic.

Original: Combining all cases proves the Lemma.
Revised: Combining all cases proves the Lemma.

Original: To prove the main theorem, we visualize the two graphs A and B zoomed at edge ( u, v ) of base graph in Figure 4 to help understanding.
Revised: To prove the main theorem, we visualize the two graphs A and B zoomed at edge ( u, v ) of base graph in Figure 4 to help understanding.

Original: To prove that Subgraph-1-WL with t ≤ 4 -egonet
Revised: To prove that Subgraph-1-WL with t ≤ 4 -egonet

Original: (we use t to prevent notation conﬂict) cannot distinguish A and B , we mainly analyze the subgraph around a u,v for both graph A and B .
Revised: (we use t to prevent notation conﬂict) cannot distinguish A and B , we mainly analyze the subgraph around a u,v for both graph A and B .

Original: For other subgraphs with different root nodes within t shortest-path-distance around nodes a u,v , a v,u , b u,v , b v,u , they follow exactly the same argument.
Revised: For other subgraphs with different root nodes within t shortest-path-distance around nodes a u,v , a v,u , b u,v , b v,u , they follow exactly the same argument.

Original: For all subgraphs with other root nodes, it’s trivial to observe that there is no difference among two subgraphs in A and B .
Revised: For all subgraphs with other root nodes, it’s trivial to observe that there is no difference among two subgraphs in A and B .

Original: We introduce some notation ﬁrst.
Revised: We introduce some notation ﬁrst.

Original: Let A [ N t ( a u,v )] and B [ N t ( a u,v )] be the two t -egonet subgraphs around a u,v in A and B respectively.
Revised: Let A [ N t ( a u,v )] and B [ N t ( a u,v )] be the two t -egonet subgraphs around a u,v in A and B respectively.

Original: Let A left [ N t ( a u,v )] be the “left” part graph (visualized in
Revised: Let A left [ N t ( a u,v )] be the “left” part graph (visualized in

Original: Figure 4, the left side of a u,v , include a u,v ) of A [ N t ( a u,v )] , and A right [ N t ( a u,v )] be the “right” part graph (visualized in Figure 4, the right side of a u,v , excluding a u,v ) of A [ N t ( a u,v )] .
Revised: Figure 4, the left side of a u,v , include a u,v ) of A [ N t ( a u,v )] , and A right [ N t ( a u,v )] be the “right” part graph (visualized in Figure 4, the right side of a u,v , excluding a u,v ) of A [ N t ( a u,v )] .

Original: Then
Revised: Then

Original: A [ N t ( a u,v )] is reconstructed by connecting A left [ N t ( a u,v )] and A right [ N t ( a u,v )] with a single edge ( a u,v , a v,u ) .
Revised: A [ N t ( a u,v )] is reconstructed by connecting A left [ N t ( a u,v )] and A right [ N t ( a u,v )] with a single edge ( a u,v , a v,u ) .

Original: B left [ N t ( a u,v )] and B right [ N t ( a u,v )] are deﬁned in the same way, such that
Revised: B left [ N t ( a u,v )] and B right [ N t ( a u,v )] are deﬁned in the same way, such that

Original: B [ N t ( a u,v )] is reconstructed by connecting ( a u,v , b v,u ) in B .
Revised: B [ N t ( a u,v )] is reconstructed by connecting ( a u,v , b v,u ) in B .

Original: When t ≤ 4 , we show that there exists an isomorphism that maps A [ N t ( a u,v )] to B [ N t ( a u,v )] .
Revised: When t ≤ 4 , we show that there exists an isomorphism that maps A [ N t ( a u,v )] to B [ N t ( a u,v )] .

Original: We construct this isomorphism by constructing an isomorphism between A left [ N t ( a u,v )] and
Revised: We construct this isomorphism by constructing an isomorphism between A left [ N t ( a u,v )] and

Original: B left [ N t ( a u,v )] and an isomorphism between A right [ N t ( a u,v )] and B right [ N t ( a u,v )]
Revised: B left [ N t ( a u,v )] and an isomorphism between A right [ N t ( a u,v )] and B right [ N t ( a u,v )]

Original: ﬁrst.
Revised: ﬁrst.

Original: Trivially, the current node ordering in A left [ N t ( a u,v )] and B left [ N t ( a u,v )] already deﬁnes an isomorphism among them, that is, mapping any a i ( b i , m S ) in A left [ N t ( a u,v )] to a i ( b i , m S ) in
Revised: Trivially, the current node ordering in A left [ N t ( a u,v )] and B left [ N t ( a u,v )] already deﬁnes an isomorphism among them, that is, mapping any a i ( b i , m S ) in A left [ N t ( a u,v )] to a i ( b i , m S ) in

Original: B left [ N t ( a u,v )] .
Revised: B left [ N t ( a u,v )] .

Original: To ﬁnd an isomorphism between A right [ N t ( a u,v )] and B right
Revised: To ﬁnd an isomorphism between A right [ N t ( a u,v )] and B right

Original: [ N t ( a u,v )] , one can easily observe that for t ≤ 3 , A right [ N t ( a u,v )] can be 1-to-1 mapped (with a v,u in A being mapped to a i ) to X 3 [ N t − 1 ( a i )] and
Revised: [ N t ( a u,v )] , one can easily observe that for t ≤ 3 , A right [ N t ( a u,v )] can be 1-to-1 mapped (with a v,u in A being mapped to a i ) to X 3 [ N t − 1 ( a i )] and

Original: B right [ N t ( a u,v )] can be 1-to-1 mapped (with b v,u in A being mapped to b i ) to X 3 [ N t − 1 ( b i )] for some i .
Revised: B right [ N t ( a u,v )] can be 1-to-1 mapped (with b v,u in A being mapped to b i ) to X 3 [ N t − 1 ( b i )] for some i .

Original: When t = 4 , A right [ N t ( a u,v )] is mapped to
Revised: When t = 4 , A right [ N t ( a u,v )] is mapped to

Original: X 3 [ N 3 ( a i )] with additional 2 nodes and B right
Revised: X 3 [ N 3 ( a i )] with additional 2 nodes and B right

Original: [ N t ( a u,v )] is mapped to X 3 [ N 3 ( b i )] with additional 2 nodes.
Revised: [ N t ( a u,v )] is mapped to X 3 [ N 3 ( b i )] with additional 2 nodes.

Original: Then applying Lemma 7, when t ≤ 4 we can construct an isomorphism between
Revised: Then applying Lemma 7, when t ≤ 4 we can construct an isomorphism between

Original: A right [ N t ( a u,v )] and N right [ N t ( a u,v )] with a v,u in A being mapped to b v,u in B .
Revised: A right [ N t ( a u,v )] and N right [ N t ( a u,v )] with a v,u in A being mapped to b v,u in B .

Original: We remark that when t = 4 , the two additional nodes outside X 3 [ N 3 ( a i )] and X 3 [ N 3 ( b i )] does not affecting applying Lemma 7.
Revised: We remark that when t = 4 , the two additional nodes outside X 3 [ N 3 ( a i )] and X 3 [ N 3 ( b i )] does not affecting applying Lemma 7.

Original: Last, the combination of the isomorphism between A left [ N t ( a u,v )] and
Revised: Last, the combination of the isomorphism between A left [ N t ( a u,v )] and

Original: B left [ N t ( a u,v )] and the isomorphism between A right [ N t ( a u,v )] and B right [ N t ( a u,v )] is actually a new isomorphism between A [ N t ( a u,v )] and B [ N t ( a u,v )] , as the edge ( a u,v , a v,u ) is mapped to edge ( a u,v , b v,u ) .
Revised: B left [ N t ( a u,v )] and the isomorphism between A right [ N t ( a u,v )] and B right [ N t ( a u,v )] is actually a new isomorphism between A [ N t ( a u,v )] and B [ N t ( a u,v )] , as the edge ( a u,v , a v,u ) is mapped to edge ( a u,v , b v,u ) .

Original: Thus, when t ≤ 4 , A [ N t ( a u,v )] and B [ N t ( a u,v )] are isomorphic and
Revised: Thus, when t ≤ 4 , A [ N t ( a u,v )] and B [ N t ( a u,v )] are isomorphic and

Original: HASH ( A [ N t ( a u,v )])
Revised: HASH ( A [ N t ( a u,v )])

Original: =
Revised: =

Original: HASH ( B [ N t ( a u,v )]) no matter what HASH ( · ) function is used.
Revised: HASH ( B [ N t ( a u,v )]) no matter what HASH ( · ) function is used.

Original: Applying the same argument to all other pairs of subgraphs in A and B implies that the histogram encodings of A and B are the same.
Revised: Applying the same argument to all other pairs of subgraphs in A and B implies that the histogram encodings of A and B are the same.

Original: Hence Subgraph-1-WL with t ≤ 4 cannot distinguish the two graphs A and B that are k -WL-failed for any k ≥ 3 .
Revised: Hence Subgraph-1-WL with t ≤ 4 cannot distinguish the two graphs A and B that are k -WL-failed for any k ≥ 3 .

Original: We remark that the theorem doesn’t imply that Subgraph-1-WL with t ≥ 5 -egonet can distinguish A and B .
Revised: We remark that the theorem doesn’t imply that Subgraph-1-WL with t ≥ 5 -egonet can distinguish A and B .

Original: This should depend on the base graph.
Revised: This should depend on the base graph.

Original: We give an informal sketch.
Revised: We give an informal sketch.

Original: Suppose that for t ≥ 5 , the left part A left [ N t ( a u,v )] and right part A right [ N t ( a u,v )] are only connected through ( a u,v , a v,u ) , that is, removing ( a u,v , a v,u ) from A [ N t ( a u,v )] resulting in A left [ N t ( a u,v )] and A right [ N t ( a u,v )] being disconnected.
Revised: Suppose that for t ≥ 5 , the left part A left [ N t ( a u,v )] and right part A right [ N t ( a u,v )] are only connected through ( a u,v , a v,u ) , that is, removing ( a u,v , a v,u ) from A [ N t ( a u,v )] resulting in A left [ N t ( a u,v )] and A right [ N t ( a u,v )] being disconnected.

Original: Then we can apply Lemma 5 multiple times with mapping some a v,u in A to b v,u in B and b v,u in A to a v,u in B, until this kind of switches reach the boundary of A [ N t ( a u,v )] , resulting in an isomorphism between A [ N t ( a u,v )] and B [ N t ( a u,v )] .
Revised: Then we can apply Lemma 5 multiple times with mapping some a v,u in A to b v,u in B and b v,u in A to a v,u in B, until this kind of switches reach the boundary of A [ N t ( a u,v )] , resulting in an isomorphism between A [ N t ( a u,v )] and B [ N t ( a u,v )] .

Original: We refer the reader to Lemma 6.2 in Cai et al.
Revised: We refer the reader to Lemma 6.2 in Cai et al.

Original: (1992), which also uses this line of argument.
Revised: (1992), which also uses this line of argument.

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: The proof is by contradiction.
Revised: The proof is by contradiction.

Original: Let G and H be two non-isomorphic graphs and | V G | = | V H | (if graph sizes are different then Subgraph-1-WL can trivially distinguish them).
Revised: Let G and H be two non-isomorphic graphs and | V G | = | V H | (if graph sizes are different then Subgraph-1-WL can trivially distinguish them).

Original: Now suppose that
Revised: Now suppose that

Original: Prop. 1 is incorrect, i.e. Subgraph-1-WL cannot distinguish G and H even provided that the two conditions (1) and (2) are satisﬁed.
Revised: Prop. 1 is incorrect, i.e. Subgraph-1-WL cannot distinguish G and H even provided that the two conditions (1) and (2) are satisﬁed.

Original: Then, for any iteration of Subgraph-1-WL, G and H would have the same histogram of subgraph colors.
Revised: Then, for any iteration of Subgraph-1-WL, G and H would have the same histogram of subgraph colors.

Original: Now we focus on iteration 0.
Revised: Now we focus on iteration 0.

Original: Formally, let color c
Revised: Formally, let color c

Original: Gv,i = HASH ( G [ N k
Revised: Gv,i = HASH ( G [ N k

Original: ( v Gi )]) and c Hv,i =
Revised: ( v Gi )]) and c Hv,i =

Original: HASH ( H [ N k ( v Hi )]) for a node order v ( v Gi maps index i to a node in G ).
Revised: HASH ( H [ N k ( v Hi )]) for a node order v ( v Gi maps index i to a node in G ).

Original: According to Condition (1): for any node reordering v G 1 , ..., v GN G and v H 1 , ..., v HN H , ∃ i ∈ [1 , max( N G , N H )] that G [ N k ( v Gi )] and H [ N k ( v Gi )] are non-isomorphic, then we know that ∃ i that G [ N k ( v Gi )] and H [ N k ( v Hi )] are non-isomorphic,
Revised: According to Condition (1): for any node reordering v G 1 , ..., v GN G and v H 1 , ..., v HN H , ∃ i ∈ [1 , max( N G , N H )] that G [ N k ( v Gi )] and H [ N k ( v Gi )] are non-isomorphic, then we know that ∃ i that G [ N k ( v Gi )] and H [ N k ( v Hi )] are non-isomorphic,

Original: hence c
Revised: hence c

Original: Gv,i (cid:54) =
Revised: Gv,i (cid:54) =

Original: c
Revised: c

Original: Hv,i since by Condition (2)
Revised: Hv,i since by Condition (2)

Original: HASH can distinguish these two subgraphs.
Revised: HASH can distinguish these two subgraphs.

Original: However as two graphs have the same histogram of subgraph colors, there must be a j (cid:54)
Revised: However as two graphs have the same histogram of subgraph colors, there must be a j (cid:54)

Original: =
Revised: =

Original: i such that c
Revised: i such that c

Original: Gv,i = c
Revised: Gv,i = c

Original: Hv,j and c Gv,j = c Hv,i .
Revised: Hv,j and c Gv,j = c Hv,i .

Original: Then we can create a new node order m by swapping v Gi and v Gj , resulting c Gm,i =
Revised: Then we can create a new node order m by swapping v Gi and v Gj , resulting c Gm,i =

Original: c
Revised: c

Original: Hm,i and c Gm,j = c
Revised: Hm,i and c Gm,j = c

Original: Hm,j .
Revised: Hm,j .

Original: This process can be repeated until having a new node order w such that ∀ i ∈ 1 , ..., | V G | , c Gw,i =
Revised: This process can be repeated until having a new node order w such that ∀ i ∈ 1 , ..., | V G | , c Gw,i =

Original: c
Revised: c

Original: Hw,i .
Revised: Hw,i .

Original: As HASH is discriminative enough according to Condition (2), this implies ∀ i ∈ 1 , ..., | V G | , G [ N k ( w Gi )] and
Revised: As HASH is discriminative enough according to Condition (2), this implies ∀ i ∈ 1 , ..., | V G | , G [ N k ( w Gi )] and

Original: H [ N k ( w Hi )] are isomorphic, which contradicts with Condition (1).
Revised: H [ N k ( w Hi )] are isomorphic, which contradicts with Condition (1).

Original: Thus, Prop. 1 must be true.
Revised: Thus, Prop. 1 must be true.

Original: A. S AMPLING BY S UBGRAPH D ROP
Revised: A. P ROOF OF P ROPOSITION

Original: [Table]
Revised: Proof. When a pair of non-isomorphic graphs G and H cannot be distinguished by GNN-AK + , for any layer l , the histogram of h ( l )) v in G and H in Eq. 7 should be the same. Which implies that for any layer l , the histogram of h ( l )) v in G and H in Eq.

Original: [Figure] [Table] [Figure] [Table]
Revised: 4 should be the same.

Original: [-3) SUM {Emb( i |Subgraph 3)} Emb(5|Subgraph 5) SUM {Emb( i |Subgraph 5)} Emb(6|Subgraph 6) SUM {Emb( i |Subgraph 6)} 
 [Figure] Propagate subgraph & centroid encodings-]
Revised: [-3) SUM {Emb( i |Subgraph 3)} Emb(5|Subgraph 5) SUM {Emb( i |Subgraph 5)} Emb(6|Subgraph 6) SUM {Emb( i |Subgraph 6)} 
 [Figure] Propagate subgraph & centroid encodings-]Then GNN-AK cannot distinguish G and H .

Original: [-Scalecontextencodings 
 SUM {Emb(8|Subgraph i )} x scale6 Emb(6|Subgraph 8) SUM {Emb( i |Subgraph 8)} [Figure]-]
Revised: [-Scalecontextencodings 
 SUM {Emb(8|Subgraph i )} x scale6 Emb(6|Subgraph 8) SUM {Emb( i |Subgraph 8)} [Figure]-]So for any pair of non-isomorphic graphs, GNN-AK cannot distinguish them if GNN-AK + cannot distinguish them.

Original: i [Figure]
Revised: Thus GNN-AK + is more powerful than GNN-AK . A. S AMPLING BY S UBGRAPH D ROP

Original: The descriptions of subgraph sampling strategies are as follows.
Revised: The descriptions of subgraph sampling strategies are as follows.

Original: Fig.
Revised: Fig.

Original: 5 shows an overview of
Revised: 5 shows an overview of

Original: SubgraphDrop 
Revised: SubgraphDrop .

Original: • Random sampling selects subgraphs randomly until every node is covered ≥ R times.
Revised: • Random sampling selects subgraphs randomly until every node is covered ≥ R times.

Original: • Farthest sampling selects subgraphs iteratively, starting at a random one and greedily selecting each subsequent one whose root node is farthest w.r.t.
Revised: • Farthest sampling selects subgraphs iteratively, starting at a random one and greedily selecting each subsequent one whose root node is farthest w.r

Original:  shortest path distance from those of already selected subgraphs, until every node is covered ≥ R times.
Revised: t. shortest path distance from those of already selected subgraphs, until every node is covered ≥ R times.

Original: • Min-set-cover sampling initially selects a subgraph randomly, and follows the greedy minimum set cover algorithm to iteratively select the subgraph containing the maximum number of uncovered nodes, until every node is covered ≥ R times.
Revised: • Min-set-cover sampling initially selects a subgraph randomly, and follows the greedy minimum set cover algorithm to iteratively select the subgraph containing the maximum number of uncovered nodes, until every node is covered ≥ R times.

Original: A. D ATASET D ESCRIPTION & S TATISTICS 
Revised: A. D ATASET D ESCRIPTION & S TATISTICS A. E XPERIMENTAL S ETUP D ETAILS

Revised: To reduce the search space, we search hyperparameters in a two-phase approach: First, we search common ones (hidden size from [64, 128], number of layers L from [2,4,5,6], (sub)graph pooling

Original: [Table]
Revised: [Table]

Original: To reduce the search space, we search hyperparameters in a two-phase approach: First, we search common ones (hidden size from [64, 128], number of layers L from [2,4,5,6], (sub)graph pooling from [SUM, MEAN] for each dataset using GIN based on validation performance, and ﬁx it for any other GNN and GNN-AK .
Revised: [Figure] [Table] from [SUM, MEAN] for each dataset using GIN based on validation performance, and ﬁx it for any other GNN and GNN-AK .

Original: While hyperparameters may not be optimal for other GNN models, the evaluation is fair as there is no beneﬁt for GNN-AK .
Revised: While hyperparameters may not be optimal for other GNN models, the evaluation is fair as there is no beneﬁt for GNN-AK .

Original: Next, we search GNN-AK exclusive ones (encoding types) over validation set using GIN-AK and keep them ﬁxed for other GNN-AK .
Revised: Next, we search GNN-AK exclusive ones (encoding types) over validation set using GIN-AK and keep them ﬁxed for other GNN-AK .

Original: We use a 1 -layer base model, 3 -hop egonets for GNN-AK , with excpetions that CIFAR10 uses 2 -hop egonet due to memory constraint; PATTERN uses random walk based subgraph with walk length=10 and repeat times=5 as its graphs are dense.
Revised: We use a 1 -layer base model, 3 -hop egonets for GNN-AK , with excpetions that CIFAR10 uses 2 -hop egonet due to memory constraint; PATTERN uses random walk based subgraph with walk length=10 and repeat times=5 as its graphs are dense.

Original: For GNN-AK -S, R = 3 is set as default.
Revised: For GNN-AK -S, R = 3 is set as default.

Original: We use farthest sampling for molecular datasets ZINC, MolHIV, and MolPCBA; to speed up further, random sampling is used for CIFAR10 whose graphs are k -NN graphs; min-set-cover sampling is used for PATTERN to adapt random walk based subgraph.
Revised: We use farthest sampling for molecular datasets ZINC, MolHIV, and MolPCBA; to speed up further, random sampling is used for CIFAR10 whose graphs are k -NN graphs; min-set-cover sampling is used for PATTERN to adapt random walk based subgraph.

Original: We use Batch Normalization and ReLU activation in all models.
Revised: We use Batch Normalization and ReLU activation in all models.

Original: For optimization we use Adam with learning rate 0.001 and weight decay 0.
Revised: For optimization we use Adam with learning rate 0.001 and weight decay 0.

Original: All experiments are repeated 3 times to calculate mean and standard derivation.
Revised: All experiments are repeated 3 times to calculate mean and standard derivation.

Original: All experiments are conducted on
Revised: All experiments are conducted on

Original: RTX-A6000 GPUs.
Revised: RTX-A6000 GPUs.

Original: A. A BLATION S TUDY
Revised: A. A BLATION S TUDY

Original: We present ablation results on various structural components of GNN-AK .
Revised: We present ablation results on various structural components of GNN-AK .

Original: Table 6 shows the performance of GIN-AK for varying size egonets with k .
Revised: Table 6 shows the performance of GIN-AK for varying size egonets with k .

Original: We ﬁnd that larger subgraphs tend to yield improvement, although runtime-performance trade-off may vary by dataset.
Revised: We ﬁnd that larger subgraphs tend to yield improvement, although runtime-performance trade-off may vary by dataset.

Original: Notably, simply 1-egonets are enough for CIFAR10 to uplift performance of the base GIN considerably.
Revised: Notably, simply 1-egonets are enough for CIFAR10 to uplift performance of the base GIN considerably.

Original: Next Table 7 illustrates the added beneﬁt of various node encodings.
Revised: Next Table 7 illustrates the added beneﬁt of various node encodings.

Original: Compared to the full design, eliminating any of the subgraph, centroid, or context encodings (Eq.s (3)–(5)) yields notably inferior  results.
Revised: Compared to the full design, eliminating any of the subgraph, centroid, or context encodings (Eq.s (3)–(5)) yields notably inferior [Table] [Table] results.

Original: Encoding without distance awareness is also subpar.
Revised: Encoding without distance awareness is also subpar.

Original: These justify the design choices in our framework and verify the practical beneﬁts of our design.
Revised: These justify the design choices in our framework and verify the practical beneﬁts of our design.

Original: As context encoding and distance-to-centroid (D2C) encoding are practical designs, which are not directly explained by Subgraph-1-WL ∗ (though D2C is supported by Subgraph-1-WL, by strengthening HASH), we conduct additional ablation studies over GNN-AK with different combinations of
Revised: As GNN-AK + is not directly explained by Subgraph-1-WL ∗ (though D2C is supported by Subgraph1-WL

Original: [Table]
Revised: [Table]

Revised: by strengthening HASH)

Original: [Table]
Revised: [Table]

Original:  removing context encoding and D2C, shown in Table 8.
Revised: we conduct additional ablation studies over GNN-AK + with different combinations of removing context encoding and D2C [Table] shown in Table 8.

Original: Note that all experiments in Table 8 are using a 1-layer base GIN model.
Revised: Note that all experiments in Table 8 are using a 1-layer base GIN model.

Original: We summarize the observations as follows
Revised: We summarize the observations as follows.

Original: • For real-world datasets (ZINC-12K, CIFAR10), We observe that the largest performance improvement over base model comes from wrapping base GNN with Eq.2 (the performance of GIN-AKw/o Ctx&D2C ), while adding context encoding and D2C monotonically improves performance of GNN-AK  .
Revised: • For real-world datasets (ZINC-12K, CIFAR10), We observe that the largest performance improvement over base model comes from wrapping base GNN with Eq.2 (the performance of GIN-AK ), while adding context encoding and D2C monotonically improves performance of GNN-AK + .

Original: • For substructure counting and graph property regression tasks, we observe D2C signiﬁcantly increases the performance of GNN-AK (supported by the theory of Subgraph-1-WL where the HASH is required to be injective).
Revised: • For substructure counting and graph property regression tasks, we observe D2C signiﬁcantly increases the performance of GIN-AK + w /o Ctx (supported by the theory of Subgraph-1-WL where the HASH is required to be injective).

Original: Speciﬁcally, the cost-free D2C feature enhances the expressiveness of the base 1-layer GIN model (similar to the beneﬁt of distance encoding shown in Li et al.
Revised: Speciﬁcally, the cost-free D2C feature enhances the expressiveness of the base 1-layer GIN model (similar to the beneﬁt of distance encoding shown in Li et al.

Original: (2020)), resulting in a more expressive GNN-AK  , which lies between Subgraph-
Revised: (2020)), resulting in a more expressive GNN-AK + , which lies between Subgraph-1-WL and

Original: 1-WL and Subgraph-1WL ∗ .
Revised: Subgraph-1-WL ∗ .

Original: We leave the exact theoretical analysis of D2C’s expressiveness beneﬁt in future work.
Revised: We leave the exact theoretical analysis of D2C’s expressiveness beneﬁt in future work.

Original: Notice that GIN-AKw/o Ctx&D2C improves the performance over the base model but not in a large margin, we next show this is due to the insufﬁciency of the number of layers of base GIN.
Revised: Notice that GIN-AK improves the performance over the base model but not in a large margin, we next show this is due to the insufﬁciency of the number of layers of base GIN

Original: [Table]
Revised: [Table]

Original: -AK without D2C.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: According to Prop.
Revised: According to Prop.

Original: 1, the base model must be discriminative enough such that
Revised: 1, the base model must be discriminative enough such that

Original: HASH ( G [ N k ( v Gi )]) (cid:54) =
Revised: HASH ( G [ N k ( v Gi )]) (cid:54) =

Original: HASH ( H [ N k
Revised: HASH ( H [ N k

Original: v Hi )]) , for Subgraph-1-WL with k -egonet to enjoy expressiveness beneﬁts
Revised: v Hi )]) , for Subgraph-1-WL with k -egonet to enjoy expressiveness beneﬁts.

Original: In addition to using D2C to increase the expressiveness of base model, another way is to practically increase the number of layers of the base model (akin to increasing the number of iterations of 1-WL, as in the deﬁnition of Subgraph-1-WL ∗ ).
Revised: In addition to using D2C to increase the expressiveness of base model, another way is to practically increase the number of layers of the base model (akin to increasing the number of iterations of 1-WL, as in the deﬁnition of Subgraph-1-WL ∗ ).

Original: We study the effect of base model’s number of layers in GNN-AK  , with and without D2C in Table 9.
Revised: We study the effect of base model’s number of layers in GNN-AK + , with and without D2C in Table 9.

Original: • Firstly, GNN-AK  with D2C is insensitive to the depth of base model, with 1-layer base model being enough to achieve great performance in counting substructures and the best performance in regressing graph properties.
Revised: • Firstly, GNN-AK + with D2C is insensitive to the depth of base model, with 1-layer base model being enough to achieve great performance in counting substructures and the best performance in regressing graph properties.

Original: We hypothesize that D2C-enhanced 1-layer GIN base model is discriminative enough for subgraphs in the dataset, and without expressiveness bottleneck of base model, increasing GNN-AK  ’s depth beneﬁts expressiveness, akin to increasing iterations of Subgraph-1-WL.
Revised: We hypothesize that D2C-enhanced 1-layer GIN base model is discriminative enough for subgraphs in the dataset, and without expressiveness bottleneck of base model, increasing GNN-AK + ’s depth beneﬁts expressiveness, akin to increasing iterations of Subgraph-1-WL.

Original: Besides, unlike counting substructure that needs local information within subgraphs, regressing graph properties needs the graph’s global information which can only be accessed with increasing GNN-AK  ’s (outer) depth.
Revised: Besides, unlike counting substructure that needs local information within subgraphs, regressing graph properties needs the graph’s global information which can only be accessed with increasing GNN-AK + ’s (outer) depth.

Original: • Secondly, GNN-AK  without D2C suffers from a trade-off between the base model’s expressiveness (depth of base model) and the number of GNN-AK  layers (outer depth), which is clearly observed in regressing graph properties.
Revised: • Secondly, GNN-AK ( + ) without D2C suffers from a trade-off between the base model’s expressiveness (depth of base model) and the number of GNN-AK ( + ) layers (outer depth), which is clearly observed in regressing graph properties.

Original: We hypothesize that without D2C the 1-layer GIN base model is not discriminative enough for subgraphs in the dataset, and with this bottleneck of base model, GNN-AK  cannot beneﬁt from increasing the outer depth.
Revised: We hypothesize that without D2C the 1-layer GIN base model is not discriminative enough for subgraphs in the dataset, and with this bottleneck of base model, GNN-AK ( + ) cannot beneﬁt from increasing the outer depth.

Original: Hence the number of layers of the base model are important for the expressiveness of GNN-AK  when D2C is not used.
Revised: Hence the number of layers of the base model are important for the expressiveness of GNN-AK ( + ) when D2C is not used.

Original: A. A DDITIONAL R ESULTS ON TU D ATASETS
Revised: A. A DDITIONAL R ESULTS ON TU D ATASETS

Original: We also report additional results on several smaller datasets from TUDataset (Morris et al., 2020a), with their statistics reported in last block of Table 5.
Revised: We also report additional results on several smaller datasets from TUDataset (Morris et al., 2020a), with their statistics reported in last block of Table 5.

Original: The training setting and evaluation procedure follows Xu et al.
Revised: The training setting and evaluation procedure follows Xu et al.

Original: exactly, where we perform 10-fold cross-validation and report the average and standard deviation of validation accuracy across the 10 folds within the cross-validation.
Revised: exactly, where we perform 10-fold cross-validation and report the average and standard deviation of validation accuracy across the 10 folds within the cross-validation.

Original: We take results of existing baselines directly from Bodnar et al.
Revised: We take results of existing baselines directly from Bodnar et al.

Original: (2021a)
Revised: (2021a)

Original: with their method labeled as CIN, for references to all baselines see Bodnar et al.
Revised: with their method labeled as CIN, for references to all baselines see Bodnar et al.

Original: (2021a).
Revised: (2021a).

Original: Due to time limitation (during rebuttal period), only GIN-AK istested with 4 GNN-AK layers, 1-layer GIN base model, hidden size 128, dropout 0, and 3 -egonets (with an exception on RDT-B in which random-walk subgraph extractor is used as 3 -egonet is too large to ﬁt in memory). Notice that this gives disadvantage to GIN-AK without hyperparameter search, resulting suboptimal performance (while still outperforms all baselines) shown in Table 10.
Revised: The result is shown in Table 10.

Original: [Table]
Revised: [Table]