Original: Pure Transformers are Powerful Graph Learners
Revised: Pure Transformers are Powerful Graph Learners

Original: Abstract
Revised: Abstract

Original: We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice.
Revised: We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice.

Original: Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer.
Revised: Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer.

Original: With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN).
Revised: With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN).

Original: When trained on a large-scale graph dataset (PCQM4Mv2), our method coined
Revised: When trained on a large-scale graph dataset (PCQM4Mv2), our method coined

Original: Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias.
Revised: Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias.

Original: Our implementation is available at https://github.com/jw9730/tokengt .
Revised: Our implementation is available at https://github.com/jw9730/tokengt .

Original:  Introduction
Revised:  Introduction

Original: In recent years, Transformer [67] has served as a versatile architecture in a broad class of machine learning problems, such as natural language processing [17, 7], computer vision [18] and reinforcement learning [9], to name a few.
Revised: In recent years, Transformer [68] has served as a versatile architecture in a broad class of machine learning problems, such as natural language processing [17, 7], computer vision [18] and reinforcement learning [9], to name a few.

Original: It is because the fully-attentional structure of Transformer is general and powerful enough to take, process and relate inputs and outputs of arbitrary structures, eliminating a need for data- and task-specific inductive bias to be baked into the network architecture.
Revised: It is because the fully-attentional structure of Transformer is general and powerful enough to take, process and relate inputs and outputs of arbitrary structures, eliminating a need for data- and task-specific inductive bias to be baked into the network architecture.

Original: Combined with large-scale training, it opens up a new chapter for building a versatile model that can solve a wide range of problems involving diverse data modalities and even a mixture of modalities [31, 30, 56].
Revised: Combined with large-scale training, it opens up a new chapter for building a versatile model that can solve a wide range of problems involving diverse data modalities and even a mixture of modalities [31, 30, 57].

Original: In graph learning domain, inspired by the breakthroughs multiple works tried combining selfattention into graph neural network (GNN) architecture where message passing was previously dominant [49].
Revised: In graph learning domain, inspired by the breakthroughs multiple works tried combining selfattention into graph neural network (GNN) architecture where message passing was previously dominant [50].

Original: As global self-attention across nodes cannot reflect the graph structure, however, these methods introduce graph-specific architectural modifications.
Revised: As global self-attention across nodes cannot reflect the graph structure, however, these methods introduce graph-specific architectural modifications.

Original: This includes restricting self-attention to local neighborhoods [68, 50, 19], using global self-attention in conjunction with message-passing
Revised: This includes restricting self-attention to local neighborhoods [69, 51, 19], using global self-attention in conjunction with message-passing

Original: GNN [57, 42, 34], and injecting edge information into global self-attention via attention bias [71, 77, 29, 53].
Revised: GNN [58, 43, 34], and injecting edge information into global self-attention via attention bias [72, 78, 29, 54].

Original: Despite decent performance, such modifications can be a limiting constraint in terms of versatility, especially considering future integration to multi-task and multi-modal general-purpose attentional architectures [31].
Revised: Despite decent performance, such modifications can be a limiting constraint in terms of versatility, especially considering future integration to multi-task and multi-modal general-purpose attentional architectures [31].

Original: In addition, deviating from pure self-attention, these methods may inherit the issues of message-passing such as oversmoothing [39, 8, 51], and become incompatible with useful engineering techniques e.g. , linear attention [64] developed for standard self-attention.
Revised: In addition, deviating from pure self-attention, these methods may inherit the issues of message-passing such as oversmoothing [40, 8, 52], and become incompatible with useful engineering techniques e.g. , linear attention [65] developed for standard self-attention.

Original: Instead, we explore the opposite direction of applying a standard Transformer directly for graphs .
Revised: Instead, we explore the opposite direction of applying a standard Transformer directly for graphs .

Original: For this, we treat all nodes and edges as independent tokens, augment them with appropriate token-wise embeddings, and feed the tokens as input to the standard Transformer.
Revised: For this, we treat all nodes and edges as independent tokens, augment them with appropriate token-wise embeddings, and feed the tokens as input to the standard Transformer.

Original: The model operates identically to Transformers used in language and vision; each node or edge is treated as a token, identical to the words in a sentence or patches of an image [67, 18].
Revised: The model operates identically to Transformers used in language and vision; each node or edge is treated as a token, identical to the words in a sentence or patches of an image [68, 18].

Original: Perhaps surprisingly, we show that this simple approach yields a powerful graph learner both in theory and practice.
Revised: Perhaps surprisingly, we show that this simple approach yields a powerful graph learner both in theory and practice.

Original: [Figure]
Revised: [Figure]

Original: Tokenized Graph Transformer (TokenGT) vvvv1 2 3        v e v e v e v e v e
Revised: Tokenized Graph Transformer (TokenGT) vvvv1 2 3        v e v e v e v e v e

Original: As a key theoretical result, we prove that with appropriate token-wise embeddings, self-attention over the node and edge tokens can approximate any permutation equivariant linear operator on a graph [46].
Revised: As a key theoretical result, we prove that with appropriate token-wise embeddings, self-attention over the node and edge tokens can approximate any permutation equivariant linear operator on a graph [47].

Original: Remarkably, we show that a very simple choice of embedding composed of node identifiers and type identifiers is sufficient for accurate approximation.
Revised: Remarkably, we show that a very simple choice of embedding composed of node identifiers and type identifiers is sufficient for accurate approximation.

Original: This provides a solid theoretical guarantee that, with the embeddings and enough attention heads, a Transformer is at least as expressive as a second-order invariant graph network (2-IGN) [46, 34], which is already more expressive than all message-passing GNNs [21].
Revised: This provides a solid theoretical guarantee that, with the embeddings and enough attention heads, a Transformer is at least as expressive as a second-order invariant graph network (2-IGN) [47, 34], which is already more expressive than all message-passing GNNs [21].

Original: This also immediately grants the model with the expressive power at least as good as the 2-dimensional Weisfeiler-Lehman (WL) graph isomorphism test [45], which is often sufficient for real-world graph data [82].
Revised: This also immediately grants the model with the expressive power at least as good as the 2-dimensional Weisfeiler-Lehman (WL) graph isomorphism test [46], which is often sufficient for real-world graph data [83].

Original: We further extend our theoretical result to hypergraphs with order- k hyperedges, showing that a Transformer with order- k generalized token embeddings is at least as expressive as k -IGN and, consequently k -WL test.
Revised: We further extend our theoretical result to hypergraphs with order- k hyperedges, showing that a Transformer with order- k generalized token embeddings is at least as expressive as k -IGN and, consequently k -WL test.

Original: We test our model, named Tokenized Graph Transformer (TokenGT), mainly on the PCQM4Mv large-scale quantum chemical property prediction dataset containing 3.7M molecular graphs [27].
Revised: We test our model, named Tokenized Graph Transformer (TokenGT), mainly on the PCQM4Mv large-scale quantum chemical property prediction dataset containing 3.7M molecular graphs [27].

Original: Even though TokenGT involves minimal graph-specific architectural modifications, it performs significantly better than all GNN baselines, showing that the advantages of Transformer architecture combined with large-scale training surpass the benefit of hard inductive bias of GNNs.
Revised: Even though TokenGT involves minimal graph-specific architectural modifications, it performs significantly better than all GNN baselines, showing that the advantages of Transformer architecture combined with large-scale training surpass the benefit of hard inductive bias of GNNs.

Original: Furthermore,
Revised: Furthermore,

Original: TokenGT achieves competitive performance compared to Transformer variants with strong graphspecific modifications [77, 29, 53].
Revised: TokenGT achieves competitive performance compared to Transformer variants with strong graphspecific modifications [78, 29, 54].

Original: Finally, we demonstrate that TokenGT can naturally utilize efficient approximations in Transformers in contrast to these variants, using kernel attention [11] that enables linear computation cost without much degradation in performance.
Revised: Finally, we demonstrate that TokenGT can naturally utilize efficient approximations in Transformers in contrast to these variants, using kernel attention [11] that enables linear computation cost without much degradation in performance.

Original:  Tokenized Graph Transformer (TokenGT)
Revised:  Tokenized Graph Transformer (TokenGT)

Original: In this section, we present the Tokenized Graph Transformer (TokenGT), a pure Transformer architecture for graphs with token-wise embeddings composed of node identifiers and type identifiers (Figure 1).
Revised: In this section, we present the Tokenized Graph Transformer (TokenGT), a pure Transformer architecture for graphs with token-wise embeddings composed of node identifiers and type identifiers (Figure 1).

Original: Our goal in this section is to provide a practical overview – for theoretical analysis of the architecture, we guide the readers to Section 3.
Revised: Our goal in this section is to provide a practical overview – for theoretical analysis of the architecture, we guide the readers to Section 3.

Original: Let G =
Revised: Let G =

Original: ( V , E ) an input graph with n nodes V = { v 1 , ..., v n } and m edges E = { e 1 , ..., e m } ⊆
Revised: ( V , E ) an input graph with n nodes V = { v 1 , ..., v n } and m edges E = { e 1 , ..., e m } ⊆

Original: V 2 , associated with features X V ∈ R n × C and X E ∈ R m × C , respectively.
Revised: V 2 , associated with features X V ∈ R n × C and X E ∈ R m × C , respectively.

Original: We treat each node and edge as an independent token (thus ( n + m ) tokens in total) and construct their features by
Revised: We treat each node and edge as an independent token (thus ( n + m ) tokens in total) and construct their features by

Original: X =
Revised: X =

Original: [ X V ; X E ] ∈ R ( n + m )
Revised: [ X V ; X E ] ∈ R ( n + m )

Original: × C .
Revised: × C .

Original: A naïve way to process a graph is to directly provide the tokens X as input to a Transformer, but it is inappropriate as graph connectivity is discarded.
Revised: A naïve way to process a graph is to directly provide the tokens X as input to a Transformer, but it is inappropriate as graph connectivity is discarded.

Original: To thoroughly represent graph structure, we augment the tokens X with token-wise embeddings, more specifically orthonormal node identifiers used for representing the connectivity of the tokens and trainable type identifiers that encode whether a token is a node or an edge.
Revised: To thoroughly represent graph structure, we augment the tokens X with token-wise embeddings, more specifically orthonormal node identifiers used for representing the connectivity of the tokens and trainable type identifiers that encode whether a token is a node or an edge.

Original: Despite the simplicity, we show that a
Revised: Despite the simplicity, we show that a

Original: Transformer applied on these embeddings is a theoretically powerful graph learner.
Revised: Transformer applied on these embeddings is a theoretically powerful graph learner.

Original: Node Identifiers
Revised: Node Identifiers

Original: The first component of token-wise embedding is the orthonormal node identifier that we use to represent the connectivity structure given in the input graph.
Revised: The first component of token-wise embedding is the orthonormal node identifier that we use to represent the connectivity structure given in the input graph.

Original: For a given input graph G =
Revised: For a given input graph G =

Original: ( V , E ) , we first produce n node-wise orthonormal vectors P ∈ R n × d p that we refer to as node identifiers.
Revised: ( V , E ) , we first produce n node-wise orthonormal vectors P ∈ R n × d p that we refer to as node identifiers.

Original: Then, we augment the tokens X with node identifiers as follows.
Revised: Then, we augment the tokens X with node identifiers as follows.

Original: • For each node v ∈ V , we augment the token X v as [ X v , P v , P v ] .
Revised: • For each node v ∈ V , we augment the token X v as [ X v , P v , P v ] .

Original: • For each edge ( u, v ) ∈ E , we augment the token X ( u,v ) as [ X ( u,v ) , P u , P v ] .
Revised: • For each edge ( u, v ) ∈ E , we augment the token X ( u,v ) as [ X ( u,v ) , P u , P v ] .

Original: Intuitively, a Transformer operating on the augmented tokens can fully recognize the connectivity structure of the graph since comparing the node identifiers between a pair of tokens reveals their incidence information.
Revised: Intuitively, a Transformer operating on the augmented tokens can fully recognize the connectivity structure of the graph since comparing the node identifiers between a pair of tokens reveals their incidence information.

Original: For instance, we can tell if an edge e = ( u, v ) is connected with a node k through dot-product (attention) since [ P u , P v ][ P k , P k ] ⊤ = 1 if and only if k ∈ ( u, v ) andotherwise.
Revised: For instance, we can tell if an edge e = ( u, v ) is connected with a node k through dot-product (attention) since [ P u , P v ][ P k , P k ] ⊤ = 1 if and only if k ∈ ( u, v ) andotherwise.

Original: This allows the Transformer to identify and exploit the connectivity structure of a graph, for instance by putting more weights on incident pairs when the local operation is important.
Revised: This allows the Transformer to identify and exploit the connectivity structure of a graph, for instance by putting more weights on incident pairs when the local operation is important.

Original: Notably, as the node identifiers P are only required to be orthonormal, we have a large degree of freedom in implementation choices.
Revised: Notably, as the node identifiers P are only required to be orthonormal, we have a large degree of freedom in implementation choices.

Original: We outline two practical methods below as examples.
Revised: We outline two practical methods below as examples.

Original: Their implementation details can be found in Appendix A.3.1.
Revised: Their implementation details can be found in Appendix A.3.1.

Original: • Orthogonal random features (ORFs), e
Revised: • Orthogonal random features (ORFs), e

Original: g. , rows of random orthogonal matrix Q ∈ R n × n obtained with QR decomposition of random Gaussian matrix G ∈ R n × n [78, 12].
Revised: g. , rows of random orthogonal matrix Q ∈ R n × n obtained with QR decomposition of random Gaussian matrix G ∈ R n × n [79, 12].

Original: • Laplacian eigenvectors obtained from eigendecomposition of graph Laplacian matrix, i.e. , rows of U from ∆ =
Revised: • Laplacian eigenvectors obtained from eigendecomposition of graph Laplacian matrix, i.e. , rows of U from ∆ =

Original: I − D − 1 / 2 AD − 1 / 2 = U ⊤ ΛU , where A ∈ R n × n is adjacency matrix, D is degree matrix, and Λ , U correspond to eigenvalues and eigenvectors respectively [20].
Revised: I − D − 1 / 2 AD − 1 / 2 = U ⊤ ΛU , where A ∈ R n × n is adjacency matrix, D is degree matrix, and Λ , U correspond to eigenvalues and eigenvectors respectively [20].

Original: Among the two methods, node identifiers generated as ORFs do not encode any information about the graph structure as they are entirely random.
Revised: Among the two methods, node identifiers generated as ORFs do not encode any information about the graph structure as they are entirely random.

Original: This means the Transformer that operates on the
Revised: This means the Transformer that operates on the

Original: ORF-based node identifiers needs to compile and recognize graph structure only from the incidence information provided by the node identifiers.
Revised: ORF-based node identifiers needs to compile and recognize graph structure only from the incidence information provided by the node identifiers.

Original: Although this is challenging, perhaps surprisingly, we empirically show in Section 5 that Transformers are strong enough to learn meaningful structural representations out of ORF-based node identifiers and outperform GNNs on large-scale task.
Revised: Although this is challenging, perhaps surprisingly, we empirically show in Section 5 that Transformers are strong enough to learn meaningful structural representations out of ORF-based node identifiers and outperform GNNs on large-scale task.

Original: In contrast to ORFs, Laplacian eigenvectors provide a kind of graph positional embeddings (graph
Revised: In contrast to ORFs, Laplacian eigenvectors provide a kind of graph positional embeddings (graph

Original: PEs) that describes the distance between nodes on a graph.
Revised: PEs) that describes the distance between nodes on a graph.

Original: Due to the positional information, it yields better performance compared to ORFs in our experiments in Section 5.
Revised: Due to the positional information, it yields better performance compared to ORFs in our experiments in Section 5.

Original: One interesting aspect of Laplacian eigenvectors is that they can be viewed as a generalization of sinusoidal positional embeddings of NLP Transformers to graphs, as the eigenvectors of 1D chain graphs are sine and cosine functions [20].
Revised: One interesting aspect of Laplacian eigenvectors is that they can be viewed as a generalization of sinusoidal positional embeddings of NLP Transformers to graphs, as the eigenvectors of 1D chain graphs are sine and cosine functions [20].

Original: Thus, by choosing Laplacian eigenvectors as node identifiers, our approach can be interpreted as a direct extension of the NLP Transformer for inputs involving relational structures.
Revised: Thus, by choosing Laplacian eigenvectors as node identifiers, our approach can be interpreted as a direct extension of the NLP Transformer for inputs involving relational structures.

Original: Type Identifiers
Revised: Type Identifiers

Original: The second component of token-wise embedding is the trainable type identifier that encodes whether a token is node or edge.
Revised: The second component of token-wise embedding is the trainable type identifier that encodes whether a token is node or edge.

Original: For a given input graph G =
Revised: For a given input graph G =

Original: ( V , E ) , we first prepare a trainable parameter matrix E =
Revised: ( V , E ) , we first prepare a trainable parameter matrix E =

Original: [ E V ; E E ] ∈ R 2 ×
Revised: [ E V ; E E ] ∈ R 2 ×

Original: d e that contains two type identifiers E V and E E for nodes and edges respectively.
Revised: d e that contains two type identifiers E V and E E for nodes and edges respectively.

Original: Then, we further augment the tokens with type identifiers as follows
Revised: Then, we further augment the tokens with type identifiers as follows

Original: • For each node v ∈ V , we augment the token [ X v , P v , P v ] as [ X v , P v , P v , E V ] .
Revised: • For each node v ∈ V , we augment the token [ X v , P v , P v ] as [ X v , P v , P v , E V ] .

Original: • For each edge ( u, v ) ∈ E , we augment the token
Revised: • For each edge ( u, v ) ∈ E , we augment the token

Original: [ X ( u,v ) , P u , P v ] as [ X ( u,v ) , P u , P v , E E ] .
Revised: [ X ( u,v ) , P u , P v ] as [ X ( u,v ) , P u , P v , E E ] .

Original: These embeddings provide information on whether a given token is a node or an edge, which is critical, e.g. , when an attention head tries to attend specifically to node tokens and ignore edge tokens.
Revised: These embeddings provide information on whether a given token is a node or an edge, which is critical, e.g. , when an attention head tries to attend specifically to node tokens and ignore edge tokens.

Original: Main Transformer With node identifiers and type identifiers, we obtain augmented token features
Revised: Main Transformer With node identifiers and type identifiers, we obtain augmented token features

Original: X in ∈ R ( n + m ) × ( C +2 d p + d e ) , which is further projected by a trainable matrix w in ∈ R ( C +2 d p + d e ) × d to be an input to Transformer.
Revised: X in ∈ R ( n + m ) × ( C +2 d p + d e ) , which is further projected by a trainable matrix w in ∈ R ( C +2 d p + d e ) × d to be an input to Transformer.

Original: For graph-level prediction, we prepend a special token [graph] with trainable embedding X [graph] ∈ R d similar to BERT [17] and ViT [18].
Revised: For graph-level prediction, we prepend a special token [graph] with trainable embedding X [graph] ∈ R d similar to BERT [17] and ViT [18].

Original: We utilize the feature of [graph] token at the output of the encoder as the graph representation, on which a linear prediction head is applied to produce the final graph-level prediction.
Revised: We utilize the feature of [graph] token at the output of the encoder as the graph representation, on which a linear prediction head is applied to produce the final graph-level prediction.

Original: Overall, the tokens
Revised: Overall, the tokens

Original: Z (0) =
Revised: Z (0) =

Original: [ X [graph] ; X in w in ] ∈ R (1+ n + m ) × d are used as the input to the main encoder.
Revised: [ X [graph] ; X in w in ] ∈ R (1+ n + m ) × d are used as the input to the main encoder.

Original: As an encoder, we adopt the standard Transformer [67], which is an alternating stack of multihead self-attention layers (MSA) and feedforward MLP layers.
Revised: As an encoder, we adopt the standard Transformer [68], which is an alternating stack of multihead self-attention layers (MSA) and feedforward MLP layers.

Original: We provide further details in Appendix A.1.1.
Revised: We provide further details in Appendix A.1.1.

Original: Inductive Bias Similar to Transformers in language and vision [17, 18], Tokenized Graph Transformer treats input nodes and edges as independent tokens and applies self-attention to them.
Revised: Inductive Bias Similar to Transformers in language and vision [17, 18], Tokenized Graph Transformer treats input nodes and edges as independent tokens and applies self-attention to them.

Original: This approach leads to much less inductive bias than current GNNs, where the sparse graph structure, or more fundamentally, permutation symmetry of graphs is deliberately baked into each layer [21,  46,45, 34].
Revised: This approach leads to much less inductive bias than current GNNs, where the sparse graph structure, or more fundamentally, permutation symmetry of graphs is deliberately baked into each layer [21, 47, 46, 34].

Original: For TokenGT, such information is provided entirely as a part of input using token-wise embeddings, and the model has to learn how to interpret and utilize the information from data.
Revised: For TokenGT, such information is provided entirely as a part of input using token-wise embeddings, and the model has to learn how to interpret and utilize the information from data.

Original: Although such weak inductive bias might raise questions on the expressiveness of the model, our theoretical analysis in Section 3 shows that TokenGT is a powerful graph learner thanks to the token-wise embeddings and expressive power of self-attention.
Revised: Although such weak inductive bias might raise questions on the expressiveness of the model, our theoretical analysis in Section 3 shows that TokenGT is a powerful graph learner thanks to the token-wise embeddings and expressive power of self-attention.

Original: For example, we show that TokenGT is more expressive than all message-passing GNNs under the framework of Gilmer et al.
Revised: For example, we show that TokenGT is more expressive than all message-passing GNNs under the framework of Gilmer et al.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: We now present our theory.
Revised: We now present our theory.

Original: Our key result is that TokenGT, a standard Transformer with node and type identifiers presented in Section 2, is provably at least as expressive as the second-order Invariant
Revised: Our key result is that TokenGT, a standard Transformer with node and type identifiers presented in Section 2, is provably at least as expressive as the second-order Invariant

Original: Graph Network (2-IGN [46]), which is built upon all possible permutation equivariant linear layers on a graph.
Revised: Graph Network (2-IGN [47]), which is built upon all possible permutation equivariant linear layers on a graph.

Original: This provides solid theoretical guarantees for TokenGT, such as being at least as powerful as the 2-WL graph isomorphism test and more expressive than all message-passing GNNs.
Revised: This provides solid theoretical guarantees for TokenGT, such as being at least as powerful as the 2-WL graph isomorphism test and more expressive than all message-passing GNNs.

Original: Our theory is based on a general framework on hypergraphs represented as higher-order tensors, which leads to the formulation of order- k TokenGT that is at least as expressive as order- k IGN ( k -IGN [46]).
Revised: Our theory is based on a general framework on hypergraphs represented as higher-order tensors, which leads to the formulation of order- k TokenGT that is at least as expressive as order- k IGN ( k -IGN [47]).

Original: 3.1 Preliminary: Permutation Symmetry and Invariant Graph Networks
Revised: 3.1 Preliminary: Permutation Symmetry and Invariant Graph Networks

Original: Representing and Processing Sets and (Hyper)Graphs For a set of n nodes, we often represent their features as X ∈ R n × d
Revised: Representing and Processing Sets and (Hyper)Graphs For a set of n nodes, we often represent their features as X ∈ R n × d

Original: where X i ∈ R
Revised: where X i ∈ R

Original: d is the feature of the i -th node.
Revised: d is the feature of the i -th node.

Original: The set is unordered and, therefore, should be treated invariant to the renumbering of the nodes.
Revised: The set is unordered and, therefore, should be treated invariant to the renumbering of the nodes.

Original: Let S n the symmetric group or the group of permutations π on [ n ] = { 1 , ..., n } .
Revised: Let S n the symmetric group or the group of permutations π on [ n ] = { 1 , ..., n } .

Original: By π · X we denote permuting rows of X with π , i.e. , ( π · X ) i =
Revised: By π · X we denote permuting rows of X with π , i.e. , ( π · X ) i =

Original: X π − 1 ( i ) .
Revised: X π − 1 ( i ) .

Original: Here, X and π · X represent the identical set for all π ∈ S n .
Revised: Here, X and π · X represent the identical set for all π ∈ S n .

Original: Generally, we consider (hyper)graphs represented as order- k tensor X ∈ R n k × d with feature X i =
Revised: Generally, we consider (hyper)graphs represented as order- k tensor X ∈ R n k × d with feature X i =

Original: X
Revised: X

Original: i 1 ,...,i k ∈ R
Revised: i 1 ,...,i k ∈ R

Original: d attached to (hyper)edge represented as multi-index i
Revised: d attached to (hyper)edge represented as multi-index i

Original: = ( i 1 , ..., i k ) ∈ [ n ] k .
Revised: = ( i 1 , ..., i k ) ∈ [ n ] k .

Original: Similar to sets, the tensor should be treated invariant to node renumbering by any π ∈ S n that acts on X by ( π · X ) i =
Revised: Similar to sets, the tensor should be treated invariant to node renumbering by any π ∈ S n that acts on X by ( π · X ) i =

Original: X π − 1 ( i ) where π − 1 ( i ) =
Revised: X π − 1 ( i ) where π − 1 ( i ) =

Original: ( π − 1 ( i 1 ) , ..., π − 1 ( i k )) .
Revised: ( π − 1 ( i 1 ) , ..., π − 1 ( i k )) .

Original: That is, X and π · X represent the identical (hyper)graph for all π .
Revised: That is, X and π · X represent the identical (hyper)graph for all π .

Original: Due to such symmetry, to build a function F ( X ) ≈ T for tensor X and target T , a suitable way is to make them invariant F ( π · X )
Revised: Due to such symmetry, to build a function F ( X ) ≈ T for tensor X and target T , a suitable way is to make them invariant F ( π · X )

Original: =
Revised: =

Original: F ( X ) when the target is a vector or equivariant F ( π · X ) =
Revised: F ( X ) when the target is a vector or equivariant F ( π · X ) =

Original: π · F ( X ) when the target is also a tensor, for all X ∈ R n k × d and π ∈ S n .
Revised: π · F ( X ) when the target is also a tensor, for all X ∈ R n k × d and π ∈ S n .

Original: In our theoretical analysis, we work on order- k
Revised: In our theoretical analysis, we work on order- k

Original: dense tensor representation X ∈ R n k × d of a graph as they can represent node features ( k = 1 ), edge features ( k = 2 ), or hyperedge features ( k > 2 ) in a unified manner.
Revised: dense tensor representation X ∈ R n k × d of a graph as they can represent node features ( k = 1 ), edge features ( k = 2 ), or hyperedge features ( k > 2 ) in a unified manner.

Original: This is interchangeable but slightly different from the sparse representation of a graph with edge set E used in Section 2.
Revised: This is interchangeable but slightly different from the sparse representation of a graph with edge set E used in Section 2.

Original: Nevertheless, in Section 5 we empirically verify that our key theoretical findings work equally well for dense and sparse graphs.
Revised: Nevertheless, in Section 5 we empirically verify that our key theoretical findings work equally well for dense and sparse graphs.

Original: Invariant Graph Network We mainly develop our theoretical analysis upon Invariant Graph
Revised: Invariant Graph Network We mainly develop our theoretical analysis upon Invariant Graph

Original: Networks (IGNs) [46, 45], a family of expressive graph networks derived from the permutation symmetry of tensor representation of graphs.
Revised: Networks (IGNs) [47, 46], a family of expressive graph networks derived from the permutation symmetry of tensor representation of graphs.

Original: Here we provide a summary.
Revised: Here we provide a summary.

Original: In general, we define:
Revised: In general, we define:

Original: Definition 1.
Revised: Definition 1.

Original: An order- k Invariant Graph Network ( k -IGN) is a function
Revised: An order- k Invariant Graph Network ( k -IGN) is a function

Original: F k : R n k × d 0 → R written as the following:
Revised: F k : R n k × d 0 → R written as the following:

Original: [Equation]
Revised: [Equation]

Original: where each L ( t ) k
Revised: where each L ( t ) k

Original: → k is equivariant linear layer [46] from R n k × d t − 1 to R n k × d t , σ is activation function, and L k → 0 is a invariant linear layer from R n k × d T to R .
Revised: → k is equivariant linear layer [47] from R n k × d t − 1 to R n k × d t , σ is activation function, and L k → 0 is a invariant linear layer from R n k × d T to R .

Original: A body of previous work have shown appealing theoretical properties of k -IGN, including universal approximation [47] and alignment to k -Weisfeiler-Lehman ( k -WL) graph isomorphism test [45, 10].
Revised: A body of previous work have shown appealing theoretical properties of k -IGN, including universal approximation [48] and alignment to k -Weisfeiler-Lehman ( k -WL) graph isomorphism test [46, 10].

Original: In particular, it is known that k -IGNs are theoretically at least as powerful as the k -WL test [45].
Revised: In particular, it is known that k -IGNs are theoretically at least as powerful as the k -WL test [46].

Original: It is also known that 2-IGNs are already more expressive [46, 34] than all message-passing GNNs under the framework of Gilmer et al.
Revised: It is also known that 2-IGNs are already more expressive [47, 34] than all message-passing GNNs under the framework of Gilmer et al.

Original: The core building block of IGN is invariant and equivariant linear layers [46] with maximal expressiveness while respecting node permutation symmetry.
Revised: The core building block of IGN is invariant and equivariant linear layers [47] with maximal expressiveness while respecting node permutation symmetry.

Original: The layers are defined as follows:
Revised: The layers are defined as follows:

Original: Definition 2.
Revised: Definition 2.

Original: An equivariant linear layer is a function
Revised: An equivariant linear layer is a function

Original: L k
Revised: L k

Original: → l : R n k × d
Revised: → l : R n k × d

Original: → R n l × d ′ written as follows for order- k input X ∈ R n k × d :
Revised: → R n l × d ′ written as follows for order- k input X ∈ R n k × d :

Original: [Equation]
Revised: [Equation]

Original: where i ∈ [ n ] l , j ∈ [ n ] k are multi-indices, w µ ∈ R d × d ′ , b λ ∈ R d
Revised: where i ∈ [ n ] l , j ∈ [ n ] k are multi-indices, w µ ∈ R d × d ′ , b λ ∈ R d

Original: ′ are weight and bias parameters, and B µ ∈ R n l + k and C λ ∈ R n l are binary basis tensors corresponding to order-
Revised: ′ are weight and bias parameters, and B µ ∈ R n l + k and C λ ∈ R n l are binary basis tensors corresponding to order-

Original: ( l + k ) and order- l equivalence classes µ and λ , respectively.
Revised: ( l + k ) and order- l equivalence classes µ and λ , respectively.

Original: Invariant linear layer is a special case of L k → l with l = 0 .
Revised: Invariant linear layer is a special case of L k → l with l = 0 .

Original: We provide the definition of the equivalence classes and basis tensors in Appendix A.1.1.
Revised: We provide the definition of the equivalence classes and basis tensors in Appendix A.1.1.

Original: For now, it is sufficient to know that the basis tensors are binary tensors that form the orthogonal basis of the full space of linear equivariant layers.
Revised: For now, it is sufficient to know that the basis tensors are binary tensors that form the orthogonal basis of the full space of linear equivariant layers.

Original: In general, in Eq.
Revised: In general, in Eq.

Original: (2) it is known that there exists bell ( k + l ) number of basis tensors B µ for the weight and bell ( l ) number of basis tensors C λ for the bias.
Revised: (2) it is known that there exists bell ( k + l ) number of basis tensors B µ for the weight and bell ( l ) number of basis tensors C λ for the bias.

Original: 3.2 Can Self-Attention Approximate Equivariant Basis?
Revised: 3.2 Can Self-Attention Approximate Equivariant Basis?

Original: Now, we present an intuition that connects Transformer (Section 2) and equivariant linear layer (Definition 2).
Revised: Now, we present an intuition that connects Transformer (Section 2) and equivariant linear layer (Definition 2).

Original: For that, we write out the multihead self-attention layer as follows:
Revised: For that, we write out the multihead self-attention layer as follows:

Original: [Equation]
Revised: [Equation]

Original: where H is number of heads, d H is head size, and w Qh , w Kh ∈ R d × d H , w
Revised: where H is number of heads, d H is head size, and w Qh , w Kh ∈ R d × d H , w

Original: Vh ∈ R d × d v w Oh ∈ R d v × d .
Revised: Vh ∈ R d × d v w Oh ∈ R d v × d .

Original: Our intuition is that the weighted sum of values with self-attention matrix α h in Eq.
Revised: Our intuition is that the weighted sum of values with self-attention matrix α h in Eq.

Original: (3) is analogous to the masked sum with basis tensor B µ in Eq.
Revised: (3) is analogous to the masked sum with basis tensor B µ in Eq.

Original: (2) up to normalization.
Revised: (2) up to normalization.

Original: This naturally leads to the following question: for a given equivariant layer L k
Revised: This naturally leads to the following question: for a given equivariant layer L k

Original: → k : R n k × d
Revised: → k : R n k × d

Original: → R n k × d , can we use a Transformer layer with multihead self-attention MSA : R N × d ′ → R N × d ′ with N = n k to accurately approximate
Revised: → R n k × d , can we use a Transformer layer with multihead self-attention MSA : R N × d ′ → R N × d ′ with N = n k to accurately approximate

Original: L k → k by having H = bell (2 k ) attention heads approximate each equivariant basis B µ ?
Revised: L k → k by having H = bell (2 k ) attention heads approximate each equivariant basis B µ ?

Original: We show that this can be possible, but only if we provide appropriate auxiliary information to input.
Revised: We show that this can be possible, but only if we provide appropriate auxiliary information to input.

Original: For example, let us consider first-order layer L 1 → 1 .
Revised: For example, let us consider first-order layer L 1 → 1 .

Original: The layer has bell (2) = 2 basis tensors B µ 1 =
Revised: The layer has bell (2) = 2 basis tensors B µ 1 =

Original: I and B µ 2 = 11 ⊤ − I for the weight, and bell (1) = 1 basis tensor C λ 1 = 1 for the bias.
Revised: I and B µ 2 = 11 ⊤ − I for the weight, and bell (1) = 1 basis tensor C λ 1 = 1 for the bias.

Original: Given an input set X ∈ R n × d it computes the following with w 1 , w 2 ∈ R d × d , b ∈ R d :
Revised: Given an input set X ∈ R n × d it computes the following with w 1 , w 2 ∈ R d × d , b ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: Now consider approximating basis tensor B µ 

Revised: Now consider approximating basis tensor B µ

Original: = I with an attention matrix α 1 .
Revised: = I with an attention matrix α 1 .

Original: The approximation is accurate when i -th query always only attends to i -th key and ignores the rest.
Revised: The approximation is accurate when i -th query always only attends to i -th key and ignores the rest.

Original: To achieve the attention structure consistently, i.e. , agnostic to input X , we need to provide auxiliary input that self-attention can "latch onto" to faithfully approximate
Revised: To achieve the attention structure consistently, i.e. , agnostic to input X , we need to provide auxiliary input that self-attention can "latch onto" to faithfully approximate

Original: α 1 ≈ I .
Revised: α 1 ≈ I .

Original: Without this, attention must entirely rely on the inputs X , which is unreliable and can lead to approximation failure, e.g. , when X has repeated rows.
Revised: Without this, attention must entirely rely on the inputs X , which is unreliable and can lead to approximation failure, e.g. , when X has repeated rows.

Original: For the auxiliary information, we prepare n node-wise orthonormal vectors P ∈ R n × d p (note that this is identical to node identifiers in Section 2), and augment the input to X in =
Revised: For the auxiliary information, we prepare n node-wise orthonormal vectors P ∈ R n × d p (note that this is identical to node identifiers in Section 2), and augment the input to X in =

Original: [ X , P ] ∈ R n × ( d + d p ) .
Revised: [ X , P ] ∈ R n × ( d + d p ) .

Original: Let us assume that the query and key projections in Eq.
Revised: Let us assume that the query and key projections in Eq.

Original: (3) ignore X and only leave P scaled by √ a with a > 0 .
Revised: (3) ignore X and only leave P scaled by √ a with a > 0 .

Original: Then attention matrix is computed as α 1 = softmax ( S ) where S ij = a P ⊤ i P j .
Revised: Then attention matrix is computed as α 1 = softmax ( S ) where S ij = a P ⊤ i P j .

Original: Here, due to the orthonormality of P , we have P ⊤ i P j = 1 only if i = j and otherwise 0, which leads to
Revised: Here, due to the orthonormality of P , we have P ⊤ i P j = 1 only if i = j and otherwise 0, which leads to

Original: S = a I .
Revised: S = a I .

Original: With a → ∞ by scaling up the query and key projection weights, the softmax becomes arbitrarily close to the hardmax operator, and we obtain the following:
Revised: With a → ∞ by scaling up the query and key projection weights, the softmax becomes arbitrarily close to the hardmax operator, and we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Thus, self-attention can utilize the auxiliary information P to achieve an input-agnostic approximation of α 1 to I .
Revised: Thus, self-attention can utilize the auxiliary information P to achieve an input-agnostic approximation of α 1 to I .

Original: Notably, we can achieve a similar approximation for B µ 2 = 

Revised: Notably, we can achieve a similar approximation for B µ 2 =

Original: ⊤ − I using the same P by flipping the sign of keys, which gives α 2 = softmax ( − a I ) due to orthonormality.
Revised: ⊤ − I using the same P by flipping the sign of keys, which gives α 2 = softmax ( − a I ) due to orthonormality.

Original: By sending a → ∞ , now attention from the i -th query to the i -th key is suppressed, and we obtain the following:
Revised: By sending a → ∞ , now attention from the i -th query to the i -th key is suppressed, and we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Note that this approximation is accurate only up to row normalization as rows of α 2 always sum to one due to softmax, while B µ 2 = 11 ⊤ − I is binary.
Revised: Note that this approximation is accurate only up to row normalization as rows of α 2 always sum to one due to softmax, while B µ 2 = 11 ⊤ − I is binary.

Original: In our proofs of the theoretical results, we perform appropriate denormalization with MLP after MSA to achieve an accurate approximation.
Revised: In our proofs of the theoretical results, we perform appropriate denormalization with MLP after MSA to achieve an accurate approximation.

Original: Overall, we see that simple auxiliary input P suffices for two attention heads to approximate the equivariant basis of L 1 → 1 accurately.
Revised: Overall, we see that simple auxiliary input P suffices for two attention heads to approximate the equivariant basis of L 1 → 1 accurately.

Original: We now question the following.
Revised: We now question the following.

Original: Given appropriate auxiliary information as input, can a Transformer layer with bell (2 k ) attention heads accurately approximate
Revised: Given appropriate auxiliary information as input, can a Transformer layer with bell (2 k ) attention heads accurately approximate

Original: L k → k by having each head approximate each equivariant basis B µ ?
Revised: L k → k by having each head approximate each equivariant basis B µ ?

Original: What would be the sufficient auxiliary input?
Revised: What would be the sufficient auxiliary input?

Original: We answer the question by showing that, with (order- k generalized) node and type identifiers presented in Section 2, Transformer layers can accurately approximate equivariant layers
Revised: We answer the question by showing that, with (order- k generalized) node and type identifiers presented in Section 2, Transformer layers can accurately approximate equivariant layers

Original: L k → k via input-agnostic head-wise approximation of each equivariant basis.
Revised: L k → k via input-agnostic head-wise approximation of each equivariant basis.

Original: 3.3 Pure Transformers are Powerful Graph Learners
Revised: 3.3 Pure Transformers are Powerful Graph Learners

Original: We now present our main theoretical results that extend the discussions in Section 3.2 to any order k .
Revised: We now present our main theoretical results that extend the discussions in Section 3.2 to any order k .

Original: Note that k
Revised: Note that k

Original: = 2 corresponds to TokenGT for graphs presented in Section 2.
Revised: = 2 corresponds to TokenGT for graphs presented in Section 2.

Original: With k > 2 , we naturally extend TokenGT to hypergraphs.
Revised: With k > 2 , we naturally extend TokenGT to hypergraphs.

Original: All proofs can be found in Appendix A.1.
Revised: All proofs can be found in Appendix A.1.

Original: We first introduce generalized node and type identifiers (Section 2) for order- k tensors X ∈ R n k × d .
Revised: We first introduce generalized node and type identifiers (Section 2) for order- k tensors X ∈ R n k × d .

Original: We define the node identifier P ∈ R n × d p as an orthonormal matrix with n rows, and the type identifier as a trainable matrix E ∈ R bell ( k ) ×
Revised: We define the node identifier P ∈ R n × d p as an orthonormal matrix with n rows, and the type identifier as a trainable matrix E ∈ R bell ( k ) ×

Original: d e that contains bell ( k ) rows E γ 1 , ..., E γ bell ( k ) , each of which is designated for an order- k equivalence class γ .
Revised: d e that contains bell ( k ) rows E γ 1 , ..., E γ bell ( k ) , each of which is designated for an order- k equivalence class γ .

Original: Then, we augment each entry of input tensor as [ X i 1 ,...,i k , P i 1 , ..., P i k , E γ ] where ( i 1 , ..., i k ) ∈ γ .
Revised: Then, we augment each entry of input tensor as [ X i 1 ,...,i k , P i 1 , ..., P i k , E γ ] where ( i 1 , ..., i k ) ∈ γ .

Original: Let us exemplify.
Revised: Let us exemplify.

Original: For k = 1 (sets), each i -th entry is augmented as [ X i , P i , E γ 1 ] , consistent with our discussion in Section 3.2.
Revised: For k = 1 (sets), each i -th entry is augmented as [ X i , P i , E γ 1 ] , consistent with our discussion in Section 3.2.

Original: For k = 2 (graphs), each ( i, i ) -th entry is augmented as [ X ii , P i , P i , E γ 1 ] and each ( i, j ) -th entry ( i ̸ = j ) is augmented as [ X ij , P i , P j , E γ 2 ] .
Revised: For k = 2 (graphs), each ( i, i ) -th entry is augmented as [ X ii , P i , P i , E γ 1 ] and each ( i, j ) -th entry ( i ̸ = j ) is augmented as [ X ij , P i , P j , E γ 2 ] .

Original: This is consistent with TokenGT in Section 2, which augments nodes with E V =
Revised: This is consistent with TokenGT in Section 2, which augments nodes with E V =

Original: E γ 1 and edges with E E
Revised: E γ 1 and edges with E E

Original: = E γ 2 .
Revised: = E γ 2 .

Original: With node and type identifiers, we obtain augmented order-
Revised: With node and type identifiers, we obtain augmented order-

Original: k tensor X in ∈ R n k ×
Revised: k tensor X in ∈ R n k ×

Original: d + kd p + d e ) .
Revised: d + kd p + d e ) .

Original: We use a trainable projection w in ∈ R ( d + kd p + d e ) × d T to map them to hidden dimension d T of a Transformer.
Revised: We use a trainable projection w in ∈ R ( d + kd p + d e ) × d T to map them to hidden dimension d T of a Transformer.

Original: We now show that self-attention on X in w in can accurately approximate equivariant basis:
Revised: We now show that self-attention on X in w in can accurately approximate equivariant basis:

Original: Lemma 1.
Revised: Lemma 1.

Original: For all X ∈ R n k × d and their augmentation X in , self-attention coefficients α h (Eq.
Revised: For all X ∈ R n k × d and their augmentation X in , self-attention coefficients α h (Eq.

Original: (3) ) computed with X in w in can approximate any basis tensor B µ ∈ R n 2 k of order- k equivariant linear layer L k
Revised: (3) ) computed with X in w in can approximate any basis tensor B µ ∈ R n 2 k of order- k equivariant linear layer L k

Original: → k (Definition 2) to arbitrary precision up to normalization.
Revised: → k (Definition 2) to arbitrary precision up to normalization.

Original: Consequently, with the node and type identifiers, a collection of bell (2 k ) attention heads can approximate the collection of all basis tensors of order- k equivariant layer.
Revised: Consequently, with the node and type identifiers, a collection of bell (2 k ) attention heads can approximate the collection of all basis tensors of order- k equivariant layer.

Original: This leads to the following:
Revised: This leads to the following:

Original: Theorem 1.
Revised: Theorem 1.

Original: For all X ∈ R n k × d and their augmentation X in , a Transformer layer with bell (2 k ) self-attention heads that operates on X in w in can approximate an order- k equivariant linear layer
Revised: For all X ∈ R n k × d and their augmentation X in , a Transformer layer with bell (2 k ) self-attention heads that operates on X in w in can approximate an order- k equivariant linear layer

Original: L k → k ( X ) (Definition 2) to arbitrary precision.
Revised: L k → k ( X ) (Definition 2) to arbitrary precision.

Original: While the approximation in Lemma 1 is only accurate up to normalization over inputs (keys) due to softmax normalization, for the approximation in Theorem 1 we perform appropriate denormalization using MLP after multihead self-attention and can obtain an accurate approximation.
Revised: While the approximation in Lemma 1 is only accurate up to normalization over inputs (keys) due to softmax normalization, for the approximation in Theorem 1 we perform appropriate denormalization using MLP after multihead self-attention and can obtain an accurate approximation.

Original: By extending the result to multiple layers, we arrive at the following:
Revised: By extending the result to multiple layers, we arrive at the following:

Original: Theorem 2.
Revised: Theorem 2.

Original: For all X ∈ R n k × d and their augmentation X in , a Transformer composed of T layers that operates on X in w in followed by sum-pooling and MLP can approximate an k -IGN F k ( X ) (Definition 1) to arbitrary precision.
Revised: For all X ∈ R n k × d and their augmentation X in , a Transformer composed of T layers that operates on X in w in followed by sum-pooling and MLP can approximate an k -IGN F k ( X ) (Definition 1) to arbitrary precision.

Original: This directly leads to the following corollary:
Revised: This directly leads to the following corollary:

Original: Corollary 1.
Revised: Corollary 1.

Original: A Transformer on node and type identifiers in Theorem 2 is at least as expressive as k -IGN composed of order- k equivariant linear layers.
Revised: A Transformer on node and type identifiers in Theorem 2 is at least as expressive as k -IGN composed of order- k equivariant linear layers.

Original: Corollary 1 allows us to draw previous theoretical results on the expressiveness of k -IGN [45, 46, 34] and use them to lower-bound the provable expressiveness of a standard Transformer:
Revised: Corollary 1 allows us to draw previous theoretical results on the expressiveness of k -IGN [46, 47, 34] and use them to lower-bound the provable expressiveness of a standard Transformer:

Original: Corollary 2.
Revised: Corollary 2.

Original: A Transformer on node and type identifiers in Theorem 2 is at least as powerful as k -WL graph isomorphism test and is more expressive than all message-passing GNNs within the framework of Gilmer et al.
Revised: A Transformer on node and type identifiers in Theorem 2 is at least as powerful as k -WL graph isomorphism test and is more expressive than all message-passing GNNs within the framework of Gilmer et al.

Original:  Related Work
Revised:  Related Work

Original: We outline relevant work including equivariant neural networks, theory on expressive power of
Revised: We outline relevant work including equivariant neural networks, theory on expressive power of

Original: Transformers and their connection to modeling equivariance, and Transformers for graphs.
Revised: Transformers and their connection to modeling equivariance, and Transformers for graphs.

Original: Equivariant Neural Networks A machine learning task is often invariant or equivariant to specific symmetry of input data, e.g. , image classification is invariant to the translation of an input image.
Revised: Equivariant Neural Networks A machine learning task is often invariant or equivariant to specific symmetry of input data, e.g. , image classification is invariant to the translation of an input image.

Original: A large body of literature advocated baking the invariance or equivariance into a neural network as a type of inductive bias ( e.g. , translation equivariance of image convolution), showing that it reduces the number of parameters and improves generalization for a wide range of learning tasks involving various geometric structures [13, 14, 72, 65, 48, 52, 59, 6, 34, 38].
Revised: A large body of literature advocated baking the invariance or equivariance into a neural network as a type of inductive bias ( e.g. , translation equivariance of image convolution), showing that it reduces the number of parameters and improves generalization for a wide range of learning tasks involving various geometric structures [13, 14, 73, 66, 49, 53, 60, 6, 34, 39].

Original: Ravanbakhsh et al.
Revised: Ravanbakhsh et al.

Original: [55] showed that any equivariant layer for discrete group actions is equivalent to a specific parameter sharing structure.
Revised: [56] showed that any equivariant layer for discrete group actions is equivalent to a specific parameter sharing structure.

Original: Zaheer et al.
Revised: Zaheer et al.

Original: (2017) [81] and Maron et al.
Revised: (2017) [82] and Maron et al.

Original: [46] derived the parameter sharing for node permutation-symmetric data (sets and (hyper)graphs), which gives the maximally expressive equivariant linear layers and
Revised: [47] derived the parameter sharing for node permutation-symmetric data (sets and (hyper)graphs), which gives the maximally expressive equivariant linear layers and

Original: k -IGN in Section 3.1.
Revised: k -IGN in Section 3.1.

Original: The work on equivariant neural networks underlie our theory of how a standard Transformer can be a powerful learner for sets and (hyper)graphs.
Revised: The work on equivariant neural networks underlie our theory of how a standard Transformer can be a powerful learner for sets and (hyper)graphs.

Original: Expressive Power of Transformers and Its Connection to Equivariance
Revised: Expressive Power of Transformers and Its Connection to Equivariance

Original: Recent work involving
Revised: Recent work involving

Original: Transformers often focus on minimizing the domain- and task-specific inductive bias and scaling the model and data so that any useful computation structure can be learned [18, 31, 30, 7, 17, 9, 38].
Revised: Transformers often focus on minimizing the domain- and task-specific inductive bias and scaling the model and data so that any useful computation structure can be learned [18, 31, 30, 7, 17, 9, 39].

Original: The success of this approach is, to some degree, attributed to the high expressive power of Transformers that allows learning diverse functions suited for the data at hand [80, 38, 3, 4, 40].
Revised: The success of this approach is, to some degree, attributed to the high expressive power of Transformers that allows learning diverse functions suited for the data at hand [81, 39, 3, 4, 41].

Original: Recent theory has shown that Transformers are expressive enough to even model certain equivariant functions [1, 15, 38].
Revised: Recent theory has shown that Transformers are expressive enough to even model certain equivariant functions [1, 15, 39].

Original: Andreoli et al.
Revised: Andreoli et al.

Original: [1] cast self-attention and convolution into a unified framework using basis tensors similar to ones in Section 3.1.
Revised: [1] cast self-attention and convolution into a unified framework using basis tensors similar to ones in Section 3.1.

Original: Cordonnier et al.
Revised: Cordonnier et al.

Original: (2020) [15] advanced the idea and showed that Transformers with relative positional encodings can approximate any image convolution layers.
Revised: (2020) [15] advanced the idea and showed that Transformers with relative positional encodings can approximate any image convolution layers.

Original: Lee et al.
Revised: Lee et al.

Original: [38] and Kim et al.
Revised: (2019) [39] and Kim et al.

Original: [34] showed that Transformers can model equivariant linear layers for sets [81], which can be viewed as the first-order case of our theory (see Section 3.2).
Revised: [34] showed that Transformers can model equivariant linear layers for sets [82], which can be viewed as the first-order case of our theory (see Section 3.2).

Original: To our knowledge, our work is the first to show that standard Transformers are expressive enough to provably model maximally expressive equivariant layers and k -IGN for (hyper)graphs with k ≥ 2 .
Revised: To our knowledge, our work is the first to show that standard Transformers are expressive enough to provably model maximally expressive equivariant layers and k -IGN for (hyper)graphs with k ≥ 2 .

Original: Transformers for Graphs Unlike in language and vision, developing Transformers for graphs is challenging due to (1) the presence of edge connectivity and (2) the absence of canonical node ordering that prevents adopting simple positional encodings [49].
Revised: Transformers for Graphs Unlike in language and vision, developing Transformers for graphs is challenging due to (1) the presence of edge connectivity and (2) the absence of canonical node ordering that prevents adopting simple positional encodings [50].

Original: To incorporate the connectivity of edges, early methods restricted self-attention to local neighborhoods (thus reducing to messagepassing) [19, 50, 68] or used global self-attention with auxiliary message-passing modules [57, 42].
Revised: To incorporate the connectivity of edges, early methods restricted self-attention to local neighborhoods (thus reducing to messagepassing) [19, 51, 69] or used global self-attention with auxiliary message-passing modules [58, 43].

Original: As message-passing suffers from limited expressive power [76] and oversmoothing [39, 8, 51], recent works often discard them and use global self-attention on nodes with heuristic modifications to process edges [77, 29, 53, 37, 41].
Revised: As message-passing suffers from limited expressive power [77] and oversmoothing [40, 8, 52], recent works often discard them and use global self-attention on nodes with heuristic modifications to process edges [78, 29, 54, 38, 42].

Original: Ying et al.
Revised: Ying et al.

Original: [77] proposed to inject edge encoding based on shortest paths through self-attention bias.
Revised: (2021) [78] proposed to inject edge encoding based on shortest paths through self-attention bias.

Original: Kreuzer et al.
Revised: Kreuzer et al.

Original: [37] proposed to incorporate edges into self-attention matrix via elementwise multiplication.
Revised: [38] proposed to incorporate edges into self-attention matrix via elementwise multiplication.

Original: On the contrary, we leave the self-attention unmodified and provide both nodes and edges with certain token-wise embeddings (Section 2) as its input.
Revised: On the contrary, we leave the self-attention unmodified and provide both nodes and edges with certain token-wise embeddings (Section 2) as its input.

Original: To incorporate graph structure into nodes, on the other hand, some approaches focus on developing graph positional encoding, e.g. , based on Laplacian eigenvectors [20, 41, 37].
Revised: To incorporate graph structure into nodes, on the other hand, some approaches focus on developing graph positional encoding, e.g. , based on Laplacian eigenvectors [20, 42, 38].

Original: While these can be directly incorporated into our work via auxiliary node identifiers for better performance, we leave this as future work.
Revised: While these can be directly incorporated into our work via auxiliary node identifiers for better performance, we leave this as future work.

Original: We further note that current graph Transformers that utilize Laplacian positional encoding rely heavily on heuristic edge encoding [29, 37] while ours does not.
Revised: We further note that current graph Transformers that utilize Laplacian positional encoding rely heavily on heuristic edge encoding [29, 38] while ours does not.

Original: Another closely related approach is the Higher-order Transformer [34] which generalizes k -IGN with masked self-attention.
Revised: Another closely related approach is the Higher-order Transformer [34] which generalizes k -IGN with masked self-attention.

Original: While it is highly complex to implement due to hard-coded head-wise equivariant masks, our method can be implemented effortlessly using any available implementation of standard
Revised: While it is highly complex to implement due to hard-coded head-wise equivariant masks, our method can be implemented effortlessly using any available implementation of standard

Original: Transformer.
Revised: Transformer.

Original: Furthermore, our method is more flexible as the model can choose to use different attention heads to focus on a specific equivariant operator ( e.g. , local propagation) if needed.
Revised: Furthermore, our method is more flexible as the model can choose to use different attention heads to focus on a specific equivariant operator ( e.g. , local propagation) if needed.

Original: We further discuss the difficulty in applying linear attention to graph Transformers in Appendix A.2.
Revised: We further discuss the difficulty in applying linear attention to graph Transformers in Appendix A.2.

Original:  Experiments
Revised:  Experiments

Original: We first conduct a synthetic experiment that directly confirms our key claims in Lemma 1 (Section 3).
Revised: We first conduct a synthetic experiment that directly confirms our key claims in Lemma 1 (Section 3).

Original: Then, we empirically explore the capability of Tokenized Graph Transformer (TokenGT) (Section 2) using the PCQM4Mv2 large-scale quantum chemistry regression dataset [27].
Revised: Then, we empirically explore the capability of Tokenized Graph Transformer (TokenGT) (Section 2) using the PCQM4Mv2 large-scale quantum chemistry regression dataset [27].

Original: We further present experiments on transductive node classification datasets involving large graphs in Appendix A.4.3.
Revised: We further present experiments on transductive node classification datasets involving large graphs in Appendix A.4.3.

Original: [Table]
Revised: [Table]

Original: No embedding
Revised: No embedding

Original: Type id. only
Revised: Type id. only

Original: Node id. only
Revised: Node id. only

Original: Node id. + type id.
Revised: Node id. + type id.

Original: Equivariant basis (cid:31)(cid:30)(cid:31)(cid:31)(cid:31) (cid:31)(cid:30)(cid:31)(cid:29)(cid:31) (cid:31)(cid:30)(cid:31)(cid:31)(cid:28) (cid:31)(cid:30)(cid:31)(cid:31)(cid:27) (cid:31)(cid:30)(cid:31)(cid:31)(cid:26) (cid:31)(cid:30)(cid:31)(cid:31)(cid:25) (cid:31)(cid:30)(cid:31)(cid:31)(cid:31) (cid:31)(cid:30)(cid:31)(cid:29)(cid:31) (cid:31)(cid:30)(cid:31)(cid:31)(cid:28) (cid:31)(cid:30)(cid:31)(cid:31)(cid:27) (cid:31)(cid:30)(cid:31)(cid:31)(cid:26) (cid:31)(cid:30)(cid:31)(cid:31)(cid:25)
Revised: Equivariant basis (cid:31)(cid:30)(cid:31)(cid:31)(cid:31) (cid:31)(cid:30)(cid:31)(cid:29)(cid:31) (cid:31)(cid:30)(cid:31)(cid:31)(cid:28) (cid:31)(cid:30)(cid:31)(cid:31)(cid:27) (cid:31)(cid:30)(cid:31)(cid:31)(cid:26) (cid:31)(cid:30)(cid:31)(cid:31)(cid:25) (cid:31)(cid:30)(cid:31)(cid:31)(cid:31) (cid:31)(cid:30)(cid:31)(cid:29)(cid:31) (cid:31)(cid:30)(cid:31)(cid:31)(cid:28) (cid:31)(cid:30)(cid:31)(cid:31)(cid:27) (cid:31)(cid:30)(cid:31)(cid:31)(cid:26) (cid:31)(cid:30)(cid:31)(cid:31)(cid:25)

Original: [Figure]
Revised: [Figure]

Original: Self-attention learns acute patterns coherent to equivariant basis when orthonormal node identifiers and type identifiers are both provided as input.
Revised: Self-attention learns acute patterns coherent to equivariant basis when orthonormal node identifiers and type identifiers are both provided as input.

Original: More images can be found in Appendix A.4.1.
Revised: More images can be found in Appendix A.4.1.

Original: 5.1 Approximating Second-Order Equivariant Basis
Revised: 5.1 Approximating Second-Order Equivariant Basis

Original: As in Theorem 1 and 2 (Section 3), our argument on the expressive power of TokenGT relies on its capability to approximate order- k permutation equivariant linear layers L k → k (Definition 2).
Revised: As in Theorem 1 and 2 (Section 3), our argument on the expressive power of TokenGT relies on its capability to approximate order- k permutation equivariant linear layers L k → k (Definition 2).

Original: Specifically, Lemma 1 states that such capability depends on the ability of each self-attention head α 1 , ..., α H (Eq.
Revised: Specifically, Lemma 1 states that such capability depends on the ability of each self-attention head α 1 , ..., α H (Eq.

Original: (3)) to accurately approximate each equivariant basis B µ 1 , ..., B µ bell (2 k )
Revised: (3)) to accurately approximate each equivariant basis B µ 1 , ..., B µ bell (2 k )

Original: (Definition 2) up to normalization.
Revised: (Definition 2) up to normalization.

Original: We verify this claim for k = 2 (second-order; graphs) in a synthetic setup using Barabási-Albert random graphs.
Revised: We verify this claim for k = 2 (second-order; graphs) in a synthetic setup using Barabási-Albert random graphs.

Original: We use a multihead self-attention layer (Eq.
Revised: We use a multihead self-attention layer (Eq.

Original: (3)) with bell (2 + 2)
Revised: (3)) with bell (2 + 2)

Original: = 15 heads and explicitly supervise head-wise attention scores α h to approximate each (normalized)
Revised: = 15 heads and explicitly supervise head-wise attention scores α h to approximate each (normalized)

Original: equivariant basis tensor B µ h by minimizing L2 loss.
Revised: equivariant basis tensor B µ h by minimizing L2 loss.

Original: Having the layer hyperparameters fixed, we provide different combinations of node and type identifiers, and test if multihead self-attention can jointly approximate all 15 equivariant basis on unseen graphs.
Revised: Having the layer hyperparameters fixed, we provide different combinations of node and type identifiers, and test if multihead self-attention can jointly approximate all 15 equivariant basis on unseen graphs.

Original: We experiment with both dense and sparse graph representations; for graphs with n nodes and m edges, the dense graph considers all n 2 pairwise edges as input as in Section 3, whereas the sparse graph considers only the present m edges as in Section 2.
Revised: We experiment with both dense and sparse graph representations; for graphs with n nodes and m edges, the dense graph considers all n 2 pairwise edges as input as in Section 3, whereas the sparse graph considers only the present m edges as in Section 2.

Original: Further details can be found in Appendix A.3.2.
Revised: Further details can be found in Appendix A.3.2.

Original: We outline the results in Table 1.
Revised: We outline the results in Table 1.

Original: Consistent with Lemma 1, self-attention achieves accurate approximation of equivariant basis only when both the orthonormal node identifiers and type identifiers are given.
Revised: Consistent with Lemma 1, self-attention achieves accurate approximation of equivariant basis only when both the orthonormal node identifiers and type identifiers are given.

Original: Here, Laplacian eigenvectors (Lap, ⃝ ) often yield slightly better results than orthogonal random features (ORF, ⃝ ) presumably due to less stochasticity.
Revised: Here, Laplacian eigenvectors (Lap, ⃝ ) often yield slightly better results than orthogonal random features (ORF, ⃝ ) presumably due to less stochasticity.

Original: Interestingly, we see that self-attention transfers the learned (pseudo-)equivariant self-attention structure to unseen graphs near perfectly.
Revised: Interestingly, we see that self-attention transfers the learned (pseudo-)equivariant self-attention structure to unseen graphs near perfectly.

Original: [Table]
Revised: [Table]

Original: a tt e n t i o n d i s t a n ce ( h o p s )
Revised: a tt e n t i o n d i s t a n ce ( h o p s )

Original: Network depth (layer) 

Revised: Network depth (layer)

Original: M e a n a tt e n t i o n d
Revised: M e a n a tt e n t i o n d

Original: i s t a n ce
Revised: i s t a n ce

Original: ( h o p s )
Revised: ( h o p s )

Original: Head
Revised: Head

Original: Head...
Revised: Head...

Original: Non-orthogonal random embeddings lead to inaccurate approximation (Random, ⃝ ), highlighting the importance of orthogonality of node identifiers.
Revised: Non-orthogonal random embeddings lead to inaccurate approximation (Random, ⃝ ), highlighting the importance of orthogonality of node identifiers.

Original: The approximation is also inaccurate when we sample ORF P t independently for each token t (ORF (first-order), ⃝ ) instead of using concatenated node identifiers [ P u , P v ] for token ( u, v ) .
Revised: The approximation is also inaccurate when we sample ORF P t independently for each token t (ORF (first-order), ⃝ ) instead of using concatenated node identifiers [ P u , P v ] for token ( u, v ) .

Original: This supports our argument in Section 2 that the incidence information implicitly provided via node identifiers plays a key role in approximation.
Revised: This supports our argument in Section 2 that the incidence information implicitly provided via node identifiers plays a key role in approximation.

Original: In Figure 2, we provide a visualization of self-attention maps learned under various node and type identifier choices.
Revised: In Figure 2, we provide a visualization of self-attention maps learned under various node and type identifier choices.

Original: Additional results can be found in Appendix A.4.1.
Revised: Additional results can be found in Appendix A.4.1.

Original: 5.2 Large-Scale Graph Learning
Revised: 5.2 Large-Scale Graph Learning

Original: An exclusive characteristic of TokenGT is its minimal graph-specific inductive bias, which requires it to learn internal computation structure largely from data.
Revised: An exclusive characteristic of TokenGT is its minimal graph-specific inductive bias, which requires it to learn internal computation structure largely from data.

Original: As such models are commonly known to work well with large-scale data [67, 18], we explore the capability of TokenGT on the PCQM4Mv quantum chemistry regression dataset [27], one of the current largest with 3.7M molecular graphs.
Revised: As such models are commonly known to work well with large-scale data [68, 18], we explore the capability of TokenGT on the PCQM4Mv quantum chemistry regression dataset [27], one of the current largest with 3.7M molecular graphs.

Original: For TokenGT, we use both node and type identifiers, and use main Transformer encoder configuration based on Graphormer [77] with 12 layers, 768 hidden dimension, and 32 attention heads.
Revised: For TokenGT, we use both node and type identifiers, and use main Transformer encoder configuration based on Graphormer [78] with 12 layers, 768 hidden dimension, and 32 attention heads.

Original: We try both ORF and Laplacian eigenvector as node identifiers, and denote corresponding models as TokenGT (ORF) and TokenGT (Lap) respectively.
Revised: We try both ORF and Laplacian eigenvector as node identifiers, and denote corresponding models as TokenGT (ORF) and TokenGT (Lap) respectively.

Original: As an ablation, we also experiment with the same Transformer without node and type identifiers, which we denote as Transformer .
Revised: As an ablation, we also experiment with the same Transformer without node and type identifiers, which we denote as Transformer .

Original: Finally, we apply the kernel attention [11] that approximates the attention computation to linear cost ( TokenGT (Lap) + Performer ).
Revised: Finally, we apply the kernel attention [11] that approximates the attention computation to linear cost ( TokenGT (Lap) + Performer ).

Original: We use AdamW optimizer with ( β 1 , β 2 ) =
Revised: We use AdamW optimizer with ( β 1 , β 2 ) =

Original: (0 . 99 , 0 . 999) and weight decay 0.1, and 60k learning rate warmup steps followed by linear decay over 1M iteration with batch size 1024.
Revised: (0 . 99 , 0 . 999) and weight decay 0.1, and 60k learning rate warmup steps followed by linear decay over 1M iteration with batch size 1024.

Original: For fine-tuning, we use 1k warmup, 0.1M training steps, and cosine learning rate decay.
Revised: For fine-tuning, we use 1k warmup, 0.1M training steps, and cosine learning rate decay.

Original: We train the models on 8 RTX 3090 GPUs for 3 days.
Revised: We train the models on 8 RTX 3090 GPUs for 3 days.

Original: Further details are in Appendix A.3.3.
Revised: Further details are in Appendix A.3.3.

Original: We provide the results in Table 2.
Revised: We provide the results in Table 2.

Original: A standard Transformer on the node and edge tokens cannot recognize graph structure and shows low performance (0.2340 valid MAE).
Revised: A standard Transformer on the node and edge tokens cannot recognize graph structure and shows low performance (0.2340 valid MAE).

Original: Yet, the picture changes as soon as we augment the tokens with node and type identifiers.
Revised: Yet, the picture changes as soon as we augment the tokens with node and type identifiers.

Original: Notably, TokenGT (ORF) achieves 0.0962 MAE, which is already better than all GNN baselines.
Revised: Notably, TokenGT (ORF) achieves 0.0962 MAE, which is already better than all GNN baselines.

Original: This is a somewhat surprising result, as both ORF and the Transformer are not aware of graph structures.
Revised: This is a somewhat surprising result, as both ORF and the Transformer are not aware of graph structures.

Original: This implies Transformer is strong enough to learn to interpret and reason over the incidence structure of tokens provided only implicitly by the node and type identifiers.
Revised: This implies Transformer is strong enough to learn to interpret and reason over the incidence structure of tokens provided only implicitly by the node and type identifiers.

Original: By further switching to Laplacian eigenvectors that encode position on graphs [20], we observe a performance boost to 0.0910 MAE, competitive to Transformers with sophisticated graph-specific modifications ( e.g. , shortest path-based spatial encoding [77]).
Revised: By further switching to Laplacian eigenvectors that encode position on graphs [20], we observe a performance boost to 0.0910 MAE, competitive to Transformers with sophisticated graph-specific modifications ( e.g. , shortest path-based spatial encoding [78]).

Original: While such methods inject graph structure into attention matrix via bias term and therefore strictly require O ( n 2 ) cost, TokenGT enables adopting kernelization for pure self-attention [11], resulting in
Revised: While such methods inject graph structure into attention matrix via bias term and therefore strictly require O ( n 2 ) cost, TokenGT enables adopting kernelization for pure self-attention [11], resulting in

Original: TokenGT (Lap) + Performer with the best performance among O ( n + m ) models (0.0935 MAE).
Revised: TokenGT (Lap) + Performer with the best performance among O ( n + m ) models (0.0935 MAE).

Original: Further discussion on the empirical performance of TokenGT can be found in Appendix A.5.
Revised: Further discussion on the empirical performance of TokenGT can be found in Appendix A.5.

Original: While our theory in Section 3 guarantees that TokenGT can reduce to an equivariant layer by learning fixed equivariant basis at each attention head, in practice, it can freely utilize multihead self-attention to learn less restricted and more useful computation structure from data.
Revised: While our theory in Section 3 guarantees that TokenGT can reduce to an equivariant layer by learning fixed equivariant basis at each attention head, in practice, it can freely utilize multihead self-attention to learn less restricted and more useful computation structure from data.

Original: To analyze such a structure, we compute the attention distance across heads and network depth by averaging pairwise token distances on a graph weighted by their attention scores (Figure 3).
Revised: To analyze such a structure, we compute the attention distance across heads and network depth by averaging pairwise token distances on a graph weighted by their attention scores (Figure 3).

Original: This distance is analogous to the number of hops in message-passing.
Revised: This distance is analogous to the number of hops in message-passing.

Original: In both TokenGT (ORF) and TokenGT (Lap), in the lowest layers, some heads attend globally over the graph while others consistently have small receptive fields (acting like a local message-passing operator).
Revised: In both TokenGT (ORF) and TokenGT (Lap), in the lowest layers, some heads attend globally over the graph while others consistently have small receptive fields (acting like a local message-passing operator).

Original: In deeper layers, the attention distances increase, and most heads attend globally.
Revised: In deeper layers, the attention distances increase, and most heads attend globally.

Original: Interestingly, this behavior is highly consistent with Vision Transformers on image patches [18], suggesting that hybrid architectures based on convolution to aid ViT [16, 79] might also work well for graphs.
Revised: Interestingly, this behavior is highly consistent with Vision Transformers on image patches [18], suggesting that hybrid architectures based on convolution to aid ViT [16, 80] might also work well for graphs.

Original: While TokenGT (ORF) shows relatively consistent attention distance over heads, TokenGT (Lap) shows higher variance, implying that it learns more diverse attention patterns.
Revised: While TokenGT (ORF) shows relatively consistent attention distance over heads, TokenGT (Lap) shows higher variance, implying that it learns more diverse attention patterns.

Original: Judging from the higher performance of TokenGT (Lap), this suggests that the graph structure information of the Laplacian eigenvector facilitates learning useful and diverse attention structures, which calls for future exploration of better node identifiers based on graph PEs [37, 41].
Revised: Judging from the higher performance of TokenGT (Lap), this suggests that the graph structure information of the Laplacian eigenvector facilitates learning useful and diverse attention structures, which calls for future exploration of better node identifiers based on graph PEs [38, 42].

Original:  Conclusion
Revised:  Conclusion

Original: We showed that Transformers directly applied to graphs can work well in both theory and practice.
Revised: We showed that Transformers directly applied to graphs can work well in both theory and practice.

Original: In the theoretical aspect, we proved that with appropriate token-wise embeddings, a Transformer on node and edge tokens is at least as expressive as k -IGN and k -WL test, making it more expressive than all message-passing GNNs.
Revised: In the theoretical aspect, we proved that with appropriate token-wise embeddings, a Transformer on node and edge tokens is at least as expressive as k -IGN and k -WL test, making it more expressive than all message-passing GNNs.

Original: For such token-wise embeddings, we showed that a combination of simple orthonormal node identifiers and trainable type identifiers suffices, which we also verified with a synthetic experiment.
Revised: For such token-wise embeddings, we showed that a combination of simple orthonormal node identifiers and trainable type identifiers suffices, which we also verified with a synthetic experiment.

Original: In an experiment with PCQM4Mv2 large-scale dataset, we show that Tokenized
Revised: In an experiment with PCQM4Mv2 large-scale dataset, we show that Tokenized

Original: Graph Transformer (TokenGT) performs significantly better than all GNNs and is competitive with
Revised: Graph Transformer (TokenGT) performs significantly better than all GNNs and is competitive with

Original: Transformer variants with strong graph-specific architectural components [77, 29, 53].
Revised: Transformer variants with strong graph-specific architectural components [78, 29, 54].

Original: While the results suggest a promising research direction, there are challenges to be addressed in future work.
Revised: While the results suggest a promising research direction, there are challenges to be addressed in future work.

Original: First, treating each node and edge as tokens requires O (( n + m ) 2 ) asymptotic cost due to the quadratic nature of self-attention.
Revised: First, treating each node and edge as tokens requires O (( n + m ) 2 ) asymptotic cost due to the quadratic nature of self-attention.

Original: While we address this to some degree with kernelization and achieve O ( n + m ) cost, other types of efficient Transformers ( e.g. , sparse) that can deliver better performance are left to be tested.
Revised: While we address this to some degree with kernelization and achieve O ( n + m ) cost, other types of efficient Transformers ( e.g. , sparse) that can deliver better performance are left to be tested.

Original: Another issue is slightly lower performance compared to the stateof-the-art.
Revised: Another issue is slightly lower performance compared to the stateof-the-art.

Original: Adopting Transformer engineering techniques from vision and language domains, such as data scaling [7, 18], deepening [69, 73], hybrid architectures [16, 79], and self-supervision [17, 7, 24], are promising.
Revised: Adopting Transformer engineering techniques from vision and language domains, such as data scaling [7, 18], deepening [70, 74], hybrid architectures [16, 80], and self-supervision [17, 7, 24], are promising.

Original: In the societal aspect, to prevent the potential risky behavior in, e.g. , decision making from graph-structured inputs, interpretability research regarding self-attention on graphs is desired.
Revised: In the societal aspect, to prevent the potential risky behavior in, e.g. , decision making from graph-structured inputs, interpretability research regarding self-attention on graphs is desired.

Original: We finish with interesting research directions that stem from our work.
Revised: We finish with interesting research directions that stem from our work.

Original: As our approach advocates viewing a graph as ( n + m ) tokens [36], it opens up new paradigms of graph learning, including autoregressive decoding, in-context learning, prompting, and multimodal learning.
Revised: As our approach advocates viewing a graph as ( n + m ) tokens [37], it opens up new paradigms of graph learning, including autoregressive decoding, in-context learning, prompting, and multimodal learning.

Original: Another interesting direction is to extend our theory and use self-attention to approximate equivariant basis for general discrete group actions, which might be a viable approach for learning equivariance from data .
Revised: Another interesting direction is to extend our theory and use self-attention to approximate equivariant basis for general discrete group actions, which might be a viable approach for learning equivariance from data .

Original: Acknowledgement This work was supported in part by Institute of Information & communications
Revised: Acknowledgement This work was supported in part by Institute of Information & communications

Original: Technology Planning & Evaluation (IITP) (No. 2022-0-00926, 2022-0-00959, 2021-0-02068, and 2019-0-00075) and the National Research Foundation of Korea (NRF) (No. 2021R1C1C1012540) grants funded by the Korea government (MSIT).
Revised: Technology Planning & Evaluation (IITP) (No. 2022-0-00926, 2022-0-00959, 2021-0-02068, and 2019-0-00075) and the National Research Foundation of Korea (NRF) (No. 2021R1C1C1012540) grants funded by the Korea government (MSIT).

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section 3 and Section 5.
Revised: [Yes] See Section 3 and Section 5.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 3.
Revised: [Yes] See Section 3.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We include them in the supplementary file.
Revised: [Yes] We include them in the supplementary file.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We include them in the supplementary file.
Revised: [Yes] We include them in the supplementary file.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 5. 4.
Revised: [Yes] See Section 5. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.2.
Revised: [Yes] See Section 5.2.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 5.2.
Revised: [Yes] See Section 5.2.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We include them in the supplementary file.
Revised: [Yes] We include them in the supplementary file.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]