Original: R EACH THE R EMOTE N EIGHBORS : D UAL -E NCODING T RANSFORMER FOR G RAPHS
Revised: R EACH THE R EMOTE N EIGHBORS : D UAL -E NCODING T RANSFORMER FOR G RAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite recent successes in natural language processing and computer vision,
Revised: Despite recent successes in natural language processing and computer vision,

Original: Transformer suffers from the scalability problem when dealing with graphs.
Revised: Transformer suffers from the scalability problem when dealing with graphs.

Original: Computing node-to-node attentions is infeasible on complicated graphs, e.g., knowledge graphs.
Revised: Computing node-to-node attentions is infeasible on complicated graphs, e.g., knowledge graphs.

Original: One solution is to consider only the near neighbors, which, however, will lose the key merit of Transformer that attends to the elements at any distance.
Revised: One solution is to consider only the near neighbors, which, however, will lose the key merit of Transformer that attends to the elements at any distance.

Original: In this paper, we propose a new Transformer architecture, named dual-encoding
Revised: In this paper, we propose a new Transformer architecture, named dual-encoding

Original: Transformer (DET), which has a structural encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors.
Revised: Transformer (DET), which has a structural encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors.

Original: The two encoders can be incorporated to boost each other’s performance.
Revised: The two encoders can be incorporated to boost each other’s performance.

Original: Our experiments demonstrate that DET achieves superior performance compared to the respective state-of-the-art attention-based methods in dealing with molecules, networks and knowledge graphs.
Revised: Our experiments demonstrate that DET achieves superior performance compared to the respective state-of-the-art attention-based methods in dealing with molecules, networks and knowledge graphs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformer has become one of the most prevalent neural models for natural language processing (NLP)
Revised: Transformer has become one of the most prevalent neural models for natural language processing (NLP)

Original: (Vaswani et al., 2017; Devlin et al., 2019).
Revised: (Vaswani et al., 2017; Devlin et al., 2019).

Original: The self-attention mechanism leveraged by
Revised: The self-attention mechanism leveraged by

Original: Transformer has already been extended to graph neural networks (GNNs), e.g., GAT (Velickovic et al., 2018) and its variants (Wu et al., 2019; Vashishth et al., 2020;
Revised: Transformer has already been extended to graph neural networks (GNNs), e.g., GAT (Velickovic et al., 2018) and its variants (Wu et al., 2019; Vashishth et al., 2020;

Original: Chen et al., 2021b; Kim & Oh, 2021).
Revised: Chen et al., 2021b; Kim & Oh, 2021).

Original: Nevertheless, these models only consider the near (usually one-hop) neighbors, which may violate the original intention of Transformer that attends to the elements at distant positions.
Revised: Nevertheless, these models only consider the near (usually one-hop) neighbors, which may violate the original intention of Transformer that attends to the elements at distant positions.

Original: Recently, Graphormer (Ying et al., 2021) starts to leverage the standard Transformer architecture for graph representation learning and has achieved superior performance on many benchmarks.
Revised: Recently, Graphormer (Ying et al., 2021) starts to leverage the standard Transformer architecture for graph representation learning and has achieved superior performance on many benchmarks.

Original: However, in its scenarios of graph property prediction, the datasets are small graphs (e.g., small molecules).
Revised: However, in its scenarios of graph property prediction, the datasets are small graphs (e.g., small molecules).

Original: The full node-to-node attention leveraged by Graphormer makes it inapplicable to large graphs with millions of nodes, such as knowledge graphs (KGs) or social networks.
Revised: The full node-to-node attention leveraged by Graphormer makes it inapplicable to large graphs with millions of nodes, such as knowledge graphs (KGs) or social networks.

Original: The same problem also appears in the computer vision (CV) area, yet has recently been tackled by patching pixels to patches and then to windows in a hierarchical fashion (Dosovitskiy et al., 2021; Liu et al., 2021a).
Revised: The same problem also appears in the computer vision (CV) area, yet has recently been tackled by patching pixels to patches and then to windows in a hierarchical fashion (Dosovitskiy et al., 2021; Liu et al., 2021b).

Original: These works inspire us to explore the possibility of using one universal Transformer architecture as the general backbone to model graphs of different sizes.
Revised: These works inspire us to explore the possibility of using one universal Transformer architecture as the general backbone to model graphs of different sizes.

Original: In addition to many self-attention-based methods considering only one-hop neighbors (Schlichtkrull et al., 2018; Wu et al., 2019; Ye et al., 2019;
Revised: In addition to many self-attention-based methods considering only one-hop neighbors (Schlichtkrull et al., 2018; Wu et al., 2019; Ye et al., 2019;

Original: Chen et al., 2021b; Kim & Oh, 2021), some existing works introduce multi-hop (usually 2- or 3-hop) neighbors (Abu-El-Haija et al., 2019; Sun et al., 2020).
Revised: Chen et al., 2021b; Kim & Oh, 2021), some existing works introduce multi-hop (usually 2- or 3-hop) neighbors (Abu-El-Haija et al., 2019; Sun et al., 2020).

Original: However, they still concentrate on the local information and fail to obtain useful information from the remote nodes.
Revised: However, they still concentrate on the local information and fail to obtain useful information from the remote nodes.

Original: Capturing the remote correlations is one of the most important characteristics of Transformer, because the rich context not only boosts the performance but also avoids over-ﬁtting for local information.
Revised: Capturing the remote correlations is one of the most important characteristics of Transformer, because the rich context not only boosts the performance but also avoids over-ﬁtting for local information.

Original: For example, attending to the distant nodes may be helpful even on a highly homophilic graph, considering the existence of enormous missing links (Ciotti et al., 2016).
Revised: For example, attending to the distant nodes may be helpful even on a highly homophilic graph, considering the existence of enormous missing links (Ciotti et al., 2016).

Original: In this paper, we propose a dual-encoding Transformer (DET).
Revised: In this paper, we propose a dual-encoding Transformer (DET).

Original: In DET, we consider two types of neighbors, i.e., structural neighbors and semantic neighbors.
Revised: In DET, we consider two types of neighbors, i.e., structural neighbors and semantic neighbors.

Original: Structural neighbors are the near neighbors leveraged by most existing GNNs (Vashishth et al., 2020; Chen et al., 2021b;
Revised: Structural neighbors are the near neighbors leveraged by most existing GNNs (Vashishth et al., 2020; Chen et al., 2021b;

Original: Kim & Oh, 2021).
Revised: Kim & Oh, 2021).

Original: By contrast, Semantic neighbors, i.e., the remote neighbors, are the semantically close nodes but may be remote from the node of interest in structure. 
Revised: By contrast, Semantic neighbors, i.e., the remote neighbors, are the semantically close nodes but may be remote from the node of interest in structure. [Figure] i n t e r [Figure]

Revised: t

Original: [Figure]
Revised: [Figure]

Revised: structuralembeddings

Original: Figure 1 shows the basic idea of DET.
Revised: Figure 1 shows the basic idea of DET.

Original: For structural encoding, we use the standard self-attention layer to encode the structural neighbors.
Revised: For structural encoding, we use the standard self-attention layer to encode the structural neighbors.

Original: For semantic encoding, we use a modiﬁed linear attention layer to encode the semantic neighbors.
Revised: For semantic encoding, we use a modiﬁed linear attention layer to encode the semantic neighbors.

Original: The dual encoding ensures both local aggregation and global connection, and also enables them to beneﬁt from each other through back propagation.
Revised: The dual encoding ensures both local aggregation and global connection, and also enables them to beneﬁt from each other through back propagation.

Original: The idea of reaching remote neighbors is inspired by MSA Transformer (Rao et al., 2021) and
Revised: The idea of reaching remote neighbors is inspired by MSA Transformer (Rao et al., 2021) and

Original: AlphaFold 2 (Jumper et al., 2021).
Revised: AlphaFold 2 (Jumper et al., 2021).

Original: They query the genetic database to fetch the similar sequences (i.e., proteins) as “family members”.
Revised: They query the genetic database to fetch the similar sequences (i.e., proteins) as “family members”.

Original: The difference is that the family members in DET (i.e., semantic neighbors) are obtained by self-supervised learning rather than asking for external resources.
Revised: The difference is that the family members in DET (i.e., semantic neighbors) are obtained by self-supervised learning rather than asking for external resources.

Original: Brieﬂy, we convert this problem to a learning task to ﬁnd the distant nodes that are as important as local neighbors.
Revised: Brieﬂy, we convert this problem to a learning task to ﬁnd the distant nodes that are as important as local neighbors.

Original: We then view local neighbors as positive examples and randomly sampled distant nodes as negative examples, constructing a standard contrastive learning.
Revised: We then view local neighbors as positive examples and randomly sampled distant nodes as negative examples, constructing a standard contrastive learning.

Original: Furthermore, we propose to use the semantic operator (cid:9) to estimate the score between the node of interest and others.
Revised: Furthermore, we propose to use the semantic operator (cid:9) to estimate the score between the node of interest and others.

Original: It is a learnable function to compel the encoder to value the semantically close nodes.
Revised: It is a learnable function to compel the encoder to value the semantically close nodes.

Original: The proposed DET is capable of achieving superior performance on various graph learning tasks.
Revised: The proposed DET is capable of achieving superior performance on various graph learning tasks.

Original: (1) For graph property prediction, DET outperforms the best-performing methods on the PCQM4MLSCv1 (Nakata Shimazaki 2017) and ZINC (Dwivedi et al. , 2020) datasets; (2) For node classiﬁcation, DET obtains competitive or better performance compared with the state-of-the-art attentionbased methods, on several prevalent benchmarks (e.g., Cora, PPI, and ogbn-arxiv)
Revised: (1) For graph property prediction, DET outperforms the best-performing methods on the PCQM4MLSCv1 (Nakata & Shimazaki, 2017) and ZINC (Dwivedi et al., 2020) datasets; (2) For node classiﬁcation, DET obtains competitive or better performance compared with the state-of-the-art attentionbased methods, on several prevalent benchmarks (e.g., Cora, PPI, and ogbn-arxiv)

Original: (Yang et al., 2016; Nakata & Shimazaki, 2017; Zitnik & Leskovec, 2017); (3) For KG completion (a.k.a., entity prediction), DET achieves the state-of-the-art performance on both FB15K-

Revised: (Yang et al., 2016; Nakata & Shimazaki, 2017; Zitnik & Leskovec, 2017); (3) For KG completion (a.k.a., entity prediction), DET achieves the state-of-the-art performance on both FB15K-

Original: (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2018).
Revised: (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2018).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: We split the related literature into two parts: self-attention and position embedding.
Revised: We split the related literature into three parts: non-local GNNs, self-attention, and position embedding.

Original: Self-attention Self-attention-based neural models, such as Transformer, have recently become the de facto choice in NLP, ranging from language modeling and machine translation (Devlin et al., 2019; Vaswani et al., 2017) to question answering (Yang et al., 2019; Yavuz et al., 2022) and sentiment analysis (Cheng et al., 2021; Xu et al., 2019a).
Revised: Non-local GNNs Some methods also investigate how to capture the relationships among disconnected nodes (Pei et al., 2020; Yao et al., 2020; Liu et al., 2021a; Min et al., 2022). Speciﬁcally,

Original: Transformer has signiﬁcant advantages over conventional sequential models like recurrent neural networks (RNNs) (Williams & Zipser, 1989; Hochreiter & Schmidhuber, 1997) in both scalability and efﬁciency. When modelling graph data, another well-used self-attention mechanism is GAT (Velickovic et al., 2018), which we call thelinear attention in this paper.
Revised: Geom-GCN (Pei et al., 2020) learns the aggregation purely based on embedding distance, while Non-local-GNNs (Liu et al., 2021a) uses the attention scores from a virtual node to other nodes as a sorting metric to ﬁnd non-local neighbors.

Original: Compared with the dot-product calculation in the original implementation, the linear attention performs concatenation of query and key nodes before a linear transformation. Furthermore, it considers only the one-hop neighbors as the keys, which is efﬁcient for embedding large graphs.
Revised: However, they only focus on modeling networks and are evaluated on multi-classiﬁcation tasks with a few classes. They also do not distinguish between remote nodes and direct neighbors.

Original: The linear attention has become the primal choice for modeling graph data (Wang et al., 2019a; Wu et al., 2019; Sun et al., 2020; Vashishth et al., 2020; Kim & Oh, 2021). Some methods also investigate how to capture the relationships among disconnected nodes (Min et al., 2022; Yao et al., 2020).
Revised: (Yao et al., 2020; Min et al., 2022) leverage hand-crafted features to ﬁnd the useful remote nodes, which are less relevant to our work. Self-attention Self-attention-based neural models, such as Transformer, have recently become the de facto choice in NLP, ranging from language modeling and machine translation (Devlin et al., 2019; Vaswani et al., 2017) to question answering (Yang et al., 2019; Yavuz et al., 2022) and sentiment analysis (Cheng et al., 2021; Xu et al., 2019a).

Original: However, they all rely on hand-crafted features.
Revised: Transformer has signiﬁcant advantages over conventional

Original: Position Embedding Position embedding is one of the most important modules of Transformer.
Revised: Position Embedding Position embedding is one of the most important modules of Transformer.

Original: Transformer variants in different ﬁelds typically customize different designs in this module.
Revised: Transformer variants in different ﬁelds typically customize different designs in this module.

Original: For example, ViT and its followers (Dosovitskiy et al., 2021; Fan et al., 2021; Han et al., 2021) sequentially index the patches and encode the indices as 1D position embeddings.
Revised: For example, ViT and its followers (Dosovitskiy et al., 2021; Fan et al., 2021; Han et al., 2021) sequentially index the patches and encode the indices as 1D position embeddings.

Original: SwinTransformer (Liu et al., 2021a;b) proposes the 2D-aware relative position biases, which employs a learnable matrix to record pairwise patch position information in the window.
Revised: SwinTransformer (Liu et al., 2021b;c) proposes the 2D-aware relative position biases, which employs a learnable matrix to record pairwise patch position information in the window.

Original: In addition to the position information, other prior knowledge can also be injected as attention biases or position embeddings into Transformer, which becomes the key to applying Transformer on graphs (Ahn et al., 2021; Chen et al., 2021a;b; Dwivedi & Bresson, 2021; Kreuzer et al., 2021;
Revised: In addition to the position information, other prior knowledge can also be injected as attention biases or position embeddings into Transformer, which becomes the key to applying Transformer on graphs (Ahn et al., 2021; Chen et al., 2021a;b; Dwivedi & Bresson, 2021; Kreuzer et al., 2021;

Original: Ying et al., 2021).
Revised: Ying et al., 2021).

Original: For example, GT (Dwivedi & Bresson, 2021) replaces the sinusoidal positional embeddings by the Laplacian eigenvectors.
Revised: For example, GT (Dwivedi & Bresson, 2021) replaces the sinusoidal positional embeddings by the Laplacian eigenvectors.

Original: Graphformer (Ying et al., 2021) encodes centrality and shortest path distance into embeddings, and then incorporates them as position embeddings into
Revised: Graphformer (Ying et al., 2021) encodes centrality and shortest path distance into embeddings, and then incorporates them as position embeddings into

Original: Transformer.
Revised: Transformer.

Original: HittER (Chen et al., 2021b) adds the edge type (i.e., relation) information of KGs when encoding entity embeddings.
Revised: HittER (Chen et al., 2021b) adds the edge type (i.e., relation) information of KGs when encoding entity embeddings.

Original: In particular, MSA Transformer (Baek et al., 2021; Jumper et al., 2021;

Original: Rao et al., 2021) leverages the multiple sequence alignment (MSA) information, which is closely related to our work.

Original: When encoding the information of one protein site, it considers not only the sites along the protein sequence but also the sites shared by the evolutionary family members.

Original: The semantic encoding proposed in this paper also exploits such “family” information, but the “family members” in DET are selected based on their learnable semantic scores.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we present the details of DET.
Revised: In this section, we present the details of DET.

Original: We start from the preliminaries and then introduce the dual-encoding process.
Revised: We start from the preliminaries and then introduce the dual-encoding process.

Original: Finally, we illustrate how to train a DET model.
Revised: Finally, we illustrate how to train a DET model.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: We ﬁrst introduce the terminologies and notations that will be used in the following sections.
Revised: We ﬁrst introduce the terminologies and notations that will be used in the following sections.

Original: Graph We deﬁne a graph as G =
Revised: Graph We deﬁne a graph as G =

Original: ( V , E ) , where V = { v 1 , v 2 , ..., v n } is the node set, and E = { e 1 , e 2 , ..., e m } is the edge set.
Revised: ( V , E ) , where V = { v 1 , v 2 , ..., v n } is the node set, and E = { e 1 , e 2 , ..., e m } is the edge set.

Original: n and m denote the number of nodes and edges, respectively.
Revised: n and m denote the number of nodes and edges, respectively.

Original: In practice, different tasks often have more complicated graph structures.
Revised: In practice, different tasks often have more complicated graph structures.

Original: For example, molecular graphs and KGs have edge types (i.e., chemical bonds and relations).
Revised: For example, molecular graphs and KGs have edge types (i.e., chemical bonds and relations).

Original: We do not discuss the details and follow the general setting to process these speciﬁc features (Chen et al., 2021b;
Revised: We do not discuss the details and follow the general setting to process these speciﬁc features (Chen et al., 2021b;

Original: Ying et al., 2021).
Revised: Ying et al., 2021).

Original: GNN and Self-attention Without loss of generality, we deﬁne a GNN as a neural network that learns a group of weights to aggregate the embeddings of the one-hop or multi-hop neighbors for the node of interest.
Revised: GNN and Self-attention Without loss of generality, we deﬁne a GNN as a neural network that learns a group of weights to aggregate the embeddings of the one-hop or multi-hop neighbors for the node of interest.

Original: In this sense, self-attention can be naturally treated as a GNN model.
Revised: In this sense, self-attention can be naturally treated as a GNN model.

Original: Let Q ∈ R n × h ,
Revised: Let Q ∈ R n × h ,

Original: K ∈ R n × h , V ∈ R n × h denote the query, key, and value matrices, respectively.
Revised: K ∈ R n × h , V ∈ R n × h denote the query, key, and value matrices, respectively.

Original: In this paper, they are the same node embedding matrix.
Revised: In this paper, they are the same node embedding matrix.

Original: h denotes the hidden layer size.
Revised: h denotes the hidden layer size.

Original: Self-attention calculates the attention scores as follows:
Revised: Self-attention calculates the attention scores as follows:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R n × n records the node-to-node attention scores.
Revised: where A ∈ R n × n records the node-to-node attention scores.

Original: We then aggregate the node embeddings with the following equation:
Revised: We then aggregate the node embeddings with the following equation:

Original: [Equation]
Revised: [Equation]

Original: where H ∈ R n × h is the output embedding matrix, with each row denoting the embedding of a node.
Revised: where H ∈ R n × h is the output embedding matrix, with each row denoting the embedding of a node.

Original: Linear Attention
Revised: Linear Attention

Original: The computational complexity of the above dot-product implementation is Ω( n 2 ) (without considering the hidden layer size).
Revised: The computational complexity of the above dot-product implementation is Ω( n 2 ) (without considering the hidden layer size).

Original: As the number of nodes increases, the cost becomes unacceptable.
Revised: As the number of nodes increases, the cost becomes unacceptable.

Original: GAT (Velickovic et al., 2018) proposes a linear self-attention implementation to mitigate this problem by only considering the one-hop neighbors: 
Revised: GAT (Velickovic et al., 2018) proposes a linear self-attention implementation to mitigate this problem by only considering the one-hop neighbors: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where B ij denotes the attention score from the node of interest v i to a neighbor v j . v i , v j ∈ R h are the embeddings of v i and v j , respectively.
Revised: where B ij denotes the attention score from the node of interest v i to a neighbor v j . v i , v j ∈ R h are the embeddings of v i and v j , respectively.

Original: b ∈ R h and W ∈ R
Revised: b ∈ R h and W ∈ R

Original: h × h are weight vector and matrix, 
Revised: h × h are weight vector and matrix, respectively.

Revised: σ is the activation function and (cid:107) denotes the concatenation operation.

Revised: The linear attention does not consider the correlations within neighbors, and thus its computational complexity is cut down from Ω( n 2 ) to Ω( m ) .

Revised: Note that, it is not necessary to use one-hop neighbors as keys in linear attention.

Revised: Some existing works also consider multi-hop neighbors (Sun et al., 2020).

Original: Dataset 1-hop 2-hop 3-hop 5-hop
Revised: Dataset 1-hop 2-hop 3-hop 5-hop

Original: WN18RR 2.
Revised: WN18RR 2.

Original: 483.8 FB15K20.
Revised: 483.8 FB15K20.

Original: [Figure]
Revised: [Figure]

Original: 1: Input: graph G =
Revised: 1: Input: graph G =

Original:  V predic tion loss  structural encoder M  semantic encoder
Revised: ( V , E ) , the main prediction loss L main , the structural encoder M st , and semantic encoder

Original: ; 3: repeat 4:  do 6:
Revised: ; 3: repeat 4: ; 5: for do 6:

Original: H  se ← M ←
Revised: H H se ← M se

Original:  ⊕ set ; respectively.
Revised: X se H ← H st ⊕ H se ; 9:

Original: σ is the activation function and (cid:107) denotes the concatenation operation. The linear attention does not consider the correlations within neighbors, and thus its computational complexity is cut down from Ω( n 2 ) to Ω( m ) . Note that, it is not necessary to use one-hop neighbors as keys in linear attention.
Revised: Compute L sn L ← main

Original: Some existing works also consider multi-hop neighbors (Sun et al., 2020).
Revised: , Y + sn ; 11:

Original: 3. S TRUCTURAL E NCODING
Revised: 3. S TRUCTURAL E NCODING

Original: The standard dot-product attention can be easily extended on the small graphs.
Revised: The standard dot-product attention can be easily extended on the small graphs.

Original: We add a virtual node v c (Devlin et al., 2019) as the context node connected with all nodes in G .
Revised: We add a virtual node v c (Devlin et al., 2019) as the context node connected with all nodes in G .

Original: Then, the output representation for v c can be regarded as an embedding of G .
Revised: Then, the output representation for v c can be regarded as an embedding of G .

Original: For the large graphs like KGs or networks, we perform self-attention on the local subgraph G i given the node of interest v i .
Revised: For the large graphs like KGs or networks, we perform self-attention on the local subgraph G i given the node of interest v i .

Original: Therefore, the output embedding for node
Revised: Therefore, the output embedding for node

Original: v i is
Revised: v i is

Original: [Equation]
Revised: [Equation]

Original:  where h st i denotes the output of the structural encoder for v i .
Revised: (cid:88) [Equation] [Equation] ) where h st i denotes the output of the structural encoder for v i .

Original: A cj denotes the attention score from the context node c to the neighbor v j .
Revised: A cj denotes the attention score from the context node c to the neighbor v j .

Original: N ( v i ) denotes the set of local neighbors (one-hop neighbors in our implementation) for v i .
Revised: N ( v i ) denotes the set of local neighbors (one-hop neighbors in our implementation) for v i .

Original: Follow the existing works (Chen et al., 2021b;
Revised: Follow the existing works (Chen et al., 2021b;

Original: Ying et al., 2021), we accordingly add the centrality, relation type, or shortest distance path information as special position embeddings to the encoder.
Revised: Ying et al., 2021), we accordingly add the centrality, relation type, or shortest distance path information as special position embeddings to the encoder.

Original: The details can be found in Appendix A.
Revised: The details can be found in Appendix A.

Original: 3. S EMANTIC E NCODING
Revised: 3. S EMANTIC E NCODING

Original: The structural features sometimes are unreliable for identifying a neighbor.
Revised: The structural features sometimes are unreliable for identifying a neighbor.

Original: For example, if the node of interest has a large amount of neighbors, many of them inevitably have similar or identical structural features (e.g., shortest path distance to v i ).
Revised: For example, if the node of interest has a large amount of neighbors, many of them inevitably have similar or identical structural features (e.g., shortest path distance to v i ).

Original: This problem is even more serious when they are all one- or two-hop neighbors.
Revised: This problem is even more serious when they are all one- or two-hop neighbors.

Original: However, if we directly aggregate more-hop nodes, the sheer quantity of available information will overwhelm the neural network.
Revised: However, if we directly aggregate more-hop nodes, the sheer quantity of available information will overwhelm the neural network.

Original: Table 1 summarizes the average frequency of entities appearing as others’ neighbors in different hops.
Revised: Table 1 summarizes the average frequency of entities appearing as others’ neighbors in different hops.

Original: We can ﬁnd that the two- or more-hop neighbors of a node are shared by many others, which is why current GNNs rarely consider multi-hop neighbors.
Revised: We can ﬁnd that the two- or more-hop neighbors of a node are shared by many others, which is why current GNNs rarely consider multi-hop neighbors.

Original: This phenomenon also reveals the over-smoothing problem to some extent.
Revised: This phenomenon also reveals the over-smoothing problem to some extent.

Original: To make the node of interest more distinguishable to the classiﬁer, weighting its one-hop neighbors is usually reasonable due to the less redundancy.
Revised: To make the node of interest more distinguishable to the classiﬁer, weighting its one-hop neighbors is usually reasonable due to the less redundancy.

Original: Therefore, we make the following hypothesis:
Revised: Therefore, we make the following hypothesis:

Original: Hypothesis 1 The one-hop neighbors are the most informative features to identify and represent the node of interest.
Revised: Hypothesis 1 The one-hop neighbors are the most informative features to identify and represent the node of interest.

Original: Recent successes (Rao et al., 2021; Jumper et al., 2021) in biological science demonstrate that using the information provided by the family members can help protein structure prediction.
Revised: Recent successes (Rao et al., 2021; Jumper et al., 2021) in biological science demonstrate that using the information provided by the family members can help protein structure prediction.

Original: Speciﬁcally, they leverage multiple sequence alignment (MSA) (the results of biological sequence alignment) to make use of the information within an evolutionary family.
Revised: Speciﬁcally, they leverage multiple sequence alignment (MSA) (the results of biological sequence alignment) to make use of the information within an evolutionary family.

Original: The protein sequences in a family are assumed to have a common ancestor or an evolutionary relationship.
Revised: The protein sequences in a family are assumed to have a common ancestor or an evolutionary relationship.

Original: Therefore, they may also share some important sub-structures in protein folding.
Revised: Therefore, they may also share some important sub-structures in protein folding.

Original: If we view the embedding of a node as a kind of sequence, then the remote neighbors used in semantic encoding should have similar embeddings to the node of interest.
Revised: If we view the embedding of a node as a kind of sequence, then the remote neighbors used in semantic encoding should have similar embeddings to the node of interest.

Original: We illustrate this idea in Figure 2, where the embeddings are vectors, i.e., ﬁx-length sequences.
Revised: We illustrate this idea in Figure 2, our basic idea is to view node embeddings as protein amino acid sequences with a ﬁxed length.

Original: Like the  MSA results obtained by searching an external genetic database, the semantic neighbors are selected based on their similarity scores to the node of interest.
Revised: Then, we can make use of the insight from the well-known MSA Transformer and AlphaFold 2. To this end, we need to ﬁnd the family members (that share similar evolutional characteristics and relationships) of our “protein”. In their original implementation, this step is done by querying an external gene database. In our case, we propose the semantic encoder to ﬁnd such family members.

Original: Therefore, we make the second hypothesis:
Revised: Therefore, we make the second hypothesis:

Original: Hypothesis 2 The distant nodes that have a high embedding similarity with the node of interest are important features for this node.
Revised: Hypothesis 2 The distant nodes that have a high embedding similarity with the node of interest are important features for this node.

Original: In this paper, we estimate the similarity score by a learnable neural function f s : R h
Revised: In this paper, we estimate the similarity score by a learnable neural function f s : R h

Original: × R h
Revised: × R h

Original: → R :
Revised: → R :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:9) is the semantic difference operator.
Revised: where (cid:9) is the semantic difference operator.

Original: In fact, the choice of (cid:9) is ﬂexible as long as it can reﬂect the similarity between v i and v j .
Revised: In fact, the choice of (cid:9) is ﬂexible as long as it can reﬂect the similarity between v i and v j .

Original: We use the weighted difference as (cid:9) , which can be easily extended to matrix operation.
Revised: We use the weighted difference as (cid:9) , which can be easily extended to matrix operation.

Original: W s ∈ R h × 1 and b s ∈ R are the weight and bias, respectively.
Revised: W s ∈ R h × 1 and b s ∈ R are the weight and bias, respectively.

Original: We use Sigmoid as activation to normalize the difference to (0 , 1) and then convert it to a similarity score.
Revised: We use Sigmoid as activation to normalize the difference to (0 , 1) and then convert it to a similarity score.

Original: Therefore, the output embedding for semantic encoding is written as follows:
Revised: Therefore, the output embedding for semantic encoding is written as follows:

Original: [Equation]
Revised: [Equation]

Original: where B ∈ R n × n records the node-to-node semantic attention scores estimated by f s in Equation (5).
Revised: where B ∈ R n × n records the node-to-node semantic attention scores estimated by f s in Equation (5).

Original: N se ( v i ) is the semantic neighbor set.
Revised: N se ( v i ) is the semantic neighbor set.

Original: In our implementation, it is sampled from the top candidates (top 10% in our setting) during training.
Revised: In our implementation, it is sampled from the top candidates (top 10% in our setting) during training.

Original: We then deﬁne the semantic neighbor fetching loss to learn f s :
Revised: We then deﬁne the semantic neighbor fetching loss to learn f s :

Original: [Equation]
Revised: [Equation]

Original: where N ( v i ) is the positive example set that includes the local neighbors ( Hypothesis 1) of v i , and
Revised: where N ( v i ) is the positive example set that includes the local neighbors ( Hypothesis 1) of v i , and

Original: N − ( v i ) is the negative example set in which the negative examples are randomly-sampled distant nodes (Hypothesis 2) of v i .
Revised: N − ( v i ) is the negative example set in which the negative examples are randomly-sampled distant nodes (Hypothesis 2) of v i .

Original: Therefore, the learning process of f s is self-supervised and can be jointly optimized with the main task loss.
Revised: Therefore, the learning process of f s is self-supervised and can be jointly optimized with the main task loss.

Original: 3. D UAL - ENCODING T RANSFORMER
Revised: 3. D UAL - ENCODING T RANSFORMER

Original: Algorithm We illustrate the implementation of DET in Algorithm 1 and summarize the overall training process as follows:
Revised: Algorithm We illustrate the implementation of DET in Algorithm 1 and summarize the overall training process as follows:

Original: We ﬁrst initialize the input embeddings and all parameters of DET.
Revised: We ﬁrst initialize the input embeddings and all parameters of DET.

Original: For each epoch (or every few epochs), we ﬁrst draw semantic neighbors from the top 10% candidates (according to Equation (5) for each node.
Revised: For each epoch (or every few epochs), we ﬁrst draw semantic neighbors from the top 10% candidates (according to Equation (5) for each node.

Original: In each batch, we feed the structural encoder with the structural neighbors of the input nodes, and the semantic encoder with the semantic neighbors.
Revised: In each batch, we feed the structural encoder with the structural neighbors of the input nodes, and the semantic encoder with the semantic neighbors.

Original: We combine the output embeddings of two encoders by weighted addition and jointly minimize the main task loss and semantic neighbor fetching loss.
Revised: We combine the output embeddings of two encoders by weighted addition and jointly minimize the main task loss and semantic neighbor fetching loss.

Original: Computational Cost We ﬁnd the total training time does not increase too much compared with the baselines.
Revised: Computational Cost We ﬁnd the total training time does not increase too much compared with the baselines.

Original: The design of the semantic operator (cid:9) in the semantic encoder is simpler than the linear attention, which only increases a small number of parameters.
Revised: The design of the semantic operator (cid:9) in the semantic encoder is simpler than the linear attention, which only increases a small number of parameters.

Original: Although fetching the semantic neighbors needs to iterate all nodes (yields a time complexity of n 2 ), we do not compute them on the ﬂy.
Revised: Although fetching the semantic neighbors needs to iterate all nodes (yields a time complexity of n 2 ), we do not compute them on the ﬂy.

Original: Instead, we update semantic neighbors of each node every few epochs, improving both efﬁciency and robustness.
Revised: Instead, we update semantic neighbors of each node every few epochs, improving both efﬁciency and robustness.

Original: Hence, the overall training time remains at the same level (see Appendix B for the detailed statistics).
Revised: Hence, the overall training time remains at the same level (see Appendix B for the detailed statistics).

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conducted experiments on a variety of benchmarks to verify the effectiveness of DET.
Revised: We conducted experiments on a variety of benchmarks to verify the effectiveness of DET.

Original: We uploaded the source code and reported the dataset statistics and parameter settings in Appendix C.
Revised: We uploaded the source code and reported the dataset statistics and parameter settings in Appendix C.

Original: 4. G RAPH P ROPERTY P REDICTION
Revised: 4. G RAPH P ROPERTY P REDICTION

Original: Datasets We evaluated DET on the graph property prediction benchmarks PCQM4M-LSCv

Revised: Datasets We evaluated DET on the graph property prediction benchmarks PCQM4M-LSCv

Original: (Hu et al., 2021) and ZINC (Dwivedi et al., 2020).
Revised: (Hu et al., 2021) and ZINC (Dwivedi et al., 2020).

Original: The former is used in the recent Open Graph
Revised: The former is used in the recent Open Graph

Original: Benchmark Large-Scale Challenge, while the latter is a popular dataset used to evaluate molecular graph representation learning methods.
Revised: Benchmark Large-Scale Challenge, while the latter is a popular dataset used to evaluate molecular graph representation learning methods.

Original: Considering that the number of nodes in each graph is very small (usually less than 50 ), we directly perform attention operations on the whole graph.
Revised: Considering that the number of nodes in each graph is very small (usually less than 50 ), we directly perform attention operations on the whole graph.

Original: Therefore, we removed the semantic neighbor fetching loss in this experiment.
Revised: Therefore, we removed the semantic neighbor fetching loss in this experiment.

Original: Baselines We compared DET with the following state-of-the-art methods: the attention-based methods GAT (Velickovic et al., 2018), GT (Dwivedi & Bresson, 2021) and Graphormer (Ying et al., 2021); and other recently developed GNNs GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017),
Revised: Baselines We compared DET with the following state-of-the-art methods: the attention-based methods GAT (Velickovic et al., 2018), GT (Dwivedi & Bresson, 2021) and Graphormer (Ying et al., 2021); and other recently developed GNNs GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017),

Original: GIN (Xu et al., 2019b), and DeeperGCN (Gilmer et al., 2017).
Revised: GIN (Xu et al., 2019b), and DeeperGCN (Gilmer et al., 2017).

Original: Results Table 2 and Table 3 summarize the experimental results measured by mean average error (MAE) on the two datasets.
Revised: Results Table 2 and Table 3 summarize the experimental results measured by mean average error (MAE) on the two datasets.

Original: Due to the inaccessibility of the testing data on PCQM4M-LSCv1, we alternatively report the MAE results on the training and validation sets.
Revised: Due to the inaccessibility of the testing data on PCQM4M-LSCv1, we alternatively report the MAE results on the training and validation sets.

Original: Overall, DET outperformed all the baseline methods.
Revised: Overall, DET outperformed all the baseline methods.

Original: Compared with Graphormer that only considers encoding structural information with Transformer, DET signiﬁcantly improved the performance, with 6.2% and 7.4% MAE decline on PCQM4M-LSCv1 and ZINC, respectively.
Revised: Compared with Graphormer that only considers encoding structural information with Transformer, DET signiﬁcantly improved the performance, with 6.2% and 7.4% MAE decline on PCQM4M-LSCv1 and ZINC, respectively.

Original: Furthermore, the number of model parameters still maintained the same level to that of baselines.
Revised: Furthermore, the number of model parameters still maintained the same level to that of baselines.

Original: We also observed that DET had more signiﬁcant advantages over other methods on ZINC.
Revised: We also observed that DET had more signiﬁcant advantages over other methods on ZINC.

Original: This is because the restriction of the model size on the ZINC dataset severely limited the capability of encoding structural information.
Revised: This is because the restriction of the model size on the ZINC dataset severely limited the capability of encoding structural information.

Original: Leveraging the semantic information becomes a more efﬁcient choice.
Revised: Leveraging the semantic information becomes a more efﬁcient choice.

Original: If we do not limit the model size of DET, it can easily achieve a MAE of 0 . 006 .
Revised: If we do not limit the model size of DET, it can easily achieve a MAE of 0 . 006 .

Original: 4. N ODE C LASSIFICATION
Revised: 4. N ODE C LASSIFICATION

Original: Datasets We evaluated DET on ﬁve benchmarks that are generally used for node representation learning.
Revised: Datasets We evaluated DET on ﬁve benchmarks that are generally used for node representation learning.

Original: Speciﬁcally, Cora, CiteSeer, and PubMed (Yang et al., 2016) are three citation network datasets commonly used in the transductive setting, while PPI (Zitnik & Leskovec, 2017) is a wellused protein-protein interaction dataset in the inductive setting.
Revised: Speciﬁcally, Cora, CiteSeer, and PubMed (Yang et al., 2016) are three citation network datasets commonly used in the transductive setting, while PPI (Zitnik & Leskovec, 2017) is a wellused protein-protein interaction dataset in the inductive setting.

Original: ogbn-arxiv (Nakata & Shimazaki, 2017) is a large citation network dataset.
Revised: ogbn-arxiv (Nakata & Shimazaki, 2017) is a large citation network dataset.

Original: Most experiment settings follow (Kim & Oh, 2021), and we repeated experiments 100 times on Cora, CiteSeer, and PubMed with random seeds, and
Revised: Most experiment settings follow (Kim & Oh, 2021), and we repeated experiments 100 times on Cora, CiteSeer, and PubMed with random seeds, and

Original: 30 times on PPI and ogbn-arxiv, to produce reliable results and ensure a fair comparison.
Revised: 30 times on PPI and ogbn-arxiv, to produce reliable results and ensure a fair comparison.

Original: Baselines We selected the attention-base methods GAT (Velickovic et al., 2018), CGAT (Wang et al., 2019a), Graph-Bert (Zhang et al., 2020) and SuperGAT SD (Kim & Oh, 2021) as baseline methods.
Revised: Baselines We selected the attention-base methods GAT (Velickovic et al., 2018), CGAT (Wang et al., 2019a), Graph-Bert (Zhang et al., 2020) and SuperGAT SD (Kim & Oh, 2021) as baseline methods.

Original: In addition, other GNN-based methods like GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017) and GCN+NS (Zheng et al., 2020) were also added for comparison.
Revised: In addition, other GNN-based methods like GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017) and GCN+NS (Zheng et al., 2020) were also added for comparison.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results The results are shown in Table 4, from which we can observe that DET outperformed the attention-based methods on most datasets except PPI.
Revised: Results The results are shown in Table 4, from which we can observe that DET outperformed the attention-based methods on most datasets except PPI.

Original: Although SuperGAT SD had better performance on this dataset, we argue that it is no contradiction to incorporate SuperGAT SD as structural encoder into DET to obtain a stronger model.
Revised: Although SuperGAT SD had better performance on this dataset, we argue that it is no contradiction to incorporate SuperGAT SD as structural encoder into DET to obtain a stronger model.

Original: Interestingly, the attention-based methods unanimously performed worse than the GCN-based method
Revised: Interestingly, the attention-based methods unanimously performed worse than the GCN-based method

Original: GCN+NS on CiteSeer.
Revised: GCN+NS on CiteSeer.

Original: SuperGAT SD and CGAT even had the same or worse results compared with the original GAT.
Revised: SuperGAT SD and CGAT even had the same or worse results compared with the original GAT.

Original: Nevertheless, we observed an improvement from GAT to DET.
Revised: Nevertheless, we observed an improvement from GAT to DET.

Original: This result empirically demonstrates the strength of leveraging semantic neighbors.
Revised: This result empirically demonstrates the strength of leveraging semantic neighbors.

Original: 4. KG C OMPLETION
Revised: 4. KG C OMPLETION

Original: Datasets We conducted experiments on the KG completion (a.k.a., entity prediction) task.
Revised: Datasets We conducted experiments on the KG completion (a.k.a., entity prediction) task.

Original: The main target is to predict the subject entity (or object entity) given an incomplete triple.
Revised: The main target is to predict the subject entity (or object entity) given an incomplete triple.

Original: We evaluated
Revised: We evaluated

Original: DET on two benchmark datasets FB15K-

Revised: DET on two benchmark datasets FB15K-

Original: (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2018), which are sampled from the real-world KGs Freebase (Bollacker et al., 2008) and
Revised: (Toutanova & Chen, 2015) and WN18RR (Dettmers et al., 2018), which are sampled from the real-world KGs Freebase (Bollacker et al., 2008) and

Original: WordNet (Miller, 1995), respectively.
Revised: WordNet (Miller, 1995), respectively.

Original: Baselines We chose the best-performing entity prediction methods as our baselines: the TransEfamily methods TransE (Bordes et al., 2013), RotatE (Sun et al., 2019), and TuckER (Balazevic et al., 2019), and the attention-based methods CoKE (Wang et al., 2019b), CompGCN (Vashishth et al., 2020), and HittER (Chen et al., 2021b).
Revised: Baselines We chose the best-performing entity prediction methods as our baselines: the TransEfamily methods TransE (Bordes et al., 2013), RotatE (Sun et al., 2019), and TuckER (Balazevic et al., 2019), and the attention-based methods CoKE (Wang et al., 2019b), CompGCN (Vashishth et al., 2020), and HittER (Chen et al., 2021b).

Original: Speciﬁcally, CoKE and HittER also leverage Transformer to encode the structural information.
Revised: Speciﬁcally, CoKE and HittER also leverage Transformer to encode the structural information.

Original: Results We report the main results on Table 5.
Revised: Results We report the main results on Table 5.

Original: It is clear that DET surpassed all the baselines across all datasets and metrics.
Revised: It is clear that DET surpassed all the baselines across all datasets and metrics.

Original: The improvement on MR (mean rank) is most signiﬁcant, which implies that
Revised: The improvement on MR (mean rank) is most signiﬁcant, which implies that

Original: DET learned better embeddings for all entities, not only for the top ones favored by Hits@1.
Revised: DET learned better embeddings for all entities, not only for the top ones favored by Hits@1.

Original: Overall, DET achieved competitive performance on all three types of tasks, which demonstrates its effectiveness and generality in modeling graphs.
Revised: Overall, DET achieved competitive performance on all three types of tasks, which demonstrates its effectiveness and generality in modeling graphs.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  F URTHER A NALYSIS
Revised:  F URTHER A NALYSIS

Original: To better understand DET, we design three experiments to explore and evaluate DET in depth.
Revised: To better understand DET, we design three experiments to explore and evaluate DET in depth.

Original: 5. I S E VERY M ODULE IN DET U SEFUL ?
Revised: 5. I S E VERY M ODULE IN DET U SEFUL ?

Original: We conducted ablation studies to verify the effectiveness of each module in DET.
Revised: We conducted ablation studies to verify the effectiveness of each module in DET.

Original: We used six datasets in different tasks and present the results in Table 6.
Revised: We used six datasets in different tasks and present the results in Table 6.

Original: We removed the modules from DET step-by-step while keeping identical hyper-parameter settings throughout the experiments.
Revised: We removed the modules from DET step-by-step while keeping identical hyper-parameter settings throughout the experiments.

Original: Semantic Neighbor Fetching The semantic neighbor fetching loss is undoubtedly important to
Revised: Semantic Neighbor Fetching The semantic neighbor fetching loss is undoubtedly important to

Original: DET.
Revised: DET.

Original: No matter if combining two encoders or only using the semantic encoder, integrating with the semantic fetching module had better performance in most cases.
Revised: No matter if combining two encoders or only using the semantic encoder, integrating with the semantic fetching module had better performance in most cases.

Original: The improvement was most notable on PubMed, where it yielded 3 .
Revised: The improvement was most notable on PubMed, where it yielded 3 .

Original: 7% and 3 .
Revised: 7% and 3 .

Original: 8% of accuracy increases, respectively.
Revised: 8% of accuracy increases, respectively.

Original: The mean rank results on WN18RR also got worse without the fetching loss.
Revised: The mean rank results on WN18RR also got worse without the fetching loss.

Original: Semantic Encoder
Revised: Semantic Encoder

Original: If we do not consider the semantic neighbor fetching loss, is the semantic encoder itself still useful to DET?
Revised: If we do not consider the semantic neighbor fetching loss, is the semantic encoder itself still useful to DET?

Original: Unfortunately, we ﬁnd the answer ambiguous.
Revised: Unfortunately, we ﬁnd the answer ambiguous.

Original: For Cora, PubMed, and WN18RR, when we did not employ the fetching loss, DET with the semantic encoder performed much worse than DET without the semantic encoder.
Revised: For Cora, PubMed, and WN18RR, when we did not employ the fetching loss, DET with the semantic encoder performed much worse than DET without the semantic encoder.

Original: But we observe that the situation was reversed on CiteSeer and FB15K-237.
Revised: But we observe that the situation was reversed on CiteSeer and FB15K-237.

Original: In fact, when we only consider one-hop neighbors, the semantic encoder without fetching loss is just a “minus” attention layer.
Revised: In fact, when we only consider one-hop neighbors, the semantic encoder without fetching loss is just a “minus” attention layer.

Original: It may have its pros and cons compared with the standard dot-product attention layer on different datasets.
Revised: It may have its pros and cons compared with the standard dot-product attention layer on different datasets.

Original: In this sense, the semantic fetching loss is who endows the semantic encoder with the characteristic.
Revised: In this sense, the semantic fetching loss is who endows the semantic encoder with the characteristic.

Original: On the ZINC dataset, where the model can apply attention operations on the whole graph, the semantic encoder was capable of estimating the semantic similarity of remote neighbors to the node of interest without the help of the fetching loss.
Revised: On the ZINC dataset, where the model can apply attention operations on the whole graph, the semantic encoder was capable of estimating the semantic similarity of remote neighbors to the node of interest without the help of the fetching loss.

Original: Therefore, we can see that the dual-encoding version of DET greatly outperformed the structural encoder only version.
Revised: Therefore, we can see that the dual-encoding version of DET greatly outperformed the structural encoder only version.

Original: Overall, the effectiveness of the semantic encoder is conditioned: it must get in touch with the remote nodes.
Revised: Overall, the effectiveness of the semantic encoder is conditioned: it must get in touch with the remote nodes.

Original: Structural Encoder
Revised: Structural Encoder

Original: The structural encoder also has merits.
Revised: The structural encoder also has merits.

Original: From the results of the 3 -rd and 5 -th rows in Table 6, we ﬁnd that it had better performance than the semantic encoder on all datasets except CiteSeer.
Revised: From the results of the 3 -rd and 5 -th rows in Table 6, we ﬁnd that it had better performance than the semantic encoder on all datasets except CiteSeer.

Original: We also noticed that only using the semantic encoder had the worst MAE on the
Revised: We also noticed that only using the semantic encoder had the worst MAE on the

Original: ZINC dataset, due to the absence of all structural information.
Revised: ZINC dataset, due to the absence of all structural information.

Original: travel proceed (0.36) precede (0.32) accompany (0.37) walk (0.65) movement (0.47) move (0.64) intercommunicate (0.60) move into (0.58) turn (0.56) change (0.59) (b) WN18RR
Revised: travel proceed (0.36) precede (0.32) accompany (0.37) walk (0.65) movement (0.47) move (0.64) intercommunicate (0.60) move into (0.58) turn (0.56) change (0.59) (b) WN18RR

Original: (a) FB15KNintendo
Revised: (a) FB15KNintendo

Original: Shigeru Miyamoto (0.71)
Revised: Shigeru Miyamoto (0.71)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Ubisoft (0.
Revised: Ubisoft (0.

Original: [Figure]
Revised: [Figure]

Original: Atlus(0.83)
Revised: Atlus(0.83)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ) structural neighbors node of interest semantic neighbors unnormalized semantic score (·
Revised: ) structural neighbors node of interest semantic neighbors unnormalized semantic score (·

Original: [Figure]
Revised: [Figure]

Original: 5. T HE C ORRELATION BETWEEN S EMANTIC E NCODING AND G RAPH H OMOPHILY
Revised: 5. T HE C ORRELATION BETWEEN S EMANTIC E NCODING AND G RAPH H OMOPHILY

Original: In the Introduction section, we mention that the semantic encoding is helpful even on graphs with high homophily.
Revised: In the Introduction section, we mention that the semantic encoding is helpful even on graphs with high homophily.

Original: Therefore, we conducted experiment to verify the correlations between the effectiveness of the semantic encoder and the homophily of datasets.
Revised: Therefore, we conducted experiment to verify the correlations between the effectiveness of the semantic encoder and the homophily of datasets.

Original: We set a hyper-parameter τ to control the combination of the structural encoder and the semantic encoder, which can be written in the following equation:
Revised: We set a hyper-parameter τ to control the combination of the structural encoder and the semantic encoder, which can be written in the following equation:

Original: [Equation]
Revised: [Equation]

Original: where h , h st , h se denote the combined output, structural output, and semantic output, respectively.
Revised: where h , h st , h se denote the combined output, structural output, and semantic output, respectively.

Original: By assigning different τ , we can control the importance of each encoder in the combination.
Revised: By assigning different τ , we can control the importance of each encoder in the combination.

Original: The experimental results are shown in Figure 3, from which we can see that the performance gap for different τ existed in all three datasets, but the trends and peaks were different.
Revised: The experimental results are shown in Figure 3, from which we can see that the performance gap for different τ existed in all three datasets, but the trends and peaks were different.

Original: On highly homophilic dataset Cora, the accuracy ﬁrst increased from τ = 0 to τ = 0 . 15 , and peaked around τ = 0 .
Revised: On highly homophilic dataset Cora, the accuracy ﬁrst increased from τ = 0 to τ = 0 . 15 , and peaked around τ = 0 .

Original: 2 , then dropped until τ = 95 .
Revised: 2 , then dropped until τ = 95 .

Original: On CoraFull and Chameleon with median and low homophily, we observe that the performance peaked around [0 , 0 . 05] and maintained steady in the interval [0 . 05 , 0 . 2] .
Revised: On CoraFull and Chameleon with median and low homophily, we observe that the performance peaked around [0 , 0 . 05] and maintained steady in the interval [0 . 05 , 0 . 2] .

Original: When τ ≥ 0 .
Revised: When τ ≥ 0 .

Original: 5 , the performance rapidly dropped to minima.
Revised: 5 , the performance rapidly dropped to minima.

Original: Therefore, we may make the following conclusions: (1) the semantic encoder has better effects when the datasets have low homophily.
Revised: Therefore, we may make the following conclusions: (1) the semantic encoder has better effects when the datasets have low homophily.

Original: The larger proportion the semantic encoding occupied, the better performance the model achieved; (2) even on the highly homophilic dataset, engaging the semantic encoder with a proper τ has signiﬁcant advantages over using only structural encoder; (3) in any cases, properly combining the output of two encoders (i.e., DET) is the best choice.
Revised: The larger proportion the semantic encoding occupied, the better performance the model achieved; (2) even on the highly homophilic dataset, engaging the semantic encoder with a proper τ has signiﬁcant advantages over using only structural encoder; (3) in any cases, properly combining the output of two encoders (i.e., DET) is the best choice.

Original: 5. H OW DOES THE S EMANTIC E NCODER H ELP THE S TRUCTURAL E NCODER ?
Revised: 5. H OW DOES THE S EMANTIC E NCODER H ELP THE S TRUCTURAL E NCODER ?

Original: It is worth exploring how the semantic encoder affects the structural encoder.
Revised: It is worth exploring how the semantic encoder affects the structural encoder.

Original: In Figure 4, we illustrate two examples on FB15K-237 and WN18RR, respectively.
Revised: In Figure 4, we illustrate two examples on FB15K-237 and WN18RR, respectively.

Original: At ﬁrst, we ﬁnd that the semantic scores for the structural neighbors are also in line with human intuition.
Revised: At ﬁrst, we ﬁnd that the semantic scores for the structural neighbors are also in line with human intuition.

Original: For example, in the left ﬁgure, the entity
Revised: For example, in the left ﬁgure, the entity

Original: USA has a very low score although it is directly connected to Nintendo by relation service_location .
Revised: USA has a very low score although it is directly connected to Nintendo by relation service_location .

Original: The verb precede and accompany obtain relatively low scores in the right ﬁgure.
Revised: The verb precede and accompany obtain relatively low scores in the right ﬁgure.

Original: These neighbors are not very related to the entities of interest from the human perspective.
Revised: These neighbors are not very related to the entities of interest from the human perspective.

Original: By contrast, some one-hop neighbors get high semantic scores, e.g., the well-known director Shigeru Miyamo of Nintendo in
Revised: By contrast, some one-hop neighbors get high semantic scores, e.g., the well-known director Shigeru Miyamo of Nintendo in

Original: FB15K-237 and the verb walk in WN18.
Revised: FB15K-237 and the verb walk in WN18.

Original: They are the more informative entities.
Revised: They are the more informative entities.

Original: For the semantic neighbors, we can see that the exploited remote neighbors are closely related to the entity of interest, as well as the structural neighbors with high semantic scores.
Revised: For the semantic neighbors, we can see that the exploited remote neighbors are closely related to the entity of interest, as well as the structural neighbors with high semantic scores.

Original: For example, Atlus is an important game developer to Nintendo .
Revised: For example, Atlus is an important game developer to Nintendo .

Original: Aggregating such information may be helpful when the model is asked to predict the games related to Nintendo .
Revised: Aggregating such information may be helpful when the model is asked to predict the games related to Nintendo .

Original: For the verb travel in WN18RR, move also shares many key features with it.
Revised: For the verb travel in WN18RR, move also shares many key features with it.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we propose a new Transformer architecture DET to deal with graphs of different types and sizes.
Revised: In this paper, we propose a new Transformer architecture DET to deal with graphs of different types and sizes.

Original: In DET, the structural encoder aggregates information from local nodes while the semantic encoder seeks the remote nodes with useful semantics.
Revised: In DET, the structural encoder aggregates information from local nodes while the semantic encoder seeks the remote nodes with useful semantics.

Original: The experimental results demonstrate the strong performance of DET on three prevalent GNN tasks across 9 benchmarks.
Revised: The experimental results demonstrate the strong performance of DET on three prevalent GNN tasks across 9 benchmarks.

Original: We hope DET can bring more insights and inspirations in developing uniﬁed Transformer architectures.
Revised: We hope DET can bring more insights and inspirations in developing uniﬁed Transformer architectures.

Original: In future, we plan to adapt DET to the NLP and CV areas.
Revised: In future, we plan to adapt DET to the NLP and CV areas.

Original: A P OSITION E MBEDDING
Revised: A P OSITION E MBEDDING

Original: There are many important graph features that can be used to identify different nodes.
Revised: There are many important graph features that can be used to identify different nodes.

Original: Thanks to learnable position embedding, these discrete features now can be encoding into embeddings and combined with the raw nodes embedding vectors.
Revised: Thanks to learnable position embedding, these discrete features now can be encoding into embeddings and combined with the raw nodes embedding vectors.

Original: Speciﬁcally, for graph property prediction task, we use the method proposed by Graphormer (Ying et al., 2021) to encode degree centrality of an arbitrary node v i as follows:
Revised: Speciﬁcally, for graph property prediction task, we use the method proposed by Graphormer (Ying et al., 2021) to encode degree centrality of an arbitrary node v i as follows:

Original: [Equation]
Revised: [Equation]

Original: where deg ( v i ) denotes the degree of the node v i , and f c : R → R h is the mapping function that converts the node degree to a learnable embedding.
Revised: where deg ( v i ) denotes the degree of the node v i , and f c : R → R h is the mapping function that converts the node degree to a learnable embedding.

Original: We also consider encoding the distances from the node of interest v i to different neighbors by the following equation:
Revised: We also consider encoding the distances from the node of interest v i to different neighbors by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where spd ( v i , v j ) denotes the shortest path distance from v i to v j , and f d : R → R h is a similar function that converts the distance to a learnable embedding.
Revised: where spd ( v i , v j ) denotes the shortest path distance from v i to v j , and f d : R → R h is a similar function that converts the distance to a learnable embedding.

Original: For KG representation learning task, we follow (Chen et al., 2021b) to encode the edge types into node embeddings, which is implemented by an additional atom triple Transformer M A .
Revised: For KG representation learning task, we follow (Chen et al., 2021b) to encode the edge types into node embeddings, which is implemented by an additional atom triple Transformer M A .

Original: Speciﬁcally, for a given triple ( v i , r ij , v j ) for the node of interest v i , where r ij denote the edge type (i.e., relationship) between v i and v j .
Revised: Speciﬁcally, for a given triple ( v i , r ij , v j ) for the node of interest v i , where r ij denote the edge type (i.e., relationship) between v i and v j .

Original: The edge type information can be encoded by the following equation:
Revised: The edge type information can be encoded by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where [ c A , v i , r ij , v j ] is the input embedding sequence.
Revised: where [ c A , v i , r ij , v j ] is the input embedding sequence.

Original: c A is the virtual node for the atom Transformer, whose output represents the edge encoding embedding.
Revised: c A is the virtual node for the atom Transformer, whose output represents the edge encoding embedding.

Original: B C OMPUTATIONAL C OST
Revised: B C OMPUTATIONAL C OST

Original: We used one 32GB V100 GPU to train all the methods for estimation and show the average training time in Table 7.
Revised: We used one 32GB V100 GPU to train all the methods for estimation and show the average training time in Table 7.

Original: Clearly, incorporating the semantic encoder did not signiﬁcantly increase the computational cost, especially on the node classiﬁcation datasets.
Revised: Clearly, incorporating the semantic encoder did not signiﬁcantly increase the computational cost, especially on the node classiﬁcation datasets.

Original: C E XPERIMENT D ETAILS C. D ATASET S ETTINGS
Revised: C E XPERIMENT D ETAILS C. D ATASET S ETTINGS

Original: We present the overall dataset statistics in Table 8.
Revised: We present the overall dataset statistics in Table 8.

Original: PCQM4M-LSCv
Revised: PCQM4M-LSCv

Original: ZINC
Revised: ZINC

Original: Cora
Revised: Cora

Original: CiteSeer
Revised: CiteSeer

Original: PubMed
Revised: PubMed

Original: PPI ogbn-arxiv
Revised: PPI ogbn-arxiv

Original: FB15KWN18RR
Revised: FB15KWN18RR

Original: Learning-rate 0 .0 .0 .0 .0 .0 .0 .0 .0 .
Revised: Learning-rate 0 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: 008 Batch-size        

Revised: 008 Batch-size

Original: # Layer        6 # Multi-heads
Revised: # Layer        6 # Multi-heads

Original:        8 τ increasing increasing
Revised:        8 τ increasing increasing

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: # Semantic neighbors - -      

Revised: # Semantic neighbors - -

Original: # Negative samples - -       0.
Revised: # Negative samples - -       0.

Original: V a li d M A
Revised: V a li d M A

Original: E
Revised: E

Original: Step
Revised: Step

Original: Structural + Semantic
Revised: Structural + Semantic

Original: Strctural (
Revised: Strctural (

Original: [Figure]
Revised: [Figure]

Original: V a li d M
Revised: V a li d M

Original: A E
Revised: A E

Original: Step
Revised: Step

Original: Structural + Semantic
Revised: Structural + Semantic

Original: Strctural
Revised: Strctural

Original: [Figure]
Revised: [Figure]

Original: Graph Property Prediction For PCQM4M-LSCv1, the model is asked to predict the DFT (density functional theory)-calculated HOMO-LUMO energy gap of given molecules.
Revised: Graph Property Prediction For PCQM4M-LSCv1, the model is asked to predict the DFT (density functional theory)-calculated HOMO-LUMO energy gap of given molecules.

Original: It contains more than 3.8M 2D molecular graphs as input, which is especially appropriate to evaluate the performance of model in large scale scenarios.
Revised: It contains more than 3.8M 2D molecular graphs as input, which is especially appropriate to evaluate the performance of model in large scale scenarios.

Original: On other hand, ZINC is a relative small datasets, where the main target is to predict the graph property regression for constrained solubility.
Revised: On other hand, ZINC is a relative small datasets, where the main target is to predict the graph property regression for constrained solubility.

Original: It is one of the most popular real-world molecular datasets for graph representation learning.
Revised: It is one of the most popular real-world molecular datasets for graph representation learning.

Original: Node Classﬁcation Cora, CiteSeer and PubMed are three citation network datasets proposed by (Yang et al., 2016).
Revised: Node Classﬁcation Cora, CiteSeer and PubMed are three citation network datasets proposed by (Yang et al., 2016).

Original: They are typically used for transductive node classiﬁcation task.
Revised: They are typically used for transductive node classiﬁcation task.

Original: PPI on the other hand is used for inductive evaluation.
Revised: PPI on the other hand is used for inductive evaluation.

Original: It consists of 24 graphs, with 20 graphs for training, and 2 for validation and 2 for testing.
Revised: It consists of 24 graphs, with 20 graphs for training, and 2 for validation and 2 for testing.

Original: All four datasets are the prevalent benchmarks used for node classiﬁcation.
Revised: All four datasets are the prevalent benchmarks used for node classiﬁcation.

Original: KG Completion FB15K-237 is the revised version of the original FB15K dataset (Bordes et al., 2013) that was used as entity prediction benchmark in last ten years.
Revised: KG Completion FB15K-237 is the revised version of the original FB15K dataset (Bordes et al., 2013) that was used as entity prediction benchmark in last ten years.

Original: However, recent studies (Dettmers et al., 2018; Toutanova & Chen, 2015) ﬁnd that the original FB15K contains a large proportion of redundant data, some of which may incur testing data leakage, the same to another well-used dataset WN18.
Revised: However, recent studies (Dettmers et al., 2018; Toutanova & Chen, 2015) ﬁnd that the original FB15K contains a large proportion of redundant data, some of which may incur testing data leakage, the same to another well-used dataset WN18.

Original: Therefore, most latest studies only use the revised datasets FB15Kand WN18RR for evaluation.
Revised: Therefore, most latest studies only use the revised datasets FB15Kand WN18RR for evaluation.

Original: FB15K-237 has more different relationships, while WN18RR is more sparse and has more different entities.
Revised: FB15K-237 has more different relationships, while WN18RR is more sparse and has more different entities.

Original: C. P ARAMETER S ETTINGS
Revised: C. P ARAMETER S ETTINGS

Original: We summarize the main hyper-parameter settings on different datasets in Table 9.
Revised: We summarize the main hyper-parameter settings on different datasets in Table 9.

Original: The more speciﬁc settings can be found in the source code.
Revised: The more speciﬁc settings can be found in the source code.

Original: For the graph property prediction tasks, we directly let the semantic encoder access all neighbors, and thus the semantic neighbor fetching loss is not used.
Revised: For the graph property prediction tasks, we directly let the semantic encoder access all neighbors, and thus the semantic neighbor fetching loss is not used.

Original: Alternatively, we adapt an increasing strategy to gradually improve the weight of the output of semantic encoder during training.
Revised: Alternatively, we adapt an increasing strategy to gradually improve the weight of the output of semantic encoder during training.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D A DDITIONAL E XPERIMENTS D. A NALYSIS ON THE T RAINING P ROCESS
Revised: D A DDITIONAL E XPERIMENTS D. A NALYSIS ON THE T RAINING P ROCESS

Original: We conducted experiments to compare the performance of structural encoder with and without the semantic encoder, in term of training steps.
Revised: We conducted experiments to compare the performance of structural encoder with and without the semantic encoder, in term of training steps.

Original: We depict the valid MAE results on ZINC in Figure 5a.
Revised: We depict the valid MAE results on ZINC in Figure 5a.

Original: At the beginning of the training [Table] we ﬁnd that the two methods do not have a visible performance gap.
Revised: At the beginning of the training, we ﬁnd that the two methods do not have a visible performance gap.

Original: The curves are tightly overlapped during step 0 to 74 , 000 .
Revised: The curves are tightly overlapped during step 0 to 74 , 000 .

Original: As the performance starts to be converged, i.e., step 74 , 000 to 148 , 000 , only using structural encoder is better than combining the two encoders.
Revised: As the performance starts to be converged, i.e., step 74 , 000 to 148 , 000 , only using structural encoder is better than combining the two encoders.

Original: However, as the valid MAE tends to be stable, the dual-encoding DET gradually outperforms the single structural encoder.
Revised: However, as the valid MAE tends to be stable, the dual-encoding DET gradually outperforms the single structural encoder.

Original: It is worth-noting that the turning point appears at the performance starts to be converged, where the input embeddings also approach the ideal positions.
Revised: It is worth-noting that the turning point appears at the performance starts to be converged, where the input embeddings also approach the ideal positions.

Original: Therefore, the semantic similarity among embeddings can be estimated more precisely, and contribute to a better semantic encoder.
Revised: Therefore, the semantic similarity among embeddings can be estimated more precisely, and contribute to a better semantic encoder.

Original: Therefore,
Revised: Therefore,

Original: DET can obtain a lower MAE than the single strctural encoder.
Revised: DET can obtain a lower MAE than the single strctural encoder.

Original: We present the result on PCQM4MLSCv1 in Figure 5b, which supports the same conclusion.
Revised: We present the result on PCQM4MLSCv1 in Figure 5b, which supports the same conclusion.

Original: D. R ESULTS ON O THER N ODE C LASSIFICATION D ATASETS
Revised: D. R ESULTS ON O THER N ODE C LASSIFICATION D ATASETS

Original: We also provide the results on additional 8 real-world datasets for node classiﬁcation.
Revised: We also provide the results on additional 8 real-world datasets for node classiﬁcation.

Original: They are: CS and Physics (Shchur et al., 2018); Cora-ML, Cora-Full and DBLP (Bojchevski & Günnemann, 2018);
Revised: They are: CS and Physics (Shchur et al., 2018); Cora-ML, Cora-Full and DBLP (Bojchevski & Günnemann, 2018);

Original: Chameleon (Rozemberczki et al., 2019); Four-Univ (Craven et al., 1998); and Wiki-CS (Mernyei &
Revised: Chameleon (Rozemberczki et al., 2019); Four-Univ (Craven et al., 1998); and Wiki-CS (Mernyei &

Original: Cangea, 2020).
Revised: Cangea, 2020).

Original: We report the accuracy on Table 10.
Revised: We report the accuracy on Table 10.

Original: It is clear that DET consistently and signiﬁcantly outperformed the attention-based methods on these datasets.
Revised: It is clear that DET consistently and signiﬁcantly outperformed the attention-based methods on these datasets.

Revised: D. C HOICES OF f s

Revised: In this section, we conducted experiments to verify the effectiveness of proposed f s , in comparison with a linear attention implementation.

Revised: The results are shown in Table 11.we can observe that our proposed attention outperformed linear attention, and the results of using linear attention were similar to those of SuperGAT and the original GAT.

Revised: SuperGAT also leverages a self-supervised contrastive loss to predict the existence of edges in the original graph.

Revised: If we use linear attention to replace our proposed attention, our method is more like a SuperGAT with two separated linear attentions to cope with node classiﬁcation and edge prediction, respectively.