Original: G RAPH T RANSFORMER
Revised: G RAPH T RANSFORMER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNN) have gained increasing research interests as a mean to the challenging goal of robust and universal graph learning.
Revised: Graph neural networks (GNN) have gained increasing research interests as a mean to the challenging goal of robust and universal graph learning.

Original: Previous GNNs have assumed single pre-ﬁxed graph structure and permitted only local context encoding.
Revised: Previous GNNs have assumed single pre-ﬁxed graph structure and permitted only local context encoding.

Original: This paper proposes a novel Graph Transformer (GTR) architecture that captures long-range dependency with global attention, and enables dynamic graph structures.
Revised: This paper proposes a novel Graph Transformer (GTR) architecture that captures long-range dependency with global attention, and enables dynamic graph structures.

Original: In particular, GTR propagates features within the same graph structure via an intra-graph message passing , and transforms dynamic semantics across multi-domain graph-structured data (e.g. images, sequences, knowledge graphs) for multi-modal learning via an inter-graph message passing .
Revised: In particular, GTR propagates features within the same graph structure via an intra-graph message passing , and transforms dynamic semantics across multi-domain graph-structured data (e.g. images, sequences, knowledge graphs) for multi-modal learning via an inter-graph message passing .

Original: Furthermore, GTR enables effective incorporation of any prior graph structure by weighted averaging of the prior and learned edges, which can be crucially useful for scenarios where prior knowledge is desired.
Revised: Furthermore, GTR enables effective incorporation of any prior graph structure by weighted averaging of the prior and learned edges, which can be crucially useful for scenarios where prior knowledge is desired.

Original: The proposed GTR achieves new state-of-the-arts across three benchmark tasks, including few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.
Revised: The proposed GTR achieves new state-of-the-arts across three benchmark tasks, including few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.

Original: Experiments show that GTR is superior in learning robust graph representations, transforming high-level semantics across domains, and bridging between prior graph structure with automatic structure learning.
Revised: Experiments show that GTR is superior in learning robust graph representations, transforming high-level semantics across domains, and bridging between prior graph structure with automatic structure learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is an increasing research interest in graph neural networks (GNN).
Revised: There is an increasing research interest in graph neural networks (GNN).

Original: Compared to traditional convolutional networks that are usually limited to grid-structured data, and recurrent networks that are designed for sequence modeling, GNN has demonstrated superiority on modeling dynamic and complex graph-structured data.
Revised: Compared to traditional convolutional networks that are usually limited to grid-structured data, and recurrent networks that are designed for sequence modeling, GNN has demonstrated superiority on modeling dynamic and complex graph-structured data.

Original: On one hand, as a mainstream category of graph neural networks, spectral approaches such as graph convolutional neural networks (GCN) (Kipf & Welling, 2017;
Revised: On one hand, as a mainstream category of graph neural networks, spectral approaches such as graph convolutional neural networks (GCN) (Kipf & Welling, 2017;

Original: Defferrard et al., 2016) generalize traditional convolutional networks in the spectral domain, and employ local receptive ﬁelds for robust graph learning.
Revised: Defferrard et al., 2016) generalize traditional convolutional networks in the spectral domain, and employ local receptive ﬁelds for robust graph learning.

Original: However, these approaches are restricted to ﬁxed graph structure, and require costly matrix computation (Zhang et al., 2018b) such as eigen decomposition.
Revised: However, these approaches are restricted to ﬁxed graph structure, and require costly matrix computation (Zhang et al., 2018b) such as eigen decomposition.

Original: The signiﬁcant shortcoming of GCNs, namely their lack of global context for updating nodes state which prevents long-range relation modeling, as well as the equal importance of nodes during aggregation, also limit its wide application.
Revised: The signiﬁcant shortcoming of GCNs, namely their lack of global context for updating nodes state which prevents long-range relation modeling, as well as the equal importance of nodes during aggregation, also limit its wide application.

Original: On the other hand, GNNs equipped with attention mechanism, such as graph attention networks (GAT) (Velickovic et al., 2018), have emerged as an effective alternative to GCNs by alleviating these issues.
Revised: On the other hand, GNNs equipped with attention mechanism, such as graph attention networks (GAT) (Velickovic et al., 2018), have emerged as an effective alternative to GCNs by alleviating these issues.

Original: GAT incorporates self-attention mechanism (Vaswani et al., 2017) which selectively attends to connected graph nodes according to their importance to the considered node.
Revised: GAT incorporates self-attention mechanism (Vaswani et al., 2017) which selectively attends to connected graph nodes according to their importance to the considered node.

Original: Despite these advantages, GATs, the same as many existing graph neural network approaches (Niepert et al., 2016; Zhang et al., 2018b), are still limited to evolution within the same graph structure, and lack of the capability of transforming representations across different graph structures, or various data domains (e.g. images, sequences, knowledge graphs, and 3D meshes).
Revised: Despite these advantages, GATs, the same as many existing graph neural network approaches (Niepert et al., 2016; Zhang et al., 2018b), are still limited to evolution within the same graph structure, and lack of the capability of transforming representations across different graph structures, or various data domains (e.g. images, sequences, knowledge graphs, and 3D meshes).

Original: The recently proposed Transformer (Vaswani et al., 2017) architecture for sequence modeling has been proven useful for tasks such as machine translation (Bahdanau et al., 2017).
Revised: The recently proposed Transformer (Vaswani et al., 2017) architecture for sequence modeling has been proven useful for tasks such as machine translation (Bahdanau et al., 2017).

Original: By addressing the inherent sequential computation shortcoming of recurrent neural networks, Transformer enables efﬁcient and paralleled computation by invoking a self-attention mechanism for global context modeling.
Revised: By addressing the inherent sequential computation shortcoming of recurrent neural networks, Transformer enables efﬁcient and paralleled computation by invoking a self-attention mechanism for global context modeling.

Original: Transformer also avoids vanishing gradients problem (Hochreiter et al., 2001), and automatically approximates relations among a dynamic range of symbols, which is crucial for learning real-world data of arbitrary structures.
Revised: Transformer also avoids vanishing gradients problem (Hochreiter et al., 2001), and automatically approximates relations among a dynamic range of symbols, which is crucial for learning real-world data of arbitrary structures.

Original: e 

Revised: e

Original: e 51 ee 21 eee’ 81 e’ 71 e’e’e’ 41 e’ 31 e’e’’ ’ ’ ’ ’ ’ ’ ’
Revised: e 51 ee 21 eee’ 81 e’ 71 e’e’e’ 41 e’ 31 e’e’’ ’ ’ ’ ’ ’ ’ ’

Original: Source Attention
Revised: Source Attention

Original: Self Attention Source graph
Revised: Self Attention Source graph

Original: Target graph
Revised: Target graph

Original: Self Attention
Revised: Self Attention

Original: Source graph
Revised: Source graph

Original: Target graph
Revised: Target graph

Original: Updated target graph
Revised: Updated target graph

Original: Self Attention …...
Revised: Self Attention …...

Original: Self Attention
Revised: Self Attention

Original: Source Attention
Revised: Source Attention

Original: Source Attention …...
Revised: Source Attention …...

Original: Source Attention hhhhhhhhh 4 hhh 6 h 7 h
Revised: Source Attention hhhhhhhhh 4 hhh 6 h 7 h

Original: Inspired by this work, we proposed a novel Graph Transformer (GTR) architecture which is a straightforward yet effective extension of the sequential Transformer architecture to a variety of complex graph-structured data, such as knowledge graphs, images, and sequences.
Revised: Inspired by this work, we proposed a novel Graph Transformer (GTR) architecture which is a straightforward yet effective extension of the sequential Transformer architecture to a variety of complex graph-structured data, such as knowledge graphs, images, and sequences.

Original: The goal of GTR is to transform a source graph to a target graph of possibly different structures by considering global context and realizing computation parallelism across all graph nodes.
Revised: The goal of GTR is to transform a source graph to a target graph of possibly different structures by considering global context and realizing computation parallelism across all graph nodes.

Original: An illustration is provided in Figure 1.
Revised: An illustration is provided in Figure 1.

Original: GTR invokes an intra-graph message passing paradigm to progressively reﬁne graph representations within the same graph structure, and an inter-graph message passing paradigm to attentively distill related semantics from source graphs for target graph learning.
Revised: GTR invokes an intra-graph message passing paradigm to progressively reﬁne graph representations within the same graph structure, and an inter-graph message passing paradigm to attentively distill related semantics from source graphs for target graph learning.

Original: By alternatively propagating information either within the same graph structure or from the source graph to the target graph, GTR is able to extract key information embedded in the source graph, and reﬁne target graph representations by considering both local and global context.
Revised: By alternatively propagating information either within the same graph structure or from the source graph to the target graph, GTR is able to extract key information embedded in the source graph, and reﬁne target graph representations by considering both local and global context.

Original: Our contributions are three-fold: 1)
Revised: Our contributions are three-fold: 1)

Original: We propose a novel Graph Transformer (GTR) which not only learns graph representations of the same structure, but also across different structures.
Revised: We propose a novel Graph Transformer (GTR) which not only learns graph representations of the same structure, but also across different structures.

Original: We propose formulations of converting various real-world data types (e.g. images, sequences, knowledge graphs) to graph-structured, and the application of GTR on converting semantics across these various types of data.
Revised: We propose formulations of converting various real-world data types (e.g. images, sequences, knowledge graphs) to graph-structured, and the application of GTR on converting semantics across these various types of data.

Original: We propose an effective way of incorporating prior graph structure with learning-based structure, and thus reconciles traditional knowledge-based and modern learningbased approaches in the down-stream tasks (e.g. few-shot learning, medical abnormality and disease classiﬁcation).
Revised: We propose an effective way of incorporating prior graph structure with learning-based structure, and thus reconciles traditional knowledge-based and modern learningbased approaches in the down-stream tasks (e.g. few-shot learning, medical abnormality and disease classiﬁcation).

Original: We conduct experiments on three benchmark tasks: few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.
Revised: We conduct experiments on three benchmark tasks: few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.

Original: Experiments show that GTR achieves new state-ofthe-arts on all tasks, demonstrating its superior capability in learning robust graph representations, propagating information across multi-domain data types, and effective utilization of prior knowledge.
Revised: Experiments show that GTR achieves new state-ofthe-arts on all tasks, demonstrating its superior capability in learning robust graph representations, propagating information across multi-domain data types, and effective utilization of prior knowledge.

Original: Speciﬁcally, on 1-shot learning on miniImageNet dataset, GTR improves the state-of-the-art by 1.32% by incorporating category relations in novel class weight generation.
Revised: Speciﬁcally, on 1-shot learning on miniImageNet dataset, GTR improves the state-of-the-art by 1.32% by incorporating category relations in novel class weight generation.

Original: GTR also improves abnormality classiﬁcation accuracy by more than 7% on all evaluated data sets, demonstrating its effectiveness on knowledge graph representation learning and transformation.
Revised: GTR also improves abnormality classiﬁcation accuracy by more than 7% on all evaluated data sets, demonstrating its effectiveness on knowledge graph representation learning and transformation.

Original: And, on graph classiﬁcation task, GTR outperforms all compared standard graph neural networks and kernel-based networks with roughly the same computational cost, demonstrating its efﬁciency.
Revised: And, on graph classiﬁcation task, GTR outperforms all compared standard graph neural networks and kernel-based networks with roughly the same computational cost, demonstrating its efﬁciency.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Graph neural networks (GNN) have attracted increasing research interests (Defferrard et al., 2016;
Revised: Graph neural networks (GNN) have attracted increasing research interests (Defferrard et al., 2016;

Original: Kipf & Welling, 2017; Monti et al., 2017) and provided ﬂexible representation learning of real-world graph-structured data.
Revised: Kipf & Welling, 2017; Monti et al., 2017) and provided ﬂexible representation learning of real-world graph-structured data.

Original: There has been work that study message passing in graph learning (Do et al., 2018; Gilmer et al., 2017; Schlichtkrull et al., 2018), and equip GNNs with attention mechanisms as which is shown powerful for sequence and image modeling (Zhang et al., 2018a; Vaswani et al., 2017; Al-Sabahi et al., 2018).
Revised: There has been work that study message passing in graph learning (Do et al., 2018; Gilmer et al., 2017; Schlichtkrull et al., 2018), and equip GNNs with attention mechanisms as which is shown powerful for sequence and image modeling (Zhang et al., 2018a; Vaswani et al., 2017; Al-Sabahi et al., 2018).

Original: However, most existing methods learn to encode the input feature into higher-level feature through selective attention over the object itself (Wang et al., 2018a; Velickovic et al., 2018), while our method works on multiple graphs, and models not only the data structure within the same graph but also the transformation rules among different graphs.
Revised: However, most existing methods learn to encode the input feature into higher-level feature through selective attention over the object itself (Wang et al., 2018a; Velickovic et al., 2018), while our method works on multiple graphs, and models not only the data structure within the same graph but also the transformation rules among different graphs.

Original: Among all related works, Transformer (Vaswani et al., 2017) can be formulated as a particular instance of Graph Transformer where input and output are both sequences, and no prior graph structure is provided.
Revised: Among all related works, Transformer (Vaswani et al., 2017) can be formulated as a particular instance of Graph Transformer where input and output are both sequences, and no prior graph structure is provided.

Original: Besides, GTR is close to graph attention networks (GAT) (Velickovic et al., 2018) in that they both employ attention mechanism for learning importance-differentiated relations among graph nodes.
Revised: Besides, GTR is close to graph attention networks (GAT) (Velickovic et al., 2018) in that they both employ attention mechanism for learning importance-differentiated relations among graph nodes.

Original: However, GTR differs from GAT in several aspects.
Revised: However, GTR differs from GAT in several aspects.

Original: First, GTR attends to all graph nodes at every graph update disregard of whether two nodes are directly connected or not, while
Revised: First, GTR attends to all graph nodes at every graph update disregard of whether two nodes are directly connected or not, while

Original: GAT only attends to directed connected neighboring nodes for a considered node by a proposed masked attention .
Revised: GAT only attends to directed connected neighboring nodes for a considered node by a proposed masked attention .

Original: Thus, GAT is still limited to differentiating importance among locally connected nodes.
Revised: Thus, GAT is still limited to differentiating importance among locally connected nodes.

Original: GTR, on the other hand, is able to capture global context as which is signiﬁcant for modeling long-range relations and fasten graph learning by allowing information propagation between implicitly connected nodes.
Revised: GTR, on the other hand, is able to capture global context as which is signiﬁcant for modeling long-range relations and fasten graph learning by allowing information propagation between implicitly connected nodes.

Original: Second, GTR incorporates both prior graph structure and graph structure learning where using prior edge weights allows the efﬁcient utilization of prior knowledge.
Revised: Second, GTR incorporates both prior graph structure and graph structure learning where using prior edge weights allows the efﬁcient utilization of prior knowledge.

Original: Third,
Revised: Third,

Original: GTR is able to transform representations across various graph structures while GAT is restricted to the same graph propagation.
Revised: GTR is able to transform representations across various graph structures while GAT is restricted to the same graph propagation.

Original:  G RAPH T RANSFORMER
Revised:  G RAPH T RANSFORMER

Original: Graph Transformer (GTR) transforms a source graph to a target graph by encoding features of the same graph structure into higher-level semantics, and fusing latent semantics between different graphs (e.g. from visual features to knowledge graphs).
Revised: Graph Transformer (GTR) transforms a source graph to a target graph by encoding features of the same graph structure into higher-level semantics, and fusing latent semantics between different graphs (e.g. from visual features to knowledge graphs).

Original: We represent a graph as G =
Revised: We represent a graph as G =

Original: ( V, E ) where
Revised: ( V, E ) where

Original: V = { v i } i =1: N is a set of nodes with each v i ∈ R d representing a node’s feature of dimension d , and N is the number of nodes in the graph.
Revised: V = { v i } i =1: N is a set of nodes with each v i ∈ R d representing a node’s feature of dimension d , and N is the number of nodes in the graph.

Original: E = { e i,j } i,j =[1 ,N ] represents a set of edges between any possible pair of nodes.
Revised: E = { e i,j } i,j =[1 ,N ] represents a set of edges between any possible pair of nodes.

Original: Here we consider the setting where each edge is associated with a scalar value indicating closeness between nodes, while it is straightforward to extend the formalism to other cases where edges are represented as non-scalar values such as vectors.
Revised: Here we consider the setting where each edge is associated with a scalar value indicating closeness between nodes, while it is straightforward to extend the formalism to other cases where edges are represented as non-scalar values such as vectors.

Original: GTR takes a source graph G =
Revised: GTR takes a source graph G =

Original: ( V, E ) as input, and transforms it to a target graph G (cid:48) =
Revised: ( V, E ) as input, and transforms it to a target graph G (cid:48) =

Original: ( V (cid:48) , E (cid:48) ) , where G and G (cid:48) are two different graphs and can have different structures and characteristics (e.g.
Revised: ( V (cid:48) , E (cid:48) ) , where G and G (cid:48) are two different graphs and can have different structures and characteristics (e.g.

Original: N (cid:54) = N (cid:48) , d (cid:54) =
Revised: N (cid:54) = N (cid:48) , d (cid:54) =

Original: d (cid:48) , and e i,j (cid:54) =
Revised: d (cid:48) , and e i,j (cid:54) =

Original: e (cid:48)
Revised: e (cid:48)

Original: i,j ).
Revised: i,j ).

Original: This property differs from many previous methods (Defferrard et al., 2016; Kipf & Welling, 2017; Velickovic et al., 2018) which are restricted to the same graph structures.
Revised: This property differs from many previous methods (Defferrard et al., 2016; Kipf & Welling, 2017; Velickovic et al., 2018) which are restricted to the same graph structures.

Original: For both source and target graph, the set of nodes V and V (cid:48) has to be given in prior, such as the vocabulary size when the considered graph is sequences, and abnormality nodes if the considered graph is an abnormality graph.
Revised: For both source and target graph, the set of nodes V and V (cid:48) has to be given in prior, such as the vocabulary size when the considered graph is sequences, and abnormality nodes if the considered graph is an abnormality graph.

Original: We consider two scenarios for the edges among graph nodes: 1) the edges are given in prior, and denoted as e s i ,t j where s i is the i th node of source graph and t j is the j th node of target graph; 2) the edges are unknown, and thus source and target nodes are represented as fully connected with uniform weights.
Revised: We consider two scenarios for the edges among graph nodes: 1) the edges are given in prior, and denoted as e s i ,t j where s i is the i th node of source graph and t j is the j th node of target graph; 2) the edges are unknown, and thus source and target nodes are represented as fully connected with uniform weights.

Original: We assume e s i ,t j as normalized, to avoid notation of averaging.
Revised: We assume e s i ,t j as normalized, to avoid notation of averaging.

Original: Two types of message passing are considered in GTR: the one from source graph to target graph denoted as inter-graph message passing , and another within the same graph denoted as intra-graph message passing .
Revised: Two types of message passing are considered in GTR: the one from source graph to target graph denoted as inter-graph message passing , and another within the same graph denoted as intra-graph message passing .

Original: 3. I NTER - GRAPH MESSAGE PASSING
Revised: 3. I NTER - GRAPH MESSAGE PASSING

Original: To distill relevant information from a source graph, the features of source nodes are transformed and passed to target nodes with their corresponding edge weights, which can be formulated as:
Revised: To distill relevant information from a source graph, the features of source nodes are transformed and passed to target nodes with their corresponding edge weights, which can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: (1) where σ is a nonlinear activation, and W s is a projection matrix of size d (cid:48)
Revised: (1) where σ is a nonlinear activation, and W s is a projection matrix of size d (cid:48)

Original: × d .
Revised: × d .

Original: Considering that the edge information between source and target graphs may not be available in many cases such as translating a sequence of words into another sequence of words, we propose to learn edge weights automatically by an attention mechanism (Vaswani et al., 2017).
Revised: Considering that the edge information between source and target graphs may not be available in many cases such as translating a sequence of words into another sequence of words, we propose to learn edge weights automatically by an attention mechanism (Vaswani et al., 2017).

Original: By doing so, target node update is enabled to consider the varying importance of the source nodes in global context.
Revised: By doing so, target node update is enabled to consider the varying importance of the source nodes in global context.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: where ˆ e s i ,t j is the attention weight of edge from source node i to target node j ; W as and W at are weights in attention mechanism to project nodes features of source graph and target graph to a common space of dimension q respectively; and Attention : R q −→ R is the attention mechanism that transforms the two projected features W as v i , W at v (cid:48)
Revised: where ˆ e s i ,t j is the attention weight of edge from source node i to target node j ; W as and W at are weights in attention mechanism to project nodes features of source graph and target graph to a common space of dimension q respectively; and Attention : R q −→ R is the attention mechanism that transforms the two projected features W as v i , W at v (cid:48)

Original: j ∈ R q to a scalar ˆ e s i ,t j as the edge’s attention weight.
Revised: j ∈ R q to a scalar ˆ e s i ,t j as the edge’s attention weight.

Original: In our experiments, Attention is parameterized as a scaled dot-product operation with multihead attention (Vaswani et al., 2017).
Revised: In our experiments, Attention is parameterized as a scaled dot-product operation with multihead attention (Vaswani et al., 2017).

Original: The attention weights are normalized across all source nodes for each target node, representing the relative importance of each source node to a target node among all source nodes.
Revised: The attention weights are normalized across all source nodes for each target node, representing the relative importance of each source node to a target node among all source nodes.

Original: The formulation can be written as:
Revised: The formulation can be written as:

Original: [Equation]
Revised: [Equation]

Original: Once obtained, the normalized attention coefﬁcients are be combined with prior edge weights in order to pass features of connected source nodes to target nodes.
Revised: Once obtained, the normalized attention coefﬁcients are be combined with prior edge weights in order to pass features of connected source nodes to target nodes.

Original: The combined features serve as the target node’s updated features with source graph knowledge encoded.
Revised: The combined features serve as the target node’s updated features with source graph knowledge encoded.

Original: We adopt weighted sum of the learned attention edge weights and prior edge weights as ﬁnal edge weights.
Revised: We adopt weighted sum of the learned attention edge weights and prior edge weights as ﬁnal edge weights.

Original: Other methods such as multiplication of the learned and prior edge weights followed by softmax also works.
Revised: Other methods such as multiplication of the learned and prior edge weights followed by softmax also works.

Original: However, in our experiments, we observed that the ﬁrst method performs better and avoids under-ﬁtting.
Revised: However, in our experiments, we observed that the ﬁrst method performs better and avoids under-ﬁtting.

Original: The formulation can be written as:
Revised: The formulation can be written as:

Original: [Equation]
Revised: [Equation]

Original: where λ is a user-deﬁned weight controlling importance of prior edges and learned edges.
Revised: where λ is a user-deﬁned weight controlling importance of prior edges and learned edges.

Original: If λ is set to 1, the edges between source graph and target graph are ﬁxed, and no attention machanism is required.
Revised: If λ is set to 1, the edges between source graph and target graph are ﬁxed, and no attention machanism is required.

Original: The formulation is then the same as Equation 1.
Revised: The formulation is then the same as Equation 1.

Original: If λ is set to 0 , the edges between source graph and target graph are completely learned by the model.
Revised: If λ is set to 0 , the edges between source graph and target graph are completely learned by the model.

Original: With the updated weight, one can obtain updated target nodes features via Equation 1.
Revised: With the updated weight, one can obtain updated target nodes features via Equation 1.

Original: 3. I NTRA - GRAPH MESSAGE PASSING
Revised: 3. I NTRA - GRAPH MESSAGE PASSING

Original: Intra-graph message passing aims at modeling the correlation among nodes of the same graph, and fusing features according to the closeness between them.
Revised: Intra-graph message passing aims at modeling the correlation among nodes of the same graph, and fusing features according to the closeness between them.

Original: Speciﬁcally, a target node is updated by combining features of neighboring nodes and itself.
Revised: Speciﬁcally, a target node is updated by combining features of neighboring nodes and itself.

Original: The formulation can be written as:
Revised: The formulation can be written as:

Original: [Equation]
Revised: [Equation]

Original: where W t is weight to project features of target nodes from dimension d to output dimension.
Revised: where W t is weight to project features of target nodes from dimension d to output dimension.

Original: To learn the edge weights through attention mechanism, one can directly apply Equations 1-4 by changing source and target nodes notation to be of the same graph.
Revised: To learn the edge weights through attention mechanism, one can directly apply Equations 1-4 by changing source and target nodes notation to be of the same graph.

Original: 3. GTR AS A MODULE / BUILDING BLOCK
Revised: 3. GTR AS A MODULE / BUILDING BLOCK

Original: As shown in Figure 1, we formulate GTR as a module denoted as GTR by ﬁrst concatenating intragraph message passing and inter-graph message passing into one step (that is, ﬁrst conduct message passing within the target graph, then conducting message passing from a source graph), then stacking multiple such steps into one module in order to progressively convert target graph features into high-level semantics.
Revised: As shown in Figure 1, we formulate GTR as a module denoted as GTR by ﬁrst concatenating intragraph message passing and inter-graph message passing into one step (that is, ﬁrst conduct message passing within the target graph, then conducting message passing from a source graph), then stacking multiple such steps into one module in order to progressively convert target graph features into high-level semantics.

Original: To denote different variants of GTR for different input or output domains, we use s to denote sequence, i to denote images, and g to denote knowledge graphs, and represent
Revised: To denote different variants of GTR for different input or output domains, we use s to denote sequence, i to denote images, and g to denote knowledge graphs, and represent

Original: GTR for image input and knowledge graph output as GTR i 2 g , GTR for knowledge graph input and knowledge graph output as GTR g 2 g and the rest possible combinations in the same manner.
Revised: GTR for image input and knowledge graph output as GTR i 2 g , GTR for knowledge graph input and knowledge graph output as GTR g 2 g and the rest possible combinations in the same manner.

Original: 3. GTR FOR M ULTIPLE D OMAINS
Revised: 3. GTR FOR M ULTIPLE D OMAINS

Original: Most real-world data types (e.g. images, sequences, graphs) can be formulated as graph-structured.
Revised: Most real-world data types (e.g. images, sequences, graphs) can be formulated as graph-structured.

Original: For example, a 2-dimensional image can be formulated as a graph whose nodes are pixels of the image where every node is connected with its neighboring pixel nodes; and a sequence of words can be formulated as a graph whose nodes are the individual words where edges among nodes are the consecutive relation among words.
Revised: For example, a 2-dimensional image can be formulated as a graph whose nodes are pixels of the image where every node is connected with its neighboring pixel nodes; and a sequence of words can be formulated as a graph whose nodes are the individual words where edges among nodes are the consecutive relation among words.

Original: If the global context of the data is considered, which is commonly adopted in attention mechanism (Vaswani et al., 2017), the graph nodes are then fullyconnected.
Revised: If the global context of the data is considered, which is commonly adopted in attention mechanism (Vaswani et al., 2017), the graph nodes are then fullyconnected.

Original: In the following, we describe the variants of GTR for different data domains by ﬁrst formulating data as graph-structured, and then perform GTR operations on it.
Revised: In the following, we describe the variants of GTR for different data domains by ﬁrst formulating data as graph-structured, and then perform GTR operations on it.

Original: [Figure]
Revised: [Figure]

Original: GTR for sequential input/output.
Revised: GTR for sequential input/output.

Original: To apply GTR for sequential input or output (e.g. a sequence of words, a sequence of retrieved items), we employ positional encoding (Vaswani et al., 2017) to GTR so as to add relative and absolute position information to the input or output sequence.
Revised: To apply GTR for sequential input or output (e.g. a sequence of words, a sequence of retrieved items), we employ positional encoding (Vaswani et al., 2017) to GTR so as to add relative and absolute position information to the input or output sequence.

Original: Speciﬁcally, we use sine and cosine functions of different frequencies:
Revised: Speciﬁcally, we use sine and cosine functions of different frequencies:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where pos is the position and i is the dimension.
Revised: where pos is the position and i is the dimension.

Original: If both input and output are sequences, GTR is close to a Transformer (Vaswani et al., 2017), however, with additional prior edge weights.
Revised: If both input and output are sequences, GTR is close to a Transformer (Vaswani et al., 2017), however, with additional prior edge weights.

Original: GTR for image input.
Revised: GTR for image input.

Original: We denote visual features of an image as I ∈ R D,W,H where D is the dimension of latent features, W and H is width and height.
Revised: We denote visual features of an image as I ∈ R D,W,H where D is the dimension of latent features, W and H is width and height.

Original: To apply GTR for image input, we ﬁrst reshape visual features by ﬂattening the 2-dimension into 1-dimension R W × H,D .
Revised: To apply GTR for image input, we ﬁrst reshape visual features by ﬂattening the 2-dimension into 1-dimension R W × H,D .

Original: Then each pixel is treated as graph node whose features are used as graph node features.
Revised: Then each pixel is treated as graph node whose features are used as graph node features.

Original:  E XPERIMENTS 4. F EW - SHOT LEARNING
Revised:  E XPERIMENTS 4. F EW - SHOT LEARNING

Original: We evaluate GTR for few-shot learning on miniImageNet dataset (Vinyals et al., 2016) which consists of 100 ImageNet classes with each with 600 samples.
Revised: We evaluate GTR for few-shot learning on miniImageNet dataset (Vinyals et al., 2016) which consists of 100 ImageNet classes with each with 600 samples.

Original: We follow the same category split proposed by Ravi et al (Ravi & Larochelle, 2017) with 64, 16, 20 for training, validation and testing respectively.
Revised: We follow the same category split proposed by Ravi et al (Ravi & Larochelle, 2017) with 64, 16, 20 for training, validation and testing respectively.

Original: An N-way K-shot learning is deﬁned as classifying N categories with each only giving
Revised: An N-way K-shot learning is deﬁned as classifying N categories with each only giving

Original: K training examples.
Revised: K training examples.

Original: We conduct 5 way 1-shot and 5-shot classiﬁcation and develop our model based on the framework proposed by Gidaris & Komodakis (2018).
Revised: We conduct 5 way 1-shot and 5-shot classiﬁcation and develop our model based on the framework proposed by Gidaris & Komodakis (2018).

Original: First, the few-shot learning comprises two stages: base category classiﬁcation, and episodic learning.
Revised: First, the few-shot learning comprises two stages: base category classiﬁcation, and episodic learning.

Original: During base category classiﬁcation learning, the images of the base categories are sampled and used for training a set of prototypes with each corresponding to a base category.
Revised: During base category classiﬁcation learning, the images of the base categories are sampled and used for training a set of prototypes with each corresponding to a base category.

Original: The forward procedure includes feature extraction via a feature extractor, and a classiﬁcation operation such as dot product and cosine similarity to classify images.
Revised: The forward procedure includes feature extraction via a feature extractor, and a classiﬁcation operation such as dot product and cosine similarity to classify images.

Original: During episodic learning, speciﬁcally testing, N categories are ﬁrst sampled from novel categories, each with K images as examples.
Revised: During episodic learning, speciﬁcally testing, N categories are ﬁrst sampled from novel categories, each with K images as examples.

Original: The K images are ﬁrst fed to feature extractor to extract features, then used as initialization of novel class prototypes.
Revised: The K images are ﬁrst fed to feature extractor to extract features, then used as initialization of novel class prototypes.

Original: The initialized novel class prototypes are further used by an attention block proposed by Gidaris & Komodakis (2018) to selectively attends to base class prototypes in order to learn novel prototypes without forgetting base category prototypes.
Revised: The initialized novel class prototypes are further used by an attention block proposed by Gidaris & Komodakis (2018) to selectively attends to base class prototypes in order to learn novel prototypes without forgetting base category prototypes.

Original: Gidaris & Komodakis (2018) updates novel category prototypes by selectively attending to available base category prototypes, and modeling the data-dependent relations among the base and novel
Revised: Gidaris & Komodakis (2018) updates novel category prototypes by selectively attending to available base category prototypes, and modeling the data-dependent relations among the base and novel

Original: [Table]
Revised: [Table]

Original: ﬁxed during episodic training.
Revised: ﬁxed during episodic training.

Original: The weight on prior edges λ (in Equation 4) is chosen by validation (see Table 6).
Revised: The weight on prior edges λ (in Equation 4) is chosen by validation (see Table 6).

Original: categories in each episodic learning.
Revised: categories in each episodic learning.

Original: However, machine learning models are often fooled by visual illusions.
Revised: However, machine learning models are often fooled by visual illusions.

Original: For example, images of different animals with the same color and similar posture taken in the same environment may have higher probability of being classiﬁed to the same class than images of the same animal postured differently and located in the different environment.
Revised: For example, images of different animals with the same color and similar posture taken in the same environment may have higher probability of being classiﬁed to the same class than images of the same animal postured differently and located in the different environment.

Original: When training samples are few and visual features may not be a reliable source, having the prior knowledge of how similar categories are can help improve visual recognition drastically.
Revised: When training samples are few and visual features may not be a reliable source, having the prior knowledge of how similar categories are can help improve visual recognition drastically.

Original: To achieve this, we propose to replace the novel category weight generation process in Gidaris & Komodakis (2018) with our GTR incorporated with a prior category relation graph.
Revised: To achieve this, we propose to replace the novel category weight generation process in Gidaris & Komodakis (2018) with our GTR incorporated with a prior category relation graph.

Original: The architecture is shown in Figure 2.
Revised: The architecture is shown in Figure 2.

Original: During an episodic learning, we treat the base and novel categories as graph nodes, and replace the attention mechanism in the weight generator in Gidaris & Komodakis (2018) with GTR.
Revised: During an episodic learning, we treat the base and novel categories as graph nodes, and replace the attention mechanism in the weight generator in Gidaris & Komodakis (2018) with GTR.

Original: We deﬁne the input source graph of GTR as the base categories whose features are their prototypes, and the target graph as novel categories whose features are initialized by the visual features of the K examples as initial target graph.
Revised: We deﬁne the input source graph of GTR as the base categories whose features are their prototypes, and the target graph as novel categories whose features are initialized by the visual features of the K examples as initial target graph.

Original: GTR evolves the target graph by sequentially attending to source graph and itself.
Revised: GTR evolves the target graph by sequentially attending to source graph and itself.

Original: We further deﬁne the edges among different graph nodes as the similarity between the category words computed by path similarity () API of wordnet (Miller, 1995) which returns the shortest path between two nodes in the is-a taxonomy.
Revised: We further deﬁne the edges among different graph nodes as the similarity between the category words computed by path similarity () API of wordnet (Miller, 1995) which returns the shortest path between two nodes in the is-a taxonomy.

Original: To encourage sparsity, we further prune edges with smaller than 0.1 scores, and normalize the remaining out-going edges per node.
Revised: To encourage sparsity, we further prune edges with smaller than 0.1 scores, and normalize the remaining out-going edges per node.

Original: The similarity scores are used as prior edge weights, and are weighted averaged with the learned edges weights during training and inference.
Revised: The similarity scores are used as prior edge weights, and are weighted averaged with the learned edges weights during training and inference.

Original: We denote the importance weight of prior edges as λ (see Equation 4), and experiment different values for λ ∈ { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } .
Revised: We denote the importance weight of prior edges as λ (see Equation 4), and experiment different values for λ ∈ { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } .

Original: The GTR in our experiment haslayers and 6 heads, and is trained with 0.1 dropout, and maximum 60 epochs.
Revised: The GTR in our experiment haslayers and 6 heads, and is trained with 0.1 dropout, and maximum 60 epochs.

Original: For base category classiﬁcation, we use 0.1 initial learning rate, and decrease it to 0.06, 0.012, 0.0024 at epoch 20, 40, 50 respectively.
Revised: For base category classiﬁcation, we use 0.1 initial learning rate, and decrease it to 0.06, 0.012, 0.0024 at epoch 20, 40, 50 respectively.

Original: For episodic training, we use 0.01 learning rate.
Revised: For episodic training, we use 0.01 learning rate.

Original: We use ConvNet128 as proposed in Gidaris & Komodakis (2018) for extracting visual features of dimension 128 × 5 × 5 .
Revised: We use ConvNet128 as proposed in Gidaris & Komodakis (2018) for extracting visual features of dimension 128 × 5 × 5 .

Original: Results.
Revised: Results.

Original: We compare to a range of state-of-the-art methods including that use convolutional networks and deep residual networks as shown in Table 1 along with the results.
Revised: We compare to a range of state-of-the-art methods including that use convolutional networks and deep residual networks as shown in Table 1 along with the results.

Original: GTR achieves a new state-of-the-art result on 1-shot learning, and has improved on 5-shot learning compared to Gidaris & Komodakis (2018).
Revised: GTR achieves a new state-of-the-art result on 1-shot learning, and has improved on 5-shot learning compared to Gidaris & Komodakis (2018).

Original: Furthermore, GTR obtains larger improvements on 1-shot than 5-shot, demonstrating that the incorporation of prior category relations takes larger effect when less training samples are given, and thus prior knowledge from textual semantic domains helps the model learn novel visual patterns more effectively and accurately.
Revised: Furthermore, GTR obtains larger improvements on 1-shot than 5-shot, demonstrating that the incorporation of prior category relations takes larger effect when less training samples are given, and thus prior knowledge from textual semantic domains helps the model learn novel visual patterns more effectively and accurately.

Original: 4. M EDICAL ABNORMALITY & DISEASE CLASSIFICATION
Revised: 4. M EDICAL ABNORMALITY & DISEASE CLASSIFICATION

Original: To demonstrate the capability of GTR on transforming graph features between graph-structured data of different domains such as from image features to knowledge graph, and from knowledge graph to another knowledge graph, we conduct the experiment on medical abnormality and disease clas-
Revised: To demonstrate the capability of GTR on transforming graph features between graph-structured data of different domains such as from image features to knowledge graph, and from knowledge graph to another knowledge graph, we conduct the experiment on medical abnormality and disease clas-

Original: [Figure]
Revised: [Figure]

Original: siﬁcation.
Revised: siﬁcation.

Original: Speciﬁcally, we formulate the abnormality and disease categories as graph nodes where the edges among nodes represent their correlation (e.g. co-occurrence frequency).
Revised: Speciﬁcally, we formulate the abnormality and disease categories as graph nodes where the edges among nodes represent their correlation (e.g. co-occurrence frequency).

Original: The learning of abnormality and disease classiﬁcation is thus learning abnormality and disease graph features whose semantics represent clinical conditions that could lead to the diagnosis of abnormalities or diseases.
Revised: The learning of abnormality and disease classiﬁcation is thus learning abnormality and disease graph features whose semantics represent clinical conditions that could lead to the diagnosis of abnormalities or diseases.

Original: In detail, we ﬁrst compile an off-the-shelf abnormality graph that contains frequent abnormalities stem from thoracic organs.
Revised: In detail, we ﬁrst compile an off-the-shelf abnormality graph that contains frequent abnormalities stem from thoracic organs.

Original: For example, ”disappearance of costophrenic angle”, ”low lung volumes”, and ”blunted costophrenic angle”.
Revised: For example, ”disappearance of costophrenic angle”, ”low lung volumes”, and ”blunted costophrenic angle”.

Original: Additionally, we design a disease graph containing common thorax diseases (e.g. nodule, pneumonia and emphysema) which are commonly concluded from the single or combined condition of abnormalities.
Revised: Additionally, we design a disease graph containing common thorax diseases (e.g. nodule, pneumonia and emphysema) which are commonly concluded from the single or combined condition of abnormalities.

Original: For example, atelectasis may be concluded if ”interval development of bandlike opacity in the left lung base” is present;
Revised: For example, atelectasis may be concluded if ”interval development of bandlike opacity in the left lung base” is present;

Original: As described in Figure 3, a set of images are ﬁrst fed into a CNN for extracting visual features which are then transformed into an abnormality graph via GTR
Revised: As described in Figure 3, a set of images are ﬁrst fed into a CNN for extracting visual features which are then transformed into an abnormality graph via GTR

Original: i 2 g .
Revised: i 2 g .

Original: A GTR g 2 g module then converts the abnormality graph into a disease graph.
Revised: A GTR g 2 g module then converts the abnormality graph into a disease graph.

Original: The node features of abnormality graph and disease graph are passed to separate classiﬁcation layers for predicting abnormality and disease categories respectively.
Revised: The node features of abnormality graph and disease graph are passed to separate classiﬁcation layers for predicting abnormality and disease categories respectively.

Original: We conduct experiments on two medical image datasets.
Revised: We conduct experiments on two medical image datasets.

Original: First, Indiana University Chest X-Ray
Revised: First, Indiana University Chest X-Ray

Original: Collection (IU X-Ray) (Demner-Fushman et al., 2015) is a public dataset consisting of 7,470 chest x-ray images paired with their corresponding diagnostic reports.
Revised: Collection (IU X-Ray) (Demner-Fushman et al., 2015) is a public dataset consisting of 7,470 chest x-ray images paired with their corresponding diagnostic reports.

Original: Each patient has a frontal view and a lateral view image, a report, and corresponding MeSH (Frey & George, 2007) tags.
Revised: Each patient has a frontal view and a lateral view image, a report, and corresponding MeSH (Frey & George, 2007) tags.

Original: We text-mined 80 most frequent abnormality labels and 14 disease labels from reports and MeSH tags respectively.
Revised: We text-mined 80 most frequent abnormality labels and 14 disease labels from reports and MeSH tags respectively.

Original: Examples of abnormality and disease labels are shown in Figure 3 and Table 3.
Revised: Examples of abnormality and disease labels are shown in Figure 3 and Table 3.

Original: CX-CHR is a private dataset of chest X-ray images collected from a professional medical institution.
Revised: CX-CHR is a private dataset of chest X-ray images collected from a professional medical institution.

Original: The dataset consists of 35,609 patients and 45,598 images.
Revised: The dataset consists of 35,609 patients and 45,598 images.

Original: Each patient has one or multiple chest x-ray images in different views (e.g. frontal and lateral), and a corresponding Chinese report.
Revised: Each patient has one or multiple chest x-ray images in different views (e.g. frontal and lateral), and a corresponding Chinese report.

Original: We select patients with no more than 2 images and obtain 33,236 patient samples in total which covers over 93% of the dataset.
Revised: We select patients with no more than 2 images and obtain 33,236 patient samples in total which covers over 93% of the dataset.

Original: We text-mine most popular abnormalities and diseases mentioned in reports, which yields 155 abnormalities and 14 most popular thorax diseases.
Revised: We text-mine most popular abnormalities and diseases mentioned in reports, which yields 155 abnormalities and 14 most popular thorax diseases.

Original: On both datasets, we randomly split the data by patients into training, validation and testing by a ratio of 7:1:2.
Revised: On both datasets, we randomly split the data by patients into training, validation and testing by a ratio of 7:1:2.

Original: There is no overlap between patients in different sets.
Revised: There is no overlap between patients in different sets.

Original: For abnormality classiﬁcation, we compare our GTR with a DenseNet (Huang et al., 2017) on classifying the same set of abnormality labels.
Revised: For abnormality classiﬁcation, we compare our GTR with a DenseNet (Huang et al., 2017) on classifying the same set of abnormality labels.

Original: For ablation study, we compare our method by solely training on abnormality classiﬁcation (GTR-1graph), and jointly training on both abnormality and disease classiﬁcation (GTR-2graphs).
Revised: For ablation study, we compare our method by solely training on abnormality classiﬁcation (GTR-1graph), and jointly training on both abnormality and disease classiﬁcation (GTR-2graphs).

Original: For disease classiﬁcation, we compare our method with DenseNet on CX-CHR dataset, and TieNet (Wang et al., 2018b) on IU X-Ray dataset.
Revised: For disease classiﬁcation, we compare our method with DenseNet on CX-CHR dataset, and TieNet (Wang et al., 2018b) on IU X-Ray dataset.

Original: We use learning rate 1 e − 3 for training and 1 e − 5 for ﬁne-tuning, and reduce by 10 times when encountering validation performance plateau.
Revised: We use learning rate 1 e − 3 for training and 1 e − 5 for ﬁne-tuning, and reduce by 10 times when encountering validation performance plateau.

Original: Results.
Revised: Results.

Original: The area under the curve (AUC) of abnormality classiﬁcation is shown in Table 2, and
Revised: The area under the curve (AUC) of abnormality classiﬁcation is shown in Table 2, and

Original: AUC of disease classiﬁcation is shown in Table 3.
Revised: AUC of disease classiﬁcation is shown in Table 3.

Original: On both abnormality and disease classiﬁcation,
Revised: On both abnormality and disease classiﬁcation,

Original: GTR achieves the best results on both dataset, demonstrating its superior capability of learning and transforming graph features.
Revised: GTR achieves the best results on both dataset, demonstrating its superior capability of learning and transforming graph features.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 4. G RAPH CLASSIFICATION
Revised: 4. G RAPH CLASSIFICATION

Original: To probe the ability of Graph Transformer on transforming same graph features from low-level semantics to high-level semantics [Table] we conduct the experiment on two benchmark protein datasets for graph classiﬁcation: PROTEINS (Borgwardt et al., 2005; Feragen et al., 2013) and D&D (Dobson & Doig, 2003).
Revised: To probe the ability of Graph Transformer on transforming same graph features from low-level semantics to high-level semantics, we conduct the experiment on two benchmark protein datasets for graph classiﬁcation: PROTEINS (Borgwardt et al., 2005; Feragen et al., 2013) and D&D (Dobson & Doig, 2003).

Original: PROTEINS contains 1113 graphs and 43472 nodes in total.
Revised: PROTEINS contains 1113 graphs and 43472 nodes in total.

Original: Each graph has nodes representing secondary structure elements (SSEs), and edges representing neighboring relations in the amino acid sequence or in 3D space (Borgwardt et al., 2005).
Revised: Each graph has nodes representing secondary structure elements (SSEs), and edges representing neighboring relations in the amino acid sequence or in 3D space (Borgwardt et al., 2005).

Original: The maximum number of nodes in a graph is 620, the average number of nodes is 39.06, and the average number of edges is 72.8.
Revised: The maximum number of nodes in a graph is 620, the average number of nodes is 39.06, and the average number of edges is 72.8.

Original: D&D contains 1178 protein graphs and 334926 nodes in total.
Revised: D&D contains 1178 protein graphs and 334926 nodes in total.

Original: The graph nodes representing amino acids are connected if they are less than 6 Angstroms apart (Dobson & Doig, 2003).
Revised: The graph nodes representing amino acids are connected if they are less than 6 Angstroms apart (Dobson & Doig, 2003).

Original: The maximum number of nodes in a graph is 5748, the average number of nodes is 284.32.
Revised: The maximum number of nodes in a graph is 5748, the average number of nodes is 284.32.

Original: We use 1e-4 learning rate and train maximum 20 epochs on one GPU.
Revised: We use 1e-4 learning rate and train maximum 20 epochs on one GPU.

Original: The graph nodes of both data sets are ﬁrst re-organized as its unique BFS ordering before feeding into the model.
Revised: The graph nodes of both data sets are ﬁrst re-organized as its unique BFS ordering before feeding into the model.

Original: We use 3 layers, 6 heads and 0.5 dropout in GTR.
Revised: We use 3 layers, 6 heads and 0.5 dropout in GTR.

Original: We pad or trim number of nodes to 100 and 500 during training and inference for PROTEINS and D&D dataset respectively.
Revised: We pad or trim number of nodes to 100 and 500 during training and inference for PROTEINS and D&D dataset respectively.

Original: We take mean of all nodes features after
Revised: We take mean of all nodes features after

Original: GTR and feed the averaged features to a linear fully-connected layer for prediction graph labels.
Revised: GTR and feed the averaged features to a linear fully-connected layer for prediction graph labels.

Original: Results.
Revised: Results.

Original: We compare with 4 kernel-based methods and 6 deep learning-based methods as shown in
Revised: We compare with 6 kernel-based methods and 6 deep learning-based methods as shown in

Original: Table 4.
Revised: Table 4.

Original: We report averaged classiﬁcation accuracy of 10-fold cross-validation in Table 4.
Revised: We report averaged classiﬁcation accuracy of 10-fold cross-validation in Table 4.

Original: Compared with kernel-based and graph neural network-based (GNN-based) baselines, our proposed method
Revised: Compared with kernel-based and graph neural network-based (GNN-based) baselines, our proposed method

Original: GTR achieves the best graph classiﬁcation performance on both datasets, demonstrating the capability of GTR on classifying large-scale and dynamic graphs with simple architecture and avoidance of sophisticated kernels or network design.
Revised: GTR achieves the best graph classiﬁcation performance on both datasets, demonstrating the capability of GTR on classifying large-scale and dynamic graphs with simple architecture and avoidance of sophisticated kernels or network design.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper introduces the Graph Transformer that generalizes the Transformer model to multidomain graph-structured data types, and enables the incorporation of prior knowledge.
Revised: This paper introduces the Graph Transformer that generalizes the Transformer model to multidomain graph-structured data types, and enables the incorporation of prior knowledge.

Original: The proposed Graph Transformer achieves state-of-the-art results on three challenging benchmark tasks, namely few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.
Revised: The proposed Graph Transformer achieves state-of-the-art results on three challenging benchmark tasks, namely few-shot learning, medical abnormality and disease classiﬁcation, and graph classiﬁcation.

Original: Graph Transformer reconciles traditional knowledge-based approaches with modern learning-based approaches, and demonstrates its signiﬁcant contributions to many tasks such as few-shot learning, and abnormality and disease classiﬁcation.
Revised: Graph Transformer reconciles traditional knowledge-based approaches with modern learning-based approaches, and demonstrates its signiﬁcant contributions to many tasks such as few-shot learning, and abnormality and disease classiﬁcation.

Original: Appendices A R ELATED W ORK
Revised: Appendices A R ELATED W ORK

Original: Our work is also close to Graph Attentional Multi-Label Learning (GAML)
Revised: Our work is also close to Graph Attentional Multi-Label Learning (GAML)

Original: (Do et al., 2018) which studies graph representation learning via attention mechanism.
Revised: (Do et al., 2018) which studies graph representation learning via attention mechanism.

Original: However, our work differs in that (1) GAML is proposed speciﬁcally for multi-label classiﬁcation while GTR can not only be formulated for multi-label classiﬁcation, but also for graph-level classiﬁcation and few-shot classiﬁcation which is shown in our three diverse experiments.
Revised: However, our work differs in that (1) GAML is proposed speciﬁcally for multi-label classiﬁcation while GTR can not only be formulated for multi-label classiﬁcation, but also for graph-level classiﬁcation and few-shot classiﬁcation which is shown in our three diverse experiments.

Original: Furthermore, GTR uses different mechanism for multi-labeling by expanding extra classiﬁcation layers to each graph node while GAML treats all class labels are auxiliary nodes which are modeled in parallel with input graph nodes.
Revised: Furthermore, GTR uses different mechanism for multi-labeling by expanding extra classiﬁcation layers to each graph node while GAML treats all class labels are auxiliary nodes which are modeled in parallel with input graph nodes.

Original: (2) GAML does not actually study transformation from one graph to another but only intra-graph message passing by formulating all label nodes and input nodes into one uniﬁed graph.
Revised: (2) GAML does not actually study transformation from one graph to another but only intra-graph message passing by formulating all label nodes and input nodes into one uniﬁed graph.

Original: However, GTR formulates input nodes and label nodes as separate graphs, and studies both intra- and inter-graph message passing.
Revised: However, GTR formulates input nodes and label nodes as separate graphs, and studies both intra- and inter-graph message passing.

Original: This formulation is structure-agnostic and allows substantial difference between input and label graphs including graph size, edges types, and graph node feature dimension.
Revised: This formulation is structure-agnostic and allows substantial difference between input and label graphs including graph size, edges types, and graph node feature dimension.

Original: GTR uses different mechanism for multi-labeling by expanding extra classiﬁcation layers to each graph node while GAML treats all class labels are auxiliary nodes which are modeled in parallel with input graph nodes.
Revised: GTR uses different mechanism for multi-labeling by expanding extra classiﬁcation layers to each graph node while GAML treats all class labels are auxiliary nodes which are modeled in parallel with input graph nodes.

Original: (4) GTR also can be easily extended to existing deep learning models such as DenseNet (Huang et al., 2017) and ResNet (He et al., 2016) without signiﬁcant change to the deep network so as to explicitly and effectively uncover label relations and feature-label relations.
Revised: (4) GTR also can be easily extended to existing deep learning models such as DenseNet (Huang et al., 2017) and ResNet (He et al., 2016) without signiﬁcant change to the deep network so as to explicitly and effectively uncover label relations and feature-label relations.

Original: On the contrary, GAML requires extra conﬁguration of the last layer of existing deep networks in order to merge all labels nodes and features nodes into the same graph, and provides no ﬂexibility to feature dimension or edge types to the extended GAML.
Revised: On the contrary, GAML requires extra conﬁguration of the last layer of existing deep networks in order to merge all labels nodes and features nodes into the same graph, and provides no ﬂexibility to feature dimension or edge types to the extended GAML.

Original: Our extensive set of experiments verify that
Revised: Our extensive set of experiments verify that

Original: GTR serves better purposes of intra- and inter- graph message passing compared to several strong baselines.
Revised: GTR serves better purposes of intra- and inter- graph message passing compared to several strong baselines.

Original: Compared to some previous/parellel work on graph transformation such as graph2vec (Narayanan et al., 2016), graph2graph (Anonymous, 2018) and graph2seq (Xu et al., 2018; Venkatakrishnan et al., 2018),  GTR proposes a novel and uniﬁed strategy of formulating and learning from multi-modal data types such as visual features, knowledge graphs and sequential text, where graph2vec
Revised: Compared to some previous/parellel work on graph transformation such as graph2vec (Narayanan et al., 2016), graph2graph (Anonymous, 2018), graph2seq (Xu et al., 2018; Venkatakrishnan et al., 2018), AWE (Ivanov & Burnaev, 2018), and FGSD (Verma & Zhang, 2017), GTR proposes a novel and uniﬁed strategy of formulating and learning from multi-modal data types such as visual features, knowledge graphs and sequential text, where graph2vec

Original: (Narayanan et al., 2016) and graph2seq (Xu et al., 2018; Venkatakrishnan et al., 2018) can be seen as particular instances of GTR with vector or sequence as output respectively.
Revised: (Narayanan et al., 2016) and graph2seq (Xu et al., 2018; Venkatakrishnan et al., 2018) can be seen as particular instances of GTR with vector or sequence as output respectively.

Original: Besides, graph2graph (Anonymous, 2018) is speciﬁcally designed for multi-label classiﬁcation and its output in the conducted experiments is not actually graphs.
Revised: Besides, graph2graph (Anonymous, 2018) is speciﬁcally designed for multi-label classiﬁcation and its output in the conducted experiments is not actually graphs.

Original: As explained in Anonymous (2018), graph2graph does not following the conventional phrasing such as Seq2Seq in which the output is a sequence, but just representing the input and output space as two separate graphs.
Revised: As explained in Anonymous (2018), graph2graph does not following the conventional phrasing such as Seq2Seq in which the output is a sequence, but just representing the input and output space as two separate graphs.

Original: However, GTR indeed studies the graph to graph transformation, and follows the convention where output can be graphs such as in medical abnormality and disease classiﬁcation task, as well as other data types (e.g., graph-level labels in graph classiﬁcation task, and classiﬁer parameters in few-shot learning).
Revised: However, GTR indeed studies the graph to graph transformation, and follows the convention where output can be graphs such as in medical abnormality and disease classiﬁcation task, as well as other data types (e.g., graph-level labels in graph classiﬁcation task, and classiﬁer parameters in few-shot learning).

Original: Thus, GTR is at the core of graph to graph transformation, and shows more ﬂexibility towards application on a diverse set of tasks.
Revised: Thus, GTR is at the core of graph to graph transformation, and shows more ﬂexibility towards application on a diverse set of tasks.

Original: GTR is also explainable as the label prediction probability and learned edges/relations between labels can be visualized directly.
Revised: GTR is also explainable as the label prediction probability and learned edges/relations between labels can be visualized directly.

Revised: Additionally, AWE (Ivanov & Burnaev, 2018) uses random walk for graph representation learning which is limited to local context encoding, and only uses predeﬁned edge weights, while GTR enables global context encoding via global attention mechanism and incorporation of predeﬁned edge weights and learnable edge weights.

Revised: FGSD (Verma & Zhang, 2017) uses graph spectral distances for learning graph features which requires complex computation such as eigendecomposion while our method does not require such cost.

Revised: Furthermore, both AWE and FGSD use SVM as algorithm for graph classiﬁcation, while our method does not use any additional classiﬁer.

Original: B M EMORY COMPLEXITY
Revised: B M EMORY COMPLEXITY

Original: In terms of memory overhead, GTR scales linearly with the graph size thanks to the proposed global attention mechanism.
Revised: In terms of memory overhead, GTR scales linearly with the graph size thanks to the proposed global attention mechanism.

Original: GTR does not require any additional memory compared to the standard Transformer (Vaswani et al., 2017) and recurrent neural networks (Ranzato et al., 2016; Donahue et al., 2015).
Revised: GTR does not require any additional memory compared to the standard Transformer (Vaswani et al., 2017) and recurrent neural networks (Ranzato et al., 2016; Donahue et al., 2015).

Original: Table 5 (second column) summarizes the memory complexity for sequence and graph outputs where Let N is the number of input graph nodes, M is the number of output graph nodes, d is the models hidden state dimension.
Revised: Table 5 (second column) summarizes the memory complexity for sequence and graph out-

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: importance weight on learned edges.
Revised: importance weight on learned edges.

Original: For GTR episodic training, we use initial learning rate 0.01, and decrease by 10 times when encountering validation performance plateau.
Revised: For GTR episodic training, we use initial learning rate 0.01, and decrease by 10 times when encountering validation performance plateau.

Original: Feature extractor is ﬁxed during episodic training.
Revised: Feature extractor is ﬁxed during episodic training.

Revised: puts where Let N is the number of input graph nodes, M is the number of output graph nodes, d is the models hidden state dimension.

Original: C T IME COMPLEXITY
Revised: C T IME COMPLEXITY

Original: Compared to many graph neural networks (Defferrard et al., 2016; Kipf & Welling, 2017; Monti et al., 2017), GTR is more parallelizable and requires signiﬁcantly less time to train as every graph message propagation learns the global relations of all nodes, as opposed to only transferring features to connected neighboring nodes in the case of graph convolutional networks (GCN), or propagating previous states to subsequent states sequentially in the case of recurrent neural networks (RNN).
Revised: Compared to many graph neural networks (Defferrard et al., 2016; Kipf & Welling, 2017; Monti et al., 2017), GTR is more parallelizable and requires signiﬁcantly less time to train as every graph message propagation learns the global relations of all nodes, as opposed to only transferring features to connected neighboring nodes in the case of graph convolutional networks (GCN), or propagating previous states to subsequent states sequentially in the case of recurrent neural networks (RNN).

Original: Furthermore, in case of graph output, GTR only requires constant training and testing time as the joint probability of nodes/labels is not estimated using chain rules but predicted in parallel via global attention mechanism.
Revised: Furthermore, in case of graph output, GTR only requires constant training and testing time as the joint probability of nodes/labels is not estimated using chain rules but predicted in parallel via global attention mechanism.

Original: Table 5 (last two columns) summarizes the training time and testing time complexity.
Revised: Table 5 (last two columns) summarizes the training time and testing time complexity.

Original: Lastly, GTR does not suffer from vanished gradients due to the same merit of global context encoding while vanished gradients being one of the biggest challenges of most RNNs and potentially GCNs where only local context encoding is enabled.
Revised: Lastly, GTR does not suffer from vanished gradients due to the same merit of global context encoding while vanished gradients being one of the biggest challenges of most RNNs and potentially GCNs where only local context encoding is enabled.

Original: Last but not least, explicitly representing latent features as knowledge graphs may hold the key to classiﬁcation performance and explainability.
Revised: Last but not least, explicitly representing latent features as knowledge graphs may hold the key to classiﬁcation performance and explainability.

Original: GTR enables intuitive visualizations for better understanding of the knowledge structure and label relations, and explanation of the model behaviors.
Revised: GTR enables intuitive visualizations for better understanding of the knowledge structure and label relations, and explanation of the model behaviors.

Original: D A BLATION STUDY ON WEIGHT OF PRIOR EDGES .
Revised: D A BLATION STUDY ON WEIGHT OF PRIOR EDGES .

Original: The results on few-shot learning using different weight of prior edges is shown in Table 6.
Revised: The results on few-shot learning using different weight of prior edges is shown in Table 6.

Original: The highest 1-shot accuracy is achieved by λ = 0 .
Revised: The highest 1-shot accuracy is achieved by λ = 0 .

Original: The highest 5-shot accuracy is achieved by λ = 0 .
Revised: The highest 5-shot accuracy is achieved by λ = 0 .

Original: The results demonstrate that: 1) GTR with prior category similarity improves few-shot learning performance over its direct baseline framework Gidaris & Komodakis (2018) on both 1-shot and 5-shot learning, and achieves the state-of-the-art performance on 1-shot learning; 2) 1-shot learning relies more on the prior knowledge of similarity between base and novel categories than 5-shot learning 3) GTR λ = .0) is slightly lower than that achieved by Gidaris & Komodakis (2018), however not statistically signiﬁcant, indicating that the attention mechanisms in both models have similar effectiveness in this task. 
Revised: The results demonstrate that: 1) GTR with prior category similarity improves few-shot learning performance over its direct baseline framework Gidaris & Komodakis (2018) on both 1-shot and 5-shot learning, and achieves the state-of-the-art performance on 1-shot learning; 2) 1-shot learning relies more on the prior knowledge of similarity between base and novel categories than 5-shot learning; 3) GTR( λ =0.0) is slightly lower than that achieved by Gidaris & Komodakis (2018), however not statistically signiﬁcant, indicating that the attention mechanisms in both models have similar effectiveness in this task. [Table]

Original: E A BLATION STUDY ON MODEL SIZE
Revised: E A BLATION STUDY ON MODEL SIZE

Original: We conduct ablation study on how performance changes with parameter size/memory usage on 1shot learning by changing feature dimensions and hidden feature dimensions, and ﬁxing all other[Table] hyper-parameters.
Revised: We conduct ablation study on how performance changes with parameter size/memory usage on 1shot learning by changing feature dimensions and hidden feature dimensions, and ﬁxing all other hyper-parameters.

Original: Table 7 summarizes the results.
Revised: Table 7 summarizes the results.

Original: It can be observed that the performance drops slightly when reducing either feature dimension or hidden feature dimension.
Revised: It can be observed that the performance drops slightly when reducing either feature dimension or hidden feature dimension.

Original: However, all changes are not statistically signiﬁcant.
Revised: However, all changes are not statistically signiﬁcant.

Original: Furthermore, the memory usage reduces greatly from 0.3409G to 0.0278G. More importantly, GTR using 512 as feature dimension and hidden feature dimension (last row of Table 7) only consumes 0.0278G memory which is less than that used by Gidaris &
Revised: Furthermore, the memory usage reduces greatly from 0.3409G to 0.0278G. More importantly, GTR using 512 as feature dimension and hidden feature dimension (last row of Table 7) only consumes 0.0278G memory which is less than that used by Gidaris &

Original: Komodakis (2018) (the ﬁrst row of Table 7), and improved result greatly by 4.3578%.
Revised: Komodakis (2018) (the ﬁrst row of Table 7), and improved result greatly by 4.3578%.

Original: This demonstrates that GTR is not only effective, but also memory efﬁcient.
Revised: This demonstrates that GTR is not only effective, but also memory efﬁcient.

Original: Lastly, all results are still larger than that of all baseline models, maintaining GTRs state-of-the-art performance.
Revised: Lastly, all results are still larger than that of all baseline models, maintaining GTRs state-of-the-art performance.

Original: A line chart of model size v.s. graph size is provided in Appendix.
Revised: A line chart of model size v.s. graph size is provided in Appendix.