Original: I NDUCTIVE REPRESENTATION LEARNING ON TEMPORAL GRAPHS
Revised: I NDUCTIVE REPRESENTATION LEARNING ON TEMPORAL GRAPHS

Original: A BSTRACT
Revised: A BSTRACT

Original: Inductive representation learning on temporal graphs is an important step toward salable machine learning on real-world dynamic networks.
Revised: Inductive representation learning on temporal graphs is an important step toward salable machine learning on real-world dynamic networks.

Original: The evolving nature of temporal dynamic graphs requires handling new nodes as well as capturing temporal patterns.
Revised: The evolving nature of temporal dynamic graphs requires handling new nodes as well as capturing temporal patterns.

Original: The node embeddings, which are now functions of time, should represent both the static node features and the evolving topological structures.
Revised: The node embeddings, which are now functions of time, should represent both the static node features and the evolving topological structures.

Original: Moreover, node and topological features can be temporal as well , whose patterns the node embeddings should also capture.
Revised: Moreover, node and topological features can be temporal as well, whose patterns the node embeddings should also capture.

Original: We propose the temporal graph attention (TGAT) layer to efﬁciently aggregate temporal-topological neighborhood features as well as learning the time-feature interactions.
Revised: We propose the temporal graph attention (TGAT) layer to efﬁciently aggregate temporal-topological neighborhood features as well as to learn the time-feature interactions.

Original: For TGAT, we use the self-attention mechanism as building block and develop a novel functional time encoding technique based on the classical Bochner’s theorem from harmonic alaysis.
Revised: For TGAT, we use the self-attention mechanism as building block and develop a novel functional time encoding technique based on the classical Bochner’s theorem from harmonic alaysis.

Original: By stacking TGAT layers, the network recognizes the node embeddings as functions of time and is able to inductively infer embeddings for both new and observed nodes as the graph evolves.
Revised: By stacking TGAT layers, the network recognizes the node embeddings as functions of time and is able to inductively infer embeddings for both new and observed nodes as the graph evolves.

Original: The proposed approach handles both node classiﬁcation and link prediction task and can be naturally extended to include the temporal edge features.
Revised: The proposed approach handles both node classiﬁcation and link prediction task, and can be naturally extended to include the temporal edge features.

Original: We evaluate our method with transductive and inductive tasks under temporal settings with two benchmark and one industrial dataset.
Revised: We evaluate our method with transductive and inductive tasks under temporal settings with two benchmark and one industrial dataset.

Original: Our TGAT model compares favorably to state-of-the-art baselines as well  the previous temporal graph embedding approaches
Revised: Our TGAT model compares favorably to state-of-the-art baselines as well as the previous temporal graph embedding approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The technique of learning lower-dimensional vector embeddings on graphs have been widely applied to graph analysis tasks (Perozzi et al., 2014; Tang et al., 2015; Wang et al., 2016) and deployed in industrial systems (Ying et al., 2018; Wang et al., 2018a).
Revised: The technique of learning lower-dimensional vector embeddings on graphs have been widely applied to graph analysis tasks (Perozzi et al., 2014; Tang et al., 2015; Wang et al., 2016) and deployed in industrial systems (Ying et al., 2018; Wang et al., 2018a).

Original: Most of the graph representation learning approaches only accept static or non-temporal graphs as input, despite the fact that many graph-structured data are time-dependent.
Revised: Most of the graph representation learning approaches only accept static or non-temporal graphs as input, despite the fact that many graph-structured data are time-dependent.

Original: In social network, citation network, question answering forum and user-item interaction system, graphs are created as temporal interactions between nodes.
Revised: In social network, citation network, question answering forum and user-item interaction system, graphs are created as temporal interactions between nodes.

Original: Using the ﬁnal state as a static portrait of the graph is reasonable in some cases, such as the proteinprotein interaction network, as long as node interactions are timeless in nature.
Revised: Using the ﬁnal state as a static portrait of the graph is reasonable in some cases, such as the proteinprotein interaction network, as long as node interactions are timeless in nature.

Original: Otherwise, ignoring the temporal information can severely diminish the modelling efforts and even causing questionable inference.
Revised: Otherwise, ignoring the temporal information can severely diminish the modelling efforts and even causing questionable inference.

Original: For instance, models may mistakenly utilize future information for predicting past interactions during training and testing if the temporal constraints are disregarded.
Revised: For instance, models may mistakenly utilize future information for predicting past interactions during training and testing if the temporal constraints are disregarded.

Original: More importantly, the dynamic and evolving nature of many graph-related problems demand an explicitly modelling of the timeliness whenever nodes and edges are added, deleted or changed over time.
Revised: More importantly, the dynamic and evolving nature of many graph-related problems demand an explicitly modelling of the timeliness whenever nodes and edges are added, deleted or changed over time.

Original: Learning representations on temporal graphs is extremely challenging, and it is not until recently that several solutions are proposed (Nguyen et al., 2018;
Revised: Learning representations on temporal graphs is extremely challenging, and it is not until recently that several solutions are proposed (Nguyen et al., 2018;

Original: Li et al., 2018; Goyal et al., 2018; Trivedi et al., 2018).
Revised: Li et al., 2018; Goyal et al., 2018; Trivedi et al., 2018).

Original: We conclude the challenges in three folds.
Revised: We conclude the challenges in three folds.

Original: Firstly , to model the temporal dynamics, node embeddings should not be only the projections of topological structures and node features but also functions of the continuous time.
Revised: Firstly , to model the temporal dynamics, node embeddings should not be only the projections of topological structures and node features but also functions of the continuous time.

Original: Therefore, in addition to the usual vector space, temporal representation learning should be operated in some functional space as well.
Revised: Therefore, in addition to the usual vector space, temporal representation learning should be operated in some functional space as well.

Original: Secondly , graph topological structures are no longer static since the nodes and edges are evolving over time, which poses temporal constraints on neighborhood aggregation methods.
Revised: Secondly , graph topological structures are no longer static since the nodes and edges are evolving over time, which poses temporal constraints on neighborhood aggregation methods.

Original: Thirdly , node features and topological structures can exhibit temporal patterns.
Revised: Thirdly , node features and topological structures can exhibit temporal patterns.

Original: For example, node interactions that took place long ago may have less impact on the current topological structure and thus the node embeddings.
Revised: For example, node interactions that took place long ago may have less impact on the current topological structure and thus the node embeddings.

Original: Also, some nodes may possess features that allows them having more regular or recurrent interactions with others.
Revised: Also, some nodes may possess features that allows them having more regular or recurrent interactions with others.

Original: We provide sketched plots for visual illustration in Figure 1.
Revised: We provide sketched plots for visual illustration in Figure 1.

Original: Similar to its non-temporal counterparts, in the real-world applications, models for representation learning on temporal graphs should be able to quickly generate embeddings whenever required, in an inductive fashion.
Revised: Similar to its non-temporal counterparts, in the real-world applications, models for representation learning on temporal graphs should be able to quickly generate embeddings whenever required, in an inductive fashion.

Original: GraphSAGE (Hamilton et al., 2017a) and graph attention network ( GAT ) (Veli ˇ ckovi ´ c et al., 2017) are capable of inductively generating embeddings for unseen nodes based on their features, however, they do not consider the temporal factors.
Revised: GraphSAGE (Hamilton et al., 2017a) and graph attention network ( GAT ) (Veli ˇ ckovi ´ c et al., 2017) are capable of inductively generating embeddings for unseen nodes based on their features, however, they do not consider the temporal factors.

Original: Most of the temporal graph embedding methods can only handle transductive tasks, since they require re-training or the computationally-expensive gradient calculations to infer embeddings for unseen nodes or node embeddings for a new timepoint.
Revised: Most of the temporal graph embedding methods can only handle transductive tasks, since they require re-training or the computationally-expensive gradient calculations to infer embeddings for unseen nodes or node embeddings for a new timepoint.

Original: In this work, we aim at developing an architecture to inductively learn representations for temporal graphs such that the time-aware embeddings (for unseen and observed nodes) can be obtained via a single network forward pass.
Revised: In this work, we aim at developing an architecture to inductively learn representations for temporal graphs such that the time-aware embeddings (for unseen and observed nodes) can be obtained via a single network forward pass.

Original: The key to our approach is the combination of the self-attention mechanism (Vaswani et al., 2017) and a novel functional time encoding technique derived from the Bochner’s theorem from classical harmonic analysis (Loomis, 2013).
Revised: The key to our approach is the combination of the self-attention mechanism (Vaswani et al., 2017) and a novel functional time encoding technique derived from the Bochner’s theorem from classical harmonic analysis (Loomis, 2013).

Original: The motivation for adapting self-attention to inductive representation learning on temporal graphs is to identify and capture relevant pieces of the temporal neighborhood information.
Revised: The motivation for adapting self-attention to inductive representation learning on temporal graphs is to identify and capture relevant pieces of the temporal neighborhood information.

Original: Both graph convolutional network ( GCN ) (Kipf & Welling, 2016a) and GAT are implicitly or explicitly assigning different weights to neighboring nodes (Veliˇckovi´c et al., 2017) when aggregating node features.
Revised: Both graph convolutional network ( GCN ) (Kipf & Welling, 2016a) and GAT are implicitly or explicitly assigning different weights to neighboring nodes (Veliˇckovi´c et al., 2017) when aggregating node features.

Original: The self-attention mechanism was initially designed to recognize the relevant parts of input sequence in natural language processing.
Revised: The self-attention mechanism was initially designed to recognize the relevant parts of input sequence in natural language processing.

Original: As a discrete-event sequence learning method, self-attention outputs a vector representation of the input sequence as a weighted sum of individual entry embeddings.
Revised: As a discrete-event sequence learning method, self-attention outputs a vector representation of the input sequence as a weighted sum of individual entry embeddings.

Original: Selfattention enjoys several advantages such as parallelized computation and interpretability (Vaswani et al., 2017).
Revised: Selfattention enjoys several advantages such as parallelized computation and interpretability (Vaswani et al., 2017).

Original: Since it captures sequential information only through the positional encoding, temporal features can not be handled.
Revised: Since it captures sequential information only through the positional encoding, temporal features can not be handled.

Original: Therefore, we are motivated to replace positional encoding with some vector representation of time.
Revised: Therefore, we are motivated to replace positional encoding with some vector representation of time.

Original: Since time is a continuous variable, the mapping from the time domain to vector space has to be functional.
Revised: Since time is a continuous variable, the mapping from the time domain to vector space has to be functional.

Original: We gain insights from harmonic analysis and propose a theoretical-grounded functional time encoding approach that is compatible with the self-attention mechanism.
Revised: We gain insights from harmonic analysis and propose a theoretical-grounded functional time encoding approach that is compatible with the self-attention mechanism.

Original: The temporal signals are then modelled by the interactions between the functional time encoding and nodes features as well as the graph topological structures.
Revised: The temporal signals are then modelled by the interactions between the functional time encoding and nodes features as well as the graph topological structures.

Original: To evaluate our approach, we consider future link prediction on the observed nodes as transductive learning task, and on the unseen nodes as inductive learning task.
Revised: To evaluate our approach, we consider future link prediction on the observed nodes as transductive learning task, and on the unseen nodes as inductive learning task.

Original: We also examine the dynamic node classiﬁcation task using node embeddings (temporal versus non-temporal) as features to demonstrate the usefulness of our functional time encoding.
Revised: We also examine the dynamic node classiﬁcation task using node embeddings (temporal versus non-temporal) as features to demonstrate the usefulness of our functional time encoding.

Original: We carry out extensive ablation studies and sensitivity analysis to show the effectiveness of the proposed functional time encoding and TGAT -layer.
Revised: We carry out extensive ablation studies and sensitivity analysis to show the effectiveness of the proposed functional time encoding and TGAT -layer.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Graph representation learning .
Revised: Graph representation learning .

Original: Spectral graph embedding models operate on the graph spectral domain by approximating, projecting or expanding the graph Laplacian (Kipf & Welling, 2016a;
Revised: Spectral graph embedding models operate on the graph spectral domain by approximating, projecting or expanding the graph Laplacian (Kipf & Welling, 2016a;

Original: Henaff et al., 2015; Defferrard et al., 2016).
Revised: Henaff et al., 2015; Defferrard et al., 2016).

Original: Since their training and inference are conditioned on the speciﬁc graph spectrum, they are not directly extendable to temporal graphs.
Revised: Since their training and inference are conditioned on the speciﬁc graph spectrum, they are not directly extendable to temporal graphs.

Original: Non-spectral approaches, such as GAT , GraphSAGE and MoNET , (Monti et al., 2017) rely on the localized neighbourhood aggregations and thus are not restricted to the training graph.
Revised: Non-spectral approaches, such as GAT , GraphSAGE and MoNET , (Monti et al., 2017) rely on the localized neighbourhood aggregations and thus are not restricted to the training graph.

Original: GraphSAGE and GAT also have the ﬂexibility to handle evolving graphs inductively.
Revised: GraphSAGE and GAT also have the ﬂexibility to handle evolving graphs inductively.

Original: To extend classical graph representation learning approaches to the temporal domain, several attempts have been done by cropping the temporal graph into a sequence of graph snapshots (Li et al., 2018; Goyal et al., 2018; Rahman et al., 2018; Xu et al., 2019b), and some others work with temporally persistent node (edges) (Trivedi et al., 2018; Ma et al., 2018).
Revised: To extend classical graph representation learning approaches to the temporal domain, several attempts have been done by cropping the temporal graph into a sequence of graph snapshots (Li et al., 2018; Goyal et al., 2018; Rahman et al., 2018; Xu et al., 2019b), and some others work with temporally persistent node (edges) (Trivedi et al., 2018; Ma et al., 2018).

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2018) proposes a node embedding method based on temporal random walk and reported state-of-the-art performances.
Revised: (2018) proposes a node embedding method based on temporal random walk and reported state-of-the-art performances.

Original: However, their approach only generates embeddings for the ﬁnal state of temporal graph and can not directly apply to the inductive setting.
Revised: However, their approach only generates embeddings for the ﬁnal state of temporal graph and can not directly apply to the inductive setting.

Original: Self-attention mechanism.
Revised: Self-attention mechanism.

Original: Self-attention mechanisms often have two components: the embedding layer and the attention layer.
Revised: Self-attention mechanisms often have two components: the embedding layer and the attention layer.

Original: The embedding layer takes an ordered entity sequence as input.
Revised: The embedding layer takes an ordered entity sequence as input.

Original: Selfattention uses the positional encoding, i.e. each position k is equipped with a vector
Revised: Selfattention uses the positional encoding, i.e. each position k is equipped with a vector

Original: p k (ﬁxed or learnt) which is shared for all sequences.
Revised: p k (ﬁxed or learnt) which is shared for all sequences.

Original: For the entity sequence e = ( e 1 , . . .
Revised: For the entity sequence e = ( e 1 , . . .

Original: , e l ) , the embedding layer takes the sum or concatenation of entity embeddings (or features) ( z ∈ R d ) and their positional encodings as input:
Revised: , e l ) , the embedding layer takes the sum or concatenation of entity embeddings (or features) ( z ∈ R d ) and their positional encodings as input:

Original: [Equation]
Revised: [Equation]

Original: where || denotes concatenation operation and d pos is the dimension for positional encoding.
Revised: where || denotes concatenation operation and d pos is the dimension for positional encoding.

Original: Selfattention layers can be constructed using the scaled dot-product attention, which is deﬁned as:
Revised: Selfattention layers can be constructed using the scaled dot-product attention, which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Q denotes the ’queries’, K the ’keys’ and V the ’values’.
Revised: where Q denotes the ’queries’, K the ’keys’ and V the ’values’.

Original: In Vaswani et al.
Revised: In Vaswani et al.

Original: (2017), they are treated as projections of the output Z e : Q = Z e W Q ,
Revised: (2017), they are treated as projections of the output Z e : Q = Z e W Q ,

Original: [Equation]
Revised: [Equation]

Original: W K and W V are the projection matrices.
Revised: W K and W V are the projection matrices.

Original: Since each row of Q , K and V represents an entity, the dot-product attention takes a weighted sum of the entity ’values’ in V where the weights are given by the interactions of entity ’query-key’ pairs.
Revised: Since each row of Q , K and V represents an entity, the dot-product attention takes a weighted sum of the entity ’values’ in V where the weights are given by the interactions of entity ’query-key’ pairs.

Original: The hidden representation for the entity sequence under the dot-product attention is then given by h e =
Revised: The hidden representation for the entity sequence under the dot-product attention is then given by h e =

Original: Attn ( Q , K , V ) .
Revised: Attn ( Q , K , V ) .

Original:  T EMPORAL G RAPH A TTENTION N ETWORK A RCHITECTURE
Revised:  T EMPORAL G RAPH A TTENTION N ETWORK A RCHITECTURE

Original: We ﬁrst derive the mapping from time domain to the continuous differentiable functional domain as the functional time encoding such that resulting formulation is compatible with self-attention mechanism as well as the backpropagation-based optimization frameworks.
Revised: We ﬁrst derive the mapping from time domain to the continuous differentiable functional domain as the functional time encoding such that resulting formulation is compatible with self-attention mechanism as well as the backpropagation-based optimization frameworks.

Original: The same idea was explored in a concurrent work (Xu et al., 2019a).
Revised: The same idea was explored in a concurrent work (Xu et al., 2019a).

Original: We then present the temporal graph attention layer and show how it can be naturally extended to incorporate the edge features.
Revised: We then present the temporal graph attention layer and show how it can be naturally extended to incorporate the edge features.

Original: 3. F UNCTIONAL TIME ENCODING
Revised: 3. F UNCTIONAL TIME ENCODING

Original: Recall that our starting point is to obtain a continuous functional mapping Φ : T → R d T from time domain to the d T -dimensional vector space to replace the positional encoding in (1).
Revised: Recall that our starting point is to obtain a continuous functional mapping Φ : T → R d T from time domain to the d T -dimensional vector space to replace the positional encoding in (1).

Original: Without loss of generality, we assume that the time domain can be represented by the interval starting from origin: T = [0 , t max ] , where t max is determined by the observed data.
Revised: Without loss of generality, we assume that the time domain can be represented by the interval starting from origin: T = [0 , t max ] , where t max is determined by the observed data.

Original: For the inner-product selfattention in (2), often the ’key’ and ’query’ matrices ( K , Q ) are given by identity or linear projection of Z e deﬁned in (1), leading to terms that only involve inner-products between positional (time) encodings.
Revised: For the inner-product selfattention in (2), often the ’key’ and ’query’ matrices ( K , Q ) are given by identity or linear projection of Z e deﬁned in (1), leading to terms that only involve inner-products between positional (time) encodings.

Original: Consider two time points t 1 , t 2 and inner product between their functional encodings (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) .
Revised: Consider two time points t 1 , t 2 and inner product between their functional encodings (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) .

Original: Usually, the relative timespan, rather than the absolute value of time, reveals critical temporal information.
Revised: Usually, the relative timespan, rather than the absolute value of time, reveals critical temporal information.

Original: Therefore, we are more interested in learning patterns related to the timespan of | t 2 − t 1 | , which should be ideally expressed by (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) to be compatible with self-attention.
Revised: Therefore, we are more interested in learning patterns related to the timespan of | t 2 − t 1 | , which should be ideally expressed by (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) to be compatible with self-attention.

Original: Formally, we deﬁne the temporal kernel K : T × T → R with K ( t 1 , t 2 ) :
Revised: Formally, we deﬁne the temporal kernel K : T × T → R with K ( t 1 , t 2 ) :

Original: = (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) and
Revised: = (cid:10) Φ( t 1 ) , Φ( t 2 ) (cid:11) and

Original: K ( t 1 , t 2 ) =
Revised: K ( t 1 , t 2 ) =

Original: ψ ( t 1 − t 2 ) , ∀ t 1 , t 2 ∈ T for some ψ : [ − t max , t max ] → R .
Revised: ψ ( t 1 − t 2 ) , ∀ t 1 , t 2 ∈ T for some ψ : [ − t max , t max ] → R .

Original: The temporal kernel is then translation-invariant, since K ( t 1 + c, t 2 + c ) =
Revised: The temporal kernel is then translation-invariant, since K ( t 1 + c, t 2 + c ) =

Original: ψ ( t 1 − t 2 ) =
Revised: ψ ( t 1 − t 2 ) =

Original: K ( t 1 , t 2 ) for any constant c .
Revised: K ( t 1 , t 2 ) for any constant c .

Original: Generally speaking, functional learning is extremely complicated since it operates on inﬁnite-dimensional spaces, but now we have transformed the problem into learning the temporal kernel K expressed by Φ .
Revised: Generally speaking, functional learning is extremely complicated since it operates on inﬁnite-dimensional spaces, but now we have transformed the problem into learning the temporal kernel K expressed by Φ .

Original: Nonetheless, we still need to ﬁgure out an explicit parameterization for Φ in order to conduct efﬁcient gradient-based optimization.
Revised: Nonetheless, we still need to ﬁgure out an explicit parameterization for Φ in order to conduct efﬁcient gradient-based optimization.

Original: Classical harmonic analysis theory, i.e. the Bochner’s theorem, motivates our ﬁnal solution.
Revised: Classical harmonic analysis theory, i.e. the Bochner’s theorem, motivates our ﬁnal solution.

Original: We point out that the temporal kernel K is positive-semideﬁnite (PSD) and continuous, since it is deﬁned via Gram matrix and the mapping Φ is continuous.
Revised: We point out that the temporal kernel K is positive-semideﬁnite (PSD) and continuous, since it is deﬁned via Gram matrix and the mapping Φ is continuous.

Original: Therefore, the kernel K deﬁned above satisfy the assumptions of the Bochner’s theorem, which we state below.
Revised: Therefore, the kernel K deﬁned above satisfy the assumptions of the Bochner’s theorem, which we state below.

Original: Theorem 1 (Bochner’s Theorem) .
Revised: Theorem 1 (Bochner’s Theorem) .

Original: A continuous, translation-invariant kernel K ( x , y ) =
Revised: A continuous, translation-invariant kernel K ( x , y ) =

Original: ψ ( x − y ) on R d is positive deﬁnite if and only if there exists a non-negative measure on R such that ψ is the
Revised: ψ ( x − y ) on R d is positive deﬁnite if and only if there exists a non-negative measure on R such that ψ is the

Original: Fourier transform of the measure.
Revised: Fourier transform of the measure.

Original: Consequently, when scaled properly, our temporal kernel K have the alternate expression:
Revised: Consequently, when scaled properly, our temporal kernel K have the alternate expression:

Original: [Equation]
Revised: [Equation]

Original: where ξ ω ( t ) = e iωt .
Revised: where ξ ω ( t ) = e iωt .

Original: Since the kernel K and the probability measure p ( ω ) are real, we extract the real part of (3) and obtain:
Revised: Since the kernel K and the probability measure p ( ω ) are real, we extract the real part of (3) and obtain:

Original: [Equation]
Revised: [Equation]

Original: The above formulation suggests approximating the expectation by the Monte Carlo integral (Rahimi & Recht, 2008), i.e. K ( t 1 , t 2 )
Revised: The above formulation suggests approximating the expectation by the Monte Carlo integral (Rahimi & Recht, 2008), i.e. K ( t 1 , t 2 )

Original: ≈ 1 d (cid:80) di
Revised: ≈ 1 d (cid:80) di

Original: =1 cos( ω i t 1 ) cos( ω i t 2 )
Revised: =1 cos( ω i t 1 ) cos( ω i t 2 )

Original: + sin( ω i t 1 ) sin( ω i t 2 ) , with ω 1 , . . .
Revised: + sin( ω i t 1 ) sin( ω i t 2 ) , with ω 1 , . . .

Original: , ω d i.i.d ∼ p ( ω ) .
Revised: , ω d i.i.d ∼ p ( ω ) .

Original: Therefore, we propose the ﬁnite dimensional functional mapping to R d as:
Revised: Therefore, we propose the ﬁnite dimensional functional mapping to R d as:

Original: [Equation]
Revised: [Equation]

Original: and it is easy to show that (cid:10) Φ d ( t 1 ) , Φ d ( t 2 ) (cid:11)
Revised: and it is easy to show that (cid:10) Φ d ( t 1 ) , Φ d ( t 2 ) (cid:11)

Original: ≈ K
Revised: ≈ K

Original: ( t 1 , t 2 ) .
Revised: ( t 1 , t 2 ) .

Original: As a matter of fact, we prove the stochastic uniform convergence of (cid:10) Φ d ( t 1 ) , Φ d ( t 2 ) (cid:11) to the underlying K ( t 1 , t 2 ) and shows that it takes only a reasonable amount of samples to achieve proper estimation, which is stated in Claim 1.
Revised: As a matter of fact, we prove the stochastic uniform convergence of (cid:10) Φ d ( t 1 ) , Φ d ( t 2 ) (cid:11) to the underlying K ( t 1 , t 2 ) and shows that it takes only a reasonable amount of samples to achieve proper estimation, which is stated in Claim 1.

Original: Claim 1.
Revised: Claim 1.

Original: Let p ( ω ) be the corresponding probability measure stated in Bochner’s Theorem for kernel function K .
Revised: Let p ( ω ) be the corresponding probability measure stated in Bochner’s Theorem for kernel function K .

Original: Suppose the feature map Φ is constructed as described above using samples { ω i } di =1 , then we only need d = Ω (cid:0) 1 (cid:15) 2 log σ 2 p t max (cid:15) (cid:1) samples to have sup
Revised: Suppose the feature map Φ is constructed as described above using samples { ω i } di =1 , then we only need d = Ω (cid:0) 1 (cid:15) 2 log σ 2 p t max (cid:15) (cid:1) samples to have sup

Original: [Equation]
Revised: [Equation]

Original: ) (cid:12)(cid:12) < (cid:15) with any probability for ∀ (cid:15) > 0 , where σ 2 p is the second momentum with respect to p ( ω ) .
Revised: ) (cid:12)(cid:12) < (cid:15) with any probability for ∀ (cid:15) > 0 , where σ 2 p is the second momentum with respect to p ( ω ) .

Original: The proof is provided in supplement material.
Revised: The proof is provided in supplement material.

Original: By applying Bochner’s theorem, we convert the problem of kernel learning to distribution learning, i.e. estimating the p ( ω ) in Theorem 1.
Revised: By applying Bochner’s theorem, we convert the problem of kernel learning to distribution learning, i.e. estimating the p ( ω ) in Theorem 1.

Original: A straightforward solution is to apply the reparameterization trick by using auxiliary random variables with a known marginal distribution as in variational autoencoders (Kingma & Welling, 2013).
Revised: A straightforward solution is to apply the reparameterization trick by using auxiliary random variables with a known marginal distribution as in variational autoencoders (Kingma & Welling, 2013).

Original: However, the reparameterization trick is often limited to certain distributions such as the ’local-scale’ family, which may not be rich enough for our purpose.
Revised: However, the reparameterization trick is often limited to certain distributions such as the ’local-scale’ family, which may not be rich enough for our purpose.

Original: For instance, when p ( ω ) is multimodal it is difﬁcult to reconstruct the underlying distribution via direct reparameterizations.
Revised: For instance, when p ( ω ) is multimodal it is difﬁcult to reconstruct the underlying distribution via direct reparameterizations.

Original: An alternate approach is to use the inverse cumulative distribution function (CDF) transformation.
Revised: An alternate approach is to use the inverse cumulative distribution function (CDF) transformation.

Original: Rezende & Mohamed (2015) propose using parameterized normalizing ﬂow , i.e. a sequence of invertible transformation functions, to approximate arbitrarily complicated CDF and efﬁciently sample from it.
Revised: Rezende & Mohamed (2015) propose using parameterized normalizing ﬂow , i.e. a sequence of invertible transformation functions, to approximate arbitrarily complicated CDF and efﬁciently sample from it.

Original: Dinh et al.
Revised: Dinh et al.

Original: (2016) further considers stacking bijective transformations, known as afﬁne coupling layer, to achieve more effective CDF estimation.
Revised: (2016) further considers stacking bijective transformations, known as afﬁne coupling layer, to achieve more effective CDF estimation.

Original: The above methods learns the inverse CDF function F − 1 θ ( . ) parameterized by ﬂow-based networks and draw samples from the corresponding distribution.
Revised: The above methods learns the inverse CDF function F − 1 θ ( . ) parameterized by ﬂow-based networks and draw samples from the corresponding distribution.

Original: On the other hand, if we consider an non-parameterized approach for estimating distribution, then learning F − 1 ( . ) and obtain d samples from it is equivalent to directly optimizing the { ω 1 , . .
Revised: On the other hand, if we consider an non-parameterized approach for estimating distribution, then learning F − 1 ( . ) and obtain d samples from it is equivalent to directly optimizing the { ω 1 , . .

Original: . , ω d } in (4) as free model parameters.
Revised: . , ω d } in (4) as free model parameters.

Original: In practice, we ﬁnd these two approaches to have highly comparable performances (see supplement material).
Revised: In practice, we ﬁnd these two approaches to have highly comparable performances (see supplement material).

Original: Therefore we focus on the non-parametric approach, since it is more parameter-efﬁcient and has faster training speed (as no sampling during training is required).
Revised: Therefore we focus on the non-parametric approach, since it is more parameter-efﬁcient and has faster training speed (as no sampling during training is required).

Original: The above functional time encoding is fully compatible with self-attention, thus they can replace the positional encodings in (1) and their parameters are jointly optimized as part of the whole model.
Revised: The above functional time encoding is fully compatible with self-attention, thus they can replace the positional encodings in (1) and their parameters are jointly optimized as part of the whole model.

Original: 3. T EMPORAL GRAPH ATTENTION LAYER
Revised: 3. T EMPORAL GRAPH ATTENTION LAYER

Original: We use v i and x i ∈ R d 0 to denote node i and its raw node features.
Revised: We use v i and x i ∈ R d 0 to denote node i and its raw node features.

Original: The proposed TGAT architecture depends solely on the temporal graph attention layer ( TGAT layer).
Revised: The proposed TGAT architecture depends solely on the temporal graph attention layer ( TGAT layer).

Original: In analogy to GraphSAGE and
Revised: In analogy to GraphSAGE and

Original: GAT , the TGAT layer can be thought of as a local aggregation operator that takes the temporal neighborhood with their hidden representations (or features) as well as timestamps as input, and the output is the time-aware representation for target node at any time point t .
Revised: GAT , the TGAT layer can be thought of as a local aggregation operator that takes the temporal neighborhood with their hidden representations (or features) as well as timestamps as input, and the output is the time-aware representation for target node at any time point t .

Original: We denote the hidden representation output for node i at time t from the l th layer l i
Revised: We denote the hidden representation output for node i at time t from the l th layer l i

Original: Similar to GAT , we perform the masked self-attention to take account of the structural information (Veli ˇ ckovi ´
Revised: Similar to GAT , we perform the masked self-attention to take account of the structural information (Veli ˇ ckovi ´

Original: c et al., 2017).
Revised: c et al., 2017).

Original: For node v 0 at time t , we consider its neighborhood
Revised: For node v 0 at time t , we consider its neighborhood

Original: N ( v 0 ; t ) =
Revised: N ( v 0 ; t ) =

Original: { v 1 , . .
Revised: { v 1 , . .

Original: . , v N } such that the interaction between v 0 and v i ∈ N ( v 0 ; t ) , which takes place at time t i , is prior to t 1 .
Revised: . , v N } such that the interaction between v 0 and v i ∈ N ( v 0 ; t ) , which takes place at time t i , is prior to t 1 .

Original: The input of TGAT layer is the neighborhood information Z = (cid:8)
Revised: The input of TGAT layer is the neighborhood information Z = (cid:8)

Original: ˜ h
Revised: ˜ h

Original: ( l − 1)( t 1 ) , . . .
Revised: ( l − 1)( t 1 ) , . . .

Original: ˜ h
Revised: ˜ h

Original: ( l − 1) N ( t N ) (cid:9) and the target node information with some time point (cid:0)
Revised: ( l − 1) N ( t N ) (cid:9) and the target node information with some time point (cid:0)

Original: ˜ h ( l − 1)( t ) , t (cid:1) .
Revised: ˜ h ( l − 1)( t ) , t (cid:1) .

Original: When l = 1 , i.e. for the ﬁrst layer, the inputs are just raw node features.
Revised: When l = 1 , i.e. for the ﬁrst layer, the inputs are just raw node features.

Original: The layer produces the time-aware representation of target node v 0 at time t , denoted by ˜ h
Revised: The layer produces the time-aware representation of target node v 0 at time t , denoted by ˜ h

Original: ( l ) 

Revised: ( l )

Original: ( t ) , as its output.
Revised: ( t ) , as its output.

Original: Due to the translation-invariant assumption for the temporal kernel, we can alternatively use { t − t 1 , . . .
Revised: Due to the translation-invariant assumption for the temporal kernel, we can alternatively use { t − t 1 , . . .

Original: , t − t N } as interaction times, since | t i − t
Revised: , t − t N } as interaction times, since | t i − t

Original: j | = (cid:12)(cid:12) ( t − t i ) − ( t − t j ) (cid:12)(cid:12) and we only care for the timespan.
Revised: j | = (cid:12)(cid:12) ( t − t i ) − ( t − t j ) (cid:12)(cid:12) and we only care for the timespan.

Original: In line with original self-attention mechanism, we ﬁrst obtain the entity-temporal feature matrix as
Revised: In line with original self-attention mechanism, we ﬁrst obtain the entity-temporal feature matrix as

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: sum)
Revised: sum)

Original: [Equation]
Revised: [Equation]

Original: 6) and forward it to three different linear projections to obtain the ’query’, ’key’ and ’value’:
Revised: 6) and forward it to three different linear projections to obtain the ’query’, ’key’ and ’value’:

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K , W V ∈ R ( d + d T ) × d h are the weight matrices that are employed to capture the interactions between time encoding and node features.
Revised: where W Q , W K , W V ∈ R ( d + d T ) × d h are the weight matrices that are employed to capture the interactions between time encoding and node features.

Original: For notation simplicity, in the following discussion we treat the dependence of the intermediate outputs on target time t as implicit.
Revised: For notation simplicity, in the following discussion we treat the dependence of the intermediate outputs on target time t as implicit.

Original: The attention weights { α i } Ni =1 of the softmax function output in (2) is given by: α i = exp (cid:0) q (cid:124)
Revised: The attention weights { α i } Ni =1 of the softmax function output in (2) is given by: α i = exp (cid:0) q (cid:124)

Original: K i (cid:1) / (cid:16) (cid:80) q exp (cid:0) q (cid:124)
Revised: K i (cid:1) / (cid:16) (cid:80) q exp (cid:0) q (cid:124)

Original: K q (cid:1)(cid:17) .
Revised: K q (cid:1)(cid:17) .

Original: The attention weight α i reveals how node i attends to the features of node v 0 within the topological structure deﬁned as N ( v 0 ; t ) after accounting for their interaction time with v 0 .
Revised: The attention weight α i reveals how node i attends to the features of node v 0 within the topological structure deﬁned as N ( v 0 ; t ) after accounting for their interaction time with v 0 .

Original: The self-attention therefore captures the temporal interactions with both node features and topological features and deﬁnes a local temporal aggregation operator on graph.
Revised: The self-attention therefore captures the temporal interactions with both node features and topological features and deﬁnes a local temporal aggregation operator on graph.

Original: The hidden representation for any node v i ∈ N ( v 0 ; t ) is given by: α i V i .
Revised: The hidden representation for any node v i ∈ N ( v 0 ; t ) is given by: α i V i .

Original: The mechanism can be effectively shared across all nodes for any time point.
Revised: The mechanism can be effectively shared across all nodes for any time point.

Original: We then take the row-wise sum from the above dot-product self-attention output as the hidden neighborhood representations , i.e.
Revised: We then take the row-wise sum from the above dot-product self-attention output as the hidden neighborhood representations , i.e.

Original: [Equation]
Revised: [Equation]

Original: To combine neighbourhood representation with the target node features, we adopt the same practice from GraphSAGE and concatenate the neighbourhood representation with the target node’s feature vector z 0 .
Revised: To combine neighbourhood representation with the target node features, we adopt the same practice from GraphSAGE and concatenate the neighbourhood representation with the target node’s feature vector z 0 .

Original: We then pass it to a feed-forward neural network to capture non-linear interactions between the features as in (Vaswani et al., 2017):
Revised: We then pass it to a feed-forward neural network to capture non-linear interactions between the features as in (Vaswani et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where ˜ h ( l ) 0 ( t ) ∈ R
Revised: where ˜ h ( l ) 0 ( t ) ∈ R

Original: d is the ﬁnal output representing the time-aware node embedding at time t for the target node.
Revised: d is the ﬁnal output representing the time-aware node embedding at time t for the target node.

Original: Therefore, the TGAT layer can be implemented for node classiﬁcation task using the semi-supervised learning framework proposed in Kipf & Welling (2016a) as well as the link prediction task with the encoder-decoder framework summarized by Hamilton et al.
Revised: Therefore, the TGAT layer can be implemented for node classiﬁcation task using the semi-supervised learning framework proposed in Kipf & Welling (2016a) as well as the link prediction task with the encoder-decoder framework summarized by Hamilton et al.

Original: (2017b).
Revised: (2017b).

Original: Veliˇckovi´c et al.
Revised: Veliˇckovi´c et al.

Original: (2017) suggests that using multi-head attention improves performances and stabilizes training for GAT .
Revised: (2017) suggests that using multi-head attention improves performances and stabilizes training for GAT .

Original: For generalization purposes, we also show that the proposed TGAT layer can be easily extended to the multi-head setting.
Revised: For generalization purposes, we also show that the proposed TGAT layer can be easily extended to the multi-head setting.

Original: Consider the dot-product self-attention outputs from a total of k different heads, i.e. h ( i ) ≡ Attn ( i ) (cid:0) q ( t ) , K ( t ) , V ( t ) (cid:1) , i = 1 , . . .
Revised: Consider the dot-product self-attention outputs from a total of k different heads, i.e. h ( i ) ≡ Attn ( i ) (cid:0) q ( t ) , K ( t ) , V ( t ) (cid:1) , i = 1 , . . .

Original: , k .
Revised: , k .

Original: We ﬁrst concatenate the k neighborhood representations into a combined vector and then carry out the same procedure:
Revised: We ﬁrst concatenate the k neighborhood representations into a combined vector and then carry out the same procedure:

Original: [Equation]
Revised: [Equation]

Original: Just like GraphSAGE , a single TGAT layer aggregates the localized one-hop neighborhood, and by stacking L TGAT layers the aggregation extends to L hops.
Revised: Just like GraphSAGE , a single TGAT layer aggregates the localized one-hop neighborhood, and by stacking L TGAT layers the aggregation extends to L hops.

Original: Similar to GAT , out approach does not restrict the size of neighborhood.
Revised: Similar to GAT , out approach does not restrict the size of neighborhood.

Original: We provide a graphical illustration of our TGAT layer in Figure 2.
Revised: We provide a graphical illustration of our TGAT layer in Figure 2.

Original: 3. E XTENSION TO INCORPORATE E DGE F EATURES
Revised: 3. E XTENSION TO INCORPORATE E DGE F EATURES

Original: We show that the TGAT layer can be naturally extended to handle edge features in a messagepassing fashion.
Revised: We show that the TGAT layer can be naturally extended to handle edge features in a messagepassing fashion.

Original: Simonovsky & Komodakis (2017) and Wang et al.
Revised: Simonovsky & Komodakis (2017) and Wang et al.

Original: (2018b) modify classical spectral-based graph convolutional networks to incorporate edge features.
Revised: (2018b) modify classical spectral-based graph convolutional networks to incorporate edge features.

Original: Battaglia et al.
Revised: Battaglia et al.

Original: (2018) propose general graph neural network frameworks where edges features can be processed.
Revised: (2018) propose general graph neural network frameworks where edges features can be processed.

Original: For temporal graphs, we consider the general setting where each dynamic edge is associated with a feature vector, i.e. the interaction between v
Revised: For temporal graphs, we consider the general setting where each dynamic edge is associated with a feature vector, i.e. the interaction between v

Original: i and v j at time t induces the feature vector x i,j ( t ) .
Revised: i and v j at time t induces the feature vector x i,j ( t ) .

Original: To propagate edge features during the TGAT aggregation, we simply extend the Z ( t ) in (6) to:
Revised: To propagate edge features during the TGAT aggregation, we simply extend the Z ( t ) in (6) to:

Original: [Equation]
Revised: [Equation]

Original: such that the edge information is propagated to the target node’s hidden representation, and then passed on to the next layer (if exists).
Revised: such that the edge information is propagated to the target node’s hidden representation, and then passed on to the next layer (if exists).

Original: The remaining structures stay the same as in Section 3.2.
Revised: The remaining structures stay the same as in Section 3.2.

Original: 3. T EMPORAL SUB - GRAPH BATCHING
Revised: 3. T EMPORAL SUB - GRAPH BATCHING

Original: Stacking L TGAT layers is equivalent to aggregate over the L -hop neighborhood.
Revised: Stacking L TGAT layers is equivalent to aggregate over the L -hop neighborhood.

Original: For each L -hop sub-graph that is constructed during the batch-wise training, all message passing directions must be aligned with the observed chronological orders.
Revised: For each L -hop sub-graph that is constructed during the batch-wise training, all message passing directions must be aligned with the observed chronological orders.

Original: Unlike the non-temporal setting where each edge appears only once, in temporal graphs two node can have multiple interactions at different time points.
Revised: Unlike the non-temporal setting where each edge appears only once, in temporal graphs two node can have multiple interactions at different time points.

Original: Whether or not to allow loops that involve the target node should be judged case-bycase.
Revised: Whether or not to allow loops that involve the target node should be judged case-bycase.

Original: Sampling from neighborhood, or known as neighborhood dropout , may speed up and stabilize model training.
Revised: Sampling from neighborhood, or known as neighborhood dropout , may speed up and stabilize model training.

Original: For temporal graphs, neighborhood dropout can be carried uniformly or weighted by the inverse timespan such that more recent interactions has higher probability of being sampled.
Revised: For temporal graphs, neighborhood dropout can be carried uniformly or weighted by the inverse timespan such that more recent interactions has higher probability of being sampled.

Original: 3. C OMPARISONS TO RELATED WORK
Revised: 3. C OMPARISONS TO RELATED WORK

Original: The functional time encoding technique and TGAT layer introduced in Section 3.1 and 3.2 solves several critical challenges, and the TGAT network intrinsically connects to several prior methods.
Revised: The functional time encoding technique and TGAT layer introduced in Section 3.1 and 3.2 solves several critical challenges, and the TGAT network intrinsically connects to several prior methods.

Original: • Instead of cropping temporal graphs into a sequence of snapshots or constructing timeconstraint random walks, which inspired most of the current temporal graph embedding methods, we directly learn the functional representation of time.
Revised: • Instead of cropping temporal graphs into a sequence of snapshots or constructing timeconstraint random walks, which inspired most of the current temporal graph embedding methods, we directly learn the functional representation of time.

Original: The proposed approach is
Revised: The proposed approach is

Original: motivated by and thus fully compatible with the well-established self-attention mechanism.
Revised: motivated by and thus fully compatible with the well-established self-attention mechanism.

Original: Also, to the best of our knowledge, no previous work has discussed the temporal-feature interactions for temporal graphs, which is also considered in our approach.
Revised: Also, to the best of our knowledge, no previous work has discussed the temporal-feature interactions for temporal graphs, which is also considered in our approach.

Original: • The TGAT layer is computationally efﬁcient compared with RNN-based models, since the masked self-attention operation is parallelizable, as suggested by Vaswani et al.
Revised: • The TGAT layer is computationally efﬁcient compared with RNN-based models, since the masked self-attention operation is parallelizable, as suggested by Vaswani et al.

Original: The per-batch time complexity of the TGAT layer with k heads and l layers can be expressed as O (cid:0) ( k ˜ N ) l (cid:1) where ˜ N is the average neighborhood size, which is comparable to GAT .
Revised: The per-batch time complexity of the TGAT layer with k heads and l layers can be expressed as O (cid:0) ( k ˜ N ) l (cid:1) where ˜ N is the average neighborhood size, which is comparable to GAT .

Original: When using multi-head attention, the computation for each head can be parallelized as well.
Revised: When using multi-head attention, the computation for each head can be parallelized as well.

Original: • The inference with TGAT is entirely inductive .
Revised: • The inference with TGAT is entirely inductive .

Original: With an explicit functional expression ˜ h ( t ) for each node, the time-aware node embeddings can be easily inferred for any timestamp via a single network forward pass.
Revised: With an explicit functional expression ˜ h ( t ) for each node, the time-aware node embeddings can be easily inferred for any timestamp via a single network forward pass.

Original: Similarity, whenever the graph is updated, the embeddings for both unseen and observed nodes can be quickly inferred in an inductive fashion similar to that of GraphSAGE , and the computations can be parallelized across all nodes.
Revised: Similarity, whenever the graph is updated, the embeddings for both unseen and observed nodes can be quickly inferred in an inductive fashion similar to that of GraphSAGE , and the computations can be parallelized across all nodes.

Original: • GraphSAGE with mean pooling (Hamilton et al., 2017a) can be interpreted as a special case of the proposed method, where the temporal neighborhood is aggregated with equal attention coefﬁcients.
Revised: • GraphSAGE with mean pooling (Hamilton et al., 2017a) can be interpreted as a special case of the proposed method, where the temporal neighborhood is aggregated with equal attention coefﬁcients.

Original: GAT is like the time-agnostic version of our approach but with a different formulation for self-attention, as they refer to the work of Bahdanau et al.
Revised: GAT is like the time-agnostic version of our approach but with a different formulation for self-attention, as they refer to the work of Bahdanau et al.

Original: We discuss the differences in detail in the Appendix.
Revised: We discuss the differences in detail in the Appendix.

Original: It is also straightforward to show our connections with the menory networks (Sukhbaatar et al., 2015) by thinking of the temporal neighborhoods as memory.
Revised: It is also straightforward to show our connections with the menory networks (Sukhbaatar et al., 2015) by thinking of the temporal neighborhoods as memory.

Original: The techniques developed in our work may also help adapting
Revised: The techniques developed in our work may also help adapting

Original: GAT and GraphSAGE to temporal settings as we show in our experiments.
Revised: GAT and GraphSAGE to temporal settings as we show in our experiments.

Original:  E XPERIMENT AND R ESULTS
Revised:  E XPERIMENT AND R ESULTS

Original: We test the performance of the proposed method against a variety of strong baselines (adapted for temporal settings when possible) and competing approaches, for both the inductive and transductive tasks on two benchmark and one large-scale industrial dataset.
Revised: We test the performance of the proposed method against a variety of strong baselines (adapted for temporal settings when possible) and competing approaches, for both the inductive and transductive tasks on two benchmark and one large-scale industrial dataset.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: Real-world temporal graphs consist of time-sensitive node interactions, evolving node labels as well as new nodes and edges.
Revised: Real-world temporal graphs consist of time-sensitive node interactions, evolving node labels as well as new nodes and edges.

Original: We choose the following datasets which contain all scenarios.
Revised: We choose the following datasets which contain all scenarios.

Original: Reddit dataset .
Revised: Reddit dataset .

Original: We use the data from active users and their posts under subreddits, leading to a temporal graph with 11,000 nodes, ∼ 700,000 temporal edges and dynamic labels indicating whether a user is banned from posting.
Revised: We use the data from active users and their posts under subreddits, leading to a temporal graph with 11,000 nodes, ∼ 700,000 temporal edges and dynamic labels indicating whether a user is banned from posting.

Original: The user posts are transformed into edge feature vectors.
Revised: The user posts are transformed into edge feature vectors.

Original: Wikipedia dataset .
Revised: Wikipedia dataset .

Original: We use the data from top edited pages and active users, yielding a temporal graph ∼ 9,300 nodes and around 160,000 temporal edges.
Revised: We use the data from top edited pages and active users, yielding a temporal graph ∼ 9,300 nodes and around 160,000 temporal edges.

Original: Dynamic labels indicate if users are temporarily banned from editing.
Revised: Dynamic labels indicate if users are temporarily banned from editing.

Original: The user edits are also treated as edge features.
Revised: The user edits are also treated as edge features.

Original: Industrial dataset .
Revised: Industrial dataset .

Original: We choose 70,000 popular products and 100,000 active customers as nodes from the online grocery shopping website 4 and use the customer-product purchase as temporal edges ( ∼million).
Revised: We choose 70,000 popular products and 100,000 active customers as nodes from the online grocery shopping website 4 and use the customer-product purchase as temporal edges ( ∼million).

Original: The customers are tagged with labels indicating if they have a recent interest in dietary products.
Revised: The customers are tagged with labels indicating if they have a recent interest in dietary products.

Original: Product features are given by the pre-trained product embeddings (Xu et al., 2020).
Revised: Product features are given by the pre-trained product embeddings (Xu et al., 2020).

Original: We do the chronological train-validation-test split with 70%-15%-15% according to node interaction timestamps.
Revised: We do the chronological train-validation-test split with 70%-15%-15% according to node interaction timestamps.

Original: The dataset and preprocessing details are provided in the supplement material.
Revised: The dataset and preprocessing details are provided in the supplement material.

Original: 4. T RANSDUCTIVE AND INDUCTIVE LEARNING TASKS
Revised: 4. T RANSDUCTIVE AND INDUCTIVE LEARNING TASKS

Original: Since the majority of temporal information is reﬂected via the timely interactions among nodes, we choose to use a more revealing link prediction setup for training.
Revised: Since the majority of temporal information is reﬂected via the timely interactions among nodes, we choose to use a more revealing link prediction setup for training.

Original: Node classiﬁcation is then treated as the downstream task using the obtained time-aware node embeddings as input.
Revised: Node classiﬁcation is then treated as the downstream task using the obtained time-aware node embeddings as input.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Transductive task examines embeddings of the nodes that have been observed in training, via the future link prediction task and the node classiﬁcation.
Revised: Transductive task examines embeddings of the nodes that have been observed in training, via the future link prediction task and the node classiﬁcation.

Original: To avoid violating temporal constraints, we predict the links that strictly take place posterior to all observations in the training data.
Revised: To avoid violating temporal constraints, we predict the links that strictly take place posterior to all observations in the training data.

Original: Inductive task examines the inductive learning capability using the inferred representations of unseen nodes, by predicting the future links between unseen nodes and classify them based on their inferred embedding dynamically.
Revised: Inductive task examines the inductive learning capability using the inferred representations of unseen nodes, by predicting the future links between unseen nodes and classify them based on their inferred embedding dynamically.

Original: We point out that it sufﬁces to only consider the future sub-graph for unseen nodes since they are equivalent to new graphs under the non-temporal setting.
Revised: We point out that it sufﬁces to only consider the future sub-graph for unseen nodes since they are equivalent to new graphs under the non-temporal setting.

Original: As for the evaluation metrics , in the link prediction tasks, we ﬁrst sample an equal amount of negative node pairs to the positive links and then compute the average precision ( AP ) and classiﬁcation accuracy .
Revised: As for the evaluation metrics , in the link prediction tasks, we ﬁrst sample an equal amount of negative node pairs to the positive links and then compute the average precision ( AP ) and classiﬁcation accuracy .

Original: In the downstream node classiﬁcation tasks, due to the label imbalance in the datasets, we employ the area under the ROC curve ( AUC ).
Revised: In the downstream node classiﬁcation tasks, due to the label imbalance in the datasets, we employ the area under the ROC curve ( AUC ).

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: Transductive task : for link prediction of observed nodes, we choose the compare our approach with the state-of-the-art graph embedding methods: GAE and VGAE (Kipf & Welling, 2016b).
Revised: Transductive task : for link prediction of observed nodes, we choose the compare our approach with the state-of-the-art graph embedding methods: GAE and VGAE (Kipf & Welling, 2016b).

Original: For complete comparisons, we also include the skip-gram-based node2vec (Grover & Leskovec, 2016) as well as the spectral-based DeepWalk model (Perozzi et al., 2014), using the same inner-product decoder as GAE for link prediction.
Revised: For complete comparisons, we also include the skip-gram-based node2vec (Grover & Leskovec, 2016) as well as the spectral-based DeepWalk model (Perozzi et al., 2014), using the same inner-product decoder as GAE for link prediction.

Original: The CDTNE model based on the temporal random walk has been reported with superior performance on transductive learning tasks (Nguyen et al., 2018), so we include CDTNE as the representative for temporal graph embedding approaches.
Revised: The CDTNE model based on the temporal random walk has been reported with superior performance on transductive learning tasks (Nguyen et al., 2018), so we include CDTNE as the representative for temporal graph embedding approaches.

Original: Inductive task : few approaches are capable of managing inductive learning on graphs even in the non-temporal setting.
Revised: Inductive task : few approaches are capable of managing inductive learning on graphs even in the non-temporal setting.

Original: As a consequence, we choose GraphSAGE and GAT as baselines after adapting them to the temporal setting.
Revised: As a consequence, we choose GraphSAGE and GAT as baselines after adapting them to the temporal setting.

Original: In particular, we equip them with the same temporal sub-graph batching describe in Section 3.4 to maximize their usage on temporal information.
Revised: In particular, we equip them with the same temporal sub-graph batching describe in Section 3.4 to maximize their usage on temporal information.

Original: Also, we implement the extended version for the baselines to include edge features in the same way as ours (in Section 3.3).
Revised: Also, we implement the extended version for the baselines to include edge features in the same way as ours (in Section 3.3).

Original: We experiment on different aggregation functions for GraphSAGE , i.e. Graph-
Revised: We experiment on different aggregation functions for GraphSAGE , i.e. Graph-

Original: SAGE -mean, GraphSAGE -pool and GraphSAGE -LSTM.
Revised: SAGE -mean, GraphSAGE -pool and GraphSAGE -LSTM.

Original: In accordance with the original work of
Revised: In accordance with the original work of

Original: Hamilton et al. (2017a), GraphSAGE -LSTM gives the best validation performance among the three approaches, which is reasonable under temporal setting since LSTM aggregation takes account of the sequential information.
Revised: Hamilton et al. (2017a), GraphSAGE -LSTM gives the best validation performance among the three approaches, which is reasonable under temporal setting since LSTM aggregation takes account of the sequential information.

Original: Therefore we report the results of GraphSAGE -LSTM.
Revised: Therefore we report the results of GraphSAGE -LSTM.

Original: In addition to the above baselines
Revised: In addition to the above baselines

Original: [Table]
Revised: [Table]

Original: we implement a version of TGAT with all temporal attention weights set to equal value ( Const-TGAT ).
Revised: we implement a version of TGAT with all temporal attention weights set to equal value ( Const-TGAT ).

Original: Finally, to show that the superiority of our approach owes to both the time encoding and the network architecture, we experiment with the enhanced GAT and GraphSAGE -mean by concatenating the proposed time encoding to the original features during temporal aggregations ( GAT+T and GraphSAGE+T ).
Revised: Finally, to show that the superiority of our approach owes to both the time encoding and the network architecture, we experiment with the enhanced GAT and GraphSAGE -mean by concatenating the proposed time encoding to the original features during temporal aggregations ( GAT+T and GraphSAGE+T ).

Original: [Table]
Revised: [Table]

Original: is the AUC .
Revised: is the AUC .

Original: 4. E XPERIMENT SETUP
Revised: 4. E XPERIMENT SETUP

Original: We use the time-sensitive link prediction loss function for training the l -layer TGAT network:
Revised: We use the time-sensitive link prediction loss function for training the l -layer TGAT network:

Original: [Equation]
Revised: [Equation]

Original: where the summation is over the observed edges on v i and v j that interact at time t ij , and σ ( . ) is the sigmoid function, Q is the number of negative samples and P n ( v ) is the negative sampling distribution over the node space.
Revised: where the summation is over the observed edges on v i and v j that interact at time t ij , and σ ( . ) is the sigmoid function, Q is the number of negative samples and P n ( v ) is the negative sampling distribution over the node space.

Original: As for tuning hyper-parameters, we ﬁx the node embedding dimension and the time encoding dimension to be the original feature dimension for simplicity, and then select the number of TGAT layers from { 1,2,3 } , the number of attention heads from { 1,2,3,4,5 } , according to the link prediction AP score in the validation dataset.
Revised: As for tuning hyper-parameters, we ﬁx the node embedding dimension and the time encoding dimension to be the original feature dimension for simplicity, and then select the number of TGAT layers from { 1,2,3 } , the number of attention heads from { 1,2,3,4,5 } , according to the link prediction AP score in the validation dataset.

Original: Although our method does not put restriction on the neighborhood size during aggregations, to speed up training, specially when using the multi-hop aggregations, we use neighborhood dropout (selected among p = { 0.1, 0.3, 0.5 } ) with the uniform sampling.
Revised: Although our method does not put restriction on the neighborhood size during aggregations, to speed up training, specially when using the multi-hop aggregations, we use neighborhood dropout (selected among p = { 0.1, 0.3, 0.5 } ) with the uniform sampling.

Original: During training, we use 0 .
Revised: During training, we use 0 .

Original: 0001 as learning rate for Reddit and Wikipedia dataset and 0 .
Revised: 0001 as learning rate for Reddit and Wikipedia dataset and 0 .

Original: 001 for the industrial dataset, with Glorot initialization and the Adam SGD optimizer.
Revised: 001 for the industrial dataset, with Glorot initialization and the Adam SGD optimizer.

Original: We do not experiment on applying regularization since our approach is parameter-efﬁcient and only requires Ω (cid:0) ( d + d T )
Revised: We do not experiment on applying regularization since our approach is parameter-efﬁcient and only requires Ω (cid:0) ( d + d T )

Original: d h + ( d h + d 0 )
Revised: d h + ( d h + d 0 )

Original: d f + d f d (cid:1) parameters for each attention head, which is independent of the graph and neighborhood size.
Revised: d f + d f d (cid:1) parameters for each attention head, which is independent of the graph and neighborhood size.

Original: Using two TGAT layers and two attention heads with dropout rate as 0.1 give the best validation performance.
Revised: Using two TGAT layers and two attention heads with dropout rate as 0.1 give the best validation performance.

Original: For inference, we inductively compute the embeddings for both the unseen and observed nodes at each time point that the graph evolves, or when the node labels are updated.
Revised: For inference, we inductively compute the embeddings for both the unseen and observed nodes at each time point that the graph evolves, or when the node labels are updated.

Original: We then use these embeddings as features for the future link prediction and dynamic node classiﬁcations with multilayer perceptron.
Revised: We then use these embeddings as features for the future link prediction and dynamic node classiﬁcations with multilayer perceptron.

Original: We further conduct ablation study to demonstrate the effectiveness of the proposed functional time encoding approach.
Revised: We further conduct ablation study to demonstrate the effectiveness of the proposed functional time encoding approach.

Original: We experiment on abandoning time encoding or replacing it with the original positional encoding (both ﬁxed and learnt).
Revised: We experiment on abandoning time encoding or replacing it with the original positional encoding (both ﬁxed and learnt).

Original: We also compare the uniform neighborhood dropout to sampling with inverse timespan (where the recent edges are more likely to be sampled), which is provided in supplement material along with other implementation details and setups for baselines.
Revised: We also compare the uniform neighborhood dropout to sampling with inverse timespan (where the recent edges are more likely to be sampled), which is provided in supplement material along with other implementation details and setups for baselines.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: The results in Table 1 and Table 2 demonstrates the state-of-the-art performances of our approach on both transductive and inductive learning tasks.
Revised: The results in Table 1 and Table 2 demonstrates the state-of-the-art performances of our approach on both transductive and inductive learning tasks.

Original: In the inductive learning task, our TGAT network signiﬁcantly improves upon the the upgraded GraphSAGE -LSTM and GAT in accuracy and average precision by at least 5 % for both metrics, and in the transductive learning task TGAT consistently outperforms all baselines across datasets.
Revised: In the inductive learning task, our TGAT network signiﬁcantly improves upon the the upgraded GraphSAGE -LSTM and GAT in accuracy and average precision by at least 5 % for both metrics, and in the transductive learning task TGAT consistently outperforms all baselines across datasets.

Original: While GAT+T and GraphSAGE+T slightly outperform or tie with GAT and GraphSAGE -LSTM, they are nevertheless outperformed by our approach.
Revised: While GAT+T and GraphSAGE+T slightly outperform or tie with GAT and GraphSAGE -LSTM, they are nevertheless outperformed by our approach.

Original: On one hand, the results suggest that the time encoding have potential to extend non-temporal graph representation learning methods to temporal settings.
Revised: On one hand, the results suggest that the time encoding have potential to extend non-temporal graph representation learning methods to temporal settings.

Original: On the other, we note that the time encoding still works the best with our network architecture which is designed for temporal graphs.
Revised: On the other, we note that the time encoding still works the best with our network architecture which is designed for temporal graphs.

Original: Overall, the results demonstrate the superiority of our approach in learning representations on temporal graphs over prior models.
Revised: Overall, the results demonstrate the superiority of our approach in learning representations on temporal graphs over prior models.

Original: We also see the beneﬁts from assigning temporal attention weights to neighboring nodes, where GAT signiﬁcantly outperforms the Const-TGAT in all three tasks.
Revised: We also see the beneﬁts from assigning temporal attention weights to neighboring nodes, where GAT signiﬁcantly outperforms the Const-TGAT in all three tasks.

Original: The dynamic node classiﬁcation outcome (in Table 3) further suggests the usefulness of our time-aware node embeddings for downstream tasks as they surpass all the baselines.
Revised: The dynamic node classiﬁcation outcome (in Table 3) further suggests the usefulness of our time-aware node embeddings for downstream tasks as they surpass all the baselines.

Original: The ablation study results of Figure 3 successfully reveals the effectiveness of the proposed functional time encoding approach in capturing temporal signals as it outperforms the positional encoding counterparts.
Revised: The ablation study results of Figure 3 successfully reveals the effectiveness of the proposed functional time encoding approach in capturing temporal signals as it outperforms the positional encoding counterparts.

Original: 4. A TTENTION A NALYSIS
Revised: 4. A TTENTION A NALYSIS

Original: To shed some insights into the temporal signals captured by the proposed TGAT , we analyze the pattern of the attention weights { α ij ( t ) } as functions of both time t and node pairs ( i, j ) in the inference stage.
Revised: To shed some insights into the temporal signals captured by the proposed TGAT , we analyze the pattern of the attention weights { α ij ( t ) } as functions of both time t and node pairs ( i, j ) in the inference stage.

Original: Firstly , we analyze how the attention weights change with respect to the timespans of previous interactions, by plotting the attention weights (cid:8) α jq ( t ij ) | q ∈ N ( v j ; t ij ) (cid:9) ∪ (cid:8)
Revised: Firstly , we analyze how the attention weights change with respect to the timespans of previous interactions, by plotting the attention weights (cid:8) α jq ( t ij ) | q ∈ N ( v j ; t ij ) (cid:9) ∪ (cid:8)

Original: α ik ( t ij ) | k ∈
Revised: α ik ( t ij ) | k ∈

Original: N ( v i ; t ij ) (cid:9) against the timespans { t ij − t jq }∪{ t ij − t ik } when predicting the link for ( v i , v j , t ij ) ∈ E (Figure 4a).
Revised: N ( v i ; t ij ) (cid:9) against the timespans { t ij − t jq }∪{ t ij − t ik } when predicting the link for ( v i , v j , t ij ) ∈ E (Figure 4a).

Original: This gives us an empirical estimation on the α (∆ t ) , where a smaller ∆ t means a more recent interaction.
Revised: This gives us an empirical estimation on the α (∆ t ) , where a smaller ∆ t means a more recent interaction.

Original: Secondly , we analyze how the topological structures affect the attention weights as time elapses.
Revised: Secondly , we analyze how the topological structures affect the attention weights as time elapses.

Original: Speciﬁcally, we focus on the topological structure of the recurring neighbours , by ﬁnding out what attention weights the model put on the neighbouring nodes with different number of reoccurrences .
Revised: Speciﬁcally, we focus on the topological structure of the recurring neighbours , by ﬁnding out what attention weights the model put on the neighbouring nodes with different number of reoccurrences .

Original: Since the functional forms of all { α ij ( . ) } are ﬁxed after training, we are able to feed in different target time t and then record their value on neighbouring nodes with different number of occurrences (Figure 4b).
Revised: Since the functional forms of all { α ij ( . ) } are ﬁxed after training, we are able to feed in different target time t and then record their value on neighbouring nodes with different number of occurrences (Figure 4b).

Original: From Figure 4a we observe that TGAT captures the pattern of having less attention on more distant interactions in all three datasets.
Revised: From Figure 4a we observe that TGAT captures the pattern of having less attention on more distant interactions in all three datasets.

Original: In Figure 4b, it is obvious that when predicting a more future interaction, TGAT will consider neighbouring nodes who have a higher number of occurrences of more importance.
Revised: In Figure 4b, it is obvious that when predicting a more future interaction, TGAT will consider neighbouring nodes who have a higher number of occurrences of more importance.

Original: The patterns of the attention weights are meaningful, since the more recent and repeated actions often have larger inﬂuence on users’ future interests.
Revised: The patterns of the attention weights are meaningful, since the more recent and repeated actions often have larger inﬂuence on users’ future interests.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: We introduce a novel time-aware graph attention network for inductive representation learning on temporal graphs.
Revised: We introduce a novel time-aware graph attention network for inductive representation learning on temporal graphs.

Original: We adapt the self-attention mechanism to handle the continuous time by proposing a theoretically-grounded functional time encoding.
Revised: We adapt the self-attention mechanism to handle the continuous time by proposing a theoretically-grounded functional time encoding.

Original: Theoretical and experimental analysis demonstrate the effectiveness of our approach for capturing temporal-feature signals in terms of both node and topological features on temporal graphs.
Revised: Theoretical and experimental analysis demonstrate the effectiveness of our approach for capturing temporal-feature signals in terms of both node and topological features on temporal graphs.

Original: Self-attention mechanism often provides useful model interpretations (Vaswani et al., 2017), which is an important direction of our future work.
Revised: Self-attention mechanism often provides useful model interpretations (Vaswani et al., 2017), which is an important direction of our future work.

Original: Developing tools to visualize the evolving graph dynamics and temporal representations efﬁciently is another important direction for both research and application.
Revised: Developing tools to visualize the evolving graph dynamics and temporal representations efﬁciently is another important direction for both research and application.

Original: Also, the functional time encoding technique has huge potential for adapting other deep learning methods to the temporal graph domain.
Revised: Also, the functional time encoding technique has huge potential for adapting other deep learning methods to the temporal graph domain.

Original: A A PPENDIX A. PROOF FOR CLAIM
Revised: A A PPENDIX A. PROOF FOR CLAIM

Original: Proof.
Revised: Proof.

Original: The proof is also shown in our concurrent work Xu et al. (2019a).
Revised: The proof is also shown in our concurrent work Xu et al. (2019a).

Original: We also provide it here for completeness.
Revised: We also provide it here for completeness.

Original: To prove the results in Claim 1, we alternatively show that under the same condition,
Revised: To prove the results in Claim 1, we alternatively show that under the same condition,

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the score S ( t 1 , t 2 ) =
Revised: Deﬁne the score S ( t 1 , t 2 ) =

Original: Φ B d ( t 1 ) (cid:48)
Revised: Φ B d ( t 1 ) (cid:48)

Original: Φ B d ( t 2 ) .
Revised: Φ B d ( t 2 ) .

Original: The goal is to derive a uniform upper bound for s ( t 1 , t 2 ) − K ( t 1 , t 2 ) .
Revised: The goal is to derive a uniform upper bound for s ( t 1 , t 2 ) − K ( t 1 , t 2 ) .

Original: By assumption S ( t 1 , t 2 ) is an unbiased estimator for K ( t 1 , t 2 ) , i.e.
Revised: By assumption S ( t 1 , t 2 ) is an unbiased estimator for K ( t 1 , t 2 ) , i.e.

Original: E [ S ( t 1 , t 2 )]
Revised: E [ S ( t 1 , t 2 )]

Original: = K ( t 1 , t 2 ) .
Revised: = K ( t 1 , t 2 ) .

Original: Due to the translation-invariant property of S and K , we let ∆( t ) ≡ s ( t 1 , t 2 ) − K
Revised: Due to the translation-invariant property of S and K , we let ∆( t ) ≡ s ( t 1 , t 2 ) − K

Original: t 1 , t 2 ) , where t ≡ t 1 − t 2 for all t 1 , t 2 ∈ [0 , t max ] .
Revised: t 1 , t 2 ) , where t ≡ t 1 − t 2 for all t 1 , t 2 ∈ [0 , t max ] .

Original: Also we deﬁne s ( t 1 − t 2 ) :
Revised: Also we deﬁne s ( t 1 − t 2 ) :

Original: =
Revised: =

Original: S ( t 1 , t 2 ) .
Revised: S ( t 1 , t 2 ) .

Original: Therefore t ∈ [ − t max , t max ] , and we use t ∈ ˜ T as the shorthand notation.
Revised: Therefore t ∈ [ − t max , t max ] , and we use t ∈ ˜ T as the shorthand notation.

Original: The LHS in (1) now becomes Pr (cid:0) sup t ∈ ˜ T | ∆
Revised: The LHS in (1) now becomes Pr (cid:0) sup t ∈ ˜ T | ∆

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: | ≥
Revised: | ≥

Original: [Equation]
Revised: [Equation]

Original: (cid:1) .
Revised: (cid:1) .

Original: Note that ˜ T ⊆ ∪ N − 1 i =

Revised: Note that ˜ T ⊆ ∪ N − 1 i =

Original: T i with T i =
Revised: T i with T i =

Original: [ − t max + 

Revised: [ − t max +

Original: it max N , − t max + 2( i +1) t max N ] for i = 1 , . . .
Revised: it max N , − t max + 2( i +1) t max N ] for i = 1 , . . .

Original: , N .
Revised: , N .

Original: So ∪ N − 1 i =

Revised: So ∪ N − 1 i =

Original: T i is a ﬁnite cover of ˜ T .
Revised: T i is a ﬁnite cover of ˜ T .

Original: Deﬁne t
Revised: Deﬁne t

Original: i = − t max + (2 i +1) t max N , then for any t ∈ T i , i = 1 , . . .
Revised: i = − t max + (2 i +1) t max N , then for any t ∈ T i , i = 1 , . . .

Original: , N we have
Revised: , N we have

Original: [Equation]
Revised: [Equation]

Original: where L ∆ = max t ∈ ˜ T (cid:107)∇ ∆( t ) (cid:107) (since ∆ is differentiable) with the maximum achieved at t ∗ .
Revised: where L ∆ = max t ∈ ˜ T (cid:107)∇ ∆( t ) (cid:107) (since ∆ is differentiable) with the maximum achieved at t ∗ .

Original: So we may bound the two events separately.
Revised: So we may bound the two events separately.

Original: For | ∆( t i ) | we simply notice that trigeometric functions are bounded between [ − 1 , 1] , and therefore − 1 ≤ Φ B d ( t 1 ) (cid:48)
Revised: For | ∆( t i ) | we simply notice that trigeometric functions are bounded between [ − 1 , 1] , and therefore − 1 ≤ Φ B d ( t 1 ) (cid:48)

Original: Φ B d ( t 2 ) ≤ 1 .
Revised: Φ B d ( t 2 ) ≤ 1 .

Original: The Hoeffding’s inequality for bounded random variables immediately gives us:
Revised: The Hoeffding’s inequality for bounded random variables immediately gives us:

Original: [Equation]
Revised: [Equation]

Original: So applying the Hoeffding-type union bound to the ﬁnite cover gives
Revised: So applying the Hoeffding-type union bound to the ﬁnite cover gives

Original: [Equation]
Revised: [Equation]

Original: For the other event we ﬁrst apply Markov inequality and obtain:
Revised: For the other event we ﬁrst apply Markov inequality and obtain:

Original: [Equation]
Revised: [Equation]

Original: Also, since E [ s ( t 1 − t 2 )]
Revised: Also, since E [ s ( t 1 − t 2 )]

Original: = ψ ( t 1 − t 2 ) , we have
Revised: = ψ ( t 1 − t 2 ) , we have

Original: [Equation]
Revised: [Equation]

Original: where σ 2 p is the second momentum with respect to p ( ω ) .
Revised: where σ 2 p is the second momentum with respect to p ( ω ) .

Original: Combining (11), (12) and (11) gives us:
Revised: Combining (11), (12) and (11) gives us:

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to examine that the RHS of (14) is a convex function of N and is minimized by
Revised: It is straightforward to examine that the RHS of (14) is a convex function of N and is minimized by

Original: N ∗ =
Revised: N ∗ =

Original: σ p (cid:113)
Revised: σ p (cid:113)

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: exp ( d(cid:15) 2 32 ) .
Revised: exp ( d(cid:15) 2 32 ) .

Original: Plug N ∗ back to (14) and we obtain (9).
Revised: Plug N ∗ back to (14) and we obtain (9).

Original: We then solve for d according to (9) and obtain the results in Claim 1.
Revised: We then solve for d according to (9) and obtain the results in Claim 1.

Original: A. C OMPARISONS BETWEEN THE ATTENTION MECHANISM OF TGAT AND GAT
Revised: A. C OMPARISONS BETWEEN THE ATTENTION MECHANISM OF TGAT AND GAT

Original: In this part, we provide detailed comparisons between the attention mechanism employed by our proposed TGAT and the GAT proposed by Veliˇckovi´c et al.
Revised: In this part, we provide detailed comparisons between the attention mechanism employed by our proposed TGAT and the GAT proposed by Veliˇckovi´c et al.

Original: Other than the obvious fact that
Revised: Other than the obvious fact that

Original: GAT does not handle temporal information, the main difference lies in the formulation of attention weights.
Revised: GAT does not handle temporal information, the main difference lies in the formulation of attention weights.

Original: While GAT depends on the attention mechanism proposed by Bahdanau et al.
Revised: While GAT depends on the attention mechanism proposed by Bahdanau et al.

Original: , our architecture refers to the self-attention mechanism of Vaswani et al.
Revised: , our architecture refers to the self-attention mechanism of Vaswani et al.

Original: Firstly, the attention mechanism used by GAT does not involve the notions of ’query’, ’key’ and ’value’ nor the dotproduct formulation introduced in (2).
Revised: Firstly, the attention mechanism used by GAT does not involve the notions of ’query’, ’key’ and ’value’ nor the dotproduct formulation introduced in (2).

Original: As a consequence, the attention weight between node v i and its neighbor v j is computed via
Revised: As a consequence, the attention weight between node v i and its neighbor v j is computed via

Original: [Equation]
Revised: [Equation]

Original: where a is a weight vector, W is a weight matrix, N ( v i ) is the neighorhood set for node v i and h
Revised: where a is a weight vector, W is a weight matrix, N ( v i ) is the neighorhood set for node v i and h

Original: i is the hidden representation of node v i .
Revised: i is the hidden representation of node v i .

Original: It is then obvious that their computation of α ij is very different from our approach.
Revised: It is then obvious that their computation of α ij is very different from our approach.

Original: In TGAT , after expanding the expressions in Section 3, the attention weight is computed by:
Revised: In TGAT , after expanding the expressions in Section 3, the attention weight is computed by:

Original: [Equation]
Revised: [Equation]

Original: Intuitively speaking, the attention mechanism of GAT relies on the parameter vector a and the
Revised: Intuitively speaking, the attention mechanism of GAT relies on the parameter vector a and the

Original: LeakyReLU(.) to capture the hidden factor interactions between entities in the sequence, while we use the linear transformation followed by the dot-product to capture pair-wise interactions of the hidden factors between entities and the time embeddings.
Revised: LeakyReLU(.) to capture the hidden factor interactions between entities in the sequence, while we use the linear transformation followed by the dot-product to capture pair-wise interactions of the hidden factors between entities and the time embeddings.

Original: The dot-product formulation is important for our approach.
Revised: The dot-product formulation is important for our approach.

Original: From the theoretical perspective, the time encoding functional form is derived according to the notion of temporal kernel K and its inner-product decomposition (Section 3).
Revised: From the theoretical perspective, the time encoding functional form is derived according to the notion of temporal kernel K and its inner-product decomposition (Section 3).

Original: As for the practical performances, we see from Table 1, 2 and 3 that even after we equip GAT with the same time encoding, the performance is still inferior to our TGAT .
Revised: As for the practical performances, we see from Table 1, 2 and 3 that even after we equip GAT with the same time encoding, the performance is still inferior to our TGAT .

Original: A. D ETAILS ON DATASETS AND PREPROCESSING
Revised: A. D ETAILS ON DATASETS AND PREPROCESSING

Original: Reddit dataset : this benchmark dataset contains users interacting with subreddits by posting under the subreddits.
Revised: Reddit dataset : this benchmark dataset contains users interacting with subreddits by posting under the subreddits.

Original: The timestamps tell us when the user makes the posts.
Revised: The timestamps tell us when the user makes the posts.

Original: The dataset uses the posts made in a one-month span, and selects the most active users and subreddits as nodes, giving a total of 11,000 nodes and around 700,000 temporal edges.
Revised: The dataset uses the posts made in a one-month span, and selects the most active users and subreddits as nodes, giving a total of 11,000 nodes and around 700,000 temporal edges.

Original: The user posts have textual features that are transformed into a 172-dimensional vector representing under the linguistic inquiry and word count (LIWC) categories (Pennebaker et al., 2001).
Revised: The user posts have textual features that are transformed into a 172-dimensional vector representing under the linguistic inquiry and word count (LIWC) categories (Pennebaker et al., 2001).

Original: The dynamic binary labels indicate if a user is banned from posting under a subreddit.
Revised: The dynamic binary labels indicate if a user is banned from posting under a subreddit.

Original: Since node features are not provided in the original dataset, we use the all-zero vector instead.
Revised: Since node features are not provided in the original dataset, we use the all-zero vector instead.

Original: Wikipedia dataset : the dataset also collects one-month of interactions induced by users’ editing the
Revised: Wikipedia dataset : the dataset also collects one-month of interactions induced by users’ editing the

Original: Wikipedia pages.
Revised: Wikipedia pages.

Original: The the top edited pages and active users are considered, leading to ∼ 9,300 nodes and around 160,000 temporal edges.
Revised: The the top edited pages and active users are considered, leading to ∼ 9,300 nodes and around 160,000 temporal edges.

Original: Similar to the Reddit dataset, we also have the ground-truth dynamic labels on whether a user is banned from editing a Wikipedia page.
Revised: Similar to the Reddit dataset, we also have the ground-truth dynamic labels on whether a user is banned from editing a Wikipedia page.

Original: User edits consist of the textual features and are also converted into 172-dimensional LIWC feature vectors.
Revised: User edits consist of the textual features and are also converted into 172-dimensional LIWC feature vectors.

Original: Node features are also not provided, so we also use the all-zero vector as well.
Revised: Node features are also not provided, so we also use the all-zero vector as well.

Original: Industrial dataset : we obtain the large-scale customer-product interaction graph from the online grocery shopping platform grocery.walmart.com .
Revised: Industrial dataset : we obtain the large-scale customer-product interaction graph from the online grocery shopping platform grocery.walmart.com .

Original: We select ∼ 70,000 most popular products and 100,000 active customers as nodes and use the customer-product purchase interactions over a one-month period as temporal edges ( ∼ 2 million).
Revised: We select ∼ 70,000 most popular products and 100,000 active customers as nodes and use the customer-product purchase interactions over a one-month period as temporal edges ( ∼ 2 million).

Original: Each purchase interaction is timestamped, which we use to construct the temporal graph.
Revised: Each purchase interaction is timestamped, which we use to construct the temporal graph.

Original: The customers are labelled with business tags, indicating if they are interested in dietary products according to their most recent purchase records.
Revised: The customers are labelled with business tags, indicating if they are interested in dietary products according to their most recent purchase records.

Original: Each product node possesses contextual features containing their name, brand, categories and short description.
Revised: Each product node possesses contextual features containing their name, brand, categories and short description.

Original: The previous LIWC categories no longer apply since the product contextual features are not natural sentences.
Revised: The previous LIWC categories no longer apply since the product contextual features are not natural sentences.

Original: We use product embedding approach (Xu et al., 2020) to embed each product’s contextual
Revised: We use product embedding approach (Xu et al., 2020) to embed each product’s contextual

Original: Reddit
Revised: Reddit

Original: Wikipedia
Revised: Wikipedia

Original: Industrial # Nodes 11, 9, 170, # Edges 672, 157, 2,135, # Feature dimension   # Feature type
Revised: Industrial # Nodes 11, 9, 170, # Edges 672, 157, 2,135, # Feature dimension   # Feature type

Original: LIWC category vector
Revised: LIWC category vector

Original: LIWC category vector documentembeddings # Timespan 30 days 30 days 30 days %
Revised: LIWC category vector documentembeddings # Timespan 30 days 30 days 30 days %

Original: Training nodes 90% 90% 90% % Unseen nodes 10% 10% 10% %
Revised: Training nodes 90% 90% 90% % Unseen nodes 10% 10% 10% %

Original: Training edges ∼ 67% ∼ 65% ∼ 64% %
Revised: Training edges ∼ 67% ∼ 65% ∼ 64% %

Original: Future edges between observed nodes ∼ 27% ∼ 28% ∼ 29% %
Revised: Future edges between observed nodes ∼ 27% ∼ 28% ∼ 29% %

Original: Future edges between unseen nodes ∼ 6% ∼ 7% ∼ 7% # Nodes with dynamic labels  5,
Revised: Future edges between unseen nodes ∼ 6% ∼ 7% ∼ 7% # Nodes with dynamic labels  5,

Original: Label type binary binary binary
Revised: Label type binary binary binary

Original: Positive label meaning banned from posting banned from editting interested in dietary products features into a 100-dimensional vector space as preprocessing.
Revised: Positive label meaning banned from posting banned from editting interested in dietary products features into a 100-dimensional vector space as preprocessing.

Original: The user nodes and edges do not possess features.
Revised: The user nodes and edges do not possess features.

Original: We then split the temporal graphs chronologically into 70%-15%-15% for training, validation and testing according to the time epochs of edges, as illustrated in Figure 5 with the Reddit dataset.
Revised: We then split the temporal graphs chronologically into 70%-15%-15% for training, validation and testing according to the time epochs of edges, as illustrated in Figure 5 with the Reddit dataset.

Original: Since all three datasets have a relatively stationary edge count distribution over time, using the 70 andpercentile time points to split the dataset results in approximately 70%-15%-15% of total edges, as suggested by Figure 5.
Revised: Since all three datasets have a relatively stationary edge count distribution over time, using the 70 andpercentile time points to split the dataset results in approximately 70%-15%-15% of total edges, as suggested by Figure 5.

Original: To ensure that an appropriate amount of future edges among the unseen nodes will show up during validation and testing, for each dataset, we randomly sample 10% of nodes, mask them during training and treat them as unseen nodes by only considering their interactions in validation and testing period.
Revised: To ensure that an appropriate amount of future edges among the unseen nodes will show up during validation and testing, for each dataset, we randomly sample 10% of nodes, mask them during training and treat them as unseen nodes by only considering their interactions in validation and testing period.

Original: This manipulation is necessary since the new nodes that show up during validation and testing period may not have much interaction among themselves.
Revised: This manipulation is necessary since the new nodes that show up during validation and testing period may not have much interaction among themselves.

Original: The statistics for the three datasets are summarized in Table 4.
Revised: The statistics for the three datasets are summarized in Table 4.

Original: Preprocessing.
Revised: Preprocessing.

Original: For the Node2vec
Revised: For the Node2vec

Original: and DeepWalk baselines who only take static graphs as input, the graph is constructed using all edges in training data regardless of temporal information.
Revised: and DeepWalk baselines who only take static graphs as input, the graph is constructed using all edges in training data regardless of temporal information.

Original: For DeepWalk , we treat the recurrent edges as appearing only once, so the graph is unweighted.
Revised: For DeepWalk , we treat the recurrent edges as appearing only once, so the graph is unweighted.

Original: Although our approach handles both directed and undirected graphs, for the sake of training stability of the baselines, we treat the graphs as undirected .
Revised: Although our approach handles both directed and undirected graphs, for the sake of training stability of the baselines, we treat the graphs as undirected .

Original: For Node2vec , we use the count of recurrent edges as their weights and construct the weighted graph.
Revised: For Node2vec , we use the count of recurrent edges as their weights and construct the weighted graph.

Original: For all three datasets, the obtained graphs in both cases are undirected and do not have isolated nodes.
Revised: For all three datasets, the obtained graphs in both cases are undirected and do not have isolated nodes.

Original: Since we choose from active users and popular items, the graphs are all connected .
Revised: Since we choose from active users and popular items, the graphs are all connected .

Original: For the graph convolutional network baselines, i.e. GAE and VGAE , we construct the same undirected weighted graph as for Node2vec .
Revised: For the graph convolutional network baselines, i.e. GAE and VGAE , we construct the same undirected weighted graph as for Node2vec .

Original: Since GAE and VGAE do not take edge features as input, we use the posts/edits as user node features.
Revised: Since GAE and VGAE do not take edge features as input, we use the posts/edits as user node features.

Original: For each user in Reddit and Wikipedia dataset, we take the average of their post/edit feature vectors as the node feature.
Revised: For each user in Reddit and Wikipedia dataset, we take the average of their post/edit feature vectors as the node feature.

Original: For the industrial dataset where user features are not available, we use the all-zero feature vector instead.
Revised: For the industrial dataset where user features are not available, we use the all-zero feature vector instead.

Original: As for the downstream dynamic node classiﬁcation task, we use the same training, validation and testing dataset as above.
Revised: As for the downstream dynamic node classiﬁcation task, we use the same training, validation and testing dataset as above.

Original: Since we aim at predicting the dynamic node labels, for Reddit and
Revised: Since we aim at predicting the dynamic node labels, for Reddit and

Original: Wikipedia dataset we predict if the user node is banned and for the industrial dataset we predict the customers’ business labels, at different time points.
Revised: Wikipedia dataset we predict if the user node is banned and for the industrial dataset we predict the customers’ business labels, at different time points.

Original: Due to the label imbalance, in each of the batch when training for the node label classiﬁer, we conduct stratiﬁed sampling such that the label distributions are similar across batches.
Revised: Due to the label imbalance, in each of the batch when training for the node label classiﬁer, we conduct stratiﬁed sampling such that the label distributions are similar across batches.

Original: A. E XPERIMENT S ETUP FOR B ASELINES
Revised: A. E XPERIMENT S ETUP FOR B ASELINES

Original: For all baselines, we set the node embedding dimension to d = 100 to keep in accordance with our approach.
Revised: For all baselines, we set the node embedding dimension to d = 100 to keep in accordance with our approach.

Original: Transductive baselines.
Revised: Transductive baselines.

Original: Since Node2vec and DeepWalk do not provide room for task-speciﬁc manipulation or hacking, we do not modify their default loss function and input format.
Revised: Since Node2vec and DeepWalk do not provide room for task-speciﬁc manipulation or hacking, we do not modify their default loss function and input format.

Original: For both approaches, we select the number of walks among { 60,80,100 } and the walk-length among { 20,30,40 } according to the validation
Revised: For both approaches, we select the number of walks among { 60,80,100 } and the walk-length among { 20,30,40 } according to the validation

Original: AP .
Revised: AP .

Original: Setting number of walks =80 and walk-length =30 give slightly better validation performance compared to others for both approaches.
Revised: Setting number of walks =80 and walk-length =30 give slightly better validation performance compared to others for both approaches.

Original: Notice that both Node2vec and DeepWalk use the sigmoid function with embedding inner-products as the decoder to predict neighborhood probabilities.
Revised: Notice that both Node2vec and DeepWalk use the sigmoid function with embedding inner-products as the decoder to predict neighborhood probabilities.

Original: So when predicting whether v i and v j will interact in the future, we use σ ( − z (cid:124) i
Revised: So when predicting whether v i and v j will interact in the future, we use σ ( − z (cid:124) i

Original: z j ) as the score, where z i and z j are the node embeddings.
Revised: z j ) as the score, where z i and z j are the node embeddings.

Original: Notice that Node2vec has the extra hyper-parameter p and q which controls the likelihood of immediately revisiting a node in the walk and interpolation between breadth-ﬁrst strategy and depth-ﬁrst strategy.
Revised: Notice that Node2vec has the extra hyper-parameter p and q which controls the likelihood of immediately revisiting a node in the walk and interpolation between breadth-ﬁrst strategy and depth-ﬁrst strategy.

Original: After selecting the optimal number of walks and walklength under p = 1 and q = 1 , we further tune the different values of p in { 0.2,0.4,0.6,0.8,1.0 } while ﬁxing q = 1 .
Revised: After selecting the optimal number of walks and walklength under p = 1 and q = 1 , we further tune the different values of p in { 0.2,0.4,0.6,0.8,1.0 } while ﬁxing q = 1 .

Original: According to validation, p = 0 . 6 and 0 . 8 give comparable optimal performance.
Revised: According to validation, p = 0 . 6 and 0 . 8 give comparable optimal performance.

Original: For the GAE and VGAE baselines, we experiment on using one, two and three graph convolutional layers as the encoder (Kipf & Welling, 2016a) and use the ReLU ( . )
Revised: For the GAE and VGAE baselines, we experiment on using one, two and three graph convolutional layers as the encoder (Kipf & Welling, 2016a) and use the ReLU ( . )

Original: as the activation function.
Revised: as the activation function.

Original: By referencing the ofﬁcial implementation, we also set the dimension of hidden layers to 200.
Revised: By referencing the ofﬁcial implementation, we also set the dimension of hidden layers to 200.

Original: Similar to previous ﬁndings, using two layers gives signiﬁcant performances to using only one layer.
Revised: Similar to previous ﬁndings, using two layers gives signiﬁcant performances to using only one layer.

Original: Adding the third layer, on the other hand, shows almost identical results for both models.
Revised: Adding the third layer, on the other hand, shows almost identical results for both models.

Original: Therefore the results reported are based on two-layer GCN as the encoder.
Revised: Therefore the results reported are based on two-layer GCN as the encoder.

Original: For GAE , we use the standard inner-product decoder as our approach and optimize over the reconstruction loss, and for VGAE , we restrict the Gaussian latent factor space (Kipf & Welling, 2016b).
Revised: For GAE , we use the standard inner-product decoder as our approach and optimize over the reconstruction loss, and for VGAE , we restrict the Gaussian latent factor space (Kipf & Welling, 2016b).

Original: Since we have eliminated the temporal information when constructing the input, we ﬁnd that the optimal hyper-parameters selected according to the tuning have similar patterns as in the previous non-temporal settings.
Revised: Since we have eliminated the temporal information when constructing the input, we ﬁnd that the optimal hyper-parameters selected according to the tuning have similar patterns as in the previous non-temporal settings.

Original: For the temporal network embedding model CTDNE , the walk length for the temporal random walk is also selected among { 60,80,100 } , where setting walk length to 80 gives slightly better validation outcome.
Revised: For the temporal network embedding model CTDNE , the walk length for the temporal random walk is also selected among { 60,80,100 } , where setting walk length to 80 gives slightly better validation outcome.

Original: The original paper considers several temporal edge selection (sampling) methods (uniform, linear and exponential) and ﬁnds uniform sampling with best performances (Nguyen et al., 2018).
Revised: The original paper considers several temporal edge selection (sampling) methods (uniform, linear and exponential) and ﬁnds uniform sampling with best performances (Nguyen et al., 2018).

Original: Since our setting is similar to theirs, we adopt the uniform sampling approach.
Revised: Since our setting is similar to theirs, we adopt the uniform sampling approach.

Original: Inductive baselines.
Revised: Inductive baselines.

Original: For the GraphSAGE and GAT baselines, as mentioned before, we train the models in an identical way as our approach with the temporal subgraph batching , despite several slight differences.
Revised: For the GraphSAGE and GAT baselines, as mentioned before, we train the models in an identical way as our approach with the temporal subgraph batching , despite several slight differences.

Original: Firstly, the aggregation layers in GraphSAGE usually considers a ﬁxed neighborhood size via sampling, whereas our approach can take an arbitrary neighborhood as input.
Revised: Firstly, the aggregation layers in GraphSAGE usually considers a ﬁxed neighborhood size via sampling, whereas our approach can take an arbitrary neighborhood as input.

Original: Therefore, we only consider the most recent d sample edges during each aggregation for all layers, and we ﬁnd d sample = 20 gives the best performance among { 10,15,20,25 } .
Revised: Therefore, we only consider the most recent d sample edges during each aggregation for all layers, and we ﬁnd d sample = 20 gives the best performance among { 10,15,20,25 } .

Original: Secondly, GAT implements a uniform neighborhood dropout.
Revised: Secondly, GAT implements a uniform neighborhood dropout.

Original: We also experiment with the inverse timespan sampling for neighborhood dropout, and ﬁnd that it gives slightly better performances but at the cost of computational efﬁciency, especially for large graphs.
Revised: We also experiment with the inverse timespan sampling for neighborhood dropout, and ﬁnd that it gives slightly better performances but at the cost of computational efﬁciency, especially for large graphs.

Original: We consider aggregating over one, two and three-hop neighborhood for both GAT and
Revised: We consider aggregating over one, two and three-hop neighborhood for both GAT and

Original: GraphSAGE .
Revised: GraphSAGE .

Original: When working with three hops, we only experiment on GraphSAGE with the mean pooling aggregation.
Revised: When working with three hops, we only experiment on GraphSAGE with the mean pooling aggregation.

Original: In general, using two hops gives comparable performance to using three hops.
Revised: In general, using two hops gives comparable performance to using three hops.

Original: Notice that computations with three-hop are costly, since the number of edges during aggregation increase exponentially to the number of hops.
Revised: Notice that computations with three-hop are costly, since the number of edges during aggregation increase exponentially to the number of hops.

Original: Thus we stick to using two hops for GraphSAGE ,
Revised: Thus we stick to using two hops for GraphSAGE ,

Original: GAT and our approach.
Revised: GAT and our approach.

Original: It is worth mentioning that when implementing GraphSAGE -LSTM, the input neighborhood sequences of LSTM are also ordered by their interaction time.
Revised: It is worth mentioning that when implementing GraphSAGE -LSTM, the input neighborhood sequences of LSTM are also ordered by their interaction time.

Original: Node classiﬁcation with baselines.
Revised: Node classiﬁcation with baselines.

Original: The dynamic node classiﬁcation with GraphSAGE and GAT can be conducted similarity to our approach, where we inductively compute the most up-to-date node embeddings and then input them as features to an MLP classiﬁer.
Revised: The dynamic node classiﬁcation with GraphSAGE and GAT can be conducted similarity to our approach, where we inductively compute the most up-to-date node embeddings and then input them as features to an MLP classiﬁer.

Original: For the transductive baselines, it is not reasonable to predict the dynamic node labels with only the ﬁxed node embeddings.
Revised: For the transductive baselines, it is not reasonable to predict the dynamic node labels with only the ﬁxed node embeddings.

Original: Instead, we combine the node embedding with the other node embedding it is interacting with when the label changes, e.g. combine the user embedding with the Wikipedia page embedding that the user attempts on editing when the system bans the user.
Revised: Instead, we combine the node embedding with the other node embedding it is interacting with when the label changes, e.g. combine the user embedding with the Wikipedia page embedding that the user attempts on editing when the system bans the user.

Original: To combine the pair of node embeddings, we experimented on summation, concatenation and bi-linear transformation.
Revised: To combine the pair of node embeddings, we experimented on summation, concatenation and bi-linear transformation.

Original: Under summation and concatenation, the combined embeddings are then used as input to an MLP classiﬁer, where the bi-linear transformation directly outputs scores for classiﬁcation.
Revised: Under summation and concatenation, the combined embeddings are then used as input to an MLP classiﬁer, where the bi-linear transformation directly outputs scores for classiﬁcation.

Original: The validation outcomes suggest that using concatenation with
Revised: The validation outcomes suggest that using concatenation with

Original: MLP yields the best performance.
Revised: MLP yields the best performance.

Original: A. I MPLEMENTATION DETAILS
Revised: A. I MPLEMENTATION DETAILS

Original: Training.
Revised: Training.

Original: We implement Node2vec using the ofﬁcial C code 5 on a 16-core Linux server with
Revised: We implement Node2vec using the ofﬁcial C code 5 on a 16-core Linux server with

Original: Gb memory.
Revised: Gb memory.

Original: DeepWalk is implemented with the ofﬁcial python code 6 .
Revised: DeepWalk is implemented with the ofﬁcial python code 6 .

Original: We refer to the PyTorch geometric library for implementing the GAE and VGAE baselines (Fey & Lenssen, 2019).
Revised: We refer to the PyTorch geometric library for implementing the GAE and VGAE baselines (Fey & Lenssen, 2019).

Original: To accommodate the temporal setting and incorporate edges features, we develop off-the-shelf implementation for GraphSAGE and GAT in PyTorch by referencing their original implementations 7 8 .
Revised: To accommodate the temporal setting and incorporate edges features, we develop off-the-shelf implementation for GraphSAGE and GAT in PyTorch by referencing their original implementations 7 8 .

Original: We also implement our model using PyTorch.
Revised: We also implement our model using PyTorch.

Original: All the deep learning models are trained on a machine with one Tesla V100 GPU.
Revised: All the deep learning models are trained on a machine with one Tesla V100 GPU.

Original: We use the Glorot initialization and the Adam SGD optimizer for all models, and apply the early-stopping strategy during training where we terminate the training process if the validation AP score does not improve for 10 epochs.
Revised: We use the Glorot initialization and the Adam SGD optimizer for all models, and apply the early-stopping strategy during training where we terminate the training process if the validation AP score does not improve for 10 epochs.

Original: Downstream node classiﬁcation.
Revised: Downstream node classiﬁcation.

Original: As we discussed before, we use the three-layer MLP as classiﬁer and the (combined) node embeddings as input features from all the experimented approaches, for all three datasets.
Revised: As we discussed before, we use the three-layer MLP as classiﬁer and the (combined) node embeddings as input features from all the experimented approaches, for all three datasets.

Original: The MLP is trained with the Glorot initialization and the Adam SGD optimizer in PyTorch as well.
Revised: The MLP is trained with the Glorot initialization and the Adam SGD optimizer in PyTorch as well.

Original: The (cid:96) 2 regularization parameter λ is selected in { 0.001, 0.01, 0.05, 0.1, 0.2 } case-by-case during training.
Revised: The (cid:96) 2 regularization parameter λ is selected in { 0.001, 0.01, 0.05, 0.1, 0.2 } case-by-case during training.

Original: The early-stopping strategy is also employed.
Revised: The early-stopping strategy is also employed.

Original: A. S ENSITIVITY ANALYSIS AND EXTRA ABLATION STUDY
Revised: A. S ENSITIVITY ANALYSIS AND EXTRA ABLATION STUDY

Original: Firstly, we focus on the output node embedding dimension as well as the functional time encoding dimension in this sensitivity analysis.
Revised: Firstly, we focus on the output node embedding dimension as well as the functional time encoding dimension in this sensitivity analysis.

Original: The reported results are averaged over ﬁve runs.
Revised: The reported results are averaged over ﬁve runs.

Original: We experiment on d ∈ { 60 , 80 , 100 , 120 , 140 } and d T ∈ { 60 , 80 , 100 , 120 , 140 } , and the results are reported in Figure 7a and 7c.
Revised: We experiment on d ∈ { 60 , 80 , 100 , 120 , 140 } and d T ∈ { 60 , 80 , 100 , 120 , 140 } , and the results are reported in Figure 7a and 7c.

Original: The remaining model setups reported in Section 4.4 are untouched when varying d or d T .
Revised: The remaining model setups reported in Section 4.4 are untouched when varying d or d T .

Original: We observe slightly better outcome when increasing either d or d T on the industrial dataset.
Revised: We observe slightly better outcome when increasing either d or d T on the industrial dataset.

Original: The patterns on Reddit and Wikipedia dataset are almost identical.
Revised: The patterns on Reddit and Wikipedia dataset are almost identical.

Original: Secondly, we compare between the two methods of learning functional encoding, i.e. using ﬂowbased model or using the non-parametric method introduced in Section 3.1.
Revised: Secondly, we compare between the two methods of learning functional encoding, i.e. using ﬂowbased model or using the non-parametric method introduced in Section 3.1.

Original: We experiment on two
Revised: We experiment on two

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ﬂow-based state-of-the-art CDF learning method: normalizing ﬂow (Rezende & Mohamed, 2015) and
Revised: ﬂow-based state-of-the-art CDF learning method: normalizing ﬂow (Rezende & Mohamed, 2015) and

Original: RealNVP (Dinh et al., 2016).
Revised: RealNVP (Dinh et al., 2016).

Original: We use the default model setups and hyper-parameters in their reference implementations 9 10 .
Revised: We use the default model setups and hyper-parameters in their reference implementations 9 10 .

Original: We provide the results in Figure 6b.
Revised: We provide the results in Figure 6b.

Original: As we mentioned before, using ﬂow-based models leads to highly comparable outcomes as the non-parametric approach, but they require longer training time since they implement sampling during each training batch.
Revised: As we mentioned before, using ﬂow-based models leads to highly comparable outcomes as the non-parametric approach, but they require longer training time since they implement sampling during each training batch.

Original: However, it is possible that carefully-tuned ﬂow-based models can lead to nontrivial improvements, which we leave to the future work.
Revised: However, it is possible that carefully-tuned ﬂow-based models can lead to nontrivial improvements, which we leave to the future work.

Original: Finally, we provide sensitivity analysis on the number of attention heads and layers for TGAT .
Revised: Finally, we provide sensitivity analysis on the number of attention heads and layers for TGAT .

Original: Recall that by stacking two layers in TGAT we are aggregating information from the two-hop neighbourhood.
Revised: Recall that by stacking two layers in TGAT we are aggregating information from the two-hop neighbourhood.

Original: For both accuracy and AP , using three-head attention and two-layers gives the best outcome.
Revised: For both accuracy and AP , using three-head attention and two-layers gives the best outcome.

Original: In general, the results are relatively stable to the number of heads, and stacking two layers leads to signiﬁcant improvements compared with using only a single layer.
Revised: In general, the results are relatively stable to the number of heads, and stacking two layers leads to signiﬁcant improvements compared with using only a single layer.

Original: The ablation study for comparing between uniform neighborhood dropout and sampling with inverse timespan is given in Figure 6a.
Revised: The ablation study for comparing between uniform neighborhood dropout and sampling with inverse timespan is given in Figure 6a.

Original: The two experiments are carried out under the same setting which we reported in Section 4.4.
Revised: The two experiments are carried out under the same setting which we reported in Section 4.4.

Original: We see that using the inverse timespan sampling gives slightly worse performances.
Revised: We see that using the inverse timespan sampling gives slightly worse performances.

Original: This is within expectation since uniform sampling has advantage in capturing the recurrent patterns, which can be important for predicting user actions.
Revised: This is within expectation since uniform sampling has advantage in capturing the recurrent patterns, which can be important for predicting user actions.

Original: On the other hand, the results also suggest the effectiveness of the proposed time encoding for capturing such temporal patterns.
Revised: On the other hand, the results also suggest the effectiveness of the proposed time encoding for capturing such temporal patterns.

Original: Moreover, we point out that using the inverse timespan sampling slows down training, particularly for large graphs where a weighted sampling is conducted within a large number of nodes for each training batch construction.
Revised: Moreover, we point out that using the inverse timespan sampling slows down training, particularly for large graphs where a weighted sampling is conducted within a large number of nodes for each training batch construction.

Original: Nonetheless, inverse timespan sampling can help capturing the more recent interactions which may be more useful for certain tasks.
Revised: Nonetheless, inverse timespan sampling can help capturing the more recent interactions which may be more useful for certain tasks.

Original: Therefore, we suggest to choose the neighborhood dropout method according to the speciﬁc use cases.
Revised: Therefore, we suggest to choose the neighborhood dropout method according to the speciﬁc use cases.