Original: T RANSFORMER E MBEDDINGS OF I RREGULARLY S PACED E VENTS AND T HEIR P ARTICIPANTS Anonymous authors
Revised: T RANSFORMER E MBEDDINGS OF I RREGULARLY S PACED E VENTS AND T HEIR P ARTICIPANTS Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The Transformer (Vaswani et al., 2017) has enjoyed remarkable success at learning representations for discrete-time sequences, such as natural language sentences.
Revised: The Transformer (Vaswani et al., 2017) has enjoyed remarkable success at learning representations for discrete-time sequences, such as natural language sentences.

Original: Yet there exists little work on generalizing its success to sequences of discrete events in continuous time.
Revised: Yet there exists little work on generalizing its success to sequences of discrete events in continuous time.

Original: We design a generative point process model for this case based on a continuous-time Transformer .
Revised: We design a generative point process model for this case based on a continuous-time Transformer .

Original: An actual or possible event at time t is embedded using attention over actual events at times < t .
Revised: An actual or possible event at time t is embedded using attention over actual events at times < t .

Original: More generally, we propose a new way to compute fact embeddings and event probabilities for neural
Revised: More generally, we propose a new way to compute fact embeddings and event probabilities for neural

Original: Datalog through time (Mei et al., 2020a), a neural-symbolic framework for event sequence modeling that automatically derives deep recurrent neural models from temporal logic programs.
Revised: Datalog through time (Mei et al., 2020a), a neural-symbolic framework for event sequence modeling that automatically derives deep recurrent neural models from temporal logic programs.

Original: Our Transformers replace the LSTM-style mechanism to capture temporal dependencies.
Revised: Our Transformers replace the LSTM-style mechanism to capture temporal dependencies.

Original: We evaluated our novel architecture on several synthetic and real-world datasets, evaluated by likelihood and prediction accuracy.
Revised: We evaluated our novel architecture on several synthetic and real-world datasets, evaluated by likelihood and prediction accuracy.

Original: On most of them, it surpasses the previous version and other strong competitors; on the others, it is comparable to the state-of-the-art.
Revised: On most of them, it surpasses the previous version and other strong competitors; on the others, it is comparable to the state-of-the-art.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Event sequences are ubiquitous in real-world applications, such as healthcare, ﬁnance, education, commerce, gaming, audio, news, security, and social media.
Revised: Event sequences are ubiquitous in real-world applications, such as healthcare, ﬁnance, education, commerce, gaming, audio, news, security, and social media.

Original: We propose an elegant way to embed events into a vector space R D .
Revised: We propose an elegant way to embed events into a vector space R D .

Original: An actual or possible event at time t ∈ R is contextually embedded using attention over actual events at other times, in a way that considers their timing.
Revised: An actual or possible event at time t ∈ R is contextually embedded using attention over actual events at other times, in a way that considers their timing.

Original: Good event embeddings could be used in a variety of downstream tasks, similar to word token embeddings in
Revised: Good event embeddings could be used in a variety of downstream tasks, similar to word token embeddings in

Original: BERT.
Revised: BERT.

Original: In particular, we consider the task of predicting the next event (generative modeling): at each time t ∈ R , all of the possible events are embedded using masked attention over the preceding actual events, and those embeddings yield the instantaneous probabilities at time t of the possible events.
Revised: In particular, we consider the task of predicting the next event (generative modeling): at each time t ∈ R , all of the possible events are embedded using masked attention over the preceding actual events, and those embeddings yield the instantaneous probabilities at time t of the possible events.

Original: Our approach is a continuous-time generalization of the Transformer (Vaswani et al., 2017).
Revised: Our approach is a continuous-time generalization of the Transformer (Vaswani et al., 2017).

Original: Transformers scale up well to very large datasets and have achieved astonishing success at generative modeling of discrete-time sequences, such as natural-language documents (Radford et al., 2019;
Revised: Transformers scale up well to very large datasets and have achieved astonishing success at generative modeling of discrete-time sequences, such as natural-language documents (Radford et al., 2019;

Original: Brown et al., 2020) and proteins (Rao et al., 2021).
Revised: Brown et al., 2020) and proteins (Rao et al., 2021).

Original: We show in this paper that they are also effective for irregularly spaced continuous-time sequences, even in lower-data regimes.
Revised: We show in this paper that they are also effective for irregularly spaced continuous-time sequences, even in lower-data regimes.

Original: We further show how to derive models from arbitrary NDTT programs.
Revised: We further show how to derive models from arbitrary NDTT programs.

Original: NDTT, or neural Datalog through time (Mei et al., 2020a), is a neural-symbolic framework for modeling event sequences.
Revised: NDTT, or neural Datalog through time (Mei et al., 2020a), is a neural-symbolic framework for modeling event sequences.

Original: An
Revised: An

Original: NDTT program speciﬁes logical rules that derive facts within a deductive database.
Revised: NDTT program speciﬁes logical rules that derive facts within a deductive database.

Original: Moreover, the facts in the database are embedded using a deep recurrent neural architecture that is automatically determined by the facts’ provenance.
Revised: Moreover, the facts in the database are embedded using a deep recurrent neural architecture that is automatically determined by the facts’ provenance.

Original: Among the derived facts at time t are the possible events at time t .
Revised: Among the derived facts at time t are the possible events at time t .

Original: As a simple NDTT example, if u is a user and d is a document and u has appropriate permissions, then u may open d .
Revised: As a simple NDTT example, if u is a user and d is a document and u has appropriate permissions, then u may open d .

Original: The embedding and instantaneous probability of this possible event at time t depend on the embeddings at time t of u and d (or more precisely, of the facts that u is a user and d is a document).
Revised: The embedding and instantaneous probability of this possible event at time t depend on the embeddings at time t of u and d (or more precisely, of the facts that u is a user and d is a document).

Original: If the event actually occurs, it may change what facts are true at times > t and how their embeddings evolve over times > t .
Revised: If the event actually occurs, it may change what facts are true at times > t and how their embeddings evolve over times > t .

Original: For instance, the NDTT program can state that if u opens d , this makes editing events possible, and also affects the future embeddings of u and d .
Revised: For instance, the NDTT program can state that if u opens d , this makes editing events possible, and also affects the future embeddings of u and d .

Original: These updated embeddings feed forward to help predict future events involving u and/or d .
Revised: These updated embeddings feed forward to help predict future events involving u and/or d .

Original: (They could also be exploited for other downstream predictive tasks, via multitask training or pretraining.)
Revised: (They could also be exploited for other downstream predictive tasks, via multitask training or pretraining.)

Original: All of these effects are controlled by the hand-crafted rules and learned parameters of the NDTT program,
Revised: All of these effects are controlled by the hand-crafted rules and learned parameters of the NDTT program,

Original: [Figure]
Revised: [Figure]

Original: which are analogous to the hand-crafted structure and learned parameters of a graphical model, in that they enforce domain-appropriate conditional independence and parameter sharing.
Revised: which are analogous to the hand-crafted structure and learned parameters of a graphical model, in that they enforce domain-appropriate conditional independence and parameter sharing.

Original: We modify the NDTT architectures to use attention.
Revised: We modify the NDTT architectures to use attention.

Original: In our architecture, a fact that can be derived in multiple ways, by different NDTT rules, uses multiple attention heads to examine the past.
Revised: In our architecture, a fact that can be derived in multiple ways, by different NDTT rules, uses multiple attention heads to examine the past.

Original: Du et al.
Revised: Du et al.

Original: (2016); Mei & Eisner (2017); Mei et al.
Revised: (2016); Mei & Eisner (2017); Mei et al.

Original: (2020a) instead summarize the past using recurrent neural networks (RNNs).
Revised: (2020a) instead summarize the past using recurrent neural networks (RNNs).

Original: Our attentive approach has three advantages over theirs:
Revised: Our attentive approach has three advantages over theirs:

Original: (cid:172)
Revised: (cid:172)

Original: A Transformer does not summarize the past.
Revised: A Transformer does not summarize the past.

Original: Our predictions at time t can examine an unboundedly large representation of the past ( R d embeddings of every event before t ), not merely a ﬁxeddimensional summary that was computed greedily from left to right (an RNN’s state at time t ).
Revised: Our predictions at time t can examine an unboundedly large representation of the past ( R d embeddings of every event before t ), not merely a ﬁxeddimensional summary that was computed greedily from left to right (an RNN’s state at time t ).

Original: (cid:173)
Revised: (cid:173)

Original: A Transformer’s computation graph is broader and shallower.
Revised: A Transformer’s computation graph is broader and shallower.

Original: The breadth makes it easier to learn long-distance inﬂuences.
Revised: The breadth makes it easier to learn long-distance inﬂuences.

Original: The shallowness does make it impossible to represent inherently deep concepts such as parity, but it enables greater parallelism: the layer- (cid:96) embeddings can be computed in parallel during training, as they depend only on layer (cid:96) − 1 and not on one another.
Revised: The shallowness does make it impossible to represent inherently deep concepts such as parity, but it enables greater parallelism: the layer- (cid:96) embeddings can be computed in parallel during training, as they depend only on layer (cid:96) − 1 and not on one another.

Original: (cid:174)
Revised: (cid:174)

Original: Our architecture is simpler and arguably more natural, while remaining competitive in our experiments.
Revised: Our architecture is simpler and arguably more natural, while remaining competitive in our experiments.

Original: To describe how embeddings vary continuously during the period ( t i , t i +1 ] between two events, Mei & Eisner (2017) had to stipulate an arbitrary family of parametric decay functions on t , and Mei et al.
Revised: To describe how embeddings vary continuously during the period ( t i , t i +1 ] between two events, Mei & Eisner (2017) had to stipulate an arbitrary family of parametric decay functions on t , and Mei et al.

Original: (2020a) had to design a complex method for pooling the parameters of these decay functions across multiple NDTT rules.
Revised: (2020a) had to design a complex method for pooling the parameters of these decay functions across multiple NDTT rules.

Original: In contrast, our embeddings are constructed “from scratch” at each time t simply by attending to the set of previous events, using t -speciﬁc query vectors that include a continuous positional embedding of t .
Revised: In contrast, our embeddings are constructed “from scratch” at each time t simply by attending to the set of previous events, using t -speciﬁc query vectors that include a continuous positional embedding of t .

Original: As t ranges over ( t i , t i +1 ] , the attention weights over the ﬁrst i events vary continuously with t , so the embeddings do so too.
Revised: As t ranges over ( t i , t i +1 ] , the attention weights over the ﬁrst i events vary continuously with t , so the embeddings do so too.

Original:  C ONTINUOUS -T IME T RANSFORMER FOR E MBEDDING E VENTS
Revised:  C ONTINUOUS -T IME T RANSFORMER FOR E MBEDDING E VENTS

Original: Suppose we observe I events over a ﬁxed time interval [0 , T ) .
Revised: Suppose we observe I events over a ﬁxed time interval [0 , T ) .

Original: Each event is denoted mnemonically as e @ t (i.e., “type e at time t ”), where e ∈ { 1 , 2 , . .
Revised: Each event is denoted mnemonically as e @ t (i.e., “type e at time t ”), where e ∈ { 1 , 2 , . .

Original: , E } is the event’s type.
Revised: , E } is the event’s type.

Original: The observed event sequence is e 1@ t 1 , e 2@ t 2 , . . .
Revised: The observed event sequence is e 1@ t 1 , e 2@ t 2 , . . .

Original: , e I @
Revised: , e I @

Original: t I , where 0 < t 1 < t 2 < . . .
Revised: t I , where 0 < t 1 < t 2 < . . .

Original: < t I < T .
Revised: < t I < T .

Original: For any observed or possible event e @ t , we can compute an embedding (cid:74) e (cid:75) ( t )
Revised: For any observed or possible event e @ t , we can compute an embedding (cid:74) e (cid:75) ( t )

Original: ∈ R D by attending to a set H ( e @ t ) of relevant events.
Revised: ∈ R D by attending to a set H ( e @ t ) of relevant events.

Original: (For the moment, imagine that H ( e @ t ) consists of all the observed events e i @
Revised: (For the moment, imagine that H ( e @ t ) consists of all the observed events e i @

Original: t i .)
Revised: t i .)

Original: More precisely, (cid:74) e (cid:75) ( t ) is the concatenation of layer-wise embeddings
Revised: More precisely, (cid:74) e (cid:75) ( t ) is the concatenation of layer-wise embeddings

Original: [Equation]
Revised: [Equation]

Original: e (cid:75) (0) ( t ) , (cid:74) e (cid:75) (1) ( t ) , . . .
Revised: e (cid:75) (0) ( t ) , (cid:74) e (cid:75) (1) ( t ) , . . .

Original: , (cid:74) e (cid:75) ( L ) ( t ) .
Revised: , (cid:74) e (cid:75) ( L ) ( t ) .

Original: For (cid:96) > 0 , the layer- (cid:96) embedding of e @ t is computed as
Revised: For (cid:96) > 0 , the layer- (cid:96) embedding of e @ t is computed as

Original: [Equation]
Revised: [Equation]

Original: where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is
Revised: where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is

Original: [Equation]
Revised: [Equation]

Original: In layer (cid:96) , v ( (cid:96) ) , k ( (cid:96) ) , and q ( (cid:96) ) are known as the value , key , and query vectors and are extracted from the layer-( (cid:96) -1) event embeddings using learned layer-speciﬁc matrices V ( (cid:96) ) , K ( (cid:96) ) , Q (
Revised: In layer (cid:96) , v ( (cid:96) ) , k ( (cid:96) ) , and q ( (cid:96) ) are known as the value , key , and query vectors and are extracted from the layer-( (cid:96) -1) event embeddings using learned layer-speciﬁc matrices V ( (cid:96) ) , K ( (cid:96) ) , Q (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As the base case, (cid:74) e (cid:75) (0) ( t ) def = (cid:74) e (cid:75) (0) is a learned embedding of the event type e .
Revised: As the base case, (cid:74) e (cid:75) (0) ( t ) def = (cid:74) e (cid:75) (0) is a learned embedding of the event type e .

Original: (cid:74) t (cid:75) denotes an embedding of the time t ; as proposed by Zuo et al.
Revised: (cid:74) t (cid:75) denotes an embedding of the time t ; as proposed by Zuo et al.

Original: (2020), we use the same sinusoidal embedding as in Vaswani et al.
Revised: (2020), we use the same sinusoidal embedding as in Vaswani et al.

Original: (2017), but with t now being real instead of integer:
Revised: (2017), but with t now being real instead of integer:

Original: [Equation]
Revised: [Equation]

Revised: and (cid:74) t (cid:75)

Revised: [Equation]

Revised: = sin(

Revised: [Equation]

Original:  We concatenate (cid:74) t (cid:75) to the rest of the embedding rather than adding it (cf.
Revised: [Equation] ) if d is even [Equation] We concatenate (cid:74) t (cid:75) to the rest of the embedding rather than adding it (cf.

Original:  He et al., 2020); furthermore we do so again at every layer.
Revised: Kitaev & Klein, 2018; He et al., 2020); furthermore we do so again at every layer.

Original: According to equations (1)–(3), to compute the layer- (cid:96) embedding of an event, we only need the layer- ( (cid:96) − 1 ) embeddings of the relevant events.
Revised: According to equations (1)–(3), to compute the layer- (cid:96) embedding of an event, we only need the layer- ( (cid:96) − 1 ) embeddings of the relevant events.

Original: This makes it possible to compute the layer- (cid:96) embeddings of all events in parallel.
Revised: This makes it possible to compute the layer- (cid:96) embeddings of all events in parallel.

Original: The set of relevant events H ( e @ t ) may be deﬁned in a task-speciﬁc way.
Revised: The set of relevant events H ( e @ t ) may be deﬁned in a task-speciﬁc way.

Original: For example, to pretrain
Revised: For example, to pretrain

Original: BERT-like embeddings (Devlin et al., 2018)
Revised: BERT-like embeddings (Devlin et al., 2018)

Original: , we might use a corrupted version of { e 1@ t 1 , . . .
Revised: , we might use a corrupted version of { e 1@ t 1 , . . .

Original: , e I @
Revised: , e I @

Original: t I } in which some e i @
Revised: t I } in which some e i @

Original: t i have been removed or replaced with maskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmask @
Revised: t i have been removed or replaced with maskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmask @

Original: t i .
Revised: t i .

Original: Such embeddings could be pretrained with a BERT-like objective and then ﬁne-tuned to predict properties of the observed events.
Revised: Such embeddings could be pretrained with a BERT-like objective and then ﬁne-tuned to predict properties of the observed events.

Original:  G ENERATIVE M ODELING OF C ONTINUOUS -T IME E VENT S EQUENCES
Revised:  G ENERATIVE M ODELING OF C ONTINUOUS -T IME E VENT S EQUENCES

Original: In this paper, we focus on the task of predicting future events given past ones.
Revised: In this paper, we focus on the task of predicting future events given past ones.

Original: At any time t , we would like to know what will happen at that time, given the actual events that happened before t .
Revised: At any time t , we would like to know what will happen at that time, given the actual events that happened before t .

Original: Our generative model is analogous to a Transformer language model (Radford et al., 2019; Brown et al., 2020), which, at each time t ∈ N , deﬁnes a probability distribution over the words in the vocabulary.
Revised: Our generative model is analogous to a Transformer language model (Radford et al., 2019; Brown et al., 2020), which, at each time t ∈ N , deﬁnes a probability distribution over the words in the vocabulary.

Original: In our setting, however, t ∈ R .
Revised: In our setting, however, t ∈ R .

Original: With probability 1, nothing happens at time t .
Revised: With probability 1, nothing happens at time t .

Original: Each possible event e in our vocabulary has only an inﬁnitesimal probability of occurring at time t .
Revised: Each possible event e in our vocabulary has only an inﬁnitesimal probability of occurring at time t .

Original: We write this probability as λ e ( t )d t where λ e ( t ) ∈ R is called the (Poisson) intensity of type- e events at time t .
Revised: We write this probability as λ e ( t )d t where λ e ( t ) ∈ R is called the (Poisson) intensity of type- e events at time t .

Original: More formally, the probability of such an event occurring during [ t, t + (cid:15) ) approaches λ e ( t ) (cid:15) as (cid:15) → + 0 .
Revised: More formally, the probability of such an event occurring during [ t, t + (cid:15) ) approaches λ e ( t ) (cid:15) as (cid:15) → + 0 .

Original: We model
Revised: We model

Original: λ e ( t ) as a function of the top-layer embedding of the possible event e @ t :λ e ( t ) def =
Revised: λ e ( t ) as a function of the top-layer embedding of the possible event e @ t :

Original: softplus( w (cid:62) e [1; (cid:74) e 
Revised: [Equation] ; (cid:74) e (cid:75) [Equation]

Original: (cid:75) L ( t )] , τ e ) where softplus( x, τ ) = τ
Revised: t )] , τ e ) where softplus( x, [Equation]

Original: log(1 + exp( x/τ )) >(5) with learnable parameters w e and τ e > 0 .
Revised: (5) with learnable parameters w e and τ e > 0 .

Original: We do this separately for each possible e @ t , computing the embedding (cid:74) e
Revised: We do this separately for each possible e @ t , computing the embedding (cid:74) e

Original: (cid:75) L ( t ) using equations (1)–(3).
Revised: (cid:75) L ( t ) using equations (1)–(3).

Original: To ensure that our model is generative, we compute (cid:74) e
Revised: To ensure that our model is generative, we compute (cid:74) e

Original: (cid:75) L ( t ) from only the previous events.
Revised: (cid:75) L ( t ) from only the previous events.

Original: Thus, we take H ( e @ t ) in equation (1) to consist of all the previously generated events e i @
Revised: Thus, we take H ( e @ t ) in equation (1) to consist of all the previously generated events e i @

Original: t i for t i < t , as well as the possible event e @ t itself.
Revised: t i for t i < t , as well as the possible event e @ t itself.

Original: Including e @ t allows e @ t itself to draw most of the attention if none of the previous events are very relevant (e.g., they are too far in the past), or all of the attention in the special case where no previous events exist (i.e., it ensures that H ( e @ t ) (cid:54) = ∅ , preventing division byin equation 1).
Revised: Including e @ t allows e @ t itself to draw most of the attention if none of the previous events are very relevant (e.g., they are too far in the past), or all of the attention in the special case where no previous events exist (i.e., it ensures that H ( e @ t ) (cid:54) = ∅ , preventing division byin equation 1).

Original: Embedding each possible e @ t separately is somewhat expensive.
Revised: Embedding each possible e @ t separately is somewhat expensive.

Original: The attention weights α (1) , . .
Revised: The attention weights α (1) , . .

Original: , α ( L ) used to compute its embedding and intensity must be recomputed for each e and t .
Revised: , α ( L ) used to compute its embedding and intensity must be recomputed for each e and t .

Original: This is more expressive but also more expensive than a standard Transformer language model, which instead models the probability of word e at time t as proportional to exp( w (cid:62) e [1; (cid:74)
Revised: This is more expressive but also more expensive than a standard Transformer language model, which instead models the probability of word e at time t as proportional to exp( w (cid:62) e [1; (cid:74)

Original: f (cid:75) L ( s )]) where (cid:74) f (cid:75) L ( s ) is the top-layer embedding of the previous word f @ s , where t = s + 1 , and can be reused for all possible choices of e @ t .
Revised: f (cid:75) L ( s )]) where (cid:74) f (cid:75) L ( s ) is the top-layer embedding of the previous word f @ s , where t = s + 1 , and can be reused for all possible choices of e @ t .

Original: Why do we not follow that cheaper approach, using the previous event f @ s in place of e @ t in equation (5)?
Revised: Why do we not follow that cheaper approach, using the previous event f @ s in place of e @ t in equation (5)?

Original: Because then λ e ( t ) would depend only on s and not on t (a language model can rely on t
Revised: Because then λ e ( t ) would depend only on s and not on t (a language model can rely on t

Original: = s + 1 , but we may have to evaluate intensities at all real-valued t ≥ s since our events are irregularly spaced).
Revised: = s + 1 , but we may have to evaluate intensities at all real-valued t ≥ s since our events are irregularly spaced).

Original: Our more expressive design also allows the attention vector used to compute λ e ( t ) to depend on e , to support the “selective attention” of section 4 below.
Revised: Our more expressive design also allows the attention vector used to compute λ e ( t ) to depend on e , to support the “selective attention” of section 4 below.

Original: As a speedup, however, we can associate each event type e with a coarse event type ¯ e that has the same selective attention pattern, and replace (cid:74) e
Revised: As a speedup, however, we can associate each event type e with a coarse event type ¯ e that has the same selective attention pattern, and replace (cid:74) e

Original: (cid:75) L ( t ) with (cid:74) ¯ e (cid:75) L ( t ) in equation (5).
Revised: (cid:75) L ( t ) with (cid:74) ¯ e (cid:75) L ( t ) in equation (5).

Original: (However, equation (5) still uses the ﬁne-grained w e .)
Revised: (However, equation (5) still uses the ﬁne-grained w e .)

Original: Now to compute λ e ( t ) , we only have to embed ¯ e @ t , which is a speedup if many of the possible event types e are associated with the same ¯ e .
Revised: Now to compute λ e ( t ) , we only have to embed ¯ e @ t , which is a speedup if many of the possible event types e are associated with the same ¯ e .

Original: In the case where we do not use selective attention, we can get away with using only a single coarse event type for the whole model—saving a runtime factor of E as in the cheaper approach.
Revised: In the case where we do not use selective attention, we can get away with using only a single coarse event type for the whole model—saving a runtime factor of E as in the cheaper approach.

Original: Note that the history H still uses ﬁne-grained embeddings, so if e @ t actually occurs, we must then compute (cid:74) 
Revised: Note that the history H still uses ﬁne-grained embeddings, so if e @ t actually occurs, we must then compute (cid:74) e (cid:75) 0 ( t ) , . . .

Original: [Equation]
Revised: , (cid:74) e (cid:75) L ( t ) .

Original: Our model falls into the family of multivariate point processes .
Revised: Our model falls into the family of multivariate point processes .

Original: Its log-likelihood is given by
Revised: Its log-likelihood is given by

Original: [Equation]
Revised: [Equation]

Original: This formula has been derived and discussed in previous work (Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).
Revised: This formula has been derived and discussed in previous work (Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).

Original: Intuitively, during parameter training, each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type k ever happened at other times.
Revised: Intuitively, during parameter training, each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type k ever happened at other times.

Original: That is why we need to embed possible events and compute their intensities.
Revised: That is why we need to embed possible events and compute their intensities.

Original:  M ULTI -H EAD S ELECTIVE A TTENTION
Revised:  M ULTI -H EAD S ELECTIVE A TTENTION

Original: We now enrich the formalism to allow selective attention.
Revised: We now enrich the formalism to allow selective attention.

Original: As in a graphical model, not all events should be able to inﬂuence one another.
Revised: As in a graphical model, not all events should be able to inﬂuence one another.

Original: Consider a scenario with two event types: eeeeeeeeeeeeeeeee means that Eve emails Adam, while fffffffffffffffff means that Frank emails Eve.
Revised: Consider a scenario with two event types: eeeeeeeeeeeeeeeee means that Eve emails Adam, while fffffffffffffffff means that Frank emails Eve.

Original: As Frank does not know when Eve emailed
Revised: As Frank does not know when Eve emailed

Original: Adam, past events of type eeeeeeeeeeeeeeeee cannot inﬂuence his behavior.
Revised: Adam, past events of type eeeeeeeeeeeeeeeee cannot inﬂuence his behavior.

Original: Therefore, H ( fffffffffffffffff @ t ) should include past events of type fffffffffffffffff but not eeeeeeeeeeeeeeeee , so that the embedding of fffffffffffffffff @ t and hence the intensity function λ fffffffffffffffff ( t ) pay zero attention to eeeeeeeeeeeeeeeee events.
Revised: Therefore, H ( fffffffffffffffff @ t ) should include past events of type fffffffffffffffff but not eeeeeeeeeeeeeeeee , so that the embedding of fffffffffffffffff @ t and hence the intensity function λ fffffffffffffffff ( t ) pay zero attention to eeeeeeeeeeeeeeeee events.

Original: In contrast, H ( eeeeeeeeeeeeeeeee @ t ) should still include past events of both types, since both are visible to Eve and can inﬂuence her behavior.
Revised: In contrast, H ( eeeeeeeeeeeeeeeee @ t ) should still include past events of both types, since both are visible to Eve and can inﬂuence her behavior.

Original: We describe this situation with the edges fffffffffffffffff <- fffffffffffffffff, eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee , eeeeeeeeeeeeeeeee <- fffffffffffffffff .
Revised: We describe this situation with the edges fffffffffffffffff <- fffffffffffffffff eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee , eeeeeeeeeeeeeeeee <- fffffffffffffffff .

Original: These edges are called rules , for reasons that will become clear in section 5.
Revised: These edges are called rules , for reasons that will become clear in section 5.

Original: Furthermore, when Eve decides whether to email Adam ( eee eee eeeee eee eee @ t ), we may reasonably suppose that she separately considers the embeddings of the past eeeeeeeeeeeeeeeee events (e.g., “when was my last relevant email to Adam?”) versus the past fffffffffffffffff events (e.g., “what have I heard from Frank?”).
Revised: Furthermore, when Eve decides whether to email Adam ( eeeeeeeeeeeeeeeee @ t ), we may reasonably suppose that she separately considers the embeddings of the past eeeeeeeeeeeeeeeee events (e.g., “when was my last relevant email to Adam?”) versus the past fffffffffffffffff events (e.g., “what have I heard from Frank?”).

Original: For this reason, we associate different attention heads with the two rules that affect eeeeeeeeeeeeeeeee , namely eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee and eeeeeeeeeeeeeeeee <- fffffffffffffffff .
Revised: For this reason, we associate different attention heads with the two rules that affect eee eee ee e ee eee ee e , namely eee eee ee e ee eee ee e <- eee eee ee e ee eee ee e and eee eee

Revised: ee e ee eee ee e <- fff fff

Revised: ff f ff fff ff f .

Original: These heads may have different parameters, so that they seek out and obtain different information from the past via different queries, keys, and values.
Revised: These heads may have different parameters, so that they seek out and obtain different information from the past via different queries, keys, and values.

Original: In general, we replace equation (1) with
Revised: In general, we replace equation (1) with

Original: [Equation]
Revised: [Equation]

Original: 8) where r in the summation ranges over rules e <- · · · .
Revised: where r in the summation ranges over rules e <- · · · .

Original: The history H r ( eeeeeeeeeeeeeeeee @ t ) contains only those past events f @ s that rule r makes visible to e , as well as e @ t itself as discussed in section 3.
Revised: The history H r ( eeeeeeeeeeeeeeeee @ t ) contains only those past events f @ s that rule r makes visible to e , as well as e @ t itself as discussed in section 3.

Original: The r -speciﬁc α
Revised: The r -speciﬁc α

Original: and v quantities are deﬁned using versions of equations (2)–(3) with r -speciﬁc parameters.
Revised: and v quantities are deﬁned using versions of equations (2)–(3) with r -speciﬁc parameters.

Original:  A TTENTIVE N EURAL D ATALOG T HROUGH T IME
Revised:  A TTENTIVE N EURAL D ATALOG T HROUGH T IME

Original: Edges such as eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple NDTT rules.
Revised: Edges such as eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple examples of rules in a Neural Datalog Through

Original: More generally, an NDTT program (Mei et al., 2020a) deﬁnes how facts should be automatically asserted into and retracted from a database, due to the occurrence of events and/or the presence of other facts.
Revised: Time program (Mei et al., 2020a). We brieﬂy review this formalism and then extend our approach from section 4 to handle all NDTT programs. Datalog. Recall that Datalog (Ceri et al., 1989) is a traditional formalism for deductive databases.

Revised: A deductive database holds both extensional facts , which are placed there by some external process, and intensional facts , which are transitively deduced from the extensional facts.

Original: Manually specifying such a program relieves the neural system of the burden of representing discrete facts about the world and learning how to modify them—which is properly the job of a database.
Revised: A Datalog program is simply a set of rules that govern these deductions: • h :- g 1 , ..., g n says to deduce h at any time t when g 1 , . . . , g n are all true (in the database). A single rule can license many deductions.

Revised: That is because the facts can have structured names, and h, g 1 , . . .

Revised: g n can be patterns that match against those names.

Revised: In the example at the bottom of page 1, we would have a rule like openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D) :- user(U), group(G), document(D), member(U,G), readable(D,G) , where U , G , and D are variables.

Revised: In English, this says that the event openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D) is possible 2 if user U is a member of some group G such that document D is readable by G .

Revised: Datalog Through Time.

Revised: Whenever extensional facts are added or removed, the intensional facts are instantly recomputed according to the deductive rules.

Revised: Datalog Through Time is an extension in which extensional facts are automatically added and removed when the database is notiﬁed of the occurrence of events.

Revised: This behavior is governed by two additional rule types:

Revised: • h <- f , g 1 , ..., g n says to add h at any time s when event f occurs and the g i are all true.

Revised: h <

Revised: - f , g 1 , ..., g n says to remove h at any time s satisfying the same conditions.

Revised: Thus, the proposition h is true at time t (i.e., appears as a fact in the database at time t ) iff either (cid:182) h is deduced at time t , or (cid:183) h was added at some time s < t and never removed at any time in ( s, t ) .

Revised: Neural Datalog Through Time.

Revised: It would be difﬁcult to train a neural architecture to encode thousands or millions of structured boolean facts about the world in its state and to modify those encoded facts systematically and transitively in response to events.

Revised: As a neuro-symbolic method,

Original:  However, if a proposition h appears as a fact in the NDTT database at time t ,then it will  also have an embedding (cid:74) h (cid:75) ( t ) —a learned representation of that fact at that time, capturing additional contextual properties.
Revised: NDTT delegates that task to a symbolic database governed by manually speciﬁed Datalog Through Time rules. However, it augments the database by neurally embedding every fact in a vector space. Thus, iff a proposition h appears as a fact in the NDTT database at time t , it will be associated not only with the simple truth value true but also with an embedding (cid:74) h (cid:75) ( t ) .

Original: Such a  representation  can be trained to be useful in downstream tasks.
Revised: This embedding is a learned representation of that fact at that time, and can be trained to be useful in downstream tasks.

Revised: It captures details of when and how that fact was established (the fact’s provenance ), since it is computed using learned parameters associated with the rules that deduced and/or added it.

Revised: For example, if we write NDTT rules user(U) <- · · · that add user(U) (perhaps redundantly) each time a relevant event affects it, then (cid:74) user(456) (cid:75) ( t ) reﬂects the embeddings and timings of all events before t that affected user(456) .

Revised: This embedding may help predict what user 456 will do in future.

Revised: In our example, if openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(456,78) is deduced with the help of user(456) , then the embedding (cid:74) open open open openopenopen openopen open openopen openopenopen open openopen(456,78) (cid:75) ( t ) —and its probability—in turn depends on (cid:74) user(456) (cid:75) ( t ) and varies along with it as t changes.

Original: In our setting, for example, if e denotes the time-varying boolean proposition that an event of type e is possible, then (cid:74) e (cid:75) ( t ) will be deﬁned at all times  when that proposition is true—i.e., at all times when e is possible—and is used in equation (5) to model the intensity function λ e  , which governs when e is likely to actually occur.
Revised: In general, if e denotes the time-varying boolean proposition that an event of type e is possible, it follows that (cid:74) e (cid:75) ( t ) will be deﬁned at all times t when that proposition is true—i.e., at all times when e is possible. This embedding is used in equation (5) to model the intensity function λ e ( t ) , which governs when e is likely to actually occur.

Original:  Our goalin this paper is to provide new formulas for the embeddings (cid:74) h (cid:75) ( t ) , based on Transformerstyle attention rather than LSTM-style recurrence.
Revised: Contribution. Our goal is to provide new formulas for the embeddings (cid:74) h (cid:75) ( t ) , based on Transformerstyle attention rather than LSTM-style recurrence.

Original: We call this attentive NDTT , or A-NDTT .
Revised: We call this attentive NDTT , or A-NDTT .

Original: The potential advantages for accuracy, efﬁciency, and simplicity were explained in section 1.
Revised: The potential advantages for accuracy, efﬁciency, and simplicity were explained in section 1.

Original: We have seen simple “launch” rules so far, but more generally, NDTT allows three kinds of rules:

Original: • h :- g 1 , ..., g n says to deduce h at any time t

Original: when g 1 , ..., g n are all true.

Original: • h <- f , g 1 , ..., g n says to launch h at any time s when event f occurs and the g i are all true.

Original: h <- f , g 1 , ..., g n says to dock h at any time s satisfying the same conditions.

Original: The effect of these actions is as follows: h is true (i.e., appears in the database) at time t if either (cid:182) h is deduced at time t , or (cid:183) h was launched at some time s < t and never docked at any time in ( s, t ) .

Original: The layer- (cid:96) embedding of h  is thengiven, under our A-NDTT approach, by
Revised: Under A-NDTT, the layer- (cid:96) embedding of h @ t is then

Original: [Equation]
Revised: [Equation]

Original: which augments equation (7).
Revised: which is an augmented version of equation (7).

Original: Supposecondition (cid:183) applies to h @ t where h was launched by rule r .
Revised: Suppose h is true at time t because it was added by rule r (i.e., condition (cid:183) ).

Original: Then the summand h ( (cid:96) )
Revised: Then the summand h ( (cid:96) )

Original: r ( t ) exists and is computed much as in equation (8), now as a sum over all launch times s .
Revised: r ( t ) exists and is computed much as in equation (8), now with attention over all add times s .

Original: In other words, H r ( h @ t ) includes just those past events f @ s such that f launched h via r at some time s < t and h was never docked at any time in ( s, t ) .
Revised: In other words, H r ( h @ t ) includes just those past events f @ s such that f added h via r at some time s < t and h was never removed at any time in ( s, t ) .

Original: It also includes h @ t itself, as before (even if h is not an event but some other kind of fact).
Revised: It also includes h @ t itself, as before.

Original: We make one enhancement: when the rule h <- f , g 1 , ..., g n causes h @ t to attend to the speciﬁc past event f @ s , we actually want attention to consider the embedding at time s of the entire launching condition f, g 1 , . ..
Revised: We make one enhancement: when the rule h <- f , g 1 , ..., g n causes h @ t to attend to the speciﬁc past event f @ s , we actually want attention to consider the embedding at time s of the entire add condition f, g 1 , . .

Original:  , g n .
Revised: . , g n .

Original: Thus the attention key of f @ s for this purpose is now deﬁned as 
Revised: Thus the attention key of f @ s for this purpose is now deﬁned as k ( (cid:96) ) ( f [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: (cid:2) 1; (cid:74)

Revised: [Equation]

Original: s (cid:75) ; (cid:74) f (cid:75) ( (cid:96) − 1) ( s ); (cid:74) g 1 (cid:75) ( (cid:96) − 1) ( s ); . . .
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Original: (cid:74) g n (cid:75) ( (cid:96) − 1) ( s ) (cid:3) (compare equation (3b)).
Revised: [Equation] ( s [Equation] (cid:3) (compare equation (3b)).

Original: Its attention value v ( (cid:96) )
Revised: Its attention value v ( (cid:96) )

Original: ( f @ s ) is deﬁned analogously, using a different matrix V ( (cid:96) ) r .
Revised: ( f @ s ) is deﬁned analogously, using a different matrix V ( (cid:96) ) r .

Original: How about theintermediate vector [ h ] ( (cid:96) ) ( t ) in equation (9)?
Revised: How about the vector [ h ] ( (cid:96) ) ( t ) in equation (9)?

Original: For each rule that deduces h , say h :- g 1 , . . .
Revised: For each rule that deduces h at time t (i.e., condition (cid:182) ), say h :- g 1 , . . .

Original: g n , we compute a nonlinear transform of the concatenated embeddings (cid:74) g
Revised: g n , we compute a nonlinear transform of the concatenated embeddings (cid:74) g

Original: i (cid:75) ( (cid:96) )  .
Revised: i (cid:75) ( (cid:96) ) ( t ) .

Original: The resulting vectors are pooled to obtain [ h ] ( (cid:96) )
Revised: The resulting vectors are pooled to obtain [ h ] ( (cid:96) )

Original: ( t ) .
Revised: ( t ) .

Original: (If there are no such rules, then [ h ] ( (cid:96) )
Revised: (If there are no such rules, then [ h ] ( (cid:96) )

Original: ( t ) = 0 .)
Revised: ( t ) = 0 .)

Original: Here we follow exactly the recipe of Mei et al.
Revised: Here we follow exactly the recipe of Mei et al.

Original: (2020a), except that we now do it anew at each level (cid:96) .
Revised: (2020a), except that we now do it anew at each level (cid:96) .

Revised: The speciﬁcation of parameter dimensionalities is discussed in Appendix A0.

Original: Mei et al.
Revised: Mei et al.

Original: (2020a) give many examples of NDTT programs.
Revised: (2020a) give many examples of NDTT programs.

Original: Here is a simple example to illustrate the use of :- rules.
Revised: Here is a simple example to illustrate the use of :- rules.

Original: Possible events are shown in orange and other possible facts in blue.
Revised: Possible events are shown in orange and other possible facts in blue.

Original: eeeeeeeeeeeeeeeee means that Eve posts a message to the company forum, while fffffffffffffffff means that Frank does so.
Revised: eeeeeeeeeeeeeeeee means that Eve posts a message to the company forum, while fffffffffffffffff means that Frank does so.

Original: Once the forum is created by a createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate event, its existence is a fact ( forum ) whose embedding ( (cid:74) forum (cid:75) )  reﬂects all messages posted  to the forum.
Revised: Once the forum is created by a create createcreatecreate create createcreatecreate create createcreatecreate create createcreatecreatecreate event, its existence is a fact (called forum ) whose embedding (called (cid:74) forum (cid:75) ) always reﬂects all messages posted so far to the forum.

Original: Until the forum is destroyed, Eve and Frank can post to it, and the embedding and intensity of such messages depends on the current state of the forum:
Revised: Until the forum is destroyed, Eve and Frank can post to it, and the embedding and intensity of such messages depends on the current state of the forum:

Original: forum <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate.

Original: forum <- eeeeeeeeeeeeeeeee.

Original: forum <- fffffffffffffffff.

Original: eeeeeeeeeeeeeeeee

Original: [Table]
Revised: [Equation]

Original: The resulting neural architecture is drawn in Figure 1b.
Revised: The resulting neural architecture is drawn in Figure 1b.

Original: If the company grows from 2 to K employees, then the program needs O ( K ) rules and hence O ( K ) parameters, which deﬁne how each employee’s messages affect the forum and vice-versa.
Revised: If the company grows from 2 to K employees, then the program needs O ( K ) rules and hence O ( K ) parameters, which deﬁne how each employee’s messages affect the forum and vice-versa.

Original: Without the :- rules, we would have to list out O ( K 2 ) rules such as eeeeeeeeeeeeeeeee <- fffffffffffffffff and hence would need O
Revised: Without the :- rules, we would have to list out O ( K 2 ) rules such as eeeeeeeeeeeeeeeee <- fffffffffffffffff and hence would need O

Original: ( K 2 ) parameters, which deﬁne how each employee’s messages affect every other employee’s messages directly; this case is drawn in Figure 1a.
Revised: ( K 2 ) parameters, which deﬁne how each employee’s messages affect every other employee’s messages directly; this case is drawn in Figure 1a.

Original: Actually, if the company has many employees E and a forum for each team T , NDTT rules can use capitalized variables to deﬁne the whole system concisely, using only O (1) rules and O (1) parameters.
Revised: Actually, if the company has many employees E and a forum for each team T , NDTT rules can use capitalized variables to deﬁne the whole system concisely, using only O (1) rules and O (1) parameters.

Original: Here the possible facts and events have structured names like messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,sales) , which would be an event in which employee eve posts a message to the sales team’s forum.
Revised: Here the possible facts and events have structured names like messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,sales) , which would be an event in which employee eve posts a message to the sales team’s forum.

Original: forum(T) <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate(T). forum(T) <- messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(E,T). messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(E,T) :- forum(T), member(E,T). !
Revised: (T), member

Original: forum(T)
Revised: E,T)

Original: <- destroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroy(T).
Revised: destroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroy(T).

Original: Additional rules could describe how the member facts and their embeddings change over time, which may in turn affect the embeddings and predictions of future messages.
Revised: Additional rules could describe how the member facts and their embeddings change over time, which may in turn affect the embeddings and predictions of future messages.

Original: See the discussion in section 1.
Revised: See the discussion in section 1.

Original:  T RAINING AND I NFERENCE
Revised:  T RAINING AND I NFERENCE

Original: We can estimate the parameters by locally maximizing the log-likelihood given in equation (6) using any stochastic gradient method.
Revised: We can estimate the parameters by locally maximizing the log-likelihood given in equation (6) using any stochastic gradient method.

Original: Details are given in Appendix A, including Monte Carlo approximations to the integral.
Revised: Details are given in Appendix A, including Monte Carlo approximations to the integral.

Original: Given the learned parameters, we may wish to make a minimum
Revised: Given the learned parameters, we may wish to make a minimum

Original: Bayes risk prediction about the next event given the past history.
Revised: Bayes risk prediction about the next event given the past history.

Original: A recipe can be found in Appendix B.
Revised: A recipe can be found in Appendix B.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multivariate point processes have been widely used in real-world applications, including document stream modeling (He et al., 2015; Du et al., 2015a), learning Granger causality (Xu et al., 2016;
Revised: Multivariate point processes have been widely used in real-world applications, including document stream modeling (He et al., 2015; Du et al., 2015a), learning Granger causality (Xu et al., 2016;

Original: Zhang et al., 2020b), network analysis (Choi et al., 2015; Etesami et al., 2016), recommendation systems (Du et al., 2015b), and social network analysis (Guo et al., 2015; Lukasik et al., 2016).
Revised: Zhang et al., 2020b), network analysis (Choi et al., 2015; Etesami et al., 2016), recommendation systems (Du et al., 2015b), and social network analysis (Guo et al., 2015; Lukasik et al., 2016).

Original: Over the recent years, various neural models have been proposed to expand the expressiveness of point processes.
Revised: Over the recent years, various neural models have been proposed to expand the expressiveness of point processes.

Original: They mostly use recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997): in particular Du et al.
Revised: They mostly use recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997): in particular Du et al.

Original: ; Mei & Eisner (2017); Xiao et al.
Revised: ; Mei & Eisner (2017); Xiao et al.

Original: (2017a;b);
Revised: (2017a;b);

Original: Omi et al.
Revised: Omi et al.

Original: Shchur et al.
Revised: Shchur et al.

Original: Mei et al.
Revised: Mei et al.

Original: (2020a)
Revised: (2020a)

Original: ; Boyd et al.
Revised: ; Boyd et al.

Original: Models of this kind enjoy continuous and inﬁnite state spaces, as well as ﬂexible transition functions, thus achieving superior performance on many real-world datasets, compared to classical models such as the Hawkes process (Hawkes, 1971).
Revised: Models of this kind enjoy continuous and inﬁnite state spaces, as well as ﬂexible transition functions, thus achieving superior performance on many real-world datasets, compared to classical models such as the Hawkes process (Hawkes, 1971).

Original: The Transformer Hawkes process (Zuo et al., 2020) and self-attentive Hawkes process (Zhang et al., 2020a) are pioneering work in using generative Transformers (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) in point processes.
Revised: The Transformer Hawkes process (Zuo et al., 2020) and self-attentive Hawkes process (Zhang et al., 2020a) are pioneering work in using generative Transformers (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) in point processes.

Original: The Transformer architecture allows their models to enjoy unboundedly large representations of histories, as well as great parallelism during training (see (cid:172) and (cid:173) in section 1).
Revised: The Transformer architecture allows their models to enjoy unboundedly large representations of histories, as well as great parallelism during training (see (cid:172) and (cid:173) in section 1).

Original: But they only embed actual events with attention; the intensities of possible events rely on a simple linear extrapolation of the embedding of the most recent actual event.
Revised: But they only embed actual events with attention; the intensities of possible events rely on a simple linear extrapolation of the embedding of the most recent actual event.

Original: 3 Our model embeds all possible events at all possible times with attention as well (see (cid:174) in section 1), thus being more ﬂexible (see section 3) and achieving better performance (see section 8).
Revised: 4 Our model embeds all possible events at all possible times with attention as well (see (cid:174) in section 1), thus being more ﬂexible (see section 3) and achieving better performance (see section 8).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: On several synthetic and real-world datasets, we evaluate our model’s held-out log-likelihood, and its success at predicting the time and type of the next event.
Revised: On several synthetic and real-world datasets, we evaluate our model’s held-out log-likelihood, and its success at predicting the time and type of the next event.

Original: We compare with multiple strong competitors.
Revised: We compare with multiple strong competitors.

Original: Experimental details are given in Appendix C.
Revised: Experimental details are given in Appendix C.

Original: We implemented our A-NDTT framework using PyTorch (Paszke et al., 2017) and pyDatalog (Carbonell et al., 2016), borrowing substantially from the public implementation of NDTT (Mei et al., 2020a).
Revised: We implemented our A-NDTT framework using PyTorch (Paszke et al., 2017) and pyDatalog (Carbonell et al., 2016), borrowing substantially from the public implementation of NDTT (Mei et al., 2020a).

Original: We then used it to implement our individual models.
Revised: We then used it to implement our individual models.

Original: Our code and datasets are available at code-public-after-paper-published .
Revised: Our code and datasets are available at code-public-after-paper-published .

Original: 8. C OMPARISON OF D IFFERENT T RANSFORMER A RCHITECTURES
Revised: 8. C OMPARISON OF D IFFERENT T RANSFORMER A RCHITECTURES

Original: We ﬁrst verify the comparative advantages of our continuous-time Transformer over the following state-of-the-art neural event models.
Revised: We ﬁrst verify the comparative advantages of our continuous-time Transformer over the following state-of-the-art neural event models.

Original: They are
Revised: They are

Original: Neural Hawkes process (NHP) (Mei & Eisner, 2017) .
Revised: Neural Hawkes process (NHP) (Mei & Eisner, 2017) .

Original: At any time t , NHP uses a continuous-time
Revised: At any time t , NHP uses a continuous-time

Original: LSTM to summarize the previous events into a multi-dimensional state vector, and conditions the intensities of all event types on that state.
Revised: LSTM to summarize the previous events into a multi-dimensional state vector, and conditions the intensities of all event types on that state.

Original: Transformer Hawkes process (THP) (Zuo et al., 2020) .
Revised: Transformer Hawkes process (THP) (Zuo et al., 2020) .

Original: THP directly applies the discrete-time generative Transformer, as discussed in section 7.
Revised: THP directly applies the discrete-time generative Transformer, as discussed in section 7 and footnote 4.

Original: Self-Attentive Hawkes Process (SAHP (Zhang et al., 2020a).
Revised: Self-Attentive Hawkes Process (SAHP (Zhang et al., 2020a).

Original: SAHP is very similar to THP but uses a slightly more ﬂexible intensity function; see footnote 3.
Revised: SAHP is very similar to THP but uses a slightly more ﬂexible intensity function; see footnote 4.

Original: For these models, we make use of their published implementation.
Revised: For the above models, we make use of their published implementation.

Original: 4 Details are in Appendix C.2.
Revised: 5 Details are in Appendix C.2.

Revised: Attentive neural Hawkes process (A-NHP)

Original: In this section, we use our unstructured generative model from section 3.
Revised: This is our unstructured generative model from section 3.

Original: Since this model does not use selective attention, we speed up the intensity computations by deﬁning them in terms of a single coarse event type, as described beneath equation (5).
Revised: Since this model does not use selective attention, we speed up the intensity computations by deﬁning them in terms of a single coarse event type, as described beneath equation (5).

Original: Thus, each event intensity λ e ( t ) is computed by attention over all previous events, where theparameters that determine attention  are independent of e (and t ).
Revised: Thus, each event intensity λ e ( t ) is computed by attention over all previous events, where the attention weights are independent of e .

Original: This parameter-sharing mechanism resembles the NHP, except that we now use a Transformer in place of an LSTM.
Revised: This parameter-sharing mechanism resembles the NHP, except that we now use a Transformer in place of an LSTM.

Original: So we call this model the attentive neural Hawkes process (A-NHP) .

Original: In a pilot experiment, we draw sequences from randomly initialized models (details in Appendix C.1.1), and ﬁt all these models on the synthetic datasets.
Revised: In a pilot experiment, we draw sequences from randomly initialized models (details in Appendix C.1.1), and ﬁt the above models on these synthetic datasets.

Original: The results are shown in Figure 2.
Revised: The results are shown in Figure 2.

Original: As shown in Figure 2c, on the synthetic NHP data, the performance of A-NHP is essentially the same as that of NHP, meaning that A-NHP is still a good choice even when it is misspeciﬁed.
Revised: As shown in Figure 2c, on the synthetic NHP data, the performance of A-NHP is essentially the same as that of NHP, meaning that A-NHP is still a good choice even when it is misspeciﬁed.

Original: This result is reassuring because A-NHP has less capacity in some ways (the circuit depth of a Transformer is ﬁxed, whereas the circuit depth of an LSTM grows with the length of the sequence) and excess capacity in other ways (the Transformer has unbounded memory whereas the LSTM has ﬁnite-dimensional memory).
Revised: This result is reassuring because A-NHP has less capacity in some ways (the circuit depth of a Transformer is ﬁxed, whereas the circuit depth of an LSTM grows with the length of the sequence) and excess capacity in other ways (the Transformer has unbounded memory whereas the LSTM has ﬁnite-dimensional memory).

Original: We also tried ﬁtting A-NHP to the synthetic A-NHP, THP, and SAHP data.
Revised: We also tried ﬁtting A-NHP to the synthetic A-NHP, THP, and SAHP data.

Original: As shown in
Revised: As shown in

Original: Figure 2d, the A-NHP data can not be well modeled by THP or SAHP, but it can be successfully modeled by—unsurprisingly—A-NHP and—maybe surprisingly—NHP.
Revised: Figure 2d, the A-NHP data can not be well modeled by THP or SAHP, but it can be successfully modeled by—unsurprisingly—A-NHP and—maybe surprisingly—NHP.

Original: On the SAHP and THP data,
Revised: On the SAHP and THP data,

Original: A-NHP still gets competitive performance (Figures 2a and 2b).
Revised: A-NHP still gets competitive performance (Figures 2a and 2b).

Original: Notably, THP ﬁts the time intervals poorly when it is misspeciﬁed, due to its restrictive handling of the passage of time (see section 7). 
Revised: Notably, THP ﬁts the time intervals poorly when it is misspeciﬁed, due to its restrictive handling of the passage of time (see section 7). [Figure] [Figure] [Figure]

Original: We then evaluated all these models on following two benchmark real-world datasets.
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: We then evaluated all these models on following two benchmark real-world datasets.

Original: MIMIC-II (Lee et al., 2011) .
Revised: MIMIC-II (Lee et al., 2011) .

Original: This dataset is a collection of de-identiﬁed clinical visit records of
Revised: This dataset is a collection of de-identiﬁed clinical visit records of

Original: Intensive Care Unit patients for 7 years.
Revised: Intensive Care Unit patients for 7 years.

Original: 6 Each patient has a sequence of hospital visit events, and each event records its time stamp and disease diagnosis.
Revised: Each patient has a sequence of hospital visit events, and each event records its time stamp and disease diagnosis.

Original: StackOverﬂow (Leskovec & Krevl, 2014) .
Revised: StackOverﬂow (Leskovec & Krevl, 2014) .

Original: This dataset represents two years of user awards on a question-answering website: each user received a sequence of badges (of 22 different types).
Revised: This dataset represents two years of user awards on a question-answering website: each user received a sequence of badges (of 22 different types).

Original: For these datasets, we used the preprocessed versions provided by Mei & Eisner (2017).

Original: More details about them can be found in Appendix C.1.2.

Original: On MIMIC-II data (Figure 3a), our A-NHP is always a co-winner on each of these tasks; but the other co-winner (NHP or THP) varies across tasks.
Revised: On MIMIC-II data (Figure 3a), our A-NHP is always a co-winner on each of these tasks; but the other co-winner (NHP or THP) varies across tasks.

Original: On StackOverﬂow data (Figure 3b), our A-NHP is clearly a winner on 2 out of 3 tasks.
Revised: On StackOverﬂow data (Figure 3b), our A-NHP is clearly a winner on 2 out of 3 tasks.

Original: Compared to NHP, A-NHP also enjoys a computational advantage, as discussed in sections 1 and 2.
Revised: Compared to NHP, A-NHP also enjoys a computational advantage, as discussed in sections 1 and 2.

Original: Empirically, training an A-NHP only took a fraction of the time that was needed to train an NHP, when sequences are reasonably long.
Revised: Empirically, training an A-NHP only took a fraction of the time that was needed to train an NHP, when sequences are reasonably long.

Original: Details can be found in Table 2 of Appendix C.
Revised: Details can be found in Table 2 of Appendix C.3.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 8. A-NDTT VS . NDTT

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Now we turn to the structured modeling approach presented in section 5.

Original: [Figure] [Figure]
Revised: We compare A-NDTT with NDTT on the RoboCup dataset proposed by Mei et al.

Original: [Figure]
Revised: (2020a).

Original: Now we evaluate A-NDTT vs. NDTT on the RoboCup dataset proposed by Mei et al.
Revised: In both cases, we used the NDTT program written by Mei et al.

Original: (2020a).
Revised: (2020a).

Revised: The rules are unchanged; the only difference is that our A-NDTT has the new continuous-time Transformer in lieu of the LSTM architecture.

Revised: We also evaluated the unstructured NHP and A-NHP models on this dataset.

Original: RoboCup (Chen & Mooney, 2008) .
Revised: RoboCup (Chen & Mooney, 2008) .

Original: This dataset logs  actions (e.g., kickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkick , passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass ) of robot  players during RoboCup Finals 2001–2004.
Revised: This dataset logs the actions (e.g., kickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkick , passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass ) of robot soccer players in the RoboCup Finals 2001–2004.

Original: The  set of possible event typesdynamically changes over time (e.g., only  ball possessor can kick or pass) as the ball is frequently transferred between players (by passing or stealing).
Revised: The ball is frequently transferred between players (by passing or stealing), and this dynamically changes the set of possible event types (e.g., only the ball possessor can kick or pass).

Original: There are K = 528 event types over all time, but only about 20 of them are possible at any given time.
Revised: There are K = 528 event types over all time, but only about 20 of them are possible at any given time.

Original: For each history, we made minimum Bayes risk predictions of the next event’s time, and that event’s participant(s) given its time and action type.
Revised: For each preﬁx of each held-out event sequence, we used minimum Bayes risk to predict the next event’s time, and to predict its participant(s) given its time and action type.

Original: On this dataset, we used the NDTT program written by Mei et al. (2020a). The rules are unchanged; the only difference is that our A-NDTT has the new continuous-time Transformer in lieu of the LSTM architecture. We also evaluated NHP and A-NHP on this dataset.
Revised: [Figure]

Original: The results are shown in Figure 4.
Revised: The results are shown in Figure 4.

Original: As in section 8.1, we found that A-NHP performed better than NHP on all the evaluation metrics;the difference on log-likelihood and event type prediction is signiﬁcant.
Revised: As in section 8.1, we ﬁnd that A-NHP performs better than NHP on all the evaluation metrics; on log-likelihood and event type prediction, A-NHP is signiﬁcantly better (paired permutation test, p < 0 . 05 ).

Original: The NDTT program injects appropriate domain knowledge (e.g., only the ball possessor can kick[Figure] or pass) into both LSTM and Transformer architectures, so both NDTT and A-NDTT substantially and signiﬁcantly outperform A-NHP.
Revised: We now inject domain knowledge into both the LSTM and Transformer approaches, by deriving architectures based on the RoboCup NDTT program (which speciﬁes, for example, that only the ball possessor can kick or pass).

Original: According to paired permutation tests, there is no signiﬁcant difference between A-NDTT and NDTT in the overall performance.
Revised: The resulting models—NDTT and A-NDTT—are substantial and signiﬁcant improvements.

Original: Perhaps NDTT already ﬁxes some of the problems with NHP, so it is not sorely in need of the Transformer’s ability to scan an unbounded history for relevant events.
Revised: The difference between A-NDTT and NDTT is not signiﬁcant: perhaps NDTT already ﬁxes some of the problems with NHP, so it is not sorely in need of the Transformer’s ability to scan an unbounded history for relevant events.

Original: However, as shown in Figure 5, their performance is not always comparable across all action types; similar ﬁgures in Figure 6 of Appendix C.6 were given by multiple runs with different random seeds. In terms ofdata ﬁtting (left ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while NDTT is better at the others. For time prediction (middle ﬁgure), A-NDTT is signiﬁcantly better at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal , kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff , and passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass events, but the differences for the other action types are not signiﬁcant.
Revised: Appendix C.6 includes more results of A-NDTT vs. NDTT broken down by action types.

Original: For action participant prediction (right ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while there is no difference for the others; both do perfectly well at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal and

Original: [Figure]

Original: events such that their dots overlap at the origin.

Original: [Figure] Additionally, while A-NDTT does not improve the overall accuracy for this particular NDTT program and dataset, it does achieve overall comparable accuracy with a simpler and shallower architecture ( (cid:173) – (cid:174) in section 1) that could, as future work, be trained on a GPU with parallelism; see Appendix C. for a detailed proposal.
Revised: Additionally, while A-NDTT does not improve the overall accuracy for this particular NDTT program and dataset, it does achieve overall comparable accuracy with a simpler and shallower architecture ( (cid:173) – (cid:174) in section 1). Like other Transformers, the A-NDTT architecture could be trained on a GPU with parallelism, as outlined in Appendix C.4 (future work).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We showed how to generalize the Transformer architecture to sequences of discrete events in continuous time.
Revised: We showed how to generalize the Transformer architecture to sequences of discrete events in continuous time.

Original: Through our novel continuous-time Transformer, we can incrementally build up rich embeddings of actual and possible events at any time t , from lower-level representations of contextual events.
Revised: Through our novel continuous-time Transformer, we can incrementally build up rich embeddings of actual and possible events at any time t , from lower-level representations of contextual events.

Original: The resulting probability model is highly ﬂexible, and enjoys great parallelism during training.
Revised: The resulting probability model is highly ﬂexible, and enjoys great parallelism during training.

Original: It outperforms other Transformer-based models on multiple synthetic and real-world datasets.
Revised: It outperforms other Transformer-based models on multiple synthetic and real-world datasets.

Original: We also showed how to integrate this architecture with NDTT, a neural-symbolic framework that automatically derives neural models from logic programs.
Revised: We also showed how to integrate this architecture with NDTT, a neural-symbolic framework that automatically derives neural models from logic programs.

Original: Our new version has shown competitive performance, despite having a simpler and shallower architecture.
Revised: Our new version has shown competitive performance, despite having a simpler and shallower architecture

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our paper develops a novel probabilistic model for event sequences.
Revised: Our paper develops a novel probabilistic model for event sequences.

Original: Good probabilistic models make it possible to impute missing events, anticipate possible future events and react accordingly.
Revised: Good probabilistic models make it possible to impute missing events, anticipate possible future events and react accordingly.

Original: They can also be used in exploratory data analysis.
Revised: They can also be used in exploratory data analysis.

Original: In addition to advancing the state-of-the-art in this area, our model may require less energy to train than previous models.
Revised: In addition to advancing the state-of-the-art in this area, our model may require less energy to train than previous models.

Original: We used multiple real-world datasets to evaluate our model and those datasets are all de-identiﬁed; see section 8 and Appendix C.1.
Revised: We used multiple real-world datasets to evaluate our model and those datasets are all de-identiﬁed; see section 8 and Appendix C.1.

Original: However, like many other machine learning models, our model may be applied to unethical ends.
Revised: However, like many other machine learning models, our model may be applied to unethical ends.

Original: For example, its abilities of better ﬁtting data and making more accurate predictions could potentially be used for unwanted tracking of individual behavior, e.g. for advertising or surveillance
Revised: For example, its abilities of better ﬁtting data and making more accurate predictions could potentially be used for unwanted tracking of individual behavior, e.g. for advertising or surveillance.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To facilitate the community to reproduce our results, we are committed to releasing our code when the paper is published.
Revised: To facilitate the community to reproduce our results, we are committed to releasing our code when the paper is published.

Original: For this submission, we made a thorough documentation of our methodology and experimental details in section 8 and Appendices A–C
Revised: For this submission, we made a thorough documentation of our methodology and experimental details in section 8 and Appendices A–C

Original: Appendices A L IKELIHOOD C OMPUTATION D ETAILS
Revised: Appendices A P ARAMETER D IMENSIONALITY S PECIFICATION FOR A-NDTT

Revised: In this section we discuss the dimensionality of the fact embeddings (cid:74) h (cid:75) ( t ) in section 5. The type of a fact in the database is given by its functor ( forum , member , createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate , etc.). All facts of the same type have embedding vectors of the same dimensionality, and these dimensionalities are declared by the NDTT program.

Revised: This is enough to determine the dimensions of the parameter matrices associated with the deduction rules (Mei et al., 2020a).

Revised: How about the add rules, however?

Revised: The form of equation (9) implies that the value vectors v ( (cid:96) )

Revised: r for add rule r have the same dimensionality as the embedding of the head of r .

Revised: The key and query vectors for rule r (as used in equation (2)) can share this dimensionality by default, although we are free to override this and specify a different dimensionality for them.

Revised: The foregoing choices determine the dimensions of the parameter matrices V ( (cid:96) )

Revised: r , K ( (cid:96) ) r , Q ( (cid:96) )

Revised: r associated with rule r .

Revised: A ﬁnal wrinkle is that while the add rule h <- f , g 1 , ..., g n mainly allows an h fact to attend to preceding f events, it also allows the h fact to attend to itself.

Revised: Unfortunately, we cannot extract h ’s attention key and value using V ( (cid:96) )

Revised: r and K ( (cid:96) ) r , because h may have a different dimensionality from f .

Revised: A reasonable approach would be to always use zero vectors for this key and value.

Revised: A L IKELIHOOD C OMPUTATION D ETAILS

Original: In this section we discuss the log-likelihood formulas in section 6.
Revised: In this section we discuss the log-likelihood formulas in section 6.

Original: The log-likelihood formula equation 6 has been derived and discussed in previous work (Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).
Revised: The log-likelihood formula equation 6 has been derived and discussed in previous work (Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).

Original: Intuitively, during parameter training, each log λ e i ( t i ) is increased to explain why e i happened at time t i while (cid:82) T t =

Revised: Intuitively, during parameter training, each log λ e i ( t i ) is increased to explain why e i happened at time t i while (cid:82) T t =

Original: λ ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .
Revised: λ ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .

Original: , E } ever happened at other times.
Revised: , E } ever happened at other times.

Original: Note that there is no log under the integral in equation (6).
Revised: Note that there is no log under the integral in equation (6).

Original: Why?
Revised: Why?

Original: The probability that there was not an event of any type in the inﬁnitesimally wide interval [ t, t + d t ) is 1 − λ ( t )d t , whose log is − λ ( t )d t .
Revised: The probability that there was not an event of any type in the inﬁnitesimally wide interval [ t, t + d t ) is 1 − λ ( t )d t , whose log is − λ ( t )d t .

Original: As discussed in section 6, the integral term in equation (6) is computed using the Monte Carlo approximation detailed by Algorithm 1 of Mei & Eisner (2017), which samples times t .
Revised: As discussed in section 6, the integral term in equation (6) is computed using the Monte Carlo approximation detailed by Algorithm 1 of Mei & Eisner (2017), which samples times t .

Original: However, at each sampled time t , that method still requires a summation over all events to obtain λ ( t ) .
Revised: However, at each sampled time t , that method still requires a summation over all events to obtain λ ( t ) .

Original: This summation can be expensive when there are many event types.
Revised: This summation can be expensive when there are many event types.

Original: This is not a serious problem for our standalone A-NHP implementation since it can leverage GPU parallelism.
Revised: This is not a serious problem for our standalone A-NHP implementation since it can leverage GPU parallelism.

Original: But for the general
Revised: But for the general

Original: A-NDTT implementation
Revised: A-NDTT implementation

Original: , it is hard to parallelize the λ k ( t ) computation over k and t .
Revised: , it is hard to parallelize the λ k ( t ) computation over k and t .

Original: In that case, we use the downsampling trick detailed in Appendix D of Mei et al.
Revised: In that case, we use the downsampling trick detailed in Appendix D of Mei et al.

Original: (2020a).
Revised: (2020a).

Original: An alternative is to use noise-contrastive estimation, which has shown success in training NHP and NDTT models (Mei et al., 2020b).
Revised: An alternative would be to replace maximum-likelihood estimation with noise-contrastive estimation, which is quite effective at training NHP and NDTT models (Mei et al., 2020b).

Original: B H OW TO P REDICT E VENTS
Revised: B H OW TO P REDICT E VENTS

Original: Section 8 include a task-based evaluation where we try to predict the time and type of the next event.
Revised: Section 8 includes a task-based evaluation where we try to predict the time and type of the next event.

Original: More precisely, for each event in each held-out sequence, we attempt to predict its time given only the preceding events, as well as its type given both its true time and the preceding events.
Revised: More precisely, for each event in each held-out sequence, we attempt to predict its time given only the preceding events, as well as its type given both its true time and the preceding events.

Original: We evaluate the time prediction with average L 2 loss (yielding a root-mean-squared error, or RMSE ) and evaluate the argument prediction with average 0-1 loss (yielding an error rate ).
Revised: We evaluate the time prediction with average L 2 loss (yielding a root-mean-squared error, or RMSE ) and evaluate the argument prediction with average 0-1 loss (yielding an error rate ).

Original:  Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle to predict the time and type with lowest expected loss.
Revised: [Table] Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle to predict the time and type with lowest expected loss.

Original: For completeness, we repeat the general recipe in this section.
Revised: For completeness, we repeat the general recipe in this section.

Original: For the i th event, its time t i has density p
Revised: For the i th event, its time t i has density p

Original: i ( t ) = λ ( t ) exp( − (cid:82) t t i − 1 λ ( t (cid:48) )
Revised: i ( t ) = λ ( t ) exp( − (cid:82) t t i − 1 λ ( t (cid:48) )

Original: d t (cid:48) ) .
Revised: d t (cid:48) ) .

Original: We choose (cid:82) ∞
Revised: We choose (cid:82) ∞

Original: t i − 1 tp i ( t )d t as the time prediction because it has the lowest expected L 2 loss.
Revised: t i − 1 tp i ( t )d t as the time prediction because it has the lowest expected L 2 loss.

Original: The integral can be estimated using i.i.d.
Revised: The integral can be estimated using i.i.d.

Original: samples of t i drawn from p
Revised: samples of t i drawn from p

Original: i ( t ) as detailed in Mei & Eisner (2017) and Mei et al. 
Revised: i ( t ) as detailed in Mei & Eisner (2017) and Mei et al. (2019).

Original: Given the next event time t i ,  the most likely typeis simply arg max e λ e ( t i ) ; that is the type prediction.
Revised: Given the next event time t i , we choose the most probable type arg max e λ e ( t i ) as the type prediction because it minimizes expected 0-1 loss.

Original: In some circumstances, one might also like to predict the most likely type out of a restricted set E (cid:48) (cid:40) { 1 , . . .
Revised: In some circumstances, one might also like to predict the most probable type out of a restricted set E (cid:48) (cid:40) { 1 , . . .

Original: , E } .
Revised: , E } .

Original: This allows one to answer questions like “If we know that some event  messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,T) happened at time t i , then which forum T did eve messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage in, given all past events?”
Revised: This allows one to answer questions like “If we know that some event of the form messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,T) happened at time t i , then what was the forum T , given all past events?”

Original: The answer will simply be arg max e ∈E (cid:48)
Revised: The answer will simply be arg max e ∈E (cid:48)

Original: λ
Revised: λ

Original: e ( t i ) .
Revised: e ( t i ) .

Original: C E XPERIMENTAL D ETAILS C. D ATASET D ETAILS
Revised: C E XPERIMENTAL D ETAILS C. D ATASET D ETAILS

Original: Table 1 shows statistics about each dataset that we use in this paper.
Revised: Table 1 shows statistics about each dataset that we use in this paper.

Original: C.1. S YNTHETIC D ATA D ETAILS
Revised: C.1. S YNTHETIC D ATA D ETAILS

Original: In this experiment, we draw data from randomly initialized NHP, A-NHP, SAHP, and THP.
Revised: In this experiment, we draw data from randomly initialized NHP, A-NHP, SAHP, and THP.

Original: For all of them, we take10 to be the number of event types.
Revised: For all of them, we take the number of event types to be E = 10 .

Original: For NHP, the dimensions of event embeddings and hidden states are all 32 ; for A-NHP, the number of layer is 2 , and the dimensions of time embeddings and event embeddings are 32 ; for SAHP, the number [Table]
Revised: For NHP, the dimensions of event embeddings and hidden states are all 32 ; for A-NHP, the number of layers ( L in our paper) is 2 , and the dimensions of time embeddings and event embeddings are 32 ; for SAHP, the number of layers is 4 , and the dimension of hidden states is 32 ; for THP, the number of layer is 7 , and the dimension of hidden states is 32 .

Original: For each model, we draw 800 , 100 , and 100 sequences for training, validation and testing, respectively.
Revised: For each model, we draw 800 , 100 , and 100 sequences for training, validation and testing, respectively.

Original: For each sequence, the sequence length is drawn from Uniform (49 , 99) .
Revised: For each sequence, the sequence length is drawn from Uniform (49 , 99) .

Original: C.1. O THER D ATA D ETAILS
Revised: C.1. O THER D ATA D ETAILS

Original: For MIMIC-II and StackOverﬂow, we used the version processed by Du et al.
Revised: For MIMIC-II and StackOverﬂow, we used the version processed by Du et al.

Original: (2016); more details (e.g., about processing) can be found in their paper.
Revised: (2016); more details (e.g., about processing) can be found in their paper.

Original: For RoboCup, we used the version processed by Chen & Mooney (2008); please refer to their paper for more details (e.g., data description, processing method, etc)
Revised: For RoboCup, we used the version processed by Chen & Mooney (2008); please refer to their paper for more details (e.g., data description, processing method, etc)

Original: C. I MPLEMENTATION D ETAILS
Revised: C. I MPLEMENTATION D ETAILS

Original: For NHP, our implementation is based on the public Github repositories at https://github.com/
Revised: For NHP, our implementation is based on the public Github repositories at https://github.com/

Original: HMEIatJHU/neurawkes (Mei & Eisner (2017), with MIT License) and https://github.com/
Revised: HMEIatJHU/neurawkes (Mei & Eisner (2017), with MIT License) and https://github.com/

Original: HMEIatJHU/neural-hawkes-particle-smoothing (Mei et al.
Revised: HMEIatJHU/neural-hawkes-particle-smoothing (Mei et al.

Original: (2019), with BSD 3-Clause “New” or “Revised” License).
Revised: (2019), with BSD 3-Clause “New” or “Revised” License).

Original: We made a considerable amount of modiﬁcations to their code (e.g., model, thinning algorithm), in order to migrate it to PyTorch 1.7.
Revised: We made a considerable amount of modiﬁcations to their code (e.g., model, thinning algorithm), in order to migrate it to PyTorch 1.7.

Original: We built the standalone GPU implementation of A-NHP upon our NHP code.
Revised: We built the standalone GPU implementation of A-NHP upon our NHP code.

Original:  For NDTT, we use the public Github repository at https://github.com/HMEIatJHU/ neural-datalog-through-time (Mei et al.
Revised: [Table] For NDTT, we used the public Github repository at https://github.com/HMEIatJHU/ neural-datalog-through-time (Mei et al.

Original: (2020a), with MIT License).
Revised: (2020a), with MIT License).

Original: We built A-NDTT upon NDTT.
Revised: We built A-NDTT upon NDTT.

Original: For THP, we use the public Github repository at https://github.com/SimiaoZuo/
Revised: For THP, we used the public Github repository at https://github.com/SimiaoZuo/

Original: Transformer-Hawkes-Process (Zuo et al.
Revised: Transformer-Hawkes-Process (Zuo et al.

Original: (2020), no license speciﬁed).
Revised: (2020), no license speciﬁed).

Original: For SAHP, we use the public Github repository at https://github.com/QiangAIResearcher/ sahp_repo (Zhang et al.
Revised: For SAHP, we used the public Github repository at https://github.com/QiangAIResearcher/ sahp_repo (Zhang et al.

Original: (2020a), no license speciﬁed).
Revised: (2020a), no license speciﬁed).

Original: C. T RAINING D ETAILS
Revised: C. T RAINING D ETAILS

Original: For each model in section 8,including the baseline models, we had to choose thedimension of their neural state (i.e., D ); for the attention-based models, we  had to choose the number of layers(i.e., L ) in the attention mechanism as well.
Revised: For each model in section 8, we had to specify various dimensionalities. For simplicity, we used a single hyperparameter D and took all vectors to be in R D . This includes the state vectors of NHP, the fact embeddings of NDTT and A-NDTT, and the query, key, and value vectors for the models with attention mechanisms (THP, SAHP, A-NHP, and A-NDTT). For the models with attention mechanisms, we also had to choose the number of layers L .

Original: We tuned these hyperparameters for each combination of model, dataset, and training size (e.g., each bar in Figures 2–4), always choosing the combination of D and L that achieved the best performance on the dev set.
Revised: We tuned these hyperparameters for each combination of model, dataset, and training size (e.g., each bar in Figures 2–4), always choosing the combination of D and L that achieved the best performance on the dev set.

Original: Our search spaces were D ∈ { 4 , 8 , 16 , 32 , 64 , 128 } and L ∈ { 1 , 2 , 3 , 4 , 5 , 6 } .
Revised: Our search spaces were D ∈ { 4 , 8 , 16 , 32 , 64 , 128 } and L ∈ { 1 , 2 , 3 , 4 , 5 , 6 } .

Original: In practice, the optimal D for a model was usually 32 or 64; the optimal L was usually 1, 2, or 3.
Revised: In practice, the optimal D for a model was usually 32 or 64; the optimal L was usually 1, 2, or 3.

Original: To train the parameters for a given model, we used the Adam algorithm (Kingma & Ba, 2015) with its default settings.
Revised: To train the parameters for a given model, we used the Adam algorithm (Kingma & Ba, 2015) with its default settings.

Original: We performed early stopping based on log-likelihood on the held-out dev set.
Revised: We performed early stopping based on log-likelihood on the held-out dev set.

Original: In our preliminary experiments, we also tried adding feed-forward layers and layer normalization (Ba et al., 2016) like in Vaswani et al. but they didn’t help.
Revised: In our preliminary experiments, we also tried adding feed-forward layers and layer normalization (Ba et al., 2016) as in the original Transformer architecture (Vaswani et al., 2017), but they didn’t help.

Original: For the experiments in section 8.1, we used the standalone PyTorch implementations for NHP and
Revised: For the experiments in section 8.1, we used the standalone PyTorch implementations for NHP and

Original: A-NHP, which are GPU-friendly.
Revised: A-NHP, which are GPU-friendly.

Original: We trained each model on an NVIDIA K80 GPU.
Revised: We trained each model on an NVIDIA K80 GPU.

Original: Table 2 shows their training time per sequence on each dataset.[Table]
Revised: Table 2 shows their training time per sequence on each dataset.

Original: We run our NDTT and A-NDTT models only on CPUs.
Revised: We run our NDTT and A-NDTT models only on CPUs.

Original: This follows Mei et al.
Revised: This follows Mei et al.

Original: (2020a), who did not ﬁnd an efﬁcient method to leverage GPU parallelism for training NDTT models.
Revised: (2020a), who did not ﬁnd an efﬁcient method to leverage GPU parallelism for training NDTT models.

Original: The machines we used for NDTT and A-NDTT are 6-core Haswell architectures.
Revised: The machines we used for NDTT and A-NDTT are 6-core Haswell architectures.

Original: On RoboCup, the training time of NDTT and A-NDTT is 62 and 149 seconds per sequence, respectively.
Revised: On RoboCup, the training time of NDTT and A-NDTT is 62 and 149 seconds per sequence, respectively.

Original: We used the same CPU implementation for NHP and A-NHP on RoboCup, rather than the standalone GPU implementation, since the RoboCup sequences are too long to ﬁt in the memory of the K80 GPU.
Revised: We used the same CPU implementation for NHP and A-NHP on RoboCup, rather than the standalone GPU implementation, since the RoboCup sequences are too long to ﬁt in the memory of the K80 GPU.

Original: We did this by expressing these models as NDTT programs.
Revised: We did this by expressing these models as NDTT programs.

Original: The NDTT program for NHP can be found in Appendix
Revised: The NDTT program for NHP can be found in Appendix

Original: F.2 of Mei et al.
Revised: F.2 of Mei et al.

Original: (2020a)
Revised: (2020a)

Original: ; the A-NDTT program for A-NHP can be found in Appendix C.5.1 below.
Revised: ; the A-NDTT program for A-NHP can be found in Appendix C.5.1 below.

Original: The training time is 66 and 95 seconds per sequence for NHP and A-NHP, respectively.
Revised: The training time is 66 and 95 seconds per sequence for NHP and A-NHP, respectively.

Original: C. T RAINING P ARALLELISM
Revised: C. T RAINING P ARALLELISM

Original: We point out that in the future, GPU parallelism could be exploited through the following procedure, given a GPU with enough memory to handle long training sequences.
Revised: We point out that in the future, GPU parallelism could be exploited through the following procedure, given a GPU with enough memory to handle long training sequences.

Original: (The layers can be partitioned across multiple GPUs if needed.)
Revised: (The layers can be partitioned across multiple GPUs if needed.)

Original: For each training minibatch, the ﬁrst step is to play each event sequence e 1@ t 1 , e 2@ t 2 , . . .
Revised: For each training minibatch, the ﬁrst step is to play each event sequence e 1@ t 1 , e 2@ t 2 , . . .

Original: , e I @
Revised: , e I @

Original: t I forward to determine the contents of the database on each interval (0 , t 1 ] , ( t 1 , t 2 ] , . . .
Revised: t I forward to determine the contents of the database on each interval (0 , t 1 ] , ( t 1 , t 2 ] , . . .

Original: , ( t i − 1 , t I ] , ( t I , T ] .
Revised: , ( t i − 1 , t I ] , ( t I , T ] .

Original: This step runs on CPU, and computes only the boolean facts (Datalog through time) without their embeddings (neural Datalog through time).
Revised: This step runs on CPU, and computes only the boolean facts (“Datalog through time”) without their embeddings (“neural Datalog through time”).

Original: Let F be the set of facts that ever appeared in the database during this minibatch and let R be the set of rules that were ever used to deduce or launch them (section 5).
Revised: Let F be the set of facts that ever appeared in the database during this minibatch and let R be the set of rules that were ever used to deduce or add them (section 5).

Original:  A computation graph of size O ( |R|· I ) can now be constructed, as illustrated in Figure 1b, to compute the embeddings (cid:74) h (cid:75) ( t ) of all facts h ∈ F at allevent times t = t i as well as all times t that are sampled for the Monte Carlo integral (Appendix A).
Revised: Furthermore, let T be the set of times consisting of { t 1 , . . . , t I } together with the times t that are sampled for the Monte Carlo integral (Appendix A). A computation graph of size O ( |R|· I ) can now be constructed, as illustrated in Figure 1b, to compute the embeddings (cid:74) h (cid:75) ( t ) of all facts h ∈ F at all times t ∈ T .

Original: The layer- (cid:96) embeddings at time t  depend on the layer-  (cid:96) − 1 embeddings at times s ≤ t , according to the launch rules in R .
Revised: The layer- (cid:96) embeddings at time t ∈ T depend on the layer- ( (cid:96) − 1) embeddings at times t i ≤ t , according to the add rules in R .

Original: The layer- (cid:96) embedding of a fact that is deduced at time t also depends on the layer- (cid:96) embeddings at time t of the facts that it is deduced from, according to the deduction rules in R ; this further increases the depth of the computation graph.
Revised: The layer- (cid:96) embedding of a fact that is deduced at time t also depends on the layer- (cid:96) embeddings at time t of the facts that it is deduced from, according to the deduction rules in R ; this further increases the depth of the computation graph.

Original: For a given fact h ∈ F , (cid:74) h (cid:75) ( (cid:96) )
Revised: For a given fact h ∈ F , (cid:74) h (cid:75) ( (cid:96) )

Original: ( t ) can be computed in parallel for all event sequences and all times t  (even times t when h is not true, although those embeddings will not be used).
Revised: ( t ) can be computed in parallel for all event sequences and all times t ∈ T (even times t when h is not true, although those embeddings will not be used).

Original: Multiple facts that are governed by the same NDTT rule  can also be handled in parallel.
Revised: Multiple facts that are governed by the same NDTT rule r ∈ R can also be handled in parallel, since they use the same r -speciﬁc parameters.

Original: Thus, a GPU can be effective for this phase.
Revised: Thus, a GPU can be effective for this phase.

Original: The computation of h ( (cid:96) ) r ( t ) in equation (9) must take care to limit its attention to just those earlier times when an event occurred that launched h via rule r , and the computation of [ h ] ( (cid:96) )
Revised: The computation of h ( (cid:96) ) r ( t ) in equation (9) must take care to limit its attention to just those earlier times when an event occurred that added h via rule r , and the computation of [ h ] ( (cid:96) )

Original: ( t ) in equation (9) must take care to consider only rules r that in fact deduce h at time t because their conditions are true at time t .
Revised: ( t ) in equation (9) must take care to consider only rules r that in fact deduce h at time t because their conditions are true at time t .

Original: This masks unwanted parts of the computation, rendering parts of the GPU idle.
Revised: This masks unwanted parts of the computation, rendering parts of the GPU idle.

Original: GPU parallelism will still be worthwhile if a substantial fraction of the computation remains unmasked—which is true for relatively homogenous settings where most facts in F hold true for a large portion of the observed interval [0 , T ) , even if their embeddings ﬂuctuate.
Revised: GPU parallelism will still be worthwhile if a substantial fraction of the computation remains unmasked—which is true for relatively homogenous settings where most facts in F hold true for a large portion of the observed interval [0 , T ) , even if their embeddings ﬂuctuate.

Original: C. A-NDTT P ROGRAMS C.5. A-NHP D ATALOG P ROGRAM
Revised: C. A-NDTT P ROGRAMS C.5. A-NHP D ATALOG P ROGRAM

Original: In this section, we show how to specify A-NHP (section 3) by an A-NDTT program:
Revised: In this section, we show how to specify A-NHP (section 3) by an A-NDTT program:

Original:  is_type(1).
Revised:  is_type(1).

Original: ... is_type( N ).
Revised: is_type( N )

Original:  :- embed (world, 8).
Revised:  :- embed (world, 8).

Original: :- event (eeeeeeeeeeeeeeeee, 8).
Revised: :- event (eeeeeeeeeeeeeeeee, 8).

Original: eeeeeeeeeeeeeeeee (N) :- world, is_type(N) :: prob(N).
Revised: eeeeeeeeeeeeeeeee(N) :- world, is_type(N) :: prob(N).

Original: world < initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit
Revised: world <- initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit.

Original: world <- eeeeeeeeeeeeeeeee(N
Revised: world <- eeeeeeeeeeeeeeeee(N).

Revised: where the embedding of world at time t attends to all past events via rule 7, and is used to derive the embeddings and intensities of all of the events eeeeeeeeeeeeeeeee (1) , . .

Revised: , eeeeeeeeeeeeeeeee ( N ) at time t , via rule 5.

Original: where the dashed underline for initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit indicates that it is an “exogenous” event that is not predicted by our model.
Revised: The dashed underline for initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit indicates that it is an “exogenous” event that is not predicted by our model.

Original: The initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit event always happens at time t = 0 .
Revised: The initinitinitinitinitinitinitinitinitinitinitinitinitinitinitinitinit event always happens at time t = 0 .

Original: The :: notation was mentioned in footnote 1.
Revised: The :: notation was mentioned in footnote 1.

Original: This program is different from the NDTT program that speciﬁes the NHP model: in that program, rule-7 will be replaced by a slightly fancier version, i.e., world <- eeeeeeeeeeeeeeeee(N), world (see Appendix F. of Mei et al.
Revised: This program is simpler than the NDTT program that speciﬁes the NHP model (see Appendix F. of Mei et al.

Original: (2020a)).
Revised: (2020a)).

Revised: In that program, rule 7 will be replaced by a slightly fancier version, namely world <- eeeeee eeeee eeeee e(N), world .

Original: That is becausethe world as an additional condition is necessary to resemble the traditional structure of an LSTM: the new LSTM cell of world is a linear combination of its old value and some input, where the linear coefﬁcients (i.e., input and forget gates) and the input depend on both eeeeeeeeeeeeeeeee(N) and the old value of world .
Revised: That is because world as an additional condition is necessary to resemble the traditional structure of an LSTM: the new LSTM cell of world is a linear combination of its old value and some input, where the linear coefﬁcients (i.e., input and forget gates) and the input depend on both eeeeeeeeeeeeeeeee(N) and the old value of world .

Original: Had we just used world <- eeeeeeeeeeeeeeeee(N) instead, the coefﬁcients and the input would only depend on eeeeeeeeeeeeeeeee(N) , giving a less rich recurrence.
Revised: Had we just used world <- eeeeeeeeeeeeeeeee(N) instead, the coefﬁcients and the input would only depend on eeeeeeeeeeeeeeeee(N) , giving a less rich recurrence.

Original: However, this additional richness is not necessary in our A-NHP because our multi-layer attention already captures sufﬁciently rich contextual information (hence “attention is all you need”).
Revised: However, this additional richness is not necessary in our A-NHP because our multi-layer attention already captures sufﬁciently rich contextual information (“attention is all you need”).

Original: C. M ORE R ESULTS
Revised: C. M ORE R ESULTS

Revised: The performance of A-NDTT and NDTT is not always comparable for speciﬁc action types, as shown in Figure 5 and Figure 6.

Revised: In terms of data ﬁtting (left ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while NDTT is better at the others.

Revised: For time prediction (middle ﬁgure), A-NDTT is signiﬁcantly better at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal , kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff , and passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass events, but the differences for the other action types are not signiﬁcant.

Revised: For action participant prediction (right ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while there is no difference for the others; both do perfectly well at the

Original: In Figure 6, we redraw Figure 5 by training A-NDTT with different random seeds.
Revised: [Figure] events such that their dots overlap at the origin. [Figure] Figure 6, we redraw Figure 5 by training A-NDTT with different random seeds.

Original: The ﬁgures look very similar, thus the per-action differences between NDTT and A-NDTT are fairly consistent.
Revised: The ﬁgures look very similar, thus the per-action differences between NDTT and A-NDTT are fairly consistent.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]