Original: T RANSFORMER E MBEDDINGS OF I RREGULARLY S PACED E VENTS AND T HEIR P ARTICIPANTS
Revised: T RANSFORMER E MBEDDINGS OF I RREGULARLY S PACED E VENTS AND T HEIR P ARTICIPANTS

Original: A BSTRACT
Revised: A BSTRACT

Original: The neural Hawkes process (Mei & Eisner, 2017) is a generative model of ir regularly spaced sequences of discrete events.
Revised: The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events.

Original: To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-speciﬁc pattern-matching rules ; future events are then conditioned on the database con - tents.
Revised: To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-speciﬁc pattern-matching rules); future events are then conditioned on the database contents.

Original: They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and possible event has a time -varying embedding that is derived from its symbolic provenance.
Revised: They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and possible event has a time-varying embedding that is derived from its symbolic provenance.

Original: In this paper, we modify both models, replacing their recurrent LSTM-based architectures with ﬂatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable.
Revised: In this paper, we modify both models, replacing their recurrent LSTM-based architectures with ﬂatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable.

Original: This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
Revised: This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It has recently become common to model event sequences by embedding each event into R D .
Revised: It has recently become common to model event sequences by embedding each event into R D .

Original: Event sequences are ubiquitous in real-world applications, such as healthcare, ﬁnance, education, commerce, gaming, audio, news, security, and social media.
Revised: Event sequences are ubiquitous in real-world applications, such as healthcare, ﬁnance, education, commerce, gaming, audio, news, security and social media.

Original: Event embeddings could be used in a variety of downstream applied tasks, similar to word token embeddings in BERT (Devlin et al., 2018).
Revised: Event embeddings could be used in a variety of downstream applied tasks, similar to word token embeddings in BERT (Devlin et al., 2018).

Original: In this paper, we embed each event using attention over the previous events, using continuous-time positional encodings so as to consider their timing.
Revised: In this paper we embed each event using attention over the previous events, using continuous-time positional encodings so as to consider their timing.

Original: To build a left-to-right generative model, we also embed possible events at future times in exactly the same way, and use their embeddings to predict their instantaneous probabilities at those times.
Revised: To build a left-to-right generative model, we also embed possible events at future times in exactly the same way, and use their embeddings to predict their instantaneous probabilities at those times.

Original: Attention-based models (Vaswani et al., 2017) have already become extremely popular for generative modeling of discrete-time sequences, such as natural-language documents (Radford et al., 2019;
Revised: Attention-based models (Vaswani et al., 2017) have already become extremely popular for generative modeling of discrete-time sequences, such as natural-language documents (Radford et al., 2019;

Original: Brown et al., 2020) and proteins (Rao et al., 2021).
Revised: Brown et al., 2020) and proteins (Rao et al., 2021).

Original: As we conﬁrm here, they are also effective for modeling sequences that are irregularly spaced in continuous time, even in lower-data regimes.
Revised: As we conﬁrm here, they are also effective for modeling sequences that are irregularly spaced in continuous time, even in lower-data regimes.

Original: Our past work on modeling event sequences (Mei & Eisner, 2017; Mei et al., 2019; 2020a;b) used neural architectures based on LSTMs (Hochreiter & Schmidhuber, 1997).
Revised: Our past work on modeling event sequences (Mei & Eisner, 2017; Mei et al., 2019; 2020a;b) used neural architectures based on LSTMs (Hochreiter & Schmidhuber, 1997).

Original: That is, predictions at time t were derived from a recurrent encoding of the sequence of timestamped events at times < t .
Revised: That is, predictions at time t were derived from a recurrent encoding of the sequence of timestamped events at times < t .

Original: However, an attention-based (Transformer-style) architecture has three potential advantages:
Revised: However, an attention-based (Transformer-style) architecture has three potential advantages:

Original: (cid:172)
Revised: (cid:172)

Original: A Transformer does not summarize the past.
Revised: A Transformer does not summarize the past.

Original: Our predictions at time t can examine an unboundedly large representation of the past (embeddings in R d of every event before t ), not merely a ﬁxeddimensional summary that was computed greedily from left to right (an LSTM’s state at time t ).
Revised: Our predictions at time t can examine an unboundedly large representation of the past (embeddings in R d of every event before t ), not merely a ﬁxeddimensional summary that was computed greedily from left to right (an LSTM’s state at time t ).

Original: (cid:173)
Revised: (cid:173)

Original: A Transformer’s computation graph is broader and shallower.
Revised: A Transformer’s computation graph is broader and shallower.

Original: The breadth makes it easier to learn long-distance inﬂuences.
Revised: The breadth makes it easier to learn long-distance inﬂuences.

Original: The shallowness does make it impossible to represent inherently deep concepts such as parity (Hahn, 2020), but it enables greater parallelism: the layer- (cid:96) embeddings can be computed in parallel during training, as they depend only on layer (cid:96) − 1 and not on one another.
Revised: The shallowness does make it impossible to represent inherently deep concepts such as parity (Hahn, 2020), but it enables greater parallelism: the layer- (cid:96) embeddings can be computed in parallel during training, as they depend only on layer (cid:96) − 1 and not on one another.

Original: (cid:174)
Revised: (cid:174)

Original: The Transformer architecture is simpler and arguably more natural, while remaining competitive in our experiments.To
Revised: The Transformer architecture is simpler and arguably more natural, while remaining competitive in our experiments.

Original:  model the temporal distribution of the next event
Revised: To model the temporal distribution of the next event, all of our models posit embeddings of possible future events that depend on the future event’s time t .

Original: all of our models posit embeddings of possible future events that depend
Revised: To accomplish this, Mei & Eisner (2017) had to stipulate an arbitrary family of parametric decay functions on t

Original: on
Revised: and

Original: the future event’s time t .

Original: To accomplish this,

Original: [Figure]
Revised: [Figure]

Original: Mei & Eisner (2017) had to stipulate an arbitrary family of parametric decay functions on t and the neuro-symbolic framework of Mei et al.
Revised: the neuro-symbolic framework of Mei et al.

Original: (2020a) required a complex method for pooling the parameters of these decay functions.
Revised: (2020a) required a complex method for pooling the parameters of these decay functions.

Original: But in our present method, no decay functions are required to allow embeddings and probabilities to drift over time.
Revised: But in our present method, no decay functions are required to allow embeddings and probabilities to drift over time.

Original: The embeddings are constructed “from scratch” at each time t simply by attending to the set of previous events, using t -speciﬁc query vectors that include a continuous positional embedding of t .
Revised: The embeddings are constructed “from scratch” at each time t simply by attending to the set of previous events, using t -speciﬁc query vectors that include a continuous positional embedding of t .

Original: As t increases, the attention weights over the previous events vary continuously with t , so the embeddings and probabilities do so as well.
Revised: As t increases, the attention weights over the previous events vary continuously with t , so the embeddings and probabilities do so as well.

Original: We present a series of increasingly sophisticated methods.
Revised: We present a series of increasingly sophisticated methods.

Original: Section 2 explains how to embed events in context (like continuous-time BERT).
Revised: Section 2 explains how to embed events in context (like continuous-time BERT).

Original: Section 3 turns this into a generative point process model that can predict the time and type of the next event (like continuous-time GPT).
Revised: Section 3 turns this into a generative point process model that can predict the time and type of the next event (like continuous-time GPT).

Original: In section 4, we allow a domain expert to write simple rules that control attention, constraining which events can “see” which previous events and with what parameters.
Revised: In section 4, we allow a domain expert to write simple rules that control attention, constraining which events can “see” which previous events and with what parameters.

Original: Finally, section 5 allows the domain expert to write more complex rules, using our previously published Datalog through time formalism (Mei et al., 2020a).
Revised: Finally, section 5 allows the domain expert to write more complex rules, using our previously published Datalog through time formalism (Mei et al., 2020a).

Original: These rules allow events to interact with a symbolic deductive database that tracks facts over time so that the neural architecture does not have to learn how to do so.
Revised: These rules allow events to interact with a symbolic deductive database that tracks facts over time so that the neural architecture does not have to learn how to do so.

Original: As in Mei et al.
Revised: As in Mei et al.

Original: (2020a), we deﬁne time-varying embeddings for all facts in the database and all events that are possible given those facts, using parameters associated with the rules that established the facts and possible events.
Revised: (2020a), we deﬁne time-varying embeddings for all facts in the database and all events that are possible given those facts, using parameters associated with the rules that established the facts and possible events.

Original: In the end, we arrive at attention-based versions of the NHP (Mei & Eisner, 2017) and NDTT (Mei et al., 2020a) frameworks, which we refer to as A-NHP (section 3) and A-NDTT (section 5).
Revised: In the end, we arrive at attention-based versions of the NHP (Mei & Eisner, 2017) and NDTT (Mei et al., 2020a) frameworks, which we refer to as A-NHP (section 3) and A-NDTT (section 5).

Original: We evaluate them in section 7, showing comparable or better accuracy.
Revised: We evaluate them in section 7, showing comparable or better accuracy.

Original: We release our code.
Revised: We release our code.

Original:  C ONTINUOUS -T IME T RANSFORMER FOR E MBEDDING E VENTS
Revised:  C ONTINUOUS -T IME T RANSFORMER FOR E MBEDDING E VENTS

Original: Suppose we observe I events over a ﬁxed time interval [0 , T ) .
Revised: Suppose we observe I events over a ﬁxed time interval [0 , T ) .

Original: Each event is denoted mnemonically as e @ t (i.e., “type e at time t ”), where e ∈ E is the type of event (drawn from a ﬁnite set E ).
Revised: Each event is denoted mnemonically as e @ t (i.e., “type e at time t ”), where e ∈ E is the type of event (drawn from a ﬁnite set E ).

Original: The observed event sequence is e 1@ t 1 , e 2@ t 2 , . . .
Revised: The observed event sequence is e 1@ t 1 , e 2@ t 2 , . . .

Original: , e I @
Revised: , e I @

Original: t I , where 0 < t 1 < t 2 < . . .
Revised: t I , where 0 < t 1 < t 2 < . . .

Original: < t I < T .
Revised: < t I < T .

Original: For any event e @ t , we can compute an embedding (cid:74) e (cid:75) ( t )
Revised: For any event e @ t , we can compute an embedding (cid:74) e (cid:75) ( t )

Original: ∈ R D by attending to its history H ( e @ t ) —a set of relevant events.
Revised: ∈ R D by attending to its history H ( e @ t ) —a set of relevant events.

Original: (For the moment, imagine that H ( e @ t ) consists of all the observed events e i @
Revised: (For the moment, imagine that H ( e @ t ) consists of all the observed events e i @

Original: t i .)
Revised: t i .)

Original: More precisely, (cid:74) e (cid:75) ( t ) is the concatenation of layer-wise embeddings (cid:74) e (cid:75) (0) ( t ) , (cid:74) e (cid:75) (1) ( t ) , . .
Revised: More precisely, (cid:74) e (cid:75) ( t ) is the concatenation of layer-wise embeddings (cid:74) e (cid:75) (0) ( t ) , (cid:74) e (cid:75) (1) ( t ) , . .

Original: , (cid:74) e (cid:75) ( L ) ( t ) .
Revised: , (cid:74) e (cid:75) ( L ) ( t ) .

Original: For (cid:96) > 0 , the layer- (cid:96) embedding of e @ t is computed as
Revised: For (cid:96) > 0 , the layer- (cid:96) embedding of e @ t is computed as

Original: [Equation]
Revised: [Equation]

Original: where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is
Revised: where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is

Original: [Equation]
Revised: [Equation]

Original: In layer (cid:96) , v ( (cid:96) ) , k ( (cid:96) ) , and q ( (cid:96) ) are known as the value , key , and query vectors and are extracted from the layer-( (cid:96) − 1 ) event embeddings using learned layer-speciﬁc matrices V (
Revised: In layer (cid:96) , v ( (cid:96) ) , k ( (cid:96) ) , and q ( (cid:96) ) are known as the value , key , and query vectors and are extracted from the layer-( (cid:96) − 1 ) event embeddings using learned layer-speciﬁc matrices V (

Original: [Table]
Revised: [Table]

Original: ) , K (
Revised: ) , K (

Original: [Table]
Revised: [Table]

Original: ) , Q ( (cid:96) ) :
Revised: ) , Q ( (cid:96) ) :

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: def
Revised: def

Original: [Table]
Revised: [Table]

Original: As the base case, (cid:74) e (cid:75)
Revised: As the base case, (cid:74) e (cid:75)

Original: [Table]
Revised: [Table]

Original: ) ( t ) def = (cid:74) e (cid:75)
Revised: ) ( t ) def = (cid:74) e (cid:75)

Original: [Table]
Revised: [Table]

Original: 0) is a learned embedding of the event type e .
Revised: 0) is a learned embedding of the event type e .

Original: (cid:74) t (cid:75) denotes an embedding of the time t .
Revised: (cid:74) t (cid:75) denotes an embedding of the time t .

Original: We cannot learn absolute embeddings for all real numbers, so we ﬁx (cid:74) t
Revised: We cannot learn absolute embeddings for all real numbers, so we ﬁx (cid:74) t

Original: [Table]
Revised: [Table]

Original: d = sin
Revised: d = sin

Original: [Table]
Revised: [Table]

Original: t/
Revised: t/

Original: [Table]
Revised: [Table]

Original: ) if
Revised: ) if

Original: [Table]
Revised: [Table]

Original: is even
Revised: is even

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where 0 ≤ d < D are the dimensions and our choices of m, M are explained in Appendix A.
Revised: where 0 ≤ d < D are the dimensions and our choices of m, M are explained in Appendix A.

Original: Crucially, to compute the layer- (cid:96) embedding of an event, equations (1)–(3) need only the layer-( (cid:96) − 1 ) embeddings of the relevant events in its history.
Revised: Crucially, to compute the layer- (cid:96) embedding of an event, equations (1)–(3) need only the layer-( (cid:96) − 1 ) embeddings of the relevant events in its history.

Original: This lets us compute the layer- (cid:96) embeddings of all events in parallel.
Revised: This lets us compute the layer- (cid:96) embeddings of all events in parallel.

Original: Note that equations (1)–(3) are simpliﬁcations of the traditional Transformer, since this ablation performed equally well in our pilot experiments (see Appendix A).
Revised: Note that equations (1)–(3) are simpliﬁcations of the traditional Transformer, since this ablation performed equally well in our pilot experiments (see Appendix A).

Original: The set of relevant events H ( e @ t ) could be deﬁned in a task-speciﬁc way.
Revised: The set of relevant events H ( e @ t ) could be deﬁned in a task-speciﬁc way.

Original: For example, to pretrain
Revised: For example, to pretrain

Original: BERT-like embeddings (Devlin et al., 2018)
Revised: BERT-like embeddings (Devlin et al., 2018)

Original: , we might use a corrupted version of { e 1@ t 1 , . . .
Revised: , we might use a corrupted version of { e 1@ t 1 , . . .

Original: , e I @
Revised: , e I @

Original: t I } in which some e i @
Revised: t I } in which some e i @

Original: t i have been removed or replaced with maskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmask @
Revised: t i have been removed or replaced with maskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmaskmask @

Original: t i .
Revised: t i .

Original: Such embeddings could be pretrained with a BERT-like objective and then ﬁne-tuned to predict properties of the observed events.
Revised: Such embeddings could be pretrained with a BERT-like objective and then ﬁne-tuned to predict properties of the observed events.

Original:  G ENERATIVE M ODELING OF C ONTINUOUS -T IME E VENT S EQUENCES
Revised:  G ENERATIVE M ODELING OF C ONTINUOUS -T IME E VENT S EQUENCES

Original: In this paper, we focus on the task of predicting future events given past ones.
Revised: In this paper, we focus on the task of predicting future events given past ones.

Original: At any time t , we would like to know what will happen at that time, given the actual events that happened before t .
Revised: At any time t , we would like to know what will happen at that time, given the actual events that happened before t .

Original: Our generative model is analogous to a Transformer language model (Radford et al., 2019; Brown et al., 2020), which, at each time t ∈ N , deﬁnes a probability distribution over the words in the vocabulary.
Revised: Our generative model is analogous to a Transformer language model (Radford et al., 2019; Brown et al., 2020), which, at each time t ∈ N , deﬁnes a probability distribution over the words in the vocabulary.

Original: In our setting, however, t ∈ R .
Revised: In our setting, however, t ∈ R .

Original: With probability 1, nothing happens at time t .
Revised: With probability 1, nothing happens at time t .

Original: Each possible event e in our vocabulary has only an inﬁnitesimal probability of occurring at time t .
Revised: Each possible event e in our vocabulary has only an inﬁnitesimal probability of occurring at time t .

Original: We write this probability as λ e ( t )d t where λ e ( t ) ∈ R
Revised: We write this probability as λ e ( t )d t where λ e ( t ) ∈ R

Original: + is called the (Poisson) intensity of type- e events at time t .
Revised: + is called the (Poisson) intensity of type- e events at time t .

Original: More formally, the probability of such an event occurring during [ t, t + (cid:15) ) approaches λ e ( t ) (cid:15) as (cid:15) → + 0 .
Revised: More formally, the probability of such an event occurring during [ t, t + (cid:15) ) approaches λ e ( t ) (cid:15) as (cid:15) → + 0 .

Original: Thus, our modeling task is to model
Revised: Thus, our modeling task is to model

Original: λ e ( t ) (as in, e.g., Hawkes, 1971; Du et al., 2016; Mei & Eisner, 2017).
Revised: λ e ( t ) (as in, e.g., Hawkes, 1971; Du et al., 2016; Mei & Eisner, 2017).

Original: We model
Revised: We model

Original: λ e ( t ) as a function of the top-layer embedding of the possible event e @ t : λ e ( t ) def =
Revised: λ e ( t ) as a function of the top-layer embedding of the possible event e @ t : λ e ( t ) def =

Original: softplus( w (cid:62) e [1; (cid:74) e
Revised: softplus( w (cid:62) e [1; (cid:74) e

Original: (cid:75) L ( t )] , τ e ) where softplus( x, τ ) =
Revised: (cid:75) L ( t )] , τ e ) where softplus( x, τ ) =

Original: [Equation]
Revised: [Equation]

Original: (1 +
Revised: (1 +

Original: [Equation]
Revised: [Equation]

Original: ( x/
Revised: ( x/

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (5) with learnable parameters w e and τ e > 0 .
Revised: (5) with learnable parameters w e and τ e > 0 .

Original: We do this separately for each possible e @ t , computing the embedding (cid:74) e
Revised: We do this separately for each possible e @ t , computing the embedding (cid:74) e

Original: (cid:75) L ( t ) using equations (1)–(3).
Revised: (cid:75) L ( t ) using equations (1)–(3).

Original: The softplus transfer function is inherited from the neural Hawkes process (Mei & Eisner, 2017).
Revised: The softplus transfer function is inherited from the neural Hawkes process (Mei & Eisner, 2017).

Original: To ensure that our model is generative, we compute (cid:74) e (cid:75) ( t ) from only previous events.
Revised: To ensure that our model is generative, we compute (cid:74) e (cid:75) ( t ) from only previous events.

Original: That is, H ( e @ t ) in equation (1) may contain any or all of the previously generated events e i @
Revised: That is, H ( e @ t ) in equation (1) may contain any or all of the previously generated events e i @

Original: t i for t i < t , but it may not contain any for which t i > t .
Revised: t i for t i < t , but it may not contain any for which t i > t .

Original: We call this model the attentive Neural Hawkes process , or A-NHP , and evaluate it in section 7.
Revised: We call this model the attentive Neural Hawkes process , or A-NHP , and evaluate it in section 7.

Original: Our model’s log-likelihood has the same form as for any autoregressive multivariate point process:
Revised: Our model’s log-likelihood has the same form as for any autoregressive multivariate point process:

Original: [Equation]
Revised: [Equation]

Original: Derivations of this formula can be found in previous work (e.g., Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).
Revised: Derivations of this formula can be found in previous work (e.g., Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).

Original: We can estimate the parameters by locally maximizing the log-likelihood (6) by any stochastic gradient method.
Revised: We can estimate the parameters by locally maximizing the log-likelihood (6) by any stochastic gradient method.

Original: Intuitively, each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .
Revised: Intuitively, each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .

Original: , E } ever happened at other times.
Revised: , E } ever happened at other times.

Original: Appendix D gives training details, including Monte Carlo approximations to the integral in equation (6), as well as noting alternative training objectives.
Revised: Appendix D gives training details, including Monte Carlo approximations to the integral in equation (6), as well as noting alternative training objectives.

Original: Given the learned parameters, we may wish to sample from the model given the past history, or make a minimum Bayes risk prediction about the next event.
Revised: Given the learned parameters, we may wish to sample from the model given the past history, or make a minimum Bayes risk prediction about the next event.

Original: Recipes can be found in Appendix E.
Revised: Recipes can be found in Appendix E.

Original: Notice that equation (5) is rather expensive compared to previous work, since it computes a deep embedding of the possible event e @ t just for the purpose of ﬁnding its intensity—and the algorithms of Appendices D–E require computing the intensities of many possible events.
Revised: Notice that equation (5) is rather expensive compared to previous work, since it computes a deep embedding of the possible event e @ t just for the purpose of ﬁnding its intensity—and the algorithms of Appendices D–E require computing the intensities of many possible events.

Original: Appendix A offers a speedup that shares embeddings among similar events, but it also explains why different events may sometimes have to be embedded differently to support the selective attention in sections 4–5 below.
Revised: Appendix A offers a speedup that shares embeddings among similar events, but it also explains why different events may sometimes have to be embedded differently to support the selective attention in sections 4–5 below.

Original:  M ULTI -H EAD S ELECTIVE A TTENTION
Revised:  M ULTI -H EAD S ELECTIVE A TTENTION

Original: We now present a simple initial version of selective attention.
Revised: We now present a simple initial version of selective attention.

Original: As in a graphical model, not all events should be able to inﬂuence one another directly.
Revised: As in a graphical model, not all events should be able to inﬂuence one another directly.

Original: Consider a scenario with two event types: eeeeeeeeeeeeeeeee means that Eve emails Adam, while fffffffffffffffff means that Frank emails Eve.
Revised: Consider a scenario with two event types: eeeeeeeeeeeeeeeee means that Eve emails Adam, while fffffffffffffffff means that Frank emails Eve.

Original: As Frank does not know when Eve emailed Adam, past events of type eeeeeeeeeeeeeeeee cannot inﬂuence his behavior.
Revised: As Frank does not know when Eve emailed Adam, past events of type eeeeeeeeeeeeeeeee cannot inﬂuence his behavior.

Original: Therefore, H ( fffffffffffffffff @ t ) should include past events of type fffffffffffffffff but not eeeeeeeeeeeeeeeee , so that the embedding of fffffffffffffffff @ t and hence the intensity function λ fffffffffffffffff ( t ) pay zero attention to eeeeeeeeeeeeeeeee events.
Revised: Therefore, H ( fffffffffffffffff @ t ) should include past events of type fffffffffffffffff but not eeeeeeeeeeeeeeeee , so that the embedding of fffffffffffffffff @ t and hence the intensity function λ fffffffffffffffff ( t ) pay zero attention to eeeeeeeeeeeeeeeee events.

Original: In contrast, H ( eeeeeeeeeeeeeeeee @ t ) should still include past events of both types, since both are visible to Eve and can inﬂuence her behavior.
Revised: In contrast, H ( eeeeeeeeeeeeeeeee @ t ) should still include past events of both types, since both are visible to Eve and can inﬂuence her behavior.

Original: We describe this situation with the edges fffffffffffffffff <- fffffffffffffffff , eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee , eeeeeeeeeeeeeeeee <- fffffffffffffffff .
Revised: We describe this situation with the edges fffffffffffffffff <- fffffffffffffffff , eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee , eeeeeeeeeeeeeeeee <- fffffffffffffffff .

Original: These are akin to the edges in a directed graphical model.
Revised: These are akin to the edges in a directed graphical model.

Original: They specify the sparsity pattern of the inﬂuence matrix (or Granger causality matrix ) that describes which past events can inﬂuence which future events.
Revised: They specify the sparsity pattern of the inﬂuence matrix (or Granger causality matrix ) that describes which past events can inﬂuence which future events.

Original: There is a long history of estimating this matrix from observed sequence data (e.g., Xu et al., 2016; Zhang et al., 2021), even with neural inﬂuence models (Zhang et al., 2020b).
Revised: There is a long history of estimating this matrix from observed sequence data (e.g., Xu et al., 2016; Zhang et al., 2021), even with neural inﬂuence models (Zhang et al., 2020b).

Original: In the present paper, however, we do not attempt to estimate this sparsity pattern, but assume it is provided by a human domain expert.
Revised: In the present paper, however, we do not attempt to estimate this sparsity pattern, but assume it is provided by a human domain expert.

Original: Incorporating such domain knowledge into the model can reduce the amount of training data that is needed.
Revised: Incorporating such domain knowledge into the model can reduce the amount of training data that is needed.

Original: Edges like eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple cases of  NDTT rules, whose general form will be given in section 5.
Revised: Edges like eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple cases of the NDTT rules in section 5 below.

Original: These rules also affect how we apply attention.
Revised: Such rules also affect how we apply attention.

Original: When Eve decides whether to email Adam ( eeeeeeeeeeeeeeeee @ t ), we may reasonably suppose that she separately considers the embeddings of the past eeeeeeeeeeeeeeeee events (e.g., “when were my last relevant emails to Adam?”) versus the past fffffffffffffffff events (e.g., “what have I heard recently from Frank?”).
Revised: When Eve decides whether to email Adam ( eeeeeeeeeeeeeeeee @ t ), we may reasonably suppose that she separately considers the embeddings of the past eeeeeeeeeeeeeeeee events (e.g., “when were my last relevant emails to Adam?”) versus the past fffffffffffffffff events (e.g., “what have I heard recently from Frank?”).

Original: Hence, we associate different attention heads with the two rules that affect eeeeeeeeeeeeeeeee , namely eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee and eeeeeeeeeeeeeeeee <- fffffffffffffffff .
Revised: Hence, we associate different attention heads with the two rules that affect eeeeeeeeeeeeeeeee , namely eeeeeeeeeeeeeeeee <- eeeeeeeeeeeeeeeee and eeeeeeeeeeeeeeeee <- fffffffffffffffff .

Original: These heads may have different parameters, so that they seek out and obtain different information from the past via different queries, keys, and values.
Revised: These heads may have different parameters, so that they seek out and obtain different information from the past via different queries, keys, and values.

Original: In general, we replace equation (1) with
Revised: In general, we replace equation (1) with

Original: [Equation]
Revised: [Equation]

Original: where r in the summation ranges over rules e <- · · · .
Revised: where r in the summation ranges over rules e <- · · · .

Original: The history H r ( e @ t ) contains only those past events f @ s that rule r makes visible to e .
Revised: The history H r ( e @ t ) contains only those past events f @ s that rule r makes visible to e .

Original: If there are no such events, or they have small attention weights (are only weakly relevant to e @ t ) as discussed in Appendix A, then rule r will contribute little or nothing to the sum in equation (7).
Revised: If there are no such events, or they have small attention weights (are only weakly relevant to e @ t ) as discussed in Appendix A, then rule r will contribute little or nothing to the sum in equation (7).

Original: The attention weights α r and vectors v r are deﬁned using versions of equations (2)–(3) with r -speciﬁc parameters.
Revised: The attention weights α r and vectors v r are deﬁned using versions of equations (2)–(3) with r -speciﬁc parameters.

Original: In short, each rule looks at the context separately, through its own attention weights determined by its own parameters.
Revised: In short, each rule looks at the context separately, through its own attention weights determined by its own parameters.

Original: The rule already speciﬁes symbolically which past events can get nonzero attention in the ﬁrst place, so it makes sense for the rule to also provide the parameters that determine the attention weights and value projections.
Revised: The rule already speciﬁes symbolically which past events can get nonzero attention in the ﬁrst place, so it makes sense for the rule to also provide the parameters that determine the attention weights and value projections.

Original: Further discussion is given in Appendix A.
Revised: Further discussion is given in Appendix A.

Original:  A TTENTIVE N EURAL D ATALOG T HROUGH T IME
Revised:  A TTENTIVE N EURAL D ATALOG T HROUGH T IME

Original: Edges such as eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple examples of rules in an NDTT program (Mei et al., 2020a, section 2).
Revised: Edges such as eeeeeeeeeeeeeeeee <- fffffffffffffffff can be regarded as simple examples of rules in an NDTT program (Mei et al., 2020a, section 2).

Original: We brieﬂy review this formalism and then extend our approach from section 4 to handle all NDTT programs.
Revised: We brieﬂy review this formalism and then extend our approach from section 4 to handle all NDTT programs.

Original: A Datalog through time (DTT) program describes possible sequences of events, much as a regular expression describes legal sequences of characters.
Revised: A Datalog through time (DTT) program describes possible sequences of events, much as a regular expression describes legal sequences of characters.

Original: A DTT program for a particular domain speciﬁes how each event automatically updates a database , adding or removing facts.
Revised: A DTT program for a particular domain speciﬁes how each event automatically updates a database , adding or removing facts.

Original: In this way, the past events e 1 , . . .
Revised: In this way, the past events e 1 , . . .

Original: , e i  construct a database.
Revised: , e i sequentially construct a database.

Original: This database  determines which event types (if any) can happen next:
Revised: This database then determines which event types (if any) can happen next:

Original: the next event e i +1 can be fffffffffffffffff only if fffffffffffffffff is currently a fact in the database.
Revised: the next event e i +1 can be fffffffffffffffff only if fffffffffffffffff is currently a fact in the database.

Original: Following Mei et al.
Revised: Thus, an event may appear in the database as a fact, meaning that the event is possible. We use variables e, f to range over events, but variables g, h to range over any facts (both events and nonevents).

Original: (2020a), we use orange to indicate the special facts such as fffffffffffffffff that specify possible events.
Revised: Literal examples of facts are shown in orange if they are events (e.g., fffffffffffffffff ), and in blue otherwise.

Original: A neural Datalog through time (NDTT) program is a DTT program augmented with some dimensionality declarations (Appendix C).
Revised: A neural Datalog through time (NDTT) program is a DTT program augmented with some dimensionality declarations (Appendix C).

Original: A rule that adds a fact to the database now also computes a vector embedding of that fact, or updates the existing embedding if the fact was already in the database.
Revised: A rule that adds a fact to the database now also computes a vector embedding of that fact, or updates the existing embedding Notice that the dimensionality of the embedded database changes as the database grows and shrinks over time.

Revised: Nonetheless, the model has a ﬁxed number of parameters associated with the ﬁxed set of rules of the NDTT program.

Revised: As we will see, rules can contain variables, allowing a small set of rules to model a large set of event types (i.e., parameter sharing).

Original: If fffffffffffffffff is a fact in the database at time t , meaning that event fffffffffffffffff is possible, then its embedding (cid:74) fffffffffffffffff (cid:75) L ( t ) determines its intensity λ fffffffffffffffff ( t ) via equation (5), as before.
Revised: If fffffffffffffffff is a fact in the database at time t , meaning that event fffffffffffffffff is possible at time t , then its embedding (cid:74) fffffffffffffffff (cid:75) L ( t ) determines its intensity λ fffffffffffffffff ( t ) via equation (5), as before.

Original: Thus, where a DTT program only describes which event sequences are possible , an NDTT program also describes how probable they are.
Revised: Thus, where a DTT program only describes which event sequences are possible , an NDTT program also describes how probable they are.

Original: Although the set of database facts isonly modiﬁed by events, their embeddings are time-sensitive and evolve as the events that added them to the database recede into the past.
Revised: Although the set of database facts is modiﬁed only when an event occurs, the facts’ embeddings are time-sensitive and evolve as the events that added them to the database recede into the past.

Original: This allows event intensities such as λ fffffffffffffffff ( t ) to wax and wane continously as time elapses.
Revised: This allows event intensities such as λ fffffffffffffffff ( t ) to wax and wane continously as time elapses.

Original: Notice that the dimensionality of the model state changes as the database grows and shrinks over time.

Original: However, the model has a ﬁxed number of parameters associated with the ﬁxed set of rules of the NDTT program.

Original: As we will see, rules can contain variables, allowing a small set of rules to model a large set of event types (i.e., parameter sharing).

Original: Datalog.
Revised: Datalog.

Revised: We now give details.

Original: Recall that Datalog (Ceri et al., 1989) is a traditional formalism for deductive databases.
Revised: We begin with Datalog (Ceri et al., 1989), a traditional formalism for deductive databases.

Original: A deductive database holds both extensional facts , which are placed there by some external process, and intensional facts , which are transitively deduced from the extensional facts.
Revised: A deductive database holds both extensional facts , which are placed there by some external process, and intensional facts , which are transitively deduced from the extensional facts.

Original: A Datalog program is simply a set of rules that govern these deductions:
Revised: A Datalog program is simply a set of rules that govern these deductions:

Original: • h :- g 1 , ..., g n says to deduce h at any time t when g 1 , . . .
Revised: • h :- g 1 , ..., g n says to deduce h at any time t when g 1 , . . .

Original: , g n are all true (in the database).
Revised: , g n are all true (in the database)

Original: A single rule can license many deductions.
Revised: A single rule can license many deductions.

Original: That is because the facts can have structured names, and h, g 1 , . . .
Revised: That is because the facts can have structured names, and h, g 1 , . . .

Original: g n can be patterns that match against those names, using capitalized identiﬁers as variables.
Revised: g n can be patterns that match against those names, using capitalized identiﬁers as variables.

Original: A model of ﬁlesystem properties might have a rule like openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D) :- user(U), group(G), document(D), member(U,G), readable(D,G) .
Revised: A model of ﬁlesystem properties might have a rule like openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D) :- user(U), group(G), document(D), member(U,G), readable(D,G) .

Original: In English, this says that U can open D at any time when user U is a member of some group G such that document D is readable by G .
Revised: In English, this says that U can open D at any time when user U is a member of some group G such that document D is readable by G .

Original: Datalog Through Time.
Revised: Datalog through time.

Original: Whenever extensional facts are added or removed, the intensional facts are instantly recomputed according to the deductive rules.
Revised: Whenever extensional facts are added or removed, the intensional facts are instantly recomputed according to the deductive rules.

Original: Datalog Through Time is an extension in which extensional facts are automatically added and removed when the database is notiﬁed of the occurrence of events.
Revised: DTT is an extension in which extensional facts are automatically added and removed when the database is notiﬁed of the occurrence of events.

Original: This behavior is governed by two additional rule types:
Revised: This behavior is governed by two additional rule types:

Original: • h <- f , g 1 , ..., g n says to add h at any time s when event f occurs and the g i are all true.
Revised: • h <- f , g 1 , ..., g n says to add h at any time s when event f occurs and the g i are all true.

Original: h <- f , g 1 , ..., g n says to remove h at any time s satisfying the same conditions.
Revised: h <- f , g 1 , ..., g n says to remove h at any time s satisfying the same conditions.

Original: Thus, the proposition h is true at time t (i.e., appears as a fact in the database at time t ) iff either (cid:182) h is deduced at time t , or (cid:183) h was added at some time s < t and never removed at any time in ( s, t ) .
Revised: Thus, the proposition h is true at time t (i.e., appears as a fact in the database at time t ) iff either (cid:182) h is deduced at time t , or (cid:183) h was added at some time s < t and never removed at any time in ( s, t ) .

Original: In our previous example, editing(U,D) <- openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D), member(U,G), writeable(U,G) records in the database that user U is editing D , once they have opened it with appropriate permissions.
Revised: In our previous example, editing(U,D) <- openopenopenopenopenopenopenopenopenopenopenopenopenopenopenopenopen(U,D), member(U,G), writeable(U,G) records in the database that user U is editing D , once they have opened it with appropriate permissions.

Original: (As a result, edit events might become possible via a deductive rule editeditediteditediteditediteditediteditediteditediteditediteditedit(U,D) :- editing(U,D) .)
Revised: (As a result, edit events might become possible via a deductive rule editeditediteditediteditediteditediteditediteditediteditediteditedit(U,D) :- editing(U,D) .)

Original: Neural Datalog Through Time.
Revised: Neural Datalog through time.

Original: It would be difﬁcult to train a neural architecture to encode thousands or millions of structured boolean facts about the world in its state and to modify thoseencoded facts systematically and transitively in response to  events.
Revised: It would be difﬁcult to train a neural architecture to encode thousands or millions of structured boolean facts about the world in its state and to systematically keep those facts up to date in response to possibly rare events.

Original: As a neuro-symbolic method, NDTT delegates that task to a symbolic database governed by manually speciﬁed Datalog Through Time rules.
Revised: As a neuro-symbolic method, NDTT delegates that task to a symbolic database governed by manually speciﬁed DTT rules.

Original: However, it  augments the database by neurally embedding every fact in a vector space. Thus, iff a proposition h appears as a fact in the NDTT database at time t , it will be associated not only with the simple truth value true but also with an embedding (cid:74) h (cid:75) ( t ) .
Revised: However, it also augments the database: iff a proposition h appears as a fact in the NDTT database at time t , it will be associated not only with the simple truth value true but also with an embedding (cid:74) h (cid:75) ( t ) .

Original: This embedding is a learned representation of that fact at that time, and can be trained to be useful in downstream tasks.
Revised: This embedding is a learned representation of that fact at that time, and can be trained to be useful in downstream tasks.

Original: It captures details of when and how that fact was established (the fact’s provenance ), since it is computed using learned parameters associated with the rules that deduced and/or added it.
Revised: It captures details of when and how that fact was established (the fact’s provenance ), since it is computed using learned parameters associated with the rules that deduced and/or added it.

Original: For example, a user’s embedding might be constructed using attention over all the past events that have affected the user, via rules of the form user(U) <- · · · .
Revised: For example, a user’s embedding might be constructed using attention over all the past events that have affected the user, via rules of the form user(U) <- · · · .

Original: This summarizes the user’s state.
Revised: This summarizes the user’s state.

Original: Similarily, a document’s embedding might be constructed using attention over all the edits to it, considering the editing user’s state at the time of the edit: document(D) <- edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D), user(U) .
Revised: Similarily, a document’s embedding might be constructed using attention over all the edits to it, considering the editing user’s state at the time of the edit: document(D) <- edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D)edit(U,D), user(U) .

Original: Embeddings from NDTT rules.
Revised: Embeddings from NDTT rules.

Original: Our goal is to provide new formulas for the embeddings (cid:74) h (cid:75) ( t ) , based on Transformer-style attention rather than LSTM-style recurrence.
Revised: Our goal is to provide new formulas for the embeddings (cid:74) h (cid:75) ( t ) , based on Transformer-style attention rather than LSTM-style recurrence.

Original: We call this attentive
Revised: We call this attentive

Original: NDTT , or A-NDTT .
Revised: NDTT , or A-NDTT .

Original: This gives a new way to map an NDTT program to a neural architecture.
Revised: This gives a new way to map an NDTT program to a neural architecture.

Original: The potential advantages for accuracy, efﬁciency, and simplicity were explained in section 1.
Revised: The potential advantages for accuracy, efﬁciency, and simplicity were explained in section 1.

Original: Intuitively, the <- rules will govern the “horizontal” ﬂow of information through time (by deﬁning attentional connections as we saw in section 4), while the :- rules will govern the “vertical” ﬂow of information at a given time (by deﬁning feed-forward connections).
Revised: Intuitively, the <- rules will govern the “horizontal” ﬂow of information through time (by deﬁning attentional connections as we saw in section 4), while the :- rules will govern the “vertical” ﬂow of information at a given time (by deﬁning feed-forward connections).

Original: These are, of course, the two major mechanisms in Transformer architectures.
Revised: These are, of course, the two major mechanisms in Transformer architectures.

Original: Under A-NDTT, the layer- (cid:96) embedding of h @ t is
Revised: Under A-NDTT, the layer- (cid:96) embedding of h @ t is

Original: [Equation]
Revised: [Equation]

Original: which is an augmented version of equation (7).
Revised: which is an augmented version of equation (7).

Original: Suppose h is true at time t because it was added by rule r (i.e., condition (cid:183) ).
Revised: Suppose h is true at time t because it was added by rule r (i.e., condition (cid:183) ).

Original: Then the summand h ( (cid:96) )
Revised: Then the summand h ( (cid:96) )

Original: r ( t ) exists and is computed much as in equation (8), now with attention over all “add times” s .
Revised: r ( t ) exists and is computed much as in equation (8), now with attention over all “add times” s .

Original: In other words, H r ( h @ t ) in equation (8) includes just those past events f @ s such that f added h via r at some time s < t and h was never removed at any time in ( s, t ) .
Revised: In other words, H r ( h @ t ) in equation (8) includes just those past events f @ s such that f added h via r at some time s < t and h was never removed at any time in ( s, t ) .

Original: It also includes h @ t itself, as before.
Revised: It also includes h @ t itself, as before.

Original: More precisely, when the rule h <- f , g 1 , ..., g n causes h @ t to attend to the speciﬁc past event f @ s , we actually want attention to consider the embedding at time s not just of f , but of the entire add condition f, g 1 , . .
Revised: More precisely, when the rule h <- f , g 1 , ..., g n causes h @ t to attend to the speciﬁc past event f @ s , we actually want attention to consider the embedding at time s not just of f , but of the entire add condition f, g 1 , . .

Original: . , g n .
Revised: . , g n .

Original: Thus, we replace f @ s with ( f, g 1 , . .
Revised: Thus, we replace f @ s with ( f, g 1 , . .

Original: . , g n ) @
Revised: . , g n ) @

Original: s throughout equation (8).
Revised: s throughout equation (8).

Original: The attention key of this add condition is deﬁned as k ( (cid:96) )
Revised: The attention key of this add condition is deﬁned as k ( (cid:96) )

Original: (( f, g 1 , . .
Revised: (( f, g 1 , . .

Original: . , g n ) @ s ) def =
Revised: . , g n ) @ s ) def =

Original: K ( (cid:96) ) r (cid:2) 1; (cid:74)
Revised: K ( (cid:96) ) r (cid:2) 1; (cid:74)

Original: [Equation]
Revised: [Equation]

Original: 1) ( s ); (cid:74) g 1 (cid:75) ( (cid:96) − 1) ( s ); . . .
Revised: 1) ( s ); (cid:74) g 1 (cid:75) ( (cid:96) − 1) ( s ); . . .

Original: (cid:74) g n (cid:75) ( (cid:96) − 1) ( s ) (cid:3) (compare equation (3b)).
Revised: (cid:74) g n (cid:75) ( (cid:96) − 1) ( s ) (cid:3) (compare equation (3b)).

Original: Its attention value v ( (cid:96) )
Revised: Its attention value v ( (cid:96) )

Original: (( f, g 1 , . .
Revised: (( f, g 1 , . .

Original: . , g n ) @ s ) is deﬁned analogously, using a different matrix V ( (cid:96) ) r .
Revised: . , g n ) @ s ) is deﬁned analogously, using a different matrix V ( (cid:96) ) r .

Original: The above handles the <- rules.
Revised: The above handles the <- rules.

Original: As for the :- rules, the vector [ h ] ( (cid:96) ) ( t ) in equation (9) sums over all the ways that h can be deduced at time t (i.e., condition (cid:182) ).
Revised: As for the :- rules, the vector [ h ] ( (cid:96) ) ( t ) in equation (9) sums over all the ways that h can be deduced at time t (i.e., condition (cid:182) ).

Original: This does not involve attention, so we exactly follow Mei et al.
Revised: This does not involve attention, so we exactly follow Mei et al.

Original: (2020a, equations (3)–(6)):
Revised: (2020a, equations (3)–(6)):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where r ranges over rules, and ( g 1 , . .
Revised: where r ranges over rules, and ( g 1 , . .

Original: g n ) ranges over all tuples of facts at time t such that h :- g 1 , . . .
Revised: g n ) ranges over all tuples of facts at time t such that h :- g 1 , . . .

Original: g n matches rule r (and thus deduces h at time t ).
Revised: g n matches rule r (and thus deduces h at time t ).

Original: The operator (cid:76) β r is a softmax-pooling operator with a learned inverse temperature β r .
Revised: The operator (cid:76) β r is a softmax-pooling operator with a learned inverse temperature β r .

Original: If h is not deduced at time t by any instantiation of r , then r has no effect on the sum (10), since pooling the empty set with (cid:76) β r returns 0 .
Revised: If h is not deduced at time t by any instantiation of r , then r has no effect on the sum (10), since pooling the empty set with (cid:76) β r returns 0 .

Original: Example.
Revised: Example.

Original: Mei et al.
Revised: Mei et al.

Original: (2020a) give many examples of NDTT programs.
Revised: (2020a) give many examples of NDTT programs.

Original: Here is a simple example to illustrate the use of :- rules.
Revised: Here is a simple example to illustrate the use of :- rules.

Original: eeeeeeeeeeeeeeeee means that Eve posts a message to the company forum, while fffffffffffffffff means that Frank does so.
Revised: eeeeeeeeeeeeeeeee means that Eve posts a message to the company forum, while fffffffffffffffff means that Frank does so.

Original: Once the forum is created by a createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate event, its existence is a fact (called forum ) whose embedding (called (cid:74) forum (cid:75) ) always reﬂects all messages posted so far to the forum.
Revised: Once the forum is created by a createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate event, its existence is a fact (called forum ) whose embedding (called (cid:74) forum (cid:75) ) always reﬂects all messages posted so far to the forum.

Original: Until the forum is destroyed, Eve and Frank can post to it, and the embedding and intensity of such messages depends on the current state of the forum:
Revised: Until the forum is destroyed, Eve and Frank can post to it, and the embeddings and intensities of their messages depend on the current state of the forum:

Original: 1 forum <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate.
Revised: 1 forum <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate.

Original: forum <- destroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroy.
Revised: forum <- destroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroy.

Original: 3 forum <- eeeeeeeeeeeeeeeee.
Revised: 3 forum <- eeeeeeeeeeeeeeeee.

Original: 4 forum <- fffffffffffffffff.
Revised: 4 forum <- fffffffffffffffff.

Original: 5 eeeeeeeeeeeeeeeee :- forum.
Revised: 5 eeeeeeeeeeeeeeeee :- forum.

Original: 6 fffffffffffffffff :- forum.
Revised: 6 fffffffffffffffff :- forum.

Original: The resulting neural architecture is drawn in Figure 1b.
Revised: The resulting neural architecture is drawn in Figure 1b.

Original: If the company grows from 2 to K employees, then the program needs O ( K ) rules and hence O ( K ) parameters, which deﬁne how each employee’s messages affect the forum and vice-versa.
Revised: If the company grows from 2 to K employees, then the program needs O ( K ) rules and hence O ( K ) parameters, which deﬁne how each employee’s messages affect the forum and vice-versa.

Original: Without the :- rules, we would have to list out O ( K 2 ) rules such as e e e eee ee
Revised: Without the :- rules, we would have to list out O ( K 2 ) rules such as e e e eee ee

Original: e ee eee e ee <- f f f fff ff f ff fff f ff and hence would need O
Revised: e ee eee e ee <- f f f fff ff f ff fff f ff and hence would need O

Original: ( K 2 ) parameters, which deﬁne how each employee’s messages affect every other employee’s messages directly; this case is drawn in Figure 1a.
Revised: ( K 2 ) parameters, which deﬁne how each employee’s messages affect every other employee’s messages directly; this case is drawn in Figure 1a.

Original: Appendix B and Figure 4 spell out an enhanced version of this example that makes use of variables, so that all K employees can be governed by a constant ( O (1) ) number of rules.
Revised: Appendix B and Figure 4 spell out an enhanced version of this example that makes use of variables, so that all K employees can be governed by a constant ( O (1) ) number of rules.

Original: Discussion.
Revised: Discussion.

Original: NDTT rules enrich the notion of “inﬂuence matrix” from section 4.
Revised: NDTT rules enrich the notion of “inﬂuence matrix” from section 4.

Revised: Events traditionally inﬂuence the intensities of subsequent events, but NDTT <- rules more generally let them inﬂuence the embeddings of subsequent facts (and hence the intensities of any events among those facts).

Revised: Furthemore, NDTT :- rules let facts inﬂuence the embeddings of contemporaneous facts.

Original: Each <- rule r in effect deﬁnes the ﬁxed sparsity pattern of a large inﬂuence matrix, along with parameters for computing its nonzero entries from context at each attention layer.
Revised: Each <- rule r can be seen as deﬁning the ﬁxed sparsity pattern of a large inﬂuence matrix, along with parameters for computing its nonzero entries from context at each attention layer.

Original: The size of this matrix is determined by the number of ways to instantiate the variables in the rule.
Revised: The size of this matrix is determined by the number of ways to instantiate the variables in the rule.

Original: The entries of the matrix are normalized versions of the attention weights α r .
Revised: The entries of the matrix are normalized versions of the attention weights α r .

Original: The inﬂuences of different <- rules r are combined by equation (9) and are modulated by nonlinearities.
Revised: The inﬂuences of different <- rules r are combined by equation (9) and are modulated by nonlinearities.

Original: Perhaps a more important innovation is that while events traditionally inﬂuence the intensities of subsequent events, NDTT and A-NDTT more generally allow facts to inﬂuence the embeddings of subsequent facts (via <- rules) as well as contemporaneous facts (via :- rules).

Original: Events and possible events are treated as special facts; the intensity of a possible event is computed from its embedding.

Original: Overall, NDTT and A-NDTT are best regarded as representation learning methods.
Revised: Overall, (A-)NDTT models learn representations, much like pretrained language models (Peters et al., 2018; Radford et al., 2019).

Original: They learncontinuously time-varying continuous embeddings of the facts in adiscretely time-varying discrete database, using a neural architecture that is derived from the logical rules that maintain the contents of this deductive database and update the contents in response to events. 
Revised: They learn continuous embeddings of the facts in a discrete database, using a neural architecture that is derived from the symbolic rules that deduce these facts and update them in response to events. The facts change at discrete times but their embeddings change continuously.

Original: We train the model to be predictive of subsequent events.
Revised: We train the model so that the embeddings of possible events predict how likely they are to occur.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multivariate point processes have been widely used in real-world applications, including document stream modeling (He et al., 2015; Du et al., 2015a), learning Granger causality (Xu et al., 2016;
Revised: Multivariate point processes have been widely used in real-world applications, including document stream modeling (He et al., 2015; Du et al., 2015a), learning Granger causality (Xu et al., 2016;

Original: Zhang et al., 2020b; 2021), network analysis (Choi et al., 2015; Etesami et al., 2016), recommendation systems (Du et al., 2015b), and social network analysis (Guo et al., 2015; Lukasik et al., 2016).
Revised: Zhang et al., 2020b; 2021), network analysis (Choi et al., 2015; Etesami et al., 2016), recommendation systems (Du et al., 2015b), and social network analysis (Guo et al., 2015; Lukasik et al., 2016).

Original: Over the recent years, various neural models have been proposed to expand the expressiveness of point processes.
Revised: Over the recent years, various neural models have been proposed to expand the expressiveness of point processes.

Original: They mostly use recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997): in particular Du et al.
Revised: They mostly use recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997): in particular Du et al.

Original: ; Mei & Eisner (2017); Xiao et al.
Revised: ; Mei & Eisner (2017); Xiao et al.

Original: (2017a;b);
Revised: (2017a;b);

Original: Omi et al.
Revised: Omi et al.

Original: Shchur et al.
Revised: Shchur et al.

Original: Mei et al.
Revised: Mei et al.

Original: (2020a)
Revised: (2020a)

Original: ; Boyd et al.
Revised: ; Boyd et al.

Original: Models of this kind enjoy continuous and inﬁnite state spaces, as well as ﬂexible transition functions, thus achieving superior performance on many real-world datasets, compared to classical models such as the Hawkes process (Hawkes, 1971).
Revised: Models of this kind enjoy continuous and inﬁnite state spaces, as well as ﬂexible transition functions, thus achieving superior performance on many real-world datasets, compared to classical models such as the Hawkes process (Hawkes, 1971).

Original: The Transformer Hawkes process (Zuo et al., 2020) and self-attentive Hawkes process (Zhang et al., 2020a) were the ﬁrst papers to adapt generative Transformers (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) to point processes.
Revised: The Transformer Hawkes process (Zuo et al., 2020) and self-attentive Hawkes process (Zhang et al., 2020a) were the ﬁrst papers to adapt generative Transformers (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) to point processes.

Original: The Transformer architecture allows their models to enjoy unboundedly large representations of histories, as well as great parallelism during training (see (cid:172) and (cid:173) in section 1).
Revised: The Transformer architecture allows their models to enjoy unboundedly large representations of histories, as well as great parallelism during training (see (cid:172) and (cid:173) in section 1).

Original: As section 3 discussed, both models—as well as subsequent attention-based models (Enguehard et al., 2020; Sharma et al., 2021)—derive the intensity
Revised: As section 3 discussed, both models—as well as subsequent attention-based models (Enguehard et al., 2020; Sharma et al., 2021)—derive the intensity

Original: λ e ( t ) from (cid:74) f (cid:75) ( s ) where f @ s is the latest actual event before t .
Revised: λ e ( t ) from (cid:74) f (cid:75) ( s ) where f @ s is the latest actual event before t .

Original: (The THP takes λ e ( t ) to be a softplus function of w (cid:62) e [1; t/s ; (cid:74) f
Revised: (The THP takes λ e ( t ) to be a softplus function of w (cid:62) e [1; t/s ; (cid:74) f

Original: (cid:75) ( s )] .
Revised: (cid:75) ( s )] .

Original: The SAHP deﬁnes λ e ( · ) as a function that exponentially decays toward an asymptote, computing the 3 parameters of this function from e and (cid:74) f (cid:75) ( s ) .)
Revised: The SAHP deﬁnes λ e ( · ) as a function that exponentially decays toward an asymptote, computing the 3 parameters of this function from e and (cid:74) f (cid:75) ( s ) .)

Original: In contrast (see (cid:174) in section 1), our model derives λ e ( t ) from (cid:74) e (cid:75) t —the embedding of the possible event e @ t , which is computed using e - and t -speciﬁc attention over all past events.
Revised: In contrast (see (cid:174) in section 1), our model derives λ e ( t ) from (cid:74) e (cid:75) t —the embedding of the possible event e @ t , which is computed using e - and t -speciﬁc attention over all past events.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2021, section 3.1) independently proposed this approach but did not evaluate it experimentally.
Revised: (2021, section 3.1) independently proposed this approach but did not evaluate it experimentally.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: On several synthetic and real-world datasets, we evaluate our model’s held-out log-likelihood, and its success at predicting the time and type of the next event.
Revised: On several synthetic and real-world datasets, we evaluate our model’s held-out log-likelihood, and its success at predicting the time and type of the next event.

Original: We compare with multiple strong competitors.
Revised: We compare with multiple strong competitors.

Original: Experimental details not given in this section can be found in Appendix F.
Revised: Experimental details not given in this section can be found in Appendix F.

Original: We implemented our A-NDTT framework using PyTorch (Paszke et al., 2017) and pyDatalog (Carbonell et al., 2016), borrowing substantially from the public implementation of NDTT (Mei et al., 2020a).
Revised: We implemented our A-NDTT framework using PyTorch (Paszke et al., 2017) and pyDatalog (Carbonell et al., 2016), borrowing substantially from the public implementation of NDTT (Mei et al., 2020a).

Original: We also built a faster, GPU-friendly PyTorch implementation of our more restricted
Revised: We also built a faster, GPU-friendly PyTorch implementation of our more restricted

Original: A-NHP model (see section 7.1 below).
Revised: A-NHP model (see section 7.1 below).

Original: Our code and datasets are available at https://github. com/yangalan123/anhp-andtt .
Revised: Our code and datasets are available at https://github. com/yangalan123/anhp-andtt .

Original: For the competing models, we made use of their published implementations.
Revised: For the competing models, we made use of their published implementations.

Original: 1 References and URLs are provided in Appendix F.2.
Revised: 1 References and URLs are provided in Appendix F.2.

Original: 7. C OMPARISON OF D IFFERENT T RANSFORMER A RCHITECTURES
Revised: 7. C OMPARISON OF D IFFERENT T RANSFORMER A RCHITECTURES

Original: We ﬁrst verify that our continuous-time Transformer is competitive with three state-of-the-art neural event models.
Revised: We ﬁrst verify that our continuous-time Transformer is competitive with three state-of-the-art neural event models.

Original: The four models we compare are
Revised: The four models we compare are

Original: Transformer Hawkes Process (THP) (Zuo et al., 2020) .
Revised: Transformer Hawkes Process (THP) (Zuo et al., 2020) .

Original: See section 6.
Revised: See section 6.

Original: Self-Attentive Hawkes Process (SAHP)
Revised: Self-Attentive Hawkes Process (SAHP)

Original: (Zhang et al., 2020a).
Revised: (Zhang et al., 2020a).

Original: See section 6.
Revised: See section 6.

Original: Neural Hawkes Process (NHP) (Mei & Eisner, 2017) .
Revised: Neural Hawkes Process (NHP) (Mei & Eisner, 2017) .

Original: This is not an attention-based model.
Revised: This is not an attention-based model.

Original: At any time t , NHP uses a continuous-time LSTM to summarize the events over [0 , t ) into a multidimensional state vector, and conditions the intensities λ e ( t ) of all event types on that state.
Revised: At any time t , NHP uses a continuous-time LSTM to summarize the events over [0 , t ) into a multidimensional state vector, and conditions the intensities λ e ( t ) of all event types on that state.

Original: Attentive Neural Hawkes Process (A-NHP)
Revised: Attentive Neural Hawkes Process (A-NHP)

Original: This is our unstructured generative model from section 3.
Revised: This is our unstructured generative model from section 3.

Original: Since this model does not use selective attention, we speed up the intensity computations by deﬁning them in terms of a single coarse event type, as described in Appendix A.
Revised: Since this model does not use selective attention, we speed up the intensity computations by deﬁning them in terms of a single coarse event type, as described in Appendix A.

Original: Thus, each event intensity λ e ( t ) is computed by attention over all previous events, where the attention weights are independent of e .
Revised: Thus, each event intensity λ e ( t ) is computed by attention over all previous events, where the attention weights are independent of e .

Original: This parameter-sharing mechanism resembles the NHP, except that we now use a
Revised: This parameter-sharing mechanism resembles the NHP, except that we now use a

Original: Transformer in place of an LSTM.
Revised: Transformer in place of an LSTM.

Original: In a pilot experiment
Revised: In a pilot experiment

Original: [Figure]
Revised: [Figure]

Original: we drew sequences from randomly initialized models of all 4 types (details in Appendix F.1.1), and then ﬁt all 4 models on each of these 4 synthetic datasets.
Revised: we drew sequences from randomly initialized models of all 4 types (details in Appendix F.1.1), and then ﬁt all 4 models on each of these 4 synthetic datasets.

Original: We ﬁnd that NHP,
Revised: We ﬁnd that NHP,

Original: SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself); see Figure 5 in Appendix F.1.1 for details.
Revised: SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself); see Figure 5 in Appendix F.1.1 for results.

Original: Thus, A-NHP is still a satisfactory choice even when it is misspeciﬁed.
Revised: Thus, A-NHP is still a satisfactory choice even when it is misspeciﬁed.

Original: This result is reassuring because A-NHP has less capacity in some ways (the circuit depth of a Transformer is ﬁxed, whereas the circuit depth of an LSTM grows with the length of the sequence) and excess capacity in other ways (the Transformer has unbounded memory whereas the LSTM has ﬁnite-dimensional memory)
Revised: This result is reassuring because A-NHP has less capacity in some ways (the circuit depth of a Transformer is ﬁxed [Figure] whereas the circuit depth of an LSTM grows with the length of the sequence) and excess capacity in other ways (the Transformer has unbounded memory whereas the LSTM has ﬁnite-dimensional memory).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We then ﬁt all 4 models to the following two benchmark real-world datasets.
Revised: We then ﬁt all 4 models to the following two benchmark real-world datasets.

Original: MIMIC-II (Lee et al., 2011) .
Revised: MIMIC-II (Lee et al., 2011) .

Original: This dataset is a collection of de-identiﬁed clinical visit records of
Revised: This dataset is a collection of de-identiﬁed clinical visit records of

Original: Intensive Care Unit patients for 7 years.
Revised: Intensive Care Unit patients for 7 years.

Original: Each patient has a sequence of hospital visit events, and each event records its time stamp and disease diagnosis.
Revised: Each patient has a sequence of hospital visit events, and each event records its time stamp and disease diagnosis.

Original: StackOverﬂow (Leskovec & Krevl, 2014) .
Revised: StackOverﬂow (Leskovec & Krevl, 2014) .

Original: This dataset represents two years of user awards on a question-answering website: each user received a sequence of badges (of 22 different types).
Revised: This dataset represents two years of user awards on a question-answering website: each user received a sequence of badges (of 22 different types).

Original: On MIMIC-II data (Figure 2a), our A-NHP is always a co-winner on each of these tasks; but the other co-winner (NHP or THP) varies across tasks.
Revised: On MIMIC-II data (Figure 2a), our A-NHP is always a co-winner on each of these tasks; but the other co-winner (NHP or THP) varies across tasks.

Original: On StackOverﬂow data (Figure 2b), our A-NHP is clearly a winner on 2 out of 3 tasks and is tied with NHP on the third.
Revised: On StackOverﬂow data (Figure 2b), our A-NHP is clearly a winner on 2 out of 3 tasks and is tied with NHP on the third.

Original: Compared to NHP, A-NHP also enjoys a computational advantage, as discussed in sections 1 and 2.
Revised: Compared to NHP, A-NHP also enjoys a computational advantage, as discussed in sections 1 and 2.

Original: Empirically, training an A-NHP only took a fraction of the time that was needed to train an NHP, when sequences are reasonably long.
Revised: Empirically, training an A-NHP only took a fraction of the time that was needed to train an NHP, when sequences are reasonably long.

Original: Details can be found in Table 2 of Appendix F.3.
Revised: Details can be found in Table 2 of Appendix F.3.

Original: 7. A-NDTT VS . NDTT
Revised: 7. A-NDTT VS . NDTT

Original: Now we turn to the structured modeling approach presented in section 5.
Revised: Now we turn to the structured modeling approach presented in section 5.

Original: We compare A-NDTT with
Revised: We compare A-NDTT with

Original: NDTT on the RoboCup dataset and IPTV dataset proposed by Mei et al.
Revised: NDTT on the RoboCup dataset and IPTV dataset proposed by Mei et al.

Original: (2020a).
Revised: (2020a).

Original: In both cases, we used the NDTT program written by Mei et al.
Revised: In both cases, we used the NDTT program written by Mei et al.

Original: (2020a).
Revised: (2020a).

Original: The rules are unchanged; the only difference is that our A-NDTT has the new continuous-time Transformer in lieu of the LSTM architecture.
Revised: The rules are unchanged; the only difference is that our A-NDTT has the new continuous-time Transformer in lieu of the LSTM architecture.

Original: We also evaluated the unstructured NHP and A-NHP models on these datasets.
Revised: We also evaluated the unstructured NHP and A-NHP models on these datasets.

Original: RoboCup (Chen & Mooney, 2008) .
Revised: RoboCup (Chen & Mooney, 2008) .

Original: This dataset logs the actions (e.g., kickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkick , passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass ) of robot soccer players in the RoboCup Finals 2001–2004.
Revised: This dataset logs the actions (e.g., kickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkick , passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass ) of robot soccer players in the RoboCup Finals 2001–2004.

Original: The ball is frequently transferred between players (by passing or stealing), and this dynamically changes the set of possible event types (e.g., only the ball possessor can kick or pass).
Revised: The ball is frequently transferred between players (by passing or stealing), and this dynamically changes the set of possible event types (e.g., only the ball possessor can kick or pass).

Original: There are K = 528 event types over all time, but only about 20 of them are possible at any given time.
Revised: There are K = 528 event types over all time, but only about 20 of them are possible at any given time.

Original: For each preﬁx of each held-out event sequence, we used minimum Bayes risk to predict the next event’s time, and to predict its participant(s) given its time and action type.
Revised: For each preﬁx of each held-out event sequence, we used minimum Bayes risk to predict the next event’s time, and to predict its participant(s) given its time and action type.

Original: IPTV (Xu et al., 2018) .
Revised: IPTV (Xu et al., 2018) .

Original: This dataset contains records of 1000 users watching 49 TV programs over the ﬁrst 11 months of 2012.
Revised: This dataset contains records of 1000 users watching 49 TV programs over the ﬁrst 11 months of 2012.

Original: Each event has the form watchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatch(U,P) .
Revised: Each event has the form watchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatchwatch(U,P) .

Original: Given each preﬁx of the test event sequence, we attempted to predict the next test event’s time t , and to predict its program P given its actual time t and user U .
Revised: Given each preﬁx of the test event sequence, we attempted to predict the next test event’s time t , and to predict its program P given its actual time t and user U .

Original: [Figure]
Revised: [Figure]

Original: The Robocup results are shown in Figure 3a.
Revised: The Robocup results are shown in Figure 3a.

Original: As in section 7.1, we ﬁnd that A-NHP performs better than NHP on all the evaluation metrics; on log-likelihood and event type prediction, A-NHP is signiﬁcantly better (paired permutation test, p < 0 . 05 ).
Revised: As in section 7.1, we ﬁnd that A-NHP performs better than NHP on all the evaluation metrics; on log-likelihood and event type prediction, A-NHP is signiﬁcantly better (paired permutation test, p < 0 . 05 ).

Original: We now inject domain knowledge into both the
Revised: We now inject domain knowledge into both the

Original: LSTM and Transformer approaches, by deriving architectures based on the RoboCup NDTT program (which speciﬁes, for example, that only the ball possessor can kick or pass).
Revised: LSTM and Transformer approaches, by deriving architectures based on the RoboCup NDTT program (which speciﬁes, for example, that only the ball possessor can kick or pass).

Original: The resulting models—
Revised: The resulting models—

Original: NDTT and A-NDTT—are substantial and signiﬁcant improvements, considerably reducing both the log-likelihood and the very high error rate on event type prediction.
Revised: NDTT and A-NDTT—are substantial and signiﬁcant improvements, considerably reducing both the log-likelihood and the very high error rate on event type prediction.

Original: NDTT and A-NDTT are not signiﬁcantly different from each other: since NDTT already knows which past events might be relevant, perhaps it is not sorely in need of the Transformer’s ability to scan an unbounded history for relevant events.
Revised: NDTT and A-NDTT are not signiﬁcantly different from each other: since NDTT already knows which past events might be relevant, perhaps it is not sorely in need of the Transformer’s ability to scan an unbounded history for relevant events.

Original: 3 Appendix F.5 includes more results of A-NDTT vs. NDTT broken down by action types.
Revised: 3 Appendix F.5 includes more results of A-NDTT vs. NDTT broken down by action types.

Original: Additionally, while A-NDTT does not improve the overall accuracy for this particular NDTT program and dataset, it does achieve overall comparable accuracy with a simpler and shallower architecture ( (cid:173) – (cid:174) in section 1).
Revised: Additionally, while A-NDTT does not improve the overall accuracy for this particular NDTT program and dataset, it does achieve overall comparable accuracy with a simpler and shallower architecture ( (cid:173) – (cid:174) in section 1).

Original: Like other Transformers, the A-NDTT architecture could be trained on a GPU with parallelism, as outlined in Appendix F.4 (future work).
Revised: Like other Transformers, the A-NDTT architecture could be trained on a GPU with parallelism, as outlined in Appendix F.4 (future work).

Original: The IPTV results are shown in Figure 3b.
Revised: The IPTV results are shown in Figure 3b.

Original: In this case, the log-likelihood of NHP can be substantially and signiﬁcantly improved either by using rules (as for Robocup) or by using attention, or both.
Revised: In this case, the log-likelihood of NHP can be substantially and signiﬁcantly improved either by using rules (as for Robocup) or by using attention, or both.

Original: The error rate on predicting the next event type is again very high for NHP, and is substantially and signiﬁcantly reduced by using rules—although not as much under the A-NDTT architecture as under the original NDTT architecture.
Revised: The error rate on predicting the next event type is again very high for NHP, and is substantially and signiﬁcantly reduced by using rules—although not as much under the A-NDTT architecture as under the original NDTT architecture.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We showed how to generalize the Transformer architecture to sequences of discrete events in continuous time.
Revised: We showed how to generalize the Transformer architecture to sequences of discrete events in continuous time.

Original: Our architecture builds up rich embeddings of actual and possible events at any time t , from lower-level representations of those events and their contexts.
Revised: Our architecture builds up rich embeddings of actual and possible events at any time t , from lower-level representations of those events and their contexts.

Original: We usually train the model so that the embedding of a possible event predicts its intensity, yielding a ﬂexible generative model that supports parallel computation of log-likelihood.
Revised: We usually train the model so that the embedding of a possible event predicts its intensity, yielding a ﬂexible generative model that supports parallel computation of log-likelihood.

Original: We showed in section 7.1 that it outperforms other
Revised: We showed in section 7.1 that it outperforms other

Original: Transformer-based models on multiple real-world datasets and also beats or ties them on multiple synthetic datasets.
Revised: Transformer-based models on multiple real-world datasets and also beats or ties them on multiple synthetic datasets.

Original: We also showed how to integrate this architecture with NDTT, a neural-symbolic framework that automatically derives neural models from logic programs.
Revised: We also showed how to integrate this architecture with NDTT, a neural-symbolic framework that automatically derives neural models from logic programs.

Original: Our attention-based modiﬁcation of NDTT has shown competitive performance, despite having a simpler and shallower architecture.
Revised: Our attention-based modiﬁcation of NDTT has shown competitive performance, despite having a simpler and shallower architecture.

Original: Our codeand datasets are available at https://github.com/yangalan123/anhp-andtt .
Revised: Our code

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was supported in part by the National Science Foundation under Grant No. 1718846.
Revised: This work was supported in part by the National Science Foundation under Grant No. 1718846.

Original: We thank Bloomberg for a Data Science Ph.D. Fellowship to the second author.
Revised: We thank Bloomberg for a Data Science Ph.D. Fellowship to the second author.

Original: We thank Minjie Xu for the suggestion of developing a Transformer version of NDTT.
Revised: We thank Minjie Xu for the suggestion of developing a Transformer version of NDTT.

Original: We thank the anonymous ICLRreviewers for discussion and for pointing out additional related work
Revised: We thank the anonymous ICLRreviewers for discussion and for pointing out additional related work

Original: Appendices A D ISCUSSION OFTHE A RCHITECTURE
Revised: Appendices A D ISCUSSION OF A RCHITECTURAL D ETAILS

Original: Simpliﬁcation.
Revised: Simpliﬁcation.

Original: Equation (1) is a simpliﬁcation of the original Transformer architecture (Vaswani et al., 2017).
Revised: Equation (1) is a simpliﬁcation of the original Transformer architecture (Vaswani et al., 2017).

Original: In the original architecture, (cid:74) e (cid:75) ( (cid:96) )
Revised: In the original architecture, (cid:74) e (cid:75) ( (cid:96) )

Original: ( t ) would be obtained as LayerNorm ( x + FFN ( (cid:96) )
Revised: ( t ) would be obtained as LayerNorm ( x + FFN ( (cid:96) )

Original: ( x )) , where x is the LayerNorm transformation (Ba et al., 2016) of the right-hand side of equation (1), and the nonlinear transform FFN ( (cid:96) ) is computed by a learned two-layer feed-forward network.
Revised: ( x )) , where x is the LayerNorm transformation (Ba et al., 2016) of the right-hand side of equation (1), and the nonlinear transform FFN ( (cid:96) ) is computed by a learned two-layer feed-forward network.

Original: In our preliminary experiments, we found that the LayerNorm and FFN steps did not help, so for simplicity and speed, we omitted them from equation (1) and from the remaining experiments.
Revised: In our preliminary experiments, we found that the LayerNorm and FFN steps did not help, so for simplicity and speed, we omitted them from equation (1) and from the remaining experiments.

Original: However, it is possible that they might help on other domains or with larger training datasets, so our code supports them via command-line arguments.
Revised: However, it is possible that they might help on other domains or with larger training datasets, so our code supports them via command-line arguments.

Original: Graceful degradation.
Revised: Graceful degradation.

Original: Another change to equation (1) (and equation (8)) is that  we included an additional summand of 1 in the denominator that normalizes the attention weights.
Revised: Another change to equation (1) (and equation (8)) is that when normalizing the attention weights, we included an additional summand of 1 in the denominator.

Original: 4 We do this so that when  H ( e @ t ) contains only events that are rather irrelevant to e @ t , it behaves roughly as if it were the empty set.
Revised: 4 We do this so that when the history H ( e @ t ) is “rather irrelevant” to e @ t , the architecture behaves roughly as if H ( e @ t ) were the empty set.

Original: More precisely, if the attention weights α on the events in H ( e @ t ) are (cid:28) 1 (perhaps because they are far in the past), then the argument to tanh in equation (1) is close to 0 (its value when H  ( e @ t ) = ∅ ), and thus (cid:74) e (cid:75) ( (cid:96) )
Revised: In equation (1), this means that (cid:74) e (cid:75) ( (cid:96) ) ( t ) will then be close to (cid:74) e (cid:75) ( (cid:96) − 1) ( t ) . Similarly, equation (7) will not be much inﬂuenced by rule r if rule r ﬁnds only events H r ( e @ t ) that it considers to be “rather irrelevant” to e @ t .

Original:  ( t ) will be close to (cid:74) e (cid:75) ( (cid:96) − 1) (  t )  .
Revised: A “rather irrelevant” history is one for which the unnormalized attention weights are small in toto , so that the denominator is dominated by the 1 summand. This may occur, for example, if events in the distant past tend to have small attention weights, and the history consist only of old events (and not too many of them). When the history is rather irrelevant, the argument to tanh in equation (1) and the summand e ( (cid:96) ) r ( t ) in equation (7) are close to 0 ; when H ( e @ t ) = ∅ , they are exactly 0 .

Original: Direct access to time embeddings.
Revised: Direct access to time embeddings.

Original: Another difference from Vaswani et al.
Revised: Another difference from Vaswani et al.

Original: (2017) is that in equation (3), we chose to concatenate (cid:74) t (cid:75) to the rest of the embedding rather than add it (cf.
Revised: (2017)—perhaps not an important one—is that in equation (3), we chose to concatenate (cid:74) t (cid:75) to the rest of the embedding rather than add it (cf.

Original: Kitaev & Klein, 2018; He et al., 2020).
Revised: Kitaev & Klein, 2018; He et al., 2020).

Original: Furthermore we did so at every layer and not just layer 0.
Revised: Furthermore we did so at every layer and not just layer 0.

Original: The intuition here is that the extraction of good key and query vectors at each layer may beneﬁt from “direct access” to (cid:74) t (cid:75) .
Revised: The intuition here is that the extraction of good key and query vectors at each layer may beneﬁt from “direct access” to (cid:74) t (cid:75) .

Original: For example, this should make it easy to learn keys and queries such that the attention weight is highest when s ≈ t − ∆ (since for every ∆ ∈ R , there exists a sparse linear operator that transforms (cid:74) t (cid:75) (cid:55)→ (cid:74) t − ∆ (cid:75) ).
Revised: For example, this should make it easy to learn keys and queries such that the attention weight is highest when s ≈ t − ∆ (since for every ∆ ∈ R , there exists a sparse linear operator that transforms (cid:74) t (cid:75) (cid:55)→ (cid:74) [Equation] (cid:75) [Equation]

Original: Dynamic range of wavelengths for time embeddings.
Revised: Range of wavelengths for time embeddings.

Original: Our time embedding (cid:74) t (cid:75) in (4) uses dimensions that are sinusoidal in t  with wavelengths forming a geometric progression from 2 πm to 2 π (5 M ) .
Revised: Our time embedding (cid:74) t (cid:75) in (4) uses dimensions that are sinusoidal in t , with wavelengths forming a geometric progression from 2 πm to 2 π (5 M ) .

Original: This generalizes the  scheme of Vaswani et al.
Revised: Setting m = 1 , M = 2000 would recover the standard scheme of Vaswani et al.

Original: (2017) (previously used in continuous time by Zuo et al.
Revised: (2017) (previously used in continuous time by Zuo et al.

Original: (2020)), who speciﬁcally take m = 1 , M = 2000 .

Original: We  set m and M from data so that we are robust across datasets with different time scales.
Revised: We instead set m and M from data so that we are robust to datasets of different time scales.

Original: Part of the intuition behind these sinusoidal embeddings is that nearby times can be distinguished by different values in their short-wavelength dimensions, whereas the long-wavelength dimensions make it easy to inspect and compare faraway times, since those dimensions are nearly linear on t ∈ [0 , M ] .
Revised: Part of the intuition behind using sinusoidal embeddings is that nearby times can be distinguished by different values in their short-wavelength dimensions, whereas the long-wavelength dimensions make it easy to inspect and compare faraway times, since those dimensions are nearly linear on t ∈ [0 , M ] .

Original: We therefore take m to be the shortest gap between any two events in the same history,
Revised: We therefore take m to be the shortest gap between any two events in the same history,

Original: [Equation]
Revised: [Equation]

Original: (11) as computed over training data, and take M greater than all T in training data (where each observed sequence is observed over an interval [0 , T ] ).
Revised: (11) as computed over training data, and take M greater than all T in training data (where each observed sequence is observed over an interval [0 , T ] ).

Original: On sequences of words, this recovers the values m =and M = 2000 that Vaswani et al. (2017) chose to model text documents.
Revised: If we were modeling sequences of words as in Brown et al. (2020), our procedure would indeed recover the values m = 1 and M = 2000 that they used to model text documents.

Revised: Multiplying or dividing all t values in the dataset by 1000 (e.g., switching between second and millisecond units) would have no effect on the time embeddings, as it would scale m and M in the same way.

Original: Coarse event embeddings for speed.
Revised: Coarse event embeddings for speed.

Original: As noted at the end of section 3, the intensity model equation (5) involves a full embedding of each e @ t .
Revised: As noted at the end of section 3, the intensity model equation (5) involves a full embedding of each e @ t .

Original: This may be expressive, but it is also expensive.
Revised: This may be expressive, but it is also expensive.

Original: The attention weight vectors α (1) , . .
Revised: The attention weight vectors α (1) , . .

Original: , α ( L ) used to compute this embedding must be computed from scratch for each e and t .
Revised: , α ( L ) used to compute this embedding must be computed from scratch for each e and t .

Original: Why is this necessary?
Revised: Why is this necessary?

Original: Like other neural sequence models—both RNN-based and Transformer-based—we derive the probability that the next sequence element is e from an inner product of the form w (cid:62) e [1; (cid:74) H ( e @ t ) (cid:75) ] , where in our equation (5), the role of the history embedding (cid:74) H ( e @ t ) (cid:75) is played by (cid:74) e (cid:75) L ( t ) .
Revised: Like other neural sequence models—both RNN-based and Transformer-based—we derive the probability that the next sequence element is e from an inner product of the form w (cid:62) e [1; (cid:74) H ( e @ t ) (cid:75) ] , where in our equation (5), the role of the history embedding (cid:74) H ( e @ t ) (cid:75) is played by (cid:74) e (cid:75) L ( t ) .

Original: However, for many previous models, the history embedding does not depend on e , so it can be computed once at each time t and reused across all e .
Revised: However, for many previous models, the history embedding does not depend on e , so it can be computed once at each time t and reused across all e .

Original: • In neural language models, typically all previous events are taken to be relevant.
Revised: • In neural language models, typically all previous events are taken to be relevant.

Original: (cid:74) H ( e @ t ) (cid:75) can then be deﬁned as an RNN encoding of the sequence all past events (Mikolov et al., 2010), or alternatively a Transformer embedding of the single most recent past event (which depends on the entire sequence of past events).
Revised: (cid:74) H ( e @ t ) (cid:75) can then be deﬁned as an RNN encoding of the sequence all past events (Mikolov et al., 2010), or alternatively a Transformer embedding of the single most recent past event (which depends on the entire sequence of past events).

Original: This does not depend on e .
Revised: This does not depend on e .

Original: • When modeling irregularly spaced events, t is not necessarily an integer, and the past events in H ( e @ t ) do not necessarily take place at 1 , 2 , . . .
Revised: • When modeling irregularly spaced events, t is not necessarily an integer, and the past events in H ( e @ t ) do not necessarily take place at 1 , 2 , . . .

Original: , t − 1 .
Revised: , t − 1 .

Original: Thus, the encoding (cid:74) H ( e @ t ) (cid:75) must somehow be improved to also consider the elapsed time t − t i since the most recent past event (Du et al., 2016; Mei & Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020a).
Revised: Thus, the encoding (cid:74) H ( e @ t ) (cid:75) must somehow be improved to also consider the elapsed time t − t i since the most recent past event (Du et al., 2016; Mei & Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020a).

Original: So now (cid:74) H ( e @ t ) (cid:75) must look at t , but it is still independent of e .
Revised: So now (cid:74) H ( e @ t ) (cid:75) must look at t , but it is still independent of e .

Original: • In contrast, in sections 4–5, we will allow the more general case where (cid:74) H
Revised: • In contrast, in sections 4–5, we will allow the more general case where (cid:74) H

Original: ( e @ t ) (cid:75) varies with e as well, since NDTT rules determine which past events should be attended to by e .
Revised: ( e @ t ) (cid:75) varies with e as well, since NDTT rules determine which past events should be attended to by e .

Original: The original NDTT paper essentially deﬁned (cid:74) H ( e @ t ) (cid:75) as the state at time t of an e -speciﬁc continuous-time LSTM, which is updated by just the events that are relevant to e .
Revised: The original NDTT paper essentially deﬁned (cid:74) H ( e @ t ) (cid:75) as the state at time t of an e -speciﬁc continuous-time LSTM, which is updated by just the events that are relevant to e .

Original: In our attention-based approach, we instead deﬁne it to be (cid:74) e
Revised: In our attention-based approach, we instead deﬁne it to be (cid:74) e

Original: (cid:75) L ( t ) , yielding equation (5).
Revised: (cid:75) L ( t ) , yielding equation (5).

Original: To reduce this computational cost we can associate each event type e with a coarse event type ¯ e that is guaranteed to have the same set of relevant past events and replace (cid:74) e (cid:75) L ( t ) with (cid:74) ¯ e (cid:75) L ( t ) in equation (5).
Revised: To reduce this computational cost, we can associate each event type e with a coarse event type ¯ e that is guaranteed to have the same set of relevant past events, and replace (cid:74) e (cid:75) L ( t ) with (cid:74) ¯ e (cid:75) L ( t ) in equation (5).

Original: (However, equation (5) still uses the ﬁne-grained w e .)
Revised: (However, equation (5) still uses the ﬁne-grained w e .)

Original: Now to compute λ e ( t ) , we only have to embed ¯ e @ t , which is a speedup if many of the possible event types e are associated with the same ¯ e .
Revised: Now to compute λ e ( t ) , we only have to embed ¯ e @ t , which is a speedup if many of the possible event types e are associated with the same ¯ e .

Original: In the case where we do not use selective attention, we can get away with using only a single coarse event type for the whole model—saving a runtime factor of |E| as in the cheaper approach.
Revised: In the case where we do not use selective attention, we can get away with using only a single coarse event type for the whole model—saving a runtime factor of |E| as in the cheaper approach.

Original: Note that the history H still uses ﬁne-grained embeddings, so if e @ t actually occurs, we must then compute (cid:74) e (cid:75) 0 ( t ) , . . .
Revised: Note that the history H still uses ﬁne-grained embeddings, so if e @ t actually occurs, we must then compute (cid:74) e (cid:75) 0 ( t ) , . . .

Original: , (cid:74) e (cid:75) L ( t ) .
Revised: , (cid:74) e (cid:75) L ( t ) .

Original: Concatenation vs. summation.
Revised: Concatenation vs. summation.

Original: Equation (7) uses summation to combine the outputs (8) of different attention heads r .
Revised: Equation (7) uses summation to combine the outputs (8) of different attention heads r .

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017) instead combined such outputs by projecting their concatenation, but that becomes trickier in our setting: different events e would need to concatenate different numbers of attention heads r (for just the rules r that can take the form e <- · · · ), resulting in projection matrices of different dimensionalities.
Revised: (2017) instead combined such outputs by projecting their concatenation, but that becomes trickier in our setting: different events e would need to concatenate different numbers of attention heads r (for just the rules r that can take the form e <- · · · ), resulting in projection matrices of different dimensionalities.

Original: Especially when NDTT rules can contain variables (section 5 below), it is not immediately obvious how one should construct these matrices or share parameters among them.
Revised: Especially when NDTT rules can contain variables (section 5 below), it is not immediately obvious how one should construct these matrices or share parameters among them.

Original: These presentational problems vanish with our simpler summation approach.
Revised: These presentational problems vanish with our simpler summation approach.

Original: Our approach loses no expressive power: projecting a concatenation of e ( (cid:96) )
Revised: Our approach loses no expressive power: projecting a concatenation of e ( (cid:96) )

Original: r ( t ) values, as Vaswani et al.
Revised: r ( t ) values, as Vaswani et al.

Original: would suggest, is equivalent to summing up an r -speciﬁc projection of e ( (cid:96) )
Revised: would suggest, is equivalent to summing up an r -speciﬁc projection of e ( (cid:96) )

Original: r ( t ) for each r , as we do, where our projection of e ( (cid:96) )
Revised: r ( t ) for each r , as we do, where our projection of e ( (cid:96) )

Original: r ( t ) has implicitly been incorporated into the projection (3a) that produces v ( (cid:96) ) r .
Revised: r ( t ) has implicitly been incorporated into the projection (3a) that produces v ( (cid:96) ) r .

Original: That is, if we can learn V ( (cid:96) )
Revised: That is, if we can learn V ( (cid:96) )

Original: r = V in equation (3a), then we can also learn V ( (cid:96) )
Revised: r = V in equation (3a), then we can also learn V ( (cid:96) )

Original: r = PV , where P is the desired projection matrix for rule r .
Revised: r = PV , where P is the desired projection matrix for rule r .

Original: To make our method fully equivalent to Vaswani et al.
Revised: To make our method fully equivalent to Vaswani et al.

Original: ’s, we would have to explicitly parameterize V ( (cid:96) )
Revised: ’s, we would have to explicitly parameterize V ( (cid:96) )

Original: r as a matrix product of the form PV , forcing it to be low-rank.
Revised: r as a matrix product of the form PV , forcing it to be low-rank.

Original: B NDTT E XAMPLE W ITH V ARIABLES
Revised: B NDTT E XAMPLE W ITH V ARIABLES

Original: The company message forum program in section 5 had only 2 users and 1 forum.
Revised: The company message forum program in section 5 had only 2 users and 1 forum.

Original: However, if the company employs many persons P and has a forum for each team T , NDTT rules can use capitalized variables to deﬁne the whole system concisely, using only O (1) rules and O (1) parameters.
Revised: However, if the company employs many persons P and has a forum for each team T , NDTT rules can use capitalized variables to deﬁne the whole system concisely, using only O (1) rules and O (1) parameters.

Original: Here the possible facts and events have structured names like messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,sales,joke) , which denotes an event in which employee eve posts a joke to the sales team’s forum.
Revised: Here the possible facts and events have structured names like messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,sales,joke) , which denotes an event in which employee eve posts a joke to the sales team’s forum.

Original: 7 forum(T) <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate(T).
Revised: 7 forum(T) <- createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate(T).

Original: 8 forum(T) <- messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(P,T,C).
Revised: 8 forum(T) <- messagemessagemessagemessage message message messagemessage message messagemessage message message messagemessagemessage message(P,T,C).

Original: 9 messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(P,T,C) :- empl(P), forum(T), content(C).
Revised: 9 messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(P,T,C) :- empl(P), forum(T), content(C).

Original: forum(T) <- destroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroydestroy(T).
Revised: forum(T) <- destroydestroydestroydestroy destroy destroy destroydestroy destroy destroydestroy destroy destroy destroydestroydestroy destroy(T).

Original: This generalizes the previous program, allowing multiple forums and saying that any employee (not just Eve and Frank) can post any type of message to any forum, affecting the embedding of that forum.
Revised: This generalizes the previous program, allowing multiple forums and saying that any employee (not just Eve and Frank) can post any type of message to any forum, affecting the embedding of that forum.

Original: We could modify rule 9 by adding an additional condition member(P,T) , so that employees can only post to forums of which they are members.
Revised: We could modify rule 9 by adding an additional condition member(P,T) , so that employees can only post to forums of which they are members.

Original: Membership could be established and tracked by
Revised: Membership could be established and tracked by

Original: 11 joinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoin(P,T) :- empl(P), forum(T).
Revised: 11 joinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoin(P,T) :- empl(P), forum(T).

Original: 12 member(P,T) <- joinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoin(P,T).
Revised: 12 member(P,T) <- joinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoinjoin(P,T).

Original: 13 empl(P) <- message messagemessage message message messagemessagemessagemessagemessagemessagemessage message message messagemessage message(P2,T,C), member(P,T).
Revised: 13 empl( P) <- messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(P2,T,C), member(P,T).

Original: Rules 8 and 13 ensure that a message to a forum affects the subsequent embedding of that forum and also the subsequent embeddings of all employees who were members of that forum when the message was sent.
Revised: Rules 8 and 13 ensure that a message to a forum affects the subsequent embedding of that forum and also the subsequent embeddings of all employees who were members of that forum when the message was sent.

Original: This may affect which employees join which forums in future and what they post to the forums, as drawn in Figure 4 in the appendices.
Revised: This may affect which employees join which forums in future, and what they post to the forums, as drawn in Figure 4 in the appendices.

Original: For further examples, see the full presentation of NDTT in Mei et al.
Revised: For further examples, see the full presentation of NDTT in Mei et al.

Original: (2020a).
Revised: (2020a).

Original: How are variables treated in the computation of embeddings?
Revised: How are variables treated in the computation of embeddings?

Original: In equations (7)–(8), r refers to a rule with variables.
Revised: In equations (7)–(8), r refers to a rule with variables.

Original: However, e refers to a speciﬁc fact, without variables.
Revised: However, e refers to a speciﬁc fact, without variables.

Original: An instantiation of r is a copy of r in which each variable has been consistently replaced by an actual value.
Revised: An instantiation of r is a copy of r in which each variable has been consistently replaced by an actual value.

Original: In our modiﬁed version of equation (8), the summations range over all values of ( f, g 1 , . .
Revised: In our modiﬁed version of equation (8), the summations range over all values of ( f, g 1 , . .

Original: . , g n ) @
Revised: . , g n ) @

Original: s such that e <- f , g 1 , ..., g n is an instantiation of r that added e at time s (i.e., an instantiation of r such that f occurred at time s and g 1 , . .
Revised: s such that e <- f , g 1 , ..., g n is an instantiation of r that added e at time s (i.e., an instantiation of r such that f occurred at time s and g 1 , . .

Original: . , g n were all true at time s ).
Revised: . , g n were all true at time s ).

Original: Thus, the attentional competition may consider ( f, g 1 , . .
Revised: Thus, the attentional competition may consider ( f, g 1 , . .

Original: . , g n ) @
Revised: . , g n ) @

Original: s values that are derived from many different instantiations of r .
Revised: s values that are derived from many different instantiations of r .

Original: Their attentional weights α ( (cid:96) )
Revised: Their attentional weights α ( (cid:96) )

Original: r are all obtained using the shared parameters associated with rule r . 

Revised: r are all obtained using the shared parameters associated with rule r .

Original: The summation in equation (7) ranges only over rules r with at least one instantiation that adds e @ t , so it skips rules that are irrelevant to e .
Revised: The summation in equation (7) ranges only over rules r with at least one instantiation that adds e @ t , so it skips rules that are irrelevant to e .

Original: C P ARAMETER D IMENSIONALITY S PECIFICATION FOR A-NDTT
Revised: C P ARAMETER D IMENSIONALITY S PECIFICATION FOR A-NDTT

Original: In this section we discuss the dimensionality of the fact embeddings (cid:74) h (cid:75) ( t ) in section 5.
Revised: In this section we discuss the dimensionality of the fact embeddings (cid:74) h (cid:75) ( t ) in section 5.

Original: As in the original NDTT paper Mei et al.
Revised: As in the original NDTT paper Mei et al.

Original: (2020a), the type of a fact in the database is given by its functor ( forum , member , createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate , etc.).
Revised: (2020a), the type of a fact in the database is given by its functor ( forum , member , createcreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreatecreate , etc.).

Original: All facts of the same type have embedding vectors of the same dimensionality, and these dimensionalities are declared by the NDTT program.
Revised: All facts of the same type have embedding vectors of the same dimensionality, and these dimensionalities are declared by the NDTT program.

Original: This is enough to determine the dimensions of the parameter matrices associated with the deduction rules (Mei et al., 2020a).
Revised: This is enough to determine the dimensions of the parameter matrices associated with the deduction rules (Mei et al., 2020a).

Original: How about the add rules, however?
Revised: How about the add rules, however?

Original: The form of equation (9) implies that the value vectors v ( (cid:96) )
Revised: The form of equation (9) implies that the value vectors v ( (cid:96) )

Original: r for add rule r have the same dimensionality as the embedding of the head of r .
Revised: r for add rule r have the same dimensionality as the embedding of the head of r .

Original: The key and query vectors for rule r (as used in equation (2)) can share this dimensionality by default, although we are free to override this and specify a different dimensionality for them.
Revised: The key and query vectors for rule r (as used in equation (2)) can share this dimensionality by default, although we are free to override this and specify a different dimensionality for them.

Original: The foregoing choices determine the dimensions of the parameter matrices V ( (cid:96) )
Revised: The foregoing choices determine the dimensions of the parameter matrices V ( (cid:96) )

Original: r , K ( (cid:96) ) r , Q ( (cid:96) )
Revised: r , K ( (cid:96) ) r , Q ( (cid:96) )

Original: r associated with rule r .
Revised: r associated with rule r .

Original: D L IKELIHOOD C OMPUTATION D ETAILS
Revised: D L IKELIHOOD C OMPUTATION D ETAILS

Original: In this section we discuss the log-likelihood formulas in section 3.
Revised: In this section we discuss the log-likelihood formulas in section 3.

Original: Derivations of the log-likelihood formula (6) can be found in previous work (e.g., Hawkes, 1971;
Revised: Derivations of the log-likelihood formula (6) can be found in previous work (e.g., Hawkes, 1971;

Original: Liniger, 2009; Mei & Eisner, 2017).
Revised: Liniger, 2009; Mei & Eisner, 2017).

Original: Derivations of this formula appear in previous work (e.g.,
Revised: Derivations of this formula appear in previous work (e.g.,

Original: Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).
Revised: Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017).

Original: Intuitively, when training to increase the loglikelihood (6), each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .
Revised: Intuitively, when training to increase the loglikelihood (6), each log λ e i ( t i ) is increased to explain why the observed event e i happened at time t i , while (cid:82) T t =0 (cid:80) Ee =1 λ e ( t )d t is decreased to explain why no event of any possible type e ∈ { 1 , . . .

Original: , E } ever happened at other times.
Revised: , E } ever happened at other times.

Original: Note that there is no log under the integral in equation (6).
Revised: Note that there is no log under the integral in equation (6).

Original: Why?
Revised: Why?

Original: The probability that there was not an event of any type in the inﬁnitesimally wide interval [ t, t + d t ) is 1 − λ ( t )d t , whose log is − λ ( t )d t .
Revised: The probability that there was not an event of any type in the inﬁnitesimally wide interval [ t, t + d t ) is 1 − λ ( t )d t , whose log is − λ ( t )d t .

Original: The integral term in equation (6) is computed using the Monte Carlo approximation given by Mei & Eisner (2017, Algorithm 1), which samples times t .
Revised: The integral term in equation (6) is computed using the Monte Carlo approximation given by Mei & Eisner (2017, Algorithm 1), which samples times t .

Original: This yields an unbiased stochastic gradient.
Revised: This yields an unbiased stochastic gradient.

Original: For the number of Monte Carlo samples, we follow the practice of Mei & Eisner (2017): namely, at training time, we match the number of samples to the number of observed events at training time, a reasonable and fast choice, but to estimate log-likelihood when tuning hyperparameters or reporting ﬁnal results, we take 10 times as many samples.
Revised: For the number of Monte Carlo samples, we follow the practice of Mei & Eisner (2017): namely, at training time, we match the number of samples to the number of observed events at training time, a reasonable and fast choice, but to estimate log-likelihood when tuning hyperparameters or reporting ﬁnal results, we take 10 times as many samples.

Original: The small remaining variance in this procedure is shown in our error bars, as explained in footnote 6.
Revised: The small remaining variance in this procedure is shown in our error bars, as explained in footnote 6.

Original: At each sampled time t , the Monte Carlo method still requires a summation over all events to obtain λ ( t ) .
Revised: At each sampled time t , the Monte Carlo method still requires a summation over all events to obtain λ ( t ) .

Original: This summation can be expensive when there are many event types.
Revised: This summation can be expensive when there are many event types.

Original: This is not a serious problem for our standalone A-NHP implementation since it can leverage GPU parallelism.
Revised: This is not a serious problem for our standalone A-NHP implementation since it can leverage GPU parallelism.

Original: But for the general A-NDTT implementation, it is hard to parallelize the λ k ( t ) computation over k and t .
Revised: But for the general A-NDTT implementation, it is hard to parallelize the λ k ( t ) computation over k and t .

Original: In that case, we use the downsampling trick detailed in Appendix D of Mei et al.
Revised: In that case, we use the downsampling trick detailed in Appendix D of Mei et al.

Original: (2020a).
Revised: (2020a).

Original: An alternative would be to replace maximum-likelihood estimation with noise-contrastive estimation, which is quite effective at training NHP and NDTT models (Mei et al., 2020b).
Revised: An alternative would be to replace maximum-likelihood estimation with noise-contrastive estimation, which is quite effective at training NHP and NDTT models (Mei et al., 2020b).

Original: E H OW TO P REDICT E VENTS
Revised: E H OW TO P REDICT E VENTS

Original: It is possible to sample event sequences exactly from an A-NHP or A-NDTT model, using the thinning algorithm that is traditionally used for autoregressive point processes (Lewis & Shedler, 1979; Liniger, 2009).
Revised: It is possible to sample event sequences exactly from an A-NHP or A-NDTT model, using the thinning algorithm that is traditionally used for autoregressive point processes (Lewis & Shedler, 1979; Liniger, 2009).

Original: In general, to apply the thinning algorithm to sample the next event at time ≥ t 0 , it is necessary to have an upper bound on { λ e ( t ) : t ∈ [ t 0 , ∞ ) } for each event type t .
Revised: In general, to apply the thinning algorithm to sample the next event at time ≥ t 0 , it is necessary to have an upper bound on { λ e ( t ) : t ∈ [ t 0 , ∞ ) } for each event type t .

Original: An explicit construction for the NHP (or NDTT) model was given by Mei & Eisner (2017, Appendix B.3).
Revised: An explicit construction for the NHP (or NDTT) model was given by Mei & Eisner (2017, Appendix B.3).

Original: For
Revised: For

Original: A-NHP and A-NDTT, observe that λ e ( t ) is a continuous real-valued function of (cid:74) t (cid:75)
Revised: A-NHP and A-NDTT, observe that λ e ( t ) is a continuous real-valued function of (cid:74) t (cid:75)

Original: (the particular function depends on e and the history of events at times < t 0 ).
Revised: (the particular function depends on e and the history of events at times < t 0 ).

Original: Since (cid:74) t (cid:75) falls in the compact set  [ − 1 , 1] d (thanks to the sinusoidal embedding (4)), it follows that λ e ( t ) is indeed bounded.
Revised: Since (cid:74) t (cid:75) falls in the compact set competition using its own parameters. One use of this mechanism would be to allocate multiple attention heads to a single rule. [Table] [ − 1 , 1] d (thanks to the sinusoidal embedding (4)), it follows that λ e ( t ) is indeed bounded.

Original: Actual numerical bounds can be computed using interval arithmetic.
Revised: Actual numerical bounds can be computed using interval arithmetic.

Original: That is, we can apply our continuous function not to a particular value of (cid:74) t (cid:75) but to all of [ − 1 , 1] d , where for any elementary continuous function f : R → R , we have deﬁned f ([ x lo , x hi ]) to return some bounded interval that contains f ( x ) for all x ∈ [ x lo , x hi ] .
Revised: That is, we can apply our continuous function not to a particular value of (cid:74) t (cid:75) but to all of [ − 1 , 1] d , where for any elementary continuous function f : R → R , we have deﬁned f ([ x lo , x hi ]) to return some bounded interval that contains f ( x ) for all x ∈ [ x lo , x hi ] .

Original: The result will be a bounded interval that contains λ e ( t ) for all t ∈ [ t 0 , ∞ ) .[Table]
Revised: The result will be a bounded interval that contains λ e ( t ) for all t ∈ [ t 0 , ∞ ) .

Original: Section 7 includes a task-based evaluation where we try to predict the time and type of just the next event.
Revised: Section 7 includes a task-based evaluation where we try to predict the time and type of just the next event.

Original: More precisely, for each event in each held-out sequence, we attempt to predict its time given only the preceding events, as well as its type given both its true time and the preceding events.
Revised: More precisely, for each event in each held-out sequence, we attempt to predict its time given only the preceding events, as well as its type given both its true time and the preceding events.

Original: We evaluate the time prediction with average L 2 loss (yielding a root-mean-squared error, or RMSE ) and evaluate the argument prediction with average 0-1 loss (yielding an error rate ).
Revised: We evaluate the time prediction with average L 2 loss (yielding a root-mean-squared error, or RMSE ) and evaluate the argument prediction with average 0-1 loss (yielding an error rate ).

Original: Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle to predict the time and type with lowest expected loss.
Revised: Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle to predict the time and type with lowest expected loss.

Original: For completeness, we repeat the general recipe in this section.
Revised: For completeness, we repeat the general recipe in this section.

Original: For the i th event, its time t i has density p
Revised: For the i th event, its time t i has density p

Original: i ( t ) = λ ( t ) exp [Equation] − [Equation]
Revised: i ( t ) = λ ( t ) exp( − (cid:82) t t i − 1 λ ( t (cid:48) )

Original: t t i − 1 λ

Original: [Equation]

Original: t [Equation] d t 
Revised: d t (cid:48) ) .

Original: [Equation]

Original: We choose (cid:82) ∞
Revised: We choose (cid:82) ∞

Original: t i − 1 tp i ( t )d t as the time prediction because it has the lowest expected L 2 loss.
Revised: t i − 1 tp i ( t )d t as the time prediction because it has the lowest expected L 2 loss.

Original: The integral can be estimated using i.i.d.
Revised: The integral can be estimated using i.i.d.

Original: samples of t i drawn from p i ( t ) by the thinning algorithm.
Revised: samples of t i drawn from p i ( t ) by the thinning algorithm.

Original: Given the next event time t i , we choose the most probable type arg max e λ e ( t i ) as the type prediction because it minimizes expected 0-1 loss.
Revised: Given the next event time t i , we choose the most probable type arg max e λ e ( t i ) as the type prediction because it minimizes expected 0-1 loss.

Original: In some circumstances, one might also like to predict the most probable type out of a restricted set E (cid:48) (cid:40) { 1 , . . .
Revised: In some circumstances, one might also like to predict the most probable type out of a restricted set E (cid:48) (cid:40) { 1 , . . .

Original: , E } .
Revised: , E } .

Original: This allows one to answer questions like “If we know that some event of the form messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,T) happened at time t i , then what was the forum T , given all past events?”
Revised: This allows one to answer questions like “If we know that some event of the form messagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessagemessage(eve,T) happened at time t i , then what was the forum T , given all past events?”

Original: The answer will simply be arg max e ∈E (cid:48)
Revised: The answer will simply be arg max e ∈E (cid:48)

Original: λ
Revised: λ

Original: e ( t i ) .
Revised: e ( t i ) .

Original: F E XPERIMENTAL D ETAILS F. D ATASET D ETAILS
Revised: F E XPERIMENTAL D ETAILS F. D ATASET D ETAILS

Original: Table 1 shows statistics about each dataset that we use in this paper.
Revised: Table 1 shows statistics about each dataset that we use in this paper.

Original: F.1. P ILOT E XPERIMENTS ON S IMULATED D ATA
Revised: F.1. P ILOT E XPERIMENTS ON S IMULATED D ATA

Original: In this experiment, we draw data from randomly initialized NHP, A-NHP, SAHP, and THP.
Revised: In this experiment, we draw data from randomly initialized NHP, A-NHP, SAHP, and THP.

Original: For all of them, we take the number of event types to be E = 10 .
Revised: For all of them, we take the number of event types to be E = 10 .

Original: For NHP, the dimensions of event embeddings and hidden states are all 32 ; for A-NHP, the number of layers ( L in our paper) is 2 , and the dimensions of time embeddings and event embeddings are 32 ; for SAHP, the number of layers is 4 , and the dimension of hidden states is 32 ; for THP, the number of layer is 7 , and the dimension of hidden states is 32 .
Revised: For NHP, the dimensions of event embeddings and hidden states are all 32 ; for A-NHP, the number of layers ( L in our paper) is 2 , and the dimensions of time embeddings and event embeddings are 32 ; for SAHP, the number of layers is 4 , and the dimension of hidden states is 32 ; for THP, the number of layer is 7 , and the dimension of hidden states is 32 .

Original: For each model, we draw 800 , 100 , and 100 sequences for training, validation and testing, respectively.
Revised: For each model, we draw 800 , 100 , and 100 sequences for training, validation and testing, respectively.

Original: For each sequence, the sequence length I is drawn from Uniform (49 , 99) .
Revised: For each sequence, the sequence length I is drawn from Uniform (49 , 99) .

Original: We take the maximum observation time T to be t I + 1 , one time step after the ﬁnal event.
Revised: We take the maximum observation time T to be t I + 1 , one time step after the ﬁnal event.

Original: We ﬁt all 4 models on each of these 4 synthetic datasets.
Revised: We ﬁt all 4 models on each of these 4 synthetic datasets.

Original: The results are graphed in Figure 5 and show that NHP, SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself).
Revised: The results are graphed in Figure 5 and show that NHP, SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself).

Original: Notably, THP ﬁts the time intervals poorly when it is misspeciﬁed, perhaps because its family of intensity functions (section 6) is not a good match for real data: THP requires that the intensity of e between events changes more slowly later in the event sequence, and that if it increases over time, it approaches linear growth rather than an asymptote.F.1. O THER D ATA D ETAILS
Revised: Notably, THP ﬁts the time intervals poorly when it is misspeciﬁed, perhaps because its family of intensity functions (section 6) is not a good match for real data: THP requires that the intensity of e between events changes more slowly later in the event sequence, and that if it increases over time, it approaches linear growth rather than an asymptote.

Original: For MIMIC-II and StackOverﬂow, we used the version processed by Du et al. (2016); more details (e.g., about processing) can be found in their paper.
Revised: Log-likelihood per event (the training objective) of the whole test event sequence [Figure] Log-likelihood per event of the event times only

Original: Log [Figure]
Revised: [Figure]

Original: likelihood per event (the training objective) of the whole test event sequence

Original: [Figure]

Original: Log-likelihood per event of the event times only
Revised: Log-likelihood per event of the event types only

Original: [Figure]
Revised: [Figure]

Original: Log-likelihood per event of the event types
Revised: F.1. O THER D ATA D ETAILS

Original: [Figure]
Revised: For MIMIC-II and StackOverﬂow, we used the version processed by Du et al. (2016); more details (e.g., about processing) can be found in their paper.

Original: For RoboCup, we used the version processed by Chen & Mooney (2008); please refer to their paper for more details (e.g., data description, processing method, etc)
Revised: For RoboCup, we used the version processed by Chen & Mooney (2008); please refer to their paper for more details (e.g., data description, processing method, etc)

Original: F. I MPLEMENTATION D ETAILS
Revised: F. I MPLEMENTATION D ETAILS

Original: For NHP, our implementation is based on the public Github repositories at https://github.com/
Revised: For NHP, our implementation is based on the public Github repositories at https://github.com/

Original: HMEIatJHU/neurawkes (Mei & Eisner (2017), with MIT License) and https://github.com/
Revised: HMEIatJHU/neurawkes (Mei & Eisner (2017), with MIT License) and https://github.com/

Original: HMEIatJHU/neural-hawkes-particle-smoothing (Mei et al.
Revised: HMEIatJHU/neural-hawkes-particle-smoothing (Mei et al.

Original: (2019), with BSD 3-Clause “New” or “Revised” License).
Revised: (2019), with BSD 3-Clause “New” or “Revised” License).

Original: We made a considerable amount of modiﬁcations to their code (e.g., model, thinning algorithm), in order to migrate it to PyTorch 1.7.
Revised: We made a considerable amount of modiﬁcations to their code (e.g., model, thinning algorithm), in order to migrate it to PyTorch 1.7.

Original: We built the standalone GPU implementation of A-NHP upon our NHP code.
Revised: We built the standalone GPU implementation of A-NHP upon our NHP code.

Original: For NDTT, we used the public Github repository at https://github.com/HMEIatJHU/ neural-datalog-through-time (Mei et al.
Revised: For NDTT, we used the public Github repository at https://github.com/HMEIatJHU/ neural-datalog-through-time (Mei et al.

Original: (2020a), with MIT License).
Revised: (2020a), with MIT License).

Original: We built A-NDTT upon NDTT.
Revised: We built A-NDTT upon NDTT.

Original: For THP, we used the public Github repository at https://github.com/SimiaoZuo/
Revised: For THP, we used the public Github repository at https://github.com/SimiaoZuo/

Original: Transformer-Hawkes-Process (Zuo et al.
Revised: Transformer-Hawkes-Process (Zuo et al.

Original: (2020), no license speciﬁed).
Revised: (2020), no license speciﬁed).

Original: For SAHP, we used the public Github repository at https://github.com/QiangAIResearcher/ sahp_repo (Zhang et al.
Revised: For SAHP, we used the public Github repository at https://github.com/QiangAIResearcher/ sahp_repo (Zhang et al.

Original: (2020a), no license speciﬁed).
Revised: (2020a), no license speciﬁed).

Revised: [Table]

Original: F. T RAINING D ETAILS
Revised: F. T RAINING D ETAILS

Original: For each model in section 7, we had to specify various dimensionalities.
Revised: For each model in section 7, we had to specify various dimensionalities.

Original: For simplicity, we used a single hyperparameter D and took all vectors to be in R D .
Revised: For simplicity, we used a single hyperparameter D and took all vectors to be in R D .

Original: This includes the state vectors of NHP, the fact embeddings of NDTT and A-NDTT, and the query, key, and value vectors for the models with attention mechanisms (THP, SAHP, A-NHP, and A-NDTT).
Revised: This includes the state vectors of NHP, the fact embeddings of NDTT and A-NDTT, and the query, key, and value vectors for the models with attention mechanisms (THP, SAHP, A-NHP, and A-NDTT).

Original: For the models with attention mechanisms, we also had to choose the number of layers L .
Revised: For the models with attention mechanisms, we also had to choose the number of layers L .

Original: We tuned these hyperparameters for each combination of model, dataset, and training size (e.g., each bar in Figures 2, 3a and 5), always choosing the combination of D and L that achieved[Table] the best performance on the dev set.
Revised: We tuned these hyperparameters for each combination of model, dataset, and training size (e.g., each bar in Figures 2, 3a and 5), always choosing the combination of D and L that achieved the best performance on the dev set.

Original: Our search spaces were D ∈ { 4 , 8 , 16 , 32 , 64 , 128 } and
Revised: Our search spaces were D ∈ { 4 , 8 , 16 , 32 , 64 , 128 } and

Original: L ∈ { 1 , 2 , 3 , 4 , 5 , 6 } .
Revised: L ∈ { 1 , 2 , 3 , 4 , 5 , 6 } .

Original: In practice, the optimal D for a model was usually 32 or 64; the optimal L was usually 1, 2, or 3.
Revised: In practice, the optimal D for a model was usually 32 or 64; the optimal L was usually 1, 2, or 3.

Original: To train the parameters for a given model, we used the Adam algorithm (Kingma & Ba, 2015) with its default settings.
Revised: To train the parameters for a given model, we used the Adam algorithm (Kingma & Ba, 2015) with its default settings.

Original: We performed early stopping based on log-likelihood on the held-out dev set.
Revised: We performed early stopping based on log-likelihood on the held-out dev set.

Original: For the experiments in section 7.1, we used the standalone PyTorch implementations for NHP and
Revised: For the experiments in section 7.1, we used the standalone PyTorch implementations for NHP and

Original: A-NHP, which are GPU-friendly.
Revised: A-NHP, which are GPU-friendly.

Original: We trained each model on an NVIDIA K80 GPU.
Revised: We trained each model on an NVIDIA K80 GPU.

Original: Table 2 shows their training time per sequence on each dataset.
Revised: Table 2 shows their training time per sequence on each dataset.

Original: For section 7.2, we run our NDTT and A-NDTT models only on CPUs.
Revised: For section 7.2, we run our NDTT and A-NDTT models only on CPUs.

Original: This follows Mei et al.
Revised: This follows Mei et al.

Original: (2020a), who did not ﬁnd an efﬁcient method to leverage GPU parallelism for training NDTT models.
Revised: (2020a), who did not ﬁnd an efﬁcient method to leverage GPU parallelism for training NDTT models.

Original: The machines we used for NDTT and A-NDTT are 6-core Haswell architectures.
Revised: The machines we used for NDTT and A-NDTT are 6-core Haswell architectures.

Original: On RoboCup, the training time of NDTT and A-NDTT was 62 and 149 seconds per sequence, respectively.
Revised: On RoboCup, the training time of NDTT and A-NDTT was 62 and 149 seconds per sequence, respectively.

Original: See
Revised: See

Original: Appendix F.4 for future work on improving the latter time by exploiting GPU parallelism.
Revised: Appendix F.4 for future work on improving the latter time by exploiting GPU parallelism.

Original: For the NHP and A-NHP models in section 7.2, we ran the specialized code for these models on CPU as well, rather than on GPU as in section 7.1, since the RoboCup sequences were too long to ﬁt in the memory of our K80 GPU.
Revised: For the NHP and A-NHP models in section 7.2, we ran the specialized code for these models on CPU as well, rather than on GPU as in section 7.1, since the RoboCup sequences were too long to ﬁt in the memory of our K80 GPU.

Original: The training time was 66 and 95 seconds per sequence for NHP and
Revised: The training time was 66 and 95 seconds per sequence for NHP and

Original: A-NHP, respectively.
Revised: A-NHP, respectively.

Original: F. T RAINING P ARALLELISM
Revised: F. T RAINING P ARALLELISM

Original: We point out that in the future, GPU parallelism could be exploited through the following procedure, given a GPU with enough memory to handle long training sequences.
Revised: We point out that in the future, GPU parallelism could be exploited through the following procedure, given a GPU with enough memory to handle long training sequences.

Original: (The layers can be partitioned across multiple GPUs if needed.)
Revised: (The layers can be partitioned across multiple GPUs if needed.)

Original: For each training minibatch, the ﬁrst step is to play each event sequence e 1@ t 1 , e 2@ t 2 , . . .
Revised: For each training minibatch, the ﬁrst step is to play each event sequence e 1@ t 1 , e 2@ t 2 , . . .

Original: , e I @
Revised: , e I @

Original: t I forward to determine the contents of the database on each interval (0 , t 1 ] , ( t 1 , t 2 ] , . . .
Revised: t I forward to determine the contents of the database on each interval (0 , t 1 ] , ( t 1 , t 2 ] , . . .

Original: , ( t i − 1 , t I ] , ( t I , T ] .
Revised: , ( t i − 1 , t I ] , ( t I , T ] .

Original: This step runs on CPU, and computes only the boolean facts (“Datalog through time”) without their embeddings (“neural Datalog through time”).
Revised: This step runs on CPU, and computes only the boolean facts (“Datalog through time”) without their embeddings (“neural Datalog through time”).

Original: Let F be the set of facts that ever appeared in the database during this minibatch and let R be the set of rules that were ever used to deduce or add them (section 5).
Revised: Let F be the set of facts that ever appeared in the database during this minibatch and let R be the set of rules that were ever used to deduce or add them (section 5).

Original: Furthermore, let T be the set of times consisting of { t 1 , . . .
Revised: Furthermore, let T be the set of times consisting of { t 1 , . . .

Original: , t I } together with the times t that are sampled for the Monte Carlo integral (Appendix D).
Revised: , t I } together with the times t that are sampled for the Monte Carlo integral (Appendix D).

Original: A computation graph of size O ( |R|· I ) can now be constructed, as illustrated in Figure 1b, to compute the embeddings (cid:74) h (cid:75) ( t ) of all facts h ∈ F at all times t ∈ T .
Revised: A computation graph of size O ( |R|· I ) can now be constructed, as illustrated in Figure 1b, to compute the embeddings (cid:74) h (cid:75) ( t ) of all facts h ∈ F at all times t ∈ T .

Original: The layer- (cid:96) embeddings at time t ∈ T depend on the layer- ( (cid:96) − 1) embeddings at times t i ≤ t , according to the add rules in R .
Revised: The layer- (cid:96) embeddings at time t ∈ T depend on the layer- ( (cid:96) − 1) embeddings at times t i ≤ t , according to the add rules in R .

Original: The layer- (cid:96) embedding of a fact that is deduced at time t also depends on the layer- (cid:96) embeddings at time t of the facts that it is deduced from, according to the deduction rules in R ; this further increases the depth of the computation graph.
Revised: The layer- (cid:96) embedding of a fact that is deduced at time t also depends on the layer- (cid:96) embeddings at time t of the facts that it is deduced from, according to the deduction rules in R ; this further increases the depth of the computation graph.

Original: For a given fact h ∈ F , (cid:74) h (cid:75) ( (cid:96) )
Revised: For a given fact h ∈ F , (cid:74) h (cid:75) ( (cid:96) )

Original: ( t ) can be computed in parallel for all event sequences and all times t ∈ T (even times t when h is not true, although those embeddings will not be used).
Revised: ( t ) can be computed in parallel for all event sequences and all times t ∈ T (even times t when h is not true, although those embeddings will not be used).

Original: Multiple facts that are governed by the same NDTT rule r ∈ R can also be handled in parallel, since they use the same r -speciﬁc parameters.
Revised: Multiple facts that are governed by the same NDTT rule r ∈ R can also be handled in parallel, since they use the same r -speciﬁc parameters.

Original: Thus, a GPU can be effective for this phase.
Revised: Thus, a GPU can be effective for this phase.

Original: The computation of h ( (cid:96) ) r ( t ) in equation (9) must take care to limit its attention to just those earlier times when an event occurred that added h via rule r , and the computation of [ h ] ( (cid:96) )
Revised: The computation of h ( (cid:96) ) r ( t ) in equation (9) must take care to limit its attention to just those earlier times when an event occurred that added h via rule r , and the computation of [ h ] ( (cid:96) )

Original: ( t ) in equation (9) must take care to consider only rules r that in fact deduce h at time t because their conditions are true at time t .
Revised: ( t ) in equation (9) must take care to consider only rules r that in fact deduce h at time t because their conditions are true at time t .

Original: This masks unwanted parts of the computation, rendering parts of the GPU idle.
Revised: This masks unwanted parts of the computation, rendering parts of the GPU idle.

Original: GPU parallelism will still be worthwhile if a substantial fraction of the computation remains unmasked—which is true for relatively homogenous settings where most facts in F hold true for a large portion of the observed interval [0 , T ) , even if their embeddings ﬂuctuate.
Revised: GPU parallelism will still be worthwhile if a substantial fraction of the computation remains unmasked—which is true for relatively homogenous settings where most facts in F hold true for a large portion of the observed interval [0 , T ) , even if their embeddings ﬂuctuate.

Original: F. M ORE R ESULTS
Revised: F. M ORE R ESULTS

Original: The performance of A-NDTT and NDTT is not always comparable for speciﬁc action types, as shown in Figure 6.
Revised: The performance of A-NDTT and NDTT is not always comparable for speciﬁc action types, as shown in Figure 6.

Original: In terms of data ﬁtting (left ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while NDTT is better at the others.
Revised: In terms of data ﬁtting (left ﬁgure), A-NDTT is signiﬁcantly better at the [Table] events while NDTT is better at the others.

Original: For time prediction (middle ﬁgure), A-NDTT is signiﬁcantly better at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal , kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff , and passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass events, but the differences for the other action types are not signiﬁcant.
Revised: For time prediction (middle ﬁgure), A-NDTT is signiﬁcantly better at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal , kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff , and passpasspasspasspasspasspasspasspasspasspasspasspasspasspasspasspass events, but the differences for the other action types are not signiﬁcant.

Original: For action participant prediction (right ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while there is no difference for the others; both do perfectly well at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal and  events such that their dots overlap at the origin.
Revised: For action participant prediction (right ﬁgure), A-NDTT is signiﬁcantly better at the kickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoffkickoff events while there is no difference for the others; both do perfectly well at the goalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoalgoal and kickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkickkick events such that their dots overlap at the origin.

Original: [Figure]
Revised: [Figure]

Original: In Figure 
Revised: In Figure 7, we show that Figure 6 does not change qualitatively when re-run with different random seeds.

Original: [Figure]
Revised: [Figure]

Original: we show that Figure 6 does not change qualitatively when re-run with different random seeds.
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]