Original: HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences
Revised: HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In this paper, we tackle the important yet under-investigated problem of makinglong-horizon prediction of event sequences.
Revised: In this paper, we tackle the important yet under-investigated problem of making long-horizon prediction of event sequences.

Original: Existing state-of-the-art models donot perform well at this task due to their autoregressive structure.
Revised: Existing state-of-the-art models do not perform well at this task due to their autoregressive structure.

Original: We propose
Revised: We propose

Original: HYPRO, a hybridly normalized probabilistic model that naturally fits this task:its first part is an autoregressive base model that learns to propose predictions;its second part is an energy function that learns to reweight the proposals suchthat more realistic predictions end up with higher probabilities.
Revised: HYPRO, a hybridly normalized probabilistic model that naturally fits this task: its first part is an autoregressive base model that learns to propose predictions; its second part is an energy function that learns to reweight the proposals such that more realistic predictions end up with higher probabilities.

Original: We also proposeefficient training and inference algorithms for this model.
Revised: We also propose efficient training and inference algorithms for this model.

Original: Experiments on tworeal-world datasets demonstrate that our proposed HYPRO model can significantlyoutperform previous models at making long-horizon predictions of future events.
Revised: Experiments on multiple real-world datasets demonstrate that our proposed HYPRO model can significantly outperform previous models at making long-horizon predictions of future events.

Original: We also conduct a range of ablation studies to investigate the effectiveness of eachcomponent of our proposed methods
Revised: We also conduct a range of ablation studies to investigate the effectiveness of each component of our proposed methods.

Original:  Introduction
Revised:  Introduction

Original: Long-horizon prediction of event sequences is essential in various real-world applied domains:
Revised: Long-horizon prediction of event sequences is essential in various real-world applied domains:

Original: • Healthcare.
Revised: • Healthcare.

Original: Given a patient’s symptoms and treatments so far, we would be interested in predicting
Revised: Given a patient’s symptoms and treatments so far, we would be interested in predicting

Original: their future health conditions over the next several months , including their prognosis and treatment.• Commercial.
Revised: their future health conditions over the next several months , including their prognosis and treatment. • Commercial.

Original: Given an online consumer’s previous purchases and reviews, we may be interested inpredicting what they would buy over the next several weeks and plan our advertisementaccordingly.• Urban planning.
Revised: Given an online consumer’s previous purchases and reviews, we may be interested in predicting what they would buy over the next several weeks and plan our advertisement

Original:  Having monitored the traffic flow of a town for the past few days, we’d like topredict its future traffic over the next few hours , which would be useful for congestion management.•
Revised: accordingly • Urban planning. Having monitored the traffic flow of a town for the past few days, we’d like to predict its future traffic over the next few hours , which would be useful for congestion management.

Original:  Similar scenarios arise in computer systems , finance , dialogue , music , etc.
Revised: • Similar scenarios arise in computer systems , finance , dialogue , music , etc.

Original: Though being important this task has been under-investigated the previous work in this researcharea has been mostly focused on the prediction of the next single event (e.g., its time and type).
Revised: Though being important, this task has been under-investigated: the previous work in this research area has been mostly focused on the prediction of the next single event (e.g., its time and type).

Original: In this paper, we show that previous state-of-the-art models suffer at making long-horizon predictions,i.e., predicting the series of future events over a given time interval .
Revised: In this paper, we show that previous state-of-the-art models suffer at making long-horizon predictions, i.e., predicting the series of future events over a given time interval .

Original: That is because those modelsare all autoregressive : predicting each future event is conditioned on all the previously predictedevents; an error can not be corrected after it is made and any error will be cascaded through all thesubsequent predictions.
Revised: That is because those models are all autoregressive : predicting each future event is conditioned on all the previously predicted events; an error can not be corrected after it is made and any error will be cascaded through all the subsequent predictions.

Original: Problems of the same kind also exist in natural language processing taskssuch as generation and machine translation (Ranzato et al., 2016; Goyal, 2021).
Revised: Problems of the same kind also exist in natural language processing tasks such as generation and machine translation (Ranzato et al., 2016; Goyal, 2021).

Original: In this paper, we propose a novel modeling framework that learns to make long-horizon prediction ofevent sequences.
Revised: In this paper, we propose a novel modeling framework that learns to make long-horizon prediction of event sequences.

Original: Our framework consists of three technical innovations:
Revised: Our main technical contributions include:

Original: • A new model.
Revised: • A new model.

Original: Our main contribution is HYPRO, a hybridly normalized neural probabilistic modelthat combines an autoregressive base model with an energy function: the  base model learns topropose plausible predictions; the energy function learns to reweight theproposals.
Revised: The key component of our framework is HYPRO, a hybridly normalized neural probabilistic model that combines an autoregressive base model with an energy function: the 36th Conference on Neural Information Processing Systems (NeurIPS 2022). base model learns to propose plausible predictions the energy function learns to reweight the

Original:  Although theproposals are generated autoregressively, the energy function reads each entire completedsequence(i
Revised: proposals Although the proposals are generated autoregressively, the energy function reads each entire completed

Original: e., true past events together with predicted future events) and learns to assign higher weights tothose which appear more realistic as a whole.•
Revised: sequence (i e. true past events together with predicted future events) and learns to assign higher weights to those which appear more realistic as a whole

Revised: Hybrid models have already demonstrated effective at natural language processing such as detecting machine-generated text (Bakhtin et al., 2019) and improving coherency in text generation (Deng et al., 2020).

Revised: We are the first to develop a model of this kind for time-stamped event sequences

Revised: Our model can use any autoregressive event model as its base model, and we choose the state-of-the-art continuous-time Transformer architecture (Yang et al., 2022) as its energy function

Original:  A family of new training objectives.
Revised: • A family of new training objectives.

Original: Our second contribution is a family of training objectives thatcan estimate the parameters of our proposed model with low computational cost.
Revised: Our second contribution is a family of training objectives that can estimate the parameters of our proposed model with low computational cost.

Original: Our trainingmethods are based on the principle of noise-contrastive estimation since the log-likelihood of ourHYPRO model involves an intractable normalizing constant (due to using energy functions).•
Revised: Our training methods are based on the principle of noise-contrastive estimation since the log-likelihood of our HYPRO model involves an intractable normalizing constant (due to using energy functions).

Original:  A new efficient inference method.
Revised: • A new efficient inference method.

Original: Another contribution is a normalized importance samplingalgorithm, which can efficiently draw the predictions of future events over a given time intervalfrom a trained HYPRO model.
Revised: Another contribution is a normalized importance sampling algorithm, which can efficiently draw the predictions of future events over a given time interval from a trained HYPRO model.

Original:  Technical Background2.1 Formulation: Generative Modeling of Event Sequences
Revised:  Technical Background 2.1 Formulation: Generative Modeling of Event Sequences

Original: We are given a fixed time interval [0 , T ] over which an event sequence is observed.
Revised: We are given a fixed time interval [0 , T ] over which an event sequence is observed.

Original: Suppose thereare I events in the sequence at times 0 < t 1 < . . .
Revised: Suppose there are I events in the sequence at times 0 < t 1 < . . .

Original: < t I ≤ T .
Revised: < t I ≤ T .

Original: We denote the sequence asx [0 ,T ] =
Revised: We denote the sequence as x [0 ,T ] =

Original: ( t 1 , k 1 ) , . . .
Revised: ( t 1 , k 1 ) , . . .

Original: , ( t I , k I ) where each k i ∈ { 1 , . . .
Revised: , ( t I , k I ) where each k i ∈ { 1 , . . .

Original: , K } is a discrete event type.
Revised: , K } is a discrete event type.

Original: Generative models of event sequences are temporal point processes.
Revised: Generative models of event sequences are temporal point processes.

Original: They are autoregressive :events are generated from left to right; the probability of ( t i , k i ) depends on the history of eventsx [0 ,t i )
Revised: They are autoregressive : events are generated from left to right; the probability of ( t i , k i ) depends on the history of events x [0 ,t i )

Original: =
Revised: =

Original: ( t 1 , k 1 ) , . . .
Revised: ( t 1 , k 1 ) , . . .

Original: , ( t i − 1 , k i − 1 ) that were drawn at times <
Revised: , ( t i − 1 , k i − 1 ) that were drawn at times <

Original: t i .
Revised: t i .

Original: They are locally normalized : if weuse p k ( t | x [0 ,t ) ) to denote the probability that an event of type k occurs over the infinitesimal interval[ t, t + dt ) , then the probability that nothing occurs will be 1 − (cid:80)
Revised: They are locally normalized : if we use p k ( t | x [0 ,t ) ) to denote the probability that an event of type k occurs over the infinitesimal interval [ t, t + dt ) , then the probability that nothing occurs will be 1 − (cid:80)

Original: Kk =1 p k ( t | x [0 ,t ) ) .
Revised: Kk =1 p k ( t | x [0 ,t ) ) .

Original: Specifically,temporal point processes define functions λ
Revised: Specifically, temporal point processes define functions λ

Original: k that determine a finite intensity λ k ( t | x [0 ,t ) ) ≥ 0for each event type k at each time t > 0 such that p k ( t | x [0 ,t ) )
Revised: k that determine a finite intensity λ k ( t | x [0 ,t ) ) ≥for each event type k at each time t > 0 such that p k ( t | x [0 ,t ) )

Original: =
Revised: =

Original: λ k ( t | x [0 ,t ) ) dt .
Revised: λ k ( t | x [0 ,t ) ) dt .

Original: Then thelog-likelihood of a temporal point process given the entire event sequence x [0 ,T ] is 
Revised: Then the log-likelihood of a temporal point process given the entire event sequence x [0 ,T ] is I (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Popular examples of temporal point processes include Poisson processes (Daley & Vere-Jones, 2007)as well as Hawkes processes (Hawkes, 1971) and their modern neural versions (Du et al., 2016; Mei& Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020; Yang et al., 2022).
Revised: Popular examples of temporal point processes include Poisson processes (Daley & Vere-Jones, 2007) as well as Hawkes processes (Hawkes, 1971) and their modern neural versions (Du et al., 2016; Mei & Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020; Yang et al., 2022).

Original: 2.2 Task and Challenge: Long-Horizon Prediction and Cascading Errors
Revised: 2.2 Task and Challenge: Long-Horizon Prediction and Cascading Errors

Original: We are interested in predicting the future events over an extended time interval ( T, T ′ ] .
Revised: We are interested in predicting the future events over an extended time interval ( T, T ′ ] .

Original: We call thistask long-horizon prediction as the boundary T ′ is so large that (with a high probability) manyevents will happen over ( T, T ′ ] .
Revised: We call this task long-horizon prediction as the boundary T ′ is so large that (with a high probability) many events will happen over ( T, T ′ ] .

Original: A principled way to solve this task works as follows: we draw manypossible future event sequences over the interval ( T, T ′ ] , and then use this empirical distribution toanswer questions such as “how many events of type k = 3 will happen over that interval”.
Revised: A principled way to solve this task works as follows: we draw many possible future event sequences over the interval ( T, T ′ ] , and then use this empirical distribution to answer questions such as “how many events of type k = 3 will happen over that interval”.

Original: A serious technical issue arises when we draw each possible future sequence.
Revised: A serious technical issue arises when we draw each possible future sequence.

Original: To draw an eventsequence from an autoregressive model, we have to repeatedly draw the next event, append it to thehistory, and then continue to draw the next event conditioned on the new history.
Revised: To draw an event sequence from an autoregressive model, we have to repeatedly draw the next event, append it to the history, and then continue to draw the next event conditioned on the new history.

Original: This process isprone to cascading errors : any error in a drawn event is likely to cause all the subsequent draws todiffer from what they should be, and such errors will accumulate.
Revised: This process is prone to cascading errors : any error in a drawn event is likely to cause all the subsequent draws to differ from what they should be, and such errors will accumulate.

Original: 2.3 Globally Normalized Models: Hope and Difficulties
Revised: 2.3 Globally Normalized Models: Hope and Difficulties

Original: An ideal fix of this issue is to develop a globally normalized model for event sequences.
Revised: An ideal fix of this issue is to develop a globally normalized model for event sequences.

Original: For anytime interval [0 , T ] , such a model will give a probability distribution that is normalized over allthe possible full sequences on [0 , T ] rather than over all the possible instantaneous subsequenceswithin each ( t, t + dt ) .
Revised: For any time interval [0 , T ] , such a model will give a probability distribution that is normalized over all the possible full sequences on [0 , T ] rather than over all the possible instantaneous subsequences within each ( t, t + dt ) .

Original: Technically, a globally normalized model assigns to each sequence x [0 ,T ] a score exp (cid:0) − E ( x [0 ,T ] ) (cid:1) where E is called energy function ; the normalized probability of x [0 ,T ] isproportional to its score: i.e., p ( x [0 ,T ] ) ∝ exp (cid:0) − E ( x [0 ,T ] ) (cid:1) .
Revised: Technically, a globally normalized model assigns to each sequence x [0 ,T ] a score exp (cid:0) − E ( x [0 ,T ] ) (cid:1) where E is called energy function ; the normalized probability of x [0 ,T ] is proportional to its score: i.e., p ( x [0 ,T ] ) ∝ exp (cid:0) − E ( x [0 ,T ] ) (cid:1) .

Original: Had we trained a globally normalized model, we wish to enumerate all the possible x [ T,T ′ ] for agiven x [0 ,T ] and select those which give the highest model probabilities p ( x [0 ,T ′ ] ) .
Revised: Had we trained a globally normalized model, we wish to enumerate all the possible x [ T,T ′ ] for a given x [0 ,T ] and select those which give the highest model probabilities p ( x [0 ,T ′ ] ) .

Original: Prediction madethis way would not suffer cascading errors: the entire x [0 ,T ′ ] was jointly selected and thus the overallcompatibility between the events had been considered.
Revised: Prediction made this way would not suffer cascading errors: the entire x [0 ,T ′ ] was jointly selected and thus the overall compatibility between the events had been considered.

Original: However, training such a globally normalized probabilistic model involves computing the normalizingconstant (cid:80) exp (cid:0) − E ( x [0 ,T ] ) (cid:1) where the summation (cid:80) is taken over all the possible sequences; it isintractable since there are infinitely many sequences.
Revised: However, training such a globally normalized probabilistic model involves computing the normalizing constant (cid:80) exp (cid:0) − E ( x [0 ,T ] ) (cid:1) where the summation (cid:80) is taken over all the possible sequences; it is intractable since there are infinitely many sequences.

Original: What’s worse, it is also intractable to exactlysample from such a model; approximate sampling is tractable but expensive.
Revised: What’s worse, it is also intractable to exactly sample from such a model; approximate sampling is tractable but expensive.

Original:  HYPRO: A Hybridly Normalized Neural Probabilistic Model
Revised:  HYPRO: A Hybridly Normalized Neural Probabilistic Model

Original: We propose HYPRO, a hybridly normalized neural probabilistic model that combines a temporalpoint process and an energy function: it enjoys both the efficiency of autoregressive models and thecapacity of globally normalized models.
Revised: We propose HYPRO, a hybridly normalized neural probabilistic model that combines a temporal point process and an energy function: it enjoys both the efficiency of autoregressive models and the capacity of globally normalized models.

Original: Our model normalizes over (sub)sequences: for any giveninterval [0 , T ] and its extension ( T, T ′ ] of interest, the model probability of the sequence x
Revised: Our model normalizes over (sub)sequences: for any given interval [0 , T ] and its extension ( T, T ′ ] of interest, the model probability of the sequence x

Original: ( T,T ′ ] is
Revised: ( T,T ′ ] is

Original: [Equation]
Revised: [Equation]

Original: where p auto is the probability under the chosen temporal point process and E θ is an energy functionwith parameters θ .
Revised: where p auto is the probability under the chosen temporal point process and E θ is an energy function with parameters θ .

Original: The normalizing constant  Z θ (cid:0) x [0 ,T ] (cid:1) def = (cid:80)x
Revised: The normalizing constant sums over all the possible continuations x ( T,T ′ ] for a given prefix x [0 ,T ] : Z θ (cid:0) x [0 ,T ] (cid:1) def = (cid:80)

Original:  ( T,T ′ ] exp (cid:0)− E θ ( x [0 ,T ′ ] ) (cid:1) sums overall the possible continuations x
Revised: x ( T,T ′ ] p auto (cid:0) x

Original: ( T,T ′ ] for a given prefix x [0 ,T ]  .
Revised: ( T,T ′ ] | x [0 ,T ] (cid:1) exp (cid:0) − E θ ( x [0 ,T ′ ] ) (cid:1) .

Original:  Our model is general: in principle, p auto can be any autoregressive model including those mentionedin section 2.1 and E θ can be any function that is able to encode an event sequence to a real number.
Revised: The key advantage of our model over autoregressive models is that: the energy function E θ is able to pick up the global features that may have been missed by the autoregressive base model p auto ; intuitively, the energy function fits the residuals that are not captured by the autoregressive model. Our model is general: in principle, p auto can be any autoregressive model including those mentioned in section 2.1 and E θ can be any function that is able to encode an event sequence to a real number.

Original: In section 5, we will introduce a couple of specific p auto and E θ and experiment with them.
Revised: In section 5, we will introduce a couple of specific p auto and E θ and experiment with them.

Original: In this section, we focus on the training method and inference algorithm.
Revised: In this section, we focus on the training method and inference algorithm.

Original: 3.1 Training Objectives
Revised: 3.1 Training Objectives

Original: Training our full model p HYPRO is to learn the parameters of the autoregressive model p auto as well asthose of the energy function E θ .
Revised: Training our full model p HYPRO is to learn the parameters of the autoregressive model p auto as well as those of the energy function E θ .

Original: Maximum likelihood estimation (MLE) is undesirable: the objectivewould be log p HYPRO (cid:0) x
Revised: Maximum likelihood estimation (MLE) is undesirable: the objective would be log p HYPRO (cid:0) x

Original: ( T,T ′ ]
Revised: ( T,T ′ ]

Original: | x [0 ,T ) (cid:1) = log p auto (cid:0) x
Revised: | x [0 ,T ] (cid:1) = log p auto (cid:0) x

Original: ( T,T ′ ]
Revised: ( T,T ′ ]

Original: | x [0 ,T ] (cid:1) − E θ ( x [0 ,T ′ ] ) − log Z θ (cid:0) x [0 ,T ] (cid:1)where the normalizing constant Z θ (cid:0) x [0 ,T ] (cid:1) is known to be uncomputable and inapproximable for alarge variety of reasonably expressive functions E θ (Lin & McCarthy, 2022).
Revised: | x [0 ,T ] (cid:1) − E θ ( x [0 ,T ′ ] ) − log Z θ (cid:0) x [0 ,T ] (cid:1) where the normalizing constant Z θ (cid:0) x [0 ,T ] (cid:1) is known to be uncomputable and inapproximable for a large variety of reasonably expressive functions E θ (Lin & McCarthy, 2022).

Original: We propose a training method that works around this normalizing constant.
Revised: We propose a training method that works around this normalizing constant.

Original: We first train p auto justlike how previous work trained temporal point processes.
Revised: We first train p auto just like how previous work trained temporal point processes.

Original: Then we use the trained p auto as a noisedistribution and learn the parameters θ of E θ by noise-contrastive estimation (NCE).
Revised: Then we use the trained p auto as a noise distribution and learn the parameters θ of E θ by noise-contrastive estimation (NCE).

Original: Precisely, wesample N noise sequences x (1) [ T,T ′ ] , . . .
Revised: Precisely, we sample N noise sequences x (1)[ T,T ′ ] , . . .

Original: , x ( N ) [ T,T ′ ] , compute the “energy” E θ ( x ( n ) [0 ,T ′ ] ) for each completedsequence x ( n ) [0 ,T ′ ] , and then plug those energies into one of the following training objectives.
Revised: , x ( N ) [ T,T ′ ] , compute the “energy” E θ ( x ( n ) [0 ,T ′ ] ) for each completed sequence x ( n ) [0 ,T ′ ] , and then plug those energies into one of the following training objectives.

Original: Note that all the completed sequences x ( n ) [0 ,T ′ ] share the same observed prefix x [0 ,T ] .
Revised: Note that all the completed sequences x ( n ) [0 ,T ′ ] share the same observed prefix x [0 ,T ] .

Original: Binary-NCE Objective.
Revised: Binary-NCE Objective.

Original: We train a binary classifier based on the energy function E θ to discriminatethe true event sequence—denoted as x (0)[0 ,T ′ ] —against the noise sequences by maximizing
Revised: We train a binary classifier based on the energy function E θ to discriminate the true event sequence—denoted as x (0)[0 ,T ′ ] —against the noise sequences by maximizing

Original: [Equation]
Revised: [Equation]

Original: where σ ( u ) =
Revised: where σ ( u ) =

Original: 1 1+exp( − u ) is the sigmoid function.
Revised: 1 1+exp( − u ) is the sigmoid function.

Original: By maximizing this objective, we are essentiallypushing our energy function E θ such that the observed sequences have low energy but the noise sequences have high energy.
Revised: By maximizing this objective, we are essentially pushing our energy function E θ such that the observed sequences have low energy but the noise sequences have high energy.

Original: As a result, the observed sequences will be more probable under our fullmodel p HYPRO while the noise sequences will be less probable : see equation (2).
Revised: As a result, the observed sequences will be more probable under our full model p HYPRO while the noise sequences will be less probable : see equation (2).

Original: Theoretical guarantees of general Binary-NCE can be found in Gutmann & Hyvärinen (2010).
Revised: Theoretical guarantees of general Binary-NCE can be found in Gutmann & Hyvärinen (2010).

Original: For general conditional probability models like ours, Binary-NCE implicitly assumes selfnormalization (Mnih & Teh, 2012; Ma & Collins, 2018): i.e., Z θ (cid:0) x
Revised: For general conditional probability models like ours, Binary-NCE implicitly assumes selfnormalization (Mnih & Teh, 2012; Ma & Collins, 2018): i.e., Z θ (cid:0) x

Original: [0 ,T ] (cid:1)
Revised: [0 ,T ] (cid:1)

Original: = 1 is satisfied.
Revised: = 1 is satisfied.

Original: This type of training objective has been used to train a hybridly normalized text generation model by
Revised: This type of training objective has been used to train a hybridly normalized text generation model by

Original: Deng et al.
Revised: Deng et al.

Original: ; see section 4 for more discussion about its relations with our work.
Revised: ; see section 4 for more discussion about its relations with our work.

Original: Multi-NCE Objective.
Revised: Multi-NCE Objective.

Original: Another option is to use Multi-NCE objective 2 , which means we maximize
Revised: Another option is to use Multi-NCE objective 2 , which means we maximize

Original: [Equation]
Revised: [Equation]

Original: By maximizing this objective, we are pushing our energy function E θ such that each observedsequence has relatively lower energy than the noise sequences sharing the same observed prefix.
Revised: By maximizing this objective, we are pushing our energy function E θ such that each observed sequence has relatively lower energy than the noise sequences sharing the same observed prefix.

Original: Incontrast, J binary attempts to make energies absolutely low (for observed data) or high (for noise data)without considering whether they share prefixes.
Revised: In contrast, J binary attempts to make energies absolutely low (for observed data) or high (for noise data) without considering whether they share prefixes.

Original: This effect is analyzed in Analysis-II of section 5.2.
Revised: This effect is analyzed in Analysis-III of section 5.2.

Original: This J multi objective also enjoys better statistical properties than J binary since it doesn’t assume selfnormalization: the normalizing constant Z θ (cid:0) x [0 ,T ] (cid:1) is neatly cancelled out in its derivation; see
Revised: This J multi objective also enjoys better statistical properties than J binary since it doesn’t assume selfnormalization: the normalizing constant Z θ (cid:0) x [0 ,T ] (cid:1) is neatly cancelled out in its derivation; see

Original: Appendix A.1 for a full derivation of both Binary-NCE and Multi-NCE.
Revised: Appendix A.1 for a full derivation of both Binary-NCE and Multi-NCE.

Original: Theoretical guarantees of Multi-NCE for discrete-time models were established by Ma & Collins(2018); Mei et al.
Revised: Theoretical guarantees of Multi-NCE for discrete-time models were established by Ma & Collins (2018); Mei et al.

Original: (2020b) generalized them to temporal point processes.
Revised: (2020b) generalized them to temporal point processes.

Original: Considering Distances Between Sequences.
Revised: Considering Distances Between Sequences.

Original: Previous work (LeCun et al., 2006; Bakhtin et al. ,2019) reported that energy functions may be better learned if the distances between samples areconsidered.
Revised: Previous work (LeCun et al., 2006; Bakhtin et al., 2019) reported that energy functions may be better learned if the distances between samples are considered.

Original: This has inspired us to design a regularization term that enforces such consideration.
Revised: This has inspired us to design a regularization term that enforces such consideration.

Original: Suppose that we can measure a well-defined “distance” between the true sequence x (0)[0 ,T ′ ] and anynoise sequence x ( n ) [0 ,T ′ ] ; we denote it as d ( n ) .
Revised: Suppose that we can measure a well-defined “distance” between the true sequence x (0)[0 ,T ′ ] and any noise sequence x ( n ) [0 ,T ′ ] ; we denote it as d ( n ) .

Original: We encourage the energy of each noise sequence to behigher than that of the observed sequence by a margin; that is, we propose the following regularization:[Equation]
Revised: We encourage the energy of each noise sequence to be higher than that of the observed sequence by a margin; that is, we propose the following regularization:

Original: [Equation]
Revised: [Equation]

Original: where β > 0 is a hyperparameter that we tune on the held-out development data.
Revised: where β > 0 is a hyperparameter that we tune on the held-out development data.

Original: With thisregularization, the energies of the sequences with larger distances will be pulled farther apart: thiswill help discriminate not only between the observed sequence and the noise sequences, but alsobetween the noise sequences themselves, thus making the energy function E θ more informed.
Revised: With this regularization, the energies of the sequences with larger distances will be pulled farther apart: this will help discriminate not only between the observed sequence and the noise sequences, but also between the noise sequences themselves, thus making the energy function E θ more informed.

Original: This method is general so the distance d can be any appropriately defined metric.
Revised: This method is general so the distance d can be any appropriately defined metric.

Original: In section 5, wewill experiment with an optimal transport distance specifically designed for event sequences. 
Revised: In section 5, we will experiment with an optimal transport distance specifically designed for event sequences. Note that the distance d in the regularization may be the final test metric.

Revised: In that case, our method is directly optimizing for the final evaluation score.

Original: Generating Noise Sequences.
Revised: Generating Noise Sequences.

Original: Generating event sequences from an autoregressive temporal pointprocess has been well-studied in previous literature.
Revised: Generating event sequences from an autoregressive temporal point process has been well-studied in previous literature.

Original: The standard way is to call the thinningalgorithm (Lewis & Shedler, 1979; Liniger, 2009).
Revised: The standard way is to call the thinning algorithm (Lewis & Shedler, 1979; Liniger, 2009).

Original: The full recipe for our setting is in Algorithm 1.
Revised: The full recipe for our setting is in Algorithm 1.

Original: 3.2 Inference Algorithm
Revised: 3.2 Inference Algorithm

Original: Inference involves drawing future sequences x ( T,T ′ ] from the trained full model p HYPRO ; due to theuncomputability of the normalizing constant Z ( x [0 ,T ] ) , exact sampling is intractable.
Revised: Inference involves drawing future sequences x ( T,T ′ ] from the trained full model p HYPRO ; due to the uncomputability of the normalizing constant Z ( x [0 ,T ] ) , exact sampling is intractable.

Original: We propose a normalized importance sampling method to approximately draw x ( T,T ′ ] from p HYPRO ;it is shown in Algorithm 2.
Revised: We propose a normalized importance sampling method to approximately draw x ( T,T ′ ] from p HYPRO ; it is shown in Algorithm 2.

Original: We first use the trained p auto to be our proposal distribution and call thethinning algorithm (Algorithm 1) to draw proposals x ⟨ 1 ⟩ [ T,T ′ ] , . . 
Revised: We first use the trained p auto to be our proposal distribution and call the thinning algorithm (Algorithm 1) to draw proposals x ⟨ 1 ⟩ [ T,T ′ ] , . . .

Original: , x ⟨ M ⟩ [ T,T ′ ] .
Revised: , x ⟨ M ⟩ [ T,T ′ ] .

Original: Then we reweight those proposals with the normalized weights w ⟨ m ⟩ that are defined as
Revised: Then we reweight those proposals with the normalized weights w ⟨ m ⟩ that are defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation] This collection of weighted proposals is used for the long-horizon prediction over the interval ( T, T ′ ] :if we want the most probable sequence, we return the x ⟨ m ⟩ [ T,T ′ ] with the largest weight w ⟨ m ⟩ ; if wewant a minimum Bayes risk prediction (for a specific risk metric), we can use existing methods (e.g.,the consensus decoding method in Mei et al.
Revised: This collection of weighted proposals is used for the long-horizon prediction over the interval ( T, T ′ ] : if we want the most probable sequence, we return the x ⟨ m ⟩ [ T,T ′ ] with the largest weight w ⟨ m ⟩ ; if we want a minimum Bayes risk prediction (for a specific risk metric), we can use existing methods (e.g., the consensus decoding method in Mei et al.

Original: to compose those weighted samples into a singlesequence that minimizes the risk.
Revised: to compose those weighted samples into a single sequence that minimizes the risk.

Original: In our experiments (section 5), we used the most probable sequence.
Revised: In our experiments (section 5), we used the most probable sequence.

Original: Note that our sampling method is biased since the weights w ⟨ m ⟩ are normalized .
Revised: Note that our sampling method is biased since the weights w ⟨ m ⟩ are normalized .

Original: Unbiasedsampling in our setting is intractable since that will need our weights to be unnormalized: i.e.,w def = p HYPRO /p auto = exp ( − E θ ) /Z which circles back to the problem of Z ’s uncomputability.
Revised: Unbiased sampling in our setting is intractable since that will need our weights to be unnormalized: i.e., w def = p HYPRO /p auto = exp ( − E θ ) /Z which circles back to the problem of Z ’s uncomputability.

Original: Experimental results in section 5 show that our method indeed works well in practice despite that it is biased. 
Revised: Experimental results in section 5 show that our method indeed works well in practice despite that it is biased. Input interest; trained autoregressive

Original:  ,T ] , T ′ , p auto , M )
Revised: M Output: a collection of weighted proposals 1: procedure NIS( x [0 ,T ] , T ′ , p auto , E θ , M ) 2: ▷ use normalized importance sampling to approximately draw M proposals from p HYPRO 3: x ⟨ 1 ⟩ [ T,T ′ ] , . . , x ⟨ M ⟩ [ T,T ′ ] ← D RAW N OISE ( x [0 ,T ] , T ′ , p auto , M )

Original: ▷ see Algorithm4: construct completed sequences x ⟨ 1 ⟩ [0 ,T ′ ] , . . .
Revised: ▷ see Algorithm4: construct completed sequences x ⟨ 1 ⟩ [0 ,T ′ ] , . . .

Original: , x ⟨ M ⟩ [0 ,T ′ ] by appending each x ⟨ m ⟩ [ T,T ′ ] to x [0 ,T ] 5: compute the exponential of minus energy e ⟨ m ⟩ = exp (cid:16) − E θ ( x ⟨ m ⟩ [0 ,T ′ ] )
Revised: , x ⟨ M ⟩ [0 ,T ′ ] by appending each x ⟨ m ⟩ [ T,T ′ ] to x [0 ,T ] 5: compute the exponential of minus energy e ⟨ m ⟩ = exp (cid:16) − E θ ( x ⟨ m ⟩ [0 ,T ′ ] )

Original: (cid:17) for each proposal 6: compute the normalized weights w ⟨ m ⟩ = e ⟨ m ⟩ / (cid:80)
Revised: (cid:17) for each proposal 6: compute the normalized weights w ⟨ m ⟩ = e ⟨ m ⟩ / (cid:80)

Original: Mm ′
Revised: Mm ′

Original: =1 e ⟨ m ′ ⟩ 7: return ( w ⟨ 1 ⟩ , x ⟨ 1 ⟩ [ T,T ′ ] ) , . . .
Revised: =1 e ⟨ m ′ ⟩ 7: return ( w ⟨ 1 ⟩ , x ⟨ 1 ⟩ [ T,T ′ ] ) , . . .

Original: , ( w ⟨ M ⟩ , x ⟨ M ⟩ [ T,T ′ ] )
Revised: , ( w ⟨ M ⟩ , x ⟨ M ⟩ [ T,T ′ ] )

Original: ▷ return the collection of weighted proposals

Original: Related work
Revised: Related work

Original: Over the recent years, various neural temporal point processes have been proposed.
Revised: Over the recent years, various neural temporal point processes have been proposed.

Original: Many of them arebuilt on recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997); they include Duet al.
Revised: Many of them are built on recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997); they include Du et al.

Original: ; Mei & Eisner (2017); Xiao et al.
Revised: ; Mei & Eisner (2017); Xiao et al.

Original: (2017a,b);
Revised: (2017a,b);

Original: Omi et al.
Revised: Omi et al.

Original: ; Shchur et al.
Revised: ; Shchur et al.

Original: Mei et al.
Revised: Mei et al.

Original: (2020a)
Revised: (2020a)

Original: ; Boyd et al.
Revised: ; Boyd et al.

Original: Some others use Transformer architectures (Vaswani et al., 2017;
Revised: Some others use Transformer architectures (Vaswani et al., 2017;

Original: Radford et al., 2019): in particular, Zuo et al.
Revised: Radford et al., 2019): in particular, Zuo et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Enguehard et al.
Revised: ; Enguehard et al.

Original: Sharma et al.
Revised: Sharma et al.

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: All these models all autoregressive: theydefine the probability distribution over event sequences in terms of a sequence of locally-normalizedconditional distributions over events given their histories.
Revised: All these models all autoregressive: they define the probability distribution over event sequences in terms of a sequence of locally-normalized conditional distributions over events given their histories.

Original: Energy-based models, which have a long history in machine learning (Hopfield, 1982; Hinton, 2002;
Revised: Energy-based models, which have a long history in machine learning (Hopfield, 1982; Hinton, 2002;

Original: LeCun et al., 2006; Ranzato et al., 2007; Ngiam et al., 2011; Xie et al., 2019), define the distributionover sequences in a different way: they use energy functions to summarize each possible sequenceinto a scalar (called energy) and define the unnormalized probability of each sequence in terms ofits energy, then the probability distribution is normalized across all sequences; thus, they are alsocalled globally normalized models.
Revised: LeCun et al., 2006; Ranzato et al., 2007; Ngiam et al., 2011; Xie et al., 2019), define the distribution over sequences in a different way: they use energy functions to summarize each possible sequence into a scalar (called energy) and define the unnormalized probability of each sequence in terms of its energy, then the probability distribution is normalized across all sequences; thus, they are also called globally normalized models.

Original: Globally normalized models are a strict generalization of locallynormalized models (Lin et al., 2021), but their normalizing constants are often uncomputable (Lin &
Revised: Globally normalized models are a strict generalization of locally normalized models (Lin et al., 2021): all the locally normalized models are globally normalized; but the converse is not true.

Original:  McCarthy, 2022).
Revised: Moreover, energy functions are good at capturing global features and structures (Pang et al., 2021; Du & Mordatch, 2019; Brakel et al., 2013). However, the normalizing constants of globally normalized models are often uncomputable (Lin & McCarthy, 2022).

Original: Existing work that is most similar to ours is the energy-based text generation modelsof Bakhtin et al.
Revised: Existing work that is most similar to ours is the energy-based text generation models of Bakhtin et al.

Original: and
Revised: and

Original: Deng et al.
Revised: Deng et al.

Original: (2020) that train energy functions to reweight the outputsgenerated by pretrained autoregressive models.
Revised: (2020) that train energy functions to reweight the outputs generated by pretrained autoregressive models.

Original: The differences are: we work on different kindsof sequential data (continuous-time event sequences vs. discrete-time natural language sentences),and thus the architectures of our autoregressive model and energy function are different from theirs;additionally, we explored a wider range of training objectives (e.g., Multi-NCE) than they did.
Revised: The differences are: we work on different kinds of sequential data (continuoustime event sequences vs. discrete-time natural language sentences), and thus the architectures of our autoregressive model and energy function are different from theirs; additionally, we explored a wider range of training objectives (e.g., Multi-NCE) than they did.

Original: The task of long-horizon prediction has drawn much attention in several machine learning areassuch as regular time series analysis (Yu et al., 2019; Le Guen & Thome, 2019), natural languageprocessing (Guo et al., 2018;
Revised: The task of long-horizon prediction has drawn much attention in several machine learning areas such as regular time series analysis (Yu et al., 2019; Le Guen & Thome, 2019), natural language processing (Guo et al., 2018;

Original: Guan et al., 2021), and speech modeling (Oord et al., 2016).
Revised: Guan et al., 2021), and speech modeling (Oord et al., 2016).

Original: Deshpandeet al.
Revised: Deshpande et al.

Original: (2021) is the best-performing to-date in long-horizon prediction of event sequences: they adopt ahierarchical architecture similar to ours and use a ranking objective based on the counts of the events.
Revised: (2021) is the best-performing to-date in long-horizon prediction of event sequences: they adopt a hierarchical architecture similar to ours and use a ranking objective based on the counts of the events.

Original: Their method can be regarded as a special case of our framework (if we let our energy function readthe counts of the events), and our method works better in practice (see section 5).
Revised: Their method can be regarded as a special case of our framework (if we let our energy function read the counts of the events), and our method works better in practice (see section 5).

Original:  Experiments
Revised:  Experiments

Original: We implemented our methods with PyTorch (Paszke et al., 2017); our code is submitted for reviewand will be public after acceptance.
Revised: We implemented our methods with PyTorch (Paszke et al., 2017). Our code can be found at https: //github.com/alipay/hypro_tpp and https://github.com/iLampard/hypro_tpp .

Original: Implementation details can be found in Appendix B.2.
Revised: Implementation details can be found in Appendix B.2.

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: Given a train set of sequences, we use the full sequences to train the autoregressive model p auto bymaximizing equation (1).
Revised: Given a train set of sequences, we use the full sequences to train the autoregressive model p auto by maximizing equation (1).

Original: To train the energy function E θ , we need to split each sequence into a prefixx [0 ,T ] and a continuation x
Revised: To train the energy function E θ , we need to split each sequence into a prefix x [0 ,T ] and a continuation x

Original: ( T,T ′ ] : we choose T and T ′ such that there are 20 event tokens within( T, T ′ ] on average.
Revised: ( T,T ′ ] : we choose T and T ′ such that there are 20 event tokens within ( T, T ′ ] on average.

Original: During testing, for each prefix x [0 ,T ] , we draw 20 weighted samples (Algorithm 2)and choose the highest-weighted one as our prediction ˆ x ( T,T ′ ] .
Revised: During testing, for each prefix x [0 ,T ] , we draw 20 weighted samples (Algorithm 2) and choose the highest-weighted one as our prediction ˆ x ( T,T ′ ] .

Original: We evaluate our predictions by:
Revised: We evaluate our predictions by:

Original: • The root of mean square error (RMSE) of the number of the tokens of each event type: for each type k , we count the number of type- k tokens in the true continuation—denoted as C k —as well asthat in the prediction—denoted as ˆ C k ; then the mean square error is(cid:114)K (cid:80)
Revised: • The root of mean square error (RMSE) of the number of the tokens of each event type: for each type k , we count the number of type- k tokens in the true continuation—denoted as C k —as well as that in the prediction—denoted as ˆ C k ; then the mean square error is

Original: Kk =1 (cid:16) C k − ˆ C k (cid:17) 2 .
Revised: [Equation]

Original: • The optimal transport distance (OTD) between event sequences defined by Mei et al.
Revised: • The optimal transport distance (OTD) between event sequences defined by Mei et al.

Original: (2019): forany given prefix x [0 ,T ] , the distance is defined as the minimal cost of editing the prediction ˆ x ( T,T ′ ](by inserting or deleting events, changing their occurrence times, and changing their types) such that it becomes exactly the same as the true continuation x
Revised: (2019): for any given prefix x [0 ,T ] , the distance is defined as the minimal cost of editing the prediction ˆ x ( T,T ′ ] (by inserting or deleting events, changing their occurrence times, and changing their types) such that it becomes exactly the same as the true continuation x

Original: ( T,T ′ ] .
Revised: ( T,T ′ ] .

Original: We did experiments on two real-world datasets (see Appendix B.1 for dataset details):
Revised: We did experiments on two real-world datasets (see Appendix B.1 for dataset details):

Original: • Taobao (Alibaba, 2018) .
Revised: • Taobao (Alibaba, 2018) .

Original: This 
Revised: This public dataset was

Original: public dataset was created and released for the2018 Tianchi Big
Revised: created and released for the

Original:  Data Competition.
Revised: Tianchi Big Data Competition.

Original: It contains time-stamped behavior records (e.g., browsing, purchasing) ofanonymized users on the online shopping platform Taobao from November 25 through December03, 2017.
Revised: It contains time-stamped behavior records (e g., browsing, purchasing) of anonymized users on the online shopping platform Taobao from November 25 through December 03, 2017.

Original: Each category group (e.g., men’s clothing) is an event type, and we have K = 17 event types.
Revised: Each category group (e.g., men’s clothing) is an event type, and we have K = 17 event types.

Original: We use the browsing sequences of the most active 2000 users; each user has a sequence.
Revised: We use the browsing sequences of the most active 2000 users; each user has a sequence.

Original: Then we randomly sampled disjoint train,

Original:  dev and test sets with 1300 , 200 and 500 sequences.
Revised: Then we randomly sampled disjoint train, dev and test sets with 1300 , 200 and 500 sequences.

Original: The time unit is 3 hours; the average inter-arrival time is 0 .
Revised: The time unit is 3 hours; the average inter-arrival time is 0 .

Original: 06 (i.e., 0 .
Revised: 06 (i.e., 0 .

Original: 18 hour),
Revised: 18 hour)

Original:  and we choose the prediction horizon T ′ − T to be 1 . 5 that approximately covers 20 event tokens
Revised: , and we choose the prediction horizon T ′ − T to be 1 . 5 that approximately covers 20 event tokens.

Original: • Taxi (Whong, 2014) .
Revised: • Taxi (Whong, 2014) .

Original: This dataset tracks the time-stamped taxi pick-up and drop-off events acrossthe five boroughs of the New York city; each (borough, pick-up or drop-off) combination defines an event type, so there are K = 10 event types in total.
Revised: This dataset tracks the time-stamped taxi pick-up and drop-off events across the five boroughs of the New York city; each (borough, pick-up or drop-off) combination defines an event type, so there are K = 10 event types in total.

Original: We work on a randomly sampled subset of 2000 drivers and each driver has a sequence.
Revised: We work on a randomly sampled subset of 2000 drivers and each driver has a sequence.

Original: We randomly sampled disjoint train, dev and test sets with 1400 , 200 and 400 sequences.
Revised: We randomly sampled disjoint train, dev and test sets with 1400 , 200 and 400 sequences.

Original: The time unit is 1 hour; the average inter-arrival time is 0 .
Revised: The time unit is 1 hour; the average inter-arrival time is 0 .

Original:  and we set the prediction horizon to be 4 . 5 that approximately covers 20 event tokens
Revised: 22 , and we set the prediction horizon to be 4 . 5 that approximately covers 20 event tokens. d u a l t p p n h p n h p - l g a t t a t t - l g h y p r o - a - b h

Original: We choose two strong autoregressive models as our base model pauto :
Revised: y p r o - a - m 2. R oo t M e a n Sq u a r e E rr o r d u a l t p p n h p n h p - l g a t t a t t - l g h y p

Revised: r

Revised: o - a - b h

Revised: y p r

Revised: o - a - m

Revised: A v e r a g e

Revised: O T D

Revised: [Figure]

Revised: d u a l t p p n h p n h p - l g a t t a t t - l g h y p

Revised: r

Revised: o - a - b h

Revised: y p r

Revised: o - a - m 1.

Revised: R oo t M e a n Sq u a r e

Revised: E rr o r d u a l t p p n h p n h p - l g a t t a t t - l g h y p

Revised: r

Revised: o - a - b h

Revised: y p r

Revised: o - a - m

Revised: A v e r a g e

Revised: O T D (

Revised: [Figure]

Revised: Taxi Data d u a l t p p n h p n h p - l g a t t a t t - l g h y p

Revised: r

Revised: o - a - b h

Revised: y p r

Revised: o - a - m 1.

Revised: R oo t M e a n Sq u a r e

Revised: E rr

Revised: o r d u a l t pp nh p nh p - l g a tt

Revised: a tt - l g

Revised: h

Revised: y

Revised: p

Revised: r

Revised: o - a - b h

Revised: y p r

Revised: o - a - m 37.

Revised: A v e r a g e O

Revised: T D

Revised: [Figure]

Revised: • StackOverflow (Leskovec & Krevl, 2014) .

Revised: This dataset has two years of user awards on a questionanswering website: each user received a sequence of badges and there are K = 22 different kinds of badges in total.

Revised: We randomly sampled disjoint train, dev and test sets with 1400 , 400 andsequences from the dataset.

Revised: The time unit is 11 days; the average inter-arrival time is 0 .

Revised: and we set the prediction horizon to be 20 that approximately covers 20 event tokens.

Revised: We choose two strong autoregressive models as our base model p auto :

Original: • Neural Hawkes process (NHP) (Mei & Eisner, 2017) .
Revised: • Neural Hawkes process (NHP) (Mei & Eisner, 2017) .

Original: It is an LSTM-based autoregressive modelthat has demonstrated effective at modeling event sequences in various domains.
Revised: It is an LSTM-based autoregressive model that has demonstrated effective at modeling event sequences in various domains.

Original: • Attentative neural Hawkes process (AttNHP)
Revised: • Attentative neural Hawkes process (AttNHP)

Original: (Yang et al., 2022) .
Revised: (Yang et al., 2022) .

Original: It is an attention-basedautoregressive model—like Transformer language model (Vaswani et al., 2017; Radford et al. ,2019)—whose performance is comparable to or better than that of the NHP as well as otherattention-based models (Zuo et al., 2020; Zhang et al., 2020).
Revised: It is an attention-based autoregressive model—like Transformer language model (Vaswani et al., 2017; Radford et al., 2019)—whose performance is comparable to or better than that of the NHP as well as other attention-based models (Zuo et al., 2020; Zhang et al., 2020).

Original: For the energy function E θ , we adapt the continuous-time Transformer module of the AttNHPmodel: the Transformer module embeds the given sequence of events x into a fixed-dimensionalvector (see section-2 of Yang et al.
Revised: For the energy function E θ , we adapt the continuous-time Transformer module of the AttNHP model: the Transformer module embeds the given sequence of events x into a fixed-dimensional vector (see section-2 of Yang et al.

Original: (2022) for details), which is then mapped to a scalar ∈ R via amulti-layer perceptron (MLP); that scalar is the energy value
Revised: (2022) for details), which is then mapped to a scalar ∈ R via a multi-layer perceptron (MLP); that scalar is the energy value

Original: E θ ( x ) .
Revised: E θ ( x ) .

Original: We first train the two base models NHP and AttNHP; they are also used as the baseline methodsthat we will compare to.
Revised: We first train the two base models NHP and AttNHP; they are also used as the baseline methods that we will compare to.

Original: To speed up energy function training, we use the pretrained weights of the
Revised: To speed up energy function training, we use the pretrained weights of the

Original: AttNHP to initialize the Transformer part of the energy function; this trick was also used in Deng et al.(2020) to bootstrap the energy functions for text generation models.
Revised: AttNHP to initialize the Transformer part of the energy function; this trick was also used in Deng et al. (2020) to bootstrap the energy functions for text generation models.

Original: As the full model p HYPRO hassignificantly more parameters than the base model p auto , we also trained larger NHP and AttNHP withcomparable amounts of parameters as extra baselines.
Revised: As the full model p HYPRO has significantly more parameters than the base model p auto , we also trained larger NHP and AttNHP with comparable amounts of parameters as extra baselines.

Original: Additionally, we also compare to the DualTPPmodel of Deshpande et al.
Revised: Additionally, we also compare to the DualTPP model of Deshpande et al.

Original: Details about model parameters are in Table 2 of Appendix B.3.
Revised: Details about model parameters are in Table 2 of Appendix B.3.

Original: 5.2 Results and Analysis
Revised: 5.2 Results and Analysis

Original: The main results are shown in Figure 1.
Revised: The main results are shown in Figure 1.

Original: The OTD depends on the hyperparameter C del , which isthe cost of deleting or adding an event token of any type, so we used a range of values of C del andreport the averaged OTD in Figure 1; OTD for each specific C del can be found in Appendix B.4.moreresults about OTD can be found in Appendix B.4.
Revised: The OTD depends on the hyperparameter C del , which is the cost of deleting or adding an event token of any type, so we used a range of values of C del and report the averaged OTD in Figure 1; OTD for each specific C del can be found in Appendix B.4.

Original: As we can see, NHP and AttNHP work the worst inmost cases.
Revised: As we can see, NHP and AttNHP work the worst in most cases.

Original: DualTPP doesn’t seem to outperform these autoregressive baselines even though it learnsto rerank sequences based on their macro statistics; we believe that it is because DualTPP’s baseautoregressive model is not as powerful as the state-of-the-art NHP and AttNHP and using macrostatistics doesn’t help enough. 
Revised: DualTPP doesn’t seem to outperform these autoregressive baselines even though it learns to rerank sequences based on their macro statistics; we believe that it is because DualTPP’s base autoregressive model is not as powerful as the state-of-theart NHP and AttNHP and using macro statistics doesn’t help enough. Our HYPRO method works significantly better than these baselines. Analysis-I: Does the Cascading Error Exist?

Original: Our HYPRO method works significantly better than these baselines.
Revised: Handling cascading errors is a key motivation for our framework. On the Taobao dataset, we empirically confirmed that this issue indeed exists. [Figure] a) Without the using distance-based regularization of equation (5)

Original: Analysis-I: Energy Function or Just More Parameters?
Revised: [Figure] We first investigate whether the event type prediction errors are cascaded through the subsequent events. We grouped the sequences based on how early the first event type prediction error was made and then compared the event type prediction error rate on the subsequent events: • when the first error is made on the first event token, the base AttNHP model has a 71.28% error rate on the subsequent events and

Revised: our

Revised: hybrid

Revised: model has a much lower 66.99% error rate.

Revised: • when the first error is made on the fifth event token, the base AttNHP model has a 58.88% error rate on the subsequent events and our hybrid model has a much lower 51.89% error rate.

Revised: • when the first error is made on the tenth event token, the base AttNHP model has a 44.48% error rate on the subsequent events and our hybrid model has a much lower 35.58% error rate.

Revised: Obviously, when mistakes are made earlier in the sequences, we tend to end up with a higher error rate on the subsequent predictions; that means event type prediction errors are indeed cascaded through the subsequent predictions.

Revised: Moreover, in each group, our hybrid model enjoys a lower prediction error; that means it indeed helps mitigate this issue.

Revised: We then investigate whether the event time prediction errors are cascaded.

Revised: For this, we performed a linear regression: the independent variable x is the absolute error of the prediction on the time of the first event token; the dependent variable y is the averaged absolute error of the prediction on the time of the subsequent event tokens.

Revised: Our fitted linear model is y = 0 .

Revised: 7965 x + 0 .

Revised: where the p-value of the coefficient of x is ≈ 0 . 0001 < 0 .

Revised: It means that the time prediction errors are also cascaded through the subsequent predictions.

Original:  The larger NHP and AttNHP have almostthe same numbers of parameters with HYPRO, but their performance is only comparable to thesmaller NHP and AttNHP.
Revised: Analysis-II: Energy Function or Just More Parameters? The larger NHP and AttNHP have almost the same numbers of parameters with HYPRO, but their performance is only comparable to the smaller NHP and AttNHP.

Revised: That is to say, simply increasing the number of parameters in an autoregressive model will not achieve the performance of using an energy function.

Revised: To further verify the usefulness of the energy function, we also compared our method with another baseline method that ranks the completed sequences based on their probabilities under the base model, from which the continuations were drawn.

Revised: This baseline is similar to our proposed HYPRO framework but its scorer is the base model itself.

Revised: In our experiments, this baseline method is not better than our method; details can be found in Appendix B.5.

Original: That is, the energy function E θ is essential to the success of HYPRO.
Revised: Overall, we can conclude that the energy function E θ is essential to the success of HYPRO.

Original: Analysis-II: Binary-NCE vs. Multi-NCE.
Revised: Analysis-III: Binary-NCE vs. Multi-NCE.

Original: Both Binary-NCE and Multi-NCE objectives aim tomatch our full model distribution p HYPRO with the true data distribution, but Multi-NCE enjoysbetter statistical properties (see section 3) and achieved better performance in our experiments (see
Revised: Both Binary-NCE and Multi-NCE objectives aim to match our full model distribution p HYPRO with the true data distribution, but Multi-NCE enjoys better statistical properties (see section 3) and achieved better performance in our experiments (see

Original: Figure 1).
Revised: Figure 1).

Original: In Figure 2a, we display the distributions of the energy scores of the observed sequences,p HYPRO -generated sequences, and p auto -generated noise sequences: as we can see, the distribution ofp HYPRO is different from that of noise data and indeed closer to that of real data. 
Revised: In Figure 2a, we display the distributions of the energy scores of the observed sequences, p HYPRO -generated sequences, and p auto -generated noise sequences: as we can see, the distribution of p HYPRO is different from that of noise data and indeed closer to that of real data. [Figure] oo t M e a n

Revised: Sq u a r e

Revised: [Figure]

Original: Analysis-III: Effects of the Distance-Based Regularization Ω .
Revised: rr [Figure] r [Figure]

Revised: [Figure]

Revised: Horizon =0.

Revised: A v e r a g e

Revised: O T D

Revised: Horizon =0.8 18.

Original:  We experimented with the proposeddistance-based regularization Ω in equation (5); as shown in Figure 3, it slightly improves both
Revised: Horizon =1. Horizon =2 55. Analysis-IV: Effects of the Distance-Based Regularization Ω . We experimented with the proposed distance-based regularization Ω in equation (5); as shown in Figure 3, it slightly improves both

Original: Binary-NCE and Multi-NCE.
Revised: Binary-NCE and Multi-NCE.

Original: As shown in Figure 2b, the regularization makes a larger difference inthe Binary-NCE case: the energies of p HYPRO -generated sequences are pushed further to the left. 
Revised: As shown in Figure 2b, the regularization makes a larger difference in the Binary-NCE case: the energies of p HYPRO -generated sequences are pushed further to the left. We did the paired permutation test to verify the statistical significance of our regularization technique; see Appendix B.6 for details.

Original: d u a l t pp nh p nh p - l g a tt a tt - l g
Revised: Overall, we found that the performance improvements of using the regularization are strongly significant in the Binary-NCE case (p-value < 0 . 05 ) but not significant in the Multi-NCE case (p-value ≈ 0 . 1 ). This finding is consistent with the observations in Figure 2. Analysis-V: Effects of Prediction Horizon. Figure 4 shows how well our method performs for different prediction horizons.

Revised: On Taobao Dataset, we experimented with horizon being 0 . 3 , 0 . 8 , 1 . 5 , 2 , corresponding to approximately 5 , 10 , 20 , 30 event tokens, and found that our HYPRO method improves significantly and consistently over the autoregressive baseline AttNHP.

Original: h
Revised: h

Original: y p
Revised: y p

Revised: r o - a - b h y p

Original: r
Revised: r

Original: o - a - b h
Revised: o - n - b h

Original: y p r
Revised: y p r

Original: o - a - m  2.
Revised: o - a - m h y p r o - n - m 2.

Original: R oo t M e a n
Revised: R oo t M e a n

Original: Sq u a r e E rr 
Revised: Sq u a r e E rr o r h y p

Original: o rd u a l t pp nh p nh p - l g a tt
Revised: r

Original:  att - l g h
Revised: o - a - b h

Original: y p r
Revised: y p r

Original: o - a - b h
Revised: o - n - b h

Original: y p r
Revised: y p r

Original: o - a - m 35. A v e r a g e O T D
Revised: o - a - m h y p r o - n - m 35.

Original: [Figure]
Revised: [Figure]

Original: a

Original: [Figure]
Revised: [Figure]

Original: h
Revised: A v e r a g e O T

Original: [Figure]
Revised: D (a Data h y p r

Original: oo
Revised: o - a - b h y p r o - n - b h

Original: [Figure]
Revised: y p r o - a - m h y p r o - n - m 1. R oo t

Original:  e a n
Revised: M e a n

Original: Sq u a r e E rr 
Revised: Sq u a r e E rr o r h y p

Original: o rd u a l t pp nh p nh p - l g a tt
Revised: r

Original:  att - l g h
Revised: o - a - b h

Original: y p r
Revised: y p r

Original: o - a - b h
Revised: o - n - b h

Original: y p r
Revised: y p r

Original: o - a - m 18. A v e r a g
Revised: o - a - m h y p r o - n - m 18.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: Analysis-IV: Effects of Prediction Horizon.
Revised: A v e

Original: Figure 4 shows how well our method performs fordifferent prediction horizons.
Revised: [Figure] D ( [Figure] ) Using different on

Original: On Taobao Dataset, we experimented with horizon being 0 . 3 , 0 . 8 , 1 . 5 , 2 ,corresponding to approximately 5 , 10 , 20 , 30 event tokens, and found that our HYPRO methodimproves significantly and consistently over the autoregressive baseline AttNHP.
Revised: Data

Original: Analysis-V: Different Energy Functions.
Revised: Analysis-VI: Different Energy Functions.

Original: So far we have only shown the results and analysis ofusing the continuous-time Transformer architecture as the energy function E θ .
Revised: So far we have only shown the results and analysis of using the continuous-time Transformer architecture as the energy function E θ .

Original: We also experimentedwith using a continuous-time LSTM (Mei & Eisner, 2017) as the energy function and found thatit never outperformed the Transformer energy function in our experiments; see Figure 5.
Revised: We also experimented with using a continuous-time LSTM (Mei & Eisner, 2017) as the energy function and found that it never outperformed the Transformer energy function in our experiments; see Figure 5.

Original: We thinkit is because the Transformer architectures are better at embedding contextual information than
Revised: We think it is because the Transformer architectures are better at embedding contextual information than

Original: LSTMs (Vaswani et al., 2017; Pérez et al., 2019; O’Connor & Andreas, 2021). 
Revised: LSTMs (Vaswani et al., 2017; Pérez et al., 2019; O’Connor & Andreas, 2021). Analysis-VII: Negative Samples.

Original: Conclusion
Revised: On the Taobao dataset, we analyzed how the number of negative samples affects training and inference.

Original: We presented HYPRO, a hybridly normalized neural probabilistic model for the task of long-horizonprediction of event sequences.
Revised: We experimented with the Binary-NCE objective without the distance regularization (i.e., hypro-a-b). The results are in Figure 6.

Original: Our model consists of an autoregressive base model and an energyfunction: the latter learns to reweight the sequences drawn from the former such that the sequences
Revised: During training, increasing the number of negative samples from 1 to 5 has brought improvements but further increasing it todoes not.

Original: [Figure] oo t M e a n Sq u a r e
Revised: During inference, the results are improved when we increase the number of samples from 5 to 20, but they stop improving when we further increase it.

Original: [Figure] rr
Revised: Throughout the paper, we used 5 in training and 20 in inference (Appendix B.3).

Original: [Figure]
Revised: [Figure]

Original:  r
Revised:  Conclusion

Original: [Figure] [Figure] [Figure]
Revised: We presented HYPRO, a hybridly normalized neural probabilistic model for the task of long-horizon prediction of event sequences.

Original: [Figure] [Figure] [Table] that appear more realistic as a whole can end up with higher probabilities under our full model.
Revised: Our model consists of an autoregressive base model and an energy function: the latter learns to reweight the sequences drawn from the former such that the sequences that appear more realistic as a whole can end up with higher probabilities under our full model.

Original: We developed two training objectives that can train our model without computing its normalizingconstant, together with an efficient inference algorithm based on normalized importance sampling.
Revised: We developed two training objectives that can train our model without computing its normalizing constant, together with an efficient inference algorithm based on normalized importance sampling.

Original: Empirically, our method outperformed current state-of-the-art autoregressive models as well as arecent non-autoregressive model designed specifically for the same task.
Revised: Empirically, our method outperformed current state-of-the-art autoregressive models as well as a recent non-autoregressive model designed specifically for the same task.

Original:  Limitations and Societal Impacts
Revised:  Limitations and Societal Impacts

Original: Limitations.
Revised: Limitations.

Original: Our method uses neural networks, which are typically data-hungry.
Revised: Our method uses neural networks, which are typically data-hungry.

Original: Although it workedwell in our experiments, it might still suffer compared to non-neural models if starved of data.
Revised: Although it worked well in our experiments, it might still suffer compared to non-neural models if starved of data.

Original: Additionally, our method requires the training sequences to be sufficiently long so it can learn tomake long-horizon predictions; it may suffer if the training sequences are short.
Revised: Additionally, our method requires the training sequences to be sufficiently long so it can learn to make long-horizon predictions; it may suffer if the training sequences are short.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: Our paper develops a novel probabilistic model for long-horizon prediction ofevent sequences.
Revised: Our paper develops a novel probabilistic model for long-horizon prediction of event sequences.

Original: By describing the model and releasing code, we hope to facilitate probabilisticmodeling of continuous-time sequential data in many domains.
Revised: By describing the model and releasing code, we hope to facilitate probabilistic modeling of continuous-time sequential data in many domains.

Original: However, like many other machinelearning models, our model may be applied to unethical ends.
Revised: However, like many other machine learning models, our model may be applied to unethical ends.

Original: For example, its abilities of betterfitting data and making more accurate predictions could potentially be used for unwanted tracking ofindividual behavior, e.g. for surveillance.
Revised: For example, its abilities of better fitting data and making more accurate predictions could potentially be used for unwanted tracking of individual behavior, e.g. for surveillance.

Original: Checklist
Revised: Acknowledgments

Original: For all authors...
Revised: This work was supported by a research gift to the last author by Adobe Research. We thank the anonymous NeurIPS reviewers and meta-reviewer for their constructive feedback. We also thank our colleagues at Ant Group for helpful discussion. A Method Details A.1 Derivation of NCE Objectives

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: Given a prefix x [0 ,T ] , we have the true continuation x (0)( T,T ′ ] and N noise samples x (1)( T,T ′ ] , . . . , x ( N ) ( T,T ′ ] . By concatenating the prefix and each (true or noise) continuation, we obtain N + 1 completed sequences x (0)[0 ,T ′ ] , x (1)[0 ,T ′ ] , . . . , x ( N ) [0 ,T ′ ] .

Revised: Binary-NCE Objective.

Original: [Yes](b) Did you describe the limitations of your work?
Revised: For each completed sequence x ( n ) [0 ,T ′ ] , we learn to classify whether it is real data or noise data. The unnormalized probability for each case is: [Equation] Then the normalized probabilities are:

Original: [Yes] See section 7(c)
Revised: [Equation] Following previous work (Mnih & Teh, 2012), we assume that the model is self-normalized, i.e., Z θ (cid:0) x [0 ,T ] (cid:1) = 1 . Then the normalized probabilities become

Revised: [Equation]

Revised: where σ is the sigmoid function.

Revised: For the true completed sequence x (0)[0 ,T ′ ] , we maximize the log probability that it is real data, i.e., log p ( it is real data ) ; for each noise sequence x ( n ) [0 ,T ′ ] , we maximize the log probability that it is noise data, i.e., log p ( it is noise data ) .

Revised: The Binary-NCE objective turns out to be equation (3), i.e.,

Revised: [Equation]

Revised: Multi-NCE Objective.

Revised: For these N +1 sequences, we learn to discriminate the true sequence against the noise sequences.

Original: Did you discuss any potential negative societal impacts of your work?
Revised: For each of them x ( n ) [0 ,T ′ ] , the following is the unnormalized probability that it is real data but all others are noise: [Equation] real, others are [Equation]

Revised: where can be rearranged to be

Revised: [Equation]

Revised: , others

Revised: [Equation]

Original: [Yes] Seesection 7.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?{+1 Z (cid:81) Nn =
 p auto is constant with respect to n . So we can ignore that term and obtain+}
Revised: Note that {+1 Z (cid:81) Nn =
 p auto is constant with respect to n . So we can ignore that term and obtain+}

Original: [Yes]
Revised: [Equation]

Revised: ] is real, others are noise

Revised: [Equation]

Revised: Therefore

Revised: [Equation]

Original: If you are including theoretical results...
Revised: we can obtain the normalized probability that x (0)[0 ,T ′ ] is real data as below [Equation] until break 14: draw k ∈ { 1 , . . , K } probability of k is ∝ λ k ( t 0 | H ) 15: append

Revised: T

Revised: Note that the normalizing constant Z doesn’t show up in the normalized probability since it has been cancelled out as a part of the 1 Z (cid:81)

Original: {+Nn =
+}
Revised: {+Nn =
+}

Revised: p auto constant.

Revised: That is, unlike the Binary-NCE case, we do not need to assume self-normalization in this Multi-NCE case.

Revised: We maximize the log probability that x (0)[0 ,T ′ ] is real data, i.e., log p (cid:16) x (0)[0 ,T ′ ] is real data (cid:17) ; the

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: Multi-NCE objective turns out to be equation (4), i.e., [Equation] A.2 Sampling Algorithm Details In section 3.2, we described a sampling method to approximately draw x ( T,T ′ ] from p HYPRO .

Revised: It calls the thinning algorithm, which we describe in Algorithm 3.

Revised: B Experimental Details B.1 Dataset Details

Revised: Taobao (Alibaba, 2018).

Revised: This dataset contains time-stamped user click behaviors on Taobao shopping pages from November 25 to December 03, 2017.

Revised: Each user has a sequence of item click events with each event containing the timestamp and the category of the item.

Revised: The categories of all items are first ranked by frequencies and the top 16 are kept while the rests are merged into one category, with each category corresponding to an event type.

Revised: We work on a subset of 2000 most active users with average sequence length 58 and then end up with K = 17 event types.

Revised: We randomly sampled disjoint train, dev and test sets with 1300 , 200 and 500 sequences from the dataset.

Revised: Given the average inter-arrival time 0 .

Revised: 06 (time unit is 3 hours), we choose the prediction horizon as 1 . 5 that approximately hasevent tokens per sequence.

Revised: Taxi (Whong, 2014).

Revised: This dataset contains time-stamped taxi pickup and drop off events with zone location ids in New York city in 2013 .

Revised: Following the processing recipe of previous work (Mei et al., 2019), each event type is defined as a tuple of (location, action).

Revised: The location is one of the 5 boroughs { Manhattan, Brooklyn, Queens, The Bronx, Staten Island } .

Original: [Yes] See sections
Revised: The action can be either pick-up or drop-off. Thus, there are K = 5 × 2 = 10 event types in total. We work on a subset of 2000 sequences of taxi pickup events with average length 39 and then end up with K = 10 event types. We randomly sampled disjoint train, dev and test sets with 1400 , 200 and 400 sequences from the dataset.

Original: 2and 3 and Appendix A.1.
Revised: Given the average inter-arrival time 0 . 22 (time unit is 1 hour), we choose the prediction horizon as 4 . 5 that approximately has 20 event tokens per sequence. [Table] StackOverflow (Leskovec & Krevl, 2014) .

Revised: This dataset has two years of user awards on a questionanswering website: each user received a sequence of badges and there are K = 22 different kinds of badges in total.

Revised: We randomly sampled disjoint train, dev and test sets with 1400 , 400 andsequences from the dataset.

Revised: The time unit is 11 days; the average inter-arrival time is 0 .

Original: (b) Did you include complete proofs of all theoretical results?
Revised: and we set the prediction horizon to be 20 that approximately covers 20 event tokens. Table 1 shows statistics about each dataset mentioned above. B.2 Implementation Details All models are implemented using the PyTorch framework (Paszke et al., 2017).

Original: [Yes] See Appendix A.1.
Revised: For the implementation of NHP, AttNHP, and thinning algorithm, we used the code from the public Github repository at https://github.com/yangalan123/anhp-andtt (Yang et al., 2022) with MIT License. For DualTPP, we used the code from the public Github repository at https://github.com/ pratham16cse/DualTPP (Deshpande et al., 2021) with no license specified.

Revised: For the optimal transport distance, we used the code from the public Github repository at https: //github.com/hongyuanmei/neural-hawkes-particle-smoothing (Mei et al., 2019) with

Revised: BSD 3-Clause License.

Original: If you ran experiments...
Revised: Our code can be found at https://github.com/alipay/hypro_tpp and https://github. com/iLampard/hypro_tpp . B.3 Training and Testing Details Training Generators.

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: For AttNHP, the main hyperparameters to tune are the hidden dimension D of the neural network and the number of layers L of the attention structure. In practice, the optimal D for a model was usually 32 or 64 ; the optimal L was usually 1 , 2 , 3 , 4 . In the experiment, we set

Revised: D = 32 , L = 2 for AttNHP and D = 32 , L = 4 for AttNHP-LG.

Revised: To train the parameters for a given generator, we performed early stopping based on log-likelihood on the held-out dev set.

Revised: Training Energy Function.

Revised: The energy function is built on NHP or AttNHP with 3 MLP layers to project the hidden states into a scalar energy value.

Revised: AttNHP is set to have the same structure as the base generator ’Att’.

Revised: NHP is set to have D = 36 so that the joint model have the comparable number of parameters with other competitors.

Revised: During training, each pair of training sample contains 1 positive sample and 5 negative samples ( N = 5 in equation 3 and 4), generated from generators.

Revised: Regarding the regularization term in equation 5, we choose β = 1 . 0 .

Revised: All models are optimized using Adam (Kingma & Ba, 2015).

Revised: Testing.

Revised: During testing, for efficiency, we generates 20 samples ( M = 20 in Algorithm 2) per test prefix and select the one with the highest weight as the prediction.

Revised: Increasing M could possibly improves the prediction performance.

Revised: Computation Cost.

Revised: All the experiments were conducted on a server with 256 G RAM, a 64 logical cores CPU (Intel(R)

Revised: Xeon(R)

Revised: Platinum 8163 CPU @ 2.50GHz) and one NVIDIA Tesla P100 GPU for acceleration.

Revised: On all the datasets, the training time of HYPRO-A and HYPRO-N is 0 .

Revised: 005 seconds per positive sequence.

Revised: For training, our batch size is 32.

Revised: For Taobao and Taxi dataset, training the baseline NHP, NHP-lg,

Revised: AttNHP, AttNHP-lg approximately takes 1 hour, 1.3 hour, 2 hours, and 3 hours, respectively (12, 16, 25, 38 milliseconds per sequence), training the continuous-time LSTM energy function and continuous-time Transformer energy function takes 20 minutes and 35 minutes (4 and 7 milliseconds per sequence pair) respectively.

Revised: [Table]

Revised: FUNCTION

Revised: [Table]

Revised: CLONE

Revised: [Table]

Revised: [Table]

Revised: For inference, inference with energy functions takes roughly 2 to 4 milliseconds.

Revised: It takes 0.2 seconds to draw a sequence from the autoregressive base model.

Revised: Our implementation can draw multiple sequences at a time in parallel: it takes only about 0.4 seconds to draw 20 sequences—only twice as drawing a single sequence.

Revised: We have released this implementation.

Revised: B.4 More OTD Results

Revised: The optimal transport distance (OTD) depends on the hyperparameter C del , which is the cost of deleting or adding an event token of any type.

Revised: In our experiments, we used a range of values of

Revised: C del ∈ { 0 . 05 , 0 . 5 , 1 , 1 . 5 , 2 , 3 , 4 } , and report the averaged OTD in Figure 1.

Revised: In this section, we show the OTD for each specific C del in Figure 7.

Revised: As we can see, for all the values of C del , our HYPRO method consistently outperforms the other methods.

Revised: B.5 Analysis Details: Baseline That Ranks Sequences by the Base Model

Revised: To further verify the usefulness of the energy function in our model, we developed an extra baseline method that ranks the completed sequences based on their probabilities under the base model, from which the continuations were drawn.

Revised: This baseline is similar to our proposed HYPRO framework but its scorer is the base model itself.

Revised: We evaluated this baseline on the Taobao dataset.

Revised: The results are in Figure 8.

Revised: As we can see, this new baseline method is no better than our method in terms of the OTD metric but much worse than all the other methods in terms of the RMSE metric.

Revised: B.6 Analysis Details: Statistical Significance

Revised: We performed the paired permutation test to validate the significance of our proposed regularization technique.

Original: perimental results (either in the supplemental material or as a URL)?
Revised: Particularly, for each model variant (hypro-a-b or hypro-a-m), we split the test data into ten folds and collected the paired test results with and without the regularization technique for [Table] [Figure] [Figure]

Revised: nhp

Original: [Yes] See thesupplemental material.
Revised: [Figure] att att-lg hypro-a [Figure] b hypro-a-m

Revised: C

Revised: [Table]

Revised: [Table]

Revised: C

Revised: [Table]

Revised: =

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: (b) Did you specify all the training details (e
Revised: [Figure] Data [Figure] [Figure]

Original: g., data splits, hyperparameters,
Revised: dualtpp nhp nhp [Figure] lg att att-lg hypro [Table]

Revised: [Figure]

Revised: [Table]

Revised: hypro

Revised: [Table]

Revised: -m

Revised: C

Original: how they
Revised: [Table] = [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: C

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: C del =

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: C del =

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: were chosen)?
Revised: [Table] [Table] [Table] [Table]

Original: [Yes] See Appendix B.1 and Appendix B.3.(c)
Revised: C del = [Table] [Figure] [Figure]

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: att

Original: [Yes] See section 5.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: [Figure] C del = 26. C del =1. [Figure]

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: C del = 46.

Original: [Yes] See Appendix B.3.
Revised: [Figure] C del = C del = [Figure]

Original: If you are using existing assets (e
Revised: each fold. Then we performed the test and computed the p-value following the recipe at https: //axon.cs.byu.edu/

Original: g., code, data, models) or curating/releasing new assets...
Revised: Dan/478/assignments/permutation_test.php .

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix B.3(b) Did you mention the license of the assets?
Revised: The results are in Figure 9. It turns out that the performance differences are strongly significant for hypro-a-b (p-value < 0 . 05 ) but not significant for hypro-a-m (p-value ≈ 0 . 1 ).

Original: [Yes] See Appendix B.2.
Revised: This is consistent with the findings in Figure 2. [Figure]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: a t t a t t

Original: [Yes]We include our code in the supplementary material.
Revised: [Figure] g h y p [Figure]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: a

Original: using/curating?
Revised: [Figure] oo [Figure]

Original: [N/A] We use existing datasets that are publicly available and already anonymized. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: M

Original: [N/A] We use existing datasets that are publicly
Revised: [Figure] n Sq u a r e E rr

Original: available and already anonymized.
Revised: o r

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: [Figure]

Original: applicable?
Revised: t t - l

Original: [N/A]
Revised: [Figure]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: h y p

Original: Board (IRB) approvals, if applicable?
Revised: r o - a

Original: [N/A]
Revised: [Figure]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: h y p r o [Figure]

Original: [N/A]
Revised: [Figure]