From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper presents a spatiotemporal point process model based on a transformer, which allows for simultaneously predicting times and location of multiple (fixed number of) future events. The reviewers did recognize that the paper is well written and the contribution is sound. A majority of reviewers were not in support of accepting the paper due to a number of issues, including limited significance and insufficient experimental evaluation. During the discussion period, the authors did not manage to persuade the reviewers to change their original assessment. However, I encourage the authors to revise the paper in light of the reviewers' comments and resubmit to another top tier venue.
justification_for_why_not_higher_score: Limited significance and insufficient experimental evaluation.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer mvAP
comment: Please note that we have addressed your questions regarding including the extra markers (such as magnitudes in the case of earthquake data) in Appendix A.2 in our paper.

In section 3.2 we have emphasized what each layer is contributing towards addressing the limitations associated with previous works.
Moreover in appendix A.3, we have highlighted the contribution of the main layers via two types of ablation studies.

We hope we have addressed the reviewer's comments regarding the novelty of the proposed network for our problem of interest.
We also would be happy to clarify any ambiguity regarding the loss we defined for the numerical experiments in equations 9, 10 and 13.

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_yeaA
title: Thank you for your patience
comment: I'm sorry I did not respond in a timely manner. I haven't gotten around to it yet, but I will take a look and update my review accordingly. Thank you for your patience.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: responses to reviewer boMt (second round)
comment: > * The described procedure, where all inter-event times are normalized to be between 0 and 1, may introduce information leakage from the test set, if the normalization is done before the train/test split. If normalization is done only using the train set, then the test set may contain an inter-event time > 1 outside of the distribution support, so the model will assign $-\infty$ log-likelihood to this event.

Please note that all normalization is done only $\textbf{AFTER}$ splitting the data into train, validation, and test.
For location and other markers shown by $M$, we use the mean and variance of the training set to normalize all data in train/validation/test. After prediction on the test, we use the same mean and variance (which is associated with the training set) to denormalize $\mathbf{x}$ and $M$ associated with the test set.

However, for time normalization we took a different approach since time is always increasing as opposed to other markers. More precisely, we use a similar sequence-time formation and normalization approach that was used in [Neural Spatio-temporal point processes](https://arxiv.org/abs/2011.04583). 

In this case, first, when forming sequences of length $n+L$, we reformulate the times $t_{1}$ to $t_{n+L}$ associated with all events to that sequence to start from or close to 0. This happens before splitting the data into train/validation/test. Next, before normalization and data splitting, we calculate the time intervals $\Delta t_{1}$ to $\Delta t_{n+L}$ associated with each sequence.

In our next step, after splitting data into train/validation/test we take the approach to put all $n$ input events between 0 and 1. The same approach is taken for the $L$ output events, meaning that after forming input and output sequences, we normalize the events' time associated with the $n$ input events to start from 0 and finish at 1, and normalize the events' time associated with the $L$ output events to start from 0 and finish at 1. In this regard, all the $n$ input and $L$ output events are normalized using the min and max of their own sequence. Thus, we $\textbf{Do Not}$ use the min and max of the train set to normalize all sequences.

One side-note here is we realized that normalizing the time intervals is not as necessary as time values, since in most cases time intervals stay in the range between 0 and 1. But in case there are intervals larger than 1, we normalize them using the same approach as time values.

However, in the test phase, we expect to **denormalize** the output time-intervals $\Delta \hat t_{n+1}$ to $\Delta \hat t_{n+L}$ that we extract from the predicted $L$ temporal distributions $\hat p(t_{n+1}|H_n)$ to $\hat p(t_{n+L}|H_{n+L-1})$ to recover the original values. 

For that matter and to reach the increasing effect of temporal events as shown by the top figures of figures 2 and 3 in our paper, in the test phase we do the following:

1) Simultaneously extract the inter-event timings $\Delta \hat t_{n+1}$ to $\Delta \hat t_{n+L}$ via sampling (mean of 1000 samples) from the predicted batched temporal distributions $\hat p(t_{n+1}|H_n)$ to $\hat p(t_{n+L}|H_{n+L-1})$.

2) Use the min and max associated with the **train set** $L$ output time intervals ${(\Delta t_{n+1}, \cdots, \Delta t_{n+L})}_{\text{train-set}}$ to denormalize the extracted time-intervals from step 1

3) Add up the denormalized $\Delta \hat t_{n+1}$ to $\Delta \hat t_{n+L}$ via:

$\hat t_{n+1} = \Delta \hat t_{n+1} + t_n$ 

$\hat t_{n+2} = \Delta \hat t_{n+2} + \hat t_{n+1}$ 

$\cdots$

$\hat t_{n+L} = \Delta \hat t_{n+L} + \hat t_{n+L-1}$ 

Please note that we use the unnormalized value of $t_n$ for the above, which is part of the input and is already given, whereas the inputs to our model for training and predicting $\hat p(t_{n+1}|H_n)$ to $\hat p(t_{n+L}|H_{n+L-1})$ are always normalized between 0 and 1, as we explained earlier in this comment.

As a final note, despite the concern regarding the distribution support being a valid concern, in our case the loss is computed still before denormalization, making sure we are not out of the distribution support. Moreover, reformulating the time associated with each sequence plays a big role in easing the denormalization concern. For example, the time interval between two events happening at hours $1.4$ and $1.6$ is $0.2$ hour which is less than $1$, whereas the same time interval is $12$ minutes and therefore is bigger than $1$. In our case, as mentioned earlier, our way of reformulating time leads to having time intervals between 0 and 1 in almost all cases (except for a few) even before normalization. Therefore, the min and max used in step 2 are safe choices for denormalizing and recovering close to original values of the extracted time associated with the test outputs.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: responses to reviewer boMt (second round)
comment: > Table 2 in the appendix suggests that the decoder is still helpful at prediction time, is there some intuition as to why this happens? According to the description in Figure 1, the input to the decoder is always just a tensor of zeros at prediction time. It seems that the effect of the input can therefore just be represented by a bias term in the model, as all inputs are the same. How can this help to more accurately predict the time and location of the future events?

Thank you for mentioning this. We have expanded the description of appendix A.3.2 regarding this matter. Also, we provide here a more detailed description, which is in line with our answer to the previous comment.

According to our answer to the previous question and as described in Section 3.2, during training the network learns to fit the distribution of multiple future events jointly as in $(p(x_{n+1}|H_{n}) , …, p(x_{n+L}|H_{n+L-1}))$. These simultaneous/batched learned distributions not only depend on history $H_{n}$ that comes from the encoder, but  also encode history dependency among themselves. This means that, for example, the distribution associated with event $x_{n+3}$ encodes dependency on events $x_{n+2}$ and $x_{n+1}$. This dependency is captured during training by the inputs to the decoder, in addition to the history dependency on events $x_{1}, …, x_{n}$, which is captured during training by the outputs of the encoder. 

Inputting events $x_{n+1}$ to $x_{n+L}$ to the decoder in the training phase, and not zero inputs, is needed to capture in-between-outputs history dependencies as we mentioned in our answer to the previous comment. This leads to tuning the decoder layer weights and biases towards estimating the spatio-temporal probability densities of several events in the future. 
However, in the test phase, we still obtain an estimation of the batched spatio-temporal probability densities despite zero-padding the input to the decoder.

This happens due to the reliance of the network on the historical input data, as well as the learned bias associated with the decoder layer. 

We also mentioned that the decoder captures short-range dependencies among the output events. Therefore, in the case of events being also short range dependent, our network highlights the importance of having a decoder, whose bias during the training phase captures that dependency.

> * [ETAS](https://www.jstor.org/stable/2288914#metadata_info_tab_contents) is the standard model used for earthquake forecasting that usually achieves very strong results in time prediction, so it should be considered as a baseline.


We have indeed used this as one of our baselines: The Hawkes model in table 1. 

Indeed equations 10, 11, and 12 in [ETAS](https://www.jstor.org/stable/2288914#metadata_info_tab_contents), introduce the same intensity function as in equation 1 in our paper. Moreover, section 2.3 in [ETAS](https://www.jstor.org/stable/2288914#metadata_info_tab_contents), mentions that the history-dependent intensity function (that was first introduced by Hawkes and Ogata) was applied to the earthquake data (proposed by the same people actually) and that is how the name changed to ETAS specific to earthquakes. However, different version of the ETAS model have been proposed by [Ogata](https://link.springer.com/article/10.1023/A:1003403601725) where in most cases we use a decaying temporal function, as we did in our baseline model.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: responses to reviewer boMt (second round)
comment: > **Part two of our response to**
> Usually, in self-supervised pre-training, we predict some parts of the input given other parts (e.g., predict $x_l$ from $x_1, \cdots, x_{l-1}$). In the proposed approach, however, the observed value of the input is provided as input to the network (i.e., we predict $x_l$ from $x_1, \cdots, x_{l}$), as shown in Figure 1. I don't understand how this training objective is theoretically justified, since the network can just use the input $x_l$ to perfectly predict the target $x_l$ during training.

In order to overcome this limitation, while maintaining a seq2seq prediction, we decided to have simultaneous/batched distribution learning. This can be implemented in TensorFlow via [TensorFlow-probability](https://www.tensorflow.org/probability/api_docs/python/tfp). In order to have a batched distribution prediction, we needed to remove the look-ahead mask layer in the decoder block of the original transformer to have a seq of events as the decoder output rather than only one event. 

Next, we also want to include the conditional dependency between events $x_{n+1}$ to $x_{n+L}$ whilst having the simultaneous prediction. As mentioned by the reviewer in their previous comment, in many cases, the output events’ occurrences are also highly dependent on each other (as short-range dependencies) which we indeed want to include in our method. If we zero-input the decoder in the training phase, no conditional dependency between $x_{n+1}$ to $x_{n+L}$ will be captured by the network and the output of the decoder will be independent hidden representations $h_{n+1}$ to $h_{n+L}$ that are only dependent on the history information $h_{1}$ to $h_{n}$ injected from the output of the encoder to the decoder. In such a case, the final batched learned distributions would be better described as $(p(x_{n+1}|H_{n}) , …, p(x_{n+L}|H_{n}))$, and $\textbf{not}$ $(p(x_{n+1}|H_{n}) , …, p(x_{n+L}|H_{n+L-1}))$. For this reason, we choose to use $x_{n+1}$ to $x_{n+L}$ as the inputs to the encoder in the training phase, so the final batched distributions are indeed modeled as possibly dependent distributions.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: responses to reviewer boMt (second round)
comment: > **Part one of our response to**
> Usually, in self-supervised pre-training, we predict some parts of the input given other parts (e.g., predict $x_l$ from $x_1, \cdots, x_{l-1}$). In the proposed approach, however, the observed value of the input is provided as input to the network (i.e., we predict $x_l$ from $x_1, \cdots, x_{l}$), as shown in Figure 1. I don't understand how this training objective is theoretically justified, since the network can just use the input $x_l$ to perfectly predict the target $x_l$ during training.

Thank you for your comment. We understand that our use of the decoder block can cause some confusion. Hence, here we first provide a quick overlook of the conventional transformer architecture, and then we detail how our proposed architecture deviates from it and we also justify this modification.

The conventional seq2seq nature of the transformer architecture as a self-supervised learning process relies on using the ground-truth $x_{n+1}$ to $x_{n+L}$ only when minimizing the loss during training via a sequential $x_{n+1}$ to $x_{n+L}$ forecasting. In that case, $x_{n+1}$ to $x_{n+L}$ is not input to the decoder during training (unlike our proposed architecture). More precisely, in the conventional transformer architecture, we use $x_1$ to $x_n$ to predict $\hat x_{n+1}$ and then use the ground truth $x_{n+1}$ to minimize the loss associated with predicting $x_{n+1}$. Before going to the next batch, still in the same sequence during training, we assume that we have correctly predicted $x_{n+1}$ (using the ground truth $x_{n+1}$). Therefore we use the encoder output as well as the ground-truth $x_{n+1}$ to predict $\hat x_{n+2}$ and use the ground truth $x_{n+2}$ to minimize the loss. This all goes on till we reach $x_{n+L}$.  

As it follows from the above description, the decoder’s output in the conventional transformer architecture is what we want to predict. However, in our proposed architecture, the purpose is different. We do not aim to predict $x_{n+1}$ to $x_{n+L}$. Instead, our goal is to predict simultaneous distributions associated with events $x_{n+1}$ to $x_{n+L}$. 

One potential way of achieving this (which might be what the reviewer is implying) is to never look at the ground truth $x_{n+1}$ to $x_{n+L}$ while training (as in the conventional transformer architecture) and do the following process:

Use $x_1$ to $x_n$ as inputs to the encoder and extract the hidden representations $h_1$ to $h_n$ as inputs to the decoder to predict the first hidden representation $h_{n+1}$ associated with event $x_{n+1}$. Input $h_{n+1}$ to the probabilistic layer, followed by the normalizing flow to output $\hat p(x_{n+1}|H_n)$ which is the learned conditional distribution associated with event $x_{n+1}$. Next use negative log-likelihood to minimize the loss associated with modeling this distribution ($p(x_{n+1}|H_n)$).

In order to model the conditional distribution associated with event $x_{n+2}$, we have to have access to the true sample $x_{n+1}$, which during the training phase, we could still use the ground truth sample $x_{n+1}$ to model $\hat p(x_{n+2}|H_{n+1})$. $\textbf{BUT}$ in the test phase we have to **sample** from the predicted $\hat p(x_{n+1}|H_n)$ to further learn/predict $\hat p(x_{n+2}|H_{n+1})$, **sample** from $\hat p(x_{n+2}|H_{n+1})$ to learn/predict $\hat p(x_{n+3}|H_{n+2})$ till we reach learning/predicting $\hat p(x_{n+L}|H_{n+L-1})$ which is $\textbf{HIGHLY}$  prone to error accumulation, as now we need the extracted samples to be perfectly close to the original samples. This is very similar to the fourth limitation we mentioned in our paper associated with other proposed networks when performing sequential prediction in a probabilistic setting.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer yeaA
comment: Please note that your concerns have been addressed in our second submitted version as follows:

The second limitation associated with previous works is rewritten on page 1 section 1 for better clarification.

We expanded our methodology in Appendix A.1 covering the mathematical description of our proposed probabilistic layers/model.

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_boMt
title: Reply to the rebuttal
comment: Thank you for the response! It answered some of my questions but, unfortunately, my main concerns hasn't been addressed: Usually, in self-supervised pre-training, we predict some parts of the input given other parts (e.g., predict $x_l$ from $x_1, ..., x_{l-1}$). In the proposed approach, however, the observed value of the input is provided as input to the network (i.e., we predict $x_l$ from $x_1, ..., x_{l-1}, x_l$), as shown in Figure 1. I don't understand how this training objective is theoretically justified, since the network can just use the input $x_l$ to perfectly predict the target $x_l$ during training.

Table 2 in the appendix suggests that the decoder is still helpful at prediction time, is there some intuition as to why this happens? According to the description in Figure 1, the input to the decoder is always just a tensor of zeros at prediction time. It seems that the effect of the input can therefore just be represented by a bias term in the model, as all inputs are the same. How can this help to more accurately predict the time and location of the future events?

There are some other concerns that I still have after the rebuttal:
- [ETAS](https://www.jstor.org/stable/2288914) is the standard model used for earthquake forecasting that usually achieves very strong results in time prediction, so it should be considered as a baseline.
- The described procedure, where all inter-event times are normalized to be between 0 and 1, may introduce information leakage from the test set, if the normalization is done before the train/test split. If normalization is done only using the train set, then the test set may contain an inter-event time > 1 outside of the distribution support, so the model will assign $-\infty$ log-likelihood to this event.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer q56W
comment: > **Weakness**
> * The baselines in experiments consider only Hawkes alternatives. I would encourage authors to at least present a comparison using other methods in SI.

Thank you very much for reviewing our work. We appreciate your note on comparing our method with others. However, we would like to mention that our work is the first one proposing simultaneous spatio-temporal multi-event forecasting in the field of studying point processes. All the papers cited in our work, are aimed at learning/ modeling an intensity function associated with the Hawkes process. Notice that none of these methods were developed to perform simultaneous multi-event forecasting. Hence, in such a setting, we think that comparing with simple baselines is acceptable.

That being said, please note that originally we wanted to implement the method in two other papers whose code for learning the intensity function (but no multi-event forecasting) was publicly available. We aimed at using the first-order moment for sequential event forecasting using their learned model. However, we faced several challenges in trying to do so:

1) A direct implementation of the work [Zhou et al](https://proceedings.mlr.press/v168/zhou22a.html) using the provided code led to a diverging loss during training for our dataset. Our guess for this behavior is that working with long sequences of data and higher dimensions demands computing more complex history dependency using their tool. More precisely, in their code, they showcase experiments on only 2D datasets (and not 3D such as in our case) and only for sequences of length 20 or less (while we look at lengths closer to 500).

2) The work of [Chen et al.](https://arxiv.org/abs/2011.04583)  proposed two main methods called the AttentiveCNF and JumpCNF given in a neural ODE structure to learn the spatiotemporal intensity function associated with the Hawkes process (namely ETAS in the earthquake domain). We applied both methods to our data and with all respect to the authors, despite a converging loss and fast training, the next event predicted by their AttentiveCNF using the first-order moment was not representative of the data to be predicted.
Their JumpCNF had some interesting visualizations for the prediction on only some examples for the next-event forecasting. However, due to having an autoregressive normalizing flow structure in their proposed ODE-CNF method, the sequential training for such big sequence lengths (497 in our case) was computationally very expensive as one needs to solve the ODE for every single event to go to the next event. Therefore, sequential forecasting (mentioned as the fourth limitation in sec(1) of our paper) demanded very huge computational resources and time since every time a new event is predicted, the whole training process had to start from the beginning with the updated sequence, including the already predicted event(s).
Their method (similar to the baselines we used in our paper) is proposed to learn the intensity function of a Hawkes process. However, they did not provide any numerical results regarding the next-event prediction log-likelihood and did not provide the code for that matter which we could use for the sake of comparison. The results in table(1) of our paper depict the log-likelihood associated with the predicted multi-events whereas theirs is solely based on fitting the learned intensity function to the historical data.

3) Many other papers that we cite do not provide any code we could use.

That all being said, we truly thank the reviewer for their positive feedback on different aspects of our paper.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer boMt
comment: > 3) Lack of ablations: The proposed model consists of multiple components, but no experiments are performed to confirm that each of these components is actually beneficial. To demonstrate that the transformer architecture leads to a meaningful improvement, it would be necessary to compare with:
> * $p(x_l)$ and $p(t_l)$ completely independent of the history, but parametrized exactly like in the paper (i.e., exponential + softsign for $t_l$, gaussian + real NVP for $x_l$)
> * Encoder only (without feeding $t_l$, $x_l$ ,$M_l$  to the decoder, i.e. just mapping $h_{t_l}$ and $h_{x_l}$ to the parameters of $p(t_l)$ and $p(x_l)$ ). Intuitively, it seems that this option will work just as well as the model with the decoder since in the full model the decoder anyway doesn't receive any input at prediction time anyway.

Thank you very much for your helpful insight on this matter. This will definitely help with improving the quality of the paper. Your comment is well-taken and currently, we are running the requested ablation tests. The results will be added to the appendix and further details on the interpretations will be given accordingly. In a later comment, we will provide more details as we get the new results. 

> **Minor comments**
> * The assumption that the future events are conditionally independent given the history (as defined in Problem 1) seems a bit restrictive: for example, if the next earthquake has a large magnitude, then the follow-up events will happen in quick succession. Lifting the conditional independence assumption could be an interesting avenue to explore.

Thank you for mentioning this point. Notice that the statement of Problem 1 does not imply conditional independence among the events to be predicted. Indeed, in our method, the first attention layer in the decoder seeks to capture this dependency.

> * Potentially missing references
> 1) Neural spatio-temporal point processes: https://arxiv.org/abs/1906.05467,
> 2) Modeling TPPs with normalizing flows: https://arxiv.org/abs/1910.08281 and https://arxiv.org/abs/1909.12127

 Thank you. We will add these references.

> **Summary Of The Review**:
> Several important aspects of the proposed approach need to be described more clearly. A more thorough comparison to the baselines and an ablation study would be really beneficial to justify the utility of the proposed model.

We thank the reviewer for carefully reading our manuscript and providing many useful points/comments that we did not carefully address in our first version of the manuscript. However, we will make sure that all these details are added and addressed more clearly in our revised version of the paper. Moreover, we will add new ablation studies and a better justification of the utility of all the blocks we used in developing our network. We hope that the answers here (which accordingly will be added to the paper) as well as the revised version will alleviate the reviewer's concerns.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer boMt
comment: > 2) Baselines: Currently, the experiments in section 4 only include very simple baseline models (homogenous Poisson, self-exciting/self-correcting processes for time & gaussian mixture for locations). There exist other approaches that definitely could be compared against:
> * Neural Spatio-temporal point process model from Chen et al. (https://arxiv.org/abs/2011.04583)

Thank you for bringing this up. Originally, we wanted to implement the method in this paper for our problem as well. However, we faced several challenges in trying to do so. Namely:

1) They have proposed two main methods called the AttentiveCNF and JumpCNF given in a neural ODE structure to learn the spatiotemporal intensity function associated with the Hawkes process (namely ETAS in the earthquake domain). We applied both methods to our data and with all respect to the authors, despite a converging loss and fast training, the next event predicted by their AttentiveCNF using the first-order moment was not representative of the data to be predicted.

2) Their JumpCNF had some interesting visualizations on only some examples for the next-event forecasting. However, due to having an autoregressive normalizing flow structure in their proposed ODE-CNF method, the sequential training for such big sequence lengths (497 in our case) was computationally very expensive as one needs to solve the ode for every single event to go to the next event. Therefore, sequential forecasting (mentioned as the fourth limitation in sec(1) of our paper) demanded very huge computational resources and time since every time a new event is predicted, the whole training process had to start from the beginning with the updated sequence, including the already predicted event(s).

3) Their method (similar to the baselines we used in our paper) is proposed to learn the intensity function of a Hawkes process. However, they did not provide any numerical results regarding the next-event prediction log-likelihood and did not provide the code for that matter which we could use for the sake of comparison. The results in table(1) of our paper depict the log-likelihood associated with the predicted multi-events whereas theirs is solely based on fitting the learned intensity function to the historical data.

> * Other neural TPP models for predicting the event time (many of which are cited in the paper)

Our original intention was to compare our method with the one developed by [Zhou et al](https://proceedings.mlr.press/v168/zhou22a.html). However, a direct implementation of the code provided in the paper led to a diverging loss during training for our dataset. Our guess for this behavior is that working with long sequences of data and higher dimensions demands computing more complex history dependency using their tool. More precisely, in their code, they showcase experiments on only 2D datasets (and not 3D such as in our case) and only for sequences of length 20 or less (while we look at lengths closer to 500).

Many other papers that we cite do not provide any code we could use. In any case, notice that none of these methods were developed to perform simultaneous multi-event forecasting. Hence, in such a setting, we think that comparing with simple baselines is acceptable.


> Figure 1 also suggests that the proposed model takes into account the magnitudes $M_l$ of the events, but it seems that none of the baselines have access to this information. If the magnitudes are indeed taken into account, it's important to also compare to the ETAS model from seismology (using the power-law kernel and, very importantly, taking into account the contribution of the magnitudes on productivity)

This is a good point. We have actually taken into account the magnitude values as added information (other markers) in the conditional GMM model used as a baseline for the location which could be found in appendix A.2 with more details. More precisely, Having a mixture of k Gaussian densities the parameters that we aim to use in the conditional GMM (appendix A.2) are conditionally learned from the historical events' markers as shown in eq(11). We will add to the description for a more clear understanding of the spacial baseline implementations. However, the magnitude is not used in the baselines for the time distribution. We will certainly take this into account for future studies. Please note that all the code and implementations are provided and will be updated accordingly.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: responses to reviewer boMt
comment: Thank you for reviewing our paper and for the detailed comments below.

> **Weaknesses**
> 1) My main concern is the clarity regarding the proposed model architecture & the training procedure. Here are some questions that need to be clarified in the updated version: 
> * During training the decoder takes the observed $(t_l,{\mathbf{x}}_l) $ as input, and tries to predict the exact same values  $(t_l,{\mathbf{x}}_l) $ as output. Intuitively, it seems that the decoder should learn to perfectly predict these values / the log-likelihood should tend to infinity as the model trains. Why is this not the case?

Thank you for pointing this out. Notice that we are learning a parametric description of a probability distribution that seeks to explain future events. It is true that during the self-supervised training, the portions of the sequence to be predicted are revealed to the architecture. Nonetheless, since i) we are doing self-supervision with a whole batch of sequences for which the same parametric form is learned, and ii) we are learning a probability distribution over the output space rather than point estimates, it is natural that the learned probabilities will not be able to perfectly concentrate around the true observed events. This is why the log-likelihood does not grow indefinitely during training.

> It is mentioned that at test time the inputs to the decoder are replaced with all zeros.
> * In this case, how can the decoder distinguish between different inputs? Is some positional encoding used?

We have used the same positional encoding used in the original paper introducing transformers in NLP ([Attention is all you need](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)). However, notice that the decoder is also fed by the output from the encoder, which is computed during testing using the true history that we use for prediction. Consequently, the decoder can distinguish between different event histories via the output of the decoder.


> * If $t_l = 0$ is fed into the model, this means that the distribution ${p({\mathbf{x}}_l| t_l)\}$ will effectively become independent of $t_l$. What is the purpose of modeling the conditional distribution at training time, if at test time the time and location are always independent?

This is mainly a choice for the evaluation of the method but the learned probability distributions can be used in practice to jointly predict both time and location by first drawing a time $\hat t_l$ from $p(t_l|H_{t_l})$ and then drawing a location from $p({\mathbf{x}} |\hat t_l, H_{t_l}) $ by fixing and using the time obtained in the first draw. Since our baselines are for either time prediction or location prediction, we decided to compare it with our method used for time prediction $p(t_l|H_{t_l})$ mentioned above or predicting the location of the next event, assuming that the time is known $p({\mathbf{x}} |t_{l_{\text{known}}}, H_{t_l}) $. In this way, the error is comparable with those that predict solely location since we are controlling for the error that could be introduced by a wrong time prediction $\hat t_l$.  Thus, the answer is we do not feed $t_l = 0$ into the model while modeling $p({\mathbf{x}} |t_{l}, H_{t_l}) $ but we feed $t_{l_{\text{known}}}$ instead.

> * The softsign function $f(z) = \frac{z}{|z| +1}$ has domain/codomain $f : (-\infty, \infty) \rightarrow (-1,1)$ . Restricting to positive values only (as $z_{t_l}$ comes from the exponential distribution), we get a function $f : (0, \infty) \rightarrow (0,1)$. However, we need both the domain and the codomain here to be $(0, \infty)$ since we're modeling the inter-event times or the arrival times. In other words, the used function is not a bijector so it does not define a valid normalizing flow transformation. Is there a mistake in my reasoning? As a side note, the softsign function has no learnable parameters, so it's unclear how exactly it increases the flexibility of the distribution $p_{t_l}$.

This is a very interesting point and thanks for mentioning it. We always normalize our time inputs to stay between 0 and 1 and location inputs are also normalized based on the mean and variance of the training dataset. Time normalization will cause the time-intervals to always stay between 0 and 1. Therefore, this map will ensure that the outputs from the exponential probabilistic layer will stay in our desired range. As mentioned in [Tensorflow-probability](https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors/Softsign) , a function can be used as a bijective layer even if it has no learnable parameters. Finally, as shown in the time-interval histograms associated with all datasets in fig(11), all follow an approximate exponential behavior meaning that we do not need very much flexibility and only have to bring the time-interval to the desired range through normalization.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer mvAP
comment: > **Clarity, Quality, Novelty And Reproducibility:**
> Overall this paper is well-written and sticks to the major theme of tackling the four challenges of existing point process modeling/prediction methods.
>I have some questions: (1) the intensity function \lambda in eq(1) does not include the marker M. Does this mean that the prediction is only for time t and location x of the next events, while not their markers M? However, in the proposed architecture shown in Fig1, the marker M is used in the self-supervised training phase. 
 
You are right in your assessment. The markers (magnitude) are used as input to the prediction (i.e., they are part of the history) but we are not estimating the magnitude of future events (just their time and location). This is consistent with the common use of Hawkes processes in seismology, where the time and location of the next event can depend on the time, location, and magnitude of historical events, but the magnitude of the new event is estimated using different strategies (For example in the case of earthquake data we need to rely on other physics laws such as the Gutenberg-Richter (GR) frequency-magnitude law to model the magnitude distribution [GR-Law](https://authors.library.caltech.edu/47734/1/185.full.pdf), [The magnitude distribution of declustered earthquakes in Southern California](https://www.pnas.org/doi/abs/10.1073/pnas.190241297), [Effects of magnitude uncertainties](https://pubs.geoscienceworld.org/ssa/bssa/article/75/6/1681/118775/Effects-of-magnitude-uncertainties-on-estimating?casa_token=FIz4zfVWJc4AAAAA:nx-qefQ9CEohXOVG39AVC40Dj7Jd-y6f-ol4daJ_-C-kKFe3EVXE3rhikU335GHwcL5pjmU), [Estimation of the Gutenberg-Richter relation allowing for individual earthquake magnitude uncertainties](https://www.sciencedirect.com/science/article/abs/pii/0040195195001824), etc.) The Hawkes model (namely ETAS for earthquakes) which to some level our developed network inherits from, is used to model/learn the intensity function associated with time and space of the events while other markers can as well be included as the historical information shown in eq(1) in our paper. Although it might be confusing, please notice that this is not in contradiction with Figure 1. We indeed use the history of the output (including the magnitudes) for the self-supervised part of our training, but during testing, the “batch of outputs” is fed with all zeros (as explained in the original submission) and the neural architecture only outputs probability distributions for the time and location of the next events.


>(2) in the numerical results Table 1, the author compared the negative log-likelihood for the predicted events, I am wondering how are these likelihoods computed for real-data examples (since we do not know the true intensities, and the true data-generating model is also different from Hawkes most likely).
 
Since the methods output a probability distribution for the time and location of the next events, what we compute is the log-likelihoods of the real events under the probability distributions output by the different methods. The smaller the negative log-likelihoods, the larger the probability of observing what actually happened given the learned probability distributions.
 
> **Summary Of The Review:**
> Overall this paper is well-written. This work proposes to combine transformer and normalizing flows to predict future multi-events under the spatio-temporal Hawkes models. Numerical results are provided to demonstrate the good performance in real data examples. However, considering that both transformer and normalizing flows are not new and they have been already been used in point processes, the contribution of this work seems incremental. Moreover, more clarification on numerical results is also needed.
 
We thank you for your review and your assessment. In our view, novelty in deep learning does not always entail proposing a new block or a new type of layer, but also proposing methodologies that combine existing blocks in a new and principled manner to study a problem of interest. We believe that our contribution falls under this second category. Moreover, the topic of simultaneous prediction of point process data was not studied before. Also, we hope to have clarified the confusion regarding the log-likelihood computation in the numerical experiments.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to Reviewer mvAP
comment: We would like to thank the reviewer for reading our manuscript and for their feedback. 
> **Weaknesses**:
> * the contribution is incremental, both the transformer and normalizing flows have been used for point processes in the literature, e.g, 
>  Mehrasa, N., Deng, R., Ahmed, M. O., Chang, B., He, J., Durand, T., ... & Mori, G. (2019). Point process flows. arXiv preprint arXiv:1910.08281.
 
Our main contribution does not lie in proposing a new constituent piece of a neural network but rather in the overall design and performance of the neural architecture, the way it is used for multi-event forecasting, and the specific application domain. 
 
We understand that both transformers (mainly the encoder part) and normalizing flows have been used in developing neural networks, and we have tried to mention all such previous works. Moreover, we state the distinctions between those works on ours. The reviewer points out another reference [1], which we now include in our modified version. With respect to this work, our main differences are as follows:

* Our tool is performing a Spatio-temporal multi-event forecasting whereas the provided paper is built upon only events' time and no space distribution prediction is provided

* We use an attention mechanism in our encoder-decoder blocks when encountering history dependency which provides informative scores regarding the historical events affecting the occurrence of current/future events, whereas their method only uses a VAE encoder-decoder to map all the historical information into a latent space that is formed by a Gaussian distribution. Therefore, all history is treated the same with no emphasis on the importance of which historical events have more influence. 

* We distinguish the space and time distribution in the latent z space indicated in our paper. We provide histogram visualizations in the appendix as the logical choices of our probabilistic layers, whereas the provided paper maps all historical events' timings to a Gaussian distribution using a VAE encoder.

* Where both our and their latent space are followed by normalizing flow layers, our architecture, due to being seq2seq is capable of simultaneous multi-event forecasting, whereas theirs is only modeling/learning the distribution associated with the next event, and in order to perform multi-event prediction, sequential estimation of the expected values is needed as the fourth limitation mentioned in section(1) of our paper.

Compared to previously proposed methods, our approach does not require sequential estimation of expected values to perform multi-event forecasting. To the best of our knowledge, this is the first work combining transformers (including the decoder block) and probabilistic layers augmented with normalizing flows to perform simultaneous spatiotemporal multi-event forecasting. Most previous works have proposed learning the Hawkes process intensity function using different deep learning blocks, while some of them claim that this could be used for the next event prediction (but with no numerical or visualization illustrations). On the other hand, our methodology is built to perform multi-event forecasting directly, and the rationale behind all the specific blocks used in our architecture is explained in Sections 3.1 and 3.2
 
>* the numerical experiments are not very clearly explained, and more clarification is needed
 
We thank the reviewer for this comment. However, we are not sure what might be unclear about the experiments as this was not a common concern among all the other reviewers. Below we reply to your concern about the clarity around the definition of the log-likelihood in the experiments. If there are other additional points that you think are still unclear in the numerical experiments, we would be happy to expand on the corresponding descriptions in the revised paper.

From: ICLR.cc/2023/Conference/Paper4975/Authors
title: Responses to reviewer yeaA
comment: Thank you for reviewing our manuscript. We hope the clarification below answers your questions and would be happy to provide further elaboration in case there are any remaining concerns.

> **Weakness**
> * Some of the claims in the introduction seem misleading. The classical Hawkes process has a decay function, which introduces a significant distinction over the event instances. I'm not sure if "(1) imposes similar behavior across all events over time meaning that every preceding event has the same form of impact on future events' occurrence" is a scientifically correct statement.

We acknowledge that the choice of words in our original submission was confusing. Our intended meaning is the following. In the classical Hawkes process, when we fit the rate of the decaying function (for example $f(t-t_i) = \frac{1}{\beta} \exp(-\frac{(t-t_i)}{\beta})$ to the data (meaning $\beta$ is fitted), the history dependency is only a matter of time-difference between two events projected on that decaying rate and the learned parameter. In this case, $\beta$ is fitted and fixed and the same $\beta$ is applied to all time differences. In the case of our network, the score assigned to time differences is variable and not fixed like $\beta$. Therefore, despite using an exponential probability distribution after the decoder, we assign scores to the time differences $(t-t_i)$ using the attention mechanism in the encoder, meaning now we are feeding a weighted time-difference (for example $\alpha_i (t-t_i))$ to the decoder block, where $\alpha_i$ is a weight vector indicating the importance of all previous times on the occurrence of the current/future time. We have modified the wording to clarify our intended message.



> * The proposed probabilistic models are not clearly defined. Most of the descriptions are very high-level and qualitative. As a result, it is hard to verify methodological novelty and reproduce/extend their results.

It is true that we have focused on providing a high-level description of the whole architecture and a rationale for the choice of the different constituent blocks as opposed to providing detailed descriptions of the inner workings of each block. Our main reason for doing this is that we consider that the main value is in the overall architecture rather than in its pieces, which have been introduced in the literature before. Also notice that, in terms of reproducing the results, we have provided code that contains all the necessary details to reproduce and extend our results.

That being said, the point is well-taken and we have expanded the description of our methodology. In particular, we now include in the appendix a more clear description of the probabilistic models/layers we proposed to use in our developed network architecture.

> **Clarity, Quality, Novelty And Reproducibility:**
> The text is overall well-written. The authors seem to have made much effort for clearer descriptions. However, in contrast, there are very limited formal descriptions on the generative process, which should be the key in verifying their technical contributions. As a result, it is virtually not possible to reproduce their model beyond the specific implementation.

Please see our response to your second weakness listed above.


> **Summary Of The Review:**
>This paper is unique in its sharp contrast between readable narratives and hard-to-follow descriptions of the probabilistic model. >Because of the very definition of the intensity function (the probability density of the next event), multi-event prediction generally requires Monte Carlo integration. The situation is similar to that of multi-ahead prediction in forecasting. See, e.g., https://ieeexplore.ieee.org/document/8594894
>The proposed framework seems to have solved that issue by introducing the normalized flow approach combined with the seq2seq transformer model. I wish I could give a much better rating, but it is hard for me to endorse the approach due to the massive lack of formal descriptions.

Thank you for your time and effort in reviewing our manuscript. As stated in the responses above, we have expanded the formal description of our methodology via additions to the appendix while keeping the streamlined high-level description of the method in the main body of the text. We hope that this balance in addition to the code availability will improve the reproducibility concern of the reviewer.

> **Correctness**: 2: Several of the paper’s claims are incorrect or not well-supported.

We would like to point to our answer to the first weakness mentioned by the reviewer to remove any ambiguity in that specific statement. If there are other statements that the reviewer deems incorrect or imprecise we would be happy to address those as well.

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_q56W
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Paper presents a transformer-based architecture to handle multi-event forecasting in spatio-temporal domains. The proposed architecture allows for multiple events forecast by formulating problem as a self-supervised problem. 
strength_and_weaknesses: Strengths
* Architecture proposed is quite novel in sense of being applied to multiple event prediction task
* All different parts of the model are well motivated and solve a purpose instead of a giant network being embedded for sake of doing deep learning. 
* Problem being attacked is important and has been something people in filed have mostly sidelined and been happy with doing sequential short predcitions. 
* well written and easy to follow paper. 

Weakness
* The baselines in experiments consider only Hawkes alternatives. I would encourage authors to at least present a comparison using other methods in SI. 
 
clarity,_quality,_novelty_and_reproducibility: I would rate this work very high on clarity, quality and novelty. For reproducibility, I found the code to be of good quality and fairly easy to follow. I hope authors will document the code a bit more once paper is accepted.  
summary_of_the_review: In general, the model and problem proposed in the paper are quite novel and of great importance. Authors have done a great job of explaining each individual parts of the model and its specific utility in solving the drawbacks from the previous work. Hence, I think this work deserves to be published. 
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_boMt
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies the problem of simultaneously predicting times and location of *multiple* (fixed number of) future events with a neural spatio-temporal point process model.
The proposed model is based on an encoder-decoder transformer architecture, and the distribution of the future events is modeled by 
strength_and_weaknesses: **Strengths:**
- The paper introduces an interesting variation of the prediction problem for spatio-temporal event data, where the goal is to predict multiple events simultaneously. A thorough analysis of the limitations of existing approaches that could be used in this task is provided. This is an important task and it would be important for practical applications of spatio-temporal point process models.
- The visualizations in section 4 are well done and provide an insight into the model's performance.

**Weaknesses:**

1. My main concern is the clarity regarding the proposed model architecture & the training procedure. Here are some questions that need to be clarified in the updated version:
    - During training the decoder takes the observed $\\{(t_l, x_l)\\}_\{l=n+1\}^\{n+L\}$ as input, and tries to predict the exact same values  $\\{(t_l, x_l)\\}_\{l=n+1\}^\{n+L\}$  as output. Intuitively, it seems that the decoder should learn to perfectly predict these values / the log-likelihood should tend to infinity as the model trains. Why is this not the case?
    - It is mentioned that at test time the inputs to the decoder are replaced with all zeros.
        - In this case, how can the decoder distinguish between different inputs? Is some positional encoding used?
        - If $t_l = 0$ is fed into the model, this means that the distribution $p(x_l | t_l)$ will effectively become independent of $t_l$. What is the purpose of modeling the conditional distribution at training time, if at test time the time and location are always independent?
    -  The softsign function $f(z) = \frac{z}{|z| + 1}$ has domain/codomain $f: (-\infty, \infty) \to (-1, 1)$. Restricting to positive values only (as $z_{t_l}$ comes from the exponential distribution), we get a function $f: (0, \infty) \to (0, 1)$. However, we need both the domain and the codomain here to be $(0, \infty)$ since we're modeling the inter-event times or the arrival times. In other words, the used function is not a bijector so it does not define a valid normalizing flow transformation. Is there a mistake in my reasoning?
        - As a side note, the softsign function has no learnable parameters, so it's unclear how exactly it increases the flexibility of the distribution $p(t_l)$.

2. Baselines: Currently, the experiments in section 4 only include very simple baseline models (homogenous Poisson, self-exciting/self-correcting processes for time & gaussian mixture for locations). There exist other approaches that definitely could be compared against:

    - Neural spatio-temporal point process model from Chen et al. (https://arxiv.org/abs/2011.04583)
    - Other neural TPP models for predicting the event time (many of which are cited in the paper)

    Figure 1 also suggests that the proposed model takes into account the magnitudes $M_l$ of the events, but it seems that none of the baselines have access to this information. If the magnitudes are indeed taken into account, it's important to also compare to the [ETAS model](https://www.ism.ac.jp/editsec/aism/pdf/050_2_0379.pdf) from seismology (using the power-law kernel and, very importantly, taking into account the contribution of the magnitudes on productivity)

3. Lack of ablations: The proposed model consists of multiple components, but no experiments are performed to confirm that each of these components is actually beneficial. To demonstrate that the transformer architecture leads to a meaningful improvement, it would be necessary to compare with:
    - $p(x_l)$ and $p(t)$ completely independent of the history, but parametrized exactly like in the paper (i.e., exponential + softsign for $t_l$, gaussian + real NVP for $x_l$)
    - Encoder only (without feeding $t_l$, $x_l$, $M_l$ to the decoder, i.e. just mapping $h_{t_i}$ and $h_{x_i}$ to the parameters of $p(t_l)$ and $p(x_l)$). Intuitively, it seems that this option will work just as well as the model with the decoder, since in the full model the decoder anyway doesn't receive any input at prediction time anyway.


**Minor comments**
- The assumption that the future events are conditionally independent given the history (as defined in Problem 1) seems a bit restrictive: for example, if the next earthquake has a large magnitude, then the follow-up events will happen in quick succession. Lifting the conditional independence assumption could be an interesting avenue to explore.
- Potentially missing references
    - Neural spatio-temporal point processes: https://arxiv.org/abs/1906.05467,
    - Modeling TPPs with normalizing flows: https://arxiv.org/abs/1910.08281 and https://arxiv.org/abs/1909.12127
clarity,_quality,_novelty_and_reproducibility: See main review.
summary_of_the_review: Several important aspects of the proposed approach need to be described more clearly. A more thorough comparison to the baselines and an ablation study would be really beneficial to justify the utility of the proposed model.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_mvAP
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: 

This paper proposes a neural architecture, combining the transformer model and normalizing flows, for multi-event forecasting of the time and location of discrete events. A variety of real spatio-temporal datasets are used to validate the state-of-the-art performance of the proposed method.

strength_and_weaknesses: Strength:
- A new architecture consists of transformer and normalizing flows, to capture the spatio-temporal distribution of multiple stochastic discrete events jointly, conditioning on historical events.
- extensive numerical experiments to demonstrate the performance of the proposed method.

Weaknesses:
- the contribution is incremental, both the transformer and normalizing flows have been used for point processes in the literature, e.g, [1] 
- the numerical experiments are not very clearly explained, and more clarification is needed

[1] Mehrasa, N., Deng, R., Ahmed, M. O., Chang, B., He, J., Durand, T., ... & Mori, G. (2019). Point process flows. arXiv preprint arXiv:1910.08281.
clarity,_quality,_novelty_and_reproducibility: Overall this paper is well-written and sticks to the major theme of tackling the four challenges of existing point process modeling/prediction methods.

I have some questions:
(1) the intensity function \lambda in eq(1) does not include the marker M. Does this mean that the prediction is only for time t and location x of the next events, while not their markers M? However, in the proposed architecture shown in Fig1, the marker M is used in the self-supervised training phase.
(2) in the numerical results Table 1, the author compared the negative log-likelihood for the predicted events, I am wondering how are these likelihoods computed for real-data examples (since we do not know the true intensities, and the true data-generating model is also different from Hawkes most likely). 

summary_of_the_review: Overall this paper is well-written. This work proposes to combine transformer and normalizing flows to predict future multi-events under the spatio-temporal Hawkes models. Numerical results are provided to demonstrate the good performance in real data examples. However, considering that both transformer and normalizing flows are not new and they have been already been used in point processes, the contribution of this work seems incremental. Moreover, more clarification on numerical results is also needed.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4975/Reviewer_yeaA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper leverages the sequence-to-sequence (seq2seq) model in NLP for multi-event prediction in the Hawkes process. The authors call the task 
the batched prediction. The authors employ
the encoder-decoder architecture that incorporates spatial-temporal aspects. They leverage the transformer architecture, where two embedding vectors capture the temporal event history and spatial dependency. They seem to have used an interesting normalization flow model to handle the temporal development of probability distribution.
strength_and_weaknesses: **Strengths**

- Proposes a new multi-event prediction framework. 
- Seems to have address the probabilistic nature (i.e., properties that require Monte Carlo or its variant) of multi-event prediction.
- Incorporated a new normalized flow approach.
 

**Weakness**

- Some of the claims in the introduction seem misleading. The classical Hawkes process has a decay function, which introduces a significant distinction over the event instances. I'm not sure if "(1) imposes similar behavior across all events over time meaning that every preceding event has the same form of impact on future events' occurrence" is a scientifically correct statement.

-  The proposed probabilistic models are not clearly defined. Most of the descriptions are very high-level and qualitative. As a result, it is hard to verify methodological novelty and reproduce/extend their results. 
clarity,_quality,_novelty_and_reproducibility: The text is overall well-written. The authors seem to have made much effort for clearer descriptions. However, in contrast, there are very limited formal descriptions on the generative process, which should be the key in verifying their technical contributions. As a result, it is virtually not possible to reproduce their model beyond the specific implementation.
summary_of_the_review: This paper is unique in its sharp contrast between readable narratives and hard-to-follow descriptions of the probabilistic model. Because of the very definition of the intensity function (the probability density of the next event), multi-event prediction generally requires Monte Carlo integration. The situation is similar to that of multi-ahead prediction in forecasting. See, e.g., https://ieeexplore.ieee.org/document/8594894 

The proposed framework seems to have solved that issue by introducing the normalized flow approach combined with the seq2seq transformer model. I wish I could give a much better rating, but it is hard for me to endorse the approach due to the massive lack of formal descriptions.  

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold