Paper: XTSFormer: Cross-Temporal-Scale Transformer for Irregular Time Event Prediction (ICLR-2024)

Reviewer comment (Novelty):
The proposed FCPE seems to be Mercer time embedding [1] parameterized by event type. It would be great to present the comparison with Mercer time embedding.

Author response:
Thank you for bringing up this related work on Mercer Time Embedding [1]. This is a great insight. As suggested, we have added experimental comparisons.

We replaced the FCPE in XTSFormer with Mercer Time Embedding and conducted experiments on four datasets. The results in the following table show that our method has slightly better performance, which may be due to the event type feature-based encoding. 

|        | Medications |      | Providers |      | Financial |      | StackOverflow |      |
|:------:|:-----------:|:----:|:---------:|:----:|:---------:|:----:|:-------------:|:----:|
|        |   Accuracy (%)  | RMSE |  Accuracy (%) | RMSE |  Accuracy (%) | RMSE |    Accuracy (%)   | RMSE |
| Mercer |     31.5    | 1.45 |    41.2   | 2.49 |    76.9   | 0.87 |      46.3     | 4.38 |
|  FCPE  |     33.5    | 1.12 |    43.9   | 2.33 |    77.5   | 0.88 |      49.4     | 4.21 |


[1] Self-attention with Functional Time Representation Learning, NeurIPS 2019.

Reviewer comment (Experiments):
The ablation studies on the NLL/RMSE with FCPE, attention, and weibull distribution should be presented.

Author response:
This is a great point. We have conducted comprehensive ablation studies to fill in the missing details as below, and we will include these studies in our revised manuscript.

### The NLL of ablation study results:

|  PE  | Multi-scale | Distribution | Meications | Providers | Financial | StackOverflow |
|:----:|:-----------:|:------------:|:----------:|:---------:|:---------:|:-------------:|
| base |     w/o     |  Exponential |    3.48    |    2.21   |    2.10   |      2.88     |
| FCPE |     w/o     |  Exponential |    2.99    |    2.15   |    2.07   |      2.41     |
| base |      w/     |  Exponential |    2.47    |    1.92   |    1.60   |      2.42     |
| FCPE |      w/     |  Exponential |    2.20    |    1.81   |    1.17   |      2.10     |
| base |     w/o     |   Weilbull   |    2.59    |    2.30   |    2.35   |      2.31     |
| FCPE |     w/o     |   Weilbull   |    2.34    |    2.10   |    2.11   |      2.09     |
| base |      w/     |   Weilbull   |    2.31    |    1.89   |    1.59   |      2.02     |
| FCPE |      w/     |   Weilbull   |    2.23    |    1.75   |    1.18   |      1.89     |

### The RMSE of ablation study results:

|  PE  | Multi-scale | Distribution | Meications | Providers | Financial | StackOverflow |
|:----:|:-----------:|:------------:|:----------:|:---------:|:---------:|:-------------:|
| base |     w/o     |  Exponential |    1.89    |    3.62   |    1.44   |      5.85     |
| FCPE |     w/o     |  Exponential |    1.80    |    3.05   |    1.45   |      5.50     |
| base |      w/     |  Exponential |    1.75    |    2.48   |    1.01   |      4.39     |
| FCPE |      w/     |  Exponential |    1.15    |    2.40   |    0.92   |      4.22     |
| base |     w/o     |   Weilbull   |    1.90    |    3.50   |    2.42   |      5.51     |
| FCPE |     w/o     |   Weilbull   |    1.82    |    2.88   |    0.89   |      4.39     |
| base |      w/     |   Weilbull   |    1.65    |    2.51   |    1.15   |      4.30     |
| FCPE |      w/     |   Weilbull   |    1.12    |    2.33   |    0.88   |      4.21     |

The results from our ablation study demonstrate the robust performance of our Feature-based Cycle-aware Positional Encoding (FCPE), multi-scale attention, and Weibull distribution across various datasets. Notably, the integration of FCPE and multi-scale attention consistently yields lower RMSE values, underlining the efficacy of these components in our model. For example, in the 'Medications' dataset, the combination of FCPE with multi-scale attention and Weibull distribution significantly reduces RMSE to 1.12 from a base RMSE of 1.90. This trend is observed across all datasets, confirming the advantages of our approach as discussed in section 4.3 of our paper.

Reviewer comment (Writing):
It is not so clear why multi-scale patterns are so important for event sequences. In introduction, there is no explanation about the benefits of multi-scale features.

Author response:
Thanks for the comment. Our idea of multi-scale temporal event patterns is motivated by rea-world applications. For example, in the electronic health record (EHR) data we received from collaborators in a hospital, irregular time medication events for the same patient are multi-scale in nature. Medications administered intra-operation (in operating rooms) are often every few minutes (fine granularity), while medication events pre- or post-operations are often every few hours (coarse granularity). Our domain collaborator in clinical practice also highlights the importance of such multi-scale patterns. Thus, we propose a method to capture the multi-scale temporal patterns inherent in such real-world irregular-time event datasets. We will add such explanations in the introduction as you suggested.

Reviewer comment (Novelty):
Multi-scale temporal attention mechanism has been widely published in previous papers [1][2]. Cycle-aware Time Embedding has been proposed in [3]. Please describe the difference between your paper and these published papers. The novelty is limited.

Author response:
Thank you for the great suggestions to clarify novelty against existing works on multi-scale temporal attention and cycle-aware time embedding. We have now discussed these two aspects one by one. 

First, the two works on multi-scale temporal attention (e.g., **TransRAC [1]**, **MS-TCT [2]**) are designed for action detection in video data which are **regular time series**. For instance, MS-TCT uses temporal convolutional layers to process time series data, which cannot be directly applied to irregular time event data in our case. In contrast, our XTSFormer focuses on multi-scale temporal attention on **irregularly timed event data** in a neural temporal point process framework. We will cite these two papers in related work and add discussions on the differences in our draft.

Second, we carefully reviewed the existing cycle-aware time embedding paper [3] you suggested.  
The Multi-Recurrent Cycle-Aware (MRCA) encoding [3] decomposes time into multiple cycles like seasons, months, weeks, and days to capture recurring temporal patterns. However, we found several differences as summarized below:

MRCA directly encodes the time into several **fixed cycles**, e.g., daily, weekly, monthly, which may not be sufficient for datasets where cycles are not well-defined. In contrast, our Feature-based Cycle-aware Positional Encoding (FCPE) has **learnable cycles representation**. It is specifically tailored for irregular temporal data. Although cycle-aware time positional encodings are not new (see [4,5] below), we added the idea of semantic-feature-based learnable cycles. Specifically, the learnable cycles in our FCPE depend on **event-specific semantic features**. This is motivated by real-world applications where different types of events may show different cyclic patterns. For example, some types of medications are taken every few minutes, while others may be every few hours or daily. Another difference is that the existing work of MRCA [3] lacks **time translation invariance**. This means it is unable to assure steady performance across the time domain, particularly when facing shifts in temporal features.

We also added experimental comparisons of our feature-based cycle-aware time positional encoding (FCPE) against existing basic cycle-aware time position encoding methods, including [4] (we call base below), and Mercer encoding [5] brought up by another reviewer (i.e., positional encoding without event feature parameterization). The results are as follows:


|        | Medications |      | Providers |      | Financial |      | StackOverflow |      |
|:------:|:-----------:|:----:|:---------:|:----:|:---------:|:----:|:-------------:|:----:|
|        |   Accuracy (%) | RMSE |  Accuracy (%) | RMSE |  Accuracy (%) | RMSE |    Accuracy (%)   | RMSE |
|  base  |     29.3    | 1.65 |    40.2   | 2.51 |    73.5   | 1.15 |      45.1     | 4.30 |
| Mercer |     31.5    | 1.45 |    41.2   | 2.49 |    76.9   | 0.87 |      46.3     | 4.38 |
|  FCPE  |     33.5    | 1.12 |    43.9   | 2.33 |    77.5   | 0.88 |      49.4     | 4.21 |

The results show that our FCPE has slightly better prediction accuracy.


[1]Hu H, Dong S, Zhao Y, et al. Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting. CVPR 2022. 

[2]Dai R, Das S, Kahatapitiya K, et al. MS-TCT: multi-scale temporal convtransformer for action detection. CVPR 2022. 

[3]Dikeoulias I, Amin S, Neumann G. Temporal knowledge graph reasoning with low-rank and model-agnostic representations.  Workshop on representation learning for NLP 2022.

[4] Zuo, Simiao, et al. "Transformer hawkes process." ICML 2020.

[5] Self-attention with Functional Time Representation Learning, NeurIPS 2019.

Reviewer comment (Presentation):
Eq.6 and Eq. 7 are unnecessary. They are the commonsense of attention mechanism and MLP.

Author response:
Thank you for the great catch on writing issues concerning equations and typos. We have fixed those in our revised draft.

Reviewer comment (Evaluation):
Section 3.2.3 is so simple. Please add more analysis about Flops and memory cost of your model.

Author response:
Thank you for the suggestions. We have added a detailed analysis of our approach's time and memory costs as below. 

### Time cost analysis (FLOPs):
Assume the number of input temporal points is $L$, the batch size is $B$, the number of heads is $h$, and the hidden dimensions are $d$. In our attention computation, the query matrix dimensions is represented by $Q \in \mathbb{R}^{B \times h \times L \times d}$. For each query point $\mathcal{Q}_i$  (where $1 \leq i \leq L$), its selective key set size is $M$ (dependent on the threshold in each level). Thus, the key matrix results from concatenating key sets of all query points, yielding a dimension of $\mathbb{R}^{B \times h \times L \times M \times d}$. Consequently, the FLOPs of the attention computation with our approach is $B \cdot h \cdot L \cdot M \cdot d$, with the key set size $M$ scaling $O(log L)$. This results in a significantly more efficient attention computation Flops, specifically $B \cdot h \cdot L \cdot \log L \cdot d$, compared to the vanilla transformer [6] computation of $B \cdot h \cdot L^2 \cdot d$.  For example, when $B=64, h= 4, L=10000, d=256$, the Flops per attention layer = $2\times 64 \times 4 \times 10000 \times \log 10000 \times 256 \approx 17.4$ billion FLOPs.

### Memory cost analysis:
We analyze the memory costs of the model theoretically. We use the same math variable definition as above. We use $\mathcal{K}_i$ to denote the key set for the $i$-th query point and key set size is $M$. The memory costs of our approach are dominated by the hierarchical attention layer, since we need to store the key set $\mathcal{K}_i$ for $L$ query points, and each key set has the dimension $M \cdot d$. Thus, the total memory cost is $O(B\cdot h  \cdot d \cdot L \cdot M)$ per layer, where $M=\log L$.  Given $B=64, h= 4, L=10000, d=256$, with floating32 precision, the memory cost is  $64\times 4\times 256 \times 10000 \times log 10000 \times 4 \approx 32.4$ GB.

Additionally, to evaluate the efficiency of our method on long event sequences, we have conducted further computational experiments using a synthetic dataset. This dataset comprises $64$ event data sequences, each with a length of $100,000$ events. Each event in a sequence is a pair: an event type and its corresponding time point. The event type is a discrete variable ranging from ${0, 1, 2, \dots, 9}$. Each event type exhibits a unique temporal occurrence pattern, modeled using a Poisson distribution. Given an event type $i$, the time point of its occurrence is sampled from an exponential distribution with rate parameter $\lambda_i$. This is given by:$P(T = t | \lambda_i) = \lambda_i e^{-\lambda_i t}$, where $T$ is the time between events. To introduce variability and simulate the irregularities observed in real-world event sequences, we perturb these time intervals by adding Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$ where $\sigma$ is a small standard deviation. Based on preliminary testing and to ensure that the perturbations were subtle yet meaningful, we set $\sigma = 0.1\times \text{mean of the sampled time interval}$.

In our experiments, we set the batch size $B=1$ and the hidden dimension $d=32$. We progressively increased the sequence length and recorded the time cost (in seconds) per epoch. The results, which we plan to include in the final version of the paper, are as follows (OOM indicates 'out of memory'):

|  Length of Sequence | 10,000 | 20,000 | 30,000 | 40,000 | 50,000 | 60,000 | 70,000 | 80,000 |
|:-------------------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|
| Vanilla Transformer [6] |   2.9  |   7.9  |  12.5  |  24.3  |  39.1  |  58.8  |  83.7  |   OOM  |
|      XTSFormer      |   3.1  |   4.5  |   6.6  |  12.1  |  13.9  |  16.7  |  20.8  |  25.3  |

The experimental results clearly demonstrate that our XTSFormer model is more efficient than the Vanilla Transformer.

[6] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).

Reviewer comment (Writing):
Each equation should be followed of commas or full stop. Please clarify it.

Author response:
Thank you for the great catch on writing issues concerning equations and typos. We have fixed those in our revised draft.

Reviewer comment (Writing):
Some formulas are not clearly defined, such as the missing definition of P(ti) in Eq. (5)

Author response:
Thank you for the great catch on writing issues concerning equations and typos. We have fixed those in our revised draft.

Reviewer comment (Writing):
The writing of the paper can be improved as there are many typos.

Author response:
Thank you for the great catch on writing issues concerning equations and typos. We have fixed those in our revised draft.

Reviewer comment (Evaluation):
Most alternative approaches consider a different structure for event type and intensity prediction, which makes a comparison not 100% fair (while I consider it not a big problem).

Author response:
Thank you for bringing up the recent work on c-NTPP (Cluster-Aware Neural Temporal Point Process), published in AAAI 2023. We compared it with our XTSFormer as you suggested. 

The c-NTPP model is different from our method: c-NTPP utilizes a sequential variational autoencoder framework to identify clusters in latent feature embeddings (in latent feature space), while our XTSFormer framework conducts hierarchical clustering in the time domain in order to capture the complex multi-scale patterns in time domain from real-world data. 

We also conducted experiments for comparison on two public datasets. The results are as follows:

|           | Financial |          |      |      | StackOverflow |          |      |      |
|:---------:|:---------:|:--------:|:----:|:----:|:-------------:|:--------:|:----:|:----:|
|           | Accuracy (%)| F1-score (%) | RMSE |  NLL |   Accuracy (%) | F1-score (%) | RMSE |  NLL |
|  c-NTPP [1]|    62.6   |   63.9   | 1.17 | 1.25 |      43.8     |   33.7   | 8.50 | 2.55 |
| XTSformer |    77.5   |   78.8   | 0.88 | 1.18 |      49.4     |   35.0   | 4.21 | 1.89 |

From the results, we can see that XTSFormer performed better in accuracy over c-NTPP on these two datasets (e.g., with a ~15% increase in accuracy on event type prediction on Financial dataset). This may be due to the importance of capturing multi-scale clustering patterns in the time domain. 

[1] Ding, Fangyu, Junchi Yan, and Haiyang Wang. "c-NTPP: learning cluster-aware neural temporal point process." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.

Reviewer comment (Experiments):
No comparison to other clustering approaches for TPP [1]. However, it is also tolerable, given that the paper came out in 2023. Also lack of comparison with clustering based on embeddings of tokens from NLP.

Author response:
Thank you for bringing up the recent work on c-NTPP (Cluster-Aware Neural Temporal Point Process), published in AAAI 2023. We compared it with our XTSFormer as you suggested. 

The c-NTPP model is different from our method: c-NTPP utilizes a sequential variational autoencoder framework to identify clusters in latent feature embeddings (in latent feature space), while our XTSFormer framework conducts hierarchical clustering in the time domain in order to capture the complex multi-scale patterns in time domain from real-world data. 

We also conducted experiments for comparison on two public datasets. The results are as follows:

|           | Financial |          |      |      | StackOverflow |          |      |      |
|:---------:|:---------:|:--------:|:----:|:----:|:-------------:|:--------:|:----:|:----:|
|           | Accuracy (%)| F1-score (%) | RMSE |  NLL |   Accuracy (%) | F1-score (%) | RMSE |  NLL |
|  c-NTPP [1]|    62.6   |   63.9   | 1.17 | 1.25 |      43.8     |   33.7   | 8.50 | 2.55 |
| XTSformer |    77.5   |   78.8   | 0.88 | 1.18 |      49.4     |   35.0   | 4.21 | 1.89 |

From the results, we can see that XTSFormer performed better in accuracy over c-NTPP on these two datasets (e.g., with a ~15% increase in accuracy on event type prediction on Financial dataset). This may be due to the importance of capturing multi-scale clustering patterns in the time domain. 

[1] Ding, Fangyu, Junchi Yan, and Haiyang Wang. "c-NTPP: learning cluster-aware neural temporal point process." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.

Thank you for the suggestion. We added comparisons of our XTSFormer model with the efficient transformer models based on grouping tokens into clusters, i.e., Fast Transformers with Clustered Attention [2] and ClusterFormer [3].

After reviewing these two papers in detail, we found that these two works are similar to ours in the sense that they also conduct efficient attention based on clustering. The main difference is that both works conduct clustering in the latent token embeddings (latent feature space), while our method conducts hierarchical clustering in the time domain (to capture the multiple temporal scale patterns). We added experimental comparisons, as you suggested, on the same two public datasets. Please note that we had to recast these two methods into neural temporal point process framework for irregular time event predictions. We maintained consistent hyper-parameters across all models for a fair comparison (details in Appendix A.2.1). The results are listed below.

We can see that while the models by [2] and [3] have better time efficiency, they fall short in prediction accuracy. The results may be due to the importance of capturing multi-scale clustering in time domain (our approach).

|                   |   Financial  |              |      |      |                 | StackOverflow |              |       |      |                 |
|:-----------------:|:------------:|:------------:|:----:|:----:|:---------------:|:-------------:|:------------:|:-----:|:----:|:---------------:|
|                   | Accuracy (%) | F1-score (%) | RMSE |  NLL | Time Cost (min) |  Accuracy (%) | F1-score (%) |  RMSE |  NLL | Time Cost (min) |
| Fast Transformers [2]|     71.6     |     69.2     | 1.57 | 1.44 |       6.9       |      43.5     |     32.8     |  5.17 | 3.04 |       12.4      |
|   ClusterFormer [3]  |     65.0     |     51.8     | 1.93 | 1.87 |       5.0       |      40.5     |     27.1     | 10.15 | 5.66 |       9.8       |
|     XTSformer     |     77.5     |     78.8     | 0.88 | 1.18 |       8.4       |      49.4     |     35.0     |  4.21 | 1.89 |       14.3      |


[2] Vyas, Apoorv, Angelos Katharopoulos, and François Fleuret. "Fast transformers with clustered attention." Advances in Neural Information Processing Systems 33 (2020): 21665-21674.
[3] Wang, Ningning, et al. "Clusterformer: Neural clustering attention for efficient and effective transformer." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Reviewer comment (Experiments):
Can you compare your approach to c-NTPP?

Author response:
Thank you for bringing up the recent work on c-NTPP (Cluster-Aware Neural Temporal Point Process), published in AAAI 2023. We compared it with our XTSFormer as you suggested. 

The c-NTPP model is different from our method: c-NTPP utilizes a sequential variational autoencoder framework to identify clusters in latent feature embeddings (in latent feature space), while our XTSFormer framework conducts hierarchical clustering in the time domain in order to capture the complex multi-scale patterns in time domain from real-world data. 

We also conducted experiments for comparison on two public datasets. The results are as follows:

|           | Financial |          |      |      | StackOverflow |          |      |      |
|:---------:|:---------:|:--------:|:----:|:----:|:-------------:|:--------:|:----:|:----:|
|           | Accuracy (%)| F1-score (%) | RMSE |  NLL |   Accuracy (%) | F1-score (%) | RMSE |  NLL |
|  c-NTPP [1]|    62.6   |   63.9   | 1.17 | 1.25 |      43.8     |   33.7   | 8.50 | 2.55 |
| XTSformer |    77.5   |   78.8   | 0.88 | 1.18 |      49.4     |   35.0   | 4.21 | 1.89 |

From the results, we can see that XTSFormer performed better in accuracy over c-NTPP on these two datasets (e.g., with a ~15% increase in accuracy on event type prediction on Financial dataset). This may be due to the importance of capturing multi-scale clustering patterns in the time domain. 

[1] Ding, Fangyu, Junchi Yan, and Haiyang Wang. "c-NTPP: learning cluster-aware neural temporal point process." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.