Paper: EasyTPP: Towards Open Benchmarking Temporal Point Processes (ICLR-2024)

Reviewer comment (Reproducibility):
The accuracy plots would be more useful as tables. For reproducibility purposes, it would be useful to the community to know the exact accuracy values (and confidence bounds) from the various methods. Therefore, in the reviewer’s opinion, all plots showing empirical results should be replaced with tables. The reviewer asks if this is possible in the camera-ready version.

Author response:
We agree that empirical results should be better illustrated with numbers instead of figures. We have replaced the figure of the main prediction result with a table in the main context and all the other tables are in Appendix D.3. We are committed to replace all the figures with tables in nice formats in the camera-ready version. See [Presentation Improvements].

Reviewer comment (Writing):
The "temporal event sequence" topic is not well-defined. In Fig 1, the reviewer does not quite know what is meant by "Temporal event sequence" as opposed to "temporal point process". Furthermore, the reviewer notes that a Hawkes process is a version of a temporal point process, suggesting that these publication sets may overlap somewhat. The reviewer asks if this is clarified in the text, or if the authors can clarify.

Author response:
In fact we have clarified this search process in Appendix E.1. In the revised version, we have added one line to refer to this section for details of how we count the articles.  See [Presentation Improvements].

We clarify again here. To find out the number of point process research papers, we search in arxiv with the three terms (Temporal point process, Hawkes process,  Temporal event sequence) that are widely used in related publications, respectively. To avoid the overlap of searched results, we search with the following advanced rule:
- Temporal point process: it corresponds to the articles whose abstracts contain the term ‘temporal point process’. 
- Hawkes process: it corresponds to the articles whose abstracts contain ‘hawkes process’ but not the term ‘temporal point process’. 
- Temporal event sequence: it corresponds to the articles whose abstracts contain the term ‘temporal event sequence’ without neither the term ‘hawkes process’ nor ‘temporal point process’.

Reviewer comment (Novelty):
The main concern is contribution. Considering the quality of the ICLR community, although it is particularly urgent to propose a unified comparison and processing framework for point processes, the contribution is still limited.

Author response:
Dear Reviewer btMZ,

We greatly appreciate the time you took to review our paper. In the rebuttal, we have clarified the contribution of our work and provided more detailed explanations for the proposed framework.

Due to the short duration of the author-reviewer discussion phase, we would appreciate your feedback on whether your main concerns have been adequately addressed. We are ready and willing to provide further explanations and clarifications if necessary. Thank you very much!

Reviewer comment (Presentation):
Unclear presentation

Author response:
In section 3, we discuss the modules in detail and use figure-3 to illustrate the pipeline of the system.

In section 4, we discuss the interface with two examples of usage (how to customize a model and how to run the pipeline)  in Listing 2 and Listing 3.

In Appendix A, from page 13 - 16, we discuss the details of design and interface such as 
- High level software architecture (section A.1): figure-8 depicts the detailed architecture with each module’s functionality annotated / explained in text.
- Why and how the system supports both Tensorflow and PyTorch optimization framework (section A.2 - A.3).
- Pseudo implementation to customize a model using the interface of the code base (section A.4)
- Pseudo implementation to run an existed model using the interface of the code base (section A.5)

Besides, an even more detailed description of system and interface could be found at the document provided in our code base: https://github.com/Anonymous0006/EasyTPP/tree/main/docs/source/ref. Please see Readme.md to find out how to generate htmls for better visualization. The explanations of the interfaces at any level are well documented and easy to search and read.

In all, around ¼ of the paper contents, plus pages of online documents,  relate to the system design and interface, could you please elaborate which part is still unclear to you？

Reviewer comment (Novelty):
This paper lacks technical innovation. I don't think a software-level abstraction is a good fit for ICLR.

Author response:
Please see [Technical Contribution and Broader Impact]

Reviewer comment (Experiments):
It's unclear what points the authors intend to demonstrate in the experiment section.

Author response:
We want to demonstrate the following key insights from the experiment:
- Insight 1: The performance of the classical model MHP is worse than the neural models across most of the evaluation tasks.
- Insight 2: There is no single winner against all the other methods on time and type prediction. Notably, all the best to-date models make poor predictions on time of future events. Moreover, on type prediction, attention-based models only outperform other architectures by a small margin. 

These insights have strong implications on our benchmarking work and future research:
- Based on insight 1, we decide that our code base focuses on neural TPPs rather than classical TPPs. 
- Insight 2 signals that we should think beyond architectural design for future directions. For the past decade, this area has been focusing on developing new architectures, but the performance of new models on the standard datasets seem to be saturating. Looking into the future, we advocate for a few new research directions that may bring significant contributions to the field, summarized in Section 6 [FUTURE RESEARCH OPPORTUNITIES], including building foundation models of event sequence, empowering the model with large language models etc. 

Please let me know if these points are clear for you.

Reviewer comment (Experiments):
The authors compare against the classical Multivariate Hawkes Process (MHP) but don't specify what type of kernel they use (I assume exponential) or structure of the excitation matrix.

Author response:
We are indeed using the exponential kernel for MHP whose dimension equals the number of the event types of the dataset. 

We have added the specification for MHP in the revised version.  See [Presentation Improvements].

Reviewer comment (Writing):
Some typos, e.g. Multivariate Hakwes Process (MHP).

Author response:
Thanks for pointing this out. We have fixed this typo in the revised version.  See [Presentation Improvements].

Reviewer comment (Reproducibility):
Is there an easy way to incorporate marks into your package? From the description in the paper, it seems like you can easily handle different event types through a multivariate TPP, but I don't see any discussion of marks. In many application settings, we can have a feature vector for each event, which can be modeled as a mark.

Author response:
Yes. It is easy to incorporate it in our package. We can modify the input data structure to include a feature vector for each event, and in the input module, we apply a feature encoder to produce a feature embedding and concat it with the event type embedding. This modification needs only a small amount of work and is compatible with the current framework.

We are committed to adding this functionality in the future version. See [Commitment].