Paper: Prompting-based Efficient Temporal Domain Generalization (ICLR-2024)

Reviewer comment (Experiments):
The problem description seems to promise more than what the eventual solution delivers. Particularly, the paper is positioned by the authors as domain generalization and promises a solution in a space where learnings from different domains can be utilized to capture information useful for predicting unknown target domains. However, after reading the solution and dataset description, this falls short of expectation as the authors focus on concept drift within the same dataset. Data from the same data generation process is divided into multiple windows, where each window is called a domain. So, data drifts indicated by the authors are within data drift over time.

Author response:
>This paper focuses on temporal domain generalization, addressing data distribution shifts over time. We followed the experimental setup of existing state-of-the-art temporal domain generalization methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)]. And we cited the state-of-the-art method results in Table 1. We are not the first to propose temporal DG, and we followed the standard practice.

Reviewer comment (Novelty):
In the literature, there have been multiple articles published on concept drift or data drifts in general over the past few decades. For example, please see Lu, Jie, et al. "Learning under concept drift: A review." IEEE transactions on knowledge and data engineering 31.12 (2018): 2346-2363. With this context, it is not clear why data windows within a dataset are termed as "domains", where it is truly not from a different domain.

Author response:
>We followed the experimental setup of existing state-of-the-art temporal domain generalization methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)], and we split the dataset following the SOTA temporal DG methods. We are not the first to propose temporal DG, we didn't propose a new way to split the dataset. We followed the standard practice.

Reviewer comment (Experiments):
Are the number of data points in each domain the same?

Author response:
>The number of data points in each domain can vary. We do not enforce a uniform number of data points across all domains, as this depends on the dataset and the nature of the temporal segments being considered.  
>
>In Line 12 of Algorithm 1, we are referring to the input tokens (PS1, PS2, ..., PS(t−1)) for the prompt generation module. These tokens represent the domain-specific prompts from previous time periods. In contrast, Line 13 describes the generation of PT(t), which is the current temporal prompt. This prompt is produced by the module using the concatenation of previous domain-specific prompts as input. The key distinction here is between the input prompts from previous domains (Line 12) and the output prompt for the current domain (Line 13).  
>
>Y represents the label for the input data in the same domain. To illustrate with the housing price prediction example: if the features of a house in 2021 (X) correspond to a value of 200K, then Y (the label) would also be 200K. In other words, both the input (X) and output (Y) are derived from the same domain, ensuring that the model learns to predict within the context of each specific temporal domain.

Reviewer comment (Reproducibility):
In Step 13 of Algorithm 1, how exactly is PT(t) generated? Is it a concatenation of previous domain-specific prompts concatenated when provided as input to gw? Particularly, what is the difference between Line 12 and Line 13?

Author response:
>The number of data points in each domain can vary. We do not enforce a uniform number of data points across all domains, as this depends on the dataset and the nature of the temporal segments being considered.  
>
>In Line 12 of Algorithm 1, we are referring to the input tokens (PS1, PS2, ..., PS(t−1)) for the prompt generation module. These tokens represent the domain-specific prompts from previous time periods. In contrast, Line 13 describes the generation of PT(t), which is the current temporal prompt. This prompt is produced by the module using the concatenation of previous domain-specific prompts as input. The key distinction here is between the input prompts from previous domains (Line 12) and the output prompt for the current domain (Line 13).  
>
>Y represents the label for the input data in the same domain. To illustrate with the housing price prediction example: if the features of a house in 2021 (X) correspond to a value of 200K, then Y (the label) would also be 200K. In other words, both the input (X) and output (Y) are derived from the same domain, ensuring that the model learns to predict within the context of each specific temporal domain.

Reviewer comment (Writing):
Given the reviewer’s understanding of the problem setup, it is unclear what exactly is Y. For example, in the housing price prediction case, is Y the house prices in the target domain (validation data) or house prices at domain t available in the training data?

Author response:
>The number of data points in each domain can vary. We do not enforce a uniform number of data points across all domains, as this depends on the dataset and the nature of the temporal segments being considered.  
>
>In Line 12 of Algorithm 1, we are referring to the input tokens (PS1, PS2, ..., PS(t−1)) for the prompt generation module. These tokens represent the domain-specific prompts from previous time periods. In contrast, Line 13 describes the generation of PT(t), which is the current temporal prompt. This prompt is produced by the module using the concatenation of previous domain-specific prompts as input. The key distinction here is between the input prompts from previous domains (Line 12) and the output prompt for the current domain (Line 13).  
>
>Y represents the label for the input data in the same domain. To illustrate with the housing price prediction example: if the features of a house in 2021 (X) correspond to a value of 200K, then Y (the label) would also be 200K. In other words, both the input (X) and output (Y) are derived from the same domain, ensuring that the model learns to predict within the context of each specific temporal domain.

Reviewer comment (Experiments):
The empirical evaluation in Tables 2 and 3 shows that the proposed method has the least error across all datasets with greater than 2 variables. However, both the synthetic data generation and temporal drifts learned by the prompts seem to work for non-abrupt changes. Does this also work for abrupt drift?

Author response:
>Our model is optimized for gradual temporal drifts. If abrupt drifts are present in the training data, it's likely our model can adapt to similar abrupt changes during testing. However, without exposure to abrupt drifts in training, the model may not effectively handle sudden changes in the test data.

Reviewer comment (Novelty):
The motivation for using prompts still lacks proper motivation. Specifically, the motivation for using prompts is claimed as "none of these prior works can generate time-sensitive prompts that capture temporal dynamics." There are tons of other ways to learn temporal dynamics, and we don't have to use prompts.

Author response:
While alternative methods exist, our approach is well-supported and aligned with current research trends in the field. We have extended prompting from NLP to a lot more applications, which is truly valuable. Most previous prompting methods only demonstrated their efficacy in NLP tasks or NLP-related neural networks like GPT or CLIP. We proposed the first prompting-based temporal DG method, which is a generic model and is applied to various tasks. This shows great potential in using prompt learning for various applications other than NLP.

Reviewer comment (Presentation):
The overall presentation is a little messy. There are some undefined notations. The authors seem to misuse \citep{} and \citet{}, and the presented references impact the overall readability.

Author response:
Thank you for your feedback. We will address the noted issues by correcting typos, clarifying undefined notations, and revising our citation format to improve readability and presentation.

Reviewer comment (Theory):
What's the mathematical formulation of the prompt?

Author response:
Our work is inspired by the success of prompt-tuning in NLP tasks [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021); Li & Liang (2021); Asai et al. (2022); Wang et al. (2023)]. As for the mathematical formulation of prompt, [Oymak et al. (2023)] developed a statistical foundation for gradient-driven prompt-tuning, examined its optimization and generalization behaviors, and investigated how it facilitates attending to context-relevant information. Additionally, they demonstrated that one-layer softmax-prompt-tuning is provably more effective than other methods, such as one-layer self-attention.

[Oymak et al. (2023)]: Oymak, S., Rawat, A.S., Soltanolkotabi, M. &amp; Thrampoulidis, C.. On the Role of Attention in Prompt-tuning. Proceedings of the 40th International Conference on Machine Learning (ICML 2023).

Reviewer comment (Experiments):
What would be the intuition of training the backbone network on the aggregated dataset? For some datasets with manually altered data distribution on each domain (like two moons), the decision boundary would be really difficult to learn if you mix all the data together.

Author response:
Through aggregating datasets, our network is designed to learn general/common features across the entire dataset, while domain-specific prompts target specific feature sets.

Reviewer comment (Experiments):
Following the previous question, the ablation study can be further designed to remove the backbone model to verify if the backbone model is truly useful.

Author response:
Sorry, We didn't understand the question. How would prompting work without a backbone model? We followed the standard practice of prompting [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)], and a pre-trained backbone model is required.

Reviewer comment (Writing):
Some sentences are quite hard to understand, e.g., "For each domain $t$, we prepend the input $X$ with a prompt $PS(t)$, which are learnable parameters." Are both $X$ and $PS(t)$ learnable?

Author response:
Prepending input with a prompt is a standard practice of prompting learning [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)]. In our model, the inputs are standard, non-learnable data from the dataset, similar to other classic models. Only the prompts are learnable parameters.

Reviewer comment (Experiments):
Not sure why some baselines are excluded in Table 3's comparison.

Author response:
Other baselines are not applicable on Time series datasets. We will clarify that in Camera Ready.

Reviewer comment (Novelty):
One of the major claims is also confusing: "Our paper presents a novel prompting-based approach to temporal domain generalization that does not require access to the target domain data". I feel like no access to the target domain data is a default rule of domain generalization.

Author response:
Yes, we agree that no access to the target domain data is a default rule of domain generalization. We just want to emphasize it and highlight it, since it is the key difference between domain adaptation and domain generalization.

Reviewer comment (Experiments):
I question whether it is appropriate to split time series data artificially into different source domains. If a domain is not explicitly defined, one could pick an arbitrary period and define it as a domain, as was done on the crypto data set. There is no sound justification for why a one month period was chosen or why it would be better than a two week period.

Author response:
>We followed state-of-the-art papers [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019) ] to split the dataset, and we split the Crypto dataset in the same way. We proposed one reasonable way of splitting the dataset, and we did not aim to propose the best way to split the dataset. Brute-force all kinds of ways splitting the dataset is a NP-hard problem, which is impossible and is also trivial for a paper.

Reviewer comment (Evaluation):
It seems that the presented method's performance largely lies in the confidence intervals of competing methods.

Author response:
>Yes, but our methods achieve the performances with a lot fewer parameters and much less training time. Normally machine learning models need to trade performance for efficiency, while our model shows great performance with better efficiency.

Reviewer comment (Evaluation):
I miss some scientific curiosity about the learned prompt representations; there is much more potential in this work than reducing it to performance metrics.

Author response:
>We agree, so we expanded our works to more tasks and datasets. Most previous prompting methods only demonstrated their efficacy in NLP tasks or NLP-related neural networks like GPT or CLIP. We proposed the first prompting-based temporal DG method, which is a generic model and is applied to various tasks. This shows great potential in using prompt learning for various applications other than NLP.

Reviewer comment (Reproducibility):
Can you please clarify what the stopping criterion is when pre-training the backbone initially? You say the domain-specific prompts are learnable parameters, can you specify how they are connected to the backbone/output? You write they are concatenated, does this mean in the initial pre-training, we need to know the size of the prompt and mask the input accordingly? Or is there a linear layer whose parameters are learned? You might want to formalize all this by introducing a second set of parameters (as $\theta$ is always frozen). Furthermore, is a new temporal prompt generator trained for each temporal prompt, or is it reused? In Figure 1, you "freeze" $P_{T2}$ but not $P_{T3}$ in the next step, why? In Fig. 1's caption, you say, "[...] finally, [...] $P_G$ is trained". This implies a sequential training but from the figure, it seems like $P_G$ is also the output of the temporal prompt generator.

Author response:
>The pre-training process is halted when the validation loss fails to improve for a predefined number of epochs (3)  

>We follow the standard practice of prompt learning to learn the prompts[Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)], namely domain-specific prompts are extra tokens, and we are not increasing the dimension of tokens. The concatenation is about increasing the length of the input sequence, and transformers can take variable-length inputs.  
To answer your question, yes prompts are learnable parameters and they are part of the inputs inputting to the backbone network. No, we don't need to know the size of the prompt for pre-training.  

>For our temporal prompt generator, we adopt a dynamic tuning approach rather than training a new generator for each temporal prompt. Specifically, the process involves loading the weights from the previous step and then fine-tuning them based on the domain using previous domains information.  

>As explained in Section 3.3, it is sequential training. The temporal prompt generator is sequentially trained to generate prompts for the upcoming new domain, and previously learned prompts are fixed.

Reviewer comment (Experiments):
Can you provide experimental results on data sets that naturally come from different domains?

Author response:
>For experimental results in Table 1, we followed state-of-the-art temporal DG methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)] and adopted commonly used datasets. Crypto dataset is a time series dataset that naturally comes from different temporal domains. Domain-specific and/or temporal prompts may contain some information to determine the existence of different domains, while generic prompt cannot since it contains shared information across different domains. We will compare our domain-specific prompts in the camera-ready version.

Reviewer comment (Evaluation):
If this is not the case, is it possible to use your method to **determine** the existence of different domains? I am thinking of comparing the learned domain-specific prompts, for example, in terms of their cosine similarity.

Author response:
>For experimental results in Table 1, we followed state-of-the-art temporal DG methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)] and adopted commonly used datasets. Crypto dataset is a time series dataset that naturally comes from different temporal domains. Domain-specific and/or temporal prompts may contain some information to determine the existence of different domains, while generic prompt cannot since it contains shared information across different domains. We will compare our domain-specific prompts in the camera-ready version.

Reviewer comment (Experiments):
How do standard forecasting methods such as ARIMA/Gaussian Processes perform on the data sets?

Author response:
>We have compared our methods with the current state-of-the-art methods. It would be great if the reviewer could kindly point us to existing state-of-the-art papers that apply these methods.

Reviewer comment (Evaluation):
Why are no standard deviations in Tables 3, 4, and 5 reported?

Author response:
>We will report that in Camera ready, they are similar to other tables.

Reviewer comment (Writing):
Given its origin in NLP, I am not sure if "prompt" is the best fitting wording in the context of time series.

Author response:
>We believe it is the best-fitting wording. We followed the exact standard practice of prompting [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)].

Reviewer comment (Evaluation):
Overclaim 2. As for the time-efficient aspect, there is no training time comparison analysis to demonstrate the claimed "time-efficiency." Especially, either pre-training or fine-tuning a transformer-based model to adapt to the specific task (temporal soft-prompt generation) are inefficient.

Author response:
The assertion that our model lacks time-efficiency is addressed in Table 3, which presents a detailed comparison of training times between our method and existing approaches. This table clearly demonstrates that our model is not only efficient in terms of parameter usage, as previously discussed, but also in training time.

Reviewer comment (Experiments):
Unfounded. The authors claim that "Only a few methods studied temporal DG problem Nasery et al. (2021); Bai et al. (2023), which are inefficient and complex to be applied to large datasets and large models," which is unfounded, no evidence supported, and without any quantitative analysis for demonstrating this assumption.

Author response:
The claim about the inefficiency and complexity of existing methods for temporal dynamic graph (DG) problems is backed by the data presented in Table 3. This table specifically illustrates how the number of parameters in the DRAIN method increases exponentially with time series datasets. Given that DRAIN relies on MLPs for encoding and decoding weights, this exponential increase in parameters makes it impractical for use with larger models.  Moreover, DRAIN requires generating weights for the backbone networks using hypernetworks, and generating weights for larger models is impractical.

Reviewer comment (Evaluation):
The performance improvement is minor and not significant, especially since the proposed method achieves inferior performance than DRAIN (state-of-the-art of TDG) on the 2-moon dataset, a basic synthetic dataset on testing TDG. The performance of the proposed method is not convincing.

Author response:
We evaluated our method on 4 synthetic datasets and 6 real datasets. Our model demonstrates a significant improvement in performance compared to **DRAIN, which is the current SOTA model**, with an average margin of 20% across various benchmarks, as detailed in Tables 1, 2, and 3 on 6 real datasets and 3 synthetic dataset, and achieved comparable result on only 1 synthetic dataset. This general superiority is indicative of the model's robustness and effectiveness in various applications including classification, regression, and time series forecasting. Let alone our method is more time-efficient and parameter-efficient.

Reviewer comment (Experiments):
The proposed framework seems to be adaptive on multiple modalities of transformer-based networks as the model backbone. However, the paper only evaluates their framework on one transformer-based network. The authors are highly encouraged to test their framework incorporated with multiple transformer-based networks.

Author response:
We appreciate the suggestion to evaluate our framework across multiple transformer-based networks. Indeed, our model is designed to be versatile and adaptable to a variety of models dealing with temporal data. While the current paper focuses on a single transformer-based network for a thorough and focused analysis, we recognize that using our model on other baselines can also be an interesting and Exploring this in future work is definitely something we find compelling and valuable.

Reviewer comment (Evaluation):
ONP has been proven to obtain no domain shifting [1], which means most of the TDG-based methods are useless in ONP. However, the proposed methods, in contrast, obtain good performance on ONP. Is there any specific reason that can explain this phenomenon?

Author response:
Temporal DG methods widely applied on ONP. Contrary to the statement that ONP exhibits no domain shift, it  has minimal drift [1]. The notable performance of our model on ONP can be attributed to the integration of general prompts (PG) in our framework. These general prompts capture shared patterns across domains and enhance the model's ability to adapt and perform effectively even in stable environments with minimal drift. As demonstrated in Table 4, combining Temporal and General Prompts plays a crucial role in boosting performance. This dual-prompt approach allows our model to maintain high accuracy and adaptability in various scenarios, including ONP, where traditional TDG-based methods might not be as effective.