Paper: PEMs: Pre-trained Epidemic Time-Series Models (ICLR-2024)

Reviewer comment (Experiments):
While the paper compares PEM with FUNNEL, a mechanistic model, it does not explore simpler disease-focused mechanistic models or even basic auto-regressive models. Additionally, the absence of testing the performance of simpler models on unseen diseases, like COVID-19, limits the model's generalization claims. Over the years many experts have submitted forecasts for COVID-19 and influenza in real-time (e.g., US COVID-19 Forecast Hub, EU COVID-19 Forecast Hub). The paper already refers to a paper on one of the hubs (Cramer 2022). Without comparison against the models used by the community, it is hard to judge how much value the proposed approach adds.

Author response:
We do compare against state-of-art epidemic forecasting models that have shown to beat past simpler statistical models on epidemic forecasting tasks and
have won past epidemic forecasting competition[1,2,3].
Therefore, we believe this is a fair comparison to provide evidence
that PEM indeed provides state-of-art forecasting performance.

Reviewer comment (Experiments):
For COVID-19 and epidemiological data in general, there is frequent data revision and backcorrection. The data that is available today is a much cleaner version compared to what was available at the time a forecast was made. Especially for COVID-19, the data revision history is available; therefore, the authors should test the dataset with the appropriate version, not the latest version. It is not clear if the authors have done so.

Author response:
All the baselines and the model are evaluated on the latest available version of
Covid-19 mortality data for a fair comparison.
We agree with the reviewer that data revision is an important
problem in epidemic forecasting. Our work doesn't address this problem and instead focuses on modeling and accurately forecasting
epidemic time-series. However, analyzing and tackling the data revision
problem for our pre-training framework is an interesting future direction.

Reviewer comment (Experiments):
Performance compared to simple transfer learning or doing some training on a single disease dataset (such as pre-training on just past flu data if the disease of interest is flu) would have been good. Randmask does not perform nearly as well as the others, so does adding this into the combination of tasks contribute meaningfully?

Author response:
Yes RANDMASK alone is the worse performing SSL task overall. However,
we still included it for pre-training because (a) it still outperforms
the variant without any pre-training, (b) and using all four SSL tasks
together provided the best performance across all benchmarks.

Reviewer comment (Evaluation):
The remarkably low RMSE values for Flu in the US look unusual. Some more context or explanations for these results would be useful. Was this normalized somehow (like per 100k)?

Author response:
The influenza incidence time-series extracted from CDC is normalized
between 0 to around 10. We direcltly use these values for training and prediction. Therefore, the RMSE values may appear very low compared to other benchmarks.

Reviewer comment (Experiments):
Have the authors compared their results against models that were used by the epidemic forecasting community during COVID-19 and ILI forecasting tasks? (These forecasts are publicly available)

Author response:
There are two main reasons why direct comparisons with the models used
by forecast hub is not feasible.
First, our model only uses past values of the time-series to forecast future dynamics. It does not ingest additional external variables.
Therefore, all the benchmarks are univariate time-series forecasting
or prediction tasks and we evaluate state-of-art baselines designed for such tasks.
Further, most models in the forecasting hubs used additional data sources and the models and datasets used are not public in most cases.

Reviewer comment (Experiments):
Was there any data pre-processing involved? How does that impact the results?

Author response:
We do not perform any data pre-processing on raw time-series data provided. Since we used Reversible Instance Normalization  as part of the model, we
do not need explicit data normalization before feeding into the model.

Reviewer comment (Evaluation):
Why is RMSE for the US so low?

Author response:
The influenza incidence time-series extracted from CDC is normalized
between 0 to around 10. We direcltly use these values for training and prediction. Therefore, the RMSE values may appear very low compared to other benchmarks.

Reviewer comment (Novelty):
sounds like your segments of time series is basically patching, similar to (Nie et al. 2022). If so, why do you achieve so much better results than PatchTST? Is it just the finetuning?

Author response:
**why do you achieve so much better results than PatchTST? Is it just the finetuning?**

It is due to our SSL pre-training. Indeed performing fine-tuning on heterogenous time-series data by designing specific SSL tasks for extracting useful background knowledge from wide range of disease datasets during pre-training is our most important technical contribution.
We note that most other SSL pre-training methods of past works are not effective
on our heterogenous pre-training problem.
Therefore, our method of effective fine-tuning enables the pre-trained models
to significantly outperform models without pre-training (such as using PatchTST without pre-training) as well as methods that use
previous SSL methods.

Reviewer comment (Theory):
I'm not sure I understand why inputting each time step as a single token would lack "semantic meaning", seeing as the output representations of a transformer for a given token are influenced by all the other tokens in the input sequence (i.e. "contextual embeddings").

Author response:
**I'm not sure I understand why inputting each time step as a single token would lack "semantic meaning", seeing as the output representations of a transformer for a given token are influenced by all the other tokens in the input sequence (i.e. "contextual embeddings")**

Providing semantically meaningful input tokens can better enable transformers to model
relations between tokens.
For example, in the context of language, LLMs use specific tokenization methods to capture semantic meaning instead of individual characters to provide significantly better performance.
Further, the inputs of time-series are real-valued numbers rather than specific character.
Therefore, these individual values of time-series observed in isolation do not provide sufficient context such as local patterns (such as peaks, spikes, etc.)
around the time-step.
therefore, we use segments of time-series that provide semantic information about local patterns.

Reviewer comment (Experiments):
How do you normalize the training process of your method wrt baselines? How can you disentangle the architecture's influence from the influence of the training process and the influence of using additional compute?

Author response:
**How do you normalize the training process of your method wrt baselines? How can you disentangle the architecture's influence from the influence of the training process and the influence of using additional compute?**

For all deep learning baselines and PEM, we train using early stopping.
We use the same compute hardware for training all models and baselines (appendix B) and we measure the training time till convergence
for all models.
PEM uses similar training time to other best baselines with similar compute and provide significantly better performance.
In fact, it takes 12-50% less time to match the performance of the baselines (Appendix Table 5).

We also measure the importance of various training choices and pre-training choices in Table 3 and show the importance of each SSL task
and linear-probing for optimal performance. We also measure the effect of segmentation in Appendix Table 9.

Reviewer comment (Presentation):
why are the results around one of the contributions (Significant improvement in data and training efficiency and adaptability to novel epidemics) only in the appendix?

Author response:
**why are the results around one of the contributions (Significant improvement in data and training efficiency and adaptability to novel epidemics) only in the appendix?**

We summarize the main results of training efficiency and data efficiency in the main paper page 8 and provide additional details in the Appendix: PEM takes 47-88% of the time taken by best baselines and uses 60-80% of total training data to outperform them.
We could summarize the training times in Table 4 using the bar graph and add to the main paper.

Reviewer comment (Experiments):
There might be bleed between the influenza data from 2001 to 2010 and the crypto data from 2006 to 2012 given the temporal alignment, but I'm not knowledgeable enough to judge how strong the overlap could be. Is the argument that, since influenza and crypto have different inherent dynamics, they shouldn't influence each other?

Author response:
**There might be bleed between the influenza data from 2001 to 2010 and the crypto data from 2006 to 2012 given the temporal alignment**

This is an interesting question. However, influenza and crypto show very different disease dynamics. Influenza peaks during winter months and spreads through the air. Crypto mostly occurs during the summer and is spread through food and water contamination.
While influenza is spread through a virus, crypto is spread through a
microscopic parasite.
We couldn't find any research linking the incidence of both diseases

Reviewer comment (Experiments):
How do you choose the value of T, i.e. the length of the input time series?

Author response:
**How do you choose the value of T, i.e. the length of the input time series?**

All the diseases in training and pre-training have a sampling rate of 1 month. Moreover, even seasonal diseases have periodicity within 12 months.
Therefore, we choose the total input length to be $T=12$.
We will add this information in the Appendix.

Reviewer comment (Experiments):
How do you justify the default choice of gamma = 0.1 for lastmask ? Seems like 0.2 performs well too.

Author response:
**How do you justify the default choice of gamma = 0.1 for lastmask ? Seems like 0.2 performs well too.**

Yes both 0.2 and 0.1 performs well depends on the downstream tasks.
However, even in cases 0.2 performs better, 0.1 is only slightly worse.
Therefore, we chose gamma as 0.1.

Reviewer comment (Experiments):
The baselines in this paper do not represent the SOTA performance. Also, the baselines are not "easy-to-replicate" methods whose code is not provide. This makes me a bit concern about the effectiveness of the comparison results.

Author response:
**The baselines in this paper do not represent the SOTA performance.**

The chosen baselines are state-of-art general forecasting and epidemic forecasting
models at the time of submission. Therefore, we are indeed comparing PEM with the previous best possible performance.

The baselines used have code released publicly. We used the same code with
mostly default hyperparameters for evaluation with hyperparameter tuning on learning rate.
We can add links to the baseline code in paper to alleviate these concerns.

Reviewer comment (Evaluation):
There is no uncertainty quantification analysis of all the methods. It seems the transformer-based models are so large so may have large predicting variance.

Author response:
**There is no uncertainty quantification analysis of all the methods. It seems the transformer-based models are so large that they may have large predicting variance**

Our method and most other baselines only focuses on forecasting and prediction accuracy and does not
provide reliable uncertainty estimates.
However, we agree with the reviewer that uncertainty quantification is an important forecasting problem, and extending our methods to produce calibrated probabilistic forecasting is an interesting research direction.

Reviewer comment (Writing):
In section SEASONDETECT, the paper first mentions that the year is divided into 4 seasons Season 1 (Dec-Feb), but later, it also says the peak season is s_1(d). Could you clarify this in a more consistent way?

Author response:
**Clarification on SEASONDETECT**

We divide the 12 months into four seasons: Dec-Feb, Mar-May, June-Aug and Sept-Nov.
For each disease dataset $d$, we assign one of these four seasons as peak season $s_1(d)$. For example, in the case of influenza $s_1(d) =$ Sept-Nov.
Based on assignment on $s_1(d)$ we assign $s_2(d), s_3(d), s_4(d)$ chronologically.
In influenza's case, $s_2(d)=$ Dec-Feb, $s_3(d)=$ Mar-May, $s_4(d)=$ June-Aug.

Reviewer comment (Reproducibility):
Data normalization, will the normalized data be reversed before computing the loss error during training?

Author response:
**Data normalization, will the normalized data be reversed before computing the loss error during training?**

Yes. Using Reversible Normalization prevents large gradients during training.

Reviewer comment (Experiments):
Why perform linear probing before fully fine-tuning?

Author response:
**Why perform linear-probing before fine-tuning?**

Kumar et. al [1] showed that doing direct fine-tuning leads to lower accuracy
as it distorts the learned representations of pre-trained models, especially for downstream tasks with distribution shift, similar to our case where we fine-tune on heterogeneous and unseen disease datasets.
Linear-probing alleviates this by adapting the final layer of the pre-trained model
to downstream tasks without distorting the representation learning of the pre-trained model.

Reviewer comment (Experiments):
The related work should be expanded. To the best of my knowledge, a series of papers [1] have been studied on SSL for general time series, also including epidemic analysis. However, they're rarely mentioned or considered as baselines for a fair comparison.

Author response:
**The related work should be expanded.**

We have described recent state-of-art SSL methods and used them as baselines.
While there are many methods that perform SSL on time-series, we focus
on methods that are applicable to any general time-series domain with
specific applications to forecasting tasks.
We would gladly add additional references to SSL methods including recent methods
described in the survey referred to by the reviewer.

Reviewer comment (Novelty):
The technical novelty against previous SSL approaches is somewhat limited.

Author response:
**The technical novelty against previous SSL approaches is somewhat limited.**

 Unlike most previous SSL papers, we are the first to deal with a different broader problem of effectively leveraging multiple heterogeneous epidemic datasets to train a general pre-train model that can be fine-tuned to a wide range of downstream epidemic analysis tasks. 

RANDMASK and LASTMASK are indeed widely used pre-training tasks in language domains and are also explored in time-series domain previously. 
However, to better extract useful epidemic dynamic information, we also introduce two novel tasks: PEAKMASK and SEASONDETECT. PEAKMASK helps identify peaks and their dynamics which is vital for epidemic forecasting since it is important to model and predict epidemic peaks in advance. Detecting seasonal information SEASONDETECT automatically is also important in successfully modeling epidemic dynamics. These tasks are carefully designed to impart useful background knowledge and patterns from multiple epidemic datasets. 

To further illustrate the importance of designing and choosing the right SSL tasks, we also compared PEM against sophisticated SOTA self-supervised methods for general time-series and show that using generic tasks that perform well in other domains to epidemic datasets leads to poor performance.

Reviewer comment (Experiments):
The baselines listed on Page 7 are insufficient. More SSL methods, particularly for general time series, should be included as baselines. For example, it would be interesting to see if PEM can outperform recently developed SSL methods in time series analysis.

Author response:
**The baselines listed on Page 7 are insufficient.**

We note that even most recent time-series SSL methods do not deal with multiple
heterogenous datasets. The consisdered
SSL baselines (TS2Vec, TNC, TS-TCC) in the paper which provide top performance for specific
benchmarks where training data is used for SSL pre-training
perform much worse than even some baselines which do not use any pre-training in our setting of pre-training on multiple datasets. This shows that even sophisticated general SSL methods cannot simply used
to pre-train on multiple datasets.

Reviewer comment (Reproducibility):
It would greatly enhance reproducibility if the authors provide the source code for their approach.

Author response:
**authors provide the source code for their approach.**

We have added link to the implementation code in the paper at page 7: [link](https://anonymous.4open.science/r/EmbedTS-3F5D/).
We also plan on releasing the weights of pre-trained models on acceptance.