Adaptive Sampling for Probabilistic Forecasting under Distribution Shift
Abstract
The world is not static: This causes real-world time series to change over time through external, and potentially disruptive, events such as macroeconomic cycles or the COVID-19 pandemic.
We present an adaptive sampling strategy that selects the part of the time series history that is relevant for forecasting.
We achieve this by learning a discrete distribution over relevant time steps by Bayesian optimization.
We instantiate this idea with a two-step method that is pre-trained with uniform sampling and then training a lightweight adaptive architecture with adaptive sampling.
We show with synthetic and real-world experiments that this method adapts to distribution shift and significantly reduces the forecasting error of the base model for three out of five datasets.
Introduction
Time series forecasting uses historical data to forecast the evolution of a time series to assist decision making in several domains, for example retail [24, 5, 32], electric load planning [30, 13], cloud computing management [25, 26] or labor planning [3].
Most time series forecasting algorithms make the assumption that the data distribution does not change over time.
However, this is violated in practice where real-world time series data is affected by drifts such as evolving customer demand or disruptive events (like the COVID-19 pandemic).
These distribution shifts will inevitably be part of the time series history and therefore forecasting algorithms need to be able to account for them.
Distribution shifts can occur with different (possibly reoccurring) patterns, but likely result in higher test error because the train and test distribution differs.
Thus, only a subset of the time series history might come from the same distribution as the test set.
Driven by this observation, we propose an adaptive sampling mechanism that explicitly feeds the model with inputs from the history that are only relevant for forecasting in the current (most recent) distribution.
Instead of sampling uniformly over the history, our method learns a discrete distribution over past time steps and explicitly allows continuous adaptation, so that the model can react and adapt to distribution shifts.
Our contribution is an adaptive sampling mechanism that learns the sampling distribution over time steps via gradient-free Bayesian optimization.
We propose a two-step framework where the forecasting model is pre-trained with uniform sampling and then a lightweight architecture is trained with adaptive sampling.
We evaluate the properties and performance of this approach with experiments on synthetic and real-world data.
This paper is organized as follows: Section 2 introduces the reader
Workshop on Distribution Shifts, 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
to the background of this paper and the types of data distribution shifts for forecasting that we address. and Section 3 introduces our method and we present experiments in Section 4.
Section 5 concludes the paper.
Note that the discussion of the related work can be found in Section A.1 in the Appendix.
Background
Notation We denote the value of time series i at time t by z i,t , where z i,t ∈ R , and corresponding time-varying covariates as x i,t ∈ R d . The past of each time series (i.e., training set) is denoted as z i, 1: T :=
( z i, 1 , . . . , z i,T ) , and the future (i.e., forecasting window or test set) as z i,T +1: T + τ :=
( z i,T +1 , . . .
, z i,T + τ ) . The number of time series in the dataset is denoted by N .
Our main assumption in this work is that we are able to observe at least some of the forthcoming shift in order to be able to react and adapt to it:
Assumption 1 Let c be a constant number of (past) time steps set a priori.
We assume that there is no (marginal) distribution shift between the most recent training window of c steps and the forecasting (test) window.
More precisely, let z i,t be the given time series at time t and let ¯ z i,t be the time series at time t obtained after removing trend, seasonalities and other effects of available features, then p [Equation] i
[Equation]
T [Equation] This assumption implies that the data with t ∈ [ T − c, T ] is the closest in time to the forecasting window, therefore it is reasonable to regard it as the most similar to the future.
In practice, we take c to be equal to the sum of context and prediction lengths, so that we can use { z i,T − c : T } N i =1 as a validation set to learn and adapt to any distribution shift.
In general, we consider the following two scenarios illustrated in Figure 1): (a) A new or reoccurring shift in the past and no shift between the recent training and test window; and (b) a shift that occurred in the past but reverted back to the "standard" regime.
Note that we do not address the case where the distribution shift occurs in the test window only, which would be more of a fundamental generalization problem (case (c) in Figure 1).
See Section A.2 in the Appendix for a detailed definition of the cases that we consider.  Our Method
Training of deep learning models in time series settings is usually done by sampling fixed-size windows uniformly at random over the available time series history (i.e., the past). Since distribution shifts happen with different (possibly reoccurring) patterns over time, uniform sampling over the training set would likely feed the algorithm with irrelevant or even misleading data to forecast in the current distribution regime.
Therefore we propose to adaptively generate these training windows depending on the regime of the validation set (and by Assumption 1, the test set). The key element of our approach is automatically learning which regions to focus on to accurately forecast the test set.
[Figure] To achieve this, we introduce a discrete distribution p ϕ ( · ) over time steps 1 , . .
, T that determines the probability of sampling a training window starting at a time point t .
In practice, p ϕ ( · ) can be any discrete distribution that is well defined over integers, such as Geometric, Negative Binomial, Poisson, or mixtures of these distributions, parameterized by ϕ .
This distribution is responsible for sampling time windows that are used to train our forecasting model (see Figure A.1 in the Appendix for an illustration).
We learn ϕ by minimizing validation loss of the model that is trained using examples generated according to p ϕ ( · ) . Ideally, the learned distribution parameters ϕ ⋆ will avoid sampling distribution shifts in history that do not correspond to the current regime.
This distribution replaces uniform sampling over the history, which cannot distinguish between distribution shifts and normal regions.
However, our model should fall back to uniform sampling if this results in lowest validation error. This suggests that no distribution shift is present in the validation window and consequently in the forecast window (by Assumption 1).
Adaptive training of a lightweight architecture In practice, we would expect to frequently reestimate ϕ ⋆ to adapt to the current distribution.
To address this, we propose a two-step procedure with a lightweight adaptive architecture that is fast to train and can be frequently re-trained.
In this sense, this approach is explicitly considering a notion of time-varying parameters.
i)
The model is pre-trained using uniform sampling over the whole training history.
Intuitively, this step aims at learning common features that are useful everywhere in time.
ii)
After freezing the weights of the pre-trained model, an adaptive lightweight architecture is attached and trained using the adaptive sampling mechanism, where the training windows are generated according to p ϕ ∗ ( · ) .
The goal of this step is to learn time-specific features that are relevant to forecast in the current distribution.
The lightweight architecture can be a short sequence of fully connected layers, or it can even be a subset of the pre-trained model.
Learn ϕ via Bayesian Optimization We now describe how we estimate the parameters ϕ of the distribution p ϕ ( · ) . Let us denote the parameters of the pre-trained model as w pre and those of the lightweight architecture as w ada . Moreover, let f w ( { z i, 1: T } ) denote the forecasts { ˆ z i, T +1: T + τ } of the model whose weights are given by w .
Given input time series { z i,t } , i = 1 , . . .
, N and t = 1 , . . , T , we set aside the last τ observations { z i,T − τ +1: T } , ∀ i , for estimating ϕ . We learn ϕ by minimizing loss on this validation channel of the model trained according to p ϕ ( · ) . More precisely, we have
[Equation]
where ℓ ( ϕ ) is the validation loss of the model that is trained using adaptive sampling p ϕ ( · ) ,
[Equation]
Here w ∗ ada ( ϕ ) are the weights obtained after fine-tuning the pre-trained model on the input data { z i, 1: T − τ } where the training examples are generated according to the distribution p ϕ ( · ) ; note the dependence of w ∗ ada on ϕ .
A crucial observation is that one can only evaluate ℓ ( ϕ ) and cannot obtain its gradients with respect to ϕ ; hence we cannot train ( w ada , ϕ ) jointly end-to-end.
We propose to treat ϕ as hyper-parameters and use Bayesian optimization [9] to find ϕ ∗ .
Assumption 1 plays a key role in this step because the most recent region in the training set is used as validation channel to choose ϕ ⋆ during Bayesian optimization. The overall approach is depicted in Figure 2 and the implementation details with DeepAR [28] as the base model are given in Algorithm 1. In our experiments we assume w ada ⊂ w pre and hence the algorithm is presented for this case. 1: procedure Ada-DeepAR ( { z i,t } , i ∈
N ] t ∈ [
T ]
Z train = { z i,t } , ∀ i, ∀ t 3: w pre ← DeepAR ( Z train ) Z ada_train = { z i,t } , ∀ i, t ∈ [1 .. T − τ ] 5:
Z validation_labels =
{ z i,t } , ∀ i, t ∈ [ T − τ + 1 .. T ] 6: ϕ
= ϕ▷
Random initialization 7: w = w pre 8:
Split w into w ada and w frozen 9: while CONVERGENCE do ▷ Solve min ϕ ℓ ( ϕ ) 10: w ada ← DeepAR ( Z ada_train , init = w , sampler = p ϕ ( · ) , fixed = w frozen ) ▷ Train with adaptive train sampler by weights w frozen 11: w = w ada ∪ w frozen 12: ˆ Z validation_preds =
( Z ada train ,
= w ) 13: ℓ
▷
Optimization 15: end while likely accurately 16: w ▷ Model is to
{ z i,t } , ∀ i, t ∈ [ T + 1 .. T + τ ] 17: end procedure
The instantiation of our model in this manuscript is called Ada-DeepAR and uses DeepAR [28] as a base model with a two-layer LSTM (other base models could be used).
The adaptive part of our model is the second LSTM layer.
Additionally, we perform Bayesian optimization using MOBSTER, which combines asynchronous successive halving and asynchronous Hyperband with gaussian-process based Bayesian optimization [17].
We use the implementation available in Syne Tune [29].
For any time series i with training history T and forecast horizon τ , we use z i,T − τ : T as a validation channel for Bayesian optimization.
We consider a geometric and a mixture of two negative binomial distributions as our sampling distributions p ϕ ( · ) .
Training details are given in Section A.4.
Experiments on synthetic data: To demonstrate the applicability of our method, we first evaluate it on two synthetic experiments that mimic cases (a) and (b) of Figure 1.
We generated an artificial sinusoidal dataset and then injected shifts or noise in different regions.
Focus on shifted regions distant in time: In this example we injected a shift in a distant portion of the training dataset and also in the validation and test sets (see the top of panel a) in Figure 3).
We show that the sampling distribution over time steps can focus on specific regions that are relevant to the current domain, regardless of when they occurred in the available history (see b1) in Figure 3).
Exclude noisy regions: In this example, we injected noise (a negative constant) in two different regions of the training dataset.
Our adaptive sampling mechanism avoids these noisy regions from the history ( b2) in Figure 3).
Interestingly, this sometimes corresponds to sampling only in the most recent window, while in other cases the distribution exploits its bimodality to focus on two distant windows. [Table]
Experiments on real-world datasets: Finally, we show preliminary results on five real-world datasets (see Section A.3 for details).
We compare the following models: our two-step training procedure with adaptive sampling ( Ada-DeepAR ), reversible instance normalization ( RevIn ) [16], and the attention-based Temporal Fusion Transformer TFT [20]. We also consider an ablation variant of our method, Ada-DeepAR - uniform , which uses the same architecture and two-step procedure as Ada-DeepAR , but performs adaptive training (i.e., fine-tuning of the adaptive weights w ada ) using uniform sampling instead of p ϕ ( · ) .
This variant is included to highlight the effectiveness of learning the sampling distribution p ϕ
by isolating the gains obtained by our model due to the fine-tuning step.
For Ada-DeepAR , Ada-DeepAR - uniform and RevIn we use DeepAR [28] as the base model.
Ada-DeepAR - uniform is different from DeepAR because of the fine-tuning step where adaptive weights are again trained one more time; this might help correcting the overfitting of the DeepAR model (and reduce test error) because of the lower capacity of adaptive weights and/or help escape local minima (and reduce training loss).
We evaluate these probabilistic methods using the continuous-ranked probability score (CRPS) [11, 23].
Table 1 summarizes the results.
Averaged over all datasets, Ada-DeepAR improves over the base model DeepAR by 8.4% (relative improvement).
For three out of five datasets, the improvements over the base model are statistically significant (paired Student’s t -test p -value < 0 . 05 , see Table 1).
Ada-DeepAR selected the mixture of two negative binomial distributions for all datasets except Covid-deaths (for which the geometric distribution was selected).
RevIn also improves over DeepAR , but with a much smaller margin.
TFT performs best on Covid-deaths , but overall worse than the other methods on these datasets.
Interestingly, Ada-DeepAR - uniform also improves the error by 6.7% over its base model.
In particular, Ada-DeepAR - uniform is the best model on the Traffic and Taxi datasets.
Our hypothesis is that adaption to distribution shift is not necessary, and perhaps even counterproductive, for these datasets.
This is also suggested by the performance of RevIn , which performs similar to Ada-DeepAR on these datasets.
Note that a uniform distribution could be seen as one of the possible distributions p ϕ ( · ) that our adaptive architecture should consider and if no other distribution reduces the validation error, the uniform distribution should be selected.
The performance differences between Ada-DeepAR and
Ada-DeepAR - uniform are statistically significant for all datasets except Traffic .
Conclusion
We introduced an adaptive sampling mechanism over the training history, whose parameters are learned using Bayesian optimization.
Our experiments show that our method can improve the forecasting error over its base model (in this work, the improvements over the base model where statistically significant for three out of five datasets).
While we find Bayesian optimization to be a powerful technique to find good adaptive sampling parameters, this approach is computationally expensive (because many parallel trials need to be evaluated) and could degrade as the number of parameters increases.
Re-framing our approach as an input-selection layer that uses the reparametrization trick for sampling could allow for cheaper end-to-end learning of the adaptive model parameters and p ϕ ( · ) .
A Appendix A.1 Related Work
The problem of learning under distribution shift for time series has been extensively studied for many years. For example, change point detection analyzes when the statistical distribution of a time series changes [27]. Another related field is adaptive control, where the system parameters are continuously adjusted to account for discrepancies of expected and observed data [15].
We focus our review on recent methods that address distribution shifts when forecasting with neural networks.
Kim et al.
[16] employs a trainable instance-wise normalization-and-denormalization layer to address non-stationary mean and variance in time series forecasting.
Arik et al.
[2] proposes Self-Adaptive Forecasting to adapt the forecasting model using test-time training by "backcasting" and using the backcast errors to signal a potential distribution shift to adjust the weights of the model before inference.
AdaRNN [7] splits the time series history into dissimilar segments and learns importance weights to combine the
RNN hidden states over these segments.
Another notion of distribution shift, adversarial attacks, has been recently considered to build more robust forecasting [35, 21].
Another class of approaches to time series forecasting use specific weighting schemes to weight the time series history to make predictions, for example exponential smoothing, the Non-Parametric
Time Series (NPTS) Forecaster [10, 1], or stratified sampling [22].
In contrast to these models, our approach adaptively learns the weighting scheme over the history.
Thus, our work can be seen as a specific instantiaten of weighted re-sampling (or importance sampling) for time-series forecasting.
Importance sampling weights the log-likelihood in supervised learning tasks by importance weights w ( x ) =
p test ( x ) /p train ( x ) where p test and p train are the respective densities from which the train/test samples are drawn [31, 14, 34, 19].
Here, we propose to avoid estimating p train and p test and instead tune the parameters of a parametric re-sampling function using Bayesian optimization.
A.2 Scenarios of distribution shift occurrence
Assumption 1 allows us to provide a precise breakdown of the specific cases of distribution shift that our model can handle.
This is important because it determines the exact applicability of the work described in this paper.
As Figure 1 shows, we identify three domains:
New or reoccurring shift: When a distribution shift is new or reoccurring in the past and there is no shift between t ∈ [ T − c, T ] and t ∈ [ T + 1 , T + τ ] .
In this case the adaptive sampling mechanism is expected to focus only on the shifted regions that are relevant for the current regime.
Reverted shift: When a shift occurred in the past but the distribution eventually reverted back to the “standard” regime.
This case is complementary to the previous one: here the adaptive sampling mechanism provides a way to exclude the shifted regions.
One can see the shifts in this case as regions with noisy or corrupted data: excluding them during training would guarantee robustness against noise.
Future shift: When the shift occurs in the forecasting window but not in the most recent region before the forecast start date.
Our adaptive sampling mechanism cannot work if we have no information regarding the distribution shift happening in the test set (no free lunch).
[-, z i,T ) , and the future (i.e., forecasting window or test set) as z i,T +1: T + τ := 
-]
A.3 Datasets
We use datasets from the Monash Time Series Forecasting Repository [12] and GluonTS [1] for our experiments.
We use the same train/test splits as described in TSBench [4].
See Table A.1 for dataset statistics.
The frequencies of our used datasets are 30 minutes (30 MIN), hourly (H), and daily (D).
In particular, we use the following datasets:
Covid-deaths : Daily COVID-19 deaths of different countries between January 22 and August [Table]
Electricity : Hourly household consumption data between January 2012 and June 2014 [8]. Traffic : Hourly occupancy rates of freeways in the San Francisco Bay area between 2015 and 2017 [8].
Solar : Hourly power output of 137 photovoltaic power stations in the US [18] Taxi : Number of taxi rides in
different locations in New York sampled
at
30 minute
windows [33].
A.4 Training Details
The parameters of the distribution over history are optimized via Bayesian Optimization [9] using the recent scalable implementation provided in Syne-Tune [29].
We leveraged an AWS EC2 instance with ≈ 42 cores and set the number of random initializations to 44 to allow for sufficient exploration of the parameter space.
That is, the optimizer starts with 44 trials, each with a different parameter configuration, and sequentially stops some of them or instantiate new ones given the information from the Gaussian Process posterior.
The optimizer stopping criterion was chosen to be the maximum number of trials taken to completion, which was set to 200.
We used the CRPS used to decide which trials were stopped or continued was the mean quantile loss over the validation [-set.
-]set.
The lightweight architecture is supposed to be simple enough to allow frequent re-training through adaptive sampling, so that the forecasting model can quickly adapt to new distributions.
Clearly, different architectures will work better with different datasets.
In all our experiments with DeepAR [28], we found that using the default two-layer LSTM with 40 hidden units for the pre-trained model, and freezing the first layer only for adaptive training yielded the best results.
To clarify, this means that w ada ⊂ w pre .
Both pre-training and adaptive training were done overepochs with a small dropout probability equal to 0 .
For TFT , we selected the hyperparameter setting from the TSBench [4] that resulted in lowest CRPS loss on the validation set for each dataset.
The hyperparameters tuned in TSBench are the context length, the number of hidden states, and the number of heads.
For the DeepAR and TFT models, we use the implementation available in GluonTS [1]. A.5 Additional Figures