O NLINE C ONTINUAL L EARNING WITH F EEDFORWARD A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently deep learning has been widely used in time-series prediction tasks.
Although a trained deep neural network model typically performs well on the training set, performance drop significantly in a test set under slight distribution shifts.
This challenge motivates the adoption of online test-time adaptation algorithms to update the prediction models in real time to improve the prediction performance.
Existing online adaptation methods optimize the prediction model by feeding back the latest prediction error computed with respect to the latest observation.
However, the feedback based approach is prone to forgetting past information.
In this work, we propose an online adaptation method with feedforward compensation, which uses critical data samples from a memory buffer, instead of the latest samples, to optimize the prediction model.
We prove that the proposed approach has a smaller error bound than previously used approaches in slow time-varying systems.
The experiments on several time-series prediction tasks show that the proposed feedforward adaptation outperforms previous adaptation methods by 12% .
In addition, the proposed feedforward adaptation method is able to estimate an uncertainty bound of the prediction that is agnostic from specific optimizers, while existing feedback adaptation could not.
I NTRODUCTION
Time-series prediction (or forecasting) has been widely studied in many fields, including control, energy management, and financial investment Box et al.
; Brockwell & Davis (2002).
Among the research applications, acquiring future trends and tendencies of the time-series data is one of the most important subjects.
With the emergence of deep learning, many deep neural network models have been proposed to solve this problem Lim & Zohren (2021), e.g., Recurrent Neural Networks
Lai et al.
and Temporal Convolutional Networks Bai et al.
Inspired by the great success of Transformer in the NLP and CV community Vaswani et al.
; Dosovitskiy et al.
Transformer-style methods have been introduced to capture long-term dependencies in time series prediction tasks Zhou et al.
Benefiting from the self-attention mechanism, Transformers obtain a great advantage in modeling long-term dependencies for sequential data Brown et al.
Although a trained Transformer model (or other big deep neural network models) typically performs well on the training set, performance can significantly drop in a slightly different test domain or under a slightly different data distribution Popel & Bojar (2018); Si et al.
In practical time-series prediction problems, there are often significant distributional discrepancies between the offline training set and the real-time testing set.
These differences may be attributed to multiple factors.
In some cases, it is too expensive to collect large unbiased training datasets, e.g., for weather prediction or medical time-series prediction.
In other cases, it may be difficult to obtain the training instances from a specific domain.
For example, in human-robot collaboration, it is hard to collect data from all potential future users.
In these cases, adaptation techniques are applied to deal with the distribution mismatch between offline training and real-time testing Blum (1998).
Besides, some tasks require the system to adapt itself after every observation.
For example, in human-robot collaboration, the robot needs to continually adapt its behaviors to different users.
In these scenarios, online adaptation techniques are often embraced Abuduweili et al.
Online adaptation is a special case of online continual learning, which continually learns from real-time streaming data.
In online adaptation, a prediction model receives sequential observations, and then an online optimization algorithm (e.g. SGD) updates the prediction model according to the prediction loss measured by the observed data.
The goal of online adaptation is to improve prediction accuracy in subsequent rounds.
Online adaptation is currently applied to many kinds of research like time-series prediction Pastor et al.
; Abuduweili & Liu (2020), image recognition
Lee & Kriegman (2005); Chen et al. (2022), and machine translation Mart´ınez-G´omez et al.
In this paper, we mainly focus on time-series prediction tasks, but the proposed methods also can be used for other online adaptation (or online learning) tasks.
Most existing online adaptation approaches are based on feedback compensation Tonioni et al. (2019), analogous to feedback control.
In feedback adaptation, a prediction model only utilizes the latest received data.
After observing a new sample, the online optimization algorithm updates the prediction model according to the prediction loss measured between the last prediction and the latest ground truth.
However, this kind of passive feedback compensation is not efficient.
In this work, we propose feedforward compensation in online adaptation to maximize information extraction from existing data, especially those that are more critical.
A critical sample is more helpful to reduce the objective (loss) of the model when the sample is selected for training.
In the proposed feedforward adaptation , we will not only have forgetting as is done in conventional online adaptation Paleologu et al.
but also enable recalling to compensate for potential shortsighted behaviors due to forgetting.
There is a balance between forgetting and recalling.
On the one hand, to rapidly learn the new function value in a time-varying system, we need to forget some of the old data.
On the other hand, too much forgetting may cause unstable and incorrect predictions when we encounter a similar pattern with historical data.
To achieve the balance between forgetting and recalling, we design a novel mechanism for feedforward compensation using a memory buffer similar to the functionality of the Hippocampus in the human brain Barron (2021).
We will maintain the memory buffer by storing recent L -steps observations (or hidden features) of samples.
When the prediction model experiences similar observations, it will pull the corresponding data (critical sample) from the memory buffer to enhance learning.
For example, in human behavior prediction tasks, a human subject may exhibit similar behavior patterns on different days.
These would be extremely difficult to discover if we only learn from the most recent data like conventional online adaptation but can be identified using the feedforward adaptation methods with memory buffer.
We can also use the related information between the current sample and critical samples to measure the uncertainty bound to the current prediction.
Our main contributions can be summarized in the following points
• By summarizing feedforward and feedback adaptation methods, we provide a general online test-time adaptation framework and prove its error bound.
• We propose a feedforward compensation for online test-time adaptation problems.
We prove that the proposed feedforward adaptation method has a smaller error bound than previously used feedback methods.
• We propose an uncertainty-bound estimation related to the feedforward approach, which is agnostic from specific optimizers.
• We conduct extensive experiments to show that the proposed feedforward adaptation is superior to conventional feedback adaptation.
P ROBLEM O VERVIEW 2. T IME - SERIES PREDICTION
The time series prediction problem is to make inferences on future time-series data given the past and current observations.
We consider a multi-step prediction problem: using recent I steps’ observations to predict future O steps’ data.
Assume the transition model is composed of a feature extractor (or Encoder) E and a predictor (or decoder)
f .
At time step t , the input to the model is
X t =
[ x t − I +1 , x t − I +2 , · · · , x t ] , which denotes the stack of I -step recent observations.
The output of the model is Y t +1 =
[ y t +1 , y t +2 , · · · , y t
+ O ] , which denotes the stack of O -step future predictions.
The observations x t , y t are vectors that may contain trajectory or feature, and x t = y t for some cases (e.g. univariate prediction).
The transition model for time series prediction can be formulated as
[Equation]
where Z t is a hidden feature representation of input X t .
Feature extractor E does not change over time, while predictor f t changes over time.
Let f t denote the ground-truth predictor, that generates ground-truth output Y t +1 .
In online adaptation, we use the parameterized model (e.g. Neural Networks)
ˆ f ( θ t , Z t ) with learnable parameters θ t to estimate the ground-truth predictor f t ( Z t ) .
This paper assumes that the encoder E is fixed in online adaptation, which could be trained offline before online adaptation or be a non-parametric feature extractor 1 .
2. O NLINE ADAPTATION
Due to the temporal nature of time series prediction, the output (future) space in prediction tasks is not fixed. Since the train set and test set are split in chronological order, train-test distribution mismatch is very common in time-series prediction. Thus, online test-time adaptation is crucial to overcome the distribution mismatch problems and make the prediction robust to time-varying and heterogeneous behaviors.
Online adaptation also can be called adaptable prediction , since it makes an inference concurring with updating model parameters.
Online adaptation explores local overfitting to minimize the prediction error: at time step t , the prediction error e t +1 is to be minimized.
The optimization objective is shown below
[Equation]
where f t ( Z t ) =
Y t +1 =
[ y t +1 , y t +2 , · · · , y t
+ O ] is the ground truth observation (to be received in the future) and ˆ f ( θ t , Z t ))
=
ˆ Y t +1 =
[ˆ y t +1 , ˆ y t +2 , · · · , ˆ y t
+ O ] is the predicted outcome from the learned model parameter θ t .
The adaptation objective can be in any ℓ p norm. In conventional (feedback) online adaptation, the objective of minimizing the prediction error in the future Eq. (3) can be approximated by minimizing the fitting error in the past, as shown below
[Equation] where 0 < λ ≤ 1 is a forgetting factor.
The model parameter θ t is updated iteratively when new observations are received.
Then a new prediction is made using the new estimate.
In the next time step, the estimate will be updated again given a new observation, and the process repeats.
It is worth noting that the observation we received at time t is y t .
The other terms in Y t remain unknown.
This paper focused on adaptation methods using one-step-ahead observation.
It is also possible to conduct online adaptation with multi-step ahead observations Abuduweili & Liu (2021).
In this paper, we propose a feedforward adaptation method, whose objective is different from feedback adaptation. In the feedforward adaptation, the objective of minimizing the prediction error in the future Eq. (3) can be approximated by minimizing the upper bound of the prediction error in the future, as shown below [Equation]
We will show the algorithms and effectiveness of the feedforward adaptation in section 3. In specific adaptation algorithms, the feedback and feedforward approach differ by sample selection strategy. Algorithm Input: Initial predictor f ( θ 0 , :) with parameters θ 0 , Feature Extractor E , Optimizer O (: , : , :)
Output: Sequence of predictions { ˆ Y t +1 } Tt = 1: for t = 1 , 2 , · · · , T do 2:
Receive the ground truth observation values x t , y t 3:
Find the critical input-output pairs ( Z s , y s +1 ) for 1 ≤ s < t , where Z s =
E
( X s ) 4:
Adaptation: θ t =
O ( θ t − 1 , ˆ y s +1 , y s +1 ) , where ˆ Y s +1 =
[ˆ y s +1 , · · · , ˆ y s + O ] =
f ( θ t − 1 , Z s ) 5:
Prediction: ˆ Y t +1 =
[ˆ y t +1 , · · · , ˆ y t
+ O ] = f ( θ t , Z t ) , where Z t = E ( X t )
We provide a general online adaptation framework as shown in algorithm 1, by incorporating conventional feedback with the proposed feedforward adaptation.
At time step t , after receiving the current observations ( x t , y t ) , we select the critical input-output pair ( Z s , y s +1 ) from historical observations.
The critical pair was used to adjust the parameters of the prediction model by an online optimizer (e.g. SGD).
Then we obtain the current prediction result with the adapted model.
The main difference between different adaptation algorithms lies in the critical pair selection strategy (line 3 in algorithm 1).
In feedback adaptation, the critical input-output pairs are composed by the latest observations Z s =
Z t − 1 .
In random adaptation, the critical input-output pairs are randomly sampled from historical observations Z s ∼ [ Z 1 , · · · , Z t − 1 ] .
We will show in section 3, in feedforward adaptation, the critical input-output pairs are the most similar samples to the current observation
Z s = arg min Z ∥
Z t − Z ∥ .
F EEDFORWARD A DAPTATION A PPROACH 3. E RROR BOUND FOR GENERAL ONLINE ADAPTATION
The theoretical analysis of the paper is based on two basic conditions about the local smoothness property of ground-truth predictors f t .
K -Lipschitz continuity condition .
For a time-step t and ∀ s ∈ [ t − L, t − 1] , we have local K
Lipschitzness for ground-truth prediction function f t and recent L steps input data:
[Equation]
(6) where K is the bound (real number) for the change of the value of the function over input space.
Intuitively, a Lipschitz continuous function is limited in how fast function value can change over input space.
It is proven that every function that has bounded first derivatives is Lipschitz continuous
Sohrab (2003).
Similar to Eq.
(6), we assume our parameterized function ˆ f t ( θ t , :) (e.g. Neural Networks) has Lipschitz continuity with constant value
[Equation]
K-Lipschitz continuity is common in machine learning because neural networks have bounded first derivatives by proper training.
δ time-varying condition.
For a time-step t and ∀ s ∈ [ t − L, t − 1] , assume the ground-truth prediction functions f t has bounded changes within recent L steps under the same input Z s :
[Equation]
where δ is the bound (real number) for change of the value of the function sequences over time on a fixed input.
A bounded δ time-varying condition is a common condition in time-series tasks because we obtain similar future time-series signals with the same input in most cases.
Theorem 1 (Error Bound of Online Adaptation) .
Under the K -Lipschitz continuity condition (Eq.
(6) and (7)) and δ time-varying condition (Eq. , the (prior) prediction error e t +1 of general online adaptation (algorithm 1) has the following upper bound:
[Equation]
Please check the proof in appendix C.1.
3. F EEDFORWARD A DAPTATION
As discussed in section 2.2, the goal of the online adaptation is to minimize the prediction error in the future Eq.
Due to the lack of ground-truth value in the current steps, it is not feasible to directly minimize the prediction error.
Conventional online adaptation methods approximate the original objective by minimizing the fitting error in the past Eq.
In this work, we approximate the original objective by minimizing the bound of the prediction error Eq. Which is equivalent to optimizing the worst-case scenarios. The worst-case performance analysis is really useful in real-world applications Roughgarden (2021).
We will show that, under the following assumption, feedforward adaptation provides better results than feedback adaptation.
Temporarily Slow Time-varying Assumption. For the δ time-varying function f t , assume δ is time-independent or it slowly changes within recent L -steps: ∂δ∂t ≈ 0 .
Note that, we only assume the ground-truth function f t varying slowly (locally) within recent Lsteps not (globally) for every step.
Thus the assumption is reasonable because the major changes in function value are caused by the change of inputs instead of the time-dependency in many realworld prediction problems.
In addition in some cases, δ is caused by random noises, then which still follow our assumptions.
Under the above assumption, we can ignore the term δ in optimization.
Then the optimization objective for the error bound is shown below:
[Equation]
However, we still can not directly minimize the above objective, because we do not know K, ˆ K .
Then a simplification is applied to the above objective to make it applicable.
We change the joint minimization over s and θ t to a bi-level optimization which first minimizes the first term of objective over sampling time-step s , then minimizes the second term of objective over parameter θ t .
Thus, the simplified objective function is shown below: [Equation]
[Equation]
In a summary, the proposed feedforward adaptation method selects the most similar samples to the current observation as critical pair ( Z s ⋆ , y s ⋆ ) by Eq.
Then using the critical pair to optimize the prediction model by Eq.
3. U NCERTAINTY E STIMATION
The error bound Eq. (9) provides uncertainty estimation of the prediction results.
Here we use estimation of ˜ K t and ˜ δ to approximate real K and δ in Eq.
We use confidence factor σ ∈ (0 , 1] to decay the error bound .
The uncertainty estimation ˆ U t +1 for prediction ˆ Y t +1 is shown below:
[Equation]
Where s = t − 1 for feedback adaptation and s = s
⋆ (Eq. (13)) for feedforward adaptation.
The confidence factor σ is a predefined hyperparameter, e.g. σ = 0 .
9 for 90% confidence of uncertainty.
˜ δ and ˜ K t are also predefined hyperparameters, e.g. ˜ δ = 0 . 001 , ˜ K = 1 .
For more accurate uncertainty estimation, we can set ˜ δ as a small fixed value, and iteratively update ˜ K t according to the estimated uncertainty ˆ U t and real error e t .
If the previous uncertainty estimation is much larger than the real error then we shrink the ˜ K t value. If the previous uncertainty estimation is much smaller than the real error then we enlarge the ˜ K t value.
The ˜ K t estimation criteria are shown below.
[Equation]
In adaptation and prediction, feedforward adaptation does not require K and δ values.
In the uncertainty estimation, we need to approximate K and δ or serve these as predefined hyperparameters, but an accurate approximation is not necessary.
Since we mainly consider the relative uncertainty estimation between different samples or different optimizers.
As shown in Eq.
(14), the inaccuracy of ˜ K and ˜ δ mainly raise a data-independent error, which is no impact on the data-selection strategy and optimizers.
The overall feedforward adaptation algorithm is shown in algorithm 2.
The 1 ∼lines of algorithm 2 is corresponding to the 1 ∼ 5 lines of algorithm 1. Buffer B is used to store recent L -step observations. ⋆ + y s ⋆ + O ] f t 3. E FFECTIVENESS OF F EEDFORWARD A DAPTATION
As described in section 3.2, under the temporarily slow time-varying assumption, feedforward adaptation achieves a smaller error bound than feedback adaptation.
In this section, we analyze more theoretical results on the online adaptation of some specific functions.
Linear time-variant function . Consider a function f t which can be separated into time-invariant function g ( z t ) and linearly time-dependent part αt .
Input z t is a random variable from the uniform distribution U (0 , 1) .
[Equation]
Lemma 1 .
We use neural network ˆ f ( θ, :) to learn the linear time-variant function Eq.
Let K be a Lipschitz constant for the ground-truth function f t , and ˆ K is a Lipschitz constant for neural network ˆ f ( θ, :) . Let B fbe and B ffe denote the prediction error bound for feedback and feedforward adaptation.
Then we have the following results.
(a) Expectation of the error bound for feedback adaptation is E [ B fbe ] = K + ˆ K+ α . (b) Expectation of {+the error bound for feedforward adaptation is E [ B ffe ] ≤ ( K + ˆ K ) · max( 1 t , 1 L +1 ) + α · min( t 2 , L +12 ) . With L = (cid:113) 2( K + ˆ K ) α − 1 , we achieve the smallest final error bound for feedforward adaptation: E [ B ffe ] ≤ (cid:113) 
+}
α ( K + ˆ K ) .
{+(c) If time-varying factor α is smaller, specifically α ≤ α ⋆ = 4 −√ 
 [Equation] adapta [Equation]+}
tion has smaller error bound than feedback adaptation. Time-invariant polynomial function . Consider a time-invariant polynomial ground-truth function f t ( z t ) with the input of random variable sampled from the uniform distribution U (0 , 1) . [Equation]
Lemma 2 .
We use linear projection ˆ f ( V t , z t ) =
V t z t to learn the time-invariant polynomial function Eq. Then we have the following results. (a) Expectation of the prediction error for feedback adaptation is E [ e fbt +1 ] ≥ 13 N − 1 N 2 (cid:80) N − 1 i
=1 W i +1 . (b) Expectation of the prediction error for feedforward adaptation is E [ e ff t +1 ] ≤ 12 max( 1 t , 1 L +1 ) (cid:80) N − 1 i =1 W
i +1 . The final error converges lim L →∞ ,t →∞ E ( e ff t +1 ) = 0 . (c) Feedforward adaptation is provably better than feedback adaptation.
The proof of Lemma 1 and Lemma 2 are provided in appendix C.2.
Thus feedforward adaptation can be used to learn time-invariant functions and slow time-varying systems.
These kinds of systems are very common in the world.
For example, the exchange rate and incidence rate for common diseases have a slow time-dependent shift over time.
E XPERIMENTS 4. S YNTHETIC E XPERIMENTS
To evaluate Lemma 1 in section 3. [Figure] we consider the following time-varying function.
[Equation]
According to Lemma
[Figure] we derive the error bound for feedback adaptation B fbe [Figure] and feedforward adaptation B ffe is shown below (please check details in appendix D.1).
[Equation]
Then we calculate the threshold α ⋆ .
If α ≤ α ⋆
[Figure]
feedforward adaptation has a smaller error bound. [Equation] [Figure] Figure 1a shows the average prediction error of feedback and feedforward adaptation under different time-varying factors α .
Note that, feedforward adaptation with L = 1 is the same as feedback adaptation.
We will show that the results in Fig. 1a correspond to the theoretical results from Lemma 1. (1) As can be seen, for a smaller time-varying factor α , feedforward adaptation with a larger buffer size L achieves a smaller prediction error. Besides, if α is larger, the performance of the feedforward adaptation drops.
These are consistent with Eq.
For example, feedforward adaptation with buffer size L = 100 achieves the smallest prediction error when α < 10 − 3 .
If α < 0 . 1 , feedforward adaptation with buffer size L = 3 performs better than feedback adaptation. But if α > 0 . 1 , feedback adaptation is better.
The threshold α ⋆ ≈ 0 .
1 is consistent with Eq.
Figure 1b illustrate the comparison between feedforward and feedback adaptation.
The first row shows the prediction results for α = 10 − 3 target function of feedforward adaptation with L =(b.1 subfigure) and feedback adaptation (b.3 subfigure).
As can be seen, feedforward adaptation can learn the ground truth more precisely.
The second row shows the prediction results for α = 10 − 2 target function.
For this case, prediction with feedforward adaptation (b.2 subfigure) has a significant shift from the ground truth, although it is still able to learn some input-dependent details.
Compared to feedback adaptation, feedforward adaptation is more focused on the input-dependent part of the ground truth function and pays less attention to the time-varying shift of the function.
These empirical results validate the conclusions in section 3.4. The experimental evaluation of
Lemma 2 is shown in appendix D.2.
4. E XPERIMENTS ON REAL WORLD DATA
In this section, we evaluate the proposed feedforward adaptation on four real-world benchmarks, including ETT (Electricity Transformer Temperature), Exchange-Rate, ILI (Influenza-like Illness), and THOR human motion trajectory dataset.
We include two models as our learnable prediction function ˆ f ( θ, :) :
Informer Zhou et al.
(2021) and simple MLP (multi-layer perception).
Experimental Design .
First we offline train models (Informer and MLP) on a train set.
After training, we will incrementally receive the data point from the test set like in real-world applications.
At each time step, we conduct an online adaptation to optimize the model with selected previous observations by feedback or feedforward compensation.
Then the prediction output is inferred from the updated model.
We evaluate the prediction results with mean squared error (MSE) and mean absolute error (MAE).
In the experiments, we only adapt the decoder of the model and make the encoder fixed.
Please check the detailed experimental design in appendix D.3.
Baselines .
We compare the proposed method with four baselines.
1) w/o adapt directly conduct prediction without adaptation.
Which is a lower bound for adaptation methods.
Feedback adaptation is the most important baseline to us.
3) Random adaptation is a method that selects the critical pair from the L -size buffer with random sampling.
Full adaptation is a method that uses all samples from the buffer to adapt the model, which is similar to offline training.
Results .
The detailed results of the experiment on SGD optimizer are shown in table 1.
In table 1, the last row denotes the performance gain of the proposed feedforward adaptation over feedback adaptation in terms of average result.
Feedforward adaptation achieves the best results on all four datasets.
Specifically, feedforward adaptation outperforms feedback adaptation by 12 . 6% , 7 . 4% , 22 . 34% , and 8 . 1% in terms of average MSE on four datasets respectively.
Feedforward adaptation is better than full adaptation, which means adapting models with more samples is not effective, because online adaptation focuses on the ability to rapidly learn and adapt in the presence of non-stationarity instead of generalization ability like offline learning. Feedforward adaptation is better than feedback and random adaptation, which means the sample selection strategy based on sample similarity in feedforward adaptation is more critical than the time-based sample selection and random sampling.
[Table] We found another interesting result by comparison between Informer and MLP on table 1.
Without adaptation, the simple MLP outperforms the Informer on average.
This is caused by the overfitting of the Informer on the training set.
However, by online adaptation, especially with feedforward adaptation, the performance of Informer is greatly increased.
For example, the performance of
Informer on the ETTh1 dataset increased from 0.211 to 0.109 by 48% and outperform the MLP model.
That shows Informer has greater representation ability but may perform poorly on the test set.
Considering this case, it is embarrassing to directly use the big Transformer-like models on real-world time-series prediction.
But online adaptation may make the big Transformer-like models powerful again.
This phenomenon emphasizes the importance of online adaptation in real-world time-series prediction. [Figure] Discussion .
We will show that the sample selection strategy in the proposed feedforward adaptation method could intrinsically mine the periodicity of the input data. Figure 2(a) shows the ETTh1 timeseries data, and Fig. 2(b) is the FFT (Fast Fourier Transformation) period analysis of the ETTh data. As can be seen ETTh1 has roughly T ≈ 500 repetition periods. Random adaptation randomly selects samples from the buffer, then the time shift between current time-step t and the time-step s for selected samples is ( t − s ) ∼ [1 , L ] ; Feedback adaptation only selects the last time sample to optimization, then ( t − s )
= 1 . This can be found from the distribution of time shift ( t − s ) in Fig. 2(c). For the proposed feedforward adaptation in Fig. 2(d), many samples were selected from ( t − s ) ≈ 500 steps earlier, which corresponds to the repetition period of T ≈ 500 .
Because feedforward adaptation selects the most similar samples to the current sample, which helps to extract the hidden periodicity of the input signal over time.
Thus the distribution of t − s is similar to FFT period analysis.
Experiments on different datasets are shown in appendix D.5.
According to theorem 1 (Eq.
(10)), the error bound is related to the sample difference ∥
Z t − Z s ∥ .
Figure 2(e) shows the real prediction error for different adaptation methods over time.
As can be seen, feedforward adaptation has the smallest prediction error, because feedforward adaptation has the smallest sample difference ∥
Z t − Z s ∥ during adaptation, as shown in Fig. 2(f).
One of the advantages of the proposed feedforward adaptation is that it could provide uncertainty estimation.
The results of prediction output and uncertainty are shown in appendix D.6.
CONCLUSION
This paper studies an effective feedforward adaptation algorithm for time-series prediction tasks.
Firstly we propose the general framework for online adaptation which includes feedback and feedforward adaptation.
Then we propose the feedforward adaptation algorithm by selecting the most similar critical samples for optimization.
We prove that, in a time-invariant or slow time-varying system, the feedforward adaptation has a smaller error bound than conventional feedback adaptation.
In the experiments, we empirically validate the effectiveness of the proposed algorithm both on synthetic and real-world data.
In the end, we show that online adaptation can greatly improve the performance of Informer or other models. A R ELATED W ORKS
Time series prediction.
Time-series prediction historically has very diverse applications across various domains Lim & Zohren (2021).
These applications trigger many methods for time-series prediction.
Modern machine learning methods provide a purely data-driven manner without domain knowledge Ahmed et al.
In recent years, deep neural networks becomes the dominant approach inspired by notable achievements of deep learning in computer vision Dosovitskiy et al. and natural language processing Vaswani et al. Among deep neural networks,
Transformer-style models are becoming more and more popular in time-series prediction tasks, since which has the potential to capture long-term dependencies Brown et al.
Most Transformerstyle models focus on long-term time series forecasting problem, including Informer Zhou et al.
(2021), Autoformer Wu et al.
(2021), and Pyraformer Liu et al.
Continual Learning also known as Lifelong learning, aims at providing incrementally updated knowledge in an ever-changing environment.
From the data statistics perspective, continual learning studies the problem of learning from non-i.i.d data, with the goal of preserving and extending the acquired knowledge.
Conventional continual learning assumes the new data arrive one task at a time, and the data distribution for each task is stationary De Lange et al.
Which is also called offline continual learning.
Different from the batch training on offline continual learning, Online
Continual Learning focuses on the more realistic problem, where data arrive one tiny batch at a time and previously seen batches from the current or the previous tasks are not accessible Mai et al.
Therefore, online continual learning is required to efficiently learn from a single sample from the online data stream in a single step.
Recent works of continual learning can be roughly divided into four families: dynamic architectures, regularization-based, rehearsal, and generative replay approach Lesort et al.
Among these methods, Rehearsal-based methods are more related to our work, which use memory replays to enhance the knowledge from the previous data or processes such as iCaRL Rebuffi et al.
(2017), GEM Lopez-Paz & Ranzato (2017).
However, most of these methods focus on offline continual learning.
Test-time Adaptation is a special setting of domain adaptation and continual learning where a trained model on the training domain has to adapt to the testing domain without accessing the training set Kundu et al.
Most test-time adaptation methods for image data utilize unsupervised domain adaptation algorithms to improve the performance on the test set such as entropy minimization Wang et al.
(2020), pseudo-labeling Li et al.
For the time-series prediction task, the input signal at the current step might be the ground-truth label of the previous step.
So (partially) supervised learning can be used for test-time adaptation on time-series tasks Abuduweili & Liu (2021).
In many real-world applications, online test time adaptation (online adaptation for short) is more reasonable than offline adaptation.
For example, in automotive driving, we only receive an observation (sample) in an online manner at each time step, instead of batch data of offline setting.
Online Adaptation continually learns from real-time streaming data.
Online adaptation is a special case of online continual learning in that the model is always offline trained on a training dataset (which can be very small), and then incrementally adapt its models in an online test dataset Cheng et al.
Unlike works that focus on avoiding catastrophic forgetting in offline continual learning Kirkpatrick et al.
(2017), online adaptation focuses on the ability to rapidly learn and adapt in the presence of non-stationarity.
Most existing online adaptation approaches are based on feedback compensation with SGD (Stochastic Gradient Descent) or EKF (Extended Kalman Filter) optimizers
Bhasin et al.
In algorithm 1, an online optimizer was used to adjust the model with the selected critical samples: θ t =
O ( θ t − 1 , ˆ y s +1 , y s +1 ) .
Unlike offline machine learning, online optimizers need to handle iterative input-output pairs with a single sample.
We assume our objective function (loss function) has suitable smoothness properties (e.g. differentiable and locally Lipschitz continuous).
SGD and EKF are two widely used optimizers for online adaptation.
Stochastic Gradient Descent (SGD) is one of the widely used optimizers in online adaptation
Kivinen et al.
SGD is memory efficient and stable.
The update formula of the SGD is shown below: [Equation]
[Equation]
where η is the learning rate.
Extended Kalman Filter (EKF) is another strong optimizer in online adaptation Jazwinski (2007).
In the original EKF, the object being estimated is the state value of a dynamic system.
In online adaptation, we can apply the EKF approach to adapt model parameters by regarding model parameters as system states.
The EKF approach has a faster convergence rate than SGD, but it is very memoryexhaustive.
When optimizing small neural networks, EKF approach has been demonstrated to be superior to the SGD-based algorithms Ruck et al.
; Abuduweili & Liu (2021).
If the adapted model size is not big, EKF is a strong optimizer for online adaptation.
The update formula of the
EKF is shown below:
[Equation]
where λ ∈ (0 , 1] is a forgetting factor, H t is the gradient matrix, G t is the Kalman gain, and P t is a matrix representing the uncertainty in the estimates of parameters.
The initial value of the matrix P t can be set as a diagonal matrix P
= p 0 · diag(1 , 1 , · · · , 1) for p 0 > 0 .
C T HEORETICAL R ESULTS ON SECTIONC. T HEOREM 1 (E RROR B OUND OF O NLINE A DAPTATION )
Bound of ground-truth difference .
If the sereis of ground-truth functions f t within recent L steps follows K -Lipschitz continuity Eq. (6) and δ time-varying conditions Eq.
[-+
-](8), then the ground-truth value Y t +1 and Y s +1 has the following property:
[Equation]
The proof is shown below:
[Equation]
Error Bound of Online Adaptation .
For time step t , the (prior) prediction error e t +1 has the following inequality:
[Equation]
The first two terms come from the difference between ground-truth Y t +1 − Y s +1 , the third term is a (posterior) fitting error for input-output tuple ( Z s , Y s +1 ) , and the last term is the difference between two predictions.
Combining Eq.
(29) with Eq.
(7), we obtain the error bound for general online adaptation is shown below:
[Equation]
C. E FFECTIVENESS OF F EEDFORWARD A DAPTATION C.2. E RROR BOUND OF ONLINE ADAPTATION ON LINEAR TIME - VARIANT FUNCTION
In this section, we compare the error bound of feedforward and feedback adaptation methods on linear time-variant systems (functions). Consider a function f t which can be separated into timeinvariant function g ( z t ) and linearly time-dependent part αt . Input z t is a random variable from the uniform distribution U (0 , 1) .
[Equation]
We use neural network ˆ f ( θ, :) to learn the above function.
Let K be a Lipschitz constant for f t (it is also equal to the Lipschitz constant for the function g ), and ˆ K is a Lipschitz constant for neural network
ˆ f ( θ, :) . The δ time-varying condition becomes ∥ f t ( z s ) − f s ( z s ) ∥
= α ∥ t − s ∥ , and time-varying factor δ = α ∥ t − s ∥ .
Then we have an error bound from Eq.
[Equation]
Now we consider the comparison of the error bound between feedforward and feedback adaptation.
Note that, the last term of the above equation ∥
y s +1 − ˆ f ( θ t , z s ) ∥ is a fitting error on input-output tuple ( z s , y s +1 ) , and the fitting error is irrelevant to feedback or feedforward compensation strategy. Besides, in over-parameterized neural networks, the fitting error is very small, even can be zero Allen-Zhu et al.
Thus we ignore the fitting error in comparison.
We now compare the expectation of the error bound. Let B e denote the prediction error bound for online adaptation, B e := Bound[ e t +1 ] .
Then we have the expectation of the error bound for online adaptation as shown below.
[Equation]
Then we use Eq. (34) to derive the expectation of the error bound for feedback and feedforward adaptation methods. The core is measurement of E [ ∥ z t − z s ∥ ] and E [ | t − s | ] . Expectation of the error bound for feedback adaptation .
In feedback adaptation, the selected input-output pairs are the latest observations z s =
z t − 1 and s = t − 1 . The current sample z t and last sample z t − 1 are independent random variables from U (0 , 1) .
The expectation of the distance between these two independent variables is 13 , then E [ ∥ z t − z s ∥ ] = E [ ∥ z t − z t − 1 ∥ ] = 13 .
We have the expectation for the error bound of feedback adaptation B fbe as shown:
[Equation]
Expectation of the error bound for feedforward adaptation . In feedforward adaptation, the selected input-output pairs are the most similar samples to the current observation z
⋆s = arg min z s ∥
z t − z s ∥ from L -size buffer, and s ⋆ = arg min s ∈
[ t − L,t − 1] ∥ Z t − Z s ∥ . The expectation E [ ∥ z t − z s ∥ ] represents the average minimum distance between current sample z t and previous samples.
If t ≤ L , E [ ∥ z t − z s ∥ ] considers the minimum distance for t samples, which is no greater than 1 t .
If t > L , E [ ∥ z t − z s ∥ ] considers the minimum distance for L + 1 samples in the buffer, which is no greater than 1 L +1 .
Similarly, E [ ∥ t − s ∥ ] is an average distance between t indices for t ≤ L , which is t 2 , or average distance from L + 1 indices for t ≥ l , which is L +12 .
We have the expectation for the error bound of feedforward adaptation B ffe as shown:
[Equation]
We mainly consider the final performance t > L , then the above equation can be simplified as:
[Equation]
With [Equation] − 1 , we achieve the smallest error bound for feedforward adaptation:
[Equation] [Equation]
When is feedforward better than feedback .
Similar to the worst-case analysis, we derive the conditions for feedforward adaptation has a smaller error bound than feedback adaptation.
To find the condition for E [ B ffe ] ≤ E [ B fbe ] , we solve the following inequality
[Equation]
Then we obtain a δ time condition on linear time-variant function for feedforward adaptation:
[Equation]
As summary, if time-varying factor α of linear time-variant function Eq. (32) is smaller, specifically α ≤ 4 −√ 12( K + ˆ K ) , feedforward adaptation has smaller error bound than feedback adaptation. α ⋆ =
[Equation]
√ 12( K + ˆ K ) is a threshold for selecting whether feedforward or feedback.
Actually, we can extend the above conclusion to a slightly larger set of functions.
α can be timedependent, but the supremum of it needs to be small.
Consider the function f t ( z t ) = g ( z t ) + h ( t ) , and let α t = ∂f t ∂t = ∂h∂t . Similar to the above analysis on linear time-variant function Eq.
(32), we have a conclusion below. Lemma 3 . Given a function f t , let α t = ∂f t ∂t .
Let K and ˆ K denote the Lipschitz constant for groundtruth f t and learnable function (e.g. neural networks)
ˆ f ( θ ) .
If sup( α t ) ≤ 4 −√ 12( K + ˆ K ) , with the proper buffer size, feedforward adaptation achieves smaller error bound than feedback adaptation.
C.2. E RROR BOUND OF ONLINE ADAPTATION ON T IME INVARIANT POLYNOMIAL
In this section, we compare the error (not error bound) of feedforward and feedback adaptation methods on time-invariant polynomial systems (functions). Consider a time-invariant polynomial ground-truth function f t ( z t ) with the input of random variable sampled from the uniform distribution U (0 , 1) .
[Equation] Assume our parameterized prediction model is a linear projection:
[Equation]
[Equation]
In online adaptation, we use the critical pair ( z s , y s +1 ) to optimize the linear prediction model at time step t . Then we have a parameter of
[Equation]
Then the absolute prediction error ( l 1 norm) of online adaptation is shown below: [Equation] = | z t | · | N
[Equation]
1 (cid:88)
i [Equation] W [Equation]
( z
[Equation]
) |
[Equation]
| z t |
[Equation]
|
[Equation]
W
[Equation]
js ) [Equation] ) | = | z t | · | z t − z s | · |
N − 1 (cid:88) i = W i + i [Equation] We consider the expectation of the prediction error.
We have the following property in statistics:
[Equation] For simplicity, assume W i > 0 .
The expectation of the prediction error is shown below.
[Equation]
Then we use Eq. (47) to derive the expectation of the error for feedback and feedforward adaptation methods. The core is measurement of E [ ∥ z t − z s ∥ ] and E [ z i − j − 1 t z js ] . Expectation of the prediction error for feedback adaptation .
In feedback adaptation, z s = z t −the current sample z t and the selected sample z s are independent random variables from U (0 , 1) . We have E ( | z t − z s | )
= {+
 (Eq.+}
Then we derive an expectation of the prediction error for feedback adaptation:
[Equation]
As can be seen, the prediction error of feedback adaptation cannot converge.
The expected prediction error E ( e fbt +1 ) in each step is lower bounded by a non-zero constant.
Expectation of the prediction error for feedforward adaptation .
In feedforward adaptation, the selected samples z s are the most similar samples to the current observation z t .
We have E ( | z t − z s | ) ≤ max( 1 t , 1 L +1 ) (Eq.
Then we derive an expectation of the prediction error for feedforward adaptation:
[Equation]
When we consider the final performance,
[Equation]
As can be seen, the expectation of the prediction error of feedforward compensation converges to zero with a large buffer size L .
When is feedforward better than feedback . In the previous appendix C.2.1, we already know that feedforward adaptation has a smaller error bound than feedback adaptation on approximate timeinvariant system ( α ≈ 0) . In this section, we furtherly prove that for the time-invariant polynomial system, feedforward adaptation has a smaller expected prediction error (not an error bound) than feedback adaptation. Thus feedforward adaptation is provably better than feedback adaptation for time-invariant or slowly time-variant systems.
D A DDITIONAL E XPERIMENTS D. S YNTHETIC E XPERIMENTS : L INEAR T IME - VARYING FUNCTION
To evaluate Lemma 1 in section 3.4, we consider the following time-varying function. [Equation] Our parameterized prediction model is a one-layer perception with Sigmoid activation function. [Equation]
[Equation]
We have The Lipschitz constant K and ˆ K for the ground-truth function f t and the one-layer perception ˆ f :
[Equation]
[Equation]
We use SGD as an optimizer in feedback and feedforward adaptation.
During training, we keep the ∥ V t ∥ bounded, i.e. ∥ V t ∥ ≤ 1 , then ˆ K = 14 .
We use Lemma 1 to calculate the error bound for feedback and feedforward adaptation:
[Equation] In feedforward adaptation, when L = (cid:113) 52 α − 1 , we get the minimum error bound: [Equation] Then we calculate the threshold α ⋆ .
If α ≤ α ⋆ , feedforward adaptation has a smaller error bound. [Equation] The experimental evaluation of the above theoretical results is shown in section 4.1. D. SYNTHETIC EXPERIMENTS: T IME - INVARIANT Q UADRATIC F UNCTION
To evaluate Lemma 2 in section 3.4, we consider the following quadratic function with random input. [Equation]
[Equation] [Equation] Our parameterized prediction model is a linear projection:
[Equation]
Set the buffer size L = 99 .
According to Lemma 2, the expected prediction error for feedback and feedforward adaptation are:
[Equation]
Fig.
3a shows the prediction error for feedback and feedforward adaptation.
As can be seen, the error curve for feedback adaptation is not converged.
While the error curve for feedforward adaptation converges by the trend of 12 t .
The experimental results are consistent with Eq.
(64) and (65).
Fig.
3a shows the prediction results for feedforward adaptation.
The dashed blue region denotes the estimated uncertainty in section 3.3.
In the figure, the ground truth is always within the estimated uncertainty, which validifies the proposed uncertainty estimation.
[Figure]
D. E XPERIMENTAL DESIGN FOR REAL - WORLD EXPERIMENTS
Dataset .
• ETT Zhou et al.
(2021) dataset contains the data collected from electricity transformers, including load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018.
Which consists of two hourly-level datasets (ETTh) and two 15-minute-level datasets (ETTm).
In our experiments, we used the first hourly-level dataset ETTh1 as a univariance prediction task.
• Exchange-Rate Lai et al.
(2018) records the daily exchange rates of eight countries from 1990 to 2016.
• ILI 2 describes the ratio of patients seen with ILI and the total number of patients.
Which includes the weekly recorded influenza-like illness (ILI) patients data from the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
• THOR Rudenko et al.
(2020) is a public dataset of human motion trajectories, recorded in a controlled indoor experiment.
Which includes the motion trajectories with diverse and accurate social human motion data in a shared indoor environment.
In our experiments, we use No. 2 ∼agent’s trajectory as a train set, No. 7 ∼ 8 as a validation set, and No. 9 ∼ 10 agent’s trajectory as a test set.
Backbone models .
We include two models as our learnable prediction function ˆ f ( θ, :) :
Informer Zhou et al.
(2021) and simple MLP (multi-layer perception).
• Informer Zhou et al.
(2021) is a widely used transformer-based time-series prediction model.
Which extends the Transformer with KL-divergence based ProbSparse attention.
• MLP is a simple but robust baseline for time-series prediction.
Our MLP consists of 2 layers.
The first layer can be considered as Encoder Z t = W · X t .
After the encoder, MLP has a layer normalization, activation function and a final linear projection Y t +
= V · Relu(LayerNorm( Z t )) .
The layer normalization and the final projection can be served as a decoder.
Note that, we did not flatten the input for MLP, the expression Z t = W · X t is a linear layer along the temporal axis.
Hyperparameters .
For offline training, we follow the strategy in Zeng et al.
In adaptation, we set the learning rate of SGD as η = 0 .
1 and set the EKF hyperparameters as p 0 = 0 .
1 , λ = 1 .
Buffer size for feedforward adaptation is L = 1000 .
For ETTh1 dataset, we use I = 192 step recent observations to predict future O = 192 step output data.
For the Exchange-Rate dataset, we use
I = 96 step recent observations to predict future O = 192 step output data.
For the ILI dataset, we use I = 36 step recent observations to predict future O = 36 step output data. For the THOR dataset, we use I = 20 step recent observations to predict future O = 20 step output data. For uncertainty estimation, we set ˜ δ = 0 , ˜ K = 1 . D. E XPERIMENTAL RESULTS ON EKF OPTIMIZERS
We conduct experiments on SGD and EKF as optimizers in adaptation.
The descriptions of SGD and
EKF are shown in appendix B. In this section, we report the detailed results of the experiment on the EKF optimizer, as shown in table 2. Similar to results on the SGD on table 1, feedforward adaptation still achieves the best results on all four datasets with EKF optimizer.
Specifically, feedforward adaptation outperforms feedback adaptation by 5 .
1% , 24 . 1% , and 2 .
6% in terms of average
MSE on four datasets respectively.
The performance gain on the ILI dataset is the most obvious.
It is reasonable because the incidence rate of common diseases in a region is a slow time-varying system, the future value is mostly dependent on the current input.
[Table]
D. S TUDY OF THE SAMPLE SELECTION STRATEGY OF DIFFERENT ADAPTATION METHODS
We will show that the sample selection strategy in the proposed feedforward adaptation method could intrinsically extract the periodicity of the input data.
The experiments on the ETTh1 dataset were shown in Fig.
In this section, we show the results of ILI, Exchange, THOR datasets.
In our experiment, we set the buffer size L = 1000 .
We use SGD optimizer to adapt the MLP model with different adaptation methods.
ILI dataset .
Figure 4(a) shows the ILI time-series data (1st dimension of the output), and Fig. 4(b) is the FFT (Fast Fourier Transformation) period analysis of the ILI data.
As can be seen ETTh has roughly T ≈ 50 repetition periods.
Random adaptation randomly selects samples from the
[Figure]
buffer, then the time shift between current time-step t and the time-step for selected samples s is ( t − s ) ∼ [1 , L ] ; Feedback adaptation only selects the last time sample to optimization, then ( t − s )
= 1 .
This can be found from the distribution of time shift ( t − s ) in Fig. 4(c).
For the proposed feedforward adaptation in Fig. 4(d), many samples were selected from ( t − s )
≈ 50 steps earlier, which corresponds to the repetition period of T ≈ 50 .
Because feedforward adaptation selects the most similar samples to the current sample, which helps to extract the hidden periodicity of the input signal over time.
Thus the distribution of t − s is similar to FFT period analysis.
[Figure]
Exchange dataset .
Figure 5(a) shows the Exchange time-series data (6th dimension of the output), and Fig. 5(b) is the FFT (Fast Fourier Transformation) period analysis of the Exchange data.
As can be seen Exchange has no apparent periodicity.
(The intensity of the FFT signal is too low, for example, the intensity of at the T ≈ 90 is about 0 .
In this case, Feedforward adaptation in
Fig. 5(d), is likely to select the recent samples like feedback adaptation in Fig. 5(c).
Because the most recent sample is a similar sample itself.
Even for this kind of no-periodic system, feedforward adaptation still outperforms feedback adaptation, because it has more freedom to select samples.
At least feedforward adaptation can select samples exactly the same as feedback adaptation.
[Figure]
THOR dataset .
Figure 6(a) shows the THOR time-series data (1st dimension of the output), and
Fig.
6(b) is the FFT (Fast Fourier Transformation) period analysis of the THOR data.
As can be seen
THOR has roughly T ≈ 1200 repetition periods.
But the buffer size is L = 1000 < T .
In this case, we only store the recent L = 1000 samples in the buffer, but the period is more larger T ≈ 1200 >
L , so the feedforward adaptation cannot extract the periodicity.
In this case, Feedforward adaptation
Fig.
6(d), is likely to select the recent samples like feedback adaptation in Fig. 6(c).
Because the most recent sample is a similar sample itself within the buffer.
In the future, we will investigate the more efficient sampling and buffer storing strategy to extract the very long time-dependency and periods.
D. P REDICTION OUTPUT AND UNCERTAINTY
One of the advantages of the proposed feedforward adaptation is it could provide uncertainty estimation as shown in algorithm 2.
Figure 7 shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) on four different datasets.
As can be seen, in most cases, the estimated uncertainty covers the real ground truth value.
Which validates the effectiveness of the proposed uncertainty estimation.
[Figure]