Learning Mixture Structure on Multi-Source Time Series for Probabilistic Forecasting
Abstract
In data-driven applications nowadays, it is increasingly desirable to collect data from different sources for enhancing performance.
In this paper, we are interested in the problem of probabilistic forecasting with multi-source time series.
We propose a neural mixture structure based probability model for learning different predictive relations and their adaptive combinations from multi-source time series.
We present the prediction and uncertainty quantification methods which are applicable to different distributions of target variables.
Additionally, given the imbalanced and unstable behaviors observed during the direct training of the proposed mixture model, we develop a phased learning method and provide a theoretical analysis.
In experimental evaluations, the mixture model trained by the phased learning exhibits competitive performance on both point and probabilistic prediction metrics.
Meanwhile, the proposed uncertainty conditioned error suggests the potential of the mixture model’s uncertainty score as a reliability indicator of predictions.
Introduction
Time series data are prevalent in many applications [1] and in this paper we are interested in the probabilistic forecasting in the setting of multi-source time series.
The data source can be distinct locations, entities, records, sensors etc., from where data is collected.
This multi-source data setting can fit into a variety of scenarios, where learning from a single source may be sub-optimal.
For instance, in finance, different operations by market participants (e g., performing transactions, putting limit orders, etc.) are separately recorded, thereby giving rise to distinct sources of features reflecting the underlying characteristics of markets [2, 3].
In environmental monitoring where sensors are usually deployed in multiple sites, each site is a data source and collectively using the data from different sensor sites would be favorable to predictive analytics [4].
Challenges.
Multi-source time series data usually carry different dynamics information and have time-varying relevance to the target variable [5, 6], as illustrated in Fig.
Meanwhile, the target variable might follow different distributions depending on applications [7].
For downstream decision making, besides point predictions, it is increasingly preferred to provide additional predictive information, e.g., uncertainty score, quantiles, etc [8, 9, 10].
All the above highlight the need for a flexible and adaptive probabilistic model structure on multi-source data.
As for the probability mixture model, it is typically composed of hierarchical components, and the interactions between these components during the training affect the end performance.
For instance, as shown in Sec. 4.2, direct optimizing our mixture model’ loss function leads to imbalanced learning behaviors of different data-source-specific components, implying that some data sources’ predictive power is inadequately or even biasedly captured.
This further hinders exploiting the full capacity of mixture models.
Contributions.
Specifically, the contribution is as follows: (1) We present the neural mixture structure based probability model on multi-source time series.
It is equipped with a representation module to encode multi-source data, and a representation sharing mechanism to serve the prediction and weight modules.
(2) We present the adaptive prediction and uncertainty quantification methods, and analyse the implication of different components in the uncertainty score.
This inference process is applicable to different target distributions, e.g., normal, and log-normal distributions in this paper.
(3) We demonstrate the imbalanced learning behaviors when directly minimizing the proposed mixture model’s loss function, and then develop the phased learning method with theoretical analysis. (4) Through experiments on real datasets from finance and environmental monitoring scenarios, our mixture model trained by the phased learning exhibits competitive performance on both point and probabilistic prediction metrics.
Meanwhile, we put forward uncertainty conditioned error analysis to compare uncertainty scores from different methods.
Related work
[Figure]
vance to the target the corresponding data source has.
The sum of all colored value ranges is 1 .
at each timestamp.
Probabilistic Forecasting on Time
Series.
Probabilistic forecasting focuses on the predictive probabilistic characteristics of target variables, and is aimed to provide a variety of predictive quantities for downstream decision making, e.g., uncertainty scores, predictive quantiles, prediction intervals, etc [9, 7, 11].
Recently, many neural network based probabilistic forecasting models have been proposed for time series data. [12, 8] respectively equipped RNNs with the linear state space models and random effect components.
[7] used
RNNs with autoregressive and covariate inputs for probabilistic forecasts.
[13, 14] focused on modeling multi-variate time series.
Stochastic sequential models [15, 16, 17, 18, 19, 20] developed stochastic temporal latent states for modeling generative dynamics in time series.
Given the setting of multi-source data in this paper, above methods mostly model the input data as a whole and could hardly differentiate various time-varying relations in multisource data, thereby potentially leading to inferior predictive performance.
Mixture Models.
Due to the flexibility and adaptivity, mixture models have been applied in a variety of areas, such as unsupervised learning [21, 22, 23], natural language processing [24, 25, 26], computer vision [27, 28, 29], etc.
Few attempts have been made to explore the mixture structure, its inference, and learning methods for probabilistic forecasting with multi-source data.
As a category of latent variable models, mixture models are normally trained through (stochastic) expectation maximization (EM) style optimizations [30].
[31] provided theoretical convergence analysis for linear mixture models.
[32] integrated variance reduction into stochastic EM algorithms.
In this paper, we explore the practical treatments for stably and balancedly learning different data source corresponding modules of the mixture model.
Problem and Proposed Model 3.1 Problem Statement
Let y t be the value of the target variable at time t .
Assume S number of different data sources, which are indexed by s = 1 , · · · , S .
The multi-source data used to predict y t is denoted by x 1: S,<t
= { x s,<t } Ss =1 , where each element x s,<t is a multi-dimensional time series from a limited window of historical data of source s prior to timestamp t .
Depending on applications, the latest timestamp in x s,<t could be t − 1 for one-step or t − h for h -step ahead prediction.
For simplicity, we skip the notations for time series dimensions and history window length.
The task is to model the predictive density function p ( y t | x 1: S,<t ) for providing predictive quantities such as point predictions, uncertainty scores and so on [33, 34].
Note that in this paper, we assume that x 1: S,<t implicitly contains the auto-regressive data of the target variable, i.e., y <t when available, either as an individual data source or being included into one.
3.2 Model Structure
The idea of the mixture structure on p ( y t | x 1: S,<t ) is to introduce a latent discrete random variable z t for differentiating various predictive relations between x 1: S,<t and y t .
In our case, z t is defined on the data source index set, i.e., z t ∈ { 1 , · · · , S } , thereby leading to data-source-wise prediction modules.
The probability mass function of z t serves as the weight module.
Meanwhile, instead of using the raw data x 1: S,<t as inputs to the weight and prediction modules, we feed a set of latent representations that are learned from x 1: S,<t by powerful neural networks.
Let h 1: S,t denote this set of representation vectors { h s,t } Ss =1 .
The factorization of the joint density function including z t and h 1: S,t as latent variables is expressed as:
[Equation]
representation
[Equation]
prediction
[Equation]
Representation Module is to encode each source’s data into the latent representations that will serve the prediction and weight modules. There are various probabilistic representation learning methods for time series, e.g., state-space models, stochastic recurrent neural networks, etc [35, 36, 20]. Since advancing the representation learning is not the focus of this paper, we choose a simple yet flexible way to formulate the representation module, namely, placing a Dirac delta distribution centered at the hidden state output of a temporal data encoder [37, 16, 38]. This module is formulated as:
[Equation]
, where δ ( a ) = 1 only when a = 0 . NN η s ( x s,<t ) is the encoder to model the temporal dynamics of source s and η s denotes the trainable parameter in it. In the experiment, we employ a LSTM as the temporal encoder, while it is flexible to use others. If it is hypothesized to there exist global latent factors governing the dynamics across data sources, the above formulation is extensible by learning a global representation from all sources and then sharing it across individual representations [39, 8].
Prediction Module is to parameterize the source-wise predictive distribution p ω s ( y t | z t =
s, h s,t ) with the learnable parameters ω s .
For instance, for y t ∈ R and the normal distribution is applied, we have y t | h s,t ∼ N ( µ s,t , σ 2 s,t ) , where µ s,t , σ 2 s,t = g ω s ( h s,t ) , and g w s ( h s,t ) can be any flexible function such as multiple dense layers parameterized by ω s [7].
For y t ∈ R + , we can choose the log-normal distribution, i.e., log y t | h s,t ∼ N ( µ s,t , σ 2 s,t ) [40, 33].
Weight Module is the probability mass function of z t :
[Equation]
, where θ s denotes the parameters of the logit function for data source s .
f θ s ( · ) can be any flexible function such as neural networks.
The trainable parameters from all logit functions constitute the parameter set of this module, i.e., θ = { θ s } Ss =1 .
Adaptive Inference and Phased Learning 4.1 Inference
We mainly present the adaptive point prediction and uncertainty quantification.
Other predictions like quantiles, intervals, etc., and their derivations are described in Appendix A.2.
For simplicity, we express the weight module as P θ
( z t = s | · ) .
Point Prediction.
It is the predictive mean of the mixture, which is the weighted sum of each source’s predictive mean.
[Equation]
, where E [ y t | z t = s, (cid:101) h s ] denotes the predictive mean by source s .
For instance, it is µ s,t for normal distribution and exp( µ s,t + 12 σ 2 s,t ) for the log-normal distribution [33].
Uncertainty Score.
In forecasting tasks, the predictive variance is commonly used as an uncertainty measure [41, 42].
With a single forward pass, our mixture model provides the uncertainty score ˆ u t consisting of two components as follows:
[Equation]
[Figure]
[Equation]
[Figure]
U n c e r t a i n t y
[Figure]
o r e
[Figure]
[Figure] P r e d i c t i o n s b y e a c h s o u r c e (c) For instance, the variance term Var ( y t | z t = s, (cid:101) h s ) is σ 2 s,t for normal distribution, and exp( σ 2 s,t − 1) · exp(2 µ s,t + σ 2 s,t ) for log-normal distribution [43].
In Eq.5, the aleatoric uncertainty quantifies predictive noise inherent in the target, while the mixture part reflects the variation among individual data sources’ predictions.
The mixture weights serve to adjust the contributions of different sources.
This is different from some uncertainty quantification methods using equal-weighted components, e.g., model snapshots, latent states [44, 45, 20].
In the following, through the testing examples in Fig.2, we give a qualitative interpretation of these two uncertainty components.
In time period 1 , the target variable stays relatively stable in Fig.2(a).
Accordingly, in Fig.2(b), the aleatoric uncertainty is low.
In Fig.2(c), the predictions by individual sources show some consensus and reside in a close value range, and thus the mixture uncertainty is also low in 1 .
On the other hand, in period 2 , when the target is fluctuating in Fig.2(a), besides the aleatoric part, the mixture uncertainty drives up the total uncertainty because of the widened prediction dispersion as shown in Fig.2(c).
Interestingly, Fig.2(c) also reveals that when the target is in a volatile period (e.g., 2 ), just some of the data sources can provide seasonably reasonable predictions, while the rest of the sources output too high or too low predictions and seem to lack enough predictive power.
4.2 Issues with Direct Learning
In this part, we analyze the imbalanced learning behaviors when directly minimizing the loss function of the mixture model.
We first define the training dataset D = (cid:8) y t , x 1: S,<t (cid:9) t ∈T , where T represents the set of timestamps at which data instances are collected.
Let Θ represent the set of trainable parameters, i.e., Θ = (cid:8) { η s , ω s } Ss =1 , θ (cid:9) .
Then, the mixture loss is expressed as:
[Equation]
Next, through Fig.3, we illustrate the imbalanced learning status of different data sources when directly minimizing the mixture loss via stochastic gradient descent (SGD) based methods [46].
This imbalanced and unstable behavior might hinder the mixture model from adequately capturing different data sources’ predictive relations and their combinations, and thus impairs the overall performance as shown in Sec.5.
Specifically, Fig.3(a) reports the source-wise prediction error which is defined by the rooted mean squared error between training true values and predictive means by each source’s prediction module over training epochs.
It shows that some error curves converge poorly, e.g., the purple one implies the corresponding source’s prediction module hardly captures useful predictive relations.
As for the converged curves, the noticeable error dispersion suggests that those prediction modules might be learned to different extents.
[Figure]
mixture model, compared with the proposed phased learning.
Each curve represents the rooted mean squared error (RMSE) between target values and predictive means by a source’s prediction module over training epochs.
The results are based on five different training runs on AIR data in Sec.5.
As a comparison,
Fig.3(b) shows the source-wise prediction error of the model trained by the proposed phased learning in Sec.4.3.
The source-wise errors stably and rapidly decline to a close range, and, likely, all prediction modules are reasonably learned.
In the zoom-in part, the spread between converged error curves still implies the different predictive powers of data sources.
Notably, in
Fig.3(b), the purple error curve also converges nicely.
It implies the corresponding data source has the predictive power, but it is not properly captured in
Fig.3(a).
In Sec.5, we will show that the overall prediction performance of the mixture model considerably benefits from this phased learning.
Lemma 4.1.
In minimizing the mixture loss via SGD optimization, given a random training sample indexed by t , the vectorized derivative of a source’s prediction and representation modules are denoted by:
[Equation]
[Equation]
where ∇ (cid:101) h s,t denotes the Jacobian matrix of source s ’s representation w.r.t.
η s .
Meanwhile, the posterior weight π s,t is defined as:
[Equation]
Then, the (stochastic) updates to the prediction and representation parameters of source s , i.e., ω s and η s , are:
[Equation]
(Derivation details in the appendix section.)
Lemma 4.1 sheds light on one potential cause of the imbalanced learning, i.e., the intertwined relation between posterior weights and parameter updates, and Sec.4.3 will present another theoretical analysis.
Specifically, at the beginning of the training, all representation, prediction, and weight modules are barely trained.
Some data sources receive low posterior weights (i.e., π s,t ), if they happen to have low (stochastic) predictive density values and/or low mixture weights, according to Eq.10.
Then, based on Eq.11, low posterior weights lead to under-weighted (stochastic) updates and might slow down or even corrupt the learning progress of corresponding source’s modules.
In turn, this posterior weight bias might persist and result in the imbalanced learning status in Fig.3(a).
4.3 Phased Learning
To relax the above issues, we propose the phased learning method splitting the training process into two phases respectively for unbiasedly learning each data source’s predictive power and then the adaptive combinations.
(The proofs in this part are in the appendix section.)
Lemma 4.2.
In the iterative learning process, at any point of Θ , there exists the upper bound of the mixture loss:
[Equation]
(12) where D t is a random training sample D t ≜ ( y t , x 1: S,<t ) ∈ D .
[Equation]
(13) and a ∗ t is a constant value, a ∗ t = min s ( { P ( z t = s | · ) } ) .
First, the impartial phase learning is inspired by Lemma 4.2.
It minimizes the upper bound Eq.13, thereby leading to the following update scheme:
[Equation]
, where g ω s ,t and g η s ,t are defined in Eq.7 and 8.
These updates only involve the prediction related parameters, i.e., the representation and prediction modules.
Meanwhile, the equal-weighted manner allows to impartially learn the predictive relation of each data source.
Theorem 4.3.
For all stochastic derivatives g ∈ { g ω s , g η s , d η s } corresponding to the data source s , suppose it is bounded E ∥ g ∥ 2 ≤ G 2 .
The trainable parameters related to a data source’s prediction are denoted by Θ s = [ ω s , η s ] ⊤ and the corresponding gradient is
G s = E [ g ω s , g η s ] ⊤ .
Over the learning step i = 0 , · · · , I , the convergence of G s to a stationary point in the direct optimization and the impartial-phase optimization are respectively:
[Equation]
Impartial phase:
I
[Equation]
, where π ∗ s ∈ (0 , 1) is the minimum of poster weights across data samples and iteration steps.
L s,i represents the negative log-likelihood of the predictions by data source s at step i .
Theorem 4.3 explains the convergence patterns of the direct and impartial-phase learning observed in Fig.3.
As shown in Eq.15, there are two more factors that impact the convergence of each data source’s prediction related parameters in the direct learning, compared with the impartial-phase learning.
The value of π ∗ s is less than 1 and implies the slow value decline of the first term on the right-hand-side of Eq.15.
Meanwhile, if the second term on the right-hand-side of Eq.15 dose not vanish completely, it converges to the neighbourhood of a stationary point, i.e., the corresponding prediction modules might be inadequately trained.
For instance, the converged error curves in
Fig.3(a) are still higher than the counterparts in Fig.3(b).
Second, the collective phase will update all modules’ parameters by switching to minimize the original mixture loss Eq.6 until the end of the training.
The weigh module is now activated to learn the adaptive combinations.
The representations and prediction modules are updated as Eq.11.
Thanks to the reasonably learned representation and prediction modules in the impartial phase, the posterior weight now in Eq.10 more reliably reflects different data sources’ relevance to the target.
In this phase, it amounts to fine-tuning the representations and prediction modules by considering the data source relevance reflected in posterior weights.
For instance, in Fig.3(b), the zoom-in figure shows that some sources’ error curves continue to decline in the collective phase.
In practice, the number of epochs of the impartial phase can be tuned as a hyper parameter.
On the experiment datasets, the range of 5 to 15 epochs is enough for the impartial phase.
Meanwhile, we found that the total number of epochs needed in the phased learning is comparable to the direct learning, i.e., having two phases does not incur the need for more epochs in total.
Compared with the direct learning with a certain number of epochs, the phased learning can mostly present enhanced performance by running the impartial and collective phase with the same budget of epochs.
Experiments
In this section, we report the experimental comparison of different models and learning methods.
The Appendix contains additional experiment details and results.
5.1 Baselines
DAR denotes DeepAR, which uses auto-regressive recurrent neural networks to model the probabilistic distribution of the target variable [7].
DF refers to the Deep Factor model, which is a global-local method based on a global neural network backbone and local probabilistic random effect models [8].
AF represents Autoformer [47], a decomposition based Transformer [48] variant with an auto-correlation mechanism.
DAR and DF are probability models, and AF is adapted for probabilistic forecasting based on [49]; hence all baselines can parameterize different distributions for the target variable and be evaluated on point and probabilistic prediction metrics.
For baselines, multi-source data are fed as a whole.
For the mixture model and baselines, probabilistic forecasts like predictive quantiles and intervals are derived via the same type of methods presented in A.2.
5.2 Datasets
Existing time series forecasting models mostly take as input single (univariate or multivariate) time series, while our mixture model works on multi (multivariate) time series from distinct sources. Thus, we use the following data from two scenarios.
For each dataset, data instances are timeordered, and we use the first 70% of points for training, the next 10% for validation, and the last 20% for testing.
AIR is from the environmental monitoring scenario [50].
There are twelve air-quality monitoring sites, and each site collects an eleven-dimensional time series including hourly air pollutant and meteorological observations, from March 2013 to February 2017.
The target variable, which is the air pollutant value reflected in PM2.5 measurements, lies in the non-negative domain, and thus the log-normal distribution is applied.
The time series from all sites forms a twelve-source data for forecasting the target variable of each site.
VOL is from the finance area, i.e., two liquid cryptocurrency exchanges, Bitfinex and Bitstamp.
From the limit order book (LOB) and transaction records of each exchange, it collects two minutelevel multi-dimensional feature time series, from May 2018 to September 2018.
The target variable is the intra-day traded volume of bitcoins.
It is the raw volume deduced by the intra-day periodicity and lies in the real value domain, and thus the normal distribution is applied.
In a cryptocurrency [Table] exchange, it records different trading related operations, e.g., transaction data stores the information about executed trades, the limited order book (LOB) records the standing orders with buy or sell limits, etc.
We formulate a multi-source data consisting of the time series from LOB and transaction features as well as historical volume.
5.3 Setup
Hyper-Parameters.
The hyper-parameter search and associated training processes are run on a server with NVIDIA A100 GPUs.
For all models, Bayesian optimization is applied to search in the hyper-parameter space [51].
Once the best hyper-parameters is fixed, each model is retrained for five times with different random seeds, and the average performance is reported.
The hyper-parameter space is detailed in Appendix.
Point Prediction Metrics.
The root-mean-square error (RMSE) and mean absolute error (MAE) are defined on true values and predictive means.
Probabilistic Prediction Metrics.
The mean negative log likelihood (NLLm) is the predictive negative log-likelihood of testing instances averaged by the number of instances.
The mean quantile loss (QLm) is the mean of quantile losses w.r.t.
a set of quantile levels [9].
A quantile loss is defined on true values and predictive quantiles to reflect how well the predictive distribution fits the target variable’s probabilistic characterises [8].
Detailed formations of these metrics are in Appendix.
5.4 Results
Comparison of Prediction Performance.
In Table 1, on each site or exchange (exc.), the mixture model outperforms baselines on the point and probabilistic prediction metrics.
It indicates that the mixture models more accurately characterize the distributional properties of the target variable.
Compared with the baselines modeling multi-source data as a whole, the outperformance of the mixture model suggests that capturing adaptive relations in multi-source data can better harness the predictive power.
Comparison of Training Methods.
Table 2 shows the results of the mixture models respectively trained by different learning methods.
The phased learning noticeably improves the performance of the mixture model on all metrics, while in some cases the mixture model trained by the direct learning under-performs the baselines in Table 1.
This observation suggests that the specialized training process is beneficial for exploiting the capacity of mixture models.
Meanwhile, the low standard errors of the phased learning imply the training process is relatively more stable, thereby leading to more consistent generalization performance.
[Table]
Comparison of Uncertainty Conditioned Errors.
The uncertainty score can an indicator of the prediction error level, namely, high uncertainty scores imply unreliable predictions and thus potentially large prediction errors, and vice versa.
Following this intuition, we propose the uncertainty conditioned prediction error to compare the uncertainty scores from different models.
Specifically, given the predictions and associated uncertainty scores, we first derive the empirical quantiles of the uncertainty scores.
Then, as shown in Fig.4, we calculate the prediction errors (RMSE) by using the predictions with the uncertainty scores falling within each interval of quantile levels
[Figure]
In Fig.4, based on the above intuition, the uncertainty conditioned error curve is expected to present a monotonic increasing pattern.
The mixture model has monotonic error curves, while the baselines
DAR and DF have some irregular errors, i [Figure] e., the error of the high uncertainty interval is lower than that of the low uncertainty interval.
Compared with the baselines only capturing the aleatoric uncertainty, the mixture model’s uncertainty score consists of the weighted aleatoric and mixture components [Figure] and it is more effective in reflecting the prediction error extent as a function of the uncertainty.
Meanwhile, the mixture model has lower errors on all quantile level intervals.
This also explains the overall competitive prediction performance of mixture models in Table 1 from the uncertainty perspective.
Conclusion
This paper focuses on the probabilistic forecasting with multi-source time series data .
We present a neural mixture structure based probability model on multi-source time series and the associated phased learning method.
The experiment results exhibit competitive performance of the proposed model and learning method on both point and probabilistic metrics.
It reveals that capturing datasource-wise predictive relations and their adaptive combinations is important for exploiting the predictive power of multi-source data.
Meanwhile, for mixture models with hierarchical and interacted components, understanding the learning behaviors of these components and accordingly customizing the learning method help realize the capacity of models.
The limitation of the present work lies in the representation learning of time series data.
It will be worth exploring more advanced representation learning methods and studying the effect on the prediction performance.
Meanwhile, the transition from the impartial to collect phase is more favored to have a systematic mechanism to avoid tedious hyper-parameter search.
A Model Specification A.1 Target Distributions
We show the flexibility of specifying the prediction modules of the mixture models through two different target distributions.
For the target variable in normal distribution, i.e., y t ∈ R , the prediction module of each data source is expressed as:
[Equation]
) µ s,t and σ 2 s,t are derived by g w s ( h s,t ) , where g w s ( · ) can be any flexible function such as dense layers parameterized by ω s [7].
The corresponding predictive mean and variance are E
[ y t | z t = s, (cid:101) h s ] = µ s,t and Var ( y t | z t = s, (cid:101) h s,t )
=
σ 2 s,t .
When y t ∈ R + and is assumed to follow the log-normal distribution, the density function is:
[Equation]
The predictive mean and variance by the data source s are E
[ y t | z t = s, (cid:101) h s ] = exp( µ s,t + 12 σ 2 s,t ) and Var ( y t | z t = s, (cid:101) h s,t ) =
exp( σ 2 s,t − 1) · exp(2 µ s,t + σ 2 s,t ) [43, 33].
A.2 Inference
In this part, we present several predictive quantities that can be derived from the mixture model.
Point Prediction.
It is the predictive mean defined as:
[Equation]
, where E [ y t | z t = s, (cid:101) h s ] denotes the predictive mean by source s .
As aforementioned, it is µ s,t for the normal distribution and exp( µ s,t + 12 σ 2 s,t ) for the log-normal distribution [33].
Uncertainty Score.
According to the variance definition, we have:
[Equation]
Plugging the following into Eq.21 gives rise to [-Eq.
-]Eq.
[Equation]
, where the variance term Var ( y t | z t
= s, (cid:101) h s,t ) is σ 2 s,t for the normal distribution, and exp( σ 2 s,t − 1) · exp(2 γ s,t + σ 2 s,t ) for the log-normal distribution [43].
Quantile Prediction.
Based on the mixture model, the predictive cumulative probability function (CDF) of the target is expressed as:
[Equation]
where (cid:101) h s,t =
NN η s ( x s,<t ) , and P ( y t | z t = s, (cid:101) h s,t ) represents the individual data source’s CDF.
The second equality is due to the delta distribution applied to p ( h 1: S,t | x 1: S,<t ) , as is presented in
Sec.3.2.
Then, for the quantile level α ∈ [0 , 1] , we resort to a numerical method by applying the root-finding algorithm to the function in Eq.
25, while sampling based method can also applied.
The solution of the root-finding algorithm is the quantile prediction ˆ y t,α .
[Equation]
Prediction Interval.
For obtaining a prediction interval corresponding to a probability range, we make use of the corresponding quantile predictions of the probability range.
For instance, a prediction interval corresponding to the probability range [0 . 1 , 0 . 9] is constructed by the root solutions of
Q (cid:0) y ; α = 0 .
1 , P ( · | x 1: S,<t ) (cid:1) = 0 and Q (cid:0) y ; α = 0 . 9 , P ( · | x 1: S,<t ) (cid:1) = 0 .
A.3 Learning
The loss function of the mixture model is:
[Equation]
where Θ = (cid:8) { η s , ω s } Ss =1 , θ (cid:9) denotes the set of all trainable parameters.
Since in this paper the delta distribution is applied to p ( h 1: S,t | x 1: S,<t ) and p ( r 1: S,t | x 1: S,<t ) , the integral over the hidden states is reduced to single points, i.e., (cid:101) h s,t = NN η s ( x s,<t ) .
B Proofs
Lemma 4.1.
In minimizing the mixture loss via SGD optimization, given a random training sample indexed by t , the vectorized derivative of a source’s prediction and representation modules are denoted by:
[Equation]
[Equation]
where ∇ (cid:101) h s,t denotes the Jacobian matrix of source s ’s representation w.r.t.
η s .
Meanwhile, the posterior weight π s,t is defined as:
[Equation]
Then, the (stochastic) updates to the prediction and representation parameters of source s , i.e., ω s and η s , are:
[Equation]
Proof.
Given a random training sample ( y t , x 1: S,<t ) , the mixture loss Eq.26 is expressed:
[Equation]
Take the derivative w
[Equation]
r
[Equation]
t. the prediction module parameter of source s ,
[Equation]
In Eq.32, the second equality is through the derivative trick of log functions, i.e., ∂f ( ω ) /∂ω = f
[Equation] log [Equation] [Equation]
Likewise, for η s , we have the following:
[Equation]
In Eq.33, the second equality is based on the derivative trick of log functions.
Because the weight module P θ ( z t = k | (cid:101) h 1: S,t ) takes as input the representations of all data sources, there exists the sum of derivatives in the first term in the last equality.
Meanwhile, since (cid:101) h 1: S,t is the output of representation modules parameterized by { η s } Ss =1 , the derivative w.r.t.
η s is through (cid:101) h s,t .
Then, the update scheme in Eq.
31 is obtained.
■
Lemma 4.2.
In the iterative minimization process of the mixture loss, at any point of Θ , there exists the following upper bound:
[Equation]
where D t is a random training sample D t ≜ ( y t , x 1: S,<t ) ∈ D .
[Equation]
and a ∗ t is a constant value, a ∗ t = min s ( { P ( z t = s | · ) } ) .
Proof.
The mixture loss on a random training sample D t is expressed:
[Equation]
where (cid:101) h s,t =
NN η s ( x s,<t ) .
Given Θ , by defining a ∗ t ≜ min s (cid:0) { P ( z t = s | · ) } Ss = [Equation] , we have
[Equation]
Eq.
37 holds, because the probability densities are non-negative.
The equality holds when all weights are equal.
Then, Eq. 37 is further expressed as follows:
[Equation]
S is the number of data sources.
The last inequality is based on Jensen’s inequality.
■
Theorem 4.3.
For all stochastic derivatives g ∈ { g ω s , g η s , d η s } corresponding to the data source s , suppose it is bounded E ∥ g ∥ 2 ≤ G 2 .
The trainable parameters related to a data source’s prediction are denoted by Θ s = [ ω s , η s ] ⊤ and the corresponding gradient is
G s = E [ g ω s , g η s ] ⊤ .
Over the learning step i = 0 , · · · , I , the convergence of G s to a stationary point in the direct optimization and the impartial-phase optimization are respectively:
[Equation]
Impartial phase:
I
[Equation]
I
[Equation]
, where π ∗ s ∈ (0 , 1) is the minimum of poster weights across data samples and iteration steps.
L s,i represents the negative log-likelihood of the predictions by data source s at step i .
Proof.
Before going to the proof details, we first recall some notations.
As defined in Lemma 4.1.
, we have the (stochastic) derivatives on a random training sample t as follows:
[Equation]
where ∇ (cid:101) h s,t denotes the Jacobian matrix of source s ’s representation module w.r.t.
η s .
Given the parameter Θ s,i at step i , the predictive performance of data source s is reflected in the negative log-likelihood which is a function of Θ s,i as:
[Equation]
Then, the empirical expected gradient of Θ s,i w.r.t.
L s,i is as follows: (the step index i is omitted for simplicity)
[Equation]
Direct optimization
In the direct optimization, as shown in Lemma [Equation] [Equation] the stochastic update to the prediction and representation parameters of a data sources s is expressed as: (cid:98) g s,i ≜ [ π s,i g ω s ,i , π s,i g η s ,i + d η s ,i ] ⊤ , where π s,i is the posterior weight.
In the following, we will drop the random training sample subscript t for simplicity when unnecessary.
Following the L -smoothness assumption of L s yields:
[Equation]
where γ is the learning step size.
π ∗ s,i is the minimum of posterior weights across data samples,
i
[Equation]
e., π ∗ s,i = min t ( { π s,t,i } t ) , and it is always less than 1 ,
i
[Equation]
e., π ∗ s,i ∈ (0 , 1) .
⟨· , ·⟩ is the inner product of two vectors.
Then, for the ( a ) and ( b ) terms
[Equation]
we have the following:
[Equation]
The first inequality is because 2 ⟨ a , b ⟩ ≤ ∥ a ∥ 2 + ∥ b ∥ 2 and the last one is due to the gradient bound assumption.
Plugging Eq.48 and 52 into Eq.46 and summing the inequality from i =
to i
= I lead to:
[Equation]
If we further take the minimum of poster weights across iteration steps by setting π ∗ s = min i ( { π ∗ s,i } i ) or equivalently π ∗ s = min t [Equation] i ( { π s,t,i } t,i ) , rearranging Eq.
53 gives:
[Equation]
Plugging in the step size γ that minimizes the RHS of Eq.
54 shows the result of the direct optimization:
[Equation]
Impartial phase optimization
In the impartial phase [Equation] the stochastic update to the prediction and representation parameters of a data source s is expressed as: ˚ g s,i ≜ [ 1 S g ω s ,i , 1 S g η s ,i ] ⊤ .
Following the L -smoothness assumption yields:
[Equation]
Then [Equation] by summing the inequality Eq.
58 from i =
to i
=
I and some rearrangements [Equation] we obtain
[Equation]
I
[Equation]
[Equation]
Plugging in the step size γ that minimizes the RHS of Eq.
59 gives the convergence:
[Equation]
C Additional Experiment Details and Results
Hyper-Parameters.
The hyper-parameter search and associated training processes are run on a server with NVIDIA A100 GPUs. For all models, Bayesian optimization is applied to search in the hyper-parameter space [51]. Once the best hyper-parameters is fixed, each model is retrained for five times with different random seeds, and the average performance is reported.
For recurrent neural networks used in DAR, DF and mixture models, the layer size and number of layers are searched over { 64 , 128 , 256 , 512 , 768 , 1024 } and { 1 , 2 , 3 } . The learning rate is chosen in the range { 1 e − 5 , 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 } following an exponential decay scheme with a rate of 0 .
every 10 epochs.
The search spaces of batch sizes, weight decay and dropout rates respectively lie in [256 , 512] , { 1 e − 7 , 5 e − 7 , 1 e − 6 , 5 e − 6 , 1 e − 5 , 5 e − 5 , 1 e − 4 , } , and { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 } Performance Metrics. Given the point prediction ˆ y t and the true value y t , the point prediction metrics are defined as follows: [Equation]
where T test represents the set of timesteps in the testing data, and |T test | is the number of testing examples.
As for the mean quantile loss (QLm), we define the (normalized) quantile loss of a given quantile level α ∈ (0 , 1) as:
[Equation]
where I takes value 1 only when the corresponding condition holds [8].
ˆ y t,α is the quantile prediction obtained through the method presented in Sec.
A.2.
Then, the QLm is the mean value of the quantile losses QL ( α ) for α = 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 .
Results.
In this part, we report the performance result on all datasets.
Table 3 and 4 show the point and probabilistic prediction performance on all datasets.
Our mixture model MIX shows competitive performance over baselines.
[Table]
Table 5 shows the performance of the mixture models trained by the direct and phased learning methods.
The phased learning constantly helps enhance the performance.
This implies that when the prediction and representation related modules of mixture models are insufficiently learned yet, simultaneously learning the prediction modules with the weight modules may fail to reliably capture the predictive power of different data sources, thereby leading to less accurate weighted combination.
Fig.
5 show the uncertainty conditioned prediction errors of all datasets.
The uncertainty quantile level on the x-axis of each sub-figure is derived from the empirical distribution of predictive uncertainties.
[Table]
[Table]
Site26.
Site30.
Exc.
A 1.
Exc.
B 1.
[Figure]