W HEN R IGID C OHERENCY H URTS : D ISTRIBUTIONAL C OHERENCY R EGULARIZATION FOR P ROBABILISTIC H IERARCHICAL T IME S ERIES F
ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Probabilistic hierarchical time-series forecasting is an important variant of timeseries forecasting, where the goal is to model and forecast multivariate time-series that have hierarchical relations.
Previous works all assume rigid consistency over the given hierarchies and do not adapt to real-world data that show deviation from this assumption.
Moreover, recent state-of-art neural probabilistic methods also impose hierarchical relations on point predictions and samples of predictive distribution.
This does not account for full forecast distributions being coherent with the hierarchy and leads to poorly calibrated forecasts.
We close both these gaps and propose PROFH I T, a probabilistic hierarchical forecasting model that jointly models forecast distributions over the entire hierarchy.
PROFH I T (1) uses a flexible probabilistic Bayesian approach and (2) introduces soft distributional coherency regularization that enables end-to-end learning of the entire forecast distribution leveraging information from the underlying hierarchy.
This enables robust and calibrated forecasts as well as adaptation to real-life data with varied hierarchical consistency.
PROFH I T provides 41-88% better performance in accuracy and 23-33% better calibration over a wide range of dataset consistency.
Furthermore,
PROFH I T can robustly provide reliable forecasts even if up to 10% of input timeseries data is missing, whereas other methods’ performance severely degrade by over 70%.
I NTRODUCTION
Time-series forecasting is an important problem that impacts decision-making in a wide range of applications.
In many real-world situations, the time-series have inherent hierarchical relations and structures.
Examples include forecasting time-series of employment (Taieb et al., 2017) measured at different geographical scales; epidemic forecasting (Reich et al., 2019) at county, state and country, etc.
Given time-series dataset with underlying hierarchical relations, the goal of Hierarchical Time-series
Forecasting ( HTSF ) is to generate accurate forecast for all time-series leveraging the hierarchical relations between time-series (Hyndman et al., 2011).
Most previous methods do not provide well-calibrated forecasts for both so-called "strong" and "weakly" consistent datasets.
Previous HTSF methods assume that the time-series values of datasets strictly satisfy the underlying hierarchical constraints and impose rigid coherency on generated forecasts i.e., forecasts strictly satisfy the hierarchical relations of dataset.
These methods can model datasets generated (Taieb et al., 2017) by first collecting data for time-series of the leaf level nodes and deriving time-series for higher-level nodes.
We call such data as strongly consistent .
For example, classical HTSF methods (Hyndman & Athanasopoulos, 2018) use a bottom-up or top-down approach where all time-series at a single level of hierarchy are modeled independently and the values of other levels are derived using the aggregation function governing the hierarchy.
In contrast, many real-world datasets are weakly consistent , i.e., they do not follow the strict constraints of the hierarchy 1 .
Such data have an underlying data generation process that may follow a hierarchical set of constraints but may contain some deviations.
These deviations can be caused by factors such as measurement or reporting error, asynchrony in data aggregation and revision pipeline, etc, as frequently observed in epidemic forecasting (Adhikari et al., 2019).
Most state-of-the-art HTSF methods are designed for strongly consistent datasets and impose rigid coherency constraints — they thus may not adapt to such deviations and can provide poor forecasts for weakly consistent datasets .
[Figure]
Moreover, previous methods do not focus on providing calibrated forecasts with precise uncertainty measures.
Traditional methods focus on point predictions only.
Recent methods like M IN T (Wickramasuriya et al., 2019), ERM (Ben Taieb & Koo, 2019) and PEMBU (Taieb et al., 2017) refine raw independent forecast distribution as a post-processing step.
This does not enable the models generating the raw forecasts to leverage underlying hierarchical relations across time-series.
End-to-end learning neural methods directly leverage hierarchical relations as part of the model architecture or learning algorithm like
H IER E2E (Rangapuram et al., 2021) and SHARQ (Han et al., 2021).
They do this and usually outperform post-processing methods by imposing hierarchical constraints on the mean or fixed quantiles of the forecast distributions.
However, these methods do not enforce hierarchical coherency on the full distributions.
Therefore, the forecasts may not be well-calibrated (Kuleshov et al., 2018) i.e., they produce unreliable prediction intervals that may not match observed probabilities from ground truth (Fisch et al., 2022).
[Table]
In this work, we fill this gap of learning well-calibrated and accurate forecasts for both strong and weakly consistent datasets leveraging underlying hierarchical relations.
We propose
PROFH I T Probabilistic
Robust Forecasting for
Hierarchical Time-series), a neural probabilistic HTSF method that provides an end-to-end Bayesian approach to model the distributions of forecasts of all time-series together (see Table 1 for a comparison).
Specifically, we introduce a novel Soft Distributional Coherency Regularization (SDCR) to tackle the challenge.
First, SDCR enables
PROFH I T to leverage hierarchical relations over entire forecast distributions to generate calibrated forecast distributions by encouraging forecast distribution of any parent node to be similar to aggregation of children nodes’ forecast distribution (Figure 1).
Second, since SDCR is a soft constraint, our model is trained to adapt to datasets with varying hierarchical consistency that allows the model to trade-off coherency for better accuracy and calibration on weakly consistent datasets.
Our main contributions are:
(1) Accurate and Calibrated Probabilistic Hierarchical Time-Series Forecasting: We propose PROFH I T, a deep probabilistic framework for modeling the distributions of each time-series together using a soft distributional coherency regularization (SDCR).
PROFH I T leverages probabilistic deep-learning models to learn priors of individual time-series and refines the priors of all time-series leveraging the hierarchy to provide accurate and well-calibrated forecasts.
(2) Adaptation to Strong and Weak Consistency via Soft Distributional Coherency Regularization: SDCR imposes soft hierarchical constraints on the full forecast distributions to help adapt the model to varying levels of hierarchical consistency.
We build a novel refinement module over raw forecast priors and leverage multi-task learning over shared parameters that enable PROFH I T to perform consistently well across the hierarchy.
(3) Evaluation Across Multiple Datasets and with Missing Data: We show that our method PROFH I T outperforms a wide variety of state-of-the-
art baselines on both accuracy and calibration, at all levels of the hierarchy, for both strong and weakly consistent datasets.
We also show training using SDCR enables PROFH I T to leverage hierarchical relations to provide robust predictions that can handle missing data values in the time-series.
P ROBLEM S TATEMENT
Consider the dataset D of N time-series over the time horizon 1 , 2 , . .
, T .
Let
y i ∈
R T be timeseries i and y ( t )
i its value at time t .
The time-series have a hierarchical relationship denoted as
T =
( G T , H T ) where G T is a tree of N nodes rooted at time-series 1 .
For a non-leaf node (timeseries) i , we denote its children as C i .
The node values are related via set of relations H T of form
H T = { y i =
(cid:80) j ∈C i ϕ
ij y j : ∀ i ∈ { 1 , 2 , . . . , N } , |C i | > 0 } where values of ϕ ij are known and time-independent real-valued constants.
Definition 1 (Consistency Error - CE) .
Given a dataset D of N time-series over the time horizon 1 , 2 , . . .
, T and aggregation relations H T as above, the dataset consistency error (CE) is defined as
[Equation]
(Intuitively, datasets with lower CE have time-series values which more strictly follow relations H T ).
Definition 2 (Strong and weak consistency) .
A dataset D is strongly consistent if E T ( D ) = 0 .
Otherwise, D is said to be weakly consistent.
Let current time-step be t .
For any 1 ≤ t 1 < t 2 ≤ t , we denote y ( t 1 : t 2 ) i
= { y ( t 1 ) i , y ( t 1 +1) i , . . .
, y ( t 2 ) i } .
Given the data D t =
[ y 1: t 1 , y 1: t 2 , . . .
, y 1: tN ] and hierarchical relations H T , a model M is trained to predict the marginal forecast distributions at time t + τ for all time-series of hierarchy leveraging past values of all time-series: { p M ( y ( t + τ )|D t ) , . . .
p M ( y ( t + τ ) N |D t ) } .
Along with accuracy of probabilistic forecasts we also evaluate forecast distributions for calibration .
We define calibration of model forecasts based on previous works (Kamarthi et al., 2021; Kuleshov et al., 2018):
Definition 3.
(Calibration Score of Model)
Given a model M we define a calibration function k M : [0 , 1] → [0 , 1] as follows:
Given a confidence c , k M ( c ) is the fraction of the predictions for which the ground truth lies within c -confidence interval.
The calibration score CS ( M ) is the total deviation between c and k M ( c ) : CS ( M ) =
(cid:82) 1
| k M ( c ) − c | dc .
A perfectly calibrated model is such that ∀ c : k M ( c )
≈ c .
Given a dataset D with underlying hierarchical relations H T , the goal of Calibrated Probabilistic
Hierarchical Forecasting is to design a model M that provides accurate and well-calibrated forecast distributions { p M ( y ( t + τ )|D t ) , . . .
p M ( y ( t + τ ) N |D t ) } across all levels of the hierarchy for both weakly and strongly consistent datasets.
M ETHODOLOGY
[Figure]
Overview
PROFH I T models the forecast distributions { P ( y ( t + τ ) i |D t ) }
Ni =1 of all time-series nodes of the hierarchy by leveraging the relations from the hierarchy to provide accurate and well-calibrated forecasts that are adaptable to varying hierarchical consistency.
Most existing methods do not attempt to model entire probabilistic distribution but focus on coherency of point forecasts or samples or fixed quantiles of the distribution (Rangapuram et al., 2021; Han et al., 2021).
This approach does not fully capture the uncertainty of the forecasts and in turn does not provide calibrated predictions.
Moreover, most methods operate on datasets that are strongly consistent over hierarchical relations.
However, many real-world datasets are weakly consistent with time-series values of all nodes of hierarchy observed simultaneously and may not follow the hierarchical relations strictly due to noise and discrepancies in collecting data at different levels.
Therefore, most previous works may not adapt well to such deviations from these constraints.
PROFH I T, on the other hand, reconciles the need to model coherency between entire forecast distributions as well as induce a soft adaptable constraint to enforce coherency via a two-step stochastic process that is trained in an end-to-end manner.
PROFH I T first produces a raw forecast distribution for each node parameterized by { (ˆ µ i , ˆ σ i ) } Ni =1 by using past values of time-series via a neural probabilistic forecasting model.
Raw forecasts of all nodes are used as priors to derive a refined set of forecast distributions parameterized by { ( µ i , σ i ) } Ni =1 via the refinement module .
The full probabilistic process of PROFH I T is depicted in Figure 2 and formally summarized as:
[Equation]
where z i , u i , z are intermediate latent variables of our probabilistic raw forecasting model TSFNP (Section 3.1).
PROFH I T’s SDCR regularizes the parameters { ( µ i , σ i ) } Ni =1 to leverage the hierarchical relations by minimizing the Distributional Coherency Error (DCE) defined as follows:
Definition 4.
(Distributional Coherency Error - DCE)
Given the forecasts at time t + τ as { p M ( y
[Equation]
} distributional coherency error (DCE) is defined as
[Equation]
where Dist is a distributional distance metric.
Leveraging distributional coherency error as a soft regularizer enforces forecast distributions to be well-calibrated while adaptively adhering to hierarchical relations of the dataset.
3. R AW FORECAST DISTRIBUTIONS FROM NGP S
NGPs (Neural Gaussian Process) (Louizos et al., 2019) are novel class of probabilistic neural models state-of-the-art accurate and calibrated predictions.
We, therefore, use a modified form of a stateof-the-art NGP model for time-series forecasting (Kamarthi et al., 2021) which we call TSFNP to derive raw forecast distributions for each time-series.
We briefly describe TSFNP’s three components: 1) Probabilistic Neural Encoder :
It encodes the input univariate time-series into a latent stochastic embedding via a GRU (Cho et al., 2014) followed by a self-attention layer (Vaswani et al., 2017): [
[Equation]
i [Equation] [Equation] 2) Stochastic Data Correlation Graph : We further leverage similar patterns of past time-series data and aggregate them as local latent variable .
Unlike E PI FNP which uses past time-series information from same node, in our multi-variate case TSFNP uses past information from all nodes.
Formally, for input sequence y ( t ′ : t ) i and each of the past sequence y j where j ∈ { 1 , . . .
, N } , we sample y j with probability exp( − γ
||
u
i − u j || 22 ) into set N i .
Then, we derive the local latent variable as
[Equation]
where Θ 1 and Θ 2 are feed-forward networks.
3) Predictive Distribution Decoder : Finally, we combine the latent embedding of input time-series, local latent variable and combined information of all past sequences to derive the parameters of the output distribution via a simple feed-forward network.
We first derive a global latent variable that combines the information from latent embeddings of all past sequences via self-attention:
[Equation]
Finally, we combine the latent embedding of input time-series, local latent variable and global latent variable to derive the raw forecast distribution modelled as a Gaussian N (ˆ µ i , ˆ
[Equation]
i ) as:
[Equation]
where Θ 3 is a feed forward network.
3. R EFINEMENT MODULE
The refinement module leverages the raw distributions of all nodes of hierarchy to produce refined forecast distributions using hierarchical relations.
Given the parameters of raw forecast distributions { ˆ µ i , ˆ σ i } Ni =1 derived from TSFNP for all time-series { y ( t ′ : t ) i } Ni =1 , the refinement module derives the refined forecast distributions denoted by parameters { µ i , σ i } Ni =1 as functions of parameters of raw forecasts of all time-series.
The refined forecasts are optimized to be more coherent using SDCR.
Since we drive the full distributions of refined forecasts to be coherent, rather than just the samples or mean statistics, the refined distributions’ calibration is also consistent with the hierarchical relations.
Let ˆ µ =
[ˆ µ 1 . .
, ˆ µ N ] and ˆ σ =
[ˆ σ 1 . .
, ˆ σ N ] be vectors of means and standard deviations of raw distributions.
We model the refined mean as a function of the raw means of all the nodes.
Formally, we derive the mean µ i of refined distribution as a weighted sum of two terms: a) ˆ µ i , the mean of raw time-series, and b) linear combination of all raw mean of all time-series:
[Equation]
{ ˆ w i } Ni =1 and { w i } i =1: N are both learnable set of parameters of the model.
sigmoid ( · ) denotes the sigmoid function.
γ i helps model the trade-off between the influence of the raw distribution of node i and the influence of the other nodes of the hierarchy.
This is useful for the model to automatically adapt to datasets with varying hierarchical consistency.
Similarly, we assume the variance of the refined distribution depends on the raw mean and variance of all the time-series.
The variance parameter σ i of the refined distribution is derived from the raw distribution parameters ˆ µ and ˆ σ as
[Equation]
where { v 1 i } Ni =1 , { v 2 i } Ni =1 and { b i } Ni =1 are parameters and c is a positive constant hyperparameter.
3. L IKELIHOOD L OSS AND R EGULARIZATION OVER H IERARCHY
We optimize the probabilistic process of Equation 2 for accuracy and calibration by leveraging hierarchical relations by training on likelihood loss on ground truth training data as well as SDCR.
Likelihood Loss To maximise the likelihood of forecasts over ground truth P ( { y ( t + τ ) i } Ni =1 |D t ) we use variational inference by approximating the posterior (cid:81) Ni =1 P ( z i , u ( j ) i |D t ) with the variational distribution (cid:81)
Ni =1 P ( u i |
y
( t ′ : t ) i )
q i ( u i | y ( t ′ : t ) i ) where q i is a feed-forward network over GRU hidden embeddings of Probabilistic Neural Encoder that parameterizes the Gaussian distribution of q i ( u i | y ( t ′ : t ) i ) .
We derive the ELBO (detailed derivation in Appendix) as
L
[Equation]
Soft Distributional Coherency Regularization
PROFH I T leverages the hierarchy relations in T and regularizes the refined distributions to be coherent.
Since PROFH I T aims to leverage hierarchical coherency for improved robustness and calibration, we regularize over the full distributions by using distributional coherency error as part of the loss function.
We use the Jensen-Shannon Divergence (Endres & Schindelin, 2003) (JSD) as the distance metric since it is a symmetric and bounded variant of the popularly used KL-Divergence distance and assumes closed form for many widely used distributions.
We derive the distributional coherency error on { ( µ i , σ i ) } Ni =1 as
[Equation]
=
[Equation]
Computation of JSD is generally intractable.
However, in our case, due to parameterization of each time-series distribution as a Gaussian we get a closed-form differentiable expression:
[Equation]
We provide the derivation of Equation 12 in the Appendix.
We use the distributional coherency error as a soft regularization term to enable PROFH I T to leverage constraints H T when generating forecast distributions.
Thus, the total loss for training is given as L =
L 1 + λ L 2 where the hyperparameter λ controls the trade off between data likelihood and coherency.
We also use the reparameterization trick to make the sampling process differentiable and we learn the parameters of all training modules via Stochastic Variational Bayes (Kingma & Welling, 2013).
The full pipeline of PROFH I T is summarized in Figure 2.
3. D ETAILS ON T RAINING
Parameter sharing across nodes Since PROFH I T’s TSFNP module forecasts for multiple nodes, we leverage the hard-parameter sharing paradigm of multi-task learning (Caruana, 1997) and use different set of parameters for Predictive Distribution Decoder (i.e., weights of Θ 3 for each timeseries i is different) whereas the parameters of other components of TSFNP are shared across all nodes (Figure 2).
Sharing parameters for Probabilistic Neural Encoder drastically lowers the number of learnable parameters since datasets can have large number of nodes (up to 512 nodes in our experiments).
Pre-training on individual time-series Before we start training for refined forecasts, we pre-train the parameters of TSFNP on given training dataset to model raw forecast distribution accurately.
We train using only a log likelihood loss to learn parameters { ˆ µ i , ˆ σ i } Ni =1 similar to Equation 10.
E XPERIMENTS
We evaluate PROFH I T over multiple datasets and compare it with state-of-the-art baselines 2 .
4. S ETUP
Baselines: We compare PROFH I T’s performance against state-of-the-art HTSF methods.
We also compare against state-of-the-art general probabilistic forecasting methods to study the importance of modeling the hierarchy for both weak and strongly consistent datasets.
(1) TSFNP (Kamarthi et al., 2021) and (2) D EEP AR (Salinas et al., 2020)
as state-of-the-art deep probabilistic forecasting models which do not exploit hierarchy relations.
(3) M IN T (Wickramasuriya et al., 2019) and (4) ERM (Ben Taieb & Koo, 2019) are methods that convert incoherent forecasts as post-processing step by framing it as an optimization problem.
Since TSFNP provided better evaluation scores compared to
D EEP AR, we performed ERM and M IN T on Monte Carlo samples of TSFNP predictive distribution.
(5) H IER E2E (Rangapuram et al., 2021) is a recent state-of-the-art deep-learning based approach that projects the raw predictions onto a space of coherent forecasts and trains the model in an end-to-end manner.
(6) SHARQ (Han et al., 2021) is another state-of-the-art deep learning based approach that reconciles forecast distributions by using quantile regressions and making the quantile values coherent.
(7) PEMBU (Taieb et al., 2017) is a post-processing method that refines raw forecasts to be distributionally coherent.
We use the mean forecast from M IN T and ERM as input forecasts for
PEMBU.
Note that we fine-tune the hyperparameters of PROFH I T and each baseline specific to each benchmark.
More details on hyperparameters are in Appendix.
We also evaluate the efficacy and contribution of our various modeling choices by performing an ablation study using the following variants of PROFH I T: (7) P-G LOBAL : We study the effect of our multi-tasking hard-parameter sharing approach (Section 3.4) by training a variant where all the parameters are shared across all the nodes.
(8) P-F INE T UNE : We also look at the efficacy of our soft regularization using both losses that adapts to optimize for both coherency and training accuracy by comparing it with a variant where the predictive distribution decoder parameters are further fine-tuned for individual nodes using only the likelihood loss.
(9) P-D EEP AR:
We evaluate our choice of using TSFNP, a previous state-of-the-art univariate forecasting model for accurate and calibrated forecasts with DeepAR, another popular probabilistic forecasting model that was used by
H IER E2E. (10) P-N O C OHERENT :
This variant is trained by completely removing the SDCR from the training.
Note that unlike P-F INE T UNE which was initially trained with SDCR before fine-tuning,
P-N O C OHERENT never uses the SDCR at any point of training routine.
Therefore P-N O C OHERENT measures the importance of explicitly regularizing over the information from the hierarchy.
[Table]
Datasets: We evaluate on a diverse set of publicly available datasets (Table 2) from different domains with varied hierarchical relations and consistency.
The bechmarking dataset and evaluation setup including forecast horizon is replicated from recent and past literature related to general HTSF as well as epidemic forecasting.
(1) Labour dataset contains monthly employment data from Feb 1978 to Dec 2020 collected from Australian Bureau of Statistics.
(2) Tourism-L (Wickramasuriya et al., 2019) contains tourism flows in different regions in Australia grouped via region and demographic.
It has two sets of hierarchy (with four and five levels), one for the mode of travel and the other for geography with the top node being the only common node of both hierarchies.
Wiki dataset collects the number of daily views of 145000 Wikipedia articles aggregated intogroups (Taieb et al., 2017).
These 150 groups are leaf nodes of a four-level hierarchy with groups of similar topics aggregated together.
(4) Flu-Symptoms contains flu incidence values called weighted influenza-like incidence (wILI) values (Reich et al., 2019) at multiple spatial scales for USA for period of 2004-2020.
The scales used are states, HHS and National level (US states are grouped into 10 HHS regions by CDC).
(5) FB-Survey provides aggregated anonymized daily indicator for the prevalence of Covid-19 symptoms based on online surveys conducted on Facebook (Delphi
Research Group, 2021) from Dec 2020 to Aug 2021 for each state and national level.
We use the state-level values to find aggregates at HHS levels.
Tourism-L , Labour and Wiki are constructed by collecting values of leaf nodes and deriving the values of time-series of other nodes of the hierarchy.
Hence, they are strongly consistent with zero CE (Definition 1).
The values of each node of hierarchy in case of Flu-Symptoms and FB-Survey are directly collected or measured.
For example, the values of Flu-Symptoms dataset are collected from public health agencies at the state, HHS and national levels and aggregated by CDC.
Due to factors like reporting discrepancies and noise they contain values in time-series that may deviate from the given hierarchical relations (Chakraborty et al., 2018).
Therefore, these datasets are weakly consistent with significant CE
(Table 2).
Evaluation metrics For a ground truth y ( t ) , let the predicted probability distribution be ˆ p y ( t ) with mean ˆ y ( t ) .
Also let ˆ F y ( t ) be the CDF.
We evaluate our model and baselines using carefully chosen metrics that are widely used in literature to measure accuracy and calibration.
Mean
Absolute Percentage Error (MAPE) is a commonly used score for point-predictions calculated as MAPE =
[Equation]
) | 3.
Log Score (LS) is a standard score used to measure accuracy of probabilistic forecasts in epidemiology (Reich et al., 2019).
LS measures the negative log likelihood of a fixed size interval around the ground truth under the predictive distribution: LS (ˆ p y , y )
= −
(cid:82) y
+ L y − L log ˆ p y
(ˆ y )
d ˆ y. Similar to (Reich et al., 2019), log likelihood of a forecast is capped at -10.
Calibration Score (CS): To measure calibration of forecasts, we use the calibration score defined in Section 2. 2.
Cumulative Ranked Probability Score (CRPS) is a widely used standard metric for evaluation of probabilistic forecasts that measures both accuracy and calibration .
Given ground truth y and the predicted probability distribution ˆ p y , let ˆ F y be the CDF.
Then, CRPS is defined as: CRPS ( ˆ F y , y )
=
(cid:82) ∞
−∞ ( ˆ F y (ˆ y ) − 1 { ˆ y > y } )
d ˆ y.
We approximate ˆ F y as a Gaussian distribution formed from samples of model to derive CRPS.
Distributional Coherency Error (DCE):
We calculate the Distributional Coherency Error (Equation 11) on output forecast distributions during inference to study how PROFH I T and baselines leverage SDCR to learn from hierarchical relations across datasets of varying consistency and trade-off coherency, calibration and accuracy, especially for weakly consistent data (Section 4.2 Q3).
4. R ESULTS
We comprehensively evaluate PROFH I T through the following questions: Q1: Does PROFH I T predict accurate calibrated forecasts?
Q2:
Does PROFH I T provide consistently better performance across all levels of the hierarchy?
Q3:
Does SDCR help PROFH I T outperform baselines on both strongly and weakly consistent datasets?
Q4: How does improved calibration affect robustness of
PROFH I T’s forecasts?
[Table]
Accuracy and calibration performance (Q1)
We evaluate all baselines, PROFH I T and its variants for all the datasets over 5 independent runs.
The average scores across all levels hierarchy are shown in Tables 3.
PROFH I T significantly outperforms all baselines in MAPE score by 13% and LS by 14%-550%.
In terms of calibration, we observe an average of 32% lower CS scores.
Finally,
PROFH I T shows 41-88% better CRPS scores.
Thus, PROFH I T adapts well to varied kinds of datasets and outperforms all baselines in both accuracy and calibration.
Performing t-test with significance α = 1% we find that all the CRPS, LS and CS scores are statistically significant compared to baselines.
On comparing the performance of PROFH I T with the variants, PROFH I T is comparable to or better than the best-performing variant in most benchmarks.
This shows that all the important model design choices (multi-task parameter sharing, distributional coherency, and joint training on both losses) of PROFH I T are important for its consistently superior performance.
Performance across the hierarchy (Q2)
Next, we look at the performance of all models across each level of hierarchy.
We compared the performance of PROFH I T with best performing baselines
H IER E2E and SHARQ for all datasets.
PROFH I T significantly outperforms the best baselines.
At the leaf nodes, which contain most data, PROFH I T outperforms best baselines by 7% in Wiki to 100% in FB-Survey .
For the top node of time-series the performance improvement is largest at 35% ( Wiki ) to 962% ( FB-Survey ).
Similarly, for calibration score, we observe an average improvement of 12% for top nodes and 18% for bottom nodes.
We show detailed results in Appendix.
PROFH I T also performs better than the variants in most higher levels of hierarchy and its performance is comparable to the best variant (P-F INE T UNE and P-G LOBAL ) at leaf nodes as well.
P-N O C OHERENT performs most poorly compared to all variant and PROFH I T, proving that SDCR is a very important contributor for consistent performance across the hierarchy in all datasets.
SDCR leads to consistently better performance across varying data consistency (Q3)
As discussed in Section 4.2, we evaluated on both strong and weakly consistent datasets.
Since most previous state-of-the-art models assume datasets to be strongly consistent, deviations from this assumptions can cause under-performance when used with weakly consistent datasets.
This is evidenced in Table 3 where most of the baselines explicitly optimize for hierarchical coherency as a hard constraint on the forecasts.
For example, PEMBU’s forecasts have better distributional coherency error (DCE) for weakly consistent datasets.
However, they perform much worse in both accuracy and calibration than even TSFNP, which does not even leverage hierarchical relations.
Since we use SDCR as soft learning constraint, PROFH I T can learn to trade-off coherency for accuracy and calibration.
Therefore, PROFH I T provides 93% better CRPS and 33% better calibration scores over best HTSF baselines.
These improvements are more pronounced at non-leaf nodes of hierarchy where PROFH I T improves by 2.8 times for Flu-Symptoms and 9.2 times for FB-Survey .
In case of strongly consistent datasets, PROFH I T provides 54% better CRPS and 23% better calibration scores while having comparable DCE to PEMBU.
We observed that soft coherency regularization and parameter sharing across nodes are vital for PROFH I T’s adaptability to varying levels of consistency.
We provide detailed analysis of these observations in the Appendix.
[Figure]
% C R P S
i n c r e a s e
TSFNPHierE2ESHARQPROFHiT (e) FB-Survey
Calibration enables robustness (Q4)
Accurate and well-calibrated models that can effectively leverage knowledge of the hierarchy can intuitively allow models to better adapt to noise/missing data.
Hence we introduce the task of Hierarchical Forecasting with Missing Values to study the robustness of models when there are missing values in time-series.
We model a situation that is encountered in many real-world applications such as Epidemic Forecasting where the past few values of time-series are missing due to various factors like data reporting delays (Chakraborty et al., 2018).
Formally, at time-period t , we are given full data up to time t − ρ .
We set ρ = 5 since it is the average forecast horizon of all datasets.
For sequence values in time period between t − ρ and t , we randomly remove k % of these values across all time-series.
The models are trained on complete time-series dataset till time t ′ =
t − ρ .
Models’ predictions are then used to fill in missing values for time t ′ to t .
Finally, we input the filled time-series to generate forecasts for the future time-steps.
We measure relative decrease in performance with increase in percentage of missing data k (Figures 3).
We observe that PROFH I T’s performance decrease with a larger fraction of missing values is much slower compared to other baselines.
Even at k = 10% , PROFH I T’s performance decreases by 10.45-26.8% compared to other baselines that typically decrease by over 70%.
Thus, PROFH I T effectively uses coherency to generate robust predictions on strong and weakly consistent datasets.
C ONCLUSION AND D ISCUSSION
We introduced PROFH I T, a probabilistic hierarchical forecasting model that produces accurate and well-calibrated forecasts using soft distributional coherency regularization (SDCR) which enables adaptablity to datasets with varying levels of hierarchical consistency.
We evaluated PROFH I T against previous state-of-the-art hierarchical forecasting baselines over wide variety of datasets and observed 41-88% improvement average improvement in accuracy and 23-33% better calibration.
PROFH I T provided best performance across the entire hierarchy as well as significantly outperformed other models in providing robust predictions when it encountered missing data where other baselines’ performance degraded by over 70%.
Our work opens new possibilities like extending to various domains where time-series values across the hierarchy may not be continuous real numbers, can not be modelled as Gaussian distributions or may have different sampling rates.
We can also explore modeling more complex structures between time-series with different aggregation relations.
PROFH I T can also be used to study anomaly detection in time-series, especially in time-periods where there are deviations from assumed coherency relations.
Similar to Kamarthi et al.
(2022), we can extend our work to include multiple sources of features and modalities of data both specific to each time-series and global to the entire hierarchy.
R Appendix for PROFH I T: Probabilistic Robust Forecasting for Hierarchical Time-series A A DDITIONAL R ELATED W ORK
Probabilistic time-series forecasting Classical probabilistic time-series forecasting methods include exponential smoothing and ARIMA (Hyndman & Athanasopoulos, 2018).
They are simple but focus on univariate time-series and model each time-series sequence independently.
Recently, deep learning based methods have been successfully applied in this area.
DeepVAR (Salinas et al., 2020) trains an auto-regressive recurrent network model on a large number of related time series to directly output the mean and variance parameters of the forecast distribution.
Other works are inspired from the space-state models and explicitly model the transition and emission components with deep learning modules such as deep Markov models (Krishnan et al., 2017) and deep state space models (Li et al., 2021;
Rangapuram et al., 2018)
Recently, EpiFNP (Kamarthi et al., 2021) has achieved state-of-art performance in epidemic forecasting.
It learns the stochastic correlations between input data and datapoints to model a flexible non-parametric distribution for univariate sequences.
Hierarchical time-series forecasting Classical works on hierarchical time-series forecasting used a two-step approach (Hyndman et al., 2011; Hyndman & Athanasopoulos, 2018) and focus on point predictions.
They first forecast for time-series only at a single level of the hierarchy and then derive the forecasts for other nodes using the hierarchical relations.
Recent methods like M IN T and ERM are post-processing steps applied on the set of forecasts at all levels of hierarchy.
M IN T (Wickramasuriya et al., 2019; Wickramasuriya, 2021) assumes that the base level forecasts are uncorrelated and unbiased and solves an optimization problem to minimize the variance of forecast errors of past predictions.
The unbiased assumption is relaxed in ERM (Ben Taieb & Koo, 2019).
Corani et al.
(2020) and Novak et al.
(2017) uses a fully Bayesian bottom-up post-processing approach using raw forecasts from full hierarchy.
Another line of works projects the raw forecasts of all time-series into a subspace of coherent forecasts.
(Erven & Cugliari, 2015) use an iterative Game-theoretic approach of minimizing forecast error and projection error.
Taieb et al.
(2017) uses copula method to refine raw forecasts to be distributionally coherent as a post-processing step.
Recent neural methods perform end-to-end learning that enables the model to leverage hierarchical relations while forecasting.
Rangapuram et al.
(2021) use a deep-learning based end-to-end approach to directly train on the projected forecasts.
SHARQ (Han et al., 2021) is another recent probabilistic deep-learning based method that uses quantile regression and regularizes for coherency at different quantiles of forecast distribution.
However, unlike our approach, these endto-end methods do not regularize for coherency over the entire distribution (Distributional Coherency) but only over fixed quantiles.
Most of these methods also are not designed for cases where the hierarchical constraints are not always consistently followed.
B C ODE AND D ATASET
We evaluated all models on a system with Intel 64 core Xeon Processor with 128 GB memory and Nvidia Tesla V100 GPU with 32 GB VRAM.
We provide an anonymized repository of our implementation of PROFH I T along with the datasets used at https://anonymous.4open.
science/r/PROFHiT-6F2F .
We will release the code and data publicly after acceptance.
C H YPERPARAMETERS C. D ATA P REPROCESSING
Most datasets used in our work assume the aggregation function to be simple summation (i.e,
ϕ ij =for all weights).
We first normalize the values of leaf time-series training data to have 0 mean and variance of 1.
Since the aggregation of values at higher levels of the hierarchy can lead to very large values in time-series, we instead divide each non-leaf time-series by the number of children.
Then the weights of hierarchical relations become ϕ ij = 1 | C i | where C i is the set of all children nodes of time-series i .
For the remaining datasets (Flu-Symptoms, FB-Symptoms) the time-series values are normalized by default and thus require no extra pre-processing.
C. M ODEL A RCHITECTURE
The architecture of TSFNP used in PROFH I T is similar to that used in the original implementation (Kamarthi et al., 2021).
The GRU unit contains 60 hidden units and is bi-directional.
Thus the local latent variable is also of dimension 60.
NN 1 and NN 2 are both 2-layered neural networks with the first layer shared between both.
Both layers have 60 hidden units.
Finally, NN 3 is a three-layer neural network with the input layer having 180 units (for the concatenated input of three 60 dimensional vectors) and the last two layers having 60 hidden units.
We found that the value of c in Equation 9 is not very sensitive and usually set it to 5.
Note that we do not explicitly model covariance between every pair of time series (like M IN T, ERM) and use a weighted combination of raw forecast parameters to derive refined forecasts.
Therefore the refinement module complexity (Section 3.2) is O ( N 2 ) which is on par with previous methods like
H IER E2E.
C. T RAINING AND E VALUATION
Given the training dataset D t we extract training dataset for each node as the set of prefix sequences { ( y ( t 1: t 2) i , y ( t 2+1) i ) : 1 ≤ t 1 ≤ t 2 < t − τ } and train the full model (TSFNP and refinement module).
We tune the hyperparameter using backtesting by validating on window t − τ to t .
Finally we train for entire training set with best hyperparameters.
For each benchmark, we used the validation set to mainly find the optimal batch size and learning rate.
We searched over batch-size of { 10 , 50 , 100 , 200 } and the optimal learning rate was usually around 0.001.
We also found the optimal λ to be around 0 .
01 for strongly consistent datasets and 0 .
001 for weakly consistent datasets.
We used early stopping with the patience of 150 epochs to prevent overfitting.
For each independent run of a model, we initialized the random seeds from 0 tofor PyTorch and NumPy.
We didn’t observe large variations due to randomness for PROFH I T and all baselines.
During evaluation, we sampled 2000 Monte-Carlo samples of the forecast distribution and used it to estimate the mean for MAPE.
We also used the samples mean and variance to evaluate LS and CS whereas used ensemble scoring to evaluate CRPS directly from the samples using properscoring package 3 .
D D ERIVATION OF L IKELIHOOD ELBO LOSS
The full predictive distribution of PROFH I T from Equation 2 can be further expanded as:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
To minimize the data likelihood P ( { y ( t + τ ) i } Ni =1 |D t ) requires intregration over latent variables { u i } Ni =1 and z i } Ni =1 .
We instead perform amortized variational inference on the latent variables similar to VAE (Kingma & Welling, 2013).
We approximate the posterior of latent variables P ( { u i } Ni =1 , { z i } Ni =1 , { N i } Ni =1 , z |{ y ( t + τ ) i } Ni =1 ) with a variational distribution Q ( u i } Ni =1 , z i } Ni =1 , { N i } Ni =1 , z |{ y ( t + τ ) i } Ni =1 ) expressed as:
[Equation]
where q ϕ is a feed-forward network over GRU embeddings of Probabilistic Neural Encoder that parameterizes to a gaussain distribution of z i .
The ELBO loss
[Equation]
get simplified to Equation 10 by cancelling of similar terms between variational and true distribution of latent variables.
E D ERIVATION OF D ISTRIBUTIONAL C OHERENCY E RROR
The Distributional Coherency Error (Equation 11) can be exactly expressed as:
[Equation]
To derive Equation [Equation] we use the following well-known result for JSD of two Gaussian Distributions (Nielsen, 2019):
Lemma 1.
Given two univariate Normal distributions P 1 = N 1 ( µ 1 , σ 1 ) and P 2 = N 2 ( µ 2 , σ 2 ) , the
JSD is
[Equation]
Consider each JSD term
[Equation]
of the summation in Equation 11.
Note that [Equation] [Equation] and P ( (cid:80) j ∈ C
i ϕ ij y t
+ τ j |{| ˆ µ j , ˆ σ j } j ∈ C i ))
is weighted sum of Gaussian variables {N ( µ j , σ j ) } j ∈ C i .
Therefore,
[Equation]
Using Lemma 1 along with Equations 18,19 we get the desired result in Equation 16. [Table]
F C ONSISTENCY OF DATASETS
We noted in Section 4.2 Q4 that Flu-Symptoms and FB-Survey are weakly consistent datasets since they do not strictly follow the aggregation relations H T unlike strongly consistent datasets
Tourism-L, Labour, Wiki .
We empirically observe this by measuring Consistency errors of all datasets (Definition 1) for entire hierarchy and at each level of the hierarchy.
The results are in Table 4.
As expected there is no deviations for strongly consistent datasets where as there is significant deviation in weakly consistent data.
G P ERFORMANCE ACROSS EACH LEVEL OF HIERARCHY
[Table]
[Table]
H D ETAILS ON D ATA I MPUTATION E XPERIMENT
Motivation: During real-time forecasting in real-world applications such as Epidemic or Sales forecasting, we encounter situations where the past few values of time-series are missing or unreliable for some of the nodes.
This is observed specifically at lower levels, due to discrepancies or delays during reporting and other factors (Chakraborty et al., 2018).
Therefore, one approach to performing forecasting in such a situation is first by imputation of missing values based on past data and then using the predicted missing values as part of the input for forecasting.
Task:
To simulate such scenarios of missing data and evaluate the robustness of PROFH I T and all baselines, we design a task called Hierarchical Forecasting with Missing Values (HFMV).
Formally, at time-period t , we are given full data for up to time t − ρ .
We show results here for ρ = 5 which is the average forecast horizon of all tasks.
For sequence values in time period between t − ρ and t , we randomly remove k % of these values across all time-series.
The goal of HFMV task is to use the given partial dataset from t − ρ to t as input along with complete dataset for time-period before t − ρ to predict future values at t + τ .
Therefore, success in HFMV implies that models are robust to missing data from recent past by effectively leveraging hierarchical relations.
Setup: We first train PROFH I T and baselines on complete dataset till time t ′ and then fill in the missing values of input sequence using the trained model.
Using the predicted missing values, we again forecast the output distribution.
For each baseline and PROFH I T, we perform multiple iterations of Monte-Carlo sampling for missing values followed by forecasting future values to generate the forecast distribution.
We estimate the evaluation scores using sample forecasts from all sampling iterations.
We compared the performance of PROFH I T with best performing baselines H IER E2E and SHARQ for each level of hierarchy of all datasets.
PROFH I T significantly outperforms the best baselines as well as the variants.
At the leaf nodes, which contains most data, PROFH I T outperforms best baselines by 7% in Wiki to 100% in FB-Survey .
For the top node of time-series the performance improvement is largest at 35% ( Wiki ) to 962% ( FB-Survey ).
We show detailed results in Table
[Figure]
Robustness of PROFH I T variants: We compare relative performance decrease with increase in percentage of missing data for PROFH I T and its variants in Figure 4.
We observe that PN O C OHERENT ’s performance deteriorates very rapidly in most benchmarks, showing the importance of SDCR for learning provides robust calibrated coherent forecasts.
The second worse-performing variant across all datasets is P-F INE T UNE which also relies less on the hierarchical relations due to fine-tuning of parameters for specific time-series.
Finally, we observe that PROFH I T and P-G LOBAL suffer the least degradation in performance since both these models prioritize integrating hierarchical coherency information which enables them to provide better estimates for imputed data for missing input and use them to generate more accurate and calibrated forecasts.
I A DAPTING TO VARYING DATASET CONSISTENCY
Observation 1.
The average improvement in performance of PROFH I T over best HTSF baselines is 72% higher for weakly consistent datasets over its improvement for strongly consistent datasets.
Since most previous state-of-art models assume datasets to be strongly consistent, deviations to this assumptions can cause under-performance when used with weakly consistent datasets.
This is evidenced in Table 3 where some of the baselines like M IN T and ERM that explicitly optimize for hierarchical coherency perform worse than even TSFNP, which does not leverage hierarchical relations, in Flu-Symptoms and FB-Survey .
Overall, we found that for weakly consistent datasets, PROFH I T provides a much larger 93% average improvement in CRPS scores over best
HTSF baselines compared to 54% average improvement for strongly consistent datasets.
These improvements are more pronounced at non-leaf nodes of hierarchy where PROFH I T improves by 2.
times for Flu-Symptoms and 9.2 times for FB-Survey .
This is because HTSF baselines which assume strong consistency do not adapt to noise at leaf nodes that compound to errors at higher levels of hierarchy.
Observation 2.
PROFH I T ’s approach to parameter sharing and soft coherency regularization helps adapt to varying hierarchical consistency.
We observe that that best performing variant for strongly consistent datasets in P-G LOBAL which is trained with both likelihood loss and SDCR (Table 3).
But its performance severely degrades for weakly consistent datasets since sharing all model parameters across all time-series makes it inflexible to model patterns and deviations specific to individual nodes.
In contrast, P-F INE T UNE and
P-N O C OHERENT performs the best among variants for weakly consistent datasets since they train separate sets of decoder parameters for each node.
But they perform poorly for strongly consistent datasets since they don’t leverage Distributional Coherency effectively.
PROFH I T combines the flexible parameter learning of P-F INE T UNE and leverage Distributional Coherency to jointly optimize the parameters like P-G LOBAL providing comparable performance to best variants over all datasets.
Observation 3.
PROFH I T ’s Refinement module automatically learns to adapt to varying hierarchical consistency.
of same time-series) where as strongly-coherent data have lower γ i (leverages the hierarchical relations).
[Table]
The design choices of the refinement module help PROFH I T to adapt to datasets of different levels of hierarchical consistency.
Specifically, by optimizing for values of { γ i } Ni =1 of Equation 8, PROFH I T aims to learn a good trade-off between leveraging prior forecasts for a time-series and hierarchical relations of forecasts from the entire hierarchy.
We study the learned values of { γ i } Ni =1 of Equation 8 used to derive refined mean.
Note that higher values of γ i indicate larger dependence on raw forecasts of node and smaller dependence of forecasts of the entire hierarchy.
We plot the average values of γ i for each of the datasets in Table 7.
We observe that strongly consistent datasets have lower values of γ i indicating that PROFH I T’s refinement module automatically learns to strongly leverage the hierarchy for these datasets compared to weakly coherent datasets.
[Table]
0.217 M IN T 0.
0.213 ERM 0.
0.209 H IER E2E 0.
0.109 SHARQ 0.
0.048 PROFH I T (Ours) 0.
0.008 Ablation
P-F INE T UNE 0.
0.004 P-G LOBAL 0.
0.119 P-D EEP AR 0.
0.065 P-N O C OHERENT 0.