Whittle Networks: A Deep Likelihood Model for Time Series
Abstract
While probabilistic circuits have been extensively explored for tabular data, less attention has been paid to time series.
Here, the goal is to estimate joint densities among the entire time series and, in turn, determining, for instance, conditional independence relations between them.
To this end, we propose the ﬁrst probabilistic circuits (PCs) approach for modeling the joint distribution of multivariate time series, called Whittle sum-product networks (WSPNs).
WSPNs leverage the Whittle approximation, casting the likelihood in the frequency domain, and place a complex-valued sum-product network, the most prominent PC, over the frequencies.
The conditional independence relations among the time series can then be determined efﬁciently in the spectral domain.
Moreover, WSPNs can naturally be placed into the deep neural learning stack for time series, resulting in Whittle Networks, opening the likelihood toolbox for training deep neural models and inspecting their behaviour.
Our experiments show that Whittle Networks can indeed capture complex dependencies between time series and provide a useful measure of uncertainty for neural networks.
1. Introduction
Probabilistic graphical models specify joint densities compactly using conditional independencies between random variables (RVs).
When faced with time series, dynamic
Bayesian networks are commonly employed.
In many applications, however, one instead often aims to infer the conditional independence relations between time series themselves, accounting for interactions at all possible lags, leading to time series graphical models (TGMs) (Tank et al.,
Consider, e.g., the stock price changes of industrial sectors shown in Fig.
1 (Left).
The drop in lateillustrates a critical need for new and fundamental understandings of the structure and dynamics of economic networks (Schweitzer et al., 2009).
Arguably, Dahlhaus (2000) introduced the ﬁrst (undirected) graphical model for stationary time series.
Speciﬁcally, for jointly Gaussian stationary time series, one transforms the series to the frequency domain and estimates a Gaussian graphical model in the resulting spectral representation.
The conditional independencies between time series are encoded by zeros in the inverse spectral density matrix.
Bach & Jordan (2004) leveraged the Whittle approximation (Whittle, 1953), casting the likelihood in the frequency domain, and
Tank et al.
(2015) proposed a Bayesian extension, making use of hyper complex inverse Wishart distribution priors.
Unfortunately, using graphical models for time series modeling has a number of important limitations.
First, inference is exponential in the worst case.
Second, the sample size required for accurate learning is worst-case exponential in scope size, i.e. the subset of variables of each potential.
Third, learning requires inference as a subroutine, i.e. it can take exponential time even with ﬁxed scopes.
To overcome these limitations of TGMs and inspired by the successes of deep probabilistic models for univariate time series (Trapp et al., 2020; Melibari et al., 2016), we introduce the ﬁrst probabilistic circuit for modeling the joint distribution of multivariate time series, called Whittle sumproduct network (WSPN).
It also leverages the Whittle approximation but places a sum-product network (SPN) (Poon & Domingos, 2011) over the frequencies.
Using SPNs in the frequency domain, however, requires different decomposition and conditioning operations for SPNs tailored towards complex-valued RVs—our main technical contributions.
The conditional independence relations among the time series, even in a directed fashion, can then be determined efﬁciently in the spectral domain, see Fig.
1 (Middle).
While WSPNs feature efﬁcient inference, learning their structure—the structure of complex-valued SPNs—can still be tedious and may not scale well to large number of RVs.
Therefore, we propose to “go down the deep neural road” one step further by generating unspecialized random structures for the SPNs (Peharz et al., 2020b; Ventola et al.,
[Figure]
2020; Stelzner et al., 2019; Kossen et al., 2020), scalable to millions of parameters and trainable in an end-to-end fashion, even together with deep neural networks (NNs) for time series.
This results in Whittle Networks, and it opens the likelihood toolbox for training deep neural models for time series and for inspecting their behaviour based on probabilistic grounds.
Our experimental results on stock market data, synthetic time series, MNIST, and hyperspectral images demonstrate that Whittle Networks can indeed capture complex dependencies between time series and provide a useful measure of uncertainty for neural networks.
To summarize, we make the following contributions:
• The ﬁrst probabilistic circuit for modeling the joint distribution of multivariate time series, called Whittle sum-product networks (WSPNs), by introducing complex-valued SPNs.
• Using WSPNs, we propose deep likelihood functions for training deep neural networks for time series in an end-to-end fashion, called Whittle Networks.
We illustrate this by introducing Whittle Autoencoder , a novel probabilistic autoencoder for time series.
To this end, we proceed as follows.
We start off by introducing WSPNs, reviewing more related work on the ﬂy.
Then, we show how to read off conditional independencies from them and how to interface them with deep neural networks, resulting in Whittle Networks.
Before concluding, we present our empirical evaluation.
2. Whittle SPNs: SPNs for Time Series
Neural networks have been widely used for time series processing.
For instance, multilayer perceptron (MLP) can work as an autoencoder (AE) for univariate time series modeling (Koskela et al., 1996).
Similarly, Convolutional Neural
Networks (CNN) have also been used for time series modeling (LeCun & Bengio, 1995).
Moreover, Recurrent Neural
Networks (Connor et al., 1994) and in particular Long ShortTerm Memory (LSTM)
Neural Networks (Gers et al., 2000) have been extensively used for neural modeling of time series.
However, the above neural models can not provide a natural probability measure of the outputs.
On the other hand, there are deep generative models that use well-deﬁned likelihood functions.
State space model and deep neural networks are combined in Rangapuram et al.
Similarly, state space model with conditional probability is investigated also for reinforcement learning in Buesing et al.
Kalchbrenner et al.
(2017) encode time, space, and color structure into a dependency chain to model video sequences.
However, these works can answer to a considerably limited set of queries because they are restricted only to forecasting and do not model the joint distribution of the complete time series.
A generative approach has been proposed in Krishnan et al.
but it offers rather limited support to exact inference.
In contrast, WSPNs lend themselves naturally to efﬁcient inference and learning as well as end-to-end training together with deep neural networks.
They also have clear probabilistic semantics.
In fact, WSPNs can be seen as generalized directed acyclic graphs (DAGs) of mixture models in the spectral domain, with sum nodes corresponding to mixtures over subsets of variables and product nodes corresponding to features or mixture components.
Speciﬁcally, they consist of the following two ingredients: Whittle Likelihood and complex-valued SPNs.
Ingredient 1: Whittle Likelihood.
The Whittle likelihood models the multivariate time series in the spectral domain.
Let X =
[ x (1) , . . .
, x ( T )] , with x ( t ) ∈ R p , be a realization of a p -dimensional
( p -D) time series with length T .
For t ∈ Z , x ( t ) is Gaussian stationary if:
[Equation]
X 1: N = { X 1 , . . .
, X N } are N independent realizations of the time series.
In spectral domain of each sequence, d n,k ∈ C p denotes the discrete Fourier coefﬁcient of the n th sequence at frequency λ k = 2 πk/T, k
[Equation]
[Equation] [Equation]
Based on the Whittle approximation assumption (Whittle, 1953), the Fourier coefﬁcients are independent complex normal RVs with mean zero:
[Equation]
where S k ∈ C
p × p is the spectral density matrix .
For a stationary time series, its spectral density matrix is deﬁned as:
[Equation]
The Whittle likelihood of the N realizations is deﬁned as:
[Equation]
The Whittle approximation holds asymptotically with large
T and it has been used in the Bayesian context.
To overcome the limitations in Whittle likelihood inference with TGMs, Whittle Networks place an SPN (Poon & Domingos, 2011)—a tractable and expressive density estimator—across the frequencies.
Ingredient 2: Complex-Valued SPNs (CoSPNs).
A complex random variable d ∈ C p that follows complex normal distribution has its real and imaginary parts being jointly normal distributed.
Based on the fact that the real and imaginary parts are coupled, it is a more sensible idea to model the real and imaginary parts as a pair of RVs in an SPN, which results in Complex-valued SPNs (CoSPNs).
More formally, a CoSPN S over a set D of p complex-valued RVs is a probabilistic model deﬁned via a DAG containing three types of nodes: input distributions (the leaves ), sums and products .
All leaves of the CoSPN are density functions over some subset Q ⊆ D of pairs of real-valued RVs.
Inner nodes are either weighted sums or products, denoted by S and P , respectively, i.e. S = (cid:80) N ∈ ch ( S ) ω S , N N and P = (cid:81) N ∈ ch ( P ) N , where ch ( · ) denotes the children of a node.
The sum weights ω S , N are assumed to be non-negative and normalized: ω S , N ≥ 0 , (cid:80) N ω S , N = 1 .
CoSPNs make use of pairwise Gaussian leaf nodes for modeling the pair of real and imaginary parts of the complex RVs, assuming that the real and imaginary parts from one complex RV are correlated.
The pairwise Gaussian leaf node is modeled using a vector of means µ d k ∈ R 2 and a covariance matrix Σ d k ∈ R 2 × 2 , as illustrated in Fig.
With the pairwise Gaussian leaf density of the complex RVs, CoSPN essentially encodes the joint density p ([ d ( R ) 1 , d ( I ) 1 ] , . . .
, [ d ( R ) p , d ( I ) p ]) .
In analogy to
SPNs, the scope of an input distribution N in a CoSPN is deﬁned as the set of RVs Y for which N is a distribution function, i.e. sc ( N ) :
= Y .
The scope of sum or product node N is recursively deﬁned as sc ( N ) =
(cid:83) N (cid:48)
∈ ch ( N ) sc
( N (cid:48) ) .
To represent a valid probability density, CoSPNs should satisfy two structural constraints (Poon & Domingos, 2011), namely completeness and decomposability.
Speciﬁcally, a
CoSPN is complete if for each sum S it holds that sc ( N (cid:48) )
=
sc ( N (cid:48)(cid:48) ) , for all N (cid:48) , N (cid:48)(cid:48) ∈ ch ( S ) .
A CoSPN is decomposable if it holds for each product P that sc ( N (cid:48) )
∩ sc ( N (cid:48)(cid:48) )
= ∅ , for all N (cid:48)
(cid:54) =
N (cid:48)(cid:48) ∈ ch ( P ) .
In that way, all nodes in a CoSPN recursively deﬁne a valid complex-valued distribution over their respective scopes: the leaves are complex-valued distributions by deﬁnition, sum nodes are mixtures of their children, and products are factorized, complex-valued distributions, assuming context-speciﬁc independence among the scopes of their children.
CoSPNs feature tractable probabilistic inference.
For example, CoSPNs allow one to compute arbitrary marginal densities:
In particular, let S ( x ) be a density over X represented by CoSPN S , and let ¯ X
=
{ X i 1 , . . .
, X i M } be a set of RVs to be marginalized.
The marginal density over Z =
X \ ¯ X can be computed as
[Equation]
The integrals can be iteratively swapped with sums and distributed over products in the CoSPN (Peharz et al., 2015).
Similar to Gens & Domingos (2013), learning a CoSPN can be done by clustering over instances, treating them as 2 p -D vectors, to learn sum nodes, and by non-parametric independence test over both real and imaginary components to learn a product node.
For the pairwise Gaussian leaf case, the non-parametric independence test needs to be adapted, such that two complex RVs are dependent if any of the four combinations of the real and imaginary components show a sort of dependency.
Random-and-Tensorized SPNs (RATSPNs) (Peharz et al., 2020b), instead, generate a random tree structure and then optimize the weights in a “classical deep learning manner”, which enables us to adapt CoSPNs together with other deep architectures.
WSPNs =
Whittle Likelihood + CoSPNs.
With the Whittle Likelihood and CoSPNs at hand, we are now ready to introduce WSPNs: CoSPNs are used to build the joint complex normal distribution of the Fourier coefﬁcients of the time series.
Note that the Fourier coefﬁcients from DFT for real-valued sequences are Hermitian-symmetric.
Therefore the negative frequency coefﬁcients are redundant, and in total only T W = (cid:98) T/ 2 (cid:99) + 1 Fourier coefﬁcients need to be modeled.
We know from the Whittle assumption that the independence of Fourier coefﬁcients of different frequencies holds for stationary time series.
In general, we assume that without the independent Fourier coefﬁcients assumption, the joint distribution still models the time series in the Fourier domain, even with more ﬂexibility in modeling both stationary and non-stationary time series.
Therefore, in contrast to
TGM limitations, two more constraints can be relaxed when modeling general time series:
• The mean of each frequency in (4) need not be 0 .
• The Fourier coefﬁcients of different frequencies need not be independent as in (6).
With the above two relaxations, the Fourier coefﬁcients of both stationary and non-stationary general time series can be modeled with WSPNs.
The Whittle likelihood of the ﬁrst
T W Fourier coefﬁcients from a general time series can be modeled with one WSPN: p ( X 1: N | Co )
≈
(cid:89) N n =1 p ( d n, 0 , . . .
, d n,T W
[Equation]
where Co denotes the structure and parameters of the
CoSPN trained from data, d n,k ∈ C p denotes the k th
Fourier coefﬁcients from the n th sample from data, and d n ∈ C p × T W denotes the ﬁrst T W Fourier coefﬁcients of one multivariate time series.
3. Opening the Blackbox of Whittle SPN
While WSPNs are tractable, they are just computational graphs and, hence, blackboxes w.r.t.
the (conditional) independence relations.
For p -D multivariate time series, there is great interest in ﬁnding the structure that represents the conditional independence among the p components in the form of graphs.
We now show how to extract it from WSPNs.
The vanilla search-based method for structure learning of graphical models usually consists of a structure learning algorithm, e.g. Hill climbing (Herskovits, 1991; G´amez et al., 2011), and a score, e.g. MDL (Rissanen, 1983).
For
DAGs, if the directions of edges are predeﬁned, the graph can be learned efﬁciently via a 3-phase algorithm presented in Cheng et al.
More generally, one can employ an order swapping algorithm (Teyssier & Koller, 2005) without providing the order of the nodes.
Moving to Whittle likelihood, it was indeed proposed for
Bayesian structure learning in Tank et al.
First, T spectral density matrices S 0: T − 1 are computed from time series X 1: N .
As the Whittle likelihood p ( X 1: N | G, S 0: T − 1 ) can be factorized given graph G , it can be maximized by searching over graph structures.
Feature-inclusion stochastic search (FINCS)
(Scott & Carvalho, 2008) is then applied to search for G that maximizes the above likelihood.
With WSPN, we can proceed as follows.
Denote the Fourier coefﬁcients from a subset s of the p components of the time series as d { s } n , and p ( d { s } n ) the marginal density.
Let
G =
( V, E ) be a decomposable graph with vertex set V
= { v 1 , . .
. , v p } and edge set E .
For DAGs, given each node v i and its parents Pa G ( v i ) , and the corresponding CoSPN Co learned from time series X 1: N , (7) can be factorized based on the graph structure by chain rule: p ( X 1: N | G, Co ) ≈
N
[Equation]
The factorization for undirected graph is similar, over cliques and separators (Tank et al., 2015).
Thus, given time series X 1: N , we ﬁrst learn a WSPN that models time series in the spectral domain.
Then, we start from an empty graph G and add edges iteratively.
To add an edge, we create a list of all possible edge candidates to form a graph, and compute the list of Whittle likelihoods from (8), given the obtained WSPN and each possible new graph.
The edge that mostly increases the Whittle likelihood is added to G .
We stop when adding an edge decreases the Whittle likelihood or when there are no more edge candidates.
Bayesian information criterion (BIC) (Schwarz, 1978) can be naturally applied to handle the complexity of the graph.
4. Whittle Networks: Putting WSPNs onto the Deep Learning Stack
Like other PCs, WSPNs can be vectorized and used within
GPU-supporting implementations (Peharz et al., 2020b;
Trapp et al., 2019; Peharz et al., 2020a).
Note that the discrete (fast) Fourier transformation (DFT) can naturally be vectorized, too.
Thus, WSPNs are differentiable and can be trained end-to-end together with NNs, resulting in
Whittle Networks.
Therefore, we introduce Whittle AE, an instantiation of
Whittle Networks, shown in Fig.
It combines a standard
AE with a proper (deep) likelihood function for time series using WSPNs.
To train the AE whose output distribution becomes similar to its input distribution, we employ two
WSPNs alongside the AE.
Each WSPN models the spectral domain of AE input and output, respectively.
The distributions of the AE input and output in the spectral domain will be pushed closer by minimizing the Kullback-Leibler (KL) divergence between the distributions modeled by the two
WSPNs.
The combined loss consists of the reconstruction error, two negative Whittle likelihoods of both WSPNs, and the KL divergence between the WSPNs.
In this way, we get a simple but effective way to equip AEs with densities and probabilistic inference.
This can be used to assess how likely the AE reconstruction originates from the input distribution.
Generally, we note that Whittle Networks can be combined with any other deep neural architecture, providing meaningful probabilities for time series.
5. Experimental Evaluation
Our intention here is to investigate the beneﬁts of modeling time series with the proposed Whittle Networks.
In particular, we investigated the following questions: (Q1)
Do WSPNs capture densities over time series better than baselines?
(Q2) Can WSPNs discover the conditional independence of time series?
(Q3)
Can Whittle Networks provide meaningful probabilities for deep neural networks?
Experimental Protocol and Datasets.
Stationary time series is of great interest since the Whittle approximation which inspired us is based on the stationary assumption.
Therefore, we use two real-world market datasets acquired from “Yahoo!
Finance Data”.
The ﬁrst one is the index values of 11 sectors from “Standard & Poor’s” ( S&P ) from
October 16, 2013 to May 24, 2019 (See Fig.
1 (Left)).
The second one is the global stock index ( Stock ) from 17 markets extracted from June 2, 1997 to June 30, 1999.
Both S&P and Stock datasets are applied ﬁrst with log-return transformation, assuming them to be stationary (St˘aric˘a & Granger, 2005), and then a sliding window of size 32, ending up inand 50 time series instances.
Simulation data from Vector
Autoregressive process ( VAR ) (Sims, 1980) is also used to discover the conditional independencies.
Details of the three stationary datasets can be found in Appendix A.
With the aim of showing that Whittle Networks can deal with not only stationary time series but also with nonstationary processes, we employ the following synthetic and real-world datasets for non-stationary time series.
The synthetic Sine data consists of 3 trigonometric sines with same frequency while different phases with Gaussian noise, 2 sine series with another frequency with different phases with Gaussian noise, and one series of pure Gaussian noise.
We shufﬂe the order of time series components as out-ofdomain (ood) samples.
The synthetic Billiards data contains simulations of 6 trajectories from the horizontal and vertical locations of 3 balls, unnatural trajectories form the ood set.
The synthetic Mackey-Glass series (Gers et al., 2002) consists of two channels and is used for forecasting test.
Although MNIST (LeCun et al., 1998) is not a typical univariate time series dataset, it is widely used in prominent time series processing works (Van Oord et al., 2016; Esteban et al., 2017; Le et al., 2015).
In this work, we use MNIST also to examine the ability of Whittle Networks in modeling time series with high dimensionality.
Finally, we used hyperspectral images with 328 wavelengths of plants for a qualitative analysis of anomaly detection.
Each vector from one pixel with length 328 can be viewed as a single univariate time series and we aim for detecting the unhealthy areas on the leaf.
Fig.
5 (Top) shows RGB views of healthy and unhealthy leaves with dark spots as infected areas.
Details of the non-stationary datasets are described in Appendix B.
(Q1)
Modeling Time Series with WSPNs.
We trained
SPNs with Gaussian leaves employing LearnSPN (Gens & Domingos, 2013), ResSPNs (Ventola et al., 2020), and
WSPNs on all datasets.
For LearnSPN and ResSPN, no pairwise constraint is applied when estimating independencies, i.e., the real and imaginary parts from one complex RV are free to split.
“-Pair” uses diagonal covariance matrices to model the real and imaginary parts independently, while “- 2d” takes full covariance matrices to jointly model the pairs as bidimensional Gaussians.
LearnSPN, WSPN-Pair, and
WSPN-2d encompass and do structure learning, while the structure of ResSPN, ResWSPN-Pair, and ResWSPN-2d are randomly generated.
Furthermore, in the spirit of having a neural likelihood gold standard, we make use of Masked
Autoencoder for Distribution Estimation (MADE) (Germain et al., 2015) with Gaussian conditionals, as implemented in Papamakarios et al.
(2017), to estimate the joint probability density of the Fourier coefﬁcients.
The number of hidden layers in MADE is set to 1 for all datasets, while the hidden units vary from 200 to 600, depending on the number of RVs in each dataset.
Note that this comparison is
[Table]
[Figure]
not fair since MADEs are highly specialized for likelihood estimation and they cannot be easily adopted for computing general inference and efﬁcient sampling due to the architectural constraints necessary for holding the autoregressive property.
As we can see from Tab. 1, WSPNs produce generally higher likelihood for training and test sets, compared to LearnSPN or ResSPN.
Moreover, modeling the complex RV with a full covariance matrix in the leaf node (“-2d”) provides higher likelihood for training/test set, and lower likelihood for ood set, compared with modeling them independently (“-
Pair”).
The likelihood of ood data from WSPN is also lower, except for MNIST data.
The reason might be the image similarity given that all digits are centered.
The differences of likelihoods are statistically signiﬁcant according to a
Wilcoxon signed-rank test with p = 0 .
05 , “-Pair” and “- 2d” perform equally better than ResSPN on MNIST and than LearnSPN on Billiards .
Additionally, WSPNs achieve high likelihoods and it is competitive with our gold standard
MADE.
On MNIST, MADE provides higher likelihoods than WSPNs mainly because: 1) MADE is a strong and powerful density estimator. 2)
MNIST is not a proper time series, since it has some “all zeros” rows as components in a multivariate series.
Fig.
4 shows trajectories of the 3 balls from Billiards .
An example of real movement is illustrated in Fig.
4 (Left).
[Figure]
Fig.
4 (Middle) shows trajectories after the inverse Fourier transform of the sampled Fourier coefﬁcients from the
WSPN trained on Billiards .
The behaviours of straight-linemoving and rebound of the balls are successfully captured by the WSPN model.
The movement of the green ball looks very realistic: it ﬁrst moves towards the upper right corner, hits the wall, and then goes down to the left.
The sample from Billiards shows the strong power of modeling the entire multivariate time series with WSPNs, without losing the characteristics of the original time series.
To provide a comparison, ood trajectories are visualized in Fig.
4 (Right).
To illustrate this qualitatively on plants , Fig.
5 shows heatmaps of Whittle likelihood, as well as the reconstruction error maps got from SAMMIE (Kerner et al., 2019).
As one can see, WSPN models the healthy pixels well and is able to give lower likelihood to pixels from infected or non-leaf areas, while SAMMIE fails to discover some infected spots.
Conditional SPNs (CSPNs) (Shao et al., 2020) can be applied to model the conditional distribution of p ( X t | X t − 1 ) in the time domain.
By employing CSPNs as compo-
[Figure]
nent, WSPNs can also model the conditional Whittle likelihood p ( d t | d t − 1 ) to perform forecasting straightforwardly.
Given data from the past, one can compute predictions by means of most probable explanation (MPE) queries.
We compared WSPN, CSPN, and LSTM as strong prediction baseline on Mackey-Glass .
Details of the experimental settings are described in Appendix C. CSPN achieved a mean squared error (MSE) of 0 . 0119 , WSPN 0 . 0022 , and LSTM 0 .
0014 but with slower convergence.
Qualitative results are shown in Fig.
6, one can see that WSPN performs better than CSPN and it has a competitive MSE with LSTM.
Even if the core objective is to model the joint distribution of the whole time series rather than computing only predictions,
WSPNs can also compute accurate forecasting.
Thus, (Q1) can be answered afﬁrmatively, WSPNs capture densities better than baselines. Q
Conditional Independencies among Time Series.
The conditional independencies of the VAR series are determined when it is created.
We extract both directed and undirected graphs from WSPN trained on VAR series to visualize its conditional independencies.
Fig.
10 shows that
WSPN successfully extracted the true conditional independencies of a 7-D VAR series.
Applying BIC helped to ﬁlter out the edges that increase the Whittle likelihood with a negligible increment.
Regarding real-world time series, the conditional independencies derived from WSPNs learned on S&P and Stock are shown in Fig.
1 and Fig. 8 respectively.
The thickness of an [Figure] edge represents the increase of Whittle likelihood obtained by adding that particular edge.
We set the maximum number of predecessors and successors to 2 .
For the comparison, we adopted hill climbing for learning (non-Bayesian) TGMs by using the OpenMarkov toolbox (Arias et al., 2019).
Regarding S&P data, the directed independence structure (DAG) shown in Fig.
1 (Middle) unveils interesting correlations among 11 sectors.
WSPNs discover the correlations between, e.g., Information Technology and Financials, Industrials and Materials, Financials and Materials.
Looking at the original time series in Fig.
1 (Left), one can observe that the indices of Materials and Financials seem to be quite synchronized, thus, the correlation between these two sectors makes sense.
At the same time, the (non-Bayesian)
TGM failed to discover some correlations, e.g., between
Industrials, Energy, and Utilities, cf.
Fig.
1 (Right).
WSPNs may also help to discover the conditional independencies (DAG) of the global stock market.
From
Fig.
8 (Left), it is clear that the central and western European countries (AT, UK, IT, FR, GE, CH, SP) are well clustered.
Moreover, Germany, France, and the UK are among the core components of the European cluster.
Most parts of the graph follow the geographic relationship, except for CA and BE.
One explanation could be that WSPN discovers the hidden correlations of stock indices between those Frenchspeaking countries.
In contrast, (Bayesian) TGM (Right) correctly ﬁnds the correlations among Asia-Paciﬁc markets (US, CA, HK, JP, AU), which are not discovered by WSPN.
However, the graph structure of the European countries is less informative compared to the WSPN one.
Both models fail to discover the correlation of JP and HK with others.
This is indeed more difﬁcult to discover because both are geographically far from the other markets.
Since the synthetic Sine dataset is relatively simple, both
WSPN and non-Bayesian TGM produce the same DAG: two edges connecting the 3 components with the same frequency, another edge between the two components with another frequency, and the Gaussian noise being independent.
The graph is relatively simple, thus, it is shown in Appendix D.
Overall, Tab. 2 summarizes the likelihoods of the condi-
[Table] [Table]
tional independence structures discovered via WSPNs.
Both
WSPNs and non-Bayesian TGMs achieve the same performance on Sine , as the discovered structures are actually the same.
On S&P and Stock , however, WSPNs achieve better performance, i.e., the conditional independencies they discovered ﬁt data better.
From the results on various datasets,
WSPN shows its ability to discover the conditional independencies from both stationary and non-stationary time series.
In general, the results clearly provide an afﬁrmative answer to (Q2) : WSPNs are able to discover the conditional independencies of time series, even better than baselines.
(Q3)
Whittle Networks: Deep Likelihoods for (Deep)
Neural Networks.
Unfortunately, the probability distributions of deep neural networks may generally not be wellcalibrated (Guo et al., 2017).
In order to investigate the ability of WSPNs to provide meaningful and calibrated probabilities for deep neural networks, we considered the
Whittle AE outlined before and shown in Fig.
It consists of two WSPNs and one AE in between.
The AE consists of an MLP with the following number of neurons for each layer: 128 − 64 − 16 − 2 − 16 − 64 − 128 , using sigmoid as the activation function.
We vectorized the WSPNs as described above using RAT-SPN (Peharz et al., 2020b).
This way, they can easily be integrated and optimized end-to-end together with the AE.
The RAT-SPN hyperparameters, see
Peharz et al.
(2020b) for details, can be found in Appendix
E. We trained the Whittle AE on MNIST by taking each image row as one univariate time series, in other words, each image forms one multivariate series.
The average log-likelihood of our selected training and test data for the input WSPN and output WSPN, see Fig.
3, are summarized in Tab. 3.
As one can clearly see, the input
WSPN provides high likelihood for training and test data
T e s t
O u tli e r O u tli e r
(digits “0-4”), while providing low likelihood for outlier (digits “5-9”).
It is even lower for outlier2 (images from
Fashion-MNIST (Xiao et al., 2017)), a very different domain.
Surprisingly, the WSPN that models the output of the AE also produces relatively lower likelihood for outputs (reconstructions) when using “outliers” as input.
Generally speaking, unlikely outputs of deep neural networks, such as ood samples, can be detected by looking at the rather low likelihood provided by the WSPNs.
Fig.depicts various inputs and their corresponding outputs from the Whittle AE.
As one can see, it is difﬁcult to tell from the reconstructed images if the input is normal or rather an ood sample.
However, the log-likelihoods of input and reconstructed output images clearly indicate that both ood sets have lower likelihood.
For instance, the autoencoder wrongly reconstructs a pullover (ﬁrst image of outlier2) as a digit “3” (second image of outlier2).
Thus, both input and output WSPNs are able to recognize the ood and its reconstruction by assigning a low likelihood—especially on the input image, even when the reconstructed image looks like an authentic digit “3”.
Although it is almost impossible to judge from the output image “3”, both likelihoods from input respectively output images illustrate that the input may be an ood sample and that the corresponding output is not trustworthy.
More results from Whittle AE can be found in Appendix F. Overall, these results clearly provide an afﬁrmative answer to (Q3) :
Whittle Networks can provide meaningful probabilities for deep neural networks.
6. Conclusion
We introduced the ﬁrst complex-valued SPN, called CoSPN, tailored for complex-valued normal distribution.
In particular, using CoSPNs and the Whittle likelihood approximation, we proposed the ﬁrst probabilistic circuits for multivariate time series, modeling temporal information as well as the conditional independence of multivariate time series implicitly in the spectral domain.
Being able to compute the likelihood of time series in a tractable fashion is critical in developing practical and scalable likelihood functions for deep neural networks of time series.
Our experimental results demonstrated that the resulting Whittle Networks can indeed provide meaningful probabilistic measures for deep neural networks.
Providing a natural measure of uncertainty makes deep neural networks easier to interpret and to use for non-AI-experts, in particular when it comes to decisionmaking.
Exploring this for other deep architectures is the most interesting avenue for future work, but one should also investigate more advanced structure learning methods for complex-valued SPNs in general.
Regarding complex values, one could bring the idea of CoSPNs to other models like normalizing ﬂows or wavelet networks.
Furthermore, based on the introduced conditional WSPN for forecasting, we envision the exploration of dynamic WSPN or Whittle
RNN as interesting future directions.
Acknowledgements
The authors thank the anonymous reviewers for their valuable feedback.
This work was supported by the Federal Ministry of Education and Research (BMBF; project “MADESI”, FKZ 01IS18043B, and Competence Center for
AI and Labour; “kompAKI”, FKZ 02L19C150), the German Science Foundation (DFG, German Research Foundation; GRK 1994/1 “AIPHES”), the Hessian Ministry of
Higher Education, Research, Science and the Arts (HMWK; projects “The Third Wave of AI” and “The Adaptive Mind”), and the Hessian research priority programme LOEWE within the project “WhiteBox”.
A. Stationary Time Series Datasets
Two datasets from ﬁnancial market index are employed in order to show the capability of modeling stationary time series with WSPNs.
The ﬁrst stationary time series dataset is formed by the index values of 11 sectors from “Standard & Poor’s” ( S&P ) from October 16, 2013 to May 24, 2019.
The second is global stock index ( Stock ) from 17 markets extracted from June 2, 1997 to June 30, 1999.
Before modeling the joint distribution in the spectral domain, the real-world market data is ﬁrst converted to its log-return: r t = 100 log( x ( t ) /x
( t − 1)) .
Both S&P and Stock datasets are transformed with a sliding window of size 32.
The S&P time series has lengthafter the log-return transformation, and thus 44 time series are extracted by sliding window without overlap.
The Stock series has length 522 after the log-return transformation.
The sliding window is enabled with a step size of 10 in order to have more time series for training.
In the end,time series instances with length 32 are extracted.
Tab.lists all the names of the Stock index and the corresponding markets.
More details of the Stock dataset can be found in Tank et al.
The VAR series is simulated from an order- 1 vector autoregressive process, with p = 7 dimensions.
Time series is simulated from the model:
[Equation]
where x ( t ) ∈
R p , A ∈ R p × p and (cid:15) ∼ N (0 , I p × p ) .
Following Tank et al.
(2015) and Songsiri & Vandenberghe (2010), we ﬁrst restrict A to be upper triangular, and set the diagonal elements to a constant A ii = 0 .
Then, the upper diagonal elements A ij are sampled from a Binomial distribution with p = 0 .
The corresponding inverse spectral density matrix of the process is:
[Equation]
The conditional independencies between time series are encoded by zeros in the inverse spectral density matrix S ( λ ) − 1 .
The matrix A is accepted when 1) the absolute values of all eigenvalues of A are less than one, making the series station- ary, and 2) the graph G determined by A is decomposable.
We generate the 7 -D series from the following matrix A =   0 .
0 .
 .
An example of the inverse spectral density matrix at frequency λ = π/ 10 is S ( λ = π/ 10) − 1 ≈   2 .¯
k     k 3 .
¯ k     2 .
¯ k   k 3 . ¯ k   k 3 .
k 3 .¯
k     k 3 .
 , (13) where k
= 1 . 5 + 0 .
3 i and ¯ k being its conjugate.
This inverse spectral density matrix implies the conditional independencies shown in Fig.
10, which is also shown in Fig.
7 (Left) in Section 5 in the paper.
From the above matrix A , a 7 -D series with length 557056 is generated.
With a sliding window of size 32 , 17408 series instances are extracted, where 16384 form the training set and 1024 the test set.
B. General Time Series Datasets
In this paper, we investigated ﬁve non-stationary time series datasets.
The synthetic Sine data consists of 6 components: 3 trigonometric sines with same frequency while different
[Table]
phases (Sine11, Sine12, and Sine13) plus Gaussian noise; 2 sine series with another frequency with different phases (Sine21 and Sine22) plus Gaussian noise; and one series of pure Gaussian noise (Gauss1).
The training and test sets have the 6 components in the order of: “Sine11, Sine12,
Sine13, Sine21, Sine22, Gauss1”, while the ood set has the following order: “Sine21, Sine22, Gauss1, Sine11, Sine12,
Sine13”.
Each time series instance has length 32 withcomponents.
In total, 16384 samples are generated for training , 1024 for test and 1024 as ood samples.
The synthetic Billiards data contains simulations of trajectories of three balls.
The balls perform elastic collisions with each other or against the walls of the environment.
The horizontal and vertical locations of the 3 balls form a 6 -dimensional state vector at each time step.
The ood data is generated by keeping the movement of one direction and replacing the movement of the other direction with
Gaussian noise.
Additionally, in the ood set the balls pass through each other instead of colliding.
Therefore, the above behaviour makes the trajectory of ood set unrealistic and unnatural.
Each trajectory has the locations of 100 time steps, which forms a multivariate time series with 6 components and a length of 100.
We generate 9700 samples for training , 300 for test and 300 as ood .
The synthetic Mackey-Glass series is simulated from:
[Equation]
where γ = 0 .
1 , β = 0 .
2 , n = 10 .
We simulate two series independently, one with a delay of τ = 17 , and another with a delay of τ = 17 / 3 , to form a 2 -D multivariate time series.
In total, 3000 series instances with length 1024 are generated, with the ﬁrst 544 steps for training and laststeps for test.
Regarding MNIST, the training set consists of all the original training samples with labels “0-4”.
The test set consists of original test samples with labels “0-4” and our ood set consists of original test samples with labels “5-9”, namely outlier1 .
The test set from Fashion-MNIST is also used as another outlier set for the Whittle Networks experiment.
In order to have a similar number of samples in the second outlier set, images labeled “Sandal, Shirt, Sneaker, Bag, and Ankle boot” are used to form outlier2 .
The images are down-sampled to 14 × 14 , with each row as one component of a 14-dimensional multivariate time series.
Finally, we used hyperspectral images of plants for a qualitative analysis of anomaly detection.
The images were taken from leaves of sugar beet either healthy or inoculated with a disease named Cercospora beticola , with 328 wavelengths from 380nm to 1010nm.
Each 328 -D vector from one pixel can be viewed as a single univariate time series.
There areclasses of pixels in one image, healthy, inoculated, and background.
Tab. 5 summarizes the statistics of the introduced datasets.
C. Forecasting of Mackey-Glass Dataset
Forecasting is performed window by window.
That is, given a window of series X t − 1 , we try to predict the value of the next window X t at once.
Conditional distribution of the two windows of series is modeled in order to do forecasting.
[Table] [Table]
[Table]
With a sliding window of size 64 with step size 32 ,windows are extracted from each training instance with length 544 .
In order to model the conditional distribution of either p ( X t | X t − 1 ) in the time domain, or p ( d t | d t − 1 ) in the Fourier domain, the ﬁrst 32 steps in one window form
X t − 1 , and the last 32 steps form X t .
The hyperparameters of both CSPN and conditional WSPN are: C = 1 , as we want to model the conditional distribution of data without class labels, and depth D = 2 , number of splittings R = 8 , number of sum nodes in regions S = 8 , input distributions per leaf region I = 4 .
Both models are trained with Adam optimizer for 10 epochs, with a learning rate of 0 .
001 and a batch size of 64 .
Details of the CSPN settings and hyperparameters can be found in Shao et al.
In the test phase with conditional distributions, the true value of X t − 1 is given, and the MPE of either p ( X t | X t − 1 ) or p ( d t | d t − 1 ) is estimated as the prediction.
Note that in the basic LSTM experiment, the initial hidden state is set to a 0 vector at the beginning of the test.
This is the reason why the prediction of LSTM from the 32 nd step (start of prediction) is far from the true value.
Regarding LSTM architecture, it is composed by stacking one LSTM recurrent layer having 32 hidden units and one linear layer that transforms the 32-dimensional vectors in 2-dimensional vectors.
We trained the network forepochs with Adam on min-max scaled data.
We employed a learning rate of 0 .
05 , batch size of 512 samples, and MSE as loss function.
The model has been implemented in PyTorch 1.7.1 using the default values for the other hyperparameters.
D. Independence Structure of Sine Dataset
In order to explore the ability of discovering conditional independencies of non-stationary time series, we apply
WSPNs to the Sine dataset and extract independence structures (DAGs) from it.
The resulting structures are shown in
Fig.
One can clearly see that the three sine components (“Sine11, Sine12 and Sine13”) that have the same frequency are highly correlated.
The other two components (“Sine and Sine22”) are also highly correlated while the Gaussian
[Figure]
noise component is independent of the others.
As a comparison, non-Bayesian TGM produces the same directed graph structure since the synthetic sine dataset is relatively simple.
E. RAT-SPN Hyperparameters
The RAT-SPN hyperparameters, see Peharz et al.
(2020b) for details, were set as follows:
C = 1 , as we want to model the joint distribution of data without class labels, and depth
D = 7 , number of splittings R = 2 , number of sum nodes in regions S = 2 , input distributions per leaf region I = 2 .
We use the Adam optimizer with a learning rate of 0 .on plants dataset, and a learning rate of 0 . 004 on MNIST dataset.
F. Whittle Network Results
Additional results from Whittle Network, instantiated as
Whittle AE, on MNIST are shown in Fig.
The results support our claim that Whittle Networks indeed provide meaningful probabilities to neural networks.
Both outlier (images of digits “5-9”) and outlier2 (images from FashionMNIST) have a lower average likelihood compared to the average likelihood of our MNIST test data which is composed of in-domain samples, in other words, unseen samples of the same “0-4” digits (i.e. labels) as the training set.
Furthermore, outlier2 (clothes images) has even lower likelihood than outlier1 (handwritten digits images) given it is from a very different domain.
This shows that
Whittle
Network is also able to clearly distinguish very different domains.
G. Computational Complexity and Running Times
The following running times were estimated on a workstation with AMD Ryzen Threadripper 1950X 16-Core Processor with 128GB of RAM.
The deep neural network experiments were executed on a GPU NVIDIA GeForce GTX 1080 Ti with 11GB of RAM.
Learning both the structure
T e s t T e s t
O u tli e r O u tli e r
O u tli e r
O u tli e r and the parameters of the WSPN on CPU takes 15min on
Sine dataset, 106min on MNIST, 23min on S&P , 76min on Stock , and 65min on Billiards .
Computing the Whittle likelihood for all training, test, and OOD data takes 8min in total on MNIST and less than 1min on the other datasets.
As a comparison, MADE takes 14min on Sine , 12min on
MNIST, 15.4s on S&P , 18s on Stock , and 7min on Billiards .
ResSPN takes 194min on Sine , 45h on MNIST, 35min on
S&P , 64min on Stock , and 248min on Billiards .
We summarize the running times (in minutes) and the number of generated edges from the conditional independencies extraction procedure performed on CPU in Tab. 6.
The results show that, when applying BIC, the complexity of the generated graphs can be reduced, resulting also in shorter running times.
When BIC is not employed, the computational complexity of the graph generation is linear in the data size N and quadratic in the number of graph nodes k , i.e. O ( N · k 2 ) .
In the forecasting scenario, both CSPN and WSPN are trained on GPU, taking 3min 24s for training 10 epochs, with BIC without BIC
[Table]
and 16s for test.
Training Whittle AE takes about 146min for 200 epochs on MNIST on GPU.
Testing the Whittle AE on MNIST takes 20s in total.