G RANGER C AUSAL S TRUCTURE R ECONSTRUCTION FROM H ETEROGENEOUS M ULTIVARIATE T IME S ERIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Granger causal structure reconstruction is an emerging topic that can uncover causal relationship behind multivariate time series data.
In many real-world systems, it is common to encounter a large amount of multivariate time series data collected from heterogeneous individuals with sharing commonalities, however there are ongoing concerns regarding its applicability in such large scale complex scenarios, presenting both challenges and opportunities for Granger causal reconstruction.
To bridge this gap, we propose a G ranger c
A usal S tructur E
R econstruction ( GASER ) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series.
In particular, we address the problem through a novel attention mechanism, called prototypical Granger causal attention.
Extensive experiments, as well as an online A/B test on an E-commercial advertising platform, demonstrate the superior performances of GASER.
I NTRODUCTION
Broadly, machine learning tasks are either predictive or descriptive in nature, often addressed by black-box methods (Guo et al., 2018).
With the power of uncovering relationship behind the data and providing explanatory analyses, causality inference has drawn increasing attention in many ﬁelds, e.g. marketing, economics, and neuroscience (Pearl, 2000; Peters et al., 2017).
Since the cause generally precedes its effects, known as temporal precedence (Eichler, 2013), recently, an increasing number of studies have focused on causal discovery from time series data.
They are commonly based on the concept of Granger causality (Granger, 1969; 1980) to investigate the causal relationship with quantiﬁcation measures.
In many real-world systems, it is common to encounter a large amount of multivariate time series (MTS) data collected from different individuals with shared commonalities, which we deﬁne as heterogeneous multivariate time series.
The underlying causal structures of such data often vary (Zhang et al., 2017; Huang et al., 2019).
For example, in the ﬁnancial market, the underlying causal drivers of stock prices are often heterogeneous across stocks of different plates.
Similar phenomenons are also observed in the sales of different products in E-commerce.
To this situation, most existing methods have to train separate models for MTS of each individual, which suffer from over-ﬁtting especially given limited training samples.
Although some works have been proposed to solve such problem (Zhang et al., 2017; Huang et al., 2019), they lack the inductive capability to do inferences for unseen samples and fall short of fully exploiting shared causal information among the heterogeneous data which often exist in practice.
For instance, the causal structures of the products belonging to the same categories are usually similar.
Such shared information presents opportunities for causal reconstruction to alleviate overﬁtting and to do inductive reasoning.
However, it is also challenging to detect common and speciﬁc causal structures simultaneously.
In this paper, we propose a G ranger c
A usal S tructur E R econstruction ( GASER ) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series data.
Our approach builds on the idea of quantifying the contributions of each variable series into the prediction of target variable via a novel designed prototypical Granger causal attention mechanism.
In order to ensure that the attention capturing Granger causality, we ﬁrst design an attention mechanism based on Granger causal attribution of the target series and then perform prototype learning that generates both shared and speciﬁc prototypes to improve the model’s robust-
ness.
Extensive experiments demonstrate the superior causal structure reconstruction and prediction performances of GASER.
In summary, our speciﬁc contributions are as follows:
• A novel framework that inductively reconstructs Granger causal structures and uncovers common structures among heterogeneous multivariate time series.
• A prototypical Granger causal attention mechanism that summarizes variable-wise contributions towards prediction and generates prototypes representing common causal structures.
• Relative extensive experiments on real-world, benchmark and synthetic datasets as well as an online A/B test on an E-commercial advertising platform that demonstrate the superior performance on the causal discovery and the prediction performance comparable to state-of-the-art methods.
GASER
In this section, we formally deﬁne the problem, introduce the architecture of GASER, present the prototypical Granger causal attention with the ﬁnal objective function.
2. P ROBLEM D EFINITION
Assuming we have a set of heterogeneous multivariate time series from N individuals,
i.e., X =
{ X i } Ni =1 , with each consisting of S time series of length T , denoted as X i =
( x 1 i , x 2 i , . . .
, x Si ) T ∈ R S × T , where x si = ( x si, 1 , x si, 2 , . . .
, x si,T ) T ∈ R T represents the s -th time series of individual i , and one of them is taken as the target series y i .
We aim to train a model that (1) reconstructs Granger causal structures among variables for each individual; (2) generates K common structures among all the N individuals, each structure represented by a prototype p k ∈ R S , k = 1 , ..., K ; and (3) learns a nonlinear mapping to predict the next value of the target variable series for each individual, i.e., ˆ y i,T
+1 =
F ( X i ) .
2. N ETWORK A RCHITECTURE
Our GASER framework consists of two parts: a set of parallel encoders, each predicting the target given the past observations, and an attention mechanism that generates prototypical Granger causal attention vectors to quantify variable-wise contributions towards prediction.
Figure 1 illustrates the overall framework of GASER.
As illustrated in Figure 1(a), for an input multivariate time series
X i , the encoder speciﬁc to s -th variable projects the time series x si into a sequence of hidden state, denoted as h si,t
= H s ( x i,t , h si,t − 1 ) .
The encoder could be any RNN models, such as LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014).
The last hidden states, { h si,T } Ss =1 , are used as the hidden embeddings of each variable.
Then the predicted next value of the target variable conditioned on historical data of variable s , denoted as ˆ y si,T +1 , can be computed by ˆ y si,T +1 =
f s ( h si,T ) , where f s ( · ) denotes the MLP network speciﬁc to variable s .
Then we obtain the prediction ˆ y i,T + by aggregating the predicted values speciﬁc to variables through the prototypical Granger causal attention described below.
2. P ROTOTYPICAL G RANGER C AUSAL A TTENTION
We propose a novel attention mechanism in GASER, namely prototypical Granger causal attention, to reconstruct Granger causal relationships for each individual and uncover common causal structures among heterogeneous individuals.
The goal is to learn attentions that can reﬂect the Granger causal strength between variables for each individual, and generate prototypes among heterogeneous individuals.
As illustrated in Figure 1(b), the idea of the prototypical Granger causal attention mechanism is as follows.
The Granger causal attribution corresponding to each individual is ﬁrst computed according to the concept of Granger causality, followed by prototype learning that summarizes common causal structures for heterogeneous individuals in the training set, and produces the attention vector speciﬁc to each individual.
The details of these two parts are described below.
2.3. G RANGER C AUSAL A TTRIBUTION
Granger causality (Granger, 1969; 1980) is a concept of causality based on prediction, which declares that if a time series x Granger-causes a time series y , then y can be better predicted using all available information than if the information apart from x had been used.
Thus, we obtain the
Granger causal attributions by comparing the prediction error when using all available information
[Figure]
Si
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
The architecture of the proposed GASER.
[Figure]
i,T + ˆ y all \ S i,T + y i,T +
[Figure]
with the error when using the information excluding one variable series.
In particular, given all the hidden embeddings { h si,T } Ss =1 of individual i , we obtain the embedding that encodes all available information and the one that encodes all available information excluding one variable s , denoted as h alli and h all \ s i respectively, by concatenating the embeddings of corresponding variables:
[Equation]
where [ · ] represents the concatenation operation.
Then we feed them into respective predictors, denoted as g all ( · ) and g s ( · ) , to get the predicted value of target and compute the squared errors:
[Equation]
where the predictor g all ( · ) and g s ( · ) can be MLP networks.
Inspired by Schwab et al.
(2019), we deﬁne the Granger causal attribution of the target variable corresponding to variable s as the decrease in error when adding s -th series to the set of available information, computed as:
[Equation]
where ReLU ( · ) is the rectiﬁed linear unit.
For each individual i , by normalising the Granger causal attribution, we obtain an attention vector that reﬂects Granger causality, namely Granger causal attention, denoted as qqq i .
The attention factor for variable s can be computed as:
[Equation]
2.3. P ROTOTYPE L EARNING
The Granger causal attention above is not robust enough to reconstruct Granger causal structure, given limited data (e.g., very short time series) of each individual in training.
We address the problem by generating Granger causal prototypes from all the individuals, under the assumption that there should be several common causal structures among heterogeneous individuals.
In particular, we assume there exist K Granger causal prototypes, denoted as { p k } Kk =1 , and compute the similarity between the Granger causal attention vector qqq i of individual i and each prototype vectors p k .
Since the attention can be seen as a distribution, we use the cosine similarity:
[Equation]
Then we output a prototype most similar to q i by sampling from the similarity distribution
d i using
Gumbel-Softmax (Maddison et al., 2017; Jang et al., 2016), which samples from a reparameterized continuous distribution approximation to the categorical one-hot distribution:
[Equation]
where GumbelSoftmax ( · ) denotes the Gumbel-Softmax function, e ∈ R K is the sample vector which approaches one-hot, and g is a vector of i.i.d.
samples drawn from Gumbel (0 , 1) distribution.
τ is the softmax temperature, and the distribution becomes discrete when τ goes to 0.
With the sample vector e , the output prototype ˆ p can be obtained as:
[Equation]
After normalizing the sampled prototype, we obtain an attention vector for individual i , denoted as r i , namely prototypical attention.
The Granger causal attention reﬂects the Granger causal structure speciﬁc to each individual, while the prototypical attention reﬂects one common Granger causal structure most similar to the Granger causal structure of each individual.
To detect the speciﬁc and common causal structures simultaneously, we summarize them together and generate the prototypical Granger causal attention a i as follows:
[Equation]
where α ∈ [0 , 1] is a hyperparameter that controls the ratio of the two attention mechanism.
Finally, the prediction of the target variable’s next value can be computed as the weighted sum of the predicted values from all variables:
[Equation]
2. L EARNING O BJECTIVE
In order to obtain accurate prediction and Granger causality structure, and generate diverse common causality structures, the objectives of GASER consist of three parts.
The ﬁrst two objective functions are to encourage accurate predictors, including the predictors f ( · ) to perform ﬁnal prediction and the auxiliary predictors g ( · ) to compute Granger attribution, and we adopt the the mean squared error (MSE) as the prediction loss function:
[Equation]
The last objective function is to avoid duplicate prototypes by a diversity regularization term that penalizes on prototypes that are similar to each other (Ming et al., 2019):
[Equation]
where γ controls the closeness to a tolerable degree.
To summarize, the loss function, denoted by L , is given by:
[Equation] [Equation]
where λ 1 and λ 2 are hyperparameters that adjust the ratios between the losses.
E XPERIMENTS
In this section, we evaluate the causal structure reconstruction performance on multivariate time series from both single individual and multiple individuals, as well as the prediction performance of
GASER.
We also conduct an online A/B test on an E-commerce advertising platform to further test
GASER in more practical situations.
3. E XPERIMENTAL S ETUP
We ﬁrst evaluate the causal structure reconstruction performances on two causal benchmark datasets.
Finance (Kleinberg, 2009) consists of simulated ﬁnancial market time series with known underlying causal structures.
Each dataset includes 25 variables of length 4,000.
For each dataset, we choose variables that are related to the most causes as the target variables to test model abilities in the relatively most challenging scenarios.
FMRI (Smith et al., 2011) contains 28 different Blood-oxygen-level dependent time series datasets with the ground-truth causal structures.
In the experiments, we evaluate on the ﬁrst 5 datasets and take the ﬁrst variable as the target as causal variables distribute relatively evenly in this dataset.
Then, we evaluate the causal structure reconstruction performance on heterogeneous individuals on synthetic data:
Synthetic data : We ﬁrst obtain the S exogenous time series through the following Non-linear Autoregressive Moving Average (NARMA) (Atiya & Parlos, 2000) generators:
[Equation]
where ε t are zero-mean noise terms of 0.01 variance, d is the order of non-linear interactions, and α s , β s and γ s are parameters speciﬁc to variable s , generated from N (0 , 0 . 1) .
Then, we generate the target series from the generated exogenous series via the formula:
[Equation]
where ω si ∈ { 0 , 1 } with 0.6 probability of being zero that controls the underlying causal relationship from the s -th variable to the target variable, ηηη si ∈ R p controls the causal strength sampling from
Unif {− 1 , 1 } , and x si,t − p : t − 1 = ( x si,t − p , x si,t − p +1 , . .
, x si,t − 1 ) T ∈ R p represents the last p historical values of variable s of sample i .
The 0-1 indicator vector ωωω i = ( ω 1 i , ω 2 i , . . .
, ω Si ) T ∈ R S is the ground-truth causal structure of i -th individual.
For the causal structure reconstruction task, we compare our method with previous causal discovery methods including linear Granger causality (Granger, 1969; L ¨ utkepohl, 2005) and TCDF (Nauta et al., 2019), as well as the interpretable neural network based prediction method, i.e., IMVLSTM (Guo et al., 2019), using the standard metrics of Area Under the Precision-Recall Curve (PR-AUC), and Area Under the ROC Curve (ROC-AUC) (Fawcett, 2006).
Since a byproduct of GASER is the time series prediction, we also evaluate the prediction performance on the real-world datasets, i.e., PM2.5 and SML:
PM2.5 contains the hourly PM2.5 and meteorological data in Beijing during Jan 2010 to Dec 2014, includes 7 variables (such as PM2.5 concentration, temperature, pressure and wind speed), and forms a multivariate time series of length 43,824.
The PM2.5 concentration is the target series.
the dataset is split into training (60%), validation (20%) and testing sets (20%).
SML is a monitoring dataset for the temperature forecasting, collected from a monitor system in a domotic house for approximately 40 days.
The data are sampled every minute and smoothed with the mean of every 15 minutes, forming the MTS of length 4,137.
We predict the dinning-room temperature with 17 relevant variable series.
The ﬁrst 3,200, the following 400 and the last 537 data points are respectively used for training, validation, and test.
We compare with linear Granger causality (VAR) and the state-of-the-art prediction models including DUAL (Qin et al., 2017) and IMV-LSTM (Guo et al., 2019), and adopt Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) as metrics.
[Table]
3. E XPERIMENTAL R ESULTS 3.2. C AUSAL S TRUCTURE R ECONSTRUCTION P ERFORMANCE ON H OMOGENEOUS M ULTIVARIATE T IME S ERIES
To evaluate the causal discovery performance on homogeneous multivariate time series, we train individual models for each dataset with the hyper-parameter α equaling 0.5.
We report PR-AUC and ROC-AUC averaged across all datasets, with the standard deviation reported in Table 1.
As can be seen, the proposed method greatly surpasses other methods.
Especially, GASER recovers the ground-truth causal structure with high score on the Finance data.
3.2. C AUSAL S TRUCTURE R ECONSTRUCTION P ERFORMANCE ON H ETEROGENEOUS M ULTIVARIATE T IME S ERIES
In this part, we evaluate the causal discovery performance on heterogeneous multivariate time series.
We denote the number of common causal structures as C , the number of variables as S and the series length as T , and generate 100 multivariate time series for each common causal structure according to Equation (14) and Equation (15), forming 100 C datasets.
For the inductive methods GASER and IMV-LSTM, we train one model using all the datasets, while for other methods, we train separate models for each dataset.
We report PR-AUC and ROC-AUC results w.r.t the variable number, the series length and the common structure number in Table 2 to 4, respectively.
We observe that
GASER outperforms other methods signiﬁcantly in all cases, and GASER ( α = 0 .
5 ) (with the Prototypical Granger causal attention) performs better than GASER ( α = 1 ) (only with Granger causal attention).
The observations demonstrate the superior causal discovery performance of GASER, the effectiveness of the prototypical Granger causal attention in GASER, and the advantages of utilizing shared commonalities among heterogeneous MTS.
Regarding the other competitors, linear Granger performs the best followed by TCDF and IMV-LSTM at most cases.
The possible reason is that linear Granger can detect Granger causal relations to some extent, though it utilizes linear model, i.e., Vector autoregression (VAR).
TCDF utilizes attention-based CNN to inference potential causals followed by a causal validation step, but the attention it proposed cannot reﬂect Granger causality, thus achieves unsatisfactory performance.
Compare to the performance on homogeneous multivariate time series, the performance of IMV-LSTM drops dramatically, which indicates that the attention mechanism in IMV-LSTM fails given heterogeneous multivariate time series.
In Table 2, we vary the number of variables to generate datasets of different complexity, and we can see that GASER outperforms other competitors consistently across different S , and achieves good performance when S is as large as 20, demonstrating our method’s capability to infer complex causal structures.
Since in practice, the size of collected data is often limited, which poses challenges to recover causal structure, thus we also vary the length of time series to see the model robustness to data of small sizes.
As can be seen in Table 3, GASER outperforms other methods across all cases, even when T is as small as 20, which demonstrates that advantage of using shared information.
We also observe that GASER ( α = 0 . 5 )
surpasses GASER ( α = 1 ) by a large margin, which demon-
[Table] [Table]
[Table]
strates that learning prototypical attention can alleviate the over-ﬁtting problem.
In Table 4, we control the causal heterogeneity by varing the number of common causal structures C
= { 3 , 5 , 7 } .
We observe that the performance of GASER decreases with increasing C .
In Figure 2, we map the learned causal attention vectors to a 2D space by the visualization tool t-SNE (Maaten & Hinton, 2008).
Individuals of different causal structures are labeled by different colors.
From the results, we observe that nodes belonging to the same causal structures are clustered together, which also demonstrates the effectiveness of our method.
3.2. P REDICTION P ERFORMANCE
We evaluate the prediction performance on the real-world datasets, i.e., PM2.5 and SML.
To evaluate the robustness of prediction and the accuracy of Granger causal attribution, we also build another datasets that only contain top 50% important variables towards prediction detected from each method.
We report the prediction results in Table 5.
As can be seen, GASER achieves the best performance on PM2.5 data with all features, demonstrating its superior prediction performances.
We also observe that GASER achieves comparable or even better performance using selected variables, while the others’ performances decrease, which indicates that effective variable selection of
GASER.
Linear Granger achieves the best performance on SML data, because the time series length of SML is short, thus providing limited training samples for neural-network-based methods.
3. O NLINE A/B T ESTS
In order to further evaluate the effectiveness of GASER in practice, an online A/B test is conducted on an E-commercial platform [Table] and the process is designed as follows:
• We ﬁrst train GASER on the historical MTS of 30,665 items.
Each MTS includes 26 variables related to searching, recommending and advertising, such as Page View (PV), Gross Merchandise Volume (GMV) and Impression Position In-Page, etc.
Here, we take the item popularity as the target series, and generate the underlying causal structure for each item.
• We randomly sample 100 items whose impression position in-page Granger-causes the item popularity with high conﬁdence, and divide them into two buckets.
For Bucket A, we adjust impression positions in-page of each item by one grid since 2019/08/19 till 2019/08/29, and ensure the intervention has little impact on other variables.
For Bucket B, we do nothing.
• We compare the improvement rate of item popularity week-on-week on the two bucket in Fig 3.
[Table]
#$% !"
#$ !"
#"% "& "#"%& "#$&
[Figure]
($’
[Figure]
’($)
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
As shown in Figure four days after the beginning of the intervention the item popularity improvement rate of Bucket A consistently outperforms that of Bucket B and the gap between the two buckets increases signiﬁcantly since 2019/08/25, which shows that the intervention, i e., adjusting the impression positions in-page causes the improvement on item popularities, thus demonstrates the right causal relationships detected by GASER.
R ELATED W ORK
Recently a considerable amount of work has been proposed for causal inference.
Classical methods, such as constraint-based methods (Pearl, 2000; Spirtes et al., 2000; Peters et al., 2013; Runge et al., 2017; Zhang et al., 2017), score-based methods (Chickering, 2002) and functional causal models (FCM) based methods (Shimizu et al., 2006), mainly focus on i.i.d data.
Under the scope of time series, causal inference is commonly based on the notion of Granger causality (Granger, 1969; 1980), and a classical way is to estimate linear Granger causality under the framework of VAR models (L ¨ utkepohl, 2005).
However, existing classicial methods fail to uncover causal structures inductively.
Neural network based methods that infer causal relationships or relations that approach causality have gained increasing popularity.
Lopez-Paz et al. (2015) learns a probability distribution classiﬁer to unveil causal relations.
Kipf et al.
(2018) proposes a neural relation inference model to infer interactions while simultaneously learning the dynamics.
Yu et al.
(2019) develops a deep generative model to recover the underlying DAG from complex data.
Attention mechanism has often been adopted to discover relations between variables.
For example, Dang et al.
(2018) discovers dynamic dependencies with multi-level attention.
Nauta et al.
(2019) studies causal discovery through attention-based neural networks with a causal validation step.
Guo et al.
(2019) proposes an interpretable multi-variable LSTM with mixture attention to extract variable importance knowledge.
However, these attention mechanisms provide no incentive to yield accurate attributions (Sundararajan et al., 2017; Schwab et al., 2019).
Since our method utilizes the concept of prototype to detect common causal structures, another line of related research is about prototype learning.
Prototype learning is a form of cased-based reasoning (Slade, 1991), which solves problems for new inputs based on similarity to prototypical cases.
Recently prototype learning has been leveraged in interpretable classiﬁcation (Bien et al., 2011;
Kim et al., 2014; Snell et al., 2017; Li et al., 2018; Chen et al., 2018) and sequence learning (Ming et al., 2019).
We incorporate the concept for Granger causal structure reconstruction on time series data for the ﬁrst time.
C ONCLUSION
We formalize the problem of Granger causal structure reconstruction from heterogeneous MTS data and propose an inductive framework GASER to solve it.
In particular, we propose a novel attention mechanism, namely prototypical Granger causal attention, which computes Granger causal attribution combined with prototype learning, to reconstruct Granger causal structures and uncover common causal structures.
The approach has been successfully evaluated by ofﬂine experiments on real-world and synthetic datasets compared to previous methods, also conﬁrmed by an online A/B test on an E-commercial platform.
A T ABLE OF N OTATIONS
[Table]
B A LGORITHM P SEUDOCODE
The full algorithm is presented in Algorithm 1.
The network parameter set Θ includes the parameters of sequence encoders and MLPs.
We adopt stochastic gradient descent (SGD) to optimize the network parameters and the prototype parameters.
To initialize the prototypes, we ﬁrst pretrain
GASER for several epochs
[Table]
and then employ k -means with cosine similarity on the Granger causal attentions { q } Ni =
[Table]
and ﬁnally we take the cluster centers as the initial prototypes.
Note that the Gumbel-softmax function is only adopted in the training phase to backpropagate
[Table]
and replaced by the argmax function in inference.
Algorithm 1: The algorithm of GASER.
[Table]
X
[Table]
{ X i } Ni =
[Table]
Number of prototypes K ; Maximum iterations MaxIter
Hyperparameters α , γ , λ 1 and λ 2 .
[Table]
Network parameters Θ ; Attention vectors { aaa i } Ni =1 ; Prototypes
C A DDITIONAL D ETAILS ON THE E XPERIMENTAL S ETUP C. D ATASETS
In this section, we provide some additional dataset details.
Finance data are available at http: //www.skleinberg.org/data.html .
We use the processed FMRI data provide by (Nauta et al., 2019).
The source and details of PM2.5 and SML are at https://archive.ics.uci.
edu/ml/datasets/Beijing+PM2.5+Data and https://archive.ics.uci.edu/ ml/datasets/SML2010 , respectively.
C. I MPLEMENTATION D ETAILS
We implement GASER in Tensorﬂow (Abadi et al., 2016) by the Adam optimizor (Kingma & Ba, 2014) with the learning rate set to 0.001.
We adopt LSTMs as the sequence encoders with the hidden states size set to 128 and the window size set to 5.
In all experiments, we ﬁrst pretrain GASER with only the Granger causal attention for 40 epochs.
The hyperparameters λ 1 and λ 2 are both set to 1, and the softmax temperature in Gumbel-softmax is set to 0.1.
C. C OMPARED M ETHODS
Linear Granger (Granger, 1969; 1980):
We conduct a Granger causality test in the context of Vector Autoregression (VAR) as described in chapter 7.6.3 in (L ¨ utkepohl, 2005) and implemented by the Statsmodels package (Seabold & Perktold, 2010).
In detail, we perform F-test at 5% signiﬁcance level.
The maximum number of lags to check for order selection is set to 5, which is larger than the causal order in the ground-truth.
TCDF (Nauta et al., 2019): TCDF learns causal structure on multivariate time series by attentionbased convolutional neural networks combined with a causal validation step.
The codes are available at https://github.com/M-Nauta/TCDF .
In all experiments, we follow the default settings as described in (Nauta et al., 2019), i.e., the signiﬁcance number (stating when an increase in loss is signiﬁcant enough to label a potential cause as true) as 0.8, the size of kernels as 4, dilation coefﬁcient as 4, the learning rate as 0.01, and adopting Adam optimizator.
DUAL (Qin et al., 2017):
It is an encoder-decoder RNN with an input attention mechanism, which forces the model pay more attention on certain driving series rather than treating all the input driving series equally.
In the experiment, we use the input-attention factors to detect important variables as (Guo et al., 2019) did.
We set the the size of hidden states for encoder and decoder to 64 and the window size to 10 as stated in the paper.
IMV-LSTM (Guo et al., 2019):
It is a multi-variable attention-based LSTM capable of both prediction and variable importance interpretation, with the attention factors reﬂecting importance of variables in prediction.
Thus, we take the learnt attention vectors as the Granger causal weights in the experiment.
The codes are available at https://github.com/KurochkinAlexey/
IMV_LSTM .
In all experiments, IMV-LSTM is implemented by Adam optimizer with the minibatch size 64, hidden layer size 128 and learning rate 0.001.
D S UPPLEMENTARY E XPERIMENTAL R ESULTS D. V ISUALIZATION OF L EARNED P ROTOTYPES
We visualize the the learned prototypes and the ground-truth causal structures in Fig.
In this experiments, we set the hyper-parameter of prototype number K equal to the ground-truth common structure number C .
From the results, we can see that the learned prototypes are similar to the ground-truth causal structures, which demonstrates the learned prototypes are interpretable.
D. V ISUALIZATION OF G RANGER C AUSAL A TTENTION
In this part, we visualize the Granger causal attention vectors over epochs during the training phase and the ground-truth causal structures in Fig.
From the results, we can see that for the later epochs, p0 p1 p x x x x x x x x x x
Learned prototypes
C0 C1 C x x x x x x x x x x
Ground-truth structures 0.
(a) C = p0 p1 p2 p3 p x x x x x x x x x x
Learned prototypes
C0 C1 C
C
C x x x x x x x x x x
Ground-truth structures −0.
(b) C = shown on the right of the ﬁgure, the Granger causal attention vectors are similar to the groundtruth structures.
It demonstrates that the proposed Granger causal attention mechanism forces the attention values to correlate with Granger causality.
D. R UNNING T IME
We compare the running time including the training time and the inference time with linear Granger method, i.e. VAR.
The experiments were carried out on a server with 64 Intel(R)
Xeon(R) E
[Table]
v4 2.50GHz processors and 512 GB RAM.
Since the linear Granger method is not inductive and has to retrain for new individuals, we set the inference time equal to the training time.
As shown in
Table 7, although GASER takes more time on the training phase, about 6.4 times slower than linear
Granger, it is about 44.5 times faster on the inference phase.
[Table]
[Figure]
[Figure]
D. P ARAMETER S ENSITIVITY
We investigate the parameter sensitivity in this section.
Speciﬁcally, we evaluate the sensitivity of
GASER to different numbers of prototypes K and different values of hyper-parameter α , γ , λ 1 and λ 2 .
The results on heterogenous MTS ( C =5, S =10, T =100) are shown in Fig.
We ﬁrst show how the number of prototypes affects the performance in Fig. 6(a).
We can see that the performance raises when K increases and achieves the best performance when the number of prototypes K is equal to the number of underlying common structures C .
Then, the performance starts to drop slowly.
Overall, the proposed method is not very sensitive to the parameter K .
Then we evaluate how the value of α affects the performance in Fig. 6(b).
The parameter α balances the weight of the Granger causal attention and the prototypical attention in our model.
We can see that when α = 0.5, the model reaches the best performance, which demonstrates both the Granger causal attention and the prototypical attention are essential in our model, and it is important to ﬁnd a good balanced point between them.
The parameter γ controls the closeness of different prototypes.
The smaller the γ , the prototypes are more diverse.
The result is shown in Figure 8(c).
We can see that when γ is too small, the result is not good.
This is intuitive as the ground-truth common structures among heterogeneous MTS are not totally different.
Too large γ also deteriorates the performance, because it will result in a number of similar or even duplicate prototypes.
Thus, it is important to determine the parameter γ carefully.
Figure 6(d) shows the the performance of GASER w.r.t.
the parameter λ 1 .
The parameter λ 1 is the weight of the auxiliary prediction loss.
From the result, we can see that the performance raises then λ 1 increases initially, and then keeps stable.
It demonstrates that the auxiliary predictors are essential to detect the Granger causality.
Finally, we show how the value of λ 2 affects the performance in Fig. 6(e).
The parameter λ 1 controls the weight of the prototype diversity regularization.
From the result, we can see that the prototype diversity loss is essential in our mobel, but we should concentrate more on choosing an appropriate weight.