E XPLAINING T IME S ERIES BY C OUNTERFACTUALS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a method to automatically compute the importance of features at every observation in time series, by simulating counterfactual trajectories given previous observations.
We deﬁne the importance of each observation as the change in the model output caused by replacing the observation with a generated one.
Our method can be applied to arbitrarily complex time series models.
We compare the generated feature importance to existing methods like sensitivity analyses, feature occlusion, and other explanation baselines to show that our approach generates more precise explanations and is less sensitive to noise in the input signals.
I NTRODUCTION
Multi-variate time series data are ubiquitous in application domains such as healthcare, ﬁnance, and others.
In such high stakes applications, explaining the model outcome is crucial to build trust among end–users.
Finding the features that drive the output of time series models is a challenging task due to complex non-linear temporal dependencies and cross-correlations in the data.
The explainability problem is signiﬁcantly exacerbated when more complex models are used.
Most of the current work in time series settings focus on evaluating globally relevant features (Yang et al., 2005; Yoon et al., 2005; Hmamouche et al., 2017).
However, often global feature importance represents relevant features for the entire population, that may not characterize local explanations for individual samples.
Therefore we focus our work on individualized feature importance in time series settings.
In addition, besides identifying relevant features, we also identify the relevant time instances for those speciﬁc features, i.e., we identify the most relevant observations.
To the best of our knowledge this is the ﬁrst sample–speciﬁc feature importance explanation benchmark at observation level for time series models.
In this work, we propose a counterfactual based method to learn the importance of every observation in a multivariate time series model.
We assign importance by evaluating the expected change in model prediction had an observation been different .
We generate plausible counterfactual observations based on signal history, to asses temporal changes in the underlying dynamics.
The choice of the counterfactual distribution affects the quality of the explanation.
By generating counterfactuals based on signal history, we ensure samples are realistic under individual dynamics, giving explanations that are more reliable compared to other ad-hoc counterfactual methods.
M ETHOD : F EED -F ORWARD C OUNTERFACTUALS FOR T IME S ERIES E XPLANATION
b l a c k - b o x o u t c o m e i n p u t s i g n a l c o un t e r f a c t u a l o b s e r v a t i o na c t u a l o b s e r v a t i o na c t u a l o u t c o m e c o un t e r f a c t u
a l o u t c o m et
i m e t i m e a c t u a l o b s e r v a t
i o nc o un t e r f a c t u
a l o u t c o m e} } f e a t u r e i m pf e a t u r e i m pc o un t e r f a c t u
a l o b s e r v a t i o n
In this section we describe our method, Feed
Forward Counterfactual (FFC), for generating explanation for time series models.
A feature is considered important if it affects the model output the most.
In time series, the dynamics of the features also change over time, which may impact model outcome.
As such it is critical to also identify the precise time points of such changes.
In order to ﬁnd important observations for highdimensional time series models, we propose to use a counterfactual based method.
Speciﬁcally,
[Figure] [Figure]
[Figure] [Figure]
Generator ...
PredictorNetwork
[Figure] [Figure]
We look into the difference of the original output y t and ˆ y
( i,t )
where observation x (
[Figure]
is replaced with a counterfactual.
we assign importance to an observation x i,t (feature i at time t ) based on its effect on the model output at time T ( > t ) .
We replace observation x i,t with a counterfactual ˆ x i,t , to evaluate this effect.
Figure 1 demonstrates how importance of a feature is characterized by replacing an observation with a counterfactual.
2. N OTATION
Multi-variate time series data is available in the form of X ( n ) ∈ R d × T (where d is the number of features with T observations over time) for n ∈ [ N ] samples.
We are interested in black-box estimators that receive observations up to time t , x t ∈ R d , and generate output y t at every time point t ∈
[ T ] .
F denotes the target black-box model we wish to explain through the proposed approach called FFC.
For exposition, throughout the paper, the index n over samples has been dropped for notational clarity.
We index features with subscript i .
x − i,t indicates features at time t excluding feature i .
The notation used for exposition work is brieﬂy summarized in Table 1.
[Table] [Table]
[Table] [Table]
2. E XPOSITION
We assign importance score to each observation x i,t for t ∈
[ T ] and i ∈
[ d ] , following the deﬁnition:
Deﬁnition 1.
Feature Importance:
The importance of the observation i at time t , denoted by
Imp ( i, t ) is deﬁned as E p (ˆ x i,t | X 0: t − 1 )
[ |F ( X 0: t ) − F ( X 0: t − 1 , x − i,t , ˆ x i,t ) | ] , where | · | denotes the absolute value and ˆ x i,t is the counterfactual sample.
That is, the importance of an observation for feature i at time t is deﬁned as the change in model output when the observation is replaced by a generated counterfactual.
The counterfactual observation can come from any distribution, however the quality of the counterfactual random variable directly affects the quality of the explanation.
We generate the counterfactual sample conditioned on signal history up to time t by sampling from the distribution p ( x i,t | X 0: t − 1 ) .
Using a conditional generator guarantees that our counterfactuals are sampled not only within domain but also speciﬁcally likely under the individual sample X ( n ) , as opposed to having a generator that models data across population.
Conditioning on the history also allows us to learn the dynamics of the signal and therefore generate a plausible counterfactual given the past observations.
p ( x t | X 0: t − 1 ) represents the distribution at time t , if there were no change in the dynamics of the signals.
The counterfactual ˆ x i,t is sampled from the marginal distribution p ( bx i,t | X 0: t − 1 ) , obtained from p ( x t | X 0: t − 1 ) .
Let F ( X 0: t − 1 , x − i,t , ˆ x i,t ) be the output of the model at time T , when x i,t is replaced by the generated counterfactual ˆ x i,t .
We estimate feature importance Imp ( i, t ) as
E p (ˆ x i,t | X 0: t − 1 ) [ |F ( X 0: t ) − F ( X 0: t − 1 , x − i,t , ˆ x i,t ) | ] , summarized in ﬁgure 2(b).
2. P ROPERTIES
Our proposed method has the following compelling properties in explaining the estimator F :
Time Importance (TI) For every time series, highlighting relevant time events for different features is important for actionability of the explanations.
For instance in a clinical setting, just knowing a feature like heart rate is relevant, is not sufﬁcient to intervene - it is also important to know when a deterioration had happened.
With FFC, the most eventful time instances can be obtained as:
[Equation]
We can thus rank time instances in order of importance.
That is, time t
[Equation]
2 , if
[Equation]
Feature Importance (FI)
At any time instance t , our method assigns importance to every feature of x i,t .
The magnitude of our importance function reﬂects relative importance.
Comparing the importance values across features gives the ﬂexibility to report a subset of important features at each time point t and also reﬂects the correlation between various features of the time series.
2. G ENERATOR M ODEL FOR CONDITIONAL DISTRIBUTION p ( x t | X 0: t − 1 )
We approximate the conditional distribution of p ( x t | X 0: t − 1 ) using a recurrent latent variable generator model G , introduced in Chung et al. (2015).
The architecture we use is provided in Figure 2(a).
The conditional generator G models p ( x t | z t − 1 ) where z t − 1 ∈ R k is the latent representation of history of the time series up to time t .
The latent representation is a continuous random variable, modeling the distribution parameters.
We only use past information in the time series to reﬂect temporal dependencies.
Using the recurrent structure allows us to model a non-stationary generative model that can also handle varying length of observations.
Implementation details of the generator are in the Appendix.
Our counterfactuals are not derived by looking at future values which could be done for reliable imputation.
Counterfactuals should represent the past dynamics.
Note that our derived feature importance is limited by the quality of imputation that may have been utilized by the black–box risk predictor.
For experimental evaluation on the effect of generator speciﬁcations on counterfactuals and the quality of explanations, see Section 4.1.
2. F EATURE I MPORTANCE A SSIGNMENT A LGORITHM
The proposed procedure is summarized in Algorithm 1.
We assume that we are given a trained block box model F , and the data (without labels) it had been trained on.
Using the training data, we ﬁrst train the generator that generates the conditional distribution, (denoted by G ).
In our implementation we model x as a multivariate Gaussian with full covariance to model all existing correlation between features.
The counterfactual ˆ x i,t is then sampled from G and passed to the black-box model to evaluate the effect on the black-box outcome.
R ELATED W ORK
A common method of explaining model performance, in time–series deep learning, is via visualization of activations of latent layers (Strobelt et al., 2018; Siddiqui et al., 2019;
Ming et al., 2017) or via
Input
F
Trained Black- predictor
Monte
Carlo samples 1: Train G 2: do 3:
4: p
: t
∼ G (
S do 6:
Sample ˆ x ( s ) i,t ∼ p ( x i,t | X 0:
− 1 ) 7: ˆ y ( s ) T
= F ( X 0: t , x − i,t , ˆ x ( s ) i,t ) 8: imp ( S ) T = | y T − ˆ y ( s ) T | 9:
Importance _ Matrix ( i, t ) =
Σ
Ss =0 imp ( S ) T S 10: Return Importance _ Matrix sensitivity analysis (Bach et al., 2015; Yang et al., 2018).
Understanding latent representations, sensitivity and its relationship to overall model behavior is useful for model debugging.
However, these but are too reﬁned to be useful to the end users like clinicians.
Attention models (Vaswani et al., 2017; Vinayavekhin et al., 2018; Xu et al., 2018) are the most commonly known explanation mechanisms for sequential data.
However, because of the complex mappings to latent space in recurrent models, attention weights cannot be directly attributed to individual observations of the time series (Guo et al., 2018).
To resolve this issue to some extent, Choi et al.
(2016) propose an attention model for mortality prediction of ICU patients based on clinical visits.
However attention weights may not be consistent as explanations (Jain and Wallace, 2019).
In vision, prior works tackle explainability from the counterfactual perspective, ﬁnding regions of the image that affect model prediction the most.
Fong and Vedaldi (2017) assumes higher importance for inputs that when replaced by an uninformative reference value, maximally change the classiﬁer output.
A criticism to such methods is that they may generate out-of-distribution counterfactuals, leading to unreliable explanations.
Chang et al.
(2019) address this issue for images using conditional generative models for inpainting regions with realistic counterfactuals.
Evaluating sample based feature importance remains largely unstudied for time series models.
While more widely studied for image classiﬁcation, (Bach et al., 2015; Fong and Vedaldi, 2017) these methods cannot be directly extended to time series models due to complex time-series dynamics.
Most efforts in this domain focus on population level feature importance (Tyralis and Papacharalampous, 2017).
Suresh et al.
(2017) is one of the few methods addressing sample based feature importance and use a method similar to Fong and Vedaldi (2017), called "feature occlusion".
They replace each time series observation by a sample from uniform noise to evaluate its effect on model outcome to attribute feature importance.
We argue that carefully choosing the counterfactual selection policy is necessary for derive reliable importances.
Speciﬁcally, replacing observations with noisy out-of-domain samples can lead to arbitrary changes to model output that are not reﬂective of systematic behavior in the domain.
Even if an observation is sampled from the domain distribution, it does not characterize temporal dynamics and dependencies well enough, potentially highlighting features that only reﬂect global model behavior, as opposed to sample speciﬁc feature importance.
We therefore model the data–distribution in order to generate reliable counterfactuals.
We demonstrate the implications of the choice of the generator (and hence the counterfactuals ) on the quality of explanation.
E VALUATION
We evaluate our explainability method for ﬁnding important features in time series on 2 simulated datasets and 2 real datasets.
Our goal is two-fold a) comparison to existing feature importance baselines in time series and b) evaluating the choice of generators on the quality of counterfactuals and explanations.
We compare to existing feature importance baselines described below:
Feature Occlusion (FO) (Suresh et al., 2017) : Method introduced in Suresh et al.
This method is an ad-hoc approach for generating counterfactuals.
When replacing x i,t with a random sample from the uniform distribution, the change in model output deﬁnes the importance for x i,t .
Augmented feature occlusion (AFO) : We augment the method introduced in Suresh et al.
(2017) by sampling counterfactuals from the bootstrapped distribution over each feature.
This avoids generating out-of-distribution samples.
Sensitivity Analysis
(SA) : This method evaluates the sensitivity of the output to every observation, by taking the derivative of y t with respect to x i,t , at every time point.
LIME (Ribeiro et al., 2016) :
One of the most commonly used explainabilty methods that assigns local importance to features.
Although LIME does not assign temporal importance, for this baseline, we use LIME at every time point to generate feature importances.
4. S IMULATED D ATA I
[Figure]
Evaluating the quality of explanations is challenging due to the lack of a gold standard/ground truth for the explanations.
Additionally, explanations are reﬂective of model behavior, therefore such evaluations are tightly linked to the reliability of the model itself.
Therefore we created the simulated environment in order to test our method.
In this experiment, we simulate a time series data such that only one feature determines the outcome.
Speciﬁcally, the outcome (label) changes to 1 as soon as a spike is observed in the relevant feature.
We keep the task fairly simple for two main reasons: 1) to ensure that the black-box classiﬁer can indeed learn the right relation between the important feature and the outcome, which allows us to focus on evaluating the quality of the explanations without worrying about the quality of the classiﬁer.
2) to have a gold standard for the explanations since the exact important event predictive of the outcome are known.
We expect the explanations to assign importance only to the one relevant feature, at the exact time of spike, even in the presence of spikes in other non-relevant features.
To simulate these data, we generate d = 3 (independent) sequences as a standard non–linear auto-regressive moving average (NARMA) time series of the form: x ( t + 1) = 0 .
5 x ( t )
+ 0 .
5 x ( t ) (cid:80) l − 1 i =
x ( t − l ) + 1 .
5 u ( t − ( l − 1))
u ( t )
+ 0 .
5 for t ∈ [80] , where the order is 2 and u ∼ Normal (0 , 0 . 01) .
We add linear trends to the features and introduce random spikes over time for every feature.
Note that since spikes are not correlated over time, no of the generators (used in FFC, AFO, FO) will learn to predict it.
The important feature in this setting is feature 1.
The complete procedure is described in Appendix A.2.1.
We train an RNN-based black-box model on this data, resulting in AUC = 0 .
99 on the test set.
Figure 7 demonstrates explanations of each of the compared approaches on simulated data fortest samples.
As shown in Figure 7(a), Sensitivity analysis does not pick up on the importance of the spike.
Feature occlusion gives false positive importance to spikes that happen in non-important signals as well as the important one.
Augmented feature occlusion resolves this problem since it samples the counterfactuals from the data distribution, however, it generates noisier results as it samples from the bootstrap distribution.
The proposed method (FFC) only assigns importance to the ﬁrst feature at the time of spike.
Hence, FFC generates fewer false relevance scores.
Note that all baseline methods provide an importance for evry sample at every time point.
The true explanation should highlight feature 1 at time points of spike.
Using this ground truth
[Table]
we evaluate the AUROC and AUPRC of the generated explanations denoted by (exp).
Table 2 summarizes the results for simulated data.
[Table]
4. S IMULATED D ATA II
The ﬁrst simulation does not necessarily evaluate feature importance under complex state dynamics as is common in applications.
In this simulation, we create a dataset with complex dynamics with known ground truth explanations.
The dataset consists of multivariate time series signals with 3 features.
A Hidden Markov Model with 2 latent states, with linear transition matrix and multivariate normal emission probabilities is used to simulate observations.
The the outcome y is a random variable, which, in state 1 , is only affected by feature 1 and in state 2 , only affected by feature 2 .
Also, we add non-stationarity to the time series by modeling the state transition probabilities as a function of time.
The ground truth explanation for output at time T is the observation x i,t where i is the feature that drives the output in the current state and t indicates when feature i became important.
In a time series setting, a full explanation for outcome at t =
T should include the most important feature variable as well as the time point of importance (here state change).
Figure 4 demonstrates assigned importance for a time series sample.
The shaded regions indicate the top 5 important observations ( x i,t ) for each method, the color indicating the corresponding feature i .
AFO, FO and FFC are able to learn the state dynamics and are able to ﬁnd the important feature of each state.
However, the top importance values in AFO and FO do not correspond to the important time points.
Only in FFC, the top important observations are indicative of state changes.
Tableshows the performance compared to ground-truth explanations for this data.
4.2. E FFECT OF GENERATOR SPECIFICATION
As mentioned earlier, the quality of explanations rely on the quality of the counterfactuals.
The counterfactuals should reﬂect the underlying latent dynamics for an individual sample.
Counterfactuals under the marginal (as used by AFO) need not be likely for a speciﬁc sample.
We demonstrate this by looking evaluating the log probability of the counterfactual under the true generator distribution p ∗ ( x t | X 0: t − 1 ) .
Results are summarized in Table 2, Column 3.
Since we simulate data using an
HMM, we can analytically derive the distribution p ∗ ( x i,t | X 0: t − 1 ) .
Details of the derivation are included in Appendix A.2.
. 4. MIMIC M ORTALITY
Explaining models based on feature importance is critical for clinical settings.
Most of clinical data come in form of time series therefore, in is important to ﬁnd critical time points in patients’ trajectories along with important features.
We evaluate our method on a benchmark mortality prediction task based on the Intensive Care Unit (ICU) time series data from MIMIC.
The MIMIC-III dataset consists
[Figure]
of de-identiﬁed EHRs for ∼ 40 , 000 ICU patients at the Beth Isreal Deaconess Medical Center,
Boston, MA.
The dataset has time series measurements such as vitals and lab results over patients
ICU stay (Johnson et al., 2016).
We use an RNN-based mortality predictor model as a black-box for evaluation.
The model is trained on 14 , 712 adult ICU admissions and reaches a classiﬁcation AUC of 0 .
007) , using 8 vital and 20 lab measurements, as well as patient static data.
More details on the model and data used are in Appendix A.3.
Following the procedure in Algorithm 1, we train a conditional generators for non-static time series features.
We compare results across all four existing methods by visualizing importance scores over time.
Figure 5 shows an example trajectory of a patient and the predicted outcome.
We plot the importance score over time for top 3 signals, selected by each method.
Shaded regions in bottom four rows indicate the most important observations, color representing the feature.
As shown in Figure
[Figure]
5, counterfactual based methods mostly agree and pick the same signals as important features.
We further evaluate this by looking into accordance scores among methods, indicating the percentage of agreement.
This analysis is provided in the Appendix A.3, and the heat map in Figuredemonstrates the average score across test samples.
However, the methods don’t agree on the exact time importance.
As we observe in Figure 5 and other patient trajectories, FFC assigns importance to observations at the precise times of signal change.
This is exactly as expected from the method.
The FFC counterfactual is conditioned on patient history and thus the counterfactual represents an observation assuming a patient had not change state.
4.3. E VALUATION USING C LINICAL A NNOTATIONS
Since evaluation of explanations can be subjective, we also use intervention information present in patient records to evaluate clinical applicability across baselines.
Clinicians intervene with a medication or procedure when there is a notable, adverse change in patient trajectory.
Therefore, picking up the most relevant features before an intervention onset is indicative of clinical validity of the method.
While we cannot directly associate an intervention with a speciﬁc cause (observation), we look at the overall distribution of signals considered important by each of the methods, prior to intervention onset.
Figure 6 shows these histograms for a number of interventions.
We see consistent assignment of importance across all methods.
This means they associate the same inﬂuential signals to the same medical intervention.
[Figure]
4. G REENHOUSE G AS (GHG) O BSERVING N ETWORK D ATA S ET
This experiment evaluates the utility of our method for attributing importance to GHG tracers across different locations in California.
The GHG data consists of 15 time series signals from 2921 grid cells.
A target time series is a synthetic signal of GHG concentrations.
We use an RNN model to estimate
GHG concentrations using all tracers.
Evaluating which tracers are most useful in reconstructing this synthetic signal can be posed as a feature importance problem for weather modeling over time
[Table]
In order to quantitatively evaluate the proposed method on real data
[Table]
we evaluate how well the method performs at selecting globally relevant methods as a proxy.
We aggregate the importance of all features over time (and training samples) and retrain the black–box by i) removing toprelevant features as indicated by each method ii) using top 3 relevant features only .
The performance summary is provided in Table 3 suggesting that among methods that derive instance wise feature importance over time, FFC also generates reasonable global relevance of features.
Results for both
MIMIC-III and GHG datasets are summarized in Table 3.
[Table]
4. S ANITY C HECK T EST FOR E XPLANATIONS
We additionally evaluate the quality of the proposed FFC method using the randomization tests proposed as ‘Sanity Checks’ in Adebayo et al.
Two randomization tests are designed to test for sensitivity of the explanations to i) the black–box model parameters using a model parameter randomization test , and ii) sensitivity to data labels using a using a data randomization test for Simulation Data II.
Data Parameter Randomization Test:
This test evaluates how different explanations are when the black–box model is trained on permuted labels (breaking the correlation between features and output label If explanations truly rely on the
output
labels, as
suggested in our deﬁnition, then the explanation quality should differ signiﬁcantly when a
model trains
on
permuted labels.
We evaluate the (cid:96) 2 distance between the generated
explanations
along with the AUROC and AUPRC
of the generated explanations Model Parameter Randomization Test:
This test evaluates how different explanation quality is when the parameters of the model are arbitrarily shufﬂed.
Signiﬁcant differences in generated explanations suggests the proposed method is sensitive to black-box model parameters.
In Adebayo et al.
, these tests are conducted for saliency map methods for images by evaluating the distance between saliency maps for different randomizations.
In our evaluation, we not only measure the (cid:96)distance between the generated explanations, but also in comparison with ground–truth explanations.
The results are included for Simulated Data II, as ground–truth explanations are available.
The results of both tests are included in Table 4.
They indicate the drops in explanation performance for both randomization tests.
The performance of the model used for model randomization test drops to 0.52 AUROC as opposed to 0.95 for the original trained model on this simulated task (Simulation Data II).
For data randomization, performance of the model drops to 0.62 from 0.95 in terms of AUROC.
AUROCs and AUPRCs for FFC drop the most, suggesting the FFC explanation method is sensitive to perturbations in output labels (as tested by the data randomization test) and to randomization in model parameters.
Signiﬁcant deterioration compared to explanation performance in Table 2 (for Simulation Data II) indicates that the proposed method passes the sanity checks.
D ISCUSSION
We propose a new deﬁnition for obtaining sample-based feature importance for high-dimensional time series data.
We evaluate the importance of each feature at every time point, to locate highly important observations.
We deﬁne important observations as those that cause the biggest change in model output had they been different from the actual observation.
This counterfactual observation is generated by modeling the conditional distribution of the underlying data dynamics.
We propose a generative model to sample such counterfactuals.
We evaluate and compare the proposed deﬁnition and algorithm to several existing approaches.
We show that our method is better at localizing important observations over time.
This is one of the ﬁrst methods that provides individual feature importance over time.
Future extension to this work will include analysis on real datasets annotated with feature importance explanations.
The method will also be extended to evaluate change in risk based on most relevant subsets of observations.
A A PPENDIX A. S IMULATED D ATA I
To simulate these data, we generate d = 3 (independent) sequences as a standard non–linear autoregressive moving average (NARMA) time series.
Note also that we add linear trends to featuresand 2 of the form: x ( t + 1) = 0 .
5 x ( t )
+ 0 .
5 x ( t ) (cid:80) l − 1 i =
x ( t − l ) + 1 .
5 u ( t − ( l − 1))
u ( t )
+ 0 .
5 + α d t for t ∈ [80] , α > 0 ( 0 .
065 for feature 2 and 0 .
003 for feature 1 ), and where the order l = 2 , u ∼ Normal (0 , 0 . 03) .
We additionally add linear trends to features.
We add spikes to each sample (uniformly at random over time) and for every feature
d following the procedure below:
[Equation]
where κ > 0 indicates the additive spike.
The label y t = 1 ∀ t > t 1 , where t 1 = min g d , i.e. the label changes to 1 when a spike is encountered in the ﬁrst feature and is 0 otherwise.
We sample our time series using the python TimeSynth 1 package.
Number of samples generated: 10000 (80%,20% split).
A.1. A DDITIONAL S AMPLES
[Figure]
i m p o r t a n c e
[Figure]
i
[Figure]
o r t a n
[Figure]
[Figure]
i m p o r t a n c e SA (a)
[Figure]
A. S IMULATED D ATA II
This simulated data is a two state HMM ( 2 states) with initial state π =
Transition probability T being:
T = (cid:20) 0 .
1 0 .(cid:21)
The emission probability in each state is a multivariate Gaussian: N ( µ 1 , Σ 1 ) and N ( µ 1 , Σ 1 ) where µ 1 =
[1 . 2 , 1 . 5 , 0 . 8] and µ 2 =
[ − 1 . 2 , − 0 .
8 , − 1 . 5] .
Marginal variance for all features in each state is 0 .
8 with only features 1 and 2 being correlated ( Σ 12 = Σ 21 = 0 .
in state 1 and only 0 and 2 om state 2 ( Σ 02 = Σ
The output y t at every step is assigned using the logit in 3.
Depending on the hidden state at time t , only one of the features contribute to the output and is deemed inﬂuential to the output.
[Equation]
A.2. D ERIVATION OF p ∗ ( x i,t | X 0: t − 1 )
The true conditional distribution can be derived using the forward algorithm (Bishop, 2006) as follows:
[Equation]
where,
[Equation]
where p ( s t − 1 | X 0: t − 1 ) is estimated using the forward algorithm.
J OINT C ONDITIONAL G ENERATIVE M ODEL
Our generator G i is trained using an RNN (GRU).
We model the latent state z t with a multivariate
Gaussian with diagonal covariance and observations with a multivariate Gaussian with full covariance.
Parameter setting of the generator for each of the experiments are shown in tables below.
[Table]
Software used: Python 3.7.3 , Pytorch 1.0.1.post
GPU Info: Quadro
CPU Info: Intel
[Table]
[Table] [Table]
) CPU
[Table] [Table]
[Table]
.50GHz
The counterfactual for observation i at time t can now be sampled by marginalizing over other features at time t .
i.e, x i,t ∼ (cid:80) x − i p (ˆ x | X 0: t − 1 ) .
A. MIMIC-III M ORTALITY E XPERIMENT
Feature selection and data processing: For this experiment, we select adult ICU admission data from the MIMIC dataset.
We use static patients’ static, vital measurements and lab result for the analysis.
The task is to predict 48 hour mortality based on 48 hours of clinical data, therefor we remove samples with less than 48 hours of data.
Table 6 presents a full list of clinical measurements used in this experiment.
The predictor model takes in new measurements every hour, and updates the mortality risk.
We quantize the time series data to hour blocks by averaging existing measurements within each hour block.
We use 2 approches for imputing missing values: 1) Mean imputatiopn for vital signals using the sklearn SimpleImputer 2 , 2) forward imputation for lab results, where we keep the value of the last lab measurement until a new value is evaluated.
We also removed patients who had all 48 quantized measurements missing for a speciﬁc feature.
Overall, 22,988 ICU admissions were extracted and training process was on a 65%,15%,20% train, validation, test set respectively.
[Table]
Parameter Settings for mortality risk predictor model:
The risk predictor model is a recurrent network with GRU cells.
All features are scaled to 0 mean, unit variance and the target is a probability score ranging [0 , 1] .
The model achieves 0 .
007) AUC on test set classiﬁcation task.
Detailed speciﬁcation of the model are presented in Table 7.
[Table]
[Figure]
Parameter Settings for conditional Generator: The recurrent network with speciﬁcations show in 8 learns a hidden latent vector h t representing the history.
h t is then concatenated with x − i,t and fed into a non-linear 1-layer MLP to model the conditional distribution p ( x i , t | X 0: t − 1 ) .
Additional importance plots are provided in Figure 9.
[Table]
[Figure]
[Table]
[Figure]
[Table]
encoding size
[Table]
m
[Table]
[Figure]
Regressor encoding size ( m )
[Figure]
Loss
[Figure]
Gradient Algorithm
Adam (learning rate = 0 . 0001 , β 1 = 0 .
9 , β 2 = 0 .
999 , weight decay = 0 )
[Figure]
Accordance testing: For this test we look into how much different baselines agree on important feature assignment.
As we observed from the experiments, counterfactual methods mostly agree on the most important features for individual samples.
We deﬁne accordance score between 2 methods as the percentage of top n signals both identiﬁed as important.
A score of 80 means on average over the test data, 80 of the assignments were similar.
This is depicted in Figure 10.
[Figure]
A. R UN - TIME ANALYSIS
In this section we compare the run-time across multiple baselines.
Table 9 shows inference runtime for all the baseline methods on a machine with Quadro 400 GPU and Intel(R)
Xeon(R) CPU E5v4 @
3.50GHz CPU.
The runtime for the counterfactual approaches (FFC, FO and AFO) depends only on the length of the time series.
It is also the case for FFC since the conditional generator models the joint distribution of all features.
This is an advantage, over approaches like LIME, the runtime depends both on the length of the signal as well as the number of features.
Overall, FFC performs reasonably compared to ad-hoc counterfactual approaches, since inference on the RNN conditional generator is efﬁcient.
This is one of the reasons that the RNN generator model is used to approximate the conditional distribution.
[Table]
A. GHG N ETWORK D ATA
Parameter Settings for Generator: The settings are provided in Table 10.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Parameter Settings for Black-Box: This black box regresses d = 15 tracer time signals to the target synthetic GHG time series for t = 327 time points.
This model is trained using a 65%,15%,20% train, validation, test set respectively.
All features are scaled to 0 mean unit variance and the target
[Figure]
[Table]
[Figure]
Encoding
[Figure]
[Table]
[Figure]
Regressor Activation
[Figure]
Gradient Algorithm
Adam (learning rate = 0 .
001 , β 1 = 0 .
9 , β 2 = 0 .
999 , weight decay = 0 )
(counterfactual) generators, while Figure 11 shows the training loss of black-box that was used to present feature important results in Section 4.4.