Original: T EMPORAL D EPENDENCIES IN F EATURE I MPORTANCE FOR T IME S ERIES P REDICTION
Revised: T EMPORAL D EPENDENCIES IN F EATURE I MPORTANCE FOR T IME S ERIES P REDICTION

Original: Anonymous authors
Revised: Meta Saba Zuberi

Original: Paper under double-blind review
Revised: Layer 6 AI Maksims Volkovs Layer 6 AI

Original: A BSTRACT
Revised: A BSTRACT

Original: Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time.
Revised: Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time.

Original: In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues.
Revised: In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues.

Original: Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score.
Revised: Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score.

Original: Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps.
Revised: Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps.

Original: We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods and explore the impact of various evaluation strategies.
Revised: We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods and explore the impact of various evaluation strategies.

Original: Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics
Revised: Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reliably explaining predictions of machine learning models is important given their wide-spread use.
Revised: Reliably explaining predictions of machine learning models is important given their wide-spread use.

Original: Explanations provide transparency and aid reliable decision making, especially in domains such as finance and healthcare, where explainability is often an ethical and legal requirement (Amann et al., 2020; Prenio & Yong, 2021).
Revised: Explanations provide transparency and aid reliable decision making, especially in domains such as finance and healthcare, where explainability is often an ethical and legal requirement (Amann et al., 2020; Prenio & Yong, 2021).

Original: Multivariate time series data is ubiquitous in these sensitive domains, however explaining time series models has been relatively under explored.
Revised: Multivariate time series data is ubiquitous in these sensitive domains, however explaining time series models has been relatively under explored.

Original: In this work we focus on saliency methods, a common approach to explainability that provides explanations by highlighting the importance of input features to model predictions (Baehrens et al., 2010; Mohseni et al., 2020).
Revised: In this work we focus on saliency methods, a common approach to explainability that provides explanations by highlighting the importance of input features to model predictions (Baehrens et al., 2010; Mohseni et al., 2020).

Original: It has been shown that standard saliency methods underperform on deep learning models used in the time series domain (Ismail et al., 2020).
Revised: It has been shown that standard saliency methods underperform on deep learning models used in the time series domain (Ismail et al., 2020).

Original: In time series data, observations of the same feature at different points in time are typically related and their order matters.
Revised: In time series data, observations of the same feature at different points in time are typically related and their order matters.

Original: Methods that aim to highlight important observations but treat them as independent face significant limitations.
Revised: Methods that aim to highlight important observations but treat them as independent face significant limitations.

Original: Furthermore, it is important to note that the same observation of a feature can vary in importance to predictions at different times, which we refer to as the temporal dependency in feature importance.
Revised: Furthermore, it is important to note that the same observation of a feature can vary in importance to predictions at different times, which we refer to as the temporal dependency in feature importance.

Original: For example, there can be a delay between important feature shifts and a change in the model’s predictions.
Revised: For example, there can be a delay between important feature shifts and a change in the model’s predictions.

Original: These temporal dynamics can be difficult for current explainability methods to capture.
Revised: These temporal dynamics can be difficult for current explainability methods to capture.

Original: To address these challenges, we propose the Win dowed Feature I mportance in T ime (WinIT), a feature removal based method which determines the importance of a given observation to the predictions over a time window.
Revised: To address these challenges, we propose the Win dowed Feature I mportance in T ime (WinIT), a feature removal based method which determines the importance of a given observation to the predictions over a time window.

Original: Feature removal-based methods generate importance by measuring the change in outcomes when a particular feature is removed.
Revised: Feature removal-based methods generate importance by measuring the change in outcomes when a particular feature is removed.

Original: However, removing a feature at a particular time does not account for the dependence between it and future observations of the same feature.
Revised: However, removing a feature at a particular time does not account for the dependence between current and future observations of the same feature.

Original: WinIT addresses this by assigning the importance of a feature at a specific time based on a difference of two scores, that each take the temporal dependence of the subsequent observations into account.
Revised: WinIT addresses this by assigning the importance of a feature at a specific time based on a difference of two scores, that each take the temporal dependence of the subsequent observations into account.

Original: To capture the varying importance of the same feature observation to predictions at different times, WinIT aggregates the importance to predictions over a window of time steps to generate its final score.
Revised: To capture the varying importance of the same feature observation to predictions at different times, WinIT aggregates the importance to predictions over a window of time steps to generate its final score.

Original: This allows our approach to identify important features that have delayed impact on the predictive distribution, thus better capturing temporal dynamics in feature importance.
Revised: This allows our approach to identify important features that have delayed impact on the predictive distribution, thus better capturing temporal dynamics in feature importance.

Original: It is well known that the evaluation of explainability methods is challenging because no ground truth identification of important features exists for real world data (Doshi-Velez & Kim, 2017).
Revised: It is well known that the evaluation of explainability methods is challenging because no ground truth identification of important features exists for real world data (Doshi-Velez & Kim, 2017).

Original: A common approach is to evaluate the impact of removing features that are highlighted as important by the explainability method (Lundberg et al., 2020) by masking them with a prior value.
Revised: A common approach is to evaluate the impact of removing features that are highlighted as important by the explainability method (Lundberg et al., 2020) by masking them with a prior value.

Original: For time series data, an additional challenge for evaluation is that since observations of the same feature are related, removing the information from an important observation is non-trivial.
Revised: For time series data, an additional challenge for evaluation is that since observations of the same feature are related, removing the information from an important observation is non-trivial.

Original: This fact is under-explored in prior work.
Revised: This fact is under-explored in prior work.

Original: We present a detailed investigation of masking strategies to remove important observations and demonstrate their large impact on explainability performance evaluation.
Revised: We present a detailed investigation of masking strategies to remove important observations and demonstrate their large impact on explainability performance evaluation.

Original: We present several masking strategies with complementary properties that we propose should be part of a comprehensive evaluation scheme.
Revised: We present several masking strategies with complementary properties that we propose should be part of a comprehensive evaluation scheme.

Original: In summary, our main contributions are:
Revised: In summary, our main contributions are:

Original: • We propose a new time series feature importance method that accounts for the temporal dependence between different observations of the same feature and computes the impact of a feature observation to predictions over a window of time steps.
Revised: • We propose a new time series feature importance method that accounts for the temporal dependence between different observations of the same feature and computes the impact of a feature observation to predictions over a window of time steps.

Original: • We conduct a detailed investigation of evaluation approaches for time series model explainability and present several complementary masking strategies that we propose should be part of a robust evaluation scheme.
Revised: • We conduct a detailed investigation of evaluation approaches for time series model explainability and present several complementary masking strategies that we propose should be part of a robust evaluation scheme.

Original: • We expand synthetic datasets from Tonekaboni et al.
Revised: • We expand synthetic datasets from Tonekaboni et al.

Original: (2020) to explicitly evaluate the ability of explainability methods to capture shifted temporal dependencies.
Revised: (2020) to explicitly evaluate the ability of explainability methods to capture shifted temporal dependencies.

Original: • We conduct extensive experiments and demonstrate that our approach leads to a significant improvement in explanation performance on real-world data, and is more stable under different evaluation settings.
Revised: • We conduct extensive experiments and demonstrate that our approach leads to a significant improvement in explanation performance on real-world data, and is more stable under different evaluation settings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A wide range of saliency methods have been proposed in the literature.
Revised: A wide range of saliency methods have been proposed in the literature.

Original: These include gradient based methods such as Integrated Gradients (Sundararajan et al., 2017), GradientSHAP (Erion et al., 2020), Deep-LIFT (Shrikumar et al., 2017), and DeepSHAP (Lundberg & Lee, 2017), which leverage gradients of model predictions with respect to input features to generate importance scores.
Revised: These include gradient based methods such as Integrated Gradients (Sundararajan et al., 2017), GradientSHAP (Erion et al., 2020), Deep-LIFT (Shrikumar et al., 2017), and DeepSHAP (Lundberg & Lee, 2017), which leverage gradients of model predictions with respect to input features to generate importance scores.

Original: Feature removal based methods such as feature occlusion (Zeiler & Fergus, 2014) and feature ablation (Suresh et al., 2017) are model-agnostic methods which remove a feature from the imput data and measure the changes in model predictions.
Revised: Feature removal based methods such as feature occlusion (Zeiler & Fergus, 2014) and feature ablation (Suresh et al., 2017) are model-agnostic methods which remove a feature from the input data and measure the changes in model predictions.

Original: Model based saliency methods, such as Choi et al.
Revised: Model based saliency methods, such as Choi et al.

Original: (2016); Song et al.
Revised: (2016); Song et al.

Original: Xu et al.
Revised: Xu et al.

Original: ; Kaji et al.
Revised: ; Kaji et al.

Original: (2019) for attention-based models, use the model architecture, in this case the attention layers, to generate importance scores.
Revised: (2019) for attention-based models, use the model architecture, in this case the attention layers, to generate importance scores.

Original: However, when applied to time series models, these explainability methods do not directly consider the temporal nature of the problem and have been shown to underperform (Ismail et al., 2020).
Revised: However, when applied to time series models, these explainability methods do not directly consider the temporal nature of the problem and have been shown to underperform (Ismail et al., 2020).

Original: In contrast there has been little work on saliency methods for time series.
Revised: In contrast there has been little work on saliency methods for time series.

Original: TSR (Ismail et al., 2020) separates the importance calculation along the time and feature input dimensions to improve performance.
Revised: TSR (Ismail et al., 2020) separates the importance calculation along the time and feature input dimensions to improve performance.

Original: FIT (Tonekaboni et al., 2020) measures each observation’s importance to the prediction change at the same time step using a KL-divergence based score.
Revised: FIT (Tonekaboni et al., 2020) measures each observation’s importance to the prediction change at the same time step using a KL-divergence based score.

Original: Dynamask (Crabb ´ e & van der
Revised: Dynamask (Crabb ´ e & van der

Original: Schaar, 2021) is a perturbation method that learns a mask of the input feature matrix to highlight important observations.
Revised: Schaar, 2021) is a perturbation method that learns a mask of the input feature matrix to highlight important observations.

Original: We propose a new feature removal based saliency method that explicitly accounts for the temporal nature of the data in a novel way.
Revised: We propose a new feature removal based saliency method that explicitly accounts for the temporal nature of the data in a novel way.

Original: To understand how our work relates to other feature removal based methods we can utilize the unified framework presented in Covert et al.
Revised: To understand how our work relates to other feature removal based methods we can utilize the unified framework presented in Covert et al.

Original: (2021) that categorizes feature removal based methods into three dimensions: feature removal method, model behaviour explained and summarization method.
Revised: (2021) that categorizes feature removal based methods into three dimensions: feature removal method, model behaviour explained and summarization method.

Original: Approaches such as feature occlusion (Zeiler & Fergus, 2014), feature ablation (Suresh et al., 2017) and RISE (Petsiuk et al., 2019) replace the removed features with a specific baseline value.
Revised: Approaches such as feature occlusion (Zeiler & Fergus, 2014), feature ablation (Suresh et al., 2017) and RISE (Petsiuk et al., 2019) replace the removed features with a specific baseline value.

Original: Like us, approaches such as FIT (Tonekaboni et al., 2020) and FIDO-CA (Chang et al., 2019) use a generator to replace the removed features.
Revised: Like us, approaches such as FIT (Tonekaboni et al., 2020) and FIDO-CA (Chang et al., 2019) use a generator to replace the removed features.

Original: In terms of model behaviour explained, like most methods we aim to explain the model prediction before and after feature removal, while methods such as INVASE (Yoon et al., 2019) measure the change in the loss instead.
Revised: In terms of model behaviour explained, like most methods we aim to explain the model prediction before and after feature removal, while methods such as INVASE (Yoon et al., 2019) measure the change in the loss instead.

Original: The summarization method of an explainability approach describes how the importance score is generated.
Revised: The summarization method of an explainability approach describes how the importance score is generated.

Original: Feature occlusion uses the L1 norm of the difference in model predictions, which does not account for the temporal nature of the problem, while FIT uses the KL-divergence between the prediction with and without a feature at the same time step.
Revised: Feature occlusion uses the L1 norm of the difference in model predictions, which does not account for the temporal nature of the problem, while FIT uses the KL-divergence between the prediction with and without a feature at the same time step.

Original: Our summarization method differs from these approaches by accounting for the importance of a feature observation over multiple time steps, improving our ability to capture temporal patterns in feature importance.
Revised: Our summarization method differs from these approaches by accounting for the importance of a feature observation over multiple time steps, improving our ability to capture temporal patterns in feature importance.

Original: [Figure]
Revised: [Figure]

Original:  O UR A PPROACH : C APTURING T EMPORAL D EPENDENCIES IN F EATURE I MPORTANCE
Revised:  O UR A PPROACH : C APTURING T EMPORAL D EPENDENCIES IN F EATURE I MPORTANCE

Original: In this section we introduce our approach WinIT.
Revised: In this section we introduce our approach WinIT.

Original: We assume that a multivariate time series has been observed up to time step T and a black-box model, f θ , has generated a predictive distribution at each time step.
Revised: We assume that a multivariate time series has been observed up to time step T and a black-box model, f θ , has generated a predictive distribution at each time step.

Original: The change in predictions over time represents an updated state of the model as new data becomes available.
Revised: The change in predictions over time represents an updated state of the model as new data becomes available.

Original: WinIT assigns importance by summarizing how a given observation has impacted model predictions over time.
Revised: WinIT assigns importance by summarizing how a given observation has impacted model predictions over time.

Original: We denote X ∈ R D
Revised: We denote X ∈ R D

Original: × T as a sample of a multi-variate time series with D features and T time steps.
Revised: × T as a sample of a multi-variate time series with D features and T time steps.

Original: We also let x t := X · ,t ∈ R D be the set of all feature observations at a particular time 1 ≤ t ≤ T and X 1: t :=
Revised: We also let x t := X · ,t ∈ R D be the set of all feature observations at a particular time 1 ≤ t ≤ T and X 1: t :=

Original: [ x 1 ; x 2 ; . .
Revised: [ x 1 ; x 2 ; . .

Original: ; x t ] ∈ R D × t .
Revised: ; x t ] ∈ R D × t .

Original: Let y t ∈ { 1 . . .
Revised: Let y t ∈ { 1 . . .

Original: K } be the label at each time step for a classification task with K classes.
Revised: K } be the label at each time step for a classification task with K classes.

Original: The model, f θ , estimates the conditional distribution p ( y t | X 1: t ) at each time step t .
Revised: The model, f θ , estimates the conditional distribution p ( y t | X 1: t ) at each time step t .

Original: Let S ⊆ { 1 , . . .
Revised: Let S ⊆ { 1 , . . .

Original: , D } be a subset of features of interest, S c its complement, and x St be the observations of that subset at time t .
Revised: , D } be a subset of features of interest, S c its complement, and x St be the observations of that subset at time t .

Original: A given observation x St can impact predictions at all subsequent time steps.
Revised: A given observation x St can impact predictions at all subsequent time steps.

Original: Our main goal is to generate an importance score for each observation that summarizes its impact on the following prediction time steps.
Revised: Our main goal is to generate an importance score for each observation that summarizes its impact on the following prediction time steps.

Original: We denote I ( S, a, b ) as the importance of the feature set S at time t = a on the prediction at time t = b , where b < a + N is bounded above by a window size N .
Revised: We denote I ( S, a, b ) as the importance of the feature set S at time t = a on the prediction at time t = b , where b < a + N is bounded above by a window size N .

Original: To compute the total importance for S at a , we aggregate I ( S, a, b ) across the window N .
Revised: To compute the total importance for S at a , we aggregate I ( S, a, b ) across the window N .

Original: 3. I MPORTANCE S CORE F ORMULATION
Revised: 3. I MPORTANCE S CORE F ORMULATION

Original: To define I ( S, a, b ) we first determine the importance i ( S ) b a to the prediction at time t
Revised: To define I ( S, a, b ) we first determine the importance i ( S ) b a to the prediction at time t

Original: = b of S over multiple time steps a . . .
Revised: = b of S over multiple time steps a . . .

Original: b , which we define in Eq.
Revised: b , which we define in Eq.

Original: 2 using a feature removal based approach.
Revised: 2 using a feature removal based approach.

Original: The importance of S at a single time step t = a on the prediction at t = b is then given by:
Revised: The importance of S at a single time step t = a on the prediction at t = b is then given by:

Original: [Equation]
Revised: [Equation]

Original: where N is the window size.
Revised: where N is the window size.

Original: Since there is a dependence between subsequent observations of the same feature, eg.
Revised: Since there is a dependence between subsequent observations of the same feature, eg.

Original: x Sa and x Sa +1 , in order to isolate the importance of feature set S at a on the prediction at b we can not simply remove a single observation, and instead we must account for the importance of subsequent observations of S .
Revised: x Sa and x Sa +1 , in order to isolate the importance of feature set S at a on the prediction at b we can not simply remove a single observation, and instead we must account for the importance of subsequent observations of S .

Original: This is done by formulating the importance in Eq.
Revised: This is done by formulating the importance in Eq.

Original: 1 as a difference between the importance over the time intervals [ a, b ] and [ a + 1 , b ] .
Revised: 1 as a difference between the importance over the time intervals [ a, b ] and [ a + 1 , b ] .

Original: This is represented pictorially in Figure 1a.
Revised: This is represented pictorially in Figure 1a.

Original: In our formulation, the importance i ( S ) ba over multiple time steps in Eq. 1 can be obtained using a variety of feature removal based importance methods (Covert et al., 2021).
Revised: In our formulation, the importance i ( S ) ba over multiple time steps in Eq. 1 can be obtained using a variety of feature removal based importance methods (Covert et al., 2021).

Original: Let X 1: b denote the input time series when all features from time steps 1 . . .
Revised: Let X 1: b denote the input time series when all features from time steps 1 . . .

Original: b are known and ˜X Sa : b :=
Revised: b are known and ˜X Sa : b :=

Original: X 1: a − 1 , X S c a : b denote the input features from 1 . . .
Revised: X 1: a − 1 , X S c a : b denote the input features from 1 . . .

Original: b when the feature set S is unknown at t ∈ [ a, b ] .
Revised: b when the feature set S is unknown at t ∈ [ a, b ] .

Original: We focus on approaches that evaluate the impact of feature removal on the model prediction and consider the following measures to compare the two distributions P = p ( y b | X 1: b ) and Q = p ( y b | ˜X
Revised: We focus on approaches that evaluate the impact of feature removal on the model prediction and consider the following measures to compare the two distributions P = p ( y b | X 1: b ) and Q = p ( y b | ˜X

Original: Sa : b ) :
Revised: Sa : b ) :

Original: [Equation]
Revised: [Equation]

Original: where M = P + Q 2 .
Revised: where M = P + Q 2 .

Original: Here PD, KL and JS denote prediction difference, KL divergence and JensenShannon divergence respectively.
Revised: Here PD, KL and JS denote prediction difference, KL divergence and JensenShannon divergence respectively.

Original: This is represented pictorially in Figure 1b, where g denotes the measure used to compare the distributions P and Q .
Revised: This is represented pictorially in Figure 1b, where g denotes the measure used to compare the distributions P and Q .

Original: Prediction difference is similar to methods such as Feature Occlusion (Suresh et al., 2017), while KL based scores are used in Tonekaboni et al.
Revised: Prediction difference is similar to methods such as Feature Occlusion (Suresh et al., 2017), while KL based scores are used in Tonekaboni et al.

Original: ; Yoon et 
Revised: ; Yoon et al

Original: (2019), however these formulations are different to that in Eq.
Revised: (2019), however these formulations are different to that in Eq.

Original: Based on empirical evaluation, we adopt Prediction Difference for i ( S ) ba in our approach and present details of the empirical comparison in Section 5.
Revised: Based on empirical evaluation, we adopt Prediction Difference for i ( S ) ba in our approach and present details of the empirical comparison in Section 5.

Original: To gain further intuition into the formulation of our importance score, we consider different scenarios depicted in Figure 2.
Revised: To gain further intuition into the formulation of our importance score, we consider different scenarios depicted in Figure 2.

Original: In our toy example, there is a signal in the feature set S at time step t = a 1 that causes the prediction to change at a future time step t = b .
Revised: In our toy example, there is a signal in the feature set S at time step t = a 1 that causes the prediction to change at a future time step t = b .

Original: For t = a 0 < a 1 , both i ( S ) ba 

Revised: For t = a 0 < a 1 , both i ( S ) ba

Original: and i ( S ) ba 

Revised: and i ( S ) ba

Original: +1 would be large as the signal is contained in both intervals [ a 0 , b ] and [ a 0 + 1 , b ] .
Revised: +1 would be large as the signal is contained in both intervals [ a 0 , b ] and [ a 0 + 1 , b ] .

Original: For t = a 2 > a 1 , both i ( S ) ba 2 and i ( S ) ba 2 +1 would be small as the signal is not contained in either [ a 2 , b ] or [ a 2 + 1 , b ] .
Revised: For t = a 2 > a 1 , both i ( S ) ba 2 and i ( S ) ba 2 +1 would be small as the signal is not contained in either [ a 2 , b ] or [ a 2 + 1 , b ] .

Original: Thus in both cases, the importance I ( S, a 0 , b ) and I ( S, a 2 , b ) would be small.
Revised: Thus in both cases, the importance I ( S, a 0 , b ) and I ( S, a 2 , b ) would be small.

Original: In contrast, I ( S, a 1 , b ) would be large as i ( S ) ba 1 is large because the signal is in [ a 1 , b ] while i ( S ) ba 1 +1 is small because the signal is not in [ a 1 +1 , b ] .
Revised: In contrast, I ( S, a 1 , b ) would be large as i ( S ) ba 1 is large because the signal is in [ a 1 , b ] while i ( S ) ba 1 +1 is small because the signal is not in [ a 1 +1 , b ] .

Original: Thus our formulation captures as important the moment of the feature change that would cause the predictions to change.
Revised: Thus our formulation captures as important the moment of the feature change that would cause the predictions to change.

Original: 3. F EATURE R EMOVAL WITH G ENERATIVE M ODEL
Revised: 3. F EATURE R EMOVAL WITH G ENERATIVE M ODEL

Original: To obtain the importance score for WinIT we need to compute p ( y b | ˜X Sa : b ) , where ˜X Sa : b =
Revised: To obtain the importance score for WinIT we need to compute p ( y b | ˜X Sa : b ) , where ˜X Sa : b :=

Original: X 1: a − 1 , X S c a : b .
Revised: X 1: a − 1 , X S c a : b .

Original: This is the predictive distribution where all feature values are known up to time step b , except for feature set S from a to b .
Revised: This is the predictive distribution where all feature values are known up to time step b , except for feature set S from a to b .

Original: Approaches to feature removal include setting features to a default value (Petsiuk et al., 2019), sampling from a fixed distribution (Suresh et al., 2017), or removing features by replacing them with samples from a conditional generative model (Chang et al., 2019;
Revised: Approaches to feature removal include setting features to a default value (Petsiuk et al., 2019), sampling from a fixed distribution (Suresh et al., 2017), or removing features by replacing them with samples from a conditional generative model (Chang et al., 2019;

Original: Tonekaboni et al., 2020).
Revised: Tonekaboni et al., 2020).

Original: We use a generative approach to replace the removed observations with plausible counterfactual alternatives.
Revised: We use a generative approach to replace the removed observations with plausible counterfactual alternatives.

Original: The marginal probability we would like to compute can be written as:
Revised: The marginal probability we would like to compute can be written as:

Original: [Equation]
Revised: [Equation]

Original: (3) where the conditional distribution is evaluated at specific values of X Sa : b and averaged over the distribution of all values of X Sa : b .
Revised: (3) where the conditional distribution is evaluated at specific values of X Sa : b and averaged over the distribution of all values of X Sa : b .

Original: To estimate the distribution of X Sa : b , we use a non-deterministic feature generator G S that uses past observations, from time 1 . . .
Revised: To estimate the distribution of X Sa : b , we use a non-deterministic feature generator G S that uses past observations, from time 1 . . .

Original: a − 1 , to generate a distribution of features S at time a . . .
Revised: a − 1 , to generate a distribution of features S at time a . . .

Original: b for a ≤ b < a + N
Revised: b for a ≤ b < a + N

Original: [Equation]
Revised: [Equation]

Original: Here the second argument in G S is the number of time steps the generator is going to generate, where b = a + n and 0 ≤ n < N .
Revised: Here the second argument in G S is the number of time steps the generator is going to generate, where b = a + n and 0 ≤ n < N .

Original: We sample L times from the distribution in Eq.
Revised: We sample L times from the distribution in Eq.

Original: 4 and average
Revised: 4 and average

Original: to obtain the desired output in Eq. 3.
Revised: to obtain the desired output in Eq. 3.

Original: Details are presented in Algorithm 1 in Appendix A.4.
Revised: Details are presented in Algorithm 1 in Appendix A.4.

Original: 3. A GGREGATION
Revised: 3. A GGREGATION

Original: Our method provides an importance score for a feature subset S at time t = a to the prediction at t = b .
Revised: Our method provides an importance score for a feature subset S at time t = a to the prediction at t = b .

Original: This allows the same feature observation to have varying importance for different prediction time steps, and provides a more granular view of temporal dependencies.
Revised: This allows the same feature observation to have varying importance for different prediction time steps, and provides a more granular view of temporal dependencies.

Original: However, it may also be of interest to provide a single importance score for each time step that summarizes the overall impact of S .
Revised: However, it may also be of interest to provide a single importance score for each time step that summarizes the overall impact of S .

Original: In this case we aggregate the importance scores across the forward prediction time window:
Revised: In this case we aggregate the importance scores across the forward prediction time window:

Original: [Equation]
Revised: [Equation]

Original: In our framework we use I ( S, a ) to compute the importance for feature set S at time point a , and use it to compare WinIT against leading baselines.
Revised: In our framework we use I ( S, a ) to compute the importance for feature set S at time point a , and use it to compare WinIT against leading baselines.

Original:  E XPLAINABILITY E VALUATION IN T IME S ERIES
Revised:  E XPLAINABILITY E VALUATION IN T IME S ERIES

Original: In most real world applications, the true importance of features to a model’s prediction are not known.
Revised: In most real world applications, the true importance of features to a model’s prediction are not known.

Original: A common evaluation strategy for feature importance based explainability methods is to remove important features and measure the difference in the model’s performance (Hooker et al., 2019;
Revised: A common evaluation strategy for feature importance based explainability methods is to remove important features and measure the difference in the model’s performance (Hooker et al., 2019;

Original: Lundberg et al., 2020; Tonekaboni et al., 2020).
Revised: Lundberg et al., 2020; Tonekaboni et al., 2020).

Original: Important features are assumed to be more informative to the model predictions, therefore the more important a removed observation is, the bigger the performance drop.
Revised: Important features are assumed to be more informative to the model predictions, therefore the more important a removed observation is, the bigger the performance drop.

Original: For time-series, the important feature is masked by replacing it with its value at the previous time step or its mean value, for example.
Revised: For time-series, the important feature is masked by replacing it with its value at the previous time step or its mean value, for example.

Original: However, since subsequent observations of the same feature are correlated, we cannot remove the information of an important observation by only masking that single value.
Revised: However, since subsequent observations of the same feature are correlated, we cannot remove the information of an important observation by only masking that single value.

Original: Tonekaboni et al.
Revised: Tonekaboni et al.

Original: (2020) removes the information in an important observation, x St , by masking all the following observations of that feature in the time series, ie.
Revised: (2020) removes the information in an important observation, x St , by masking all the following observations of that feature in the time series, ie.

Original: x St ′ (cid:55)→ ˆ x St ′ =
Revised: x St ′ (cid:55)→ ˆ x St ′ =

Original: x St − 1 ∀ t ′ ≥ t .
Revised: x St − 1 ∀ t ′ ≥ t .

Original: We refer to this ask the END masking strategy.
Revised: We refer to this ask the END masking strategy.

Original: However, when evaluating explainability methods by removing important features using END masking, the actual number of observations masked may be very different depending on where they are in the time series - and thus provide an unfair advantage methods that flag eariler time points as important.
Revised: However, when evaluating explainability methods by removing important features using END masking, the actual number of observations masked may be very different depending on where they are in the time series - and thus provide an unfair advantage methods that flag earlier time points as important.

Original: On the other hand, forcing the number of masked observations to be equal may also introduce bias as information spread over more time steps will be masked less.
Revised: On the other hand, forcing the number of masked observations to be equal may also introduce bias as information spread over more time steps will be masked less.

Original: To this end, we adapt three masking strategies, defined below, to investigate their impact on results: (1) END :
Revised: To this end, we adapt three masking strategies, defined below, to investigate their impact on results: (1) END :

Original: When an observation is declared important, we mask all the subsequent observations of that feature until the end; (2) STD :
Revised: When an observation is declared important, we mask all the subsequent observations of that feature until the end; (2) STD :

Original: When an observation is declared important, we mask all the subsequent observations, until the feature value has changed more than α standard deviations with respect to the first masked observation.
Revised: When an observation is declared important, we mask all the subsequent observations, until the feature value has changed more than α standard deviations with respect to the first masked observation.

Original: (3) STD-BAL : We designate a fixed number of masked observations, n m .
Revised: (3) STD-BAL : We designate a fixed number of masked observations, n m .

Original: Here, we iterate over the most important feature-time ( S, t ) instances (in order of importance) and apply STD until the number of masked observations reaches n m .
Revised: Here, we iterate over the most important feature-time ( S, t ) instances (in order of importance) and apply STD until the number of masked observations reaches n m .

Original: The STD strategy stops masking once the observed value has changed significantly, reflecting the dynamics of the underlying time series.
Revised: The STD strategy stops masking once the observed value has changed significantly, reflecting the dynamics of the underlying time series.

Original: It is therefore more conservative than the END approach, as it will mask fewer observations.
Revised: It is therefore more conservative than the END approach, as it will mask fewer observations.

Original: Furthermore, unlike the END approach, it does not favour explainability approaches that identify important observations earlier in the time series.
Revised: Furthermore, unlike the END approach, it does not favour explainability approaches that identify important observations earlier in the time series.

Original: STD-BAL follows a similar approach to STD, but fixes the number of observations masked.
Revised: STD-BAL follows a similar approach to STD, but fixes the number of observations masked.

Original: By comparing STD and
Revised: By comparing STD and

Original: STD-BAL, we can see the impact on evaluation from fixing the number of important observations to remove versus the number of observations that are masked.
Revised: STD-BAL, we can see the impact on evaluation from fixing the number of important observations to remove versus the number of observations that are masked.

Original: As we will show, the choice of masking strategy can lead to very different conclusions about the performance of competing explainability approaches.
Revised: As we will show, the choice of masking strategy can lead to very different conclusions about the performance of competing explainability approaches.

Original: Given their complementary focus, we propose different masking strategies should be part of a robust evaluation scheme for time series explainability.
Revised: Given their complementary focus, we propose different masking strategies should be part of a robust evaluation scheme for time series explainability.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For all experiments we usea 1-layerrecurrent GRU model for f θ with 200 hidden units.
Revised: For all experiments we use 1-layer GRU models for f θ with 200 hidden units.

Original: We use Adam optimizer with learning rate 10 − 3  and weight decay 10 − 3 to train the model.
Revised: We use Adam optimizer with learning rate 10 − 3 ( 10 − 4 for MIMIC-III) and weight decay 10 − 3 to train the model.

Original: In practice, S can be a single feature or a set selected by prior domain knowledge.
Revised: In practice, S can be a single feature or a set selected by prior domain knowledge.

Original: For example, in the healthcare setting, S can be a subset containing certain lab measurements, or indicators of differentchronic
Revised: For example, in the healthcare setting, S can be a subset containing certain lab measurements, or indicators of different

Original: [Table]
Revised: [Table]

Original:  diseases.
Revised: chronic diseases.

Original: For simplicity, in all experiments we set S
Revised: For simplicity, in all experiments we set S

Original: =
Revised: =

Original: { i } to contain a single feature.
Revised: { i } to contain a single feature.

Original: We also use a 1-layer GRU model for the generator with 50 hidden units, and train it by fitting a Gaussian distribution with diagonal covariance to reconstruct each feature N time steps forward.
Revised: We also use a 1-layer GRU model for the generator with 50 hidden units, and train it by fitting a Gaussian distribution with diagonal covariance to reconstruct each feature N time steps forward.

Original: We use
Revised: We use

Original: Adam optimizer with learning rate 10 − 4 , weight delay 10 − 3 and 300 epochs with early stopping.
Revised: Adam optimizer with learning rate 10 − 4 , weight delay 10 − 3 and 300 epochs with early stopping.

Original: To estimate the predicted distribution with masked features we average L = 3 Monte Carlo samples from the generator, as taking more samples does not improve performance significantly.
Revised: To estimate the predicted distribution with masked features we average L = 3 Monte Carlo samples from the generator, as taking more samples does not improve performance significantly.

Original: We measure stability by splitting the training set into 5 folds and report results averaged across the folds with corresponding standard deviation error bars.
Revised: We measure stability by splitting the training set into 5 folds and report results averaged across the folds with corresponding standard deviation error bars.

Original: To demonstrate that WinIT performs equally well on other common architectures we apply it to a 3-Layer GRU, LSTM, and ConvNet on MIMIC-III and results are shown in Appendix A.6.
Revised: To demonstrate that WinIT performs equally well on other common architectures we apply it to a 3-Layer GRU, LSTM, and ConvNet on MIMIC-III and results are shown in Appendix A.6.

Original: All the results with STD masking are with α = 1 .
Revised: All the results with STD masking are with α = 1 .

Original: All experiments were performed with 40 Intel Xeon CPU@2.20GHz cores and Nvidia Titan V GPU.
Revised: All experiments were performed with 40 Intel Xeon CPU@2.20GHz cores and Nvidia Titan V GPU.

Original: We compare our approach against leading baselines: Gradient-based methods .
Revised: We compare our approach against leading baselines: Gradient-based methods .

Original: This includes methods that use the gradient of the prediction with respect to the feature-time input.
Revised: This includes methods that use the gradient of the prediction with respect to the feature-time input.

Original: We compare against
Revised: We compare against

Original: Deep-LIFT (Shrikumar et al., 2017), DeepSHAP (Lundberg & Lee, 2017) and Integrated Gradients (IG) (Sundararajan et al., 2017).
Revised: Deep-LIFT (Shrikumar et al., 2017), DeepSHAP (Lundberg & Lee, 2017) and Integrated Gradients (IG) (Sundararajan et al., 2017).

Original: For these comparisons we make use of the implementation provided in the Captum library (Kokhlikyan et al., 2020).
Revised: For these comparisons we make use of the implementation provided in the Captum library (Kokhlikyan et al., 2020).

Original: Feature occlusion (FO) (Suresh et al., 2017)
Revised: Feature occlusion (FO) (Suresh et al., 2017)

Original: Feature importance is assigned based on the difference in prediction when each feature i is replaced with a random sample from the uniform distribution.
Revised: Feature importance is assigned based on the difference in prediction when each feature i is replaced with a random sample from the uniform distribution.

Original: We also include Augmented Feature Occlusion (AFO) which was introduced in Tonekaboni et al.
Revised: We also include Augmented Feature Occlusion (AFO) which was introduced in Tonekaboni et al.

Original: (2020) to avoid generating out-of-distribution samples.
Revised: (2020) to avoid generating out-of-distribution samples.

Original: Feature Importance in Time (FIT)
Revised: Feature Importance in Time (FIT)

Original: (Tonekaboni et al., 2020) assigns importance to an observation based on the KL-divergence between the predictive distribution and a counterfactual where the rest of the features are unobserved at the last time step.
Revised: (Tonekaboni et al., 2020) assigns importance to an observation based on the KL-divergence between the predictive distribution and a counterfactual where the rest of the features are unobserved at the last time step.

Original: Dynamask (Crabb´e & van der
Revised: Dynamask (Crabb ´ e & van der

Original: Schaar, 2021) is a perturbation-based method for time series data that learns a mask for the input sequence, that is almost binary.
Revised: Schaar, 2021) is a perturbation-based method for time series data that learns a mask for the input sequence, that is almost binary.

Original: 5. S IMULATED DATASET
Revised: 5. S IMULATED DATASET

Original: Dataset Spike is a benchmark experiment presented in Tonekaboni et al.
Revised: Dataset Spike is a benchmark experiment presented in Tonekaboni et al.

Original: (2020) which uses a multivariate dataset composed of 3 random NARMA time series with random spikes.
Revised: (2020) which uses a multivariate dataset composed of 3 random NARMA time series with random spikes.

Original: The label isuntil a spike occurs in the first feature, at which point it changes to 1 for the rest of the sample.
Revised: The label isuntil a spike occurs in the first feature, at which point it changes to 1 for the rest of the sample.

Original: We also add a new delayed version of this dataset, Delayed Spike, where the point at which the label changes to 1 is two time steps after the spike occurs.
Revised: We also add a new delayed version of this dataset, Delayed Spike, where the point at which the label changes to 1 is two time steps after the spike occurs.

Original: In this case, the observation that causes the label to change to 1 is the first spike two time steps before.
Revised: In this case, the observation that causes the label to change to 1 is the first spike two time steps before.

Original: We also include the synthetic benchmark dataset State (Tonekaboni et al., 2020), which uses a non-stationary Hidden Markov Model to generate observations over time and presents a more challenging task.
Revised: We also include the synthetic benchmark dataset State (Tonekaboni et al., 2020), which uses a non-stationary Hidden Markov Model to generate observations over time and presents a more challenging task.

Original: Evaluation Since here we know the exact reason for the positive label, we can use this reason as the ground truth importance to evaluate the explanations.
Revised: Evaluation Since here we know the exact reason for the positive label, we can use this reason as the ground truth importance to evaluate the explanations.

Original: The explanation label at the specific observations that cause the prediction label to change is 1, otherwise it is 0.
Revised: The explanation label at the specific observations that cause the prediction label to change is 1, otherwise it is 0.

Original: We evaluate our feature importance using AUPRC.
Revised: We evaluate our feature importance using AUPRC.

Original: In addition, we introduce the mean rank metric – for each instance we compute the rank of the feature importance score of the positive ground truth observations within that instance.
Revised: In addition, we introduce the mean rank metric – for each instance we compute the rank of the feature importance score of the positive ground truth observations within that instance.

Original: We take the average of this rank across all test instances.
Revised: We take the average of this rank across all test instances.

Original: See Appendix A.5 for further discussion of the metrics.
Revised: See Appendix A.5 for further discussion of the metrics.

Original: Mean rank is especially effective in assessing the performance of the spike and spike-delay datasets, as there is only one positive ground truth observation per instance.
Revised: Mean rank is especially effective in assessing the performance of the spike and spike-delay datasets, as there is only one positive ground truth observation per instance.

Original: [Table]
Revised: [Table]

Original: Results
Revised: Results.

Original: The results on simulated data are presented in Table 1.
Revised: The results on simulated data are presented in Table 1.

Original: We see that WinIT outperforms other methods on the spike and delayed spike datasets in AUPRC and is on par with Dynamask in mean rank.
Revised: We see that WinIT outperforms other methods on the spike and delayed spike datasets in AUPRC and is on par with Dynamask in mean rank.

Original: Unlike WinIT, the performance of other feature removal based methods, FO, AFO and FIT, degrades significantly for the delayed spike experiment compared to spike.
Revised: Unlike WinIT, the performance of other feature removal based methods, FO, AFO and FIT, degrades significantly for the delayed spike experiment compared to spike.

Original: This indicates that the feature removal based importance formulation in WinIT is more robust to delayed importance signals.
Revised: This indicates that the feature removal based importance formulation in WinIT is more robust to delayed importance signals.

Original: Similarly we note that Dynamask and the gradient based approaches are able to adapt to the delayed signal.
Revised: Similarly we note that Dynamask and the gradient based approaches are able to adapt to the delayed signal.

Original: The strong performance on the state dataset also shows WinIT’s robustness to non-stationary signals.
Revised: The strong performance on the state dataset also shows WinIT’s robustness to non-stationary signals.

Original: Saliency maps for the delayed spike experiment are shown in
Revised: Saliency maps for the delayed spike experiment are shown in

Original: Figure 3.
Revised: Figure 3.

Original: WinIT captures the important feature in this setting, while gradient-based method, IG, also highlights subsequent spikes.
Revised: WinIT captures the important feature in this setting, while gradient-based method, IG, also highlights subsequent spikes.

Original: A more complete set of saliency maps are shown in Appendix A.2.
Revised: A more complete set of saliency maps are shown in Appendix A.2.

Original: Note, unlike other methods Dynamask encourages the importance score to be almost binary.
Revised: Note, unlike other methods Dynamask encourages the importance score to be almost binary.

Original: As a result it generates a substantial number of repeated values and this impacts its evaluation on simulated data.
Revised: As a result it generates a substantial number of repeated values and this impacts its evaluation on simulated data.

Original: The results in Table 1 use the trapezoid rule to compute AUPRC, following Crabb´e & van der Schaar (2021).
Revised: The results in Table 1 use the trapezoid rule to compute AUPRC, following Crabb ´ e & van der Schaar (2021).

Original: Without interpolation (Davis & Goadrich, 2006), Dynamask’s AUPRC is much lower ( 0 . 086 ± 0 . {+
+}
Revised: Without interpolation (Davis & Goadrich, 2006), Dynamask’s AUPRC is much lower ( 0 . 038 ± 0 . {+
+}

Original: 001 for the state experiment), while other methods AUPRC values remain similar.
Revised: for the state experiment), while other methods AUPRC values remain similar.

Revised: Similar problem arises for mean rank.

Original: See Appendix A.5 for further analysis.
Revised: See Appendix A.5.2 for further analysis.

Original: 5. MIMIC-III MORTALITY
Revised: 5. MIMIC-III MORTALITY

Original: Dataset MIMIC-III is a multivariate clinical time series dataset with a range of vital and lab measurements taken over time for around 40,000 patients at the Beth Israel Deaconess Medical Center in
Revised: Dataset MIMIC-III is a multivariate clinical time series dataset with a range of vital and lab measurements taken over time for around 40,000 patients at the Beth Israel Deaconess Medical Center in

Original: Boston, MA (Johnson et al., 2016).
Revised: Boston, MA (Johnson et al., 2016).

Original: It is widely used in healthcare and medical AI-related research.
Revised: It is widely used in healthcare and medical AI-related research.

Original: There are multiple tasks associated, including mortality, length-of-stay prediction, and phenotyping (Wang et al., 2020).
Revised: There are multiple tasks associated, including mortality, length-of-stay prediction, and phenotyping (Wang et al., 2020).

Original: We follow the pre-processing procedure described in Tonekaboni et al.
Revised: We follow the pre-processing procedure described in Tonekaboni et al.

Original: and use 8 vitals and 20 lab measurements hourly over a 48-hour period to predict patient mortality.
Revised: and use 8 vitals and 20 lab measurements hourly over a 48-hour period to predict patient mortality.

Original: Evaluation We report performance using both global and local evaluation, as is done in Tonekaboni et al.
Revised: Evaluation We report performance using both global and local evaluation, as is done in Tonekaboni et al.

Original: The global method corresponds to measuring the impact of removing the observations with the highest (top 5% ) importance scores across the test set.
Revised: The global method corresponds to measuring the impact of removing the observations with the highest (top 5% ) importance scores across the test set.

Original: While the local method corresponds to removing the observations with the highest (top K = 50 ) importance scores within each instance.
Revised: While the local method corresponds to removing the observations with the highest (top K = 50 ) importance scores within each instance.

Original: For both experiments we present the metrics AUC drop and average prediction change when important features are masked.
Revised: For both experiments we present the metrics AUC drop and average prediction change when important features are masked.

Original: Average prediction change, defined as the average absolute prediction change, allows us to evaluate the explainability method without relying on the ground truth labels in the dataset.
Revised: Average prediction change, defined as the average absolute prediction change, allows us to evaluate the explainability method without relying on the ground truth labels in the dataset.

Original: Our base model has an AUC of 0 .
Revised: Our base model has an AUC of 0 .

Original: 767 ± 0 .
Revised: 81 ± 0 .

Original: 018 in the test set.
Revised: 003 in the test set.

Original: For the masking method, we use the END masking method from Tonekaboni et al.
Revised: For the masking method, we use the END masking method from Tonekaboni et al.

Original: (2020) for comparison purposes (See Section 4).
Revised: (2020) for comparison purposes (See Section 4).

Original: We explore the impact of other masking strategies in Section 5.3.
Revised: We explore the impact of other masking strategies in Section 5.3.

Original: Results.
Revised: Results.

Original: Our main results are presented in Table 2.
Revised: Our main results are presented in Table 2.

Original: WinIT uses a window size of 10 .
Revised: WinIT uses a window size of 10 .

Original: We can see that
Revised: We can see that

Original: WinIT significantly outperforms all the other methods for both the global Top 5% and local K =experiments.
Revised: WinIT significantly outperforms all the other methods for both the global top 5% and local K =experiments.

Original: WinIT improves over baselines both in terms of identifying observations that impact
Revised: WinIT improves over baselines both in terms of identifying observations that impact

Original: AUC most and the change in prediction.
Revised: AUC most and the change in prediction.

Original: WinIT has a greater than 50% increase in AUC Drop and 17% increase in prediction change over the leading baseline Dynamask on the top 5% experiment.
Revised: WinIT has a 44% increase in AUC Drop and 15% increase in prediction change over the leading baseline Dynamask on the top 5% experiment.

Original: Compared to other feature removal based methods, FO, AFO and FIT, the gain is even larger.
Revised: Compared to other feature removal based methods, FO, AFO and FIT, the gain is even larger.

Original: 5. A DDITIONAL A NALYSIS
Revised: 5. A DDITIONAL A NALYSIS

Original: Impact of masking strategy.
Revised: Impact of masking strategy.

Original: As discussed in Section 4, when evaluating explainability methods by removing features and measuring the impact on performance, the strategy used to mask observations can introduce bias into the evaluation.
Revised: As discussed in Section 4, when evaluating explainability methods by removing features and measuring the impact on performance, the strategy used to mask observations can introduce bias into the evaluation.

Original: In Figure 4, we show results for the K = 50 MIMIC-III mortality task using the masking strategies outlined in Section 4.
Revised: In Figure 4, we show results for the K = 50 MIMIC-III mortality task using the masking strategies outlined in Section 4.

Original: We see that masking has a large impact on the performance.
Revised: We see that masking has a large impact on the performance.

Original: WinIT outperforms other methods across all masking strategies, however the relative performance of gradient versus feature removal based methods, FO, AFO and FIT, changes between the END and STD/STD-BAL methods.
Revised: WinIT outperforms other methods across all masking strategies, however the relative performance of gradient versus feature removal based methods, FO, AFO and FIT, changes between the END and STD/STD-BAL methods.

Original: Although for a given explainability method the AUC drop and prediction change is highest with the END strategy, which tends to mask the most observations, this approach penalizes approaches (such as gradient-based methods) that have a bias towards more recent observations.
Revised: Although for a given explainability method the AUC drop and prediction change is highest with the END strategy, which tends to mask the most observations, this approach penalizes approaches (such as gradient-based methods) that have a bias towards more recent observations.

Original: This can be seen in Figure 5, which shows the counts of masked observations at each time step for different explainability methods 1 .
Revised: This can be seen in Figure 5, which shows the counts of masked observations at each time step for different explainability methods 1 .

Original: We see that the gradient-based method IG masks more recent time steps for all masking strategies, likely due to the vanishing gradients phenomenon (Pascanu et al., 2012).
Revised: We see that the gradient-based method IG masks more recent time steps for all masking strategies, likely due to the vanishing gradients phenomenon (Pascanu et al., 2012).

Original: In contrast the masked observations for methods such as
Revised: In contrast the masked observations for methods such as

Original: AFO, WinIT and Dynamask are more evenly distributed across time steps.
Revised: AFO, WinIT and Dynamask are more evenly distributed across time steps.

Original: STD masking reduces the evaluation bias of END masking against explanation methods that flag features later in the time series.
Revised: STD masking reduces the evaluation bias of END masking against explanation methods that flag features later in the time series.

Original: While STD-BAL makes sure the total number of observation masked are roughly the same across all methods.
Revised: While STD-BAL makes sure the total number of observation masked are roughly the same across all methods.

Original: Given their impact on evaluation, these masking strategies should be used as part of a broader set of evaluation methods in order to make the evaluation of time series explainability methods more robust. {+Window Size AU P RC    
+}
Revised: Given their impact on evaluation, these masking strategies should be used as part of a broader set of evaluation methods in order to make the evaluation of time series explainability methods more robust. {+Window Size AU P RC    
+}

Original: Effect of window size N .
Revised: Window Size M ea n

Revised: R a nk (a)    200 .0 .0 .0 .0 .0 .

Revised: Window Size

Revised: AU C

Revised: D r op

Revised: END

Original:  WinIT assigns importance to observations by aggregating the impact on predictions over a window of subsequent time steps.
Revised: STD STD-BAL (b) Effect of window size N . WinIT assigns importance to observations by aggregating the impact on predictions over a window of subsequent time steps.

Original: As the window size increases, WinIT captures longer range interactions between important signals and changes in predictions, which can lead to better performance since some signals can be heavily delayed.
Revised: As the window size increases, WinIT captures longer range interactions between important signals and changes in predictions, which can lead to better performance since some signals can be heavily delayed.

Original: In Figure 6a we analyze this effect for different window sizes on the delayed spike dataset, where the delay is exactly two time steps.
Revised: In Figure 6a we analyze this effect for different window sizes on the delayed spike dataset, where the delay is exactly two time steps.

Original: As expected we see that as N increases to 3 , the performance, substantially improves.
Revised: As expected we see that as N increases to 3 , the performance, substantially improves.

Original: Note that
Revised: Note that

Original: N = 3 corresponds to looking backward two time steps (plus the current one).
Revised: N = 3 corresponds to looking backward two time steps (plus the current one).

Original: We also see that the performance quickly stabilizes after N = 3 .
Revised: We also see that the performance quickly stabilizes after N = 3 .

Original: A similar pattern can be observed for the MIMIC-III mortality task shown in Figure 6b.
Revised: A similar pattern can be observed for the MIMIC-III mortality task shown in Figure 6b.

Original: We see that the AUC Drop increases with window size and then stabilizes after N = 10 , with a large[Figure] range of choices of the hyper-parameter N leading to similar results.
Revised: We see that the AUC Drop increases with window size and then stabilizes, with a large range of choices of the hyper-parameter N leading to similar results.

Original: We also note that there is a linear relationship between run time and window size, as shown in Appendix A.4, allowing for a reasonable hyper-parameter sweep to find a good choice of window size.
Revised: We also note that there is a linear relationship between run time and window size, as shown in Appendix A.4, allowing for a reasonable hyper-parameter sweep to find a good choice of window size.

Original: This pattern holds for different masking strategies, demonstrating WinIT’s robustness to evaluation strategy.
Revised: This pattern holds for different masking strategies, demonstrating WinIT’s robustness to evaluation strategy.

Original: [Table]
Revised: [Table]

Original: Effects of Different i ( S ) ba .
Revised: Effects of Different i ( S ) ba .

Original: We investigate the impact on performance of using different definitions of the importance to the prediction at t =
Revised: We investigate the impact on performance of using different definitions of the importance to the prediction at t =

Original: b of S over multiple time steps [ a, b ] , i ( S ) ba , defined in Eq. 2.
Revised: b of S over multiple time steps [ a, b ] , i ( S ) ba , defined in Eq. 2.

Original: In Table 3, we show the AUC drop on the MIMIC-III mortality task using STD masking.
Revised: In Table 3, we show the AUC drop on the MIMIC-III mortality task using STD masking.

Original: We see that WinIT-PD gives some gain over the other approaches when dropping the top 50 observations for each time series.
Revised: We see that WinIT-PD gives some gain over the other approaches when dropping the top 50 observations for each time series.

Original: We opt to use the prediction difference in our formulation.
Revised: We opt to use the prediction difference in our formulation.

Original: It is worth noting that all three approaches outperform the other baselines.
Revised: It is worth noting that all three approaches outperform the other baselines.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we propose WinIT, a feature removal based explainability method that produces a feature importance score for each observation in a multivariate time series.
Revised: In this paper we propose WinIT, a feature removal based explainability method that produces a feature importance score for each observation in a multivariate time series.

Original: Unlike prior approaches,
Revised: Unlike prior approaches,

Original: WinIT calculates the importance as the difference of the importance over multiple time steps and aggregates the importance of each observation over a window of predictions.
Revised: WinIT calculates the importance as the difference of the importance over multiple time steps and aggregates the importance of each observation over a window of predictions.

Original: Empirically, WinIT outperforms leading baselines on both synthetic and real-world datasets with more consistent performance under different evaluation strategies.
Revised: Empirically, WinIT outperforms leading baselines on both synthetic and real-world datasets with more consistent performance under different evaluation strategies.

Original: For future work, we would like to tackle two possible limitations in WinIT.
Revised: For future work, we would like to tackle two possible limitations in WinIT.

Original: First, we would like to explore approaches to automatically determine the window size parameter introduced in our method.
Revised: First, we would like to explore approaches to automatically determine the window size parameter introduced in our method.

Original: Secondly we would like to extend to time series explainability approaches that do not take the form of saliency maps, with a single importance for each observation, but which instead focus on identifying important higher-level patterns or trends over a series of observations, as we believe this form of output would further aid interpretability.
Revised: Secondly we would like to extend to time series explainability approaches that do not take the form of saliency maps, with a single importance for each observation, but which instead focus on identifying important higher-level patterns or trends over a series of observations, as we believe this form of output would further aid interpretability.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure the reproducibility of our work we describe the implementation details of our approach and conduct evaluation experiments on publicly available datasets.
Revised: To ensure the reproducibility of our work we describe the implementation details of our approach and conduct evaluation experiments on publicly available datasets.

Original: In addition, we follow the procedures outlined in previous studies to process the data.
Revised: In addition, we follow the procedures outlined in previous studies to process the data.

Original: Upon the acceptance of the manuscript, our work will be publicly available on a GitHub repository, which includes data generation code for simulated datasets and detailed settings for experiments. 
Revised: The code for our work is publicly available at https://github.com/layer6ai-labs/WinIT , which includes the detailed settings for experiments. A CKNOWLEDGMENTS

Original: A link to  our GitHub URL will be added here.
Revised: We would like to thank the anonymous reviewers for their feedback, which helped improve our paper. We would also like to thank Gabriel Loaiza-Ganem for useful discussions. This work was conducted while Clayton Rooke and Jonathan Smith were at Layer 6 AI

Original: A A PPENDIX A. S UMMARY OF N OTATION
Revised: A A PPENDIX A. S UMMARY OF N OTATION

Original: A summary of the notation used throughout the paper is provided in Table 4.
Revised: A summary of the notation used throughout the paper is provided in Table 4.

Original: A. S ALIENCY M APS
Revised: [Table]

Original: Saliency maps are one way of visualizing feature importance and have been found to be helpful for interpreting predictions for image classification in user studies
Revised: Saliency maps are one way of visualizing feature importance and have been found to be helpful for interpreting predictions for image classification in user studies

Original: Alqaraawi et al.
Revised: Alqaraawi et al.

Original: In the multivariate time series setting Shen et al.
Revised: In the multivariate time series setting Shen et al.

Original: (2020) found visualizations can also improve interpretation.
Revised: (2020) found visualizations can also improve interpretation.

Original: Saliency maps for the simulated spike and delayed spike datasets are shown in Figure 7 andrespectively.
Revised: Saliency maps for the simulated spike and delayed spike datasets are shown in Figure 7 andrespectively.

Original: These maps illustrate that WinIT captures the most important features in both the spike and delayed spike setting.
Revised: These maps illustrate that WinIT captures the most important features in both the spike and delayed spike setting.

Original: The gradient-based methods, capture the important observation, but also highlight subsequent spikes that do not impact the change in label.
Revised: The gradient-based methods, capture the important observation, but also highlight subsequent spikes that do not impact the change in label.

Original: We also shows the saliency maps for the MIMIC-III mortality experiments in Figure 9.
Revised: We also shows the saliency maps for the MIMIC-III mortality experiments in Figure 9.

Original: We see that gradient-based methods tend to highlight later observations in the time series as important.
Revised: We see that gradient-based methods tend to highlight later observations in the time series as important.

Original: In contrast FO and AFO tend to flag earlier points in the time series.
Revised: In contrast FO and AFO tend to flag earlier points in the time series.

Original: FIT tends to mark many features within the same time step important, while the salient observations are more distributed in time for
Revised: FIT tends to mark many features within the same time step important, while the salient observations are more distributed in time for

Original: WinIT and Dynamask.
Revised: WinIT and Dynamask.

Original: A. A GGREGATION M ETHODS FOR W IN IT
Revised: A. A GGREGATION M ETHODS FOR W IN IT

Original: In section 3.3, we introduce a method of aggregation to compute the importance of feature set S at timestep t = a by taking the mean of the importance scores across a time window, according to
Revised: In section 3.3, we introduce a method of aggregation to compute the importance of feature set S at timestep t = a by taking the mean of the importance scores across a time window, according to

Original: Eq. 5.
Revised: Eq. 5.

Original: Here we consider an an alternate choice of aggregation, given below:
Revised: Here we consider an an alternate choice of aggregation, given below:

Original: [Equation]
Revised: [Equation]

Original: where ˆ N = max( N − 1 , T − a ) .
Revised: where ˆ N = max( N − 1 , T − a ) .

Original: Table 5 shows the results for STD masking on the MIMIC-III mortality task with different aggregation choices.
Revised: Table 5 shows the results for STD masking on the MIMIC-III mortality task with different aggregation choices.

Original: We see that while the performance for different aggregation methods are similar, taking the mean gives the best results, in terms of both AUC Drop and prediction change.
Revised: We see that while the performance for different aggregation methods are similar, taking the mean gives the best results, in terms of both AUC Drop and prediction change.

Original: One explanation for this is that if an observation is important for multiple future predictions, it is more important than the case when an observation is important for a single future prediction.
Revised: One explanation for this is that if an observation is important for multiple future predictions, it is more important than the case when an observation is important for a single future prediction.

Original: A. A LGORITHM AND R UN TIME A NALYSIS
Revised: A. A LGORITHM AND R UN TIME A NALYSIS

Original: We provide an outline of the WinIT algorithm in Algorithm 1.
Revised: We provide an outline of the WinIT algorithm in Algorithm 1.

Original: We also explore the run time for different window sizes, N , and present our results in Figure 10.
Revised: We also explore the run time for different window sizes, N , and present our results in Figure 10.

Original: We see that the run time has a linear relationship with the window size.
Revised: We see that the run time has a linear relationship with the window size.

Original: Since the importance calculation for each feature is independent in our approach, we are able to parallelize this computation which improves the run time.
Revised: Since the importance calculation for each feature is independent in our approach, we are able to parallelize this computation which improves the run time.

Original: A. E VALUATION OF S IMULATED DATA 
Revised: A. E VALUATION OF S IMULATED DATA A.5. M ETRICS FOR S PIKE AND D ELAYED S PIKE D ATASETS

Original: Metrics for Spike and Delayed Spike Datasets

Original: In this section we provide insight into the metrics used to evaluate the simulated spike and delayed spike datasets.
Revised: In this section we provide insight into the metrics used to evaluate the simulated spike and delayed spike datasets.

Original: These datasets have only one positively labelled observation for each time series and the ground truth explanation is known.
Revised: These datasets have only one positively labelled observation for each time series and the ground truth explanation is known.

Original: Here[Table] we consider AUPRC and mean rank, which we use for evaluation and AUROC, another common metric, which we show is less suitable for this case
Revised: Here we consider AUPRC and mean rank, which we use for evaluation and AUROC, another common metric, which we show is less suitable for this case.

Original: [Table]
Revised: [Table]

Original: For simplicity
Revised: For simplicity

Original: [Table]
Revised: [Table]

Original: we assume all the feature importance values are distinct.
Revised: we assume all the feature importance values are distinct.

Original: Suppose there are n observations in a time series.
Revised: Suppose there are n observations in a time series.

Original: Let 1 ≤ r ≤ n be the rank of the feature importance among all observations in the time series.
Revised: Let 1 ≤ r ≤ n be the rank of the feature importance among all observations in the time series.

Original: AUROC can be interpreted as the probability that the prediction of the positively-labelled observation is greater than that of a negatively labelled observation.
Revised: AUROC can be interpreted as the probability that the prediction of the positively-labelled observation is greater than that of a negatively labelled observation.

Original: As there is 1 positively-labelled 
Revised: As there is 1 positively-labelled observation of rank r in our case, this is the case for n − r out of a total of n − 1 comparisons. Thus the AUROC is n − r n − 1 . [Figure]

Original: [Figure]
Revised: [Figure]

Original: 17: Return I
Revised: Return I

Original: [Figure]
Revised: [Figure]

Original: observation of rank r in our case, this is the case for n − r out of a total of n − 1 comparisons.

Original: Thus the AUROC is n − r n − 1 .

Original: For AUPRC, as there is only 1 positively labelled observation, the recall can only be 1 or 0.
Revised: For AUPRC, as there is only 1 positively labelled observation, the recall can only be 1 or 0.

Original: When the recall is 1, the best precision is 1 /r as the positively-labelled observation is of rank r .
Revised: When the recall is 1, the best precision is 1 /r as the positively-labelled observation is of rank r .

Original: When the recall is 0, if the rank r ̸ = 1 , the precision is also 0.
Revised: When the recall is 0, if the rank r ̸ = 1 , the precision is also 0.

Original: Thus, for r > 1 , there are only 2 points in the precision-recall curve, (0 , 0) and (1 , 1 r ) .
Revised: Thus, for r > 1 , there are only 2 points in the precision-recall curve, (0 , 0) and (1 , 1 r ) .

Original: This means AUPRC is 12 r for r > 1 .
Revised: This means AUPRC is 12 r for r > 1 .

Original: For r = 1 , AUPRC= by definition.
Revised: For r = 1 , AUPRC= by definition.

Original: To summarize, when rank is r , AUROC= n − r n − 1 and AUPRC= 12 r .
Revised: To summarize, when rank is r , AUROC= n − r n − 1 and AUPRC= 12 r .

Original: Since AUROC depends on n , which
Revised: Since AUROC depends on n , which

Original: AUPRC only depends on r , AUROC is less sensitive to changes in performance than AUPRC.
Revised: AUPRC only depends on r , AUROC is less sensitive to changes in performance than AUPRC.

Original: For example, our simulated spike dataset contains 3 features and 80 time points, therefore the number of observations is n = 240 .
Revised: For example, our simulated spike dataset contains 3 features and 80 time points, therefore the number of observations is n = 240 .

Original: If the positively-labelled feature is the 4th most important feature, i.e. r = 4 , the AUROC is 0 .
Revised: If the positively-labelled feature is the 4th most important feature, i.e. r = 4 , the AUROC is 0 .

Original: 9874 while the AUPRC is 0 .
Revised: 9874 while the AUPRC is 0 .

Original: If the rank is 10 , AUROC is 0 .
Revised: If the rank is 10 , AUROC is 0 .

Original: 958 and
Revised: 958 and

Original: AUPRC is 0 .
Revised: AUPRC is 0 .

Original: This shows that AUPRC is much more sensitive to changes in rank that AUROC.
Revised: This shows that AUPRC is much more sensitive to changes in rank that AUROC.

Revised: To compare performance across explainability methods on the spike and delayed spike datasets, we argue that mean rank and AUPRC provide the clearest comparison.

Original: [Table]
Revised: [Table]

Original: To compare performance across explainability methods on the spike and delayed spike datasets, we argue that mean rank and AUPRC provide the clearest comparison.
Revised: E VALUATING D YNAMASK ON S IMULATED D ATASETS

Original: Evaluating Dynamask on Simulated Datasets As described in Section 5, Dynamask (Crabb ´ e & van der Schaar, 2021) is expected to produce almost binary masks, while the importance scores for other methods are expected to be continuous values.
Revised: As described in Section [Table] Dynamask (Crabb ´ e & van der Schaar, 2021) is expected to produce almost binary masks, while the importance scores for other methods are expected to be continuous values.

Original: This makes the evaluation of Dynmask on simulated data very sensitive to the way in which ties are resolved.
Revised: This makes the evaluation of Dynmask on simulated data very sensitive to the way in which ties are resolved.

Original: For completeness we present the impact of these choices on the AUPRC and mean rank metrics for the State dataset in Table 6.
Revised: For completeness we present the impact of these choices on the AUPRC and mean rank metrics for the State dataset in Table 6.

Original: In particular we present mean rank where ties are resolved by taking the minimum rank and the average rank, where we refer to the latter as mean average rank.
Revised: In particular we present mean rank where ties are resolved by taking the minimum rank and the average rank, where we refer to the latter as mean average rank.

Original: We also present the results of AUPRC computed with and without interpolation.
Revised: We also present the results of AUPRC computed with and without interpolation.

Original: We refer to AUPRC without interpolation as Average Precision.
Revised: We refer to AUPRC without interpolation as Average Precision.

Original: In the main text we use AUPRC with interpolation following the presentation in Crabb ´ e & van der Schaar (2021), and note that elsewhere in the literature (Davis &
Revised: In the main text we use AUPRC with interpolation following the presentation in Crabb´e & van der Schaar (2021), and note that elsewhere in the literature (Davis &

Original: Goadrich, 2006) it is argued that average precision is preferred.
Revised: Goadrich, 2006) it is argued that average precision is preferred.

Revised: We see that Dynamask performance drops significantly when interpolation is not used in computing AUPRC and average vs minimum rank is used to resolve ties.

Original: Figure 11 shows the precision-recall curve and the two different types of interpolations.
Revised: Figure 11 shows the precision-recall curve and the two different types of interpolations.

Original: A. A LTERNATIVE MODEL ARCHITECTURES FOR MIMIC-III MORTALITY
Revised: A. A LTERNATIVE MODEL ARCHITECTURES FOR MIMIC-III MORTALITY

Original: In this section we show that WinIT continues to perform well on different model architectures.
Revised: In this section we show that WinIT continues to perform well on different model architectures.

Original: We show the performance on the MIMIC-III Mortality dataset on three additional architectures: 3Layer GRU, LSTM and ConvNet.
Revised: We show the performance on the MIMIC-III Mortality dataset on three additional architectures: 3Layer GRU, LSTM and ConvNet.

Original: We show the results using different masking scheme and show the performance for the K = 50 task.
Revised: We show the results using different masking scheme and show the performance for the K = 50 task.

Original: The performance of explainability methods follows a similar trend for different model architectures. [Figure] [Figure]
Revised: The performance of explainability methods follows a similar trend for different model architectures

Original: A. T RAINING C URVES FOR G ENERATORS
Revised: A. T RAINING C URVES FOR G ENERATORS

Original: The training curves for some selected generators are shown in Figure 15.
Revised: The training curves for some selected generators are shown in Figure 15.

Original: We apply early stopping to avoid overfitting.
Revised: We apply early stopping to avoid overfitting.