Original: TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
Revised: TimeSHAP: Explaining Recurrent Models through Sequence Perturbations

Original: Abstract
Revised: Abstract

Original: Recurrent neural networks are a standard building block in numerous machine learning domains, from natural language processing to time-series classiﬁcation.
Revised: Recurrent neural networks are a standard building block in numerous machine learning domains, from natural language processing to time-series classiﬁcation.

Original: While their application has grown ubiquitous, understanding of their inner workings is still lacking.
Revised: While their application has grown ubiquitous, understanding of their inner workings is still lacking.

Original: In practice, the complex decision-making in these models is seen as a black-box, creating a tension between accuracy and interpretability.
Revised: In practice, the complex decision-making in these models is seen as a black-box, creating a tension between accuracy and interpretability.

Original: Moreover, the ability to understand the reasoning process of a model is important in order to debug it and, even more so, to build trust in its decisions.
Revised: Moreover, the ability to understand the reasoning process of a model is important in order to debug it and, even more so, to build trust in its decisions.

Original: Although considerable research effort has been guided towards explaining black-box models in recent years, recurrent models have received relatively little attention.
Revised: Although considerable research effort has been guided towards explaining black-box models in recent years, recurrent models have received relatively little attention.

Original: Any method that aims to explain decisions from a sequence of instances should assess, not only feature importance, but also event importance, an ability that is missing from stateof-the-art explainers.
Revised: Any method that aims to explain decisions from a sequence of instances should assess, not only feature importance, but also event importance, an ability that is missing from stateof-the-art explainers.

Original: In this work, we contribute to ﬁlling these gaps by presenting
Revised: In this work, we contribute to ﬁlling these gaps by presenting

Original: TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP’s sound theoretical footing and strong empirical results.
Revised: TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP’s sound theoretical footing and strong empirical results.

Original: As the input sequence may be arbitrarily long, we further propose a pruning method that is shown to dramatically improve its efﬁciency in practice.
Revised: As the input sequence may be arbitrarily long, we further propose a pruning method that is shown to dramatically improve its efﬁciency in practice.

Original:  Introduction
Revised:  Introduction

Original: Recurrent neural networks (RNN) models, such as LSTMs [1] and GRUs [2], are state-of-the-art for numerous sequential decision tasks, from speech recognition [3] to language modelling [4] and time-series classiﬁcation [5].
Revised: Recurrent neural networks (RNN) models, such as LSTMs [1] and GRUs [2], are state-of-the-art for numerous sequential decision tasks, from speech recognition [3] to language modelling [4] and time-series classiﬁcation [5].

Original: However, while the application of RNNs has grown ubiquitous and its performance has steadily increased, understanding of its inner workings is still lacking.
Revised: However, while the application of RNNs has grown ubiquitous and its performance has steadily increased, understanding of its inner workings is still lacking.

Original: In practice, the complex decision-making processes in these models is seen as a black-box, creating a tension between accuracy and interpretability.
Revised: In practice, the complex decision-making processes in these models is seen as a black-box, creating a tension between accuracy and interpretability.

Original: Understanding the decision-making processes of complex models may be crucial in order to detect and correct ﬂawed reasonings, such as those stemming from spurious correlations in the training data.
Revised: Understanding the decision-making processes of complex models may be crucial in order to detect and correct ﬂawed reasonings, such as those stemming from spurious correlations in the training data.

Original: Models that rely on such spurious correlations, known as “Clever Hans” models 1 [7], may have strong test results but generalize poorly when deployed in the real-world.
Revised: Models that rely on such spurious correlations, known as “Clever Hans” models 1 [7], may have strong test results but generalize poorly when deployed in the real-world.

Original: By explaining the reasoning in a given model, we simultaneously gather insight into how it may be improved and may advance human understanding of the underlying task, as previously unknown patterns are uncovered by the explanation.
Revised: By explaining the reasoning in a given model, we simultaneously gather insight into how it may be improved and may advance human understanding of the underlying task, as previously unknown patterns are uncovered by the explanation.

Original: Additionally, understanding the model’s reasoning may be a requirement in certain real-world applications, as exempliﬁed in GDPR’s “right to explanation” [8] (although its reach is contested [9, 10]).
Revised: Additionally, understanding the model’s reasoning may be a requirement in certain real-world applications, as exempliﬁed in GDPR’s “right to explanation” [8] (although its reach is contested [9, 10]).

Original: Just as humans are biased and sometimes discriminatory towards certain groups of the population, so too can deep learning (DL) models be [11, 12, 13].
Revised: Just as humans are biased and sometimes discriminatory towards certain groups of the population, so too can deep learning (DL) models be [11, 12, 13].

Original: Regulators want to be able to peek under the “model’s hood” in order to audit for potential discriminatory reasoning.
Revised: Regulators want to be able to peek under the “model’s hood” in order to audit for potential discriminatory reasoning.

Original: Although humans may be less accurate, and certainly less scalable than DL models, they can offer some form of after-the-fact reasoning supporting their decisions.
Revised: Although humans may be less accurate, and certainly less scalable than DL models, they can offer some form of after-the-fact reasoning supporting their decisions.

Original: For all the beneﬁts and efﬁciencies DL has brought about, in order for the community to trust these models, it must be possible to explain their reasoning, at least to a level that humans can understand.
Revised: For all the beneﬁts and efﬁciencies DL has brought about, in order for the community to trust these models, it must be possible to explain their reasoning, at least to a level that humans can understand.

Original: In recent years, numerous methods have been put forth for explaining DL models [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Revised: In recent years, numerous methods have been put forth for explaining DL models [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].

Original: However, RNNs pose a distinct challenge, as their predictions are not only a function of the immediate input sample, but also of the previous input samples and the context (hidden state) drawn thereafter.
Revised: However, RNNs pose a distinct challenge, as their predictions are not only a function of the immediate input sample, but also of the previous input samples and the context (hidden state) drawn thereafter.

Original: Blindly applying state-of-the-art DL explainers to RNNs often disregards the importance of the hidden state, distributing all feature importance solely through the features of the current input (as illustrated by Figure 1).
Revised: Blindly applying state-of-the-art DL explainers to RNNs often disregards the importance of the hidden state, distributing all feature importance solely through the features of the current input (as illustrated by Figure 1).

Original: Recently, in landmark work, Lundberg and Lee [25] uniﬁed a large number of explanation methods into a single family of “additive feature attribution methods”.
Revised: Recently, in landmark work, Lundberg and Lee [25] uniﬁed a large number of explanation methods into a single family of “additive feature attribution methods”.

Original: The authors further proved that there is a unique solution to this task that fulﬁlls three crucial properties of explanations, and dubbed this implementation KernelSHAP.
Revised: The authors further proved that there is a unique solution to this task that fulﬁlls three crucial properties of explanations, and dubbed this implementation KernelSHAP.

Original: However, no work yet extended this method to explaining timestep-wise
Revised: However, no work yet extended this method to explaining timestep-wise

Original: RNN predictions.
Revised: RNN predictions.

Original: With this in mind, we propose TimeSHAP, a model-agnostic recurrent explainer suited for tabular sequence learning that leverages KernelSHAP’s strong theoretical foundations and empirical results.
Revised: With this in mind, we propose TimeSHAP, a model-agnostic recurrent explainer suited for tabular sequence learning that leverages KernelSHAP’s strong theoretical foundations and empirical results.

Original: TimeSHAP extends the framework put forth by Lundberg and Lee [25] to the recurrent model setting.
Revised: TimeSHAP extends the framework put forth by Lundberg and Lee [25] to the recurrent model setting.

Original: By doing so, we enable explanation, not only of which features are most important to a recurrent model, but also which previous events had the largest impact on a current prediction.
Revised: By doing so, we enable explanation, not only of which features are most important to a recurrent model, but also which previous events had the largest impact on a current prediction.

Original: As sequences are arbitrarily long, we further propose a pruning method that increases real-world accuracy and efﬁciency considerably.
Revised: As sequences are arbitrarily long, we further propose a pruning method that increases real-world accuracy and efﬁciency considerably.

Original: We analyze local and global explanations of an RNN using our method, and ﬁnd multiple instances in which these are crucial for debugging the underlying predictor.
Revised: We analyze local and global explanations of an RNN using our method, and ﬁnd multiple instances in which these are crucial for debugging the underlying predictor.

Original: The contributions of TimeSHAP can be summarized as follows:
Revised: The contributions of TimeSHAP can be summarized as follows:

Original: • explanations of both feature- and event-wise importance in sequence predictions; • a new perturbation function suited for the recurrent setting; • a coalition pruning algorithm that dramatically increases the method’s efﬁciency in practice; • an empirical analysis of our method on a real world banking dataset.
Revised: • explanations of both feature- and event-wise importance in sequence predictions; • a new perturbation function suited for the recurrent setting; • a coalition pruning algorithm that dramatically increases the method’s efﬁciency in practice; • an empirical analysis of our method on a real world banking dataset.

Original:  Related Work
Revised:  Related Work

Original: Research on machine learning model explainers can generally be subdivided into two categories: model-agnostic, and model-speciﬁc explainers.
Revised: Research on machine learning model explainers can generally be subdivided into two categories: model-agnostic, and model-speciﬁc explainers.

Original: Model speciﬁc explainers exploit characteristics of the model’s inner workings or architecture to obtain more accurate explanations of its reasoning [26].
Revised: Model speciﬁc explainers exploit characteristics of the model’s inner workings or architecture to obtain more accurate explanations of its reasoning [26].

Original: The task of explaining RNNs is often tackled by using attention mechanisms [27, 28, 29, 30].
Revised: The task of explaining RNNs is often tackled by using attention mechanisms [27, 28, 29, 30].

Original: However, whether attention can in fact explain a model’s behavior is debatable and a known source of controversy in the ML community [31, 32, 33].
Revised: However, whether attention can in fact explain a model’s behavior is debatable and a known source of controversy in the ML community [31, 32, 33].

Original: DL models, in which RNNs are included, can also be explained using gradient-based methods.
Revised: DL models, in which RNNs are included, can also be explained using gradient-based methods.

Original: These explainers attribute a weight w i to each feature, representing the importance, or saliency, of the i-th feature, based on the partial derivatives of the prediction function f ( x ) with respect to the input x i : w i =
Revised: These explainers attribute a weight w i to each feature, representing the importance, or saliency, of the i-th feature, based on the partial derivatives of the prediction function f ( x ) with respect to the input x i : w i =

Original: (cid:12)(cid:12)(cid:12) ∂f ( x ) ∂x i (cid:12)(cid:12)(cid:12) [14, 15, 16].
Revised: (cid:12)(cid:12)(cid:12) ∂f ( x ) ∂x i (cid:12)(cid:12)(cid:12) [14, 15, 16].

Original: Another family of DL explainers is that of layer-wise relevance propagation.
Revised: Another family of DL explainers is that of layer-wise relevance propagation.

Original: In the network’s backward pass, starting from the output neuron, the relevance (which initially corresponds to the predicted score) is propagated iteratively from higher layers to lower layers, according to some rules [18, 19, 20, 21].
Revised: In the network’s backward pass, starting from the output neuron, the relevance (which initially corresponds to the predicted score) is propagated iteratively from higher layers to lower layers, according to some rules [18, 19, 20, 21].

Original: However, when explaining sequential inputs, DLspeciﬁc methods focus on features instead of events, leaving event relevance as a largely unexplored research direction.
Revised: However, when explaining sequential inputs, DLspeciﬁc methods focus on features instead of events, leaving event relevance as a largely unexplored research direction.

Original: Regarding RNN-speciﬁc explainers [22, 23, 24], these are often inﬂexible with regards to the model’s architecture.
Revised: Regarding RNN-speciﬁc explainers [22, 23, 24], these are often inﬂexible with regards to the model’s architecture.

Original: For instance, if the RNN is a building block of a larger DL model, preceded/succeeded by other types of layers, it is not the direct input to the RNN we want to explain but the input to the model.
Revised: For instance, if the RNN is a building block of a larger DL model, preceded/succeeded by other types of layers, it is not the direct input to the RNN we want to explain but the input to the model.

Original: Hence, a model-agnostic explainer may be better suited to explain these architectures, as real-world models are seldom built only with recurrent layers.
Revised: Hence, a model-agnostic explainer may be better suited to explain these architectures, as real-world models are seldom built only with recurrent layers.

Original: Model-agnostic explainers are substantially more ﬂexible, thus often preferred in real-world applications.
Revised: Model-agnostic explainers are substantially more ﬂexible, thus often preferred in real-world applications.

Original: These explainers generally rely on post-hoc access to a model’s predictions under various settings, such as perturbations of its input [34].
Revised: These explainers generally rely on post-hoc access to a model’s predictions under various settings, such as perturbations of its input [34].

Original: A perturbation h x of the input vector x ∈
Revised: A perturbation h x of the input vector x ∈

Original: X m is the result of converting all values of a coalition of features z ∈ { 0 , 1 } m to the original input space X m , such that z i = 1 means that a feature i takes its original value x i , and z i = 0 means that a feature i takes some uninformative background value b i representing its removal.
Revised: X m is the result of converting all values of a coalition of features z ∈ { 0 , 1 } m to the original input space X m , such that z i = 1 means that a feature i takes its original value x i , and z i = 0 means that a feature i takes some uninformative background value b i representing its removal.

Original: Hence, the input perturbation function h x is given as follows:
Revised: Hence, the input perturbation function h x is given as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is the component-wise product.
Revised: where (cid:12) is the component-wise product.

Original: The vector b ∈ X m represents an uninformative input sample, which is often taken to be the zero vector [35], b = 0 , or to be composed of the average feature values in the input dataset [25], b i = x i .
Revised: The vector b ∈ X m represents an uninformative input sample, which is often taken to be the zero vector [35], b = 0 , or to be composed of the average feature values in the input dataset [25], b i = x i .

Original: Lundberg and Lee [25] uniﬁed this and other explainers (both model-agnostic and model-speciﬁc) into a single family of “additive feature attribution methods”.
Revised: Lundberg and Lee [25] uniﬁed this and other explainers (both model-agnostic and model-speciﬁc) into a single family of “additive feature attribution methods”.

Original: Moreover, the authors prove that there is a single solution to this family of methods that fulﬁlls both local accuracy (the explanation model should match the complex model locally), missingness (features that are set to be missing should have no impact on the predictions), and consistency (if a feature’s contribution increases then its attributed importance should not decrease).
Revised: Moreover, the authors prove that there is a single solution to this family of methods that fulﬁlls both local accuracy (the explanation model should match the complex model locally), missingness (features that are set to be missing should have no impact on the predictions), and consistency (if a feature’s contribution increases then its attributed importance should not decrease).

Original: Those authors put forth KernelSHAP, a model-agnostic explainer that fulﬁlls these three properties.
Revised: Those authors put forth KernelSHAP, a model-agnostic explainer that fulﬁlls these three properties.

Original: KernelSHAP approximates the local behavior of a complex model f with a linear model of feature importance g ,
Revised: KernelSHAP approximates the local behavior of a complex model f with a linear model of feature importance g ,

Original: such that g ( z ) ≈ f
Revised: such that g ( z ) ≈ f

Original: ( h x ( z )) .
Revised: ( h x ( z )) .

Original: The task of learning the explanation model g is cast as a cooperative game where a reward ( f ( x ) , the score of the original model) must be distributed fairly among the players ( i ∈ { 1 , . . .
Revised: The task of learning the explanation model g is cast as a cooperative game where a reward ( f ( x ) , the score of the original model) must be distributed fairly among the players ( i ∈ { 1 , . . .

Original: , m } , the features).
Revised: , m } , the features).

Original: The optimal reward distribution is given by the
Revised: The optimal reward distribution is given by the

Original: Shapley values formulation [36].
Revised: Shapley values formulation [36].

Original: However, obtaining the true Shapley values for all features would imply generating all possible coalitions of the input, z ∈ { 0 , 1 } m , which scales exponentially with m the number of features in the model, O (2 m ) .
Revised: However, obtaining the true Shapley values for all features would imply generating all possible coalitions of the input, z ∈ { 0 , 1 } m , which scales exponentially with m the number of features in the model, O (2 m ) .

Original: As this task is computationally intractable, KernelSHAP approximates the true values by randomly sampling feature coalitions [37].
Revised: As this task is computationally intractable, KernelSHAP approximates the true values by randomly sampling feature coalitions [37].

Original: The authors further show that a single coalition weighing kernel, π x ( z ) , and a single loss metric, L ( f, g, π x ) , lead to optimal approximations of the Shapley values:
Revised: The authors further show that a single coalition weighing kernel, π x ( z ) , and a single loss metric, L ( f, g, π x ) , lead to optimal approximations of the Shapley values:

Original: [Equation]
Revised: [Equation]

Original: L ( f, g, π x
Revised: L ( f, g, π x

Original: [Equation]
Revised: [Equation]

Original: = (cid:88)
Revised: = (cid:88)

Original: z ∈
Revised: z ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m [ f ( h x ( z )) − g
Revised: m [ f ( h x ( z )) − g

Original: [Equation]
Revised: [Equation]

Original: z )] 2 ·
Revised: z )] 2 ·

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ) (3) where | z | is the number of non-zero elements of z , and L is the squared loss used for learning g .
Revised: ) (3) where | z | is the number of non-zero elements of z , and L is the squared loss used for learning g .

Original: Despite being widely adopted by the ML community, KernelSHAP cannot be applied out-of-the-box to recurrent models.
Revised: Despite being widely adopted by the ML community, KernelSHAP cannot be applied out-of-the-box to recurrent models.

Original: Doing so disregards an important part of the model: the hidden state, which carries information on the previous predictions.
Revised: Doing so disregards an important part of the model: the hidden state, which carries information on the previous predictions.

Original: Hence, the current KernelSHAP framework cannot be applied to recurrent models due to the following shortcomings: (1) assuming independence between subsequent model predictions, and (2) disregarding the importance of the hidden state h as a feature.
Revised: Hence, the current KernelSHAP framework cannot be applied to recurrent models due to the following shortcomings: (1) assuming independence between subsequent model predictions, and (2) disregarding the importance of the hidden state h as a feature.

Original: There have been few approaches to extend this method to recurrent settings, but with debatable implementations.
Revised: There have been few approaches to extend this method to recurrent settings, but with debatable implementations.

Original: Ho et al.
Revised: Ho et al.

Original: [38] use KernelSHAP to explain RNN predictions on an ICU mortality dataset [39].
Revised: [38] use KernelSHAP to explain RNN predictions on an ICU mortality dataset [39].

Original: However, their implementation perturbs only the time-step t being explained (shortcoming (1)), and distributes the model’s score f
Revised: However, their implementation perturbs only the time-step t being explained (shortcoming (1)), and distributes the model’s score f

Original: ( x ( t ) , h ( t − 1) ) solely through the input features x ( t ) (shortcoming (2)).
Revised: ( x ( t ) , h ( t − 1) ) solely through the input features x ( t ) (shortcoming (2)).

Original:  Methodology
Revised:  Methodology

Original: Firstly, our goal is to enable explaining sequence models while preserving three desirable properties of importance attribution stemming from the Shapley values: local accuracy, missingness, and consistency [40].
Revised: Firstly, our goal is to enable explaining sequence models while preserving three desirable properties of importance attribution stemming from the Shapley values: local accuracy, missingness, and consistency [40].

Original: Secondly, as sequence models handle one extra dimension (spanning the sequence of input events), we aim to explain both feature importance and event importance.
Revised: Secondly, as sequence models handle one extra dimension (spanning the sequence of input events), we aim to explain both feature importance and event importance.

Original: Finally, our method should be resource-efﬁcient, as calculating the true Shapley values is trivial but computationally intractable.
Revised: Finally, our method should be resource-efﬁcient, as calculating the true Shapley values is trivial but computationally intractable.

Original: Hence, we put forth TimeSHAP, a model-agnostic recurrent explainer with sound theoretical footing and strong empirical results.
Revised: Hence, we put forth TimeSHAP, a model-agnostic recurrent explainer with sound theoretical footing and strong empirical results.

Original: 3.1 RNN Preliminaries
Revised: 3.1 RNN Preliminaries

Original: Although our method can be used to explain any sequence model, throughout this paper we will use the example of a vanilla RNN, as it is both simple and widely used.
Revised: Although our method can be used to explain any sequence model, throughout this paper we will use the example of a vanilla RNN, as it is both simple and widely used.

Original: Other types of recurrent models that could be used include the long short-term memory (LSTM) [1], the gated recurrent unit (GRU) [2], or conditional random ﬁeld (CRF) [41].
Revised: Other types of recurrent models that could be used include the long short-term memory (LSTM) [1], the gated recurrent unit (GRU) [2], or conditional random ﬁeld (CRF) [41].

Original: The predictions of a sequence model at a given time-step t are a function, not only of the current input x ( t ) , but also of its previous inputs x ( t − 1) , x ( t − 2) , . . .
Revised: The predictions of a sequence model at a given time-step t are a function, not only of the current input x ( t ) , but also of its previous inputs x ( t − 1) , x ( t − 2) , . . .

Original: , x (0) .
Revised: , x (0) .

Original: For RNN, this recurrence is achieved indirectly through a hidden state h that aims to encode all relevant information from previous time-steps.
Revised: For RNN, this recurrence is achieved indirectly through a hidden state h that aims to encode all relevant information from previous time-steps.

Original: As such, an RNN’s prediction, ˆ y ( t ) =
Revised: As such, an RNN’s prediction, ˆ y ( t ) =

Original: f ( x ( t ) , h ( t − 1) ) , is given as follows [42]:
Revised: f ( x ( t ) , h ( t − 1) ) , is given as follows [42]:

Original: [Equation]
Revised: [Equation]

Original: where b and c are learnable bias vectors, U, V and W are learnable weight matrices, and σ is a nonlinearity (often chosen to be the hyperbolic tangent).
Revised: where b and c are learnable bias vectors, U, V and W are learnable weight matrices, and σ is a nonlinearity (often chosen to be the hyperbolic tangent).

Original: Moreover, a ( t ) is known as the activation, h ( t ) as the hidden state, o ( t ) is the output, and ˆ y is a vector of probabilities over the output.
Revised: Moreover, a ( t ) is known as the activation, h ( t ) as the hidden state, o ( t ) is the output, and ˆ y is a vector of probabilities over the output.

Original: If the
Revised: If the

Original: RNN is the last layer of the model, then ˆ y ( t ) may be used directly as the prediction.
Revised: RNN is the last layer of the model, then ˆ y ( t ) may be used directly as the prediction.

Original: Otherwise, o ( t ) is passed to the following layers, and the prediction is given by the last layer in the forward-pass.
Revised: Otherwise, o ( t ) is passed to the following layers, and the prediction is given by the last layer in the forward-pass.

Original: 3.2 TimeSHAP
Revised: 3.2 TimeSHAP

Original: TimeSHAP builds upon KernelSHAP [25], a state-of-the-art model agnostic explainer, and extends it to work on sequential data.
Revised: TimeSHAP builds upon KernelSHAP [25], a state-of-the-art model agnostic explainer, and extends it to work on sequential data.

Original: Our method produces both feature-wise and event-wise explanations.
Revised: Our method produces both feature-wise and event-wise explanations.

Original: Hence, TimeSHAP attributes an importance value to each feature/event in the input, such that it reﬂects the degree to which that feature/event affected the ﬁnal prediction.
Revised: Hence, TimeSHAP attributes an importance value to each feature/event in the input, such that it reﬂects the degree to which that feature/event affected the ﬁnal prediction.

Original: In order to explain a sequential input, X ∈ R d × l , with l events and d features per event, our method ﬁts a linear explainer g that approximates the local behavior of a complex explainer f by minimizing the loss given by
Revised: In order to explain a sequential input, X ∈ R d × l , with l events and d features per event, our method ﬁts a linear explainer g that approximates the local behavior of a complex explainer f by minimizing the loss given by

Original: Equation 3.
Revised: Equation 3.

Original: As events are simply features in a temporal dimension, and the algorithm for explaining features x ∈ R 1 × l and events x ∈ R d × 1 is conceptually equal, we will henceforth use the word feature to mean both rows and columns of X ∈ R d × l .
Revised: As events are simply features in a temporal dimension, and the algorithm for explaining features x ∈ R 1 × l and events x ∈ R d × 1 is conceptually equal, we will henceforth use the word feature to mean both rows and columns of X ∈ R d × l .

Original: Thus, the formula for g is:
Revised: Thus, the formula for g is:

Original: [Equation]
Revised: [Equation]

Original: where the bias term w 

Revised: where the bias term w

Original: = f
Revised: = f

Original: ( h X ( 0 )) corresponds to the model’s output with all features toggled off (dubbed base score ), the weights w i , i ∈ { 1 , . . .
Revised: ( h X ( 0 )) corresponds to the model’s output with all features toggled off (dubbed base score ), the weights w i , i ∈ { 1 , . . .

Original: , m } , correspond to the importance of each feature, and either m =
Revised: , m } , correspond to the importance of each feature, and either m =

Original: d or m =
Revised: d or m =

Original: l depending on which dimension is being explained.
Revised: l depending on which dimension is being explained.

Original: The perturbation function h X : { 0 , 1 } m (cid:55)→ R d × l maps a coalition z ∈ { 0 , 1 } m to the original input space R d × l .
Revised: The perturbation function h X : { 0 , 1 } m (cid:55)→ R d × l maps a coalition z ∈ { 0 , 1 } m to the original input space R d × l .

Original: Note that the sum of all feature importances corresponds to the difference between the model’s score f ( X ) =
Revised: Note that the sum of all feature importances corresponds to the difference between the model’s score f ( X ) =

Original: f ( h X ( 1 )) and the base score f
Revised: f ( h X ( 1 )) and the base score f

Original: ( h X ( 0 )) .
Revised: ( h X ( 0 )) .

Original: Input perturbations are generated differently depending on which dimension is being explained.
Revised: Input perturbations are generated differently depending on which dimension is being explained.

Original: The perturbation function described in Equation 1 is suited to explain a single dimension of features.
Revised: The perturbation function described in Equation 1 is suited to explain a single dimension of features.

Original: We extend this function to the recurrent (and bi-dimensional) setting as follows.
Revised: We extend this function to the recurrent (and bi-dimensional) setting as follows.

Original: Given a matrix
Revised: Given a matrix

Original: B ∈ R d × l representing an uninformative input (the absence of discriminative features or events), a perturbation h (cid:48)
Revised: B ∈ R d × l representing an uninformative input (the absence of discriminative features or events), a perturbation h (cid:48)

Original: X along the features axis (the rows) of the input matrix X ∈ R d × l is the result of mapping a coalition vector z ∈ { 0 , 1 } d to the original input space R d × l , such that z i = 1 means that row i takes its original value
Revised: X along the features axis (the rows) of the input matrix X ∈ R d × l is the result of mapping a coalition vector z ∈ { 0 , 1 } d to the original input space R d × l , such that z i = 1 means that row i takes its original value

Original: X i, : , and z i = 0 means that row i takes the background uninformative value
Revised: X i, : , and z i = 0 means that row i takes the background uninformative value

Original: B i, : .
Revised: B i, : .

Original: Thus, when z i = 

Revised: Thus, when z i =

Original: the feature i is essentially toggled off for all events of the sequence.
Revised: the feature i is essentially toggled off for all events of the sequence.

Original: This is formalized as follows:
Revised: This is formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, a perturbation h ∗ X along the events axis (the columns) of the input matrix
Revised: On the other hand, a perturbation h ∗ X along the events axis (the columns) of the input matrix

Original: X ∈ R d × l is the result of mapping a coalition vector z ∈ { 0 , 1 } l to the original input space R d × l , such that z j = 1 means that column j takes its original value X : ,j , and z j = 0 means that column j takes the value B : ,j .
Revised: X ∈ R d × l is the result of mapping a coalition vector z ∈ { 0 , 1 } l to the original input space R d × l , such that z j = 1 means that column j takes its original value X : ,j , and z j = 0 means that column j takes the value B : ,j .

Original: Thus, when z j = 0 all features of event j are toggled off.
Revised: Thus, when z j = 0 all features of event j are toggled off.

Original: This is formalized as follows:
Revised: This is formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: Hence, when explaining features h X = h (cid:48)
Revised: Hence, when explaining features h X = h (cid:48)

Original: X , and when explaining events h X
Revised: X , and when explaining events h X

Original: = h ∗ X .
Revised: = h ∗ X .

Original: This change in the perturbation function is the sole implementation difference between explaining events and features.
Revised: This change in the perturbation function is the sole implementation difference between explaining events and features.

Original: Moreover, the perturbation of X according to a null-vector coalition z = 0 is the same regardless of which dimension is being perturbed, h (cid:48) X ( 0 )
Revised: Moreover, the perturbation of X according to a null-vector coalition z = 0 is the same regardless of which dimension is being perturbed, h (cid:48) X ( 0 )

Original: =
Revised: =

Original: h ∗ X ( 0 ) , and equally for z = 1 , h (cid:48)
Revised: h ∗ X ( 0 ) , and equally for z = 1 , h (cid:48)

Original: X ( 1 ) =
Revised: X ( 1 ) =

Original: h ∗ X ( 1 ) .
Revised: h ∗ X ( 1 ) .

Original: In our setting, we deﬁne the background matrix B ∈ R l × d to be composed of the average feature values in the training dataset:
Revised: In our setting, we deﬁne the background matrix B ∈ R l × d to be composed of the average feature values in the training dataset:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3.2.1 Pruning
Revised: 3.2.1 Pruning

Original: One glaring issue with TimeSHAP is that the number of event (temporal) coalitions scales exponentially with the length of the observed sequence, just as in KernelSHAP number of feature coalitions scales exponentially with the number of input features.
Revised: One glaring issue with TimeSHAP is that the number of event (temporal) coalitions scales exponentially with the length of the observed sequence, just as in KernelSHAP number of feature coalitions scales exponentially with the number of input features.

Original: Moreover, in a recurrent setting, the input sequence can be arbitrarily long, making this a serious issue that we address by proposing a temporal-coalition pruning algorithm.
Revised: Moreover, in a recurrent setting, the input sequence can be arbitrarily long, making this a serious issue that we address by proposing a temporal-coalition pruning algorithm.

Original: It is common for real-world events to be preceded by a long history of past events (e.g., the whole transaction history of a client), only a few of which are relevant to the current prediction.
Revised: It is common for real-world events to be preceded by a long history of past events (e.g., the whole transaction history of a client), only a few of which are relevant to the current prediction.

Original: Additionally, recurrent models are known to seldom encode information from events in the distant past [43].
Revised: Additionally, recurrent models are known to seldom encode information from events in the distant past [43].

Original: Based on this insight, we group together older unimportant events as a single feature, thereby reducing the number of coalitions by a factor of 2 i − 1 , where i is the number of grouped events.
Revised: Based on this insight, we group together older unimportant events as a single feature, thereby reducing the number of coalitions by a factor of 2 i − 1 , where i is the number of grouped events.

Original: Essentially, we lose the granularity between the importance of these grouped events, but their importance (albeit small) is not disregarded.
Revised: Essentially, we lose the granularity between the importance of these grouped events, but their importance (albeit small) is not disregarded.

Original: The pruning method, deﬁned in Algorithm 1, consists in splitting the input sequence X ∈ R d × l into two sub-sequences X : , 1: i , X : ,i +1: l , i ∈ { 1 , . . .
Revised: The pruning method, deﬁned in Algorithm 1, consists in splitting the input sequence X ∈ R d × l into two sub-sequences X : , 1: i , X : ,i +1: l , i ∈ { 1 , . . .

Original: , l − 1 } , ( X : ,l being the most recent event) and computing the true Shapley values for each.
Revised: , l − 1 } , ( X : ,l being the most recent event) and computing the true Shapley values for each.

Original: Computing these Shapley values amounts to 2 2 = 4 total coalitions (for each i ).
Revised: Computing these Shapley values amounts to 2 2 = 4 total coalitions (for each i ).

Original: Our objective is to ﬁnd the largest i such that the importance value for X : , 1: i falls below a given importance threshold η .
Revised: Our objective is to ﬁnd the largest i such that the importance value for X : , 1: i falls below a given importance threshold η .

Original: Algorithm 1 Temporal Coalition Pruning
Revised: Algorithm 1 Temporal Coalition Pruning

Original: Input: input sequence X , model to explain f , tolerance η , 1: for i ∈ { l − 1 , l − 2 , . .
Revised: Input: input sequence X , model to explain f , tolerance η , 1: for i ∈ { l − 1 , l − 2 , . .

Original: , 1 } do (cid:46)
Revised: , 1 } do (cid:46)

Original: Starting from the end of the sequence 2:
Revised: Starting from the end of the sequence 2:

Original: Z ← { [0 , 0] , [0 , 1] , [1 , 0] , [1 , 1] } (cid:46)
Revised: Z ← { [0 , 0] , [0 , 1] , [1 , 0] , [1 , 1] } (cid:46)

Original: Full set of coalitions to use for each i 3: w 1 , w 2 ← KernelSHAP ( (cid:46) Call adapted KernelSHAP model =
Revised: Full set of coalitions to use for each i 3: w 1 , w 2 ← KernelSHAP ( (cid:46) Call adapted KernelSHAP model =

Original: f , input =
Revised: f , input =

Original: [ X : , 1: i , X : ,i +1: l ] , (cid:46) X given as composed of only two features perturbation = h ∗ X , (cid:46) Parameterized by our temporal perturbation function coalitions =
Revised: [ X : , 1: i , X : ,i +1: l ] , (cid:46) X given as composed of only two features perturbation = h ∗ X , (cid:46) Parameterized by our temporal perturbation function coalitions =

Original: Z ) (cid:46)
Revised: Z ) (cid:46)

Original: Employing only 2 2 coalitions (SHAP sees only 2 features) 4: if | w 1 | < η then aggregate all events up
Revised: Employing only 2 2 coalitions (SHAP sees only 2 features) 4: if | w 1 | < η then aggregate all events up

Original: The computational cost of this pruning algorithm scales only linearly, O ( l ) , with the number of events.
Revised: The computational cost of this pruning algorithm scales only linearly, O ( l ) , with the number of events.

Original: Consequently, when employing pruning, the run-time of TimeSHAP when explaining the events’ axis is reduced from O (2 l ) to O (2 l − i ) .
Revised: Consequently, when employing pruning, the run-time of TimeSHAP when explaining the events’ axis is reduced from O (2 l ) to O (2 l − i ) .

Original: We will empirically show in Section 4.1 that events in the distant past seldom affect the model’s score, leading to l − i (cid:28) l .
Revised: We will empirically show in Section 4.1 that events in the distant past seldom affect the model’s score, leading to l − i (cid:28) l .

Original: Taking the best-case scenario of a recurrent model whose run-time scales linearly with l , such as a recurrent neural model ( e.g.,
Revised: Taking the best-case scenario of a recurrent model whose run-time scales linearly with l , such as a recurrent neural model ( e.g.,

Original: RNN, LSTM, GRU ), TimeSHAP’s run-time with temporal-coalition pruning totals O ( l · 2 l − i ) .
Revised: RNN, LSTM, GRU ), TimeSHAP’s run-time with temporal-coalition pruning totals O ( l · 2 l − i ) .

Original: On the other hand, when explaining the features’ axis, TimeSHAP’s computational complexity is unaffected by the pruning algorithm, totalling O ( l · 2 d ) .
Revised: On the other hand, when explaining the features’ axis, TimeSHAP’s computational complexity is unaffected by the pruning algorithm, totalling O ( l · 2 d ) .

Original: 3.2.2 Algorithm
Revised: 3.2.2 Algorithm

Original: TimeSHAP’s objective is to answer the following questions: “Which events and features contributed the most to the current prediction?”
Revised: TimeSHAP’s objective is to answer the following questions: “Which events and features contributed the most to the current prediction?”

Original: and “What was their inﬂuence on the model’s score?”
Revised: and “What was their inﬂuence on the model’s score?”

Original: Hence, when explaining the features axis, TimeSHAP ﬁts a linear explainer g that approximates the local behavior of a complex explainer f by minimizing the loss given in Equation 3, parameterized by our perturbation function h (cid:48)
Revised: Hence, when explaining the features axis, TimeSHAP ﬁts a linear explainer g that approximates the local behavior of a complex explainer f by minimizing the loss given in Equation 3, parameterized by our perturbation function h (cid:48)

Original: X .
Revised: X .

Original: On the other hand, when explaining the events’ axis, TimeSHAP ﬁrst (optionally) prunes the input sequence’s coalitions (as per Algorithm 1), and then ﬁts the linear explainer, parameterized by our perturbation function h ∗ X .
Revised: On the other hand, when explaining the events’ axis, TimeSHAP ﬁrst (optionally) prunes the input sequence’s coalitions (as per Algorithm 1), and then ﬁts the linear explainer, parameterized by our perturbation function h ∗ X .

Original: Note that, as the number of coalitions scales exponentially with the sequence’s length on the axis that is being explained (reduced by the pruning factor), it may not be tractable to exhaustively evaluate all coalitions.
Revised: Note that, as the number of coalitions scales exponentially with the sequence’s length on the axis that is being explained (reduced by the pruning factor), it may not be tractable to exhaustively evaluate all coalitions.

Original: In this case, similarly to KernelSHAP, we randomly sample coalitions from the pool of coalitions up to a predetermined number of draws.
Revised: In this case, similarly to KernelSHAP, we randomly sample coalitions from the pool of coalitions up to a predetermined number of draws.

Original:  Experiments
Revised:  Experiments

Original: To validate our method, we trained a recurrent deep learning model on a large-scale real-world banking dataset.
Revised: To validate our method, we trained a recurrent deep learning model on a large-scale real-world banking dataset.

Original: The model is composed of an embedding layer for categorical variables, followed by a GRU layer [2], and subsequently followed by a feed forward layer
Revised: The model is composed of an embedding layer for categorical variables, followed by a GRU layer [2], and subsequently followed by a feed forward layer

Original: The task consists in predicting account takeover fraud, a form of identity theft where a fraudster gains access to a victim’s bank account,
Revised: The task consists in predicting account takeover fraud, a form of identity theft where a fraudster gains access to a victim’s bank account,

Original: [Table]
Revised: [Table]

Original: Percentile
Revised: Percentile

Original: [Table]
Revised: [Table]

Original: enabling them to place unauthorized transactions.
Revised: enabling them to place unauthorized transactions.

Original: The data is tabular, consisting of approximately 20M events, including clients’ transactions, logins, or enrollments 2 , as well as corresponding geolocation and demographics data.
Revised: The data is tabular, consisting of approximately 20M events, including clients’ transactions, logins, or enrollments 2 , as well as corresponding geolocation and demographics data.

Original: We run TimeSHAP on 1 K randomly chosen sequences that were predicted positive by the model.
Revised: We run TimeSHAP on 1 K randomly chosen sequences that were predicted positive by the model.

Original: We set the maximum number of coalition samples to n_samples = 32 K .
Revised: We set the maximum number of coalition samples to n_samples = 32 K .

Original: Regarding pruning, we employ our proposed temporal-coalition pruning algorithm, as it promotes exponentially faster execution at no cost to the explanation’s reliability (only with decreased granularity on unimportant older events).
Revised: Regarding pruning, we employ our proposed temporal-coalition pruning algorithm, as it promotes exponentially faster execution at no cost to the explanation’s reliability (only with decreased granularity on unimportant older events).

Original: For sequences with a number of events/features higher than log 2 ( n_samples ) , pruning does not impact performance directly, improving instead the accuracy of results, as longer sequences would need a higher number of coalition samples to accurately compute Shapley values.
Revised: For sequences with a number of events/features higher than log 2 ( n_samples ) , pruning does not impact performance directly, improving instead the accuracy of results, as longer sequences would need a higher number of coalition samples to accurately compute Shapley values.

Original: We can choose a pruning tolerance value that enables exhaustively computing the Shapley values for most input sequences within the allocated n_samples budget.
Revised: We can choose a pruning tolerance value that enables exhaustively computing the Shapley values for most input sequences within the allocated n_samples budget.

Original: 4.1 Pruning method results
Revised: 4.1 Pruning method results

Original: Table 1 details average, median, and maximum number of events for unpruned sequences, and for sequences pruned with varying tolerance levels for Algorithm 1.
Revised: Table 1 details average, median, and maximum number of events for unpruned sequences, and for sequences pruned with varying tolerance levels for Algorithm 1.

Original: The percentage of sequences whose length, | X | , is under log 2 ( n_samples ) ≈ 15 is shown in the fourth row.
Revised: The percentage of sequences whose length, | X | , is under log 2 ( n_samples ) ≈ 15 is shown in the fourth row.

Original: This represents the percentage of input sequences whose Shapley values can be exactly computed by exhaustively evaluating all 2 | X | coalitions.
Revised: This represents the percentage of input sequences whose Shapley values can be exactly computed by exhaustively evaluating all 2 | X | coalitions.

Original: The Shapley values for all sequences longer than log 2 ( n_samples ) are estimated by randomly sampling coalitions.
Revised: The Shapley values for all sequences longer than log 2 ( n_samples ) are estimated by randomly sampling coalitions.

Original: We note that, for the original sequences, we can only compute exact Shapley values for 10% of the samples.
Revised: We note that, for the original sequences, we can only compute exact Shapley values for 10% of the samples.

Original: For the median original sequence, with a total number of coalitions on the order of 2 | X | = 2 139 , 32K sampled coalitions represents 10 − 36 % of the total universe of coalitions.
Revised: For the median original sequence, with a total number of coalitions on the order of 2 | X | = 2 139 , 32K sampled coalitions represents 10 − 36 % of the total universe of coalitions.

Original: Hence, pruning is not only resource-efﬁcient but also a necessary step in order to achieve accurate results.
Revised: Hence, pruning is not only resource-efﬁcient but also a necessary step in order to achieve accurate results.

Original: When using η = 0 .
Revised: When using η = 0 .

Original: 01 , we can compute exact Shapley values for 36.5% of the input samples.
Revised: 01 , we can compute exact Shapley values for 36.5% of the input samples.

Original: On the other hand, when using η = 0 . 025 , we can compute exact values for 58.3% of the input samples.
Revised: On the other hand, when using η = 0 . 025 , we can compute exact values for 58.3% of the input samples.

Original: Hence, we choose η = 0 .
Revised: Hence, we choose η = 0 .

Original: 025 as our pruning tolerance, providing a balance between explanations’ consistency, run-time, and granularity.
Revised: 025 as our pruning tolerance, providing a balance between explanations’ consistency, run-time, and granularity.

Original: The last row of Table 1 shows the relative standard deviation 3 [44] (RSD) of the Shapley values obtained over 10 runs of TimeSHAP for different pruning levels.
Revised: The last row of Table 1 shows the relative standard deviation 3 [44] (RSD) of the Shapley values obtained over 10 runs of TimeSHAP for different pruning levels.

Original: As expected, lower pruning tolerances (lower η ) lead to ﬁner-grained event-level explanations (higher number of explained events) but with lower reliability (higher RSD values).
Revised: As expected, lower pruning tolerances (lower η ) lead to ﬁner-grained event-level explanations (higher number of explained events) but with lower reliability (higher RSD values).

Original: In fact, there is a strict negative relation between pruning tolerance and RSD values.
Revised: In fact, there is a strict negative relation between pruning tolerance and RSD values.

Original: Running TimeSHAP on the original sequences (equivalent to η = 0 ) leads to very high variance (RSD 1 . 23 ), while the highest pruning tolerance η = 0 .
Revised: Running TimeSHAP on the original sequences (equivalent to η = 0 ) leads to very high variance (RSD 1 . 23 ), while the highest pruning tolerance η = 0 .

Original: 05 leads to relatively low variance (RSD 0 . 15 ).
Revised: 05 leads to relatively low variance (RSD 0 . 15 ).

Original: Once again, η = 0 .
Revised: Once again, η = 0 .

Original: 025 (RSD 0 . 23 ) achieves a well-balanced combination of metrics.
Revised: 025 (RSD 0 . 23 ) achieves a well-balanced combination of metrics.

Original: 4.2 Local explanations
Revised: 4.2 Local explanations

Original: We analyze TimeSHAP’s local explanations on two predicted-positive sequences, hence labeled A and B. Sequence A has a model score of f ( A ) = 0 . 57 , and a total length of 47 events.
Revised: We analyze TimeSHAP’s local explanations on two predicted-positive sequences, hence labeled A and B. Sequence A has a model score of f ( A ) = 0 . 57 , and a total length of 47 events.

Original: Sequence B
Revised: Sequence B

Original: [Figure]
Revised: [Figure]

Original: has a model score of f ( B ) = 0 .
Revised: has a model score of f ( B ) = 0 .

Original: 84 , and a total length of 286 events.
Revised: 84 , and a total length of 286 events.

Original: As a convention, we dub the current event’s index (the most recent) as t = 0 , and use negative indices for older events (the event at index t = − 1 immediately precedes the event at index t = 0 , and so on).
Revised: As a convention, we dub the current event’s index (the most recent) as t = 0 , and use negative indices for older events (the event at index t = − 1 immediately precedes the event at index t = 0 , and so on).

Original: Figures 2a and 2d show the Shapley values (importance) respectively for Sequences A and B, when split into two disjoint sub-sequences at a given index t .
Revised: Figures 2a and 2d show the Shapley values (importance) respectively for Sequences A and B, when split into two disjoint sub-sequences at a given index t .

Original: This corresponds to the application of the for loop in Algorithm 1, continuing even after the pruning condition has been fulﬁlled.
Revised: This corresponds to the application of the for loop in Algorithm 1, continuing even after the pruning condition has been fulﬁlled.

Original: Figure 2d only displays the ﬁrst 100 indexes as displaying all 286 events would clutter the Figure.
Revised: Figure 2d only displays the ﬁrst 100 indexes as displaying all 286 events would clutter the Figure.

Original: As expected, the aggregate importance of older events (from the beginning of the sequence up to index t ) suffers a steep decrease as its distance to the current event increases.
Revised: As expected, the aggregate importance of older events (from the beginning of the sequence up to index t ) suffers a steep decrease as its distance to the current event increases.

Original: This trend corroborates our hypothesis and supports our coalition pruning algorithm.
Revised: This trend corroborates our hypothesis and supports our coalition pruning algorithm.

Original: When considering the coalition pruning tolerance η = 0 .
Revised: When considering the coalition pruning tolerance η = 0 .

Original: 025 , Sequence A is pruned to 11 events, grouping the 36 older events together.
Revised: 025 , Sequence A is pruned to 11 events, grouping the 36 older events together.

Original: Similarly, sequence B is pruned to 9 events, grouping the last 277 events together.
Revised: Similarly, sequence B is pruned to 9 events, grouping the last 277 events together.

Original: Sequence A ’s event-wise explanations are shown in Figure 2b, and its feature-wise explanations in
Revised: Sequence A ’s event-wise explanations are shown in Figure 2b, and its feature-wise explanations in

Original: Figure 2c.
Revised: Figure 2c.

Original: We conclude that there are two events crucial for the model’s prediction: the transaction being explained ( t = 0 ), with a Shapley value of 0.36, and another transaction, 4 events before ( t = − 4 ), with a Shapley value of 0.17.
Revised: We conclude that there are two events crucial for the model’s prediction: the transaction being explained ( t = 0 ), with a Shapley value of 0.36, and another transaction, 4 events before ( t = − 4 ), with a Shapley value of 0.17.

Original: Between the two relevant transactions, there are three logins with little to no importance (events − 3 ≤ t ≤ − 1 ).
Revised: Between the two relevant transactions, there are three logins with little to no importance (events − 3 ≤ t ≤ − 1 ).

Original: Prior (in temporal order) to event t = − 4 , there are 5 logins and 1 transaction with reduced importance, which were nonetheless left unpruned by Algorithm 1.
Revised: Prior (in temporal order) to event t = − 4 , there are 5 logins and 1 transaction with reduced importance, which were nonetheless left unpruned by Algorithm 1.

Original: Regarding feature importances, we observe that the most relevant features are, in decreasing order of importance, the transaction’s amount , IP feature D , and the clients’ age .
Revised: Regarding feature importances, we observe that the most relevant features are, in decreasing order of importance, the transaction’s amount , IP feature D , and the clients’ age .

Original: When inspecting the raw feature data, we observe that the amount transferred at both transactions t =and t = − 4 is unusually high, a known account takeover indicator.
Revised: When inspecting the raw feature data, we observe that the amount transferred at both transactions t =and t = − 4 is unusually high, a known account takeover indicator.

Original: This is in accordance with the simultaneous high event importance for t = 0 and t = − 4 , together with the high feature importance for the transaction amount.
Revised: This is in accordance with the simultaneous high event importance for t = 0 and t = − 4 , together with the high feature importance for the transaction amount.

Original: Moreover, we observe that the client’s age is relatively high, another well-known fraud indicator, as elderly clients are often more susceptible to being victims of fraud [45].
Revised: Moreover, we observe that the client’s age is relatively high, another well-known fraud indicator, as elderly clients are often more susceptible to being victims of fraud [45].

Original: When analysing IP feature D , although this feature does not show any strange behavior, it assumes a value that is frequent throughout the dataset.
Revised: When analysing IP feature D , although this feature does not show any strange behavior, it assumes a value that is frequent throughout the dataset.

Original: Upon further inspection we conclude that the IP belongs to a cloud hosting provider, which domain experts conﬁrm to be suspicious behavior.
Revised: Upon further inspection we conclude that the IP belongs to a cloud hosting provider, which domain experts conﬁrm to be suspicious behavior.

Original: Sequence B ’s event-wise explanations are shown in Figure 2e, and its feature-wise explanations in Figure 2f.
Revised: Sequence B ’s event-wise explanations are shown in Figure 2e, and its feature-wise explanations in Figure 2f.

Original: Regarding event importance, we conclude that the most relevant events are events at indices − 4 and − 1 with their respective Shapley values of 0 .
Revised: Regarding event importance, we conclude that the most relevant events are events at indices − 4 and − 1 with their respective Shapley values of 0 .

Original: 48 and 0 .
Revised: 48 and 0 .

Original: 24 followed by events −( 0 . 089 ) and − 3 ( 0 . 049 ).
Revised: 24 followed by events −( 0 . 089 ) and − 3 ( 0 . 049 ).

Original: Interestingly, for this sequence, the most relevant event is not the current input ( t = 0 ), with near null contribution to the score ( 0 . 001 ).
Revised: Interestingly, for this sequence, the most relevant event is not the current input ( t = 0 ), with near null contribution to the score ( 0 . 001 ).

Original: The event types for the sequence of events from t = − 4 to t = − 2 are enrollment - login - transaction , a well-known pattern that is repeated on numerous stolen accounts.
Revised: The event types for the sequence of events from t = − 4 to t = − 2 are enrollment - login - transaction , a well-known pattern that is repeated on numerous stolen accounts.

Original: This sequence of events encodes a change of account settings, e.g.: a password change (enrollment), followed by a login into the captured account, and subsequently followed by one or more fraudulent transactions.
Revised: This sequence of events encodes a change of account settings, e.g.: a password change (enrollment), followed by a login into the captured account, and subsequently followed by one or more fraudulent transactions.

Original: Interestingly, events t = − 1 and t = − 2 are transactions that succeed the fraudulent enrollment and login, but precede the current transaction ( t = 0 ).
Revised: Interestingly, events t = − 1 and t = − 2 are transactions that succeed the fraudulent enrollment and login, but precede the current transaction ( t = 0 ).

Original: The information up to t = − 1 is already sufﬁcient for the model to correctly identify the account as compromised, corroborated by the low contribution of the transaction at t = 0 .
Revised: The information up to t = − 1 is already sufﬁcient for the model to correctly identify the account as compromised, corroborated by the low contribution of the transaction at t = 0 .

Original: Regarding feature importances, the most relevant features are related to the transaction type , event type , the clients’ age , and the location .
Revised: Regarding feature importances, the most relevant features are related to the transaction type , event type , the clients’ age , and the location .

Original: The feature transaction type indicates a ﬁner grained event taxonomy for when event type = “transaction”.
Revised: The feature transaction type indicates a ﬁner grained event taxonomy for when event type = “transaction”.

Original: When inspecting the raw feature data, we observe that the client is in the elderly age range, which, as previously mentioned, may indicate a more susceptible demographic.
Revised: When inspecting the raw feature data, we observe that the client is in the elderly age range, which, as previously mentioned, may indicate a more susceptible demographic.

Original: When analysing the location features Location feature A and Location feature D , we observe a discrepancy between the location of the enrollment, login and transactions from the account’s history.
Revised: When analysing the location features Location feature A and Location feature D , we observe a discrepancy between the location of the enrollment, login and transactions from the account’s history.

Original: This discrepancy in physical location is highly suspicious and indicates that there was an enrollment on the account from a previously unused location.
Revised: This discrepancy in physical location is highly suspicious and indicates that there was an enrollment on the account from a previously unused location.

Original: 4.3 Global explanations
Revised: 4.3 Global explanations

Original: Supplying a data scientist with global explanations can enable an overview of the model’s decision process, revealing which features, or events in the case of TimeSHAP, are relevant to the model and which ones are not.
Revised: Supplying a data scientist with global explanations can enable an overview of the model’s decision process, revealing which features, or events in the case of TimeSHAP, are relevant to the model and which ones are not.

Original: This provides an insight into the model decision process guiding a data scientist analysis of the model.
Revised: This provides an insight into the model decision process guiding a data scientist analysis of the model.

Original: To obtain these global explanations, TimeSHAP is used to explain N sequences and drawing conclusions from aggregations and visualizations of the N local explanations.
Revised: To obtain these global explanations, TimeSHAP is used to explain N sequences and drawing conclusions from aggregations and visualizations of the N local explanations.

Original: To obtain global explanations presented in this work, 1 K randomly sampled positive-predicted sequences were explained using TimeSHAP and aggregations on both the event-wise and feature-wise
Revised: To obtain global explanations presented in this work, 1 K randomly sampled positive-predicted sequences were explained using TimeSHAP and aggregations on both the event-wise and feature-wise

Original: By analysing the global event-wise explanations, we conclude that
Revised: By analysing the global event-wise explanations, we conclude that

Original: the latest event ( t = 0 ) has, on average, the highest event importance throughout each sequence, with an average Shapley value of 0 . 28 .
Revised: the latest event ( t = 0 ) has, on average, the highest event importance throughout each sequence, with an average Shapley value of 0 . 28 .

Original: At the same time, we observe that events between indices − 1 and − 5 are often of high importance as well, with average contributions ranging from 0 .
Revised: At the same time, we observe that events between indices − 1 and − 5 are often of high importance as well, with average contributions ranging from 0 .

Original: 03 to 0 .
Revised: 03 to 0 .

Original: From analyzing global feature-wise explanations, we conclude that the most relevant features for this model, having a predominately positive contribution to the score ( i.e. , serving as fraud indicators), are related to the transaction type (average contribution 0 . 28 ), the clients’ age ( 0 . 092 ), the event type ( 0 . 09 ), and related to location and IP ( 0 . 08 ).
Revised: From analyzing global feature-wise explanations, we conclude that the most relevant features for this model, having a predominately positive contribution to the score ( i.e. , serving as fraud indicators), are related to the transaction type (average contribution 0 . 28 ), the clients’ age ( 0 . 092 ), the event type ( 0 . 09 ), and related to location and IP ( 0 . 08 ).

Original: These features are in accordance with our domain knowledge of account takeover, where the event and transaction type together with the location and
Revised: These features are in accordance with our domain knowledge of account takeover, where the event and transaction type together with the location and

Original: IP features encode account behaviors, and the age of the client can, in most cases, indicate a possibly vulnerable accounts.
Revised: IP features encode account behaviors, and the age of the client can, in most cases, indicate a possibly vulnerable accounts.

Original:  Conclusion
Revised:  Conclusion

Original: While considerable effort has been guided towards explaining deep learning models, recurrent models have received comparatively little attention.
Revised: While considerable effort has been guided towards explaining deep learning models, recurrent models have received comparatively little attention.

Original: With this in mind, we presented TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP’s sound theoretical footing and strong empirical results.
Revised: With this in mind, we presented TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP’s sound theoretical footing and strong empirical results.

Original: TimeSHAP carries three main contributions: (1) a new perturbation function suited for the recurrent setting, (2) explanations of both feature- and event-wise importance, and (3) a sequence pruning method that dramatically increases the method’s efﬁciency in practise.
Revised: TimeSHAP carries three main contributions: (1) a new perturbation function suited for the recurrent setting, (2) explanations of both feature- and event-wise importance, and (3) a sequence pruning method that dramatically increases the method’s efﬁciency in practice.

Original: TimeSHAP enables data scientists to see which features were most important for a model’s prediction, but also which past events impact the current prediction the most.
Revised: TimeSHAP enables data scientists to see which features were most important for a model’s prediction, but also which past events impact the current prediction the most.

Original: We apply our method on a largescale real-world dataset on account takeover fraud.
Revised: We apply our method on a largescale real-world dataset on account takeover fraud.

Original: We notice potentially discriminatory reasoning based on the client’s age, as it is shown to increase the model’s fraud score on several occasions.
Revised: We notice potentially discriminatory reasoning based on the client’s age, as it is shown to increase the model’s fraud score on several occasions.

Original: Overall, we ﬁnd multiple instances whose explanations are crucial for debugging the underlying predictor.
Revised: Overall, we ﬁnd multiple instances whose explanations are crucial for debugging the underlying predictor.

Original: Acknowledgements
Revised: Acknowledgements

Original: The project CAMELOT (reference POCI-01-0247-FEDER-045915) leading to this work is coﬁnanced by the ERDF - European Regional Development Fund through the Operational Program for
Revised: The project CAMELOT (reference POCI-01-0247-FEDER-045915) leading to this work is coﬁnanced by the ERDF - European Regional Development Fund through the Operational Program for

Original: Competitiveness and Internationalisation - COMPETE 2020, the North Portugal Regional Operational
Revised: Competitiveness and Internationalisation - COMPETE 2020, the North Portugal Regional Operational

Original: Program - NORTE 2020 and by the Portuguese Foundation for Science and Technology - FCT under the CMU Portugal international partnership
Revised: Program - NORTE 2020 and by the Portuguese Foundation for Science and Technology - FCT under the CMU Portugal international partnership