Original: S WITCHED LINEAR PROJECTIONS AND INACTIVE STATE SENSITIVITY FOR DEEP NEURAL NETWORK INTERPRETABILITY
Revised: S WITCHED LINEAR PROJECTIONS AND INACTIVE STATE SENSITIVITY FOR DEEP NEURAL NETWORK INTERPRETABILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of a single linear projection in the input space.
Revised: We introduce switched linear projections for expressing the activity of a neuron in a ReLU-based deep neural network in terms of a single linear projection in the input space.

Original: The method works by isolating the active subnetwork, a series of linear transformations, that completely determine the entire computation of the deep network for a given input instance.
Revised: The method works by isolating the active subnetwork, a series of linear transformations, that completely determine the entire computation of the deep network for a given input instance.

Original: We also propose that for interpretability it is instructive and meaningful to focus on the patterns that deactive the neurons in the network
Revised: We also propose that for interpretability it is instructive and meaningful to focus on the patterns that deactive the neurons in the network

Original: , which are ignored by the exisiting methods that implicitly track only the active aspect of the network’s computation.
Revised: , which are ignored by the exisiting methods that implicitly track only the active aspect of the network’s computation.

Original: We introduce a novel interpretability method for the inactive state sensitivity (Insens).
Revised: We introduce a novel interpretability method for the inactive state sensitivity (Insens).

Original: Comparison against existing methods shows that Insens is robust (in the presence of noise), complete (in terms of patterns that affect the computation) and a very effective interpretability method for deep neural networks
Revised: Comparison against existing methods shows that Insens is robust (in the presence of noise), complete (in terms of patterns that affect the computation) and a very effective interpretability method for deep neural networks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It is notoriously hard to interpret how deep networks accomplish the tasks for which they are trained.
Revised: It is notoriously hard to interpret how deep networks accomplish the tasks for which they are trained.

Original: At the same time, due to the pervasiveness of deep learning in numerous aspects of computing, it is increasingly important to gain understanding of how they work.
Revised: At the same time, due to the pervasiveness of deep learning in numerous aspects of computing, it is increasingly important to gain understanding of how they work.

Original: There are risks associated with the possibility that a neural network might not be “looking” at the “right” patterns (Nguyen et al.
Revised: There are risks associated with the possibility that a neural network might not be “looking” at the “right” patterns (Nguyen et al.

Original: ; Geirhos et al., 2019), as well as opportunities to learn from the network’s capable of better than human performance (Sadler & Regan, 2019).
Revised: ; Geirhos et al., 2019), as well as opportunities to learn from the network’s capable of better than human performance (Sadler & Regan, 2019).

Original: Hence, there is ongoing effort to improve the interpretation and interpretability of the internal representation of neural networks.
Revised: Hence, there is ongoing effort to improve the interpretation and interpretability of the internal representation of neural networks.

Original: What makes this interpretation of the inside of a neural network hard is the high dimensionality and the distributed nature of its internal computation.
Revised: What makes this interpretation of the inside of a neural network hard is the high dimensionality and the distributed nature of its internal computation.

Original: Aside from the ﬁrst hidden layer, neurons operate in an abstract high-dimensional space.
Revised: Aside from the ﬁrst hidden layer, neurons operate in an abstract high-dimensional space.

Original: If that was not hard enough, the analysis of individual components of the network (such as activity of individual neurons) is rarely instructive, since it is the intricate relationships and interplay of those components that contain the “secret sauce”.
Revised: If that was not hard enough, the analysis of individual components of the network (such as activity of individual neurons) is rarely instructive, since it is the intricate relationships and interplay of those components that contain the “secret sauce”.

Original: The two broad approaches to dealing with this complexity is to either use simpler interpretable models to approximate what a neural network does, or to trace back the elements of the computation into the input space in order to make the internal dynamics relatable to the input.
Revised: The two broad approaches to dealing with this complexity is to either use simpler interpretable models to approximate what a neural network does, or to trace back the elements of the computation into the input space in order to make the internal dynamics relatable to the input.

Original: In the latter approach we are typically interested in neurons’ sensitivity – how the changes in network input affect their output, and decomposition – how different components of the input contribute to the output.
Revised: In the latter approach we are typically interested in neurons’ sensitivity – how the changes in network input affect their output, and decomposition – how different components of the input contribute to the output.

Original: In this paper we propose a straightforward and elegant method for expressing the computation of an arbitrary neuron’s activity to a single linear projection in the input space.
Revised: In this paper we propose a straightforward and elegant method for expressing the computation of an arbitrary neuron’s activity to a single linear projection in the input space.

Original: This projection consists of a switched weight vector and a switched bias that easily lend themselves to sensitivity analysis (analogous to gradient-based sensitivity) and decomposition of the internal computation.
Revised: This projection consists of a switched weight vector and a switched bias that easily lend themselves to sensitivity analysis (analogous to gradient-based sensitivity) and decomposition of the internal computation.

Original: We also introduce a new approach for interpretability analysis, called inactive state sensitivity (Insens), which uses switched linear projections to aggregate the contribution of patterns in the input that deactivate neurons in the network.
Revised: We also introduce a new approach for interpretability analysis, called inactive state sensitivity (Insens), which uses switched linear projections to aggregate the contribution of patterns in the input that deactivate neurons in the network.

Original: We demonstrate on several networks and image-based datasets that Insens provides a comprehensive picture of a deep network’s internal computation.
Revised: We demonstrate on several networks and image-based datasets that Insens provides a comprehensive picture of a deep network’s internal computation.

Original: The only constraint for the proposed methods is that the network must use ReLU activation functions for its hidden neurons.
Revised: The only constraint for the proposed methods is that the network must use ReLU activation functions for its hidden neurons.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Previous work on deep learning interpretability is extensive with a wide variety of methods and approaches – Simonyan et al.
Revised: Previous work on deep learning interpretability is extensive with a wide variety of methods and approaches – Simonyan et al.

Original: ; Zeiler & Fergus; Bach et al. (2015); Mahendran & Vedaldi;
Revised: ; Zeiler & Fergus; Bach et al. (2015); Mahendran & Vedaldi;

Original: Montavon et al.
Revised: Montavon et al.

Original: ; Sundararajan et al.
Revised: ; Sundararajan et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2019) being just a selection of the most prominent efforts in this area.
Revised: (2019) being just a selection of the most prominent efforts in this area.

Original: Our work on the single linear projection follows the approach akin to Lee et al.
Revised: Our work on the single linear projection follows the approach akin to Lee et al.

Original: and Erhan et al.
Revised: and Erhan et al.

Original: (2009), where the objective was to interpret the computation performed by an arbitrary neuron for a particular input vector as a projection in the input space.
Revised: (2009), where the objective was to interpret the computation performed by an arbitrary neuron for a particular input vector as a projection in the input space.

Original: However, whereas these previous attempts were based on Deep Belief Nets (Hinton et al., 2006) and required an approximation of the said projection, our method is a forward computation that gives the neuron’s activity in terms of a linear projection in the input space.
Revised: However, whereas these previous attempts were based on Deep Belief Nets (Hinton et al., 2006) and required an approximation of the said projection, our method is a forward computation that gives the neuron’s activity in terms of a linear projection in the input space.

Original: It works for any neural network, including convolutional ones, as long as all hidden neurons use piecewise linear activation functions.
Revised: It works for any neural network, including convolutional ones, as long as all hidden neurons use piecewise linear activation functions.

Original: All existing methods for interpretability of deep learning, due to the nature of ReLU computation, necessarily provide information only about the active subnetwork of the ReLU-based architecture.
Revised: All existing methods for interpretability of deep learning, due to the nature of ReLU computation, necessarily provide information only about the active subnetwork of the ReLU-based architecture.

Original: Our own observations, as well as other evidence showing that in practice neural networks produce a relatively low number of activation regions (Hanin & Rolnick, 2019), lead us to the hypothesis that the analysis of the patterns in the input that switch neurons off gives an excellent picture of a network’s sensitivity.
Revised: Our own observations, as well as other evidence showing that in practice neural networks produce a relatively low number of activation regions (Hanin & Rolnick, 2019), lead us to the hypothesis that the analysis of the patterns in the input that switch neurons off gives an excellent picture of a network’s sensitivity.

Original: We also take the view that too much interpretation in interpretability introduces the risk of showing us what we expect to see and not what the network is actually focussing on.
Revised: We also take the view that too much interpretation in interpretability introduces the risk of showing us what we expect to see and not what the network is actually focussing on.

Original: For instance, in Deep Taylor Decomposition (Montavon et al., 2017) choices of different root-points for the decomposition of the relevance function lead to different rules for Layerwise Relevance Propagation (LRP)(Bach et al., 2015), which can lead to different interpretations of what is important in the input.
Revised: For instance, in Deep Taylor Decomposition (Montavon et al., 2017) choices of different root-points for the decomposition of the relevance function lead to different rules for Layerwise Relevance Propagation (LRP)(Bach et al., 2015), which can lead to different interpretations of what is important in the input.

Original: The LRP- α 1 β 0 rule, for example, emphasises the computation over the positive weights in the network while discounting the relevance of the information passing through the negative weights.
Revised: The LRP- α 1 β 0 rule, for example, emphasises the computation over the positive weights in the network while discounting the relevance of the information passing through the negative weights.

Original: This rule is justiﬁed by assumptions about desired properties of the explanation, but this comes with a risk of analysis that gives the answer we want to get and not what is truly happening inside the network.
Revised: This rule is justiﬁed by assumptions about desired properties of the explanation, but this comes with a risk of conﬁrmation bias.

Original: Insens is an attempt to take into account the patterns in the input that cause the neurons inside the network to produce zero output.
Revised: Insens is an attempt to take into account the patterns in the input that cause the neurons inside the network to produce zero output.

Original: The information related to the inactive network may seem irrelevant, since inactive neurons do not directly contribute to the computation of the overall output.
Revised: The information related to the inactive network may seem irrelevant, since inactive neurons do not directly contribute to the computation of the overall output.

Original: However, there is something in the input that switches a particular set of neurons off, thus regulating the active computation, and as we show in this paper, this something carries a lot of meaningful information.
Revised: However, there is something in the input that switches a particular set of neurons off, thus regulating the active computation, and as we show in this paper, this something carries a lot of meaningful information.

Original:  S WITCHED LINEAR PROJECTIONS
Revised:  S WITCHED LINEAR PROJECTIONS

Original: The basis of the switched network concept is the fact that neurons that produce output of zero do not contribute to the computation of the overall output of the network.
Revised: The basis of the switched network concept is the fact that neurons that produce output of zero do not contribute to the computation of the overall output of the network.

Original: The notion of dead neurons, that is neurons that always output zero, is not new, nor is the realisation that these neurons, along with their connecting weights, can be taken out the network without any impact on the computation.
Revised: The notion of dead neurons, that is neurons that always output zero, is not new, nor is the realisation that these neurons, along with their connecting weights, can be taken out the network without any impact on the computation.

Original: In a switched projection, we treat the zero-output neurons as temporarily dead for a given instance of input.
Revised: In a switched projection, we treat the zero-output neurons as temporarily dead for a given instance of input.

Original: We refer to these neurons as inactive , since they may become active for a different network input.
Revised: We refer to these neurons as inactive , since they may become active for a different network input.

Original: Thus we isolate the subnetwork of the active neurons in a given computation.
Revised: Thus we isolate the subnetwork of the active neurons in a given computation.

Original: As it happens, for a ReLU neuron the active neurons are those that pass their activity, the weighted sum of its inputs plus bias, directly to its output 1 .
Revised: As it happens, for a ReLU neuron the active neurons are those that pass their activity, the weighted sum of its inputs plus bias, directly to its output 1 .

Original: This means that a subnetwork of active ReLU neurons is just a series of linear transformations, which is equivalent to a single linear transformation.
Revised: This means that a subnetwork of active ReLU neurons is just a series of linear transformations, which is equivalent to a single linear transformation.

Original: As a result, we can express the computation performed by any neuron in a ReLU network as a projection onto a switched weight vector in the input space plus the switched bias.
Revised: As a result, we can express the computation performed by any neuron in a ReLU network as a projection onto a switched weight vector in the input space plus the switched bias.

Original: The term switched indicates that this weight and bias vector changes when the state of the network changes, the state corresponding to the particular combination of the active and inactive neurons in the network.
Revised: The term switched indicates that this weight and bias vector changes when the state of the network changes, the state corresponding to the particular combination of the active and inactive neurons in the network.

Original: Figure 1 illustrates the concept graphically, and a formal description is given in the following theorem:
Revised: Figure 1 illustrates the concept graphically, and a formal description is given in the following proposition:

Original: Theorem 1 (Switched linear projections) .
Revised: Proposition 1 (Switched linear projections) .

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be a vector of inputs,
Revised: d be a vector of inputs,

Original: w li ∈ R
Revised: w li ∈ R

Original: U l − 1 the weight vector, and b li ∈ R the bias of neuron i in layer l (with U l − 1 inputs from the previous layer).
Revised: U l − 1 the weight vector, and b li ∈ R the bias of neuron i in layer l (with U l − 1 inputs from the previous layer).

Original: Let the activity of a neuron i in layer l be deﬁned as:
Revised: Let the activity of a neuron i in layer l be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (1) where W l = (cid:2) w Tl. . .
Revised: (1) where W l = (cid:2) w Tl. . .

Original: w TlU l (cid:3) , T denotes transpose, b l = [ b l. . .
Revised: w TlU l (cid:3) , T denotes transpose, b l = [ b l. . .

Original: b lU l ] and σ r ( v ) =
Revised: b lU l ] and σ r ( v ) =

Original: max( v, 0) is the ReLU activation function.
Revised: max( v, 0) is the ReLU activation function.

Original: If we deﬁne an input-dependent state of the network as
Revised: If we deﬁne an input-dependent state of the network as

Original: [Equation]
Revised: [Equation]

Revised: and

Original: [Equation]
Revised: [Equation]

Original:  x ) (cid:1) b l. . .
Revised: (cid:0) x ) (cid:1) b l. . .

Original: ˙ σ r (cid:0) v lU l ( x ) (cid:1) b lU l (cid:3) , where ˙ σ r ( v ) =
Revised: ˙ σ r (cid:0) v lU l ( x ) (cid:1) b lU l (cid:3) , where ˙ σ r ( v ) =

Original: dσ r ( v ) dv , then for (cid:98) w Tli ( x ) =
Revised: dσ r ( v ) dv , then for (cid:98) w Tli ( x ) =

Original: W ( x ) 1 W ( x ) 2 . . .
Revised: W ( x ) 1 W ( x ) 2 . . .

Original: W ( x ) l − 1 w Tli and (cid:98) b li ( x ) =
Revised: W ( x ) l − 1 w Tli and (cid:98) b li ( x ) =

Original: b ( x ) 1 W ( x ) 2 . . .
Revised: b ( x ) 1 W ( x ) 2 . . .

Original: W ( x ) l − 1 w Tli + b ( x ) 2 W ( x ) 3 . . . W ( x ) l − 1 w Tli 
Revised: W ( x ) l − 1 w Tli + b ( x ) 2 W ( x ) 3 . . . W ( x ) l − 1 w Tli + . . .

Original: [Equation]

Original: + b l − 1 w Tli [Equation] b li , we have
Revised: + b l − 1 w Tli + b li , we have

Original: [Equation]
Revised: [Equation]

Original: The proof is provided in Appendix A. Note that the ReLU derivative, ˙ σ r ( v ) , is just a convenient deﬁnition for a step function, so
Revised: The proof is provided in Appendix A. Note that the ReLU derivative, ˙ σ r ( v ) , is just a convenient deﬁnition for a step function, so

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: To simplify the notation, whenever referring to the parameters of the switched projection (cid:98) w , (cid:98) b , as well as activity v , we will drop the explicit dependency on x .
Revised: To simplify the notation, whenever referring to the parameters of the switched projection [Equation] b , as well as activity v , we will drop the explicit dependency on x .

Original: While Figure 1 illustrates the switching concepts on a small fully connected network, switched linear projections can be computed for networks with convolutional as well as pooling layers.
Revised: While Figure 1 illustrates the switching concepts on a small fully connected network, switched linear projections can be computed for networks with convolutional as well as pooling layers.

Original: A convolutional layer is just a special case of a fully connected layer with many weights being zero and groups of neurons constrained to share the weight values on their connections.
Revised: A convolutional layer is just a special case of a fully connected layer with many weights being zero and groups of neurons constrained to share the weight values on their connections.

Original: For max pooling, the neurons that do not win the competition, and thus their output does not affect the computation from then on, are deemed to be inactive regardless of the output they produce.
Revised: For max pooling, the neurons that do not win the competition, and thus their output does not affect the computation from then on, are deemed to be inactive regardless of the output they produce.

Original: 3. S ENSITIVITY
Revised: 3. S ENSITIVITY

Original: Equation 2 makes it obvious that a given neuron’s switched weight vector is just the derivative of its activity with respect to the network input, (cid:98) w = ∂v ( x ) x .
Revised: Equation 2 makes it obvious that a given neuron’s switched weight vector is just the derivative of its activity with respect to the network input, (cid:98) w = ∂v ( x ) x .

Original: Thus, the switched weight vector is analogous to gradient-based sensitivity analysis.
Revised: Thus, the switched weight vector is analogous to gradient-based sensitivity analysis.

Original: Figure 2 shows the heatmaps of the switched weight sensitivity for the same set of hidden neurons with different inputs from the MNIST-trained 2CONV neural network (for details on the network architecture featured in this paper see Appendix
Revised: Figure 2 shows the heatmaps of the switched weight sensitivity for the same set of hidden neurons with different inputs from the MNIST-trained 2CONV neural network (for details on the network architecture featured in this paper see Appendix

Original: B).
Revised: B).

Original: In this visualisation we show normalised (cid:98) w , with intensity of red corresponding to the larger positive value, and the intensity of blue the negative value.
Revised: In this visualisation we show normalised (cid:98) w , with intensity of red corresponding to the larger positive value, and the intensity of blue the negative value.

Original: The neurons were chosen from the 2nd 
Revised: The neurons were chosen from the 2nd convolutional layer and the penultimate fully connected layer respectively such that for the four

Original: [Figure]
Revised: [Table]

Revised: 0 .v

Revised: = 39 .b

Revised: b = − 4 .v

Revised: = 21 .b

Revised: b = − 2 .v

Revised: = 28 .b

Revised: b = − 3 .v

Revised: = − 4 .b

Revised: b = 0 .v

Revised: = 88 .b

Revised: b = 5 .v = − 122 .b

Revised: b = 7 .v

Revised: = 399 .b

Revised: b = − 39 .v

Revised: = 180 .b

Revised: b = − 22 .v

Original: convolutional layer and the penultimate fully connected layer respectively such that for the four considered inputs some neurons were always active, some inactive, and others sometimes active and sometimes not.
Revised: = − 14 .b b = 3 . + b (cid:98) b considered inputs some neurons were always active, some inactive, and others sometimes active and sometimes not.

Original: Figure 2 makes it clear that a given neuron is not necessarily sensitive to the same pattern for different network inputs – this is most evident in the sensitivity of the neurons of the fully connected layer.
Revised: Figure 2 makes it clear that a given neuron is not necessarily sensitive to the same pattern for different network inputs – this is most evident in the sensitivity of the neurons of the fully connected layer.

Original: Also note that some neurons in the convolutional layers are active despite the fact that they only “see” the part of the input image that is “empty” (all pixels are black).
Revised: Also note that some neurons in the convolutional layers are active despite the fact that they only “see” the part of the input image that is “empty” (all pixels are black).

Original: This leads us to the conclusion that a given neuron is not necessarily a detector of a particular pattern in the input space, which is often the underlying assumption of the existing interpretability techniques.
Revised: This leads us to the conclusion that a given neuron is not necessarily a detector of a particular pattern in the input space, which is often the underlying assumption of the existing interpretability techniques.

Original: Something also apparent in our switched projection analysis, though not evident from Figure 2, is that for a given input most of the neurons in the network are inactive.
Revised: Something also apparent in our switched projection analysis, though not evident from Figure 2, is that for a given input most of the neurons in the network are inactive.

Original: On average, only 17% of the neurons were active for a given MNIST input in this architecture.
Revised: On average, only 17% of the neurons were active for a given MNIST input in this architecture.

Original: The fact that only a subset of neurons are active in a given computation is not a quirk of one speciﬁc network, as observed by (Hanin & Rolnick, 2019).
Revised: The fact that only a subset of neurons are active in a given computation is not a quirk of one speciﬁc network, as observed by (Hanin & Rolnick, 2019).

Original: Switched linear projections give us an interpretation of a deep network as a set of linear, input-dependent, transformations.
Revised: Switched linear projections give us an interpretation of a deep network as a set of linear, input-dependent, transformations.

Original: Something about the input activates a subset of neurons, but also keeps all the remaining neurons inactive.
Revised: Something about the input activates a subset of neurons, but also keeps all the remaining neurons inactive.

Original: Traditional sensitivity analysis, as well as the one shown in Figure 2, shows a direction (or magnitude) of the gradient of the input that would increase neurons activity provided the same state of the network remained unchanged.
Revised: Traditional sensitivity analysis, as well as the one shown in Figure 2, shows a direction (or magnitude) of the gradient of the input that would increase neurons activity provided the same state of the network remained unchanged.

Original: However, often in these networks the state does change even after small perturbations of the input, often resulting in same classiﬁcation, but different input gradient.
Revised: However, often in these networks the state does change even after small perturbations of the input, often resulting in same classiﬁcation, but different input gradient.

Original: We reason that the analysis of the state, including information about what makes the neurons inactive, provides a missing and likely very important aspect, to interpretability than analysis of the active subnetwork alone.
Revised: We reason that the analysis of the state, including information about what makes the neurons inactive, provides a missing and likely very important aspect, to interpretability than analysis of the active subnetwork alone.

Original: 3. D ECOMPOSITION
Revised: 3. D ECOMPOSITION

Original: Switched linear projections can decompose the activity of a neuron into contributions from its input, in our case the pixels.
Revised: Switched linear projections can decompose the activity of a neuron into contributions from its input, in our case the pixels.

Original: For this we propose another re-interpretation of the computation of the output that will allow us to distribute the bias over the attributes of the input vector.
Revised: For this we propose another re-interpretation of the computation of the output that will allow us to distribute the bias over the attributes of the input vector.

Original: Note that for a linear projection
Revised: Note that for a linear projection

Original: [Equation]
Revised: [Equation]

Original: where c = x − v ww T w , b = − cw T , and c ∈ R d .
Revised: where c = x − v ww T w , b = − cw T , and c ∈ R d .

Original: The vector c can be thought of as a translation of the coordinate system to a neuron-centered one, where w goes through the origin at c .
Revised: The vector c can be thought of as a translation of the coordinate system to a neuron-centered one, where w goes through the origin at c .

Original: Montavon et al.
Revised: Montavon et al.

Original: (2017) call this vector the nearest root point , but we will refer to it as the neuron’s centre .
Revised: (2017) call this vector the nearest root point , but we will refer to it as the neuron’s centre .

Original: Figure 3 provides a geometric interpretation of c in a simple 2D scenario.
Revised: Figure 3 provides a geometric interpretation of c in a simple 2D scenario.

Original: Since c ∈ R d , we can break down the computation of v such that
Revised: Since c ∈ R d , we can break down the computation of v such that

Original: [Equation]
Revised: [Equation]

Original: and take ( x j − c j ) w j to be the contribution of the input component j (in our examples, a single pixel) to the computation of neuron’s activity.
Revised: and take ( x j − c j ) w j to be the contribution of the input component j (in our examples, a single pixel) to the computation of neuron’s activity.

Original: Since the switched linear projection is equivalent to weights and bias of a single neuron, we can compute the switched centre
Revised: Since the switched linear projection is equivalent to weights and bias of a single neuron, we can compute the switched centre

Original: [Equation]
Revised: [Equation]

Original: The switched centre (cid:98) c can be thought of as a concept related to the  reference  subtracted from the input in DeepLIFT ? .
Revised: The switched centre (cid:98) c is related to the concept of reference in DeepLIFT (Shrikumar et al., 2017) that gets subtracted from the input in order to extract a pattern of interest.

Original: However, whereas in DeepLIFT the centre is external to the model, and used for examination of perturbations it induces in the output, our proposed centre is a component of the actual computation of the network’s output; one could say, (cid:98) c is a given neuron’s inherent reference .
Revised: However, whereas in DeepLIFT the reference is external to the model, and used for examination of perturbations it induces in the output, our proposed centre is a component of the actual computation of the network’s output; one could say, [Equation] c is a given neuron’s inherent reference .

Original: Visualisations based on decomposition of active neurons do not offer anything more than existing methods.
Revised: Visualisations based on decomposition of active neurons do not offer anything more than existing methods.

Original: However, the concept of neuron’s centre will be used in the method we propose next, for interpretability based on the sensitivity of the inactive network.
Revised: However, the concept of neuron’s centre will be used in the method we propose next, for interpretability based on the sensitivity of the inactive network.

Original:  I NACTIVE STATE SENSITIVITY
Revised:  I NACTIVE STATE SENSITIVITY

Original: Since a switched linear projection can be found for any neuron in the network, it can just as easily relate what it is about the input that drives a neuron into the negative just as well as positive.
Revised: Since a switched linear projection can be found for any neuron in the network, it can just as easily relate what it is about the input that drives a neuron into the negative just as well as positive.

Original: Tracking the patterns in the input that make the neuron more inactive tells us about the aspects of the input that would ensure the stability of the network’s state.
Revised: Tracking the patterns in the input that make the neuron more inactive tells us about the aspects of the input that would ensure the stability of the network’s state.

Original: The bigger the magnitude of the negative activity, the less likely the inactive neurons are to switch on, and thus change the switched projection.
Revised: The bigger the magnitude of the negative activity, the less likely the inactive neurons are to switch on, and thus change the switched projection.

Original: Some of the inactive neurons are closer and others further away from the point where they would activate.
Revised: Some of the inactive neurons are closer and others further away from the point where they would activate.

Original: Hence, we propose a deﬁnition of inactive sensitivity based on switched linear projection that takes the magnitude of activity into account,
Revised: Hence, we propose a deﬁnition of inactive sensitivity based on switched linear projection that takes the magnitude of activity into account,

Original: [Equation]
Revised: [Equation]

Original: where v i = ( x − (cid:98) c i ) (cid:98) w Ti . The difference x − (cid:98) c i can be thought of as the component of the input that projects onto a neuron’s switched weight vector (cid:98) w i , and thus is responsible for the activity v i .
Revised: The difference x − (cid:98) c i can be thought of as the component of the input that projects onto a neuron’s switched weight vector (cid:98) w i in the coordinate system centred at (cid:98) c i .

Original: Note that (cid:98) ω
Revised: Note that (cid:98) ω

Original: i is still a vector in the direction of (cid:98) w i , and so it is a measure of a neuron’s sensitivity, but its magnitude is proportional to the absolute value of activity.
Revised: i is still a vector in the direction of (cid:98) w i , and so it is a measure of a neuron’s sensitivity, but its magnitude is proportional to the absolute value of activity.

Original: In an attempt to capture the information about the state of the network, we propose averaging the sensitivity of all inactive neurons in a given layer of the network:
Revised: In an attempt to capture the information about the state of the network, we propose averaging the sensitivity of all inactive neurons in a given layer of the network:

Original: [Equation]
Revised: [Equation]

Original: where I l is a set of all inactive neurons, v li ≤ 0 in layer l of the network (excluding the output neurons), and (cid:98) c i is the switched centre of neuron i ; m i is the number of times neuron i is on when the network is evaluated over the set of N training samples.
Revised: where I l is a set of all inactive neurons, v li ≤ 0 in layer l of the network (excluding the output neurons), and [Equation] c i is the switched centre of neuron i ; m i is the number of times neuron i is on when the network is evaluated over the set of N training samples.

Original: We refer to (cid:98) Ω ( x ) l ∈ R d as the inactive state sensitivity (Insens) of the network with respect to the input x .
Revised: We refer to [Equation] as the inactive state sensitivity (Insens) of the network with respect to the input x .

Original: The m i
Revised: The m i

Original: /N
Revised: /N

Original: factor weights the importance of a neuron based on its activity over the training set; a neuron that is never on, m i = 0 , might represent an arbitrary pattern that has no useful information, while the neuron that is usually on, but not for the input for which (cid:98) Ω ( x ) l ∈ R d is computed, is taken to carry more weight.
Revised: factor weights the importance of a neuron based on its activity over the training set; a neuron that is never on, m i = 0 , might represent an arbitrary pattern that has no useful information, while the neuron that is usually on, but not for the input for which [Equation] d is computed, is taken to carry more weight.

Original: 4. E VALUATION
Revised: 4. E VALUATION

Original: We evaluated Insens interpretability against plain gradient sensitivity, Deep Taylor decomposition (Montavon et al., 2017), Integrated gradients (Sundararajan et al., 2017) and Layerwise relevance propagation (Bach et al., 2015) as implemented by the iNNvestigate toolbox (Alber et al., 2018).
Revised: We evaluated Insens interpretability against plain gradient sensitivity, Deep Taylor decomposition (Montavon et al., 2017), Integrated gradients (Sundararajan et al., 2017) and Layerwise relevance propagation (Bach et al., 2015) as implemented by the iNNvestigate toolbox (Alber et al., 2018).

Original: For visualisations of the Insens-based interpretability we simply show (cid:98) Ω ( x ) l as a heatmap, with intensity of the red pixels relating the magnitude of its positive components, and the intensity of blue the magnitude of the negative component of the vector.
Revised: For visualisations of the Insens-based interpretability we simply show [Equation] as a heatmap, with intensity of the red pixels relating the magnitude of its positive components, and the intensity of blue the magnitude of the negative component of the vector.

Original: Since the individual (cid:98) ω i gives a weighted gradient with respect to the input, and thus the change vector that would make neuron i less prone to become active, we take the average (cid:98)Ω ( x ) l ∈ R
Revised: Since the individual (cid:98) ω i gives a weighted gradient with respect to the input, and thus the change vector that would make neuron i less prone to become active, we take the average (cid:98)

Original: [Equation]
Revised: [Equation]

Original:  to be indicative of the pattern in the input that is related to the stability of the network’s state in layaer l induced by x .
Revised: d to be indicative of the pattern in the input that is related to the stability of the network’s state in layaer l induced by x .

Original: In all evaluations we examine visualisation of a 2CONV neural network (described in Appendix B)
Revised: In all evaluations we examine visualisation of a 2CONV neural network (described in Appendix B)

Original: First, we examine visualisations of (cid:98) Ω ( x ) L − 1 ∈ R d , that is the inactive sensitivity over the penultimate layer of the network trained on the MNIST dataset (Lecun et al., 1998).
Revised: First, we examine visualisations of (cid:98) Ω ( x ) L − 1 ∈ R d , that is the inactive sensitivity over the penultimate layer of the network trained on the MNIST dataset (Lecun et al., 1998).

Original: We reason (and conﬁrm below with saliency checks) that the last layer before the softmax output provides most information related to network’s decision making out of all the hidden layers.
Revised: We reason (and conﬁrm below with saliency checks) that the last layer before the softmax output provides most information related to network’s decision making out of all the hidden layers.

Original: Figure 4 shows visualisations from existing methods against the Insens heatmaps for different instances of the input and the same network.
Revised: Figure 4 shows visualisations from existing methods against the Insens heatmaps for different instances of the input and the same network.

Original: In the ﬁrst instance we use clean MNIST input (on which test accuracy of 2CONV is over 99%).
Revised: In the ﬁrst instance we use clean MNIST input (on which test accuracy of 2CONV is over 99%).

Original: Note that Insens visualisation shows something that other methods hint at but do not show explicitly – that the network is sensitive to the black and white contrast of the digit and its outline.
Revised: Note that Insens visualisation shows something that other methods hint at but do not show explicitly – that the network is sensitive to the black and white contrast of the digit and its outline.

Original: The red and blue areas in the Insens heatmaps suggest that respectively lightening and darkening these regions would make the current network state more stable (though in this case it is not possible to make black pixels darker nor white pixels any lighter).
Revised: The red and blue areas in the Insens heatmaps suggest that respectively lightening and darkening these regions would make the current network state more stable (though in this case it is not possible to make black pixels darker nor white pixels any lighter).

Original: To verify the information provided by
Revised: To verify the information provided by

Original: Insens, we added different types of noise to the images, also shown in Figure 4, and used them as input to the same network.
Revised: Insens, we added different types of noise to the images, also shown in Figure 4, and used them as input to the same network.

Original: We wanted to conﬁrm that adding noise to the background, but not the digit outline, as suggested by Insens, does not impact performance.
Revised: We wanted to conﬁrm that adding noise to the background, but not the digit outline, as suggested by Insens, does not impact performance.

Original: Indeed, testing with images that contain random gaussian noise in the background, but not within the 3-pixel outline of the digit (Figure 4 top-right) still gives 87% accuracy over 10000 test images.
Revised: Indeed, testing with images that contain random gaussian noise in the background, but not within the 3-pixel outline of the digit (Figure 4 top-right) still gives 87% accuracy over 10000 test images.

Original: If we use images with random gaussian background everywhere (except the digit itself) or images with the same number of clean pixels as in the 3-pixel outline, but located around the frame of the image (Figure 4 bottom-left and bottom-right), the accuracy drops to 32% and 45% respectively.
Revised: If we use images with random gaussian background everywhere (except the digit itself) or images with the same number of clean pixels as in the 3-pixel outline, but located around the frame of the image (Figure 4 bottom-left and bottom-right), the accuracy drops to 32% and 45% respectively.

Original: This conﬁrms that Insens patterns for the penultimate layer provide meaningful information about the network’s sensitivity.
Revised: This conﬁrms that Insens patterns for the penultimate layer provide meaningful information about the network’s sensitivity.

Original: Also note that the other methods tend to lose the patterns of the digits in the visualisations over noisy images, whereas Insens still shows the digits, to which the network is still sensitive, albeit also being affected by the noise.
Revised: Also note that the other methods tend to lose the patterns of the digits in the visualisations over noisy images, whereas Insens still shows the digits, to which the network is still sensitive, albeit also being affected by the noise.

Original: In Figure 5 we show evaluations of (cid:98) Ω ( x ) L − 1 -based Insens and other methods for two additional 2CONV networks – one trained on smallNORB (LeCun et al.) and the other on CIFAR10 dataset (Krizhevsky, 2009) (for details of the training see Appendix B).
Revised: In Figure 5 we show evaluations of (cid:98) Ω ( x ) L − 1 -based Insens and other methods for two additional 2CONV networks – one trained on smallNORB (LeCun et al.) and the other on CIFAR10 dataset (Krizhevsky, 2009) (for details of the training see Appendix B).

Original: Insens visualisation for the penultimate layer of smallNORB and CIFAR10 datasets might not seem all that different from other methods.
Revised: Insens visualisation for the penultimate layer of smallNORB and CIFAR10 datasets might not seem all that different from other methods.

Original: However, with Insens we can examine the network state of a speciﬁc layer in the network, obtaining information about what the network pays attention to at different stages of its computation.
Revised: However, with Insens we can examine the network state of a speciﬁc layer in the network, obtaining information about what the network pays attention to at different stages of its computation.

Original: In Figure 6 we show Insens visualisation for two convolutional layers ( l = 1 and l = 3 ) as well as the penultimate, fully connected layer ( l = 5) of the 2CONV network trained on smallNORB and
Revised: In Figure 6 we show Insens visualisation for two convolutional layers ( l = 1 and l = 3 ) as well as the penultimate, fully connected layer ( l = 5) of the 2CONV network trained on smallNORB and

Original: CIFAR10.
Revised: CIFAR10.

Original: There are no Insens visualisations for the max-pooling layers, as these will contain only either active, or the arbitrary inactive chosen neurons by the max-pooling operation.
Revised: There are no Insens visualisations for the max-pooling layers, as these will contain only either active, or the arbitrary inactive chosen neurons by the max-pooling operation.

Original: Finally, to provide an objective measure of the quality of Insens visualisations, we perform sanity and saliency  as prescribed by ? .
Revised: Finally, to provide an objective measure of the quality of Insens visualisations, we perform sanity and saliency checks as prescribed by Adebayo et al.

Original: In these tests we measure correlation between interpretability visualisations for different networks; in one instance a 2CONV network trained on a given dataset against an untrained (randomly initialised) network, and in the second instance the same dataset trained network against a network trained on the same data with randomly shufﬂed  labels.
Revised: In these tests we measure correlation between interpretability visualisations for different networks; in each instance a 2CONV network trained on a given dataset against an untrained (randomly initialised) network, and in the second instance the same dataset trained network against a network trained on identical input with randomly shufﬂed target labels.

Original: High correlation between visualisations from the same input between the trained and either randomly initialised or random label trained network suggest that an interpretability method is model agnostic, most likely showing an echo of the input.
Revised: High correlation between visualisations from the same input between the trained and either randomly initialised or random label trained network suggests that an interpretability method is model agnostic, most likely showing an echo of the input.

Original: In Figure 7 we show average Spearman rank-order correlation coefﬁcients between visualisationsfor the 2CONV network derived from a sample ofrandomly chosen images from the CIFAR10 dataset.
Revised: In Figure 7 we show average Spearman rank-order correlation coefﬁcients between visualisations from 20 instances of independently trained 2CONV networks, for each using 100 randomly chosen images from the CIFAR10 dataset.

Original: In the ﬁrst instance, we check the correlation of the penultimate layer-based Insens against other methods.
Revised: In the ﬁrst instance, we check the correlation of the penultimate layer-based Insens against other methods.

Original: Insens correlation is low, on par with the Gradient and Integrated Gradient methods.
Revised: Insens correlation is low, on par with the Gradient and Integrated Gradi- ent methods.

Original: Note that the correlations between visualisation of the DeepTaylor method, which arguably provides the most visually striking visualisations, are very high, suggesting that most of the provided information is independent of the network’s state.
Revised: Note that the correlations between visualisation of the DeepTaylor method, which arguably provides the most visually striking visualisations, are very high, suggesting that most of the provided information is independent of the network’s state.

Original: In the second instance we examine the correlations over Insens from different layers of the 2CONV architecture.
Revised: In the second instance we examine the correlations over Insens from different layers of the 2CONV architecture.

Original: It is very obvious that this correlation increases the closer the layer is to the input.
Revised: It is very obvious that this correlation increases the closer the layer is to the input.

Original: We believe this is not an artefact of the Insens method, but an indication of the initial passing and then gradual ﬁltering of information passing through the network.
Revised: We believe this is not an artefact of the Insens method, but an indication of the initial passing and then gradual ﬁltering of information passing through the network.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The switched linear projection is an interpretation of the computation inside a ReLU network that distinguishes between the active and inactive parts of the deep neural network architecture.
Revised: The switched linear projection is an interpretation of the computation inside a ReLU network that distinguishes between the active and inactive parts of the deep neural network architecture.

Original: The active subnetwork tends to be a smaller subset of the deep network (see Appendix B.1 for details) and the linear projection it provides is somewhat arbitrary, in the sense that it does not matter what the orientation of the switched weight vector is, as long as it produces the desired output.
Revised: The active subnetwork tends to be a smaller subset of the deep network (see Appendix B.1 for details) and the linear projection it provides is somewhat arbitrary, in the sense that it does not matter what the orientation of the switched weight vector is, as long as it produces the desired output.

Original: Hence interpretability analysis based on the active subnetwork may not give the full picture of the patterns from the input that the network relies on in order to produce its computation.
Revised: Hence interpretability analysis based on the active subnetwork may not give the full picture of the patterns from the input that the network relies on in order to produce its computation.

Original: Inactive state sensitivity (Insens), the proposed method for interpretability of ReLU networks, aggregates weighted sensitivity over a layer of inactive neurons.
Revised: Inactive state sensitivity (Insens), the proposed method for interpretability of ReLU networks, aggregates weighted sensitivity over a layer of inactive neurons.

Original: This sensitivity relates the gradient of the input that would potentially drive the activity of the inactive neurons further away from zero, thus corresponding to the patterns in a particular input that would keep the state of the particular layer stable and the output decision the same.
Revised: This sensitivity relates the gradient of the input that would potentially drive the activity of the inactive neurons further away from zero, thus corresponding to the patterns in a particular input that would keep the state of the particular layer stable and the output decision the same.

Original: As the name implies the method isolates the patterns in the input to which the network, in a sense, is insensitive .
Revised: As the name implies the method isolates the patterns in the input to which the network, in a sense, is insensitive .

Original: Our evaluations show that these patterns give a comprehensive picture of what and at what point of the computation the network reacts to patterns in a given input.
Revised: Our evaluations show that these patterns give a comprehensive picture of what and at what point of the computation the network reacts to patterns in a given input.

Revised: It is worth noting that it is possible to extend Insens by summing over all, active and inactive, neurons in Equation 8.

Revised: In our experiments we found that inclusion of active neurons did not change the visualisations in a signiﬁcant way, and we reasoned that the effect of these neurons is already accounted for in the computation of the switched linear projections on which the Insens evaluation is based on.

Revised: However, it might be worth investigating in the future whether there are circumstances, datasets, or types of problems where inclusion of active state affects the sensitivity visualisations.

Original: Since switched linear projections are just an interpretation of theactual computation inside a neural network, they may also become a useful tool for complexity analysis of deep networks.
Revised: Since switched linear projections are just an interpretation of the computation inside a neural network, they may also become a useful tool for complexity analysis of deep networks.

Original: For instance, it might be possible to develop new regularisation methods based on switched weights, biases and centres of the neurons in the network.
Revised: For instance, it might be possible to develop new regularisation methods based on switched weights, biases and centres of the neurons in the network.

Original: It remains to be investigated how the nature of the inactive subnetwork, and potential ways of manipulating it during training, would affect generalisation.
Revised: It remains to be investigated how the nature of the inactive subnetwork, and potential ways of manipulating it during training, would affect generalisation.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X
Revised: We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X

Original: GPU used for this research.
Revised: GPU used for this research.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF P ROPOSITION

Original: Theorem 1 (Switched linear projections) .
Revised: Proposition 1 (Switched linear projections) .

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be a vector of inputs,
Revised: d be a vector of inputs,

Original: w li ∈ R
Revised: w li ∈ R

Original: U l − 1 the weight vector, and b li ∈ R the bias of neuron i in layer l (with U l − 1 inputs from the previous layer).
Revised: U l − 1 the weight vector and b li ∈ R the bias of neuron i in layer l (with U l − 1 inputs from the previous layer)

Original: Let the activity of a neuron i in layer l be deﬁned as:
Revised: Let the activity of a neuron i in layer l be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (1) where W l = (cid:2) w Tl
Revised: [Equation] (cid:16) . . [Equation] (cid:0)

Original: [Equation]
Revised: [Equation]

Original:  w TlU l (cid:3) , T denotes transpose, b l = [ b l. . .
Revised: [Equation] (cid:1) . . . (cid:17) w li + b li , (1) where W l = (cid:2) w Tl. . . w TlU l (cid:3) , T denotes transpose, b l = [ b l. . .

Original: b lU l ] and σ r ( v ) =
Revised: b lU l ] and σ r ( v ) =

Original: max( v, 0) is the ReLU activation function.
Revised: max( v, 0) is the ReLU activation function.

Original: If we deﬁne an input-dependent state of the network as
Revised: If we deﬁne an input-dependent state of the network as

Original: [Equation]
Revised: [Equation]

Original:  and [Equation] ˙
Revised: ˙ σ r (cid:0) v l 1 ( x ) (cid:1) w Tl. . . ˙ σ r (cid:0) v lU l ( x ) (cid:1) w TlU l (cid:3) and b ( x ) l = (cid:2) ˙

Original: σ r ( v l 1 (cid:0) x ) (cid:1) b l. . .
Revised: σ r ( v l(cid:0) x ) (cid:1) b l. . .

Original: ˙ σ r (cid:0) v lU l ( x ) (cid:1) b lU l (cid:3) , where ˙ σ r ( v ) =
Revised: ˙ σ r (cid:0) v lU l ( x ) (cid:1) b lU l (cid:3) , where ˙ σ r ( v ) =

Original: dσ r ( v ) dv , then for (cid:98) w Tli ( x ) =
Revised: dσ r ( v ) dv , then for (cid:98) w Tli ( x ) =

Original: W ( x ) 1 W ( x ) 2 . . .
Revised: W ( x ) 1 W ( x ) 2 . . .

Original: W ( x ) l − 1 w Tli and (cid:98) b li ( x ) =
Revised: W ( x ) l − 1 w Tli and (cid:98) b li ( x ) =

Revised: [Equation]

Original: b ( x ) 1 W ( x ) 2 . . .
Revised: ( x ) 1 W ( x ) 2 . . .

Original: W ( x ) l − 1 w Tli + b ( x ) 2 W ( x ) 3 . . .
Revised: W ( x ) l − 1 w Tli + b ( x ) 2 W ( x ) 3 . . .

Original: W ( x ) l − 1 w Tli [Equation]
Revised: W ( x ) l − 1 w Tli + . . .

Original: [Equation]

Original:  l − 1 w Tli + b li , we have 
Revised: + b l − 1 w Tli + b li , we have v

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By deﬁnition from Equation 1, the activity of neuron i in layer l is
Revised: By deﬁnition from Equation 1, the activity of neuron i in layer l is

Original: [Equation]
Revised: [Equation]

Original: (9) where w lij is the weight on the connection between neuron j of layer l − 1 and neuron i of layer l , and
Revised: (9) where w lij is the weight on the connection between neuron j of layer l − 1 and neuron i of layer l , and

Original: b li is the bias of neuron i in layer l .
Revised: b li is the bias of neuron i in layer l .

Original: Since σ r ( v ) = (cid:26) v v > 0 , otherwise , and ˙ σ r ( v ) =
Revised: Since σ r ( v ) = (cid:26) v v > 0 , otherwise , and ˙ σ r ( v ) =

Original: dσ r ( v ) dv = (cid:26) 1 v >0 otherwise , we have σ r (cid:16) v l − 1 j ( x ) (cid:17)
Revised: dσ r ( v ) dv = (cid:26) 1 v >0 otherwise , we have σ r (cid:16) v l − 1 j ( x ) (cid:17)

Original: w lij = (cid:26) v l − 1 j ( x ) w lij v > 0 , otherwise , and thus σ [Equation]
Revised: w lij = (cid:26) v l − 1 j ( x ) w lij v > 0 , otherwise , and thus σ r (cid:16) v l − 1 k ( x ) (cid:17) w lik = v l − 1 k ( x ) ˙ σ r (cid:16) v l − 1 k ( x ) (cid:17) w lik .

Original: As a result 
Revised: As a result v

Original: [Equation]
Revised: [Equation]

Original:  where A is the set of neurons with activity v > 0 , the active neurons.
Revised: [Table] where A is the set of neurons with activity v > 0 , the active neurons.

Original: Substituting the expression for activity from Equation 10 into its recursive deﬁnition in Equation 9, where v 0 i ( x ) =x i , reveals [Table]
Revised: Substituting the expression for activity from Equation 10 into its recursive deﬁnition in Equation 9, where v 0 i ( x ) =

Original:  that the overall computation is a series of linear transformations of x equivalent to a single linear transformation
Revised: x i , reveals that the overall computation is a series of linear transformations of x equivalent to a single linear transformation

Original: [Equation]
Revised: [Equation]

Original: where(cid:98) w Tli ( x ) = W ( x ) 1 W ( x ) 2 . . W ( x ) l − 1 w Tli and (cid:98) b li ( x ) = b ( x ) 1 W ( x ) 2 . . . W ( x ) l − 1 w Tli + b ( x ) 2 W ( x ) 3 . . .
Revised: where

Original: W ( x ) l − 1 w Tli + . . . + b l − 1 w Tli + b li .
Revised: [Equation]

Original: B 2CONV ARCHITECTURE
Revised: B 2CONV ARCHITECTURE

Original: This network consists of two convolutional layers each followed by a maxpool layer that downsamples the feature map, followed by a single fully connected layer of 512 neurons and K output neurons, where K is the number of classes in the dataset of interest (see Table 1 for details).
Revised: This network consists of two convolutional layers each followed by a maxpool layer that downsamples the feature map, followed by a single fully connected layer of 512 neurons and K output neurons, where K is the number of classes in the dataset of interest (see Table 1 for details).

Original: For training this network we used Adam optimiser minimising the softmax cross-entropy without regularisation.
Revised: For training this network we used Adam optimiser minimising the softmax cross-entropy without regularisation.

Original: In all training a portion of the training set was set aside for validation for early stopping.
Revised: In all training a portion of the training set was set aside for validation for early stopping.

Original: For the ﬁrst, MNIST evaluation, the network was trained on cleaned MNIST data to test accuracy of 99.4%.
Revised: For the ﬁrst, MNIST evaluation, the network was trained on cleaned MNIST data to test accuracy of 99.4%.

Original: The network was then tested on several variant of the MNIST dataset: MNIST Gauss images were given random Gaussian background (test accuracy of 31.7%), MNIST outline images with same Gaussian background except for outline within 3 pixels of the digit (test accuracy of 86.0%), and MNIST frame with Gaussian background and the same number of clean pixels as in the
Revised: The network was then tested on several variant of the MNIST dataset: MNIST Gauss images were given random Gaussian background (test accuracy of 31.7%), MNIST outline images with same Gaussian background except for outline within 3 pixels of the digit (test accuracy of 86.0%), and MNIST frame with Gaussian background and the same number of clean pixels as in the

Original: MNIST outline version of a given image around the frame (test accuracy of 44.9%).
Revised: MNIST outline version of a given image around the frame (test accuracy of 44.9%).

Original: For the second, smallNORB evaluation, the network was trained four times on variants of the dataset differing in the encoding of the images: smallNORB with grayscale intensity given by value between 0 and 1 for black to white pixel respectively, smallNORB neg with grayscale intensity between 0 and 1 for white to black pixel, smallNORB mean with grayscale between -1 and +1 for black to white pixel and smallNORB neg mean with grayscale between -1 and 1 for white to black pixel.
Revised: For the second, smallNORB evaluation, the network was trained four times on variants of the dataset differing in the encoding of the images: smallNORB with grayscale intensity given by value between 0 and 1 for black to white pixel respectively, smallNORB neg with grayscale intensity between 0 and 1 for white to black pixel, smallNORB mean with grayscale between -1 and +1 for black to white pixel and smallNORB neg mean with grayscale between -1 and 1 for white to black pixel.

Original: The classifciation accuracy on these four networks tested on 24300 image of each dataset was respectively 84.3%, 87.2%, 81.4% and 83.6%.
Revised: The classifciation accuracy on these four networks tested on 24300 image of each dataset was respectively 84.3%, 87.2%, 81.4% and 83.6%.

Original: For the third, CIFAR-10 evaluation, a single network was trained on augmented (by random contrast adjustment and crop to a 24x24 size) images of the CIFAR-10 dataset giving test accuracy of 81.0%.
Revised: For the third, CIFAR-10 evaluation, a single network was trained on augmented (by random contrast adjustment and crop to a 24x24 size) images of the CIFAR-10 dataset giving test accuracy of 81.0%.

Original: B. N UMBER OF ACTIVE NUEURONS
Revised: B. N UMBER OF ACTIVE NUEURONS

Original: Table 2 shows the average percentage of neurons that are active during input-output mappings for the trained networks evaluated in this paper.
Revised: Table 2 shows the average percentage of neurons that are active during input-output mappings for the trained networks evaluated in this paper.

Original: C A DVERSARIAL EXAMPLE EVALUATION
Revised: C M ULTIPLE OBJECTS OF INTEREST AND ADVERSARIAL EXAMPLES

Original: We include an evaluation of the penultimate layer Insens and comparison to other methods on adversarial examples.
Revised: We evaluate Insens against other methods on a classiﬁcation task over images with two objects – one to of interest and the another to be ignored. For this experiment we created a dataset where the [Table] input consists of an even and odd MNIST digit concatenated in random order.

Revised: The training label identiﬁes the even digit, making it the object of interest.

Revised: We call this dataset MNISTdbl.

Revised: Figureshows heatmap visualisations for penultimate-layer Insens and other methods for a 2CONV network trained on this dataset to achieve 99.5% test accuracy.

Revised: All the methods seem to emphasise the object of interest mostly by making the even digit brighter and more prominent in the visualisation.

Original: Figure 8 show visualisations from the MNIST and smallNORB trained 2CONV models.
Revised: LRP visualisation seems to also differentiate the two objects by colour – with the object of interest highlighted in red and the other in blue. For a deeper evaluation of these interpretability methods, we obtain visualisations from the same 2CONV network (trained on the MNISTdbl images) using different types of adversarial examples: with only the object to ignore present (Figure 9), two objects of interest in a single image (Figure 10) and two objects to ignore in a single image (Figure 11). Through the same red/blue colour highlighting scheme as in the previous example, LRP-based visualisations tend to communicate rather well which objects in the image are those of interest and which ones are the ones the network has learned to ignore. It does not provide too much explanation of the predictions that the network outputs for these examples.

Revised: DeepTaylor and Integrated Gradients methods highlight the digit used for prediction in the two even digits case.

Revised: Insens visualisation might be providing information about the decisions with respect to the output predictions, but other than the fact that the objects of interest tend to get better outlines, it is hard to say what the nature of these decisions might be.

Revised: Most intuitive expectation is for the predicted digit to show up in the visualisation.

Revised: However, this is making an assumption about the way the network makes its decisions, and, as other adversarial examples show, it’s not necessarily the case that the network perceives the shape of the shape of the digits like we do.

Revised: Finally, we include an evaluation of the penultimate layer Insens and comparison to other methods on gradient-based adversarial examples, where a correctly classiﬁed image has been modiﬁed in the direction of the input gradient obtained from a trained 2CONV network to increase the activity on the output corresponding to the wrong label.

Revised: Figure 12 shows visualisations from different interpretability methods over different adversarial examples derived from the same image for the

Revised: MNIST and smallNORB trained 2CONV networks.

Original: Figure 9 shows the difference between the visualisations from the real and adversarial input.
Revised: Figure 13 shows the difference between the visualisations from the real and adversarial input.

Original: [Table]
Revised: Once again, the predicted digit does not show up in any of the visualisations, though again, Insens does indicate subtle changes, especially in the MNIST examples, with parts of the digits missing and some extra bits showing up.