T HE M INIMAL F EATURE R EMOVAL P ROBLEM IN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present the minimal feature removal problem for neural networks, a combinatorial problem which has interesting potential applications for improving interpretability and robustness of neural network predictions.
For a given input to a trained neural network, our aim is to compute a smallest set of input features so that the model prediction changes when these features are disregarded by setting them to a given uninformative baseline value.
We show that computing such minimal subsets of features is computationally intractable for fully-connected neural networks with ReLU nonlinearities. We show, however, that the problem becomes solvable in polynomial time by a greedy algorithm for monotonic networks.
We then show that our tractability result extends seamlessly to more advanced neural network architectures such as convolutional and graph neural networks under suitable monotonicity assumptions
I NTRODUCTION
Deep Neural Networks have experienced unprecedented success in areas such as image analysis, natural language processing, speech recognition, and data science, with systems outperforming humans in a wide range of tasks Krizhevsky et al.
Hannun et al.
; LeCun et al.
Schmidhuber (2015); Silver et al.
As the use of neural models becomes widespread in complex applications, however, task performance is no longer the only driver of system design, and criteria such as safety, fairness, and robustness have gained significant prominence in recent years
Kazim & Koshiyama (2021).
Improving model interpretability is an important step towards fulfilling these criteria: if models can explain their predictions, it becomes easier to ensure that predictions based on them are safe and fair.
This is, however, notoriously challenging, as neural models are ‘black boxes’ where predictions rely on numeric calculations in high-dimension embedding spaces.
A wealth of different explanation approaches have been proposed in recent years: rule-based methods generate explanations in the form of logic rules, which are inherently interpretable Cucala et al.
; Dhurandhar et al.
(2018); attribution-based methods assign a score to input features quantifying their contribution to the prediction relative to a baseline Sundararajan et al.
; Sundararajan & Najmi (2020); Ancona et al.
; example-based methods explain predictions by retrieving training examples that are most similar to the given input Koh & Liang (2017);
Li et al.
(2018); and perturbation-based methods generate corrections to an input causing the model to change its output Zhang et al.
Goyal et al.
; Lucic et al.
; Bajaj et al.
Evaluating and improving the robustness of neural predictions is also an important and challenging problem Carlini & Wagner (2017);
Hendrycks & Dietterich (2019).
Indeed, neural networks are highly sensitive to a wide range of adversarial attacks, which creates significant vulnerabilities and safety risks when these models are applied in mission-critical applications Szegedy et al.
Katz et al.
Explainability and robustness properties often go hand-in-hand; for instance, attribution-based explanation methods are vulnerable to adversarial attacks since small changes in the input can lead to significant changes on the assigned attribution scores Ghorbani et al.
; Dombrowski et al.
; Yeh et al.
In this paper we introduce the minimal feature removal problem : a combinatorial problem with interesting potential applications for improving the interpretability and robustness of neural model predictions.
Given an (application dependent) baseline representing absence of information and an input feature vector to a trained network, our aim is to compute a smallest set S of input features so that the prediction changes when these features are set to their corresponding baseline value.
The minimality requirement ensures that S can be interpreted as a form of explanation which captures the ‘essence’ of the corresponding prediction by identifying genuinely irrelevant features that can be disregarded without an observable effect.
Furthermore, the number of features in S provides an intuitive measure for the robustness of the prediction since our definition formally guarantees that disregarding any smaller number of input features cannot change the prediction.
Finally, S may also be interpreted as a form of adversarial attack and, in particular, a type of ℓ 0 -perturbation Kotyan &
Vargas (2022), where the goal is to change the prediction by identifying changes to a discrete set of input features; indeed, as discussed in Li et al. (2022); Hosseini et al.
(2019), ℓ 0 -perturbations cannot be seen as a continuous optimisation problem and require a combinatorial approach.
Our approach effectively combines ideas from attribution-based and perturbation-based methods.
As in attribution-based explanation methods, we introduce an uninformative baseline as reference and, similarly to perturbation-based methods for explanation and adversarial attack, our identified set of features constitutes a minimal input correction with an observable effect.
Our notion of feature removal is, however, rather different from related perturbation-based approaches Zhang et al.
Lucic et al.
; Bajaj et al.
; Fong & Vedaldi (2017);
Goyal et al.
(2019) in that the aim is to ‘toggle off’ features using the baseline rather than identifying arbitrary value changes.
Our contributions are as follows.
After introducing the notion of minimal feature removal, we focus on fully-connected networks as a starting point and show that computing minimal feature removal sets is an intractable problem, even with ReLU activations.
If, however, the matrix weights in the network are non-negative and the non-linear activations of the network are continuous, differentiable almost everywhere and monotonic (as is the case with standard activations) we can show that minimal feature removal sets are computable in polynomial time by a greedy algorithm.
To show correctness of our greedy algorithm, we exploit the theoretical properties of the integrated gradients method Sundararajan et al.
(2017), thus establishing an interesting connection between our approach and the theory of attribution methods; our algorithm, however, does not rely on the application of any such attribution method, and only requires the ability to run the neural network as a black box.
We then further generalise our results to more advanced neural architectures such as convolutional neural networks (CNNs) Goodfellow et al.
; Krizhevsky et al.
and graph neural networks (GNNs) Gori et al.
(2005); Kipf & Welling (2017) and show that the problem remains tractable under suitable generalisations of the monotonicity requirement.
Although our intractability result is applicable to a many network architectures used in practice in applications such as image analysis, our greedy algorithm can still be exploited for a wide range of learning tasks where the underpinning function is monotonic Marques-Silva et al. ; Cano et al. ; Cucala et al.
When efficiently computable, minimal feature removal sets may provide interesting insights on the interpretability and the robustness of predictions.
Thus, our algorithm can become a valuable tool for both designers and users of neural networks.
P RELIMINARIES
Notation.
We let bold-face lowercase letters denote real-valued vectors, and typically use x for input feature vectors and x ′ for baselines.
Given vector x , we use x i to denote its i -th component.
Given vectors x , x ′ ∈ R n and a subset S ⊆ { 1 , ..., n } of their components, we denote with x S | x ′ the vector obtained from x by setting each component x i with i ∈ S to x ′ i .
We use bold-face capital letters to denote real-valued matrices and, given matrix M , we denote its ( i, j ) component as M i,j .
Finally, given a function f : R n (cid:55)→ R , we denote with ( ∇ f ) i the i -th component of its gradient.
Fully-Connected Neural Networks.
A fully-connected neural network (FCN) with L ≥ 1 layers and input dimension n is a tuple N = ⟨{ W ℓ } 1 ≤ ℓ ≤ L , { b ℓ } 1 ≤ ℓ ≤ L , { σ ℓ } 1 ≤ ℓ ≤ L } ⟩ .
For each layer ℓ ∈ { 1 , . . .
, L } , the integer d ℓ ∈ N is the width of layer ℓ and we require d L = 1 and define d 0 = n ; matrix W ℓ ∈ R d ℓ × d ℓ − 1 is a weight matrix ; vector b ℓ ∈ R d ℓ is a bias vector ; and σ ℓ : R (cid:55)→ R is a polytime-computable activation function applied component-wise to vectors.
The application of network N to an input feature vector x ∈ R n generates a sequence x 1 , . . .
, x L of vectors defined as x ℓ = σ ℓ ( h ℓ ) , where x 0 = x and h ℓ = W ℓ · x ℓ − 1 + b ℓ .
The result N ( x ) of applying N to x is the scalar x L .
Thus, the neural network realises a function N : R n (cid:55)→ R .
When for each ℓ ∈ { 1 , . . . , L } , σ ℓ is the rectified linear unit (ReLU), i.e. σ ℓ ( x ) = max(0 , x ) , we say that N is a ReLU FCN.
We consider the application of neural networks to classification problems.
To avoid trivialities, we focus on classifying an input in two classes based on a numeric threshold t ∈ R .
In this setting, the prediction of a network N on input x is given by the result of comparing N ( x ) to the threshold t .
B ACKGROUND ON A TTRIBUTION - BASED M ETHODS
Attribution methods Sundararajan et al.
; Sundararajan & Najmi (2020); Shapley (1953) are a family of explanation techniques which, given as input function f : R n (cid:55)→ R , a vector x ∈ R n and a baseline vector x ′ ∈ R n , assign a numerical score or contribution C fi ( x , x ′ ) to each component i ∈ { 1 , ..., n } .
Attribution methods are often designed to fulfil some (or all) of the following axioms for all functions R n (cid:55)→ R and vectors x , x ′ ∈ R n , components 1 ≤ i ≤ n and coefficients λ 1 , λ 2 ∈ R .
• Completeness: f ( x ) − f ( x ′ )
= (cid:80) nj =1 C fj
( x , x ′ ) .
• Zero-contribution: C f i ( x , x ′ )
=
whenever f ( y ) =
f ( y 1 , ..., y i − 1 , z, y i +1 , ...
y n ) for each y ∈ R n and each z ∈ R .
• Symmetry:
C fi ( x , x ′ ) =
C fj ( x , x ′ )
if x i = x j , x ′ i
= x ′ j and f ( y 1 , ..., y i , ..., y j , ...
y n )
=
f ( y 1 , ..., y j , ..., y i , ...
y n ) for each y ∈ R n .
• Linearity: C λ 1 f 1 + λ 2 f 2 i ( x , x ′ )
=
λ 1 C f
i ( x , x ′ )
+ λ 2 C f
i ( x , x ′ ) .
Completeness ensures that contributions add up to the change in value of the function.
Zerocontribution ensures that arguments not influencing the value of the function are assigned 0 as contribution.
Symmetry ensures that arguments playing a symmetric role are assigned the same contribution.
Finally, linearity ensures that contributions for a function expressed as a linear combination of other functions can be computed as a linear combination of their contributions.
A wide range of attribution-based methods have been proposed.
The Shapley values method Shapley (1953) is one of the most popular thanks to its nice properties.
Calculating Shapley values is, however, intractable, which has motivated research on approximation techniques Ancona et al.
Other popular attribution methods have been designed specifically for neural networks; these include Layer-wise Relevance Propagation Bach et al.
DeepLIFT Shrikumar et al.
Deep Taylor decompositions Montavon et al.
(2017), and Saliency Maps Simonyan et al.
Adebayo et al.
(2018); Dabkowski & Gal (2017);
Chang et al.
We will exploit the properties of Integrated Gradients Sundararajan et al.
; Aumann & Shapley (1974), which is an attribution method applicable to continuous functions that are differentiable almost everywhere.
The contribution of each argument i of a such function f for input vector x
and baseline vector x ′ is defined as follows:
[Equation]
Integrated gradients is the only path-based attribution method satisfying all of the aforementioned axioms Friedman (2004).
Furthermore, it is well-suited for functions realised by neural networks, which typically satisfy its continuity and differentiability requirements.
M INIMAL F EATURE R EMOVAL S ETS
In this section, we first present our notion of a feature removal set and then show that computing a minimal such set is an intractable problem for fully-connected neural networks.
Given a trained FCN N providing a prediction on an input feature vector x , we aim at identifying a smallest set of ‘most relevant’ features for the prediction.
Similarly to attribution-based methods, we consider an uninformative baseline x ′ for comparison where the use of a suitable baseline can be exploited to ‘toggle off’ or disregard certain features; however, while attribution-based methods determine feature relevance by assigning a numeric value to each component of the input x , our approach is based in the intuition that the most relevant features are those that result in a change of prediction when set to the baseline value.
Definition 1.
Let N be a FCN with L layers and input dimension n , let t be a numeric threshold, let x ∈ R n be such that N ( x ) > t , and let x ′ ∈ R n be a baseline vector.
A feature removal set for
N ( x ) > t relative to x ′ is a subset S ⊆ { 1 , . . . , n } satisfying N ( x S | x ′ ) ≤ t .
The size | S | of S is the number of elements it contains.
Furthermore, we say S is minimal if no S ′
⊂ { 1 , . . . , n } such that | S ′
| < | S | is a feature removal set for N ( x ) > t relative to x ′ .
Note that this definition implies that a feature removal set S cannot be the empty set.
Although different baselines may lead to different feature removal sets for the same prediction, a natural baseline independent from the input can often be identified Sundararajan et al.
We next show that the decision version of our problem, even restricted to ReLU FCNs, is NPcomplete.
Thus, to identify a minimal feature removal set, one may need to consider exponentiallymany combinations of input features.
Theorem 1.
The following problem is NP-complete: given as input vectors x , x ′ ∈ R n , a ReLU FCN N of input dimension n , a numeric threshold t such that N ( x ) > t , and 1 ≤ k ≤ n , decide whether there exists a feature removal set for N ( x ) > t relative to x ′ of size at most k .
Proof.
We show NP-hardness using a reduction from SUBSET-SUM, which is the problem of checking, given as input integers a 1 , ..., a m and ξ , whether there exists S ⊆ { 1 , . . .
, m } such that (cid:80) i ∈
S a i = ξ .
In the context of this proof, let us denote with 0 m the m -dimensional column null vector and with 1 m the m -dimensional column vector with all components equal to 1 .
We map an instance a 1 , ..., a m , ξ of SUBSET-SUM to an instance of our problem by setting k = m , t = 0 , x = 1 m +1 , x ′ =
m +
and N the 2-layer ReLU network defined as given next.
In the first layer, W 1 is a (3 × m +1) matrix with values a 1 [-= 
-], . . .
, a m ,[-σ 1 i ( y ) is 
 if y i =-] 0 in the first row, − a 1 , [-and 
-]. . .
, − a m , 0 in the second row, and having 1 m +1 in the third row; furthermore, b 11 = − (cid:80) mi =1 a i − ξ , b 12 = [-
-]{+(cid:80) mi =
+}
a i + ξ and b 13 = − m .
In the second layer, W 2 = 1 3 , b 2 = 0 [-, and σ 3 ( z ) = 
-].
With input x = 1 m +1 , we have that h 11 = − ξ , h 12 = ξ , and h 13 = 1 which yields that h 2 ≥ 1 .
By construction, we thus have N ( x ) > 0 , so each valid input to SUBSET-SUM is mapped to a valid input of our problem.
Assume there is a feature removal set S for N ( x ) > 0 relative to x ′ of size at most k
= m .
By definition, N ( x S | x ′ ) ≤ 0 .
We claim that, by construction of N , S is a solution to the corresponding subset sum instance.
Indeed, we must have h 2 ≤ 0 by property of the ReLU and by construction, h 2 ≥
(as a sum of ReLU activations), thus we must have h 2 = 0 . This enforces that h 11 ≤ 0 and h 12 ≤ 0 by property of the ReLU (we also have h {+13 = 0 since S ̸ = ∅ ). Since h 11 = − h 12 , this yields h 
+}
= h 12 = 0 . By construction, with input x S | x ′ , we have h 11 = (cid:80) i ∈{ 1 ,...,m }− S a i − (cid:80) mi =1 a i − ξ = − ( ξ − (cid:80) i ∈ S a i ) .
It follows that S is a solution to (cid:80) i ∈
S a i = ξ .
For the converse, let S be a solution to SUBSET-SUM.
We claim that S is also a solution to our problem.
Indeed, S ̸ = ∅ , thus with input x S | x ′ , we have h 13 = 0 . Furthermore, (cid:80) i ∈ S a i = ξ thus h 1 1 = h 12 = 0 which yields h {+2 = 
+}
and therefore N ( x S | x ′ ) ≤ 0 .
Membership in NP follows since a set S ⊆ { 1 , . . . , m } of size at most k provides a certificate.
In particular, S is a feature removal set if N ( x S | x ′ ) ≤ t , which is verifiable in polynomial time.
The proof of the theorem shows that intractability depends on the dimension of the feature and baseline vectors, but not on their concrete values.
Neural networks, however, can have a large number of input features, which makes the computation of minimal feature removal sets problematic.
Thus, our focus in the remainder of this paper will be on identifying additional requirements on the trained neural model ensuring that minimal feature removal sets can be computed in polynomial time.
M ONOTONIC N EURAL N ETWORKS
A key observation in the proof of Theorem 1 is that the weights in the network have to take both positive and negative values to reduce to SUBSET-SUM.
This observation suggests the following definition of a monotonic FCN.
Definition 2.
A fully-connected neural network N = ⟨{ W ℓ } 1 ≤ ℓ ≤ L , { b ℓ } 1 ≤ ℓ ≤ L , { σ ℓ } 1 ≤ ℓ ≤ L } ⟩ is monotonic if, for each 1 ≤ ℓ ≤ L , weights W ℓ are non-negative and function σ ℓ is continuous everywhere, differentiable almost everywhere, and monotonic.
In the remainder of this section we present our main result, which establishes that minimal feature removal sets can be computed in polynomial time for monotonic FCNS by means of a greedy algorithm.
Thus, our intractability result in Theorem 1 may not apply to all FCNs used in practice.
5. P ROPERTIES OF M ONOTONIC N EURAL N ETWORKS
Consider a trained FCN N where the weights and activation functions satisfy the requirements in Definition 2.
The continuity and differentiability requirements ensure that the gradient of N can be computed for each input vector x .
In turn, as we show next, the monotonicity requirement ensures that each component of the gradient of N at x can be expressed as a sum where (1) the number of elements in the sum is fixed for N (i.e., it does not depend on x ) and it is the same for all components; and (2) each element of the sum consists of a product involving a value that depends on x but which is always non-negative , and two coefficients that do not depend on x .
This key property of the gradient in monotonic FCNs allows us to exploit the theoretical properties of the integrated gradients attribution method.
In particular, we can show that, by setting a component of the input vector x to the baseline, we are not altering the relative order of the integrated gradient attributions (computed as in equation 1) associated with the remaining components.
These properties, which are established by the following technical lemma, constitute the basis of our greedy algorithm for computing minimal feature removal sets.
Lemma 1.
For each monotonic FCN N with input dimension n , there exists a non-negative integer
M , positive coefficients { A m } 1 ≤ m ≤ M , real coefficients { B i } 1 ≤ i ≤ n , and functions { g m } 1 ≤ m ≤ M from R n to R ≥ 0 , such that the following identities are satisfied for each x , x ′ ∈ R n and each i, j ∈ { 1 , ..., n } :
[Equation]
and
[Equation]
where p ij ( τ ) = x { j }| x ′ + τ ( x { i }| x ′ − x { j }| x ′ ) .
Proof.
We show equation 2 by induction on the number L of layers in N .
If L = 1 , then N = ⟨ W , b, σ ⟩ with W an n -dimensional vector, b a scalar and σ : R (cid:55)→ R .
Using the chain rule, we have ( ∇N ) i ( x ) =
W i · ( Dσ )( W · x + b ) , where Dσ is the derivative of σ in Euler’s notation.
This expression is of the form equation 2 with M = 1 , A 1 = 1 ≥ 0 , B i
= W i if σ is monotonically increasing (dually, B i = − W i if σ is monotonically decreasing), and g 1 ( x ) =
( Dσ )( W · x + b ) if σ is monotonically increasing (dually, g 1 = − ( Dσ )( W · x + b ) if σ is monotonically decreasing).
In both cases, monotonicity of σ ensures that g 1 ( x ) ≥ 0 for any x .
For the inductive case, assume that equation 2 holds for every network with L − 1 layers.
The application of N = ⟨{ W ℓ } 1 ≤ ℓ ≤ L , { b ℓ } 1 ≤ ℓ ≤ L , { σ ℓ } 1 ≤ ℓ ≤ L } ⟩ with L layers to x is defined as σ L ( h L ( x )) .
Thus, we can apply the chain rule and the definition of h L to obtain the following identity:
[Equation]
Here, N j is specified by ⟨{ W ℓ } 1 ≤ ℓ ≤ L − 2 , W L − 1 j , { b ℓ } 1 ≤ ℓ ≤ L − 2 , b L − 1 j , { σ ℓ } 1 ≤ ℓ ≤ L − 1 } ⟩ , where
W L − 1 j and b L − 1 j are the j -th row of W L − 1 and the j -th element of b L − 1 , respectively.
We apply the inductive hypothesis to obtain the value of the gradient for each 1 ≤ j ≤ d L − 1 , which is given by ( ∇N j ) i ( x ) = B j i (cid:80) M {+j m j =
 A j m j g j m j ( x ) .+}
But now, we can replace the value Algorithm Input: monotonic FCN N , vectors x , x ′ ∈ R n and threshold t ∈ R such that N ( x ) > t . for 1 ≤ j [-m j =
-]≤ n do c
j ← N ( x { j }| x ′ ) end for
I ← list of indices obtained from sorting { c j } 1 ≤ j ≤ n in ascending order (ties broken arbitrarily). S ← ∅ for 1 ≤ j ≤ n do S ← S ∪ I [ j ] if N ( x S | x ′ ) ≤ t then return S end for of the gradients in the sum of equation 4 with these values and show the statement of the lemma by instantiating equation 2 with M = (cid:80)
d L − 1 j =1 M j , A m =
W
Lj A jm j , B i =
± B ji and g m ( x ) =
±
g jm j ( x ) · ( Dσ L )( h L ( x )) , where the sign in ± is determined by whether σ is monotonically increasing or decreasing.
Again, by inductive hypothesis, it follows that A m ≥ 0 and g m ( x ) ≥for each m and x .
We now show equation 3.
Let us consider the attribution for N defined in equation 1.
Assume i ̸
= j (otherwise the equation holds trivially).
By replacing the value of the gradient in equationwith equation 2, we obtain C N i ( x , x ′ ) =
B i ( x i − x ′ i ) {+(cid:80) Mm =
 A m+} (cid:82) 1 0 g m ( x ′ + τ ( x − x ′ ))
d τ.
Since integrated gradients satisfy the completeness and zero contribution axioms, we can compute the difference N ( x { i }| x ′ ) − N ( x { j }| x ′ ) as the sum of contributions C
N i ( x { i }| x ′ , x { j }| x ′ ) and
C N j ( x { i }| x ′ , x { j }| x ′ ) to obtain ( x ′ i − x i ) (cid:82) 1 0 ( ∇N ) i (cid:0) p ij
[Table]
[Table]
p
[Table]
τ ) (cid:1)
d τ.
equation 2 provides the values for the gradients ( ∇N ) i ( p ij ( τ )) and ( ∇N ) j ( p ij ( τ )) , which we can replace in the previous expression to finally derive equation 3.
5. A G REEDY A LGORITHM FOR C OMPUTING M INIMAL F EATURE R EMOVAL S ETS
In this section, we propose a greedy algorithm for computing a minimal feature removal set for a given prediction of a monotonic FCN.
Algorithm 1 proceeds as detailed next.
In each iteration of the first loop, the algorithm sets each individual input feature to the baseline value (while leaving the remaining components unchanged) and applies the input FCN to the resulting vector.
The values obtained by each of these applications of the FCN are then sorted in ascending order.
In the second loop, the algorithm successively assigns the components of x to the baseline in the order established in the previous step until the prediction no longer holds.
The algorithm then returns S consisting of all features that were set to the baseline.
Our algorithm is quadratic in the number of input features: both loops require linearly many applications of the FCN, and each application is feasible in linear time in the number of features Goodfellow et al.
The algorithm’s correctness relies on equation 3 in Lemma 1, which ensures that, when set to the baseline, each of the features selected by the algorithm in the second loop yields the largest change (amongst all other possible feature choices) in the evaluation of the network, thus bringing the network’s output as close as possible to the prediction threshold.
As a result, the output feature removal set S is guaranteed to contain a smallest number of features.
Theorem 2.
Algorithm 1 computes a minimal feature removal set for N ( x ) > t relative to x ′ .
Proof.
It suffices to show that, for each j ∈ { 1 , ..., n } , the choice of I [ j ] in the second loop yields the largest change in the evaluation of N .
That is, for each 1 ≤ j ≤ n and j ≤ k ≤ n we have
N ( x S | x ′ ) −N ( x ( S ∪ I
[ j ]) | x ′ ) ≥ N ( x S | x ′ ) −N ( x ( S ∪ I
[ k ]) | x ′ ) .
By construction of list I , the inequality
N ( x I [ j ] | x ′ ) − N ( x I [ k ] | x ′ ) ≤ 0 holds for each 1 ≤ j ≤ k ≤ n .
We apply equation 3 in Lemmatogether with the fact that A m ≥ 0 and g m ( x ) ≥ 0 for each m and x (and hence (cid:82) 1 {+
+}
g m ( p ij ( τ )) d τ ≥ 0 ) to obtain[-(cid:80) Mm =
 A m-] ( B I [ j ] ( x ′ I
[ j ] − x I
[ j ] ) − B I [ k ] ( x ′ I
[ k ] − x I [ k ] )) ≤ 0 .
Since { I [ j ] , I [ k ] } ⊆ { 1 , ..., n } − S , we have (cid:16) x S | x ′ (cid:17)
I [ j ] = x I [ j ] and (cid:16) x S | x ′ (cid:17)
I [ k ] = x I [ k ] .
By applying equation 3 and the previous inequality, we finally obtain N ( x ( S ∪ I [ j ]) | x ′ ) ≤ N ( x ( S ∪ I [ k ]) | x ′ ) .
Note that, although the correctness of our algorithm relies on the properties of integrated gradients, the algorithm itself only relies on the ability to apply the input network as a ‘black box’ and hence does not involve the computation of any attribution values.
G ENERALISATION TO C ONVOLUTIONAL AND G RAPH N EURAL N ETWORKS
In this section, we generalise our approach to more advanced neural architectures.
Our main result is that, under monotonicity assumptions, Algorithm 1 continues to work verbatim if the input network N is a standard Convolutional (or Graph) Neural Network.
In particular, we require that the weights are non-negative and that the non-linearities in N can be equivalently expressed as vector fields that are continuous, differentiable almost everywhere and with a Jacobian that is either at leasteverywhere or at most 0 everywhere; the latter is the natural generalisation of the monotonicity requirement to vector fields, and it is satisfied by the standard non-linearities included in practical neural architectures.
To establish this result, we start by introducing as a theoretical tool the notion of a generalised FCN , where activation functions are defined as vector fields (and hence are no longer restricted to scalar functions applied component-wise to vectors).
As discussed later on, this will give us the flexibility needed to mathematically capture the application of CNNs as well as certain variants of GNNs.
Definition 3.
A generalised FCN (GFCN) with L ≥ 1 layers and input dimension n ∈ N is a tuple
N = ⟨{ W ℓ } 1 ≤ ℓ ≤ L , { b ℓ } 1 ≤ ℓ ≤ L , { σ ℓ } 1 ≤ ℓ ≤ L } ⟩ .
For each ℓ ∈ { 1 , . . .
, L } , integers d ℓ ∈ N and
D ℓ ∈ N represent the width and the pre-activation width of layer ℓ , respectively; we require d L =and define d 0 = n ; matrix W ℓ ∈ R D ℓ × d ℓ − 1 is a real-valued weight matrix; vector b ℓ ∈ R D ℓ is the bias vector; and the activation function σ ℓ : R D ℓ (cid:55)→ R d ℓ is a polytime-computable vector field.
The application of a GFCN N to a vector x ∈ R n is defined as given in the preliminaries for FCNs.
The monotonicity requirement for FCNs can be seamlessly lifted to GFCNs, by establishing the natural requirements on the Jacobian of the each activation function in the network.
In particular, we require the values of the Jacobian to be either non-negative everywhere or at most 0 everywhere.
Definition 4.
A GFCN N = ⟨{ W ℓ } 1 ≤ ℓ ≤ L , { b ℓ } 1 ≤ ℓ ≤ L , { σ ℓ } 1 ≤ ℓ ≤ L } ⟩ is monotonic if for each layer 1 ≤ ℓ ≤ L , the weights W ℓ are non-negative and the Jacobian J ℓ of σ ℓ : R D ℓ (cid:55)→ R d ℓ satisfies one of the following properties:
• ( J ℓ ( y )) ij ≥ 0 for each 1 ≤ i ≤ d ℓ , each 1 ≤ j ≤ D ℓ , and each vector y ∈ R D ℓ , OR • ( J ℓ ( y )) ij ≤ 0 for each 1 ≤ i ≤ d ℓ , each 1 ≤ j ≤ D ℓ , and each vector y ∈ R D ℓ .
We next show that Algorithm 1 remains correct if the input network is a monotonic GCFN.
This is so because the generalised monotonicity requirement in Definition 4 ensures that equation 2 in
Lemma 1 remains true if N is a monotonic GFCN.
Theorem 3.
Algorithm 1 computes a minimal feature removal set for N ( x ) > t relative to x ′ when applied to a monotonic GFCN N , vectors x , x ′ ∈ R n and threshold t ∈ R such that N ( x ) > t .
Proof.
It suffices to show that the statement in Lemma 1 also holds if N is a GFCN and, in particular, that equation 2 remains true.
To this end, we set up a similar induction on the number L of layers in N as we did in the proof of Lemma 1.
If N has a single layer ( L = 1) , then N = ⟨ W , b, σ ⟩ , with W an ( D 1 × n ) -matrix, b an D 1 -dimensional vector and σ a scalar field σ : R D 1 (cid:55)→ R .
By the chain rule, each component of the gradient of N is given by ( ∇N ) i ( x ) =
(cid:80) D 1 j =1 W 1 i,j (cid:0) J 1 (cid:0) W 1 · x + b 1 (cid:1)(cid:1) 1 ,j , where J 1 is the Jacobian of σ 1 which, in this case, is a row vector.
This expression is of the form equation 2 with M = D 1 , A m = 1 , B i = ± W 1 i,j and g m ( x ) = ± (cid:0) J L (cid:0) σ L (cid:0) W 1 · x + b 1 (cid:1)(cid:1)(cid:1) 1 ,j where the sign ± is chosen depending on the type of monotonicity of the Jacobian (c.f. Definition 4).
Observe that g m ( x ) ≥ 0 by the monotonicity assumption from Definition 4.
For the inductive case, assume equation 2 holds for each GFCN with L − 1 layers.
The application of N with L layers to x is given by N ( x ) =
σ L (cid:0) h L ( x ) (cid:1) ; thus, by the chain rule, each component of the gradient ( ∇N ) i is now given as the sum (cid:80)
D L j =1 (cid:80) d L − 1 k =1 W Lk,j ( ∇N k ) i ( x ) (cid:0) J L (cid:0) h L ( x ) (cid:1)(cid:1) 1 ,j .
Here, N k
= { W ℓ } 1 ≤ ℓ ≤ L − 1 , { b ℓ } 1 ≤ ℓ ≤ L − 1 , { σ ℓ } 1 ≤ ℓ ≤ L − 2 } , σ L − 1 k ⟩ , where σ L − 1 k : R D L − 1 (cid:55)→ R is the k -th component of the vector field σ L − 1 : R D L − 1 (cid:55)→ R d L − 1 .
Following the proof of Lemma 1, the application of the inductive hypothesis to GFCN N k yields (cid:0) ∇N k (cid:1) i ( x )
= B ki (cid:80) M k m k {+=
+}
A km k g km k ( x ) .
As a result, ( ∇N ) i can be put into form equation 2 by taking M = (cid:80)
D L j =1 (cid:80) d L − 1 k =1 M k , A m =
A km k W Lk ≥ 0 , B i = ± B ki , and g m ( x ) = ±
g km k ( x ) · (cid:0)
J L (cid:0) h L ( x ) (cid:1)(cid:1) 1 ,j .
The latter is again non-negative by assumption on the Jacobian.
We finally argue that GFCNs are powerful enough to capture CNNs with non-negative weights (and certain variants of monotonic GNNs with non-negative weights) in a way that ensures correctness of Algorithm 1 when applied directly to such neural architectures, provided that their non-linearities satisfy our monotonicity requirements.
Consider a CNN C and a fixed dimension of input images.
We argue that there is a GFCN N simulating the application of C to any image with the given dimensions.
Thus, correctness of Algorithm 1 applied to C is implied by the correctness of the algorithm applied to N stated in Theorem 3.
To see this, first note that the application of convolutional filters within a layer can be simulated by matrix multiplication Goodfellow et al.
In turn, pooling operations can be seen as nonlinearities returning a vector of smaller dimension, hence the definition of σ ℓ as a vector field.
Batch normalisation in inference mode can be seen as a linear transformation.
Finally, pooling operations composed with standard non-linear activation functions can be simulated by first applying pooling, then multiplying by the identity matrix, and finally applying the standard activation function.
Similarly, GFCNs can also capture basic graph neural networks for which the feature update function is defined as
[Equation]
(cid:17) , where h ℓu denotes the vector of features of node u at layer ℓ , and N ( u ) denotes the neighbours of node u in the input graph.
We argue that, given such GNN G and a fixed topology of input graphs (i.e., a fixed set of nodes, edges, and fixed dimensions of the feature vector associated to each node), there exists a GFCN N simulating the application of G to any graph with the given topology.
To see this, note that one can flatten the input so that each component of the input space corresponds to a feature of a particular node.
The aggregation over neighbourhoods can be captured by matrix multiplication where the matrix has zeros where the corresponding nodes are not in the same neighbourhood, and the matrix has the corresponding weight values where the nodes are in the same neighbourhood.
The non-linear activation function is then applied component-wise to each node feature, so it can be reproduced by applying the activation function component-wise to each component of the transformed input.
This correspondence also generalises to Graph Convolutional Networks (GCNs) Kipf & Welling (2017).
We note, however, that GFCNs cannot adequately capture arbitrary message-passing GNNs since there exists aggregatation functions which cannot be simulated by matrix multiplication followed by the application of a non-linear map.
We can conclude that our greedy algorithm can be directly applied to CNNs and GNNs with nonnegative weights used in practice by using the networks as a ‘black box’ (and hence without the need of explicitly constructing the corresponding GFCN).
In particular, the non-linearities used in practice (max-pooling, average-pooling, ReLu, sigmoid, tanh) satisfy our requirements.
C ONCLUSION AND F UTURE W ORK
We have proposed to study a combinatorial problem, the minimal feature removal problem , inspired by ideas from attribution-based and perturbation-based explanation methods.
Our intractability result constitutes a theoretical limitation of what can be achieved by neural network explanations. Our minimal feature removal sets can be efficiently computed for certain neural architectures, and could be used to probe the reliability of the predictions on certain tasks.
Furthermore, minimal feature removal set sizes can be used as a measure of robustness of predictions.
In line with existing work Alvarez Melis & Jaakkola (2018); Erion et al.
; Ismail et al.
(2021), our approach could also be incorporated in the training process to improve network’s robustness by generating additional training examples.
Scalability remains a potential issue in our approach since since our greedy algorithm is quadratic in the number of features; we expect, however, that parallelising predictions can help in that regard.
We hope that our work can motivate further theoretical analysis of neural network explainability
Sanity