A L EARNING T HEORETIC P ERSPECTIVE ON L OCAL E XPLAINABILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we explore connections between interpretable machine learning and learning theory through the lens of local approximation explanations.
First, we tackle the traditional problem of performance generalization and bound the testtime accuracy of a model using a notion of how locally explainable it is.
Second, we explore the novel problem of explanation generalization which is an important concern for a growing class of ﬁnite sample-based local approximation explanations.
Finally, we validate our theoretical results empirically and show that they reﬂect what can be seen in practice.
I NTRODUCTION
There has been a growing interest in interpretable machine learning, which seeks to help people understand their models.
While interpretable machine learning encompasses a wide range of problems, it is a fairly uncontroversial hypothesis that there exists a trade-off between a model’s complexity and general notions of interpretability .
This hypothesis suggests a seemingly natural connection to the ﬁeld of learning theory, which has thoroughly explored relationships between a function class’s complexity and generalization .
However, formal connections between interpretability and learning theory remain relatively unstudied.
Though there are several notions of conveying interpretability, one common and ﬂexible approach is to use local approximations.
Formally, local approximation explanations (which we will refer to as “local explanations”) provide insight into a model’s behavior as follows: for any black-box model f 2 F and input x , the explanation system produces a simple function, g x ( x 0 ) 2 G local , which approximates f in a speciﬁed neighborhood, x⇠ N x .
Crucially, the freedom to specify both G local and N x grants local explanations great versatility.
In this paper, we provide two connections between learning theory and how well f can be approximated locally (i.e. the ﬁdelity of local explanations).
Our ﬁrst result studies the standard problem of performance generalization by relating test-time performance to a notion of local interpretability.
As it turns out, our focus on local explanations leads us to unique tools and insights from a learning theory point of view.
Our second result identiﬁes and addresses an unstudied – yet important – question regarding explanation generalization .
This question pertains to a growing class of explanation systems, such as MAPLE (Plumb et al., 2018) and RL-LIM (Yoon et al., 2019), which we call ﬁnite sample-based local explanations 1 .
These methods learn their local approximations using a common ﬁnite sample drawn from D (in contrast to local approximation methods such as LIME (Ribeiro et al., 2016)) and, as a result, run the risk of overﬁtting to this ﬁnite sample.
In light of this, we answer the following question: for these explanation-learning systems, how well do they generalize to data not seen during training?
We address these questions with two bounds, which we outline now.
Regarding performance generalization, we derive our ﬁrst main result, Theorem 1, which bounds the expected test mean squared error (MSE) of any f in terms of its MSE over the m samples in the training set, S = { ( x i , y i ) } mi =1 :
[Equation]
Regarding explanation generalization for ﬁnite sample-based explanation-learning systems, we apply a similar proof technique to obtain Theorem 2, which bounds the quality of the system’s explanations on unseen data in terms of their quality on the data on which the system was trained:
[Equation]
⇤
[Equation]
+
[Equation]
O ⇣
[Equation]
R
[Equation]
( G local ) ⌘ | {z }
[Table]
Before summarizing our contributions, we discuss the key new terms and their relationship.
• Interpretability terms: The terms involving MNF correspond to Mirrored Neighborhood Fidelity, a metric we use to measure local explanation quality.
As we discuss in Section 3, this is a reasonable modiﬁcation of the commonly used Neighborhood Fidelity ( NF ) metric (Ribeiro et al., 2016; Plumb et al., 2018).
Intuitively, we generally expect MNF to be larger when the neighborhood sizes are larger since the g x 0 are required to extrapolate farther.
• Complexity term: This term measures the complexity of the local explanation system g in terms of (a) the complexity of the local explanation class G local and (b) ⇢ S , a quantity that we deﬁne and refer to as neighborhood disjointedness factor .
As we discuss in Section 4, ⇢ S is a value in [1 , p m ] (where m = | S | ) that is proportional to the level of disjointedness of the neighborhoods for points in the sample S .
Intuitively, we expect ⇢ S to be larger when the neighborhoods sizes are smaller since smaller neighborhoods will overlap less.
Notably, both our bounds capture the following key trade-off: as neighborhood widths increase,
MNF increases but ⇢ S decreases.
As such, our bounds are non-trivial only if N x can be chosen such that MNF remains small but ⇢ S grows slower than ˜ O
( p m ) (since ˆ R S G local decays as ˜ O
1 / p m
We summarize our main contributions as follows:
(1) We make a novel connection between performance generalization and local explainability, arriving at Theorem 1.
Given the relationship between MNF and ⇢ S , this bound roughly captures that an easier-to-interpret f enjoys better generalization guarantees, a potentially valuable result when reasoning about F is difﬁcult (e.g. for neural networks).
Further, our proof technique may be of independent theoretical interest as it provides a new way to bound the Rademacher complexity of a randomized function (see Section 4).
(2) We motivate and explore an important generalization question about expected explanation quality.
Speciﬁcally, we arrive at Theorem 2, a bound for test MNF in terms of training MNF .
This bound suggests that practitioners can better guarantee good local explanation quality (measured by MNF ) using methods which encourage the neighborhood widths to be wider (see Section 5).
(3) We verify empirically on UCI Regression datasets that our results non-trivially reﬂect the two types of generalization in practice.
First, we demonstrate that ⇢ can indeed exhibit slower than ˜ O
( p m ) growth without signiﬁcantly increasing the MNF terms.
Also, for Theorem 2, we show that the generalization gap indeed improves with larger neighborhoods (see Section 6).
(4) Primarily to aid in our theoretical results, we propose MNF as a novel yet reasonable measure of local explainability.
Additionally, we argue that this metric presents a promising avenue for future study, as it may naturally complement NF and offer a unique advantage when evaluating local explanations on “realistic” on-distribution data (see Section 3).
R ELATED W ORK
Interpretability meets learning theory: Semenova et al.
(2019) study the performance generalization of models learned from complex classes when they can be globally well-approximated by simpler (e.g. interpretable) classes.
In such cases, their theory argues that if the complex class has many models that perform about as optimally on training data, generalization from the complex class can be more closely bounded in terms of the simpler class’s complexity.
In our corresponding results, we similarly aim to avoid involving the larger class’s complexity.
However, we directly study generalization via a function’s local explainability, rather than instantiate abstract ”complex” and “simple” classes for global approximations.
The two are fundamentally different technical problems; standard learning theory results cannot be directly applied as they are for single-function global approximations.
Statistical localized regression: (Fan, 1993; Fan & Gijbels, 1996) are canonical results which bound the squared error of a nonparametric function deﬁned using locally ﬁt models.
These local models are both simple (e.g. linear) and similarly trained by weighting real examples with a kernel (i.e. neighborhood).
However, in these works, each local model is only used to make a prediction at its source point and the theory requires shrinking the kernel width towards 0 as the sample size grows.
We instead ﬁt local models as explanations for a trained model (i.e. which is considered the “true regression function”) and more importantly, care about the performance of each local model over whole (non-zero) neighborhoods.
Unlike localized regression, this allows us to use uniform convergence to bound test error with empirical and generalization terms.
While the previous results do not have empirical terms, the learning rates are exponential in the number of samples.
Learning Theory: One line of related work also studies how to explain generalization of overparameterized classes.
As standard uniform convergence on these classes often leads to vacuous bounds, a general approach that has followed from (Nagarajan & Kolter, 2019; Zhang et al., 2017;
Neyshabur et al., 2014) has been to study implications of different biases placed on the learned models.
We study what would happen if an overparameterized model had an unexplored type of bias, one that is inspired by local explainability.
Additionally, our work’s technical approach also parallels another line of existing results which likewise try to apply uniform convergence on a separate surrogate class.
This includes PAC-Bayesian bounds, a large family of techniques that come from looking at a stochastic version of in parameter space (McAllester, 1998; 2003; Langford &
Caruana, 2002; Langford & Shawe-Taylor, 2003).
In a different vein, some results in deep learning look at compressed/sparsiﬁed/explicitly regularized surrogates of neural networks (Arora et al., 2018; Dziugaite & Roy, 2017).
In our case, the surrogate class is a collection of local explanations.
M IRRORED N EIGHBORHOOD F IDELITY
In order to connect local explanations to generalization, recall that we study a measure of local interpretability which we call “mirrored neighborhood ﬁdelity” ( MNF ).
As we explain below, this quantity comes from a slight modiﬁcation to an existing measure of interpretability, namely, that of neighborhood ﬁdelity ( NF ).
To deﬁne our terms, we use the following notations.
Let X be an input space and let D be a distribution over X ⇥ Y where Y ✓ R .
Let F be a class of functions f : X !
Y .
For our theoretical results, we speciﬁcally assume that
Y is bounded as Y =
[  B, B ] for some B > 0 (though this does not matter until following sections).
In order to provide local explanations, we need to ﬁx a local neighborhood around each x 2 X .
To this end, for any x , let N x correspond to some distribution denoting a local neighborhood at x e.g., typically this is chosen to be a distribution centered at x .
For any distribution N , we use p N ( x ) to denote its density at x .
Now, let G be a class of explainers g : X ⇥ X !
Y such that for each x 2 X , the local explanation g ( x, · ) :
X !
Y belongs to a class of (simple) functions (e.g. linear), G local .
In short, we denote g ( x, · ) as g x ( · ) and we’ll use g ( x, · ) to locally approximate f in the neighborhood deﬁned by N x .
The accuracy of this local approximation is usually quantiﬁed by a term called “neighbhorhood ﬁdelity” which is deﬁned as follows (Ribeiro et al., 2016; 2018; Plumb et al., 2018; 2020)
[Equation]
To verbally interpet this, let us call x as the “source” point which gives rise to a local explanation
g x ( · ) and x 0 the “target” point that we try to ﬁt using g .
To compute NF ( f, g ) , we need to do the following: for each source point x , we ﬁrst compute the average error in the ﬁt of g x ( · ) over target points x 0 in the local neighborhood of the source point x (i.e., N x ); then, we globally average this error across draws of the source point x ⇠ D .
Now, to deﬁne MNF , we take the same expression as NF but swap x and x 0 within the innermost expectation (without modifying the expectations).
In other words, we now sample a target point x from D , and sample a source point x 0 from a distribution over points near x .
Since this distribution is over source points rather than target points, just for the sake of distinguishing, we’ll call this a mirrored neighborhood distribution and denote it as N mir x .
Formally we deﬁne this measure of local interpretability below, following which we explain how to understand it:
Deﬁnition 3.1.
(Mirrored Neighborhood Fidelity)
We deﬁne MNF : F ⇥ G !
R as
[Equation]
and with an abuse of notation, we let MNF ( f, g, x ) :=
E x 0 ⇠ N mir x
[Equation]
Understanding MNF .
It is helpful to parse the expression for MNF in two different ways.
First, we can think of it as measuring the error in approximating every target point x 2 X through a randomized locally-approximating function g x 0 ( · ) where x 0 is randomly drawn from the local neighborhood
N mir x .
A second way to parse this is in a manner similar to how we parsed NF .
To do this, ﬁrst we note that the expectations in MNF can be swapped around and rewritten equivalently as follows:
[Equation]
where D † and N † x 0 are suitably deﬁned distributions (deﬁned in Appendix A) that can be thought of as modiﬁed counterparts of D and N mir x 0 respectively.
With this rewritten expression, one can read
MNF like NF : for each source point (here that is x 0 ), we compute the average error in the ﬁt of the corresponding local function ( g x 0 ( · ) ) over target points ( x ) in the local neighborhood of the source point ( N † x 0 ); this error is then globally averaged over different values of the source point ( x 0 ⇠ D † ).
While both NF and MNF are closely related measures of local interpretability for f , studying MNF allows us to make connections between local interpretability and different notions of generalization (Sections 4 and 5).
At a high-level, our results don’t apply to NF because in NF , the overall distribution of the points that are “ﬁt” to calculate NF (i.e., the target points) is not the same as D (instead being D convolved with N x ).
However, we want this target distribution to be the same as D (which is the case in MNF ) for us to be able to neatly bound the test error term via the local interpretability term.
Otherwise, we would have to end up introducing many cumbersome terms.
Furthermore, MNF may also be of interest to the interpretability community, as it potentially offers a unique advantage over NF when the intended usage of local explanations is centered around understanding how the model works on the speciﬁc learning task it was trained on.
Speciﬁcally, we present as an exploratory argument that selecting the target point distribution to be D rather than D perturbed by N x (as for NF ) may better emphasize the ability for explanations to accurately convey how well g will predict at realistic points.
This is relevant for ML (and deep learning particularly) because (a) high-dimensional datasets often exhibit signiﬁcant feature dependencies and adherence to lower dimensional manifolds; (b) f can often be highly unpredictable and unstable when extrapolating beyond the training data.
Thus, when one measures NF with standard neighborhood choices that ignore feature dependencies (i.e. most commonly N x = N ( x,  I ) ), the resulting target distribution may concentrate signiﬁcantly on regions that are non-relevant to the actual task at hand.
As can be shown in a toy example, this can lead to overemphasis on ﬁtting noisy off-manifold behavior, deteriorating the ﬁt of explanations relative to task-relevant input regions (we defer a more detailed presentation of this point, as well as other trade-offs between NF and MNF to Appendix A).
G ENERALIZATION OF M ODEL P ERFORMANCE VIA MNF
The generalization error of the function f is typically bounded by some notion of the representational capacity/complexity of F .
While standard results bound complexity in terms of parameter count, there is theoretical value in deriving bounds involving other novel terms.
By doing so, we can understand how regularizing for those terms can affect the representation capacity, and in turn, the generalization error of f .
Especially when f ’s complexity may be intractable to bound on its own, introducing these terms provides a potentially useful new way to understand f ’s generalization.
Here speciﬁcally, we are interested in establishing a general connection between the representation complexity and the local explainability of any f .
This naturally requires coming up with a notion that appropriately quantiﬁes the complexity of G , which we discuss in the ﬁrst part of this section.
As we shall see, G ’s complexity can be expressed in terms of G local , which is generally less complex and more amenable to standard analysis than F in practical settings where interpretability is desired.
In the second part, we then relate this quantity to the generalization of f to derive our ﬁrst main result.
Key technical challenge: bounding the complexity of G .
The overall idea behind how one could tie the notions of generalization and local interpretability is fairly intuitive.
For example, consider a simpliﬁed setting where we approximate f by dividing X into K disjoint pieces/neighborhoods, and then approximating each neighborhood via a simple (say, linear) model.
Then, one could bound the generalization error of f as the sum of two quantities: ﬁrst, the error in approximating f via the piecewise linear model, and second, a term involving the complexity of the piecewise linear model.
It is straightforward to show that this complexity grows polynomially with the piece-count,
K , and also the complexity of the simple local approximator (see Appendix C.0.1).
Similarly, one could hope to bound the generalization error of f in terms of MNF ( f, g ) and the complexity of G .
However, the key challenge here is that the class G is a much more complex class than the above class of piecewise linear models.
For example, a straightforward piece-count-based complexity bound would be inﬁnitely large since there are effectively inﬁnitely many unique pieces in g .
Our core technical contribution here is to bound the (Rademacher) complexity of G in this more complex local-interpretability setting.
At a high level, the resulting bound (which will be stated shortly) grows with “the level of overlap” between the neighborhoods { N mir x | x 2 X} , quantiﬁed as:
Deﬁnition 4.1.
Given a dataset S 2 ( X ⇥ Y ) m , we deﬁne the disjointedness factor ⇢ S as
[Equation]
Understanding the disjointedness factor.
⇢ S can be interpreted as bounding the “effective number” of pieces induced by the set of neighborhood distributions { N mir x | x 2 X} .
This turns out to be a quantity that lies in [1 , p m ] (shown formally in Appendix Fact B.1).
To intuit about this quantity, it is helpful to consider its behavior in extreme scenarios.
First, consider the case where N mir x is the same distribution (say N ) regardless of x i.e., neighborhoods are completely overlapping.
Then, ⇢ S =
R x 0 2 X ( p N ( x 0 )) dx 0 = 1 .
In the other extreme, consider if neighborhoods centered on the training data are all disjoint with supports
X 1 , . . .
, X | S | .
Here the integral splits into m summands as: ⇢ S =
[Equation]
( x
[Equation]
dx 0 = p m .
Thus, intuitively ⇢ S grows from 1 to p m as the level of overlap between the neighborhoods N mir x 1 , . . .
, N mir x | S | reduces.
For intuition at non-extreme values, we show in Appendix B.2 that in a simple setting, ⇢ =
p m 1  k (where 0  k  1 ) if every neighborhood is just large enough to encompass a 1 / m 1  k fraction of mass of the distribution D .
Rademacher complexity of G .
We now use ⇢ S to bound the Rademacher complexity of G .
Recall that the Rademacher complexity of a function class H consisting of h : X ! R is deﬁned as [Equation] ⇤ , where the  i ’s are drawn i.i.d.
from a uniform distribution over {  1 , 1 } .
Intuitively this captures the complexity of H by measuring how well it can ﬁt random labels.
Standard results allow us to use this to bound the generalization error for functions in H .
Now, in order to deﬁne the Rademacher complexity of G (which consists of a different kind functions whose domain is X ⇥ X ), it is useful to think of g as a randomized function .
Speciﬁcally, at any target point x , the output of g is a random variable g x 0 ( x ) where the randomness comes from x 0 ⇠ N mir x .
Then, in Lemma 4.1, we take this randomization into account to deﬁne and bound the complexity of G (which we use prove our main results).
To keep our statement general, we consider a generic loss function L : R ⇥ R !
R (e.g., the squared error loss is L ( y, y 0 ) =
( y  y 0 ) 2 ).
Whenever L satisﬁes a standard Lipschitz assumption, we can bound the complexity of G composed with the loss function L , in terms of ⇢ S , the complexity of G local and the Lipschitzness of L :
Lemma 4.1.
(see Appendix Lemma D.1 for full, precise statement) Let L ( · , y 0 ) be a c -Lipschitz function w.r.t.
y
in that for all y 1 , y 2
[  B, B ] , | L ( y 1 , y 0 )
L ( y 2 , y 0 ) |  c | y 1  y 2 | .
Then, the empirical Rademacher complexity of
G under the loss function L is deﬁned and bounded as:
[Equation]
where ~ is uniformly distributed over {  1 , 1 } m .
We note that the proof technique employed here may be of independent theoretical interest as it provides a novel way to bound the complexity of a randomized function.
Although techniques like
PAC-Bayes provide ways to do this, they do not apply here since the stochasticity in the function is of a different form.
Main result.
With the above key lemma in hand, we are now ready to prove our main result, which bounds the generalization error of f in terms of the complexity of G , thereby establishing a connection between model generalization and local interpretability.
Theorem 1.
(see Appendix Theorem 3 for full, precise statement)
With probability over 1   over the draws of S = { ( x 1 , y 1 ) , . . .
, ( x m , y m ) } ⇠ D m , for all f 2 F and for all g 2 G , we have (ignoring ln 1 /  factors):
[Equation]
and ˆ R S ( G local ) is the empirical Rademacher complexity of G local deﬁned as ˆ R S ( G local ) :
=
E
[Equation]
[Equation]
This result decomposes the test error of f into four quantities.
The ﬁrst quantity corresponds to the training error of f on the training set S .
The second and the third correspond to the mirrored neighborhood ﬁdelity of f with respect to g (computed on test and training data respectively).
The fourth and ﬁnal quantity corresponds to a term that bounds the complexity of G in terms of the “disjointedness factor” and the complexity of the simpler function class G local .
Takeaway.
A key aspect of this bound is the trade-off that it captures with varying neighborhood widths.
Consider shrinking the neighborhood widths to smaller and smaller values, in turn creating less and less overlap between the neighborhoods of the training data.
Then, on the one hand, we’d observe that the complexity term (the fourth term on the R.H.S) increases.
Speciﬁcally, since ˆ
R S ( G local ) typically scales as O (1 / p m ) , as we go from the one extreme of full overlap to the other extreme of complete disjointedness, the complexity term would increase from O (1 / p m ) to O (1) (eventually rendering the bound trivial).
On the other hand, as the widths decrease, the ﬁdelity terms (the second and the third term) would likely decrease – this is because the simple functions in G local would ﬁnd it easier and easier to approximate the shrinking neighborhoods.
This tradeoff is intuitive.
A function f that is hardly amenable to being ﬁt by local explanations would require extremely tiny neighborhoods for G local to locally approximate it (i.e. make the MNF terms small).
For example, in an extreme case, when the neighborhoods N mir x are set be point masses at x , it is trivially easy to ﬁnd g x ( · ) 2 G local with no approximation error.
Thus, the complexity term would be too large in this case, implying that a hard-to-interpret f results in bad generalization.
On the other hand, when f is easy to interpret, then we’d expect it to be well-approximated by G local even with wider neighborhoods.
This allows one to afford smaller values for both the complexity and MNF terms.
In other words, an easy-to-interpret f enjoys better generalization guarantees.
Caveats.
Our bound has two limitations worth noting.
First, for high-dimensional datasets (like image datasets), practical choices of N x can lead to almost no overlap between neighborhoods, thus rendering the bound trivial in practice.
This potentially poor dimension-dependence is a caveat similarly shared by bounds for non-parametric local regression, whereby increasing d results in an exponential increase in the required sample size (Fan, 1993; Fan & Gijbels, 1996).
Nevertheless, for low-dimensional datasets, we show in the experiments that for practical choices of the neighborhood distributions, there is sufﬁcient neighborhood overlap to achieve values of ⇢ S that are o ( p m ) .
A second caveat is that the second quantity, MNF ( f, g ) , requires unlabeled test data to be computed, which may be limiting if one is interested in numerically computing this bound in practice.
It is however possible to get a bound without this dependence, although only on the test error of g rather than f (see Appendix Theorem 4).
Nevertheless, we believe that the above bound has theoretical value in how it establishes a connection between the interpretability of f and its generalization.
G ENERALIZATION OF L OCAL E XPLAINABILITY
We now turn our attention to a more subtle kind of generalization that is both unstudied yet important.
Typically, the way g x 0 is learned at any source point x 0 is by ﬁtting a ﬁnite set of points sampled near x 0 , with the hope that this ﬁt generalizes to unseen, neighboring target points.
Naturally, we would want to ask: how well do the explanations g x 0 themselves generalize in this sense?
This question is straightforward to answer in settings like LIME.
In particular, assume that we learn g x 0 by sampling a set S x 0 of nearby points from a Gaussian centered at x 0 , and that we care about the ﬁt of g x 0 generalizing to the same Gaussian.
Here, we can argue via a standard Rademacher complexity based bound that the error of g x 0 on the Gaussian is bounded by its training error on S x 0 and the complexity term R S x 0 ( G local ) . We can apply the same argument individually for the explanation in each x 0 , because we have a fresh dataset S x 0 for each x 0 .
If one has the resources to sample more points to create a larger S x 0 , these bounds will naturally also tighten.
However, consider ﬁnite sample-based local explanation settings like MAPLE (Plumb et al., 2018) and RL-LIM (Yoon et al., 2019) where the training procedure is vastly different from this: in these procedures, the goal is to learn local explanations g x 0 in a way that is sensitive to the local structure of the (unknown) underlying data distribution D .
So, instead of ﬁtting the g x 0 to samples drawn from an arbitrarily deﬁned Gaussian distribution, here one ﬁrst draws a ﬁnite sample S from the underlying distribution D (and then labels it using f ).
Then, across all x 0 2 X , one reuses a reweighted version of the same dataset S (typically, points x in S that are near x 0 are weighted more) and then learns a g x 0 that ﬁts this reweighted dataset.
Contrast this with the former setting, where for each x 0 , one has access to a fresh dataset (namely, S x 0 ) to learn g x 0 .
This distinction then makes it interesting to wonder when the reuse of a common dataset S could cause the explanations to generalize poorly.
Motivated by this question, we present Theorem 2.
By using Lemma 4.1, we provide a bound on the “test MNF” (which corresponds to the ﬁt of g x 0 on the unseen data averaged across D ) in terms of the “train MNF” (which corresponds to the ﬁt of g x 0 on S , averaged across x 0 ) and the complexity term from Lemma 4.1.
We must however caution the reader that this theorem does not answer the exact question posed in the above paragraph; it only addresses it indirectly as we discuss shortly.
Theorem 2.
(see Appendix Theorem 2-full for full, precise statement)
For a ﬁxed function f , with high probability 1   over the draws of S ⇠ D m , for all g 2 G , we have (ignoring ln 1 /  factors):
[Equation]
Understanding the overall bound.
We ﬁrst elaborate on how this bound provides an (indirect) answer to our question about how well explanations generalize.
Consider a procedure like MAPLE that learns g using the ﬁnite dataset S .
For each x 0 2 X , we would expect this procedure to have learned a g x 0 that ﬁts well on at least those target points x in S that are near x 0 .
In doing so, it’s reasonable to expect the training procedure to have implicitly controlled the “train MNF ” term The reasoning for this is that the train MNF computes the error in the ﬁt of g x 0 on S for different values of x 0 , and sums these up in a way that errors corresponding to nearby values of ( x, x 0 ) are weighted more (where the weight is given by p N mir x ( x 0 ) ).
Now, our bound suggests that when this train MNF is minimized, this carries over to test MNF too (provided the complexity term is not large).
That is, we can say that the ﬁt of g x 0 generalizes well to unseen, nearby target points x that lie outside of S .
The indirectness of our result.
Existing ﬁnite sample-based explainers do not explicitly minimize the train MNF term (e.g., MAPLE minimizes an error based upon NF ).
However, as argued above, they have implicit control over train MNF .
Hence, our bound essentially treats MNF as a surrogate for reasoning about the generalization of the explanations learned by an arbitrary procedure.
As such, our bound does not comment on how well the exact kind of ﬁdelity metric used during training generalizes to test data.
Nevertheless, we hope that this result offers a concrete ﬁrst step towards quantifying the generalization of explanations.
Furthermore, we also note that one could also imagine a novel explanation-learning procedure that does explicitly minimize the train MNF term to learn g ; in such a case our bound would provide a direct answer to how well its explanations generalize.
Indeed, we derive such a theoretically-principled algorithm in Appendix A.
Takeaway.
While the above bound captures a similar trade-off with neighborhood width as the
Theorem 1, it is worth pausing to appreciate the distinct manner in which this tradeoff arises here.
In particular, when the width is too small, we know that the complexity term approaches O ( p m ) and generalization is poor.
Intuitively, this is because in this case, the procedure for learning g x 0 would have been trained to ﬁt very few datapoints from S that would have fallen in the small neigbhorhood of x 0 .
On the other hand, when the neighborhoods are large, this issue would not persist which is captured by the fact that ⇢ S approaches O (1) .
However, with large neighborhoods, it may also be hard to ﬁnd functions in G local that ﬁt so many points in S .
Overall, one practical takeaway from this bound is that it is important to not excessively shrink the neighborhood widths if one wants explanations that generalize well for predicting how f behaves at unseen points (see Section 6).
Caveats.
We remark that this particular bound applies only when the dataset S is used to learn only g i.e., f and the neighborhoods must be learned beforehand with separate data 2 .
This sort of a framework is typical when deriving theoretical results for models like random forests, where it greatly aids analysis to assume that the trees’ splits and their decisions are learned from independent datasets (i.e. two halves of an original dataset) (Arlot & Genuer, 2014).
Now, if one is interested in a bound where S is also used to simultaneously learn f , the only change to the bound is an added factor corresponding to the complexity of F .
Another caveat is that our bound only tells us how well the explanations g x 0 generalize on average over different values of x 0 .
This does not tell us anything about the quality of the generalization of g x 0 for an arbitrary value of x 0 .
That being said, just as average accuracy remains a central metric for performance (despite ignoring discrepancies across inputs), average MNF can still be a useful quantity for evaluating an explainer’s overall performance.
E MPIRICAL R ESULTS
We present two sets of empirical results to illustrate the the usefulness of our bounds.
First, we demonstrate that ⇢ S grows much smaller than O ( p m ) which, as stated before, establishes that our bounds yield meaningful convergence rates.
Second, we show that Theorem 2 accurately reﬂects the relationship between explanation generalization (Theorem 2) and the width of N mir x used to both generate and evaluate explanations.
Setup.
For both experiments, we use several regression datasets from the UCI collection (Dua &
Graff, 2017) and standardize each feature to have mean 0 and variance 1.
We train neural networks as our “black-box” models with the same setup as in (Plumb et al., 2020), using both their nonregularized and ExpO training procedures.
The latter explicitly regularizes for NF during training, which we ﬁnd also decreases MNF on all datasets.
For generating explanations, we deﬁne G local to be linear models and optimize each g x using the empirical MNF minimizer (see Appendix A).
Finally, we approximate ⇢ S using a provably accurate sampling-based approach (see Appendix E).
Growth-rate of ⇢ S .
In Figure 1 (top), we track the sample dependence of ⇢ S for various neighborhoods of width  (setting N mir x = N ( x,  I ) ).
We speciﬁcally approximate the growth rate as polynomial, estimating the exponent by taking the overall slope of a log-log plot of ⇢ S over m .
To cover a natural range for each dataset,  is varied to be between the smallest and half the largest inter-example l 2 distances.
In these plots, while small  result in a large exponent for ⇢ S and large  cause g to intuitively saturate towards a global linear model, we observe that there do exist values of  , where both these terms are in control i.e., we observe that we can achieve a growth rate of approximately O ( m 0 . 2 ) without causing g to saturate and MNF metrics to rise sharply.
Generalization and neighborhood size.
As per the setting of Theorem 2, we generate all explanations using data not used to learn the black-box model.
Speciﬁcally, we split the original test data into two halves, using only the ﬁrst half for explanation training and the second for explanation testing .
We plot MNF as measured over these two subsets of examples in Figure 1 (bottom).
From the results, it is evident that a generalization gap between train and test MNF exists.
Further, recall that Theorem 2 predicts that this gap decreases as wider neighborhoods are used, a phenomena reﬂected in most of these plots.
As a result, while training MNF monotonically increases with larger neighborhoods, test MNF always decreases at certain ranges of  .
C ONCLUSION AND F UTURE W ORK
In this work, we have studied two novel connections between learning theory and local explanations.
We believe these results may be of use in guiding the following directions of future work: (1) developing new local explanation algorithms inspired by our theory and the metric of MNF ; (2) resolving caveats or otherwise strengthening the theory presented in this paper; and (3) exploring applications of our techniques beyond interpretability, such as the general problem of deep learning generalization or others that require reasoning about the complexity of randomized functions