On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Interpretable and explainable machine learning has seen a recent surge of interest.
We focus on safety as a key motivation behind the surge and make the relationshipbetween safety and interpretability more quantitative.
Toward assessing safety,we introduce the concept of maximum deviation via an optimization problemto ﬁnd the largest deviation of a supervised learning model from a referencemodel regarded as safe.
We then show how interpretability facilitates this safetyassessment.
For models including decision trees, generalized linear and additivemodels, the maximum deviation can be computed exactly and efﬁciently.
Fortree ensembles, which are not regarded as interpretable, discrete optimizationtechniques can still provide informative bounds.
For a broader class of piecewise
Lipschitz functions, we leverage the multi-armed bandit literature to show thatinterpretability produces tighter (regret) bounds on the maximum deviation.
Wepresent case studies, including one on mortgage approval, to illustrate our methodsand the insights about models that may be obtained from deviation maximization.
Introduction
Interpretable and explainable machine learning (ML) has seen a recent surge of interest because it isviewed as a key pillar in making models trustworthy, with implications on fairness, reliability, andsafety [1].
In this paper, we focus on safety as a key reason behind the demand for explainability.
The motivation of safety has been discussed at a qualitative level by several authors [2–5] and isperhaps clearest in the dichotomy between directly interpretable models vs. post hoc explanationsof black-box models.
The former have been called “inherently safe” [3] and promoted as the onlyalternative in high-risk applications [5].
The crux of this argument is that post hoc explanations leavea gap between the explanation and the model producing predictions.
Thus, unusual data points mayappear to be harmless based on the explanation, but truly cause havoc.
These works however havenot quantitatively addressed the questions: 1) What does safety mean for such models, and 2) howexactly does interpretability aid safety?
Towards answering the ﬁrst question, we make a conceptual contribution in the form of an optimizationproblem, intended as a tool for assessing the safety of supervised learning (i.e. predictive) models.
Viewing these models as functions mapping an input space to an output space, a key way in whichthese models can cause harm is through grossly unexpected outputs, corresponding to inputs thatare poorly represented in training data.
Accordingly, we approach safety assessment for a model bydetermining where it deviates the most from the output of a reference model and by how much (i.e., itsmaximum deviation ).
The reference model, which represents expected behavior and is deemed to besafe, could be a model well-understood by domain experts or one that has been extensively “tried andtested.”
The maximization is done over a certiﬁcation set , a large subset of the input space intendedto cover all conceivable inputs to the model.
These concepts are discussed further in Section 2.
Towards answering the second question, in Section 4 we discuss computation of the maximumdeviation for different model classes and show how this is facilitated by interpretability.
For modelclasses regarded as interpretable, including trees, generalized linear and additive models, the maximum deviation can be computed exactly and efﬁciently by exploiting the model structure.
For treeensembles, which are not regarded as interpretable, discrete optimization techniques can exploittheir composition in terms of trees to provide anytime bounds on the maximum deviation.
Thecase of trees is also generalized in a different direction to a broader class of piecewise Lipschitzfunctions, which we argue cover many popular interpretable functions.
Here we show that the beneﬁtof interpretability is signiﬁcantly tighter regret bounds on the maximum deviation compared withgeneral black-box functions, leveraging results from the multi-armed bandit literature.
More broadly,the development of tailored methods for additional model classes is beyond the scope of this ﬁrstwork on the maximum deviation approach (the black-box optimization of Section 4.4 is applicable toall models but obviously not tailored).
We discuss in Appendix B.5 some possible approaches and theresearch gaps to be overcome for neural networks and to make use of post hoc explanations, whichapproximate a model locally [6–8] or globally [9, 10].
In Section 5, we present case studies that illustrate the deviation maximization methods in Section 4 fordecision trees, linear and additive models, and tree ensembles.
It is seen that deviation maximizationprovides insights about models through studying the feature combinations that lead to extreme outputs.
These insights can in turn direct further investigation and invite domain expert input.
We also quantifyhow the maximum deviation depends on model complexity and the size of the certiﬁcation set.
Overall, our discussion provides a more quantitative basis for safety assessment of predictive modelsand for preferring more interpretable models due to the greater ease of performing this assessment.
Assessing Safety Through Maximum Deviation
We are given a supervised learning model f , which is a function mapping an input feature space X toan output space Y .
We wish to assess the safety of this model by ﬁnding its largest deviation froma given reference model f 0 : X
(cid:55)→ Y representing expected behavior.
To do this, we additionallyrequire 1) a measure of deviation D : Y × Y
(cid:55)→ R + , where R + is the set of non-negative reals, and2) a certiﬁcation set C ⊆ X over which the deviation is maximized.
Then the problem to be solved is
[Equation]
The deviation is worst-case because the maximization is over all x ∈ C ; further implications of thisare discussed in Appendix C. Note that (1) is different than typical robust training where the focus isto learn a model that minimizes some worst case loss, as opposed to ﬁnding regions in X where twoalready trained models differ signiﬁcantly.
We view problem (1) as a means toward the goal of evaluating safety.
In particular, a large deviationvalue is not necessarily indicative of a safety risk, as two models may differ signiﬁcantly for validreasons.
For example, one model may capture a useful pattern that the other does not.
What largedeviation values do indicate, however, is a (possibly) sufﬁcient reason for further investigation.
Hence,the maximizing solutions in (1) (i.e., the arg max ) are of as much operational interest as the maximumvalues (this will be illustrated in Section 5).
We now elaborate on elements in problem formulation (1).
Output space Y .
In the case of regression, Y is the set of reals R or an interval thereof.
In the caseof binary classiﬁcation, while Y could be { 0 , 1 } or {− 1 , +1 } , these limit the possible deviations tobinary values as well (“same” or “different”).
Thus to provide more informative results, we take Y tobe the space of real-valued scores that are thresholded to produce a binary label.
For example, y couldbe a predicted probability in [0 , 1] or a log-odds ratio in R .
Similarly for multi-class classiﬁcationwith M classes, Y ⊂ R M could be a M -dimensional space of real-valued scores.
In Appendix A, wediscuss considerations in choosing the deviation function D as well as models that abstain.
Reference model f 0 .
The premise of the reference model is that it should capture expected behaviorwhile being “safe”.
The simplest case is for f 0 to be a constant function representing a baseline value,for example zero or a mean prediction.
We consider the more general case where f 0 may vary with x .
For example, f 0 may be a model validated against domain knowledge or constrained by it (or mightnot arise from ML at all).
Such models are prevalent in high-stakes domains where interpretability is critical.
Examples are consumer credit scoring algorithms, medical risk and diagnostic scores, anddecision trees for predicting semiconductor yield.
The reference model could also be one that is notnecessarily informed by domain knowledge but has been extensively tested and deployed so that weregard it as safe.
In this case, f could be a new version of the model, trained on more recent data orimproved in some fashion, and we wish to evaluate its safety before deploying it in place of f 0 .
Certiﬁcation set C .
The premise of the certiﬁcation set is that it contains all inputs that the modelmight conceivably be exposed to.
This may include inputs that are highly improbable but notphysically or logically impossible (for example, a severely hypothermic human body temperature of27°C).
Thus, while C might be based on the support set of a probability distribution or data sample, itdoes not depend on the likelihood of points within the support.
The set C may also be a strict supersetof the training data domain.
For example, a model may have been trained on data for males, and wewould now like to determine its worst-case behavior on an unseen population of females.
For tabular or lower-dimensional data, C might be the entire input space X .
For non-tabular orhigher-dimensional data, the choice C =
X may be too unrepresentative because the manifold ofrealistic inputs is lower in dimension.
In this case, if we have a dataset { x i } ni =1 , one possibility is touse a union of (cid:96) p balls centered at x i ,
C = n (cid:91) i =
[Equation]
B pr [
[Equation]
The set C is thus comprised of points somewhat close to the n observed examples x i , but the radius rdoes not have to be “small”.
Related Work
In previous work on safety and interpretability in ML, the authors of [3, 11] give qualitative accountssuggesting that directly interpretable models are an inherently safe design because humans caninspect them to ﬁnd spurious elements; in this paper, we attempt to make those qualitative suggestionsmore quantitative and automate some of the human inspection.
Furthermore, several other authorshave highlighted safety as a goal for interpretability [2, 4, 12, 13, 5], but again without quantitativedevelopment.
Moreover, the lack of consensus on how to measure interpretability motivates therelationship that we explore between interpretability and the ease of evaluating safety.
In the area of ML veriﬁcation, robustness certiﬁcation methods aim to provide guarantees thatthe classiﬁcation remains constant within a radius (cid:15) of an input point, while output reachability isconcerned with characterizing the set of outputs corresponding to a region of inputs [14].
A majordifference in our work is that we consider two models, a model f to be assessed and a referencef 0 , whereas the above notions of robustness and reachability involve a single model.
Anotherimportant difference is that our focus is global , over a comprehensive set C , rather than local to smallneighborhoods around input points; a local focus is especially true of neural network veriﬁcation[15–24].
We also study the role of interpretability in safety veriﬁcation.
Works in robust optimizationapplied to ML minimize the worst-case probability of error, but this worst case is over parameters off rather than values of x [25].
Thomas et al.
[26] present a framework where during model training, aset of safety tests is speciﬁed by the model designer in order to accept or reject the possible solution.
We build on related literature on robustness and explainability that deals speciﬁcally with treeensembles.
Mixed-integer programming (MIP) and discrete optimization have been proposed to ﬁndthe smallest input perturbation to ‘evade’ a classiﬁer [27] and to obtain counterfactual explanations[28].
MIP approaches are computationally intensive however.
To address this Chen et al.
[29]introduce graph based approaches for veriﬁcation on trees.
Their central idea, which we use, is todiscretize veriﬁcation computations onto a graph constructed from the way leaves intersect.
Theveriﬁcation problem is transformed to ﬁnding all maximum cliques.
Devos et al.
[30] expand on thisidea by providing anytime bounds by probing unexplored nodes.
Safety has become a critical issue in reinforcement learning (RL) with multiple works focusingon making RL policies safe [31–34].
There are two broad themes [35]: (i) a safe and veriﬁablepolicy is learned at the outset by enforcing certain constraints, and (ii) post hoc methods are used toidentify bad regimes or failure points of an existing policy.
Our current proposal is complementary tothese works as we focus on the supervised learning setup viewed from the lens of interpretability.
Nonetheless, ramiﬁcations of our work in the RL context are brieﬂy discussed in Appendix C.
Deviation Maximization for Speciﬁc Model Classes
In this section, we discuss approaches to computing the maximum deviation (1) for f belonging tovarious model classes.
We show the beneﬁt of interpretable model structure in different guises.
Exactand efﬁcient computation is possible for decision trees, and generalized linear and additive modelsin Sections 4.1 and 4.2.
In Section 4.3, the composition of tree ensembles in terms of trees allowsdiscrete optimization methods to provide anytime bounds.
For a general class of piecewise Lipschitzfunctions in Section 4.4, the application of multi-arm bandit results yields tighter regret bounds onthe maximum deviation.
While some of the results in this section may be less surprising, one of ourcontributions is to identify precise properties that allow them to hold.
We also show that intuitivemeasures of model complexity, such as the number of leaves or pieces or smoothness of functions,have an additional interpretation in terms of the complexity of maximizing deviation.
More broadly,the development of methods speciﬁc to additional model classes is beyond the scope of a singlework.
We discuss in Appendix B.5 possible approaches and the advances needed for neural networks(beyond applying the black-box methods of Section 4.4) and to make use of post hoc explanations.
To develop mathematical results and efﬁcient algorithms, we will sometimes assume that the referencemodel f 0 is from the same class as f .
In Appendix C, we discuss the case where f 0 may not beglobally interpretable, but may be so in local regions.
We will also sometimes assume that thecertiﬁcation set C and other sets are Cartesian products.
This means that C = (cid:81)
d j =1 C j , where fora continuous feature j , C j =
[ X j , X j ] is an interval, and for a categorical feature j , C j is a set ofcategories.
We mention relaxations of the Cartesian product assumption in Appendix B.2.
4.1 Trees
We begin with the case where f and f 0 are both decision trees.
A decision tree with L leaves partitionsthe input space X into L corresponding parts, which we also refer to as ‘leaves’.
We consider onlynon-oblique trees.
In this case, each leaf is described by a conjunction of conditions on individualfeatures and is therefore a Cartesian product as deﬁned above.
With L l ⊂ X denoting the l th leaf andy l ∈ Y the output value assigned to it, tree f is described by the function
[Equation]
and similarly for tree f 0 with leaves L 0 m and outputs y 0 m , m = 1 , . . .
, L 0 .
As discussed in [36, 37],rule lists where each rule is a condition on a single feature are one-sided trees in the above sense.
The partitioning of X by decision trees and their piecewise-constant nature simplify the computationof the maximum deviation (1).
Speciﬁcally, the maximization can be restricted to pairs of leaves( l, m ) for which the intersection L l ∩ L 0 m ∩ C is non-empty.
The intersection of two leaves L l ∩ L 0 mis another Cartesian product, and we assume that it is tractable to determine whether C intersects agiven Cartesian product (see examples in Appendix B.1).
For visual representation and later use in Section 4.3, it is useful to deﬁne a bipartite graph, with Lnodes representing the leaves L l of f on one side and L 0 nodes representing the leaves L 0 m of f 0 onthe other.
Deﬁne the edge set E = { ( l, m ) :
L l ∩ L 0 m ∩ C (cid:54) = ∅} ; clearly |E| ≤ L 0 L .
Then
[Equation]
We summarize the complexity of deviation maximization for decision trees as follows.
This is a slightreﬁnement of [29, Thm. 1] in the case K = 2 , see Appendix B.1 for details.
Proposition 1.
Let f and f 0 be decision trees as in (3) with L and L 0 leaves respectively, and E bethe bipartite edge set of leaf intersections deﬁned above.
Then the maximum deviation (1) can becomputed with |E| evaluations as shown in (4) .
4.2 Linear and additive models
In this subsection, we assume that f is a generalized additive model (GAM) given by
[Equation]
where each f j is an arbitrary function of feature x j .
In the case where f j ( x j ) =
w j x j for allcontinuous features x j , where w j is a real coefﬁcient, (5) is a generalized linear model (GLM).
Wediscuss the treatment of categorical features in Appendix B.2.
The invertible link function g : R (cid:55)→ Ris furthermore assumed to be monotonically increasing.
This assumption is satisﬁed by common
GAM link functions: identity, logit ( g ( y ) = log( y/ (1 − y )) ), and logarithmic.
Equation (5) implies that Y ⊂ R and the deviation D ( y, y 0 ) is a function of two scalars y and y 0 .
Forthis scalar case, we make the following intuitively reasonable assumption throughout the subsection.
Assumption 1.
For y, y 0 ∈ Y ⊆ R , 1) D ( y, y 0 )
=
whenever y = y 0 ; 2) D
( y, y 0 ) is monotonicallynon-decreasing in y for y ≥ y
and non-increasing in y for y ≤ y 0 .
Our approach is to exploit the additive form of (5) by reducing problem (1) to the optimization
[Equation]
for different choices of S ⊂ X
and where + corresponds to max and − to min .
We discuss belowhow this can be done for two types of reference model f 0 : decision tree (which includes the constantcase L 0 = 1 ) and additive.
For the ﬁrst case, we prove the following result in Appendix B.2:
Proposition 2.
Let f be a GAM as in (5) and S be a subset of X where f 0 ( x ) ≡ y 0 is constant.
Thenif Assumption 1 holds,
[Equation]
Tree-structured f 0 .
Since f 0 is piecewise constant over its leaves L 0 m , m = 1 , . . .
, L 0 , we take Sto be the intersection of C with each L 0 m in turn and apply Proposition 2.
The overall maximum isthen obtained as the maximum over the leaves,
[Equation]
This reduces (1) to solving 2 L 0 instances of (6).
Additive f 0 .
For this case, we make the additional assumption that the link function g in (5) is theidentity function, as well as Assumption 2 below.
The implication of these assumptions is discussedin Appendix B.2.
Assumption 2.
D ( y, y 0 )
=
D ( y − y 0 ) is a function only of the difference y − y 0 .
Then f 0 ( x ) = (cid:80) dj
=1 f 0 j ( x j ) and the difference f ( x ) − f 0 ( x ) is also additive.
Using Assumptions 2,1 and a similar argument as in the proof of Proposition 2, the maximum deviation is again obtainedby maximizing and minimizing an additive function, resulting in two instances of (6) with S
= C :
[Equation]
Computational complexity of (6) .
For the case of nonlinear additive f , we additionally assume that
C is a Cartesian product.
It follows that S =
(cid:81)
dj
=1 S j is a Cartesian product (see Appendix B.2 forthe brief justiﬁcation) and (6) separates into one-dimensional optimizations over S j ,
[Equation]
The computational complexity of (8) is thus (cid:80) dj
=1 C j , where C j is the complexity of the j th onedimensional optimization.
We discuss different cases of C j in Appendix B.2; the important point isthat the overall complexity is linear in d .
In the GLM case where (cid:80) dj =1 f j ( x j ) =
w T x , problem (6) is simpler and it is less important that Cbe a Cartesian product.
In particular, if C is a convex set, so too is S (again see Appendix B.2 forjustiﬁcation).
Hence (6) is a convex optimization problem.
4.3 Tree ensembles
We now extend the idea used for single decision trees in Section 4.1 to tree ensembles.
This classcovers several popular methods such as Random Forests and Gradient Boosted Trees.
It can alsocover rule ensembles [38, 39] as a special case, as explained in Appendix B.3.
We assume f is a treeensemble consisting of K trees and f 0 is a single decision tree.
Let L l k denote the l th leaf of the k thtree in f for l = 1 , . . .
, L k , and L 0 m be the m th leaf f 0 , for m = 1 , . . .
, L 0 .
Correspondingly let y l kand y
m denote the prediction values associated with each leaf.
Deﬁne a graph G ( V , E ) , where there is a vertex for each leaf in f and f 0 , i.e.
[Equation]
Construct an edge for each overlapping pair of leaves in V , i.e.
[Equation]
This graph is a K + 1 -partite graph as leaves within an individual tree do not intersect and arean independent set.
Denote M to be the adjacency matrix of G .
Following Chen et al.
[29], amaximum clique S of size K
+ 1 on such a graph provides a discrete region in the feature space witha computable deviation.
A clique is a subset of nodes all connected to each other; a maximum cliqueis one that cannot be expanded further by adding a node.
The model predictions y c and y 0 c can beensembled from leaves in S .
Denote by D ( S ) the deviation computed from the clique S .
Maximizingover all such cliques solves (1).
However, complete enumeration is expensive, so informative bounds,either using the merge procedure in Chen et al.
[29] or the heuristic function in Devos et al.
[30] canbe used.
We use the latter which exploits the K + 1 -partite structure of G .
Speciﬁcally, we adapt the anytime bounds of Devos et al.
[30] as follows.
At each step of theenumeration procedure, an intermediate clique S contains selected leaves from trees in [1 , . . .
, k ] andunexplored trees in [ k + 1 , . .
, K + 1] .
For each unexplored tree, we select a valid candidate leafthat maximizes deviation, i.e.
[Equation]
Using these worst-case leaves, a heuristic function
[Equation]
provides an upper (dual) bound.
In practice, this dual bound is tight and therefore very useful duringthe search procedure to prune the search space.
Each K + 1 clique provides a primal bound, so thesearch can be terminated early before examining all trees if the dual bound is less than the primalbound.
We adapt the search procedure of Mirghorbani and Krokhmal [40] to include the pruningarguments.
Appendix B.3 presents the full algorithm.
Starting with an empty clique, the procedureadds a single node from each tree to create an intermediate clique.
If the size of the clique is K
+ 1the primal bound is updated.
Otherwise, the dual bound is computed.
A node compatibility vector isused to keep track of all feasible additions.
When the search is terminated at any step, the maximumdeviation is bounded by ( D lb , D ub ) .
The algorithm works for the entire feature space.
When the certiﬁcation set C is a union of balls as in(2), some additional considerations are needed.
First, we can disregard leaves that do not intersectwith C during the graph construction phase.
A validation step to ensure that the leaves of a clique allintersect with the same ball in C is also needed.
4.4 Piecewise Lipschitz Functions
We saw the beneﬁts of having speciﬁc (deterministic) interpretable functions as well as their extensionsin the context of safety.
Now consider a richer class of functions that may also be randomized withﬁnite variance.
In this case let f and f 0 denote the mean values of the learned and reference functionsrespectively.
We consider the case where each function is either interpretable or black box, where thelatter implies that query access is the only realistic way of probing the model.
This leads to threecases where either both functions are black box or interpretable, or one is black box.
What we care about in all these cases 1 is to ﬁnd the maximum (and minimum) of a function ∆( x ) =
f ( x ) − f 0 ( x ) .
Let us consider ﬁnding only the maximum of ∆ as the other case is symmetric.
Given that f and f 0can be random functions ∆ is also a random function and if ∆ is black box a standard way to optimizeit is either using Bayesian Optimization (BO) [41] or tree search type bandit methods [42, 43].Werepurpose
some of the results from this latter literature in our context showcasing the beneﬁt ofinterpretability from a safety standpoint.
To do this we ﬁrst deﬁne relevant terms.
Deﬁnition 1 (Simple Regret [42]) .
If f ∗C denotes the optimal value of the function f on the certiﬁcation set C , then the simple regret r C q after querying the f function q times and obtaining a solution x qis given by, r C q ( f ) =
f ∗C − f ( x q ) .
Deﬁnition 2 (Order β c-Lipschitz) .
Given a (normalized) metric (cid:96) a function f is c-Lipschitzcontinuous of order β >
if for any two inputs x , y and for c > 0 we have, | f ( x ) − f ( y ) | ≤c · (cid:96) ( x, y ) β .
Deﬁnition 3 (Near optimality dimension [42]) .
If N ( C , (cid:96), (cid:15) ) is the maximum number of (cid:15) radiusballs one can ﬁt in C given the metric (cid:96) and C (cid:15)
= { x ∈ C| f
( x ) ≥ f ∗C − (cid:15) } , then for c > 0 the c-nearoptimality dimension is given by, υ = max (cid:16) lim sup (cid:15) → 0 ln N ( C c(cid:15) ,(cid:96),(cid:15) )
ln( (cid:15) − 1 ) , 0 (cid:17) .
Intuitively, simple regret measures the deviation between our current best and the optimal solution.
The Lipschitz condition bounds the rate of change of the function.
Near optimality dimensionmeasures the set size for which the function has close to optimal values.
The lower the value of υ , theeasier it is to ﬁnd the optimum.
We now deﬁne what it means to have an interpretable function.
Assumption 3 (Characterizing an Interpretable Function) .
If a function f is interpretable, then wecan (easily) ﬁnd 1 ≤ m (cid:28) n partitions {C (1) , ..., C ( m ) } of the certiﬁcation set C such that thefunction f
( i ) =
{ f ( x ) | x ∈ C ( i ) } ∀ i ∈ { 1 , ..., m } in each partition is c-Lipschitz of order β .
Note that the (interpretable) function overall does not have to be c-Lipschitz of bounded order, ratheronly in the partitions .
This assumption is motivated by observing different interpretable functions.
For example, in the case of decision trees the m partitions could be its leaves, where typically thefunction is a constant in each leaf ( c = 0 ).
For rule lists as well a ﬁxed prediction is usually madeby each rule.
For a linear function one could consider the entire input space (i.e. m = 1 ), where forbounded slope α the function would also satisfy our assumption ( c = α and β = 1 ).
Examples ofmodels that are not piecewise constant or globally Lipschitz are oblique decision trees (Murthy etal., 1994), regression trees with linear functions in the leaves, and functional trees.
Moreover, m islikely to be small so that the overall model is interpretable (viz.
shallow trees or small rules).
Withthe above deﬁnitions and Assumption 3 we now provide the simple regret for the function ∆ .1.
Both black box models: If both f and f 0 are black box then it seems no gains could be made inestimating the maximum of ∆ over standard results in bandit literature.
Hence, using Hierarchical
Optimistic Optimization (HOO) with assumptions such as C being compact and ∆ being weakly
Lipschitz [42] with near optimality dimension υ the simple regret after q queries is:
[Equation]
Both interpretable models: If both f and f 0 are interpretable, then for each function basedon Assumption 3 we can ﬁnd m 1 and m 0 partitions of C respectively where the functions are c
1and c 0 -Lipschitz of order β 1 and β 0 respectively.
Now if we take non-empty intersections of thesepartitions where we could have a maximum of m 1 m 0 partitions, the function ∆ in these partitionswould be c = 2 max( c 0 , c 1 ) -Lipschitz of order β = min( β 0 , β 1 ) as stated next (proof in appendix).
Proposition 3.
If functions h 0 and h 1 are c 0 and c 1 Lipschitz of order β 0 and β 1 respectively, thenthe function h
= h
− h 1 is c-Lipschtiz of order β , where c = 2 max( c 0 , c 1 ) and β = min( β 0 , β 1 ) .
Given that ∆ is smooth in these partitions with underestimated smoothness of order β , the simpleregret after q i queries in the i th partition C ( i ) with near optimality dimension υ
i based on HOO is:
[Equation]
(cid:17) , where υ i ≤ dβ .
If we divide the overall query budget q across the π ≤ m 0 m 1 non-empty partitions equally, then the bound will be scaled by π 1 / ( υ i +2) when expressedas a function of q .
Moreover, the regret for the entire C can then be bounded by the maximum regretacross these partitions leading to
[Equation]
Notice that for a model to be interpretable m 0 and m 1 are likely to be small (i.e. shallow trees orsmall rule lists or linear model where m = 1 ) leading to a “smallish" π and υ can be much >> dβ incase 1.
Hence, interpretability reduces the regret in estimating the maximum deviation.3.
Black box and interpretable model: Making no further assumptions on the black box model andassuming ∆ satisﬁes properties mentioned in case 1, the simple regret has the same behavior as (13).
This is expected as the black box model could be highly non-smooth.
Case Studies
We present case studies to serve three purposes: 1) show that deviation maximization can lead toinsights about models, 2) illustrate the maximization methods developed in Section 4, and 3) quantifythe dependence of the maximum deviation on model complexity and certiﬁcation set size (mostly in
Appendix D).
Two datasets are featured: a sample of US Home Mortgage Disclosure Act (HMDA)data (see Appendix D.2 for details), meant as a proxy for a mortgage approval scenario, and the UCI
Adult Income dataset [44], a standard tabular dataset with mixed data types.
A subset of results isshown in this section with full results, experimental details, and an additional Lending Club datasetin Appendix D. Since these are binary classiﬁcation datasets, we take the deviation function D tobe the absolute difference between predicted probabilities of class 1.
For the certiﬁcation set C , weconsider a union of (cid:96) ∞ balls (2) centered at test set instances.
While we have used the test set here,any not necessarily labelled dataset would sufﬁce.
The case r = 0 yields a ﬁnite set consisting onlyof the test set, while r → ∞ corresponds to C being the entire domain X .
To demonstrate insights from deviation maximization, we study the solutions that maximize deviation(the arg max in (1)) and discuss three examples below.
Identiﬁcation of an artifact: The ﬁrst example comes from the Adult Income dataset, where thereference model f 0 is a decision tree (DT) and f is an Explainable Boosting Machine (EBM) [45], atype of GAM (plots of both in Appendix D.3).
Here, the capital loss feature is the largest contributorto the maximum deviation (the discussion below Table 4 explains how this is determined), and Table 8in Appendix D.3 shows that as the certiﬁcation set radius r increases, the maximizing values ofcapital loss converge to the interval [1598 , 1759] .
The plot of the GAM shape function f j for capitalloss in Figure 1 shows that this interval corresponds to a curiously low value of the function.
Thislow region may be an artifact warranting further investigation since it seems anomalous compared tothe rest of the function, and since individuals who report capital losses on their income tax returns tooffset capital gains usually have high income (hence high log-odds score).
Note that this potentialartifact was automatically identiﬁed through deviation maximization.
For the next two examples, we consider a simpliﬁed mortgage approval scenario using the HMDAdataset.
Suppose that a DT f 0 (shown in Figure 3 in Appendix D.2) has been trained to make ﬁnaldecisions on mortgage applications.
Domain experts have determined that this DT is sensible andsafe and are now looking to improve upon it by exploring EBMs.
(Logistic regression (LR) modelsare deferred to Appendix D.2 because they do not have higher balanced accuracy than f 0 .)
Conﬂict between f , f 0 : We ﬁrst examine solutions that result in the most positive difference betweenthe predicted probabilities of an EBM f with parameter max_bins =32 and the DT f 0 .
These alloccur in a leaf of f 0 (leaf 2 in Figure 3) where the applicant’s debt-to-income (DTI) ratio is too high( > 52%) and f 0 predicts a low probability of approval.
The other salient feature of the solutionsis that they all have ‘preapproval’=1, indicating that a preapproval was requested, which is given alarge weight by the EBM f in favor of approval (see Figure 4 in Appendix D.2, and Table 4 for morefeature values).
Thus f and f 0 are in conﬂict.
Among different ways in which the conﬂict could beresolved, a domain expert might decide that f 0 remains correct in rejecting risky applicants with high
DTI, even if there is a preapproval request and the new EBM model puts a high weight on it.
[Table] [Table]
[Figure] [Figure]
Trend toward extreme points, deviation can be good: We now look at solutions that yield themost negative difference between the predicted probabilities of f and f 0 , for r ≤ 0 . 8 .
Table 1 showsthe 6 features that contribute most to the deviation (again see Appendix D.2 for details).
All of thesepoints lie in a leaf of f 0 (leaf 14 in Figure 3, denoted L 14 ) that excludes several clearer-cut cases,with the result being a less conﬁdent predicted probability of 0 .
652 from f 0 .
The feature valuesthat maximize deviation tend toward extreme points of the region L 14 .
Speciﬁcally, the values ofthe continuous features debt-to-income ratio, loan-to-value ratio, property value, and income allmove in the direction of application denial.
For the latter three features, the boundary of L 14 isreached as soon as r = 0 .
1 , whereas for debt-to-income ratio, this occurs at r = 0 .
The movementtoward extremes is expected for this f since its relevant shape functions f
j are mostly increasingor decreasing, as seen in Figure 4 in Appendix D.2.
The same behavior is observed in other GAMand
LR examples in Appendix D.
In this example, a domain expert might conclude that the largedeviation is in fact desirable because f is providing varying predictions in L 14 in ways that makesense, as opposed to the constant given by f 0 .
This shows that deviation maximization can work inboth directions, identifying where the reference model and its assumptions may be too simplistic.
Maximum deviation vs. number of trees in a RF:
In Figure 2, we highlight one result from a setof such results in Appendix D, showing maximum deviation as a function of model complexity, herequantiﬁed by the number of estimators (trees) in a Random Forest (RF).
This is a demonstration ofthe method in Section 4.3, which in general provides bounds on the maximum deviation.
In thiscase, the upper (“dual”) bound is informative enough to actually show a decrease as the number ofestimators increases.
The larger number of estimators increases averaging and may serve to make themodel smoother.
Conclusion
We have considered the relationship between interpretability and safety in supervised learning throughtwo main contributions: First, the proposal of maximum deviation as a means toward assessing safety,and second, discussion of approaches to computing maximum deviation and how these are simpliﬁedby interpretable model structure.
We believe that there is much more to explore in this relationship.
Appendices C and B.5 provide further discussion of several topics and future directions.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix C in particular.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix C.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code is
proprietary at this time due to our institutional obligations.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix D.
(c) Did you report error bars (e g., with respect to the random seed after running exper-
iments multiple times)?
[N/A] We report case studies assessing the safety of ﬁxed
models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] The HMDA data comes from the US
government and does not appear to have a license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table]
information or offensive content?
[Yes] None of the data contain personally identiﬁable [Table]
information, and the US Consumer Finance Protection Bureau indicates that they have
[Table]
modiﬁed the HMDA data to protect applicant and borrower privacy. [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Additional Problem Formulation Details
Reference model examples Below we give several examples of possible reference models.
Thesecan be categorized as 1) existing domain-speciﬁc models, 2) interpretable ML models validated bydomain knowledge, and 3) extensively tested and deployed models.
The ﬁrst two categories areprevalent in high-stakes domains where interpretability is critical. Existing domain-speciﬁc models: These models originate from an application domain and may not be based on ML at all. For example in consumer ﬁnance, several industry-standard
models compute credit scores from a consumer’s credit information (the FICO score is thebest-known in the US).
Similarly in medicine, scoring systems (sparse linear models withsmall integer coefﬁcients) abound for assessing various risks (the CHADS 2 score for stroke
risk is well-known, see the "Scoring Systems: Applications and Prior Art" section of [46] for
a list of others).
These models have been used for decades by thousands of practitioners so
they are very well understood.
They may very well be improved upon by a more ML-based
model
but for such a model to gain acceptance with domain experts, any large deviations
from existing models need to be examined and understood.
Interpretable models validated by domain knowledge: Here, an interpretable ML model
is learned from data and is validated by domain experts in some way, for example byselecting important input features or by carefully inspecting the trained model.
We provide
two real examples:
In semiconductor manufacturing, process engineers typically want
decision trees [47] to model their respective manufacturing process (e
g. Etching, Polishing,
Rapid Thermal Processing, etc.)
since they are comfortable understanding and explaining
them to their superiors, which is critical especially when things go out-of-spec.
Hence, a
tree built from data (or any model in general) would only be allowed to make automated
measurement predictions if the features it highlights (viz.
pressures,
gas
ﬂows
, temperatures
)make
sense
for the speciﬁc process.
Similarly, in predicting failures of industrial assetssuch as wind turbines, some failure data is available to train models but experts in thesesystems (e.g. engineers) may also be consulted.
They have knowledge that can help validatethe model, for example which components are most likely to cause failures or whichenvironmental variables (e.g. temperature) are most inﬂuential.3.
Extensively tested and deployed models: A reference model may also be one that is notnecessarily informed by domain knowledge but has been extensively tested, deployed, and/orapproved by a regulator.
For medical devices that use ML models, the US Food and Drug
Administration (FDA) has instituted a risk-based regulatory system.
Any system updatesor changes, for instance changes in model architecture, retraining based on new data, orchanges in intended use (e.g. use for pediatric cases for devices approved only for adults),need to either seek new approvals or demonstrate “substantial equivalence” by providingsupporting evidence that the revised model is similar to a previously approved device.
In thelatter case, the reference model is the approved device and small maximum deviation servesas evidence of equivalence.
As another example, consider a ML-based recommendationmodel for products of an online retailer or articles on a social network, where because ofthe scale, a tree ensemble may be used for its fast inference time as well as its modelingﬂexibility.
In this case, a model that has been deployed for some time could be the referencemodel, since it has been extensively tested during this time even though human validation ofit may be limited.
When a new version of the model is trained on newer data or improved insome fashion, ﬁnding its maximum deviation from the reference model can serve as onesafety check before deploying it in place of the reference model.
Deviation function D
For the case where the inputs y , y 0 to D are real-valued scalars (whichcovers binary classiﬁcation and regression), while Assumption 1 was stated as a sufﬁcient conditionfor tractable optimization with GAMs, it is also an intuitively reasonable requirement: the deviationshould increase the farther y is from y 0 in either direction.
In addition, symmetry may be desirable,i.e. D ( y, y 0 )
=
D ( y 0 , y ) , to not favor one of the two models over the other.
{+Both Assumptions 1and 2 as well as symmetry are satisﬁed by monotonically increasing functions D ( | y − y 0 | ) of theabsolute difference, for example powers | y − y 
+}
| p for p > 0 .
For the case where y, y 0 ∈ R M as in multi-class classiﬁcation, it may be advantageous for D ( y, y 0 ) todecompose into a sum over output dimensions: D ( y, y 0 )
{+= (cid:80) Mk =1 D k ( y k , y 0 k ) , where y k , y 
+}
k are the components of y , y 0 .
{+For example, the p th power of the (cid:96) p distance (cid:107) y − y 
+}
(cid:107) pp
=
{+(cid:80) Mk =1 | y k − y 
+}
k | pis separable in this manner.
Models that abstain The formulation in Section 2 can also accommodate models that abstain frompredicting (and possibly defer to a human expert or other fallback system).
If f ( x ) = ∅ , representingabstention, then we may set D ( ∅ , y 0 ) =
d for any y 0 ∈ Y , where d > 0 is an intermediate value lessthan the maximum value that D can take [48].
The value d might also be less than a “typically bad”value for D , to reward the model for abstaining when it is uncertain.
B Additional Details on Deviation Maximization for Speciﬁc Model ClassesB.1 Trees
Rule lists A rule list as deﬁned by Yang et al.
[36], Angelino et al.
[37] is a nested sequence of
IF-THEN-ELSE statements, where the IF condition is a conjunctive rule and the THEN consequentis an output value.
If each rule involves a single feature (i.e., the conjunctions are of degree 1 ), suchrule lists are one-sided trees in the sense of Section 4.1.
The number of leaves in the equivalent treeis equal to the number of rules in the list (including the last default rule).
Intersection of C with a Cartesian product If C = (cid:81) dj
=1 C j is also a Cartesian product, thendetermining whether the intersection is non-empty amounts to checking whether all of the coordinatewise intersections with C j , j = 1 , . . .
, d , are non-empty.
If C is not a Cartesian product but is a unionof (cid:96) ∞ balls (which are Cartesian products), then the intersection is non-empty if the intersection withany one ball is non-empty.
Relationship between Proposition 1 and [29, Theorem 1]
In the case of a K = 2 -tree ensemble,[29, Theorem 1] bounds the complexity of exact robustness veriﬁcation as min { O ( n 2 ) , O ((4 n ) d } =
O ( n 2 ) , where n is the maximum number of leaves in a tree and we assume that the feature dimensiond ≥ 2 .
In Proposition 1, we account for the possibly different numbers of leaves L and L 0 in the twotrees f and f 0 , and we exactly enumerate the edges, |E| ≤ L 0 L ≤ n 2 .
Additive reference model For the case where f is a decision tree and f 0 is a generalized additivemodel, if the deviation function is symmetric, D ( y, y 0 )
=
D ( y 0 , y ) , then this case is covered in
Section 4.2.
B.2 Linear and Additive Models
Categorical features A function f j ( x j ) of a categorical feature x j can be represented in two ways,depending on whether f is a GAM or a GLM.
In the GAM case, we may use the native representationin which x j takes values in a ﬁnite set X j of categories.
In the GLM case, x j is one-hot encoded intomultiple binary-valued features x jk , one for each category k .
Then any function f j can be representedas a linear function,
[Equation]
where w jk is the value of f j for category k .
Implication of Assumption
The second condition implies that the deviation increases or staysthe same as y moves away from y 0 in either direction.
Proof of Proposition 2.
Let x ∈ S and S ( x ) = (cid:80)
dj
=1 f j ( x j ) .
Under Assumption 1.1, if S ( x ) =
g ( y 0 ) , then
[Equation]
As S ( x ) increases from g ( y 0 ) , f ( x ) also increases because g − 1 is an increasing function, and
D ( f ( x ) , y 0 ) increases or stays the same due to Assumption 1.2.
Similarly, as S ( x ) decreases fromg ( y 0 ) , f ( x ) decreases, and D ( f ( x ) , y 0 ) again increases or stays the same.
It follows that to maximize
D ( f ( x ) , y 0 ) , it sufﬁces to separately maximize and minimize S ( x ) , compute the resulting values of
D ( f ( x ) , y 0 ) , and take the larger of the two.
This yields the result.
Implication of Assumption 2 and identity link function g
These two assumptions imply that thedeviation is measured on the difference between f and f 0 in the space in which they are additive.
For example, if f and f 0 are logistic regression models predicting the probability of belonging toone of the classes, the difference is taken in the log-odds (logit) domain.
It is left to future work todetermine other assumptions under which problem (1) is tractable when f and f 0 are both additive.
Cartesian product C implies Cartesian product S
In the cases of constant and additive f 0 , S = C .
In the decision tree case, since each leaf is a Cartesian product L 0 m = (cid:81)
dj
=1 R mj , the intersections
S =
L 0 m ∩ C are also Cartesian products (cid:81)
dj
=1 S j where S j = R mj ∩ C j .
One-dimensional optimization complexities C j
For discrete-valued x j , C j is proportional to thenumber of allowed values |S j | .
For continuous x j , it is common to use spline functions or treeensembles as f j in constructing GAMs.
In the former case, C j is proportional to the number ofknots.
In the latter, the tree ensemble can be converted to a piecewise constant function and C j isthen proportional to the number of pieces.
Lastly in the case where f j ( x j ) =
w j x j is linear and
S j =
[ X j , X j ] is an interval, C j
= O (1) because it sufﬁces to evaluate the two endpoints.
Convex S
If C is a convex set, then in the cases of constant and additive f 0 , S
= C is alsoconvex.
In the case of tree-structured f 0 , S =
L 0 m ∩ C and each leaf L 0 m can be represented asa convex set, with interval constraints on continuous features and set membership constraints oncategorical features.
The latter can be represented as x jk = 0 constraints on the one-hot encoding(see “Categorical features” paragraph above) for non-allowed categories k .
Hence S is also convex.
As a speciﬁc example, suppose that S is the product of independent constraints on each categoricalfeature and an (cid:96) p norm constraint on the continuous features jointly.
The maximization over eachcategorical feature has complexity C j = |S j | as noted above, while the maximization of w T x overcontinuous features lying in an (cid:96) p ball has closed-form solutions for the common cases p = 1 , 2 , ∞ .
Relaxations of the Cartesian product assumption If the certiﬁcation set C is not a Cartesianproduct, then one way to still bound the maximum deviation is to ﬁnd the smallest Cartesian product
C that contains C and maximize deviation over C .
As long as it is relatively easy to optimize linearfunctions over C , then constructing such a Cartesian product is similarly easy.
Another conceivablerelaxation of the Cartesian product assumption is a Cartesian product of low-dimensional sets, notjust one-dimensional.
B.3 Tree Ensembles
The full algorithm for clique search from Section 4.3 is presented in Algorithm 1.
It uses Z as a nodecompatibility vector to keep track of valid leaves and B a set of trees/partites not yet covered by themaximum clique.
The algorithm starts with and empty clique S and anytime bounds as 0 .
It starts thesearch with the smallest tree to limit the search space.
This is typically f 0 .
Each leaf is added to theintermediate clique S in turn (Line 6).
A stronger primal bound can be achieved if the traversal isordered in a meaningful way.
In particular, starting with nodes with the highest heuristic functionvalue H ( S ) aids the algorithm to focus on better areas of the search space.
If the size of the clique is K + 1 the primal bound is updated.
Otherwise, the dual bound is computed.
If the node is promising, the algorithm recurses to the next level.
When the search is terminated atany step, the maximum deviation is bounded by ( D lb , D ub ) .
Rule ensembles Similar to the tree ensembles considered in Section 4.3, a rule ensemble is a linearcombination of conjunctive rules, where the antecedent is a conjunction of conditions on individualfeatures, and the consequent takes a real value if the antecedent is true and zero otherwise.
Theyare produced by algorithms such as SLIPPER [49], that of Rückert and Kramer [50], RuleFit [38],
ENDER [39] and have also been referred to as generalized linear rule models [51].
A rule ensemblecan be converted into a tree ensemble by converting each conjunctive rule into an IF-THEN-ELSE rule
Require: M adjacency matrix, H heuristic function 1: Z
[ i ] = 1 ∀ i ∈ V , B = { 1 , 2 , . .
, K + 1 } , S = ∅ (cid:46)
All nodes valid, all trees uncovered 2: Q =
Enumerate( Z, B, S ) 3: Initialize: D lb = 0 , D ub =(cid:46)
Anytime bounds 4: function Enumerate( Z, B, S ): 5: t = arg max b {| Z b | (cid:12)(cid:12) b ∈ B } (cid:46)
Uncovered tree with fewest valid nodes 6: for i in Z t do 7:
Z [ i ] =
(cid:46)
Mark node as incompatible 8:
S = S ∪ { i } (cid:46) Add to candidate clique 9: if | S | = K + 1 then 10:
D lb = max ( D lb , D ( S ))
(cid:46) Update primal bound 11:
Q = Q ∪ S (cid:46)
Add to set of max cliques 12:
S =
S \ { i } (cid:46) Backtrack 13: else 14:
Z t +1 =
Z t ∧ M ( i ) (cid:46) Update valid nodes 15:
B = B \ { t } (cid:46) Update uncovered trees 16:
D ub = max ( D ub , H ( S ))
(cid:46) Update dual bound 17: if D ub > D lb then 18:
Enumerate( Z t +1 , B, S ) (cid:46) Recurse to next level 19: end if 20:
S =
S \ { i } (cid:46) Backtrack 21:
B = B ∪ { t } 22: end if 23: end for list, which is a one-sided tree (see Appendix B.1). Speciﬁcally, the conditions in the conjunction aretaken in any order, each condition is negated to become an IF condition, and the THEN consequentsare all output values of zero. The ﬁnal ELSE consequent, which is reached if all the IF conditions arefalse (and hence the original rule holds), returns the output value of the original rule. The number ofleaves in the resulting tree equals the number of conditions in the conjunction plus one.
B.4 Piecewise Lipschitz Functions
Proof of Proposition 3.
Consider two inputs x and y then, [Equation] where, c = 2 max( c 0 , c 1 ) and β = min( β 0 , β 1 ) . Other choices for D ( ., . )
: The results assumed D ( ., . ) to be the identity function, where ∆ =
D ( f 0 , f ) .
This choice of function clearly satisﬁes Assumptions 1 and 2.
Again consistent with theseassumptions we look at some other choices for D ( ., . ) .
If D ( ., . ) were an afﬁne function with apositive scaling such as D ( y 0 , y )
=
α ( y 0 − y )
+ b where α > 0 , then our result in equation 14 wouldbe unchanged as only the Lipschitz constant of ∆ would change, but not its (underestimated) order.
If the function were a polynomial or exponential however, no such guarantees can be made and wewould be back to case 1.
B.5 Other Model Classes: Neural Networks and Post Hoc Explanations
For model classes beyond the ones discussed in Section 4, it appears to be a greater challenge to obtainreasonably tractable algorithms that guarantee exact computation of or bounds on the maximumdeviation.
Here we outline some future directions for neural networks and post hoc explanations.
Robustness veriﬁcation for neural networks has attracted a great deal of attention and made considerable progress, with exact approaches including satisﬁability modulo theory [21] and mixed integer programming [22, 23], and incomplete methods that compute bounds using bound propagation [19, 16], linear programming and duality [15, 17, 18], and semideﬁnite programming [20, 24].
However, all of these methods consider a single model, effectively comparing it to a constant.
Robustness veriﬁcation is thus essentially a single-model case of our problem (1) in which f 0 is aconstant (and with an appropriate choice of the deviation function D ( f, f 0 ) ).
While we may expectthat solutions to a two-model veriﬁcation problem would leverage existing robustness veriﬁcationmethods, developing such solutions remains for future work.
Furthermore, evaluation of robustnessveriﬁcation methods has largely been limited to local neighborhoods around input points (with typicalradii (cid:15) ≤ 0 .
1 in terms of normalized feature values).
This limitation may also need to be addressed toenable evaluation of maximum deviation in the way envisioned in this paper.
It is also natural to ask whether post hoc explanations for the model can help.
One way in whichthis could occur is if the post hoc explanation approximates the model f by a simpler model ˆ fand if the deviation function D satisﬁes the triangle inequality D ( f ( x ) , f 0 ( x )) ≤ D ( f ( x ) , ˆ f ( x ))
+
D ( ˆ f ( x ) , f 0 ( x )) .
Then the maximum deviation in (1) would be bounded as
[Equation]
While we may choose ˆ f to be interpretable so that the rightmost maximization is tractable, the middlemaximization asks for a uniform bound on the deviation between f and ˆ f , i.e, the ﬁdelity of ˆ f .
Weare not aware of a post hoc explanation method that provides such a guarantee.
Indeed, in general,the middle maximization might not be any easier than the left-hand one that we set out to bound.
A (practical) possibility may be to perform quantile regression [52] for a large enough quantile to learnˆ f , as opposed to minimizing expected error as is typically done.
This may be an interesting directionto explore in the future as quantile regression algorithms are available for varied model classesincluding linear models, tree ensembles [53] and even neural networks [54].
More investigation isneeded into whether quantile regression methods can provide approximate guarantees on the middleterm in (15).
Assuming that uniform proxies in the above sense can be constructed, then for certain modalities orapplications it may be possible to train highly accurate proxies.
For instance for tabular data, Random
Forests or boosted trees might very well replicate the behavior of a neural network, in which casethe machinery introduced in Section 4.3 could be used.
Even for other modalities such as text andimages, interpretable models such as Neural Additive Models (NAMs) [55] and continued fractionnetworks (CoFrNets) [56] may prove to be sufﬁcient in some cases. Finally, there are recent architectures such as Lipschitz neural networks [57] which are adversariallyrobust and hence valuable in practice. Our analysis presented in Section 4.4 for piecewise Lipschitzmodels would be applicable here, where the simple regret of standard bandit algorithms for a givennumber of queries could be reduced to (14) as opposed to (13). C Further Discussion
Worst-case approach The formulation of (1) as the worst case over a certiﬁcation set representsa deliberate choice to depend as little as possible on a probability distribution or a dataset sampledfrom one.
As stated in Section 2, Certiﬁcation Set paragraph, C can depend at most on (an expandedversion of) the support set of a distribution.
The reason for this choice is because safety is an out-ofdistribution notion: harmful outputs often arise precisely because they were not anticipated in thedata.
The trade-off inherent in this choice is that the maximum deviation may be more conservativethan needed.
The high maximum deviation values in e.g. Figure 11 may reﬂect this.
Given deﬁnition(1) as a starting point in this paper, future work could consider variations that depend more on adistribution and are thus less conservative, but may also offer a weaker safety guarantee.
Choice of reference model
The proposed deﬁnition of maximum deviation (1) depends on thechoice of reference model f 0 .
Different choices will lead to different deviation values and, perhapsmore importantly, different combinations of features that maximize the deviation.
We have discussedpossible choices in Section 2, and the results in Section 4 indicate that, as with the assessed model f ,interpretable forms for f 0 can ease the computation of maximum deviation.
Beyond these guidelines,it is up to ML practitioners and domain experts to decide on appropriate reference models for their application (and there may be beneﬁt to considering more than one).
We mention an additionalconcern with the reference model in the Ethics Discussion.
For some real applications it may be difﬁcult to come up with a globally interpretable reference model.
But speciﬁc to particular scenarios it may be possible.
For instance, it might be difﬁcult to providegeneral rules for how to drive a car, but in speciﬁc scenarios such as there being an obstacle in front,one can suggest that you stop or turn, which is a simple rule.
So our machinery could potentially beapplied at a local level where the reference model is interpretable in that locality.
This might help in“spot checking" a deployed model and estimating its safety by computing these maximum deviationsin scrupulously selected (challenging) scenarios.
Impossible inputs in certiﬁcation set Mathematically simple sets such as Cartesian products and(cid:96) p balls permit simpler algorithms for optimizing functions over them.
Accordingly, these sets havebeen the focus of not only the present work but also the related literature on ML veriﬁcation andadversarial robustness.
However, they may not serve to exclude inputs that are physically or logicallyimpossible from the certiﬁcation set C , and thus, the resulting maximum deviation values may be toolarge and conservative.
Here it is important to distinguish between impossible inputs and those thatare merely implausible (i.e., with low probability).
Techniques for capturing implausibility have beenproposed for contrastive/counterfactual explanations [8, 58], whereas we expect the set of impossibleinputs to be smaller and more constrained.
As a simple example from the Adult Income dataset, ifwe agree that a wife/husband is deﬁned to be of female/male gender (regardless of the gender of thespouse), then the cross combinations male-wife and female-husband cannot occur.
Future work canconsider the representation and handling of such constraints.
White-box vs. grey-box models In this paper, we have assumed full “white-box” access to bothf and f 0 , namely complete knowledge of their structure and parameters.
Interesting questions mayarise when this assumption is relaxed to different “grey-box” possibilities.
For example, one couldfurther investigate the third case in Section 4.4, where one of f, f 0 is black-box and the other iswhite-box interpretable.
There may exist assumptions that we have not identiﬁed that would improvethe query complexity compared to generic black-box optimization.
Other interpretability-safety relationships This paper has focused on one relationship betweenthe interpretability of a model and the safety of its outputs.
It has not addressed other ways inwhich interpretability/explainability can affect the risk of a model (in the plain English sense, not theexpectation of a loss function).
For example, in regulated industries such as consumer ﬁnance, notproviding explanations or providing inadequate ones can lead to legal, ﬁnancial, and reputational risks.
On the other hand, providing explanations is associated with its own risks [59].
These include theleakage of personal information or model information (intellectual property), an increase in appealsof decisions for the decision-making entity, and strategic manipulation of attributes (i.e. “gaming”)by individuals to gain more favorable outcomes.
Applicability to RL settings In RL, if one views the actions as labels and state representationas features, one can build a tree, albeit likely a deep/wide one, to represent exactly the RL policy,where the probability distribution over the actions can be viewed as the class distribution in a normalsupervised setting.
Rolling up the states, creating leaves with multiple states, and simply averagingthe probabilities for each action would yield smaller trees that approximate the policy.
Our worklays a foundation where in principle we can also compare f and f 0 that are policies using such treerepresentations.
This may be related to a popular global explainability method [60] that samplespolicies and builds trees to explain them.
Ethics The safety of machine learning systems has been called out by the European Commission’sregulatory framework [61].
The commission states seven key dimensions to be evaluated and auditedby a cross-disciplinary team: (i) human agency and oversight, (ii) technical robustness and safety,(iii) privacy and data governance, (iv) transparency, (v) diversity, non-discrimination and fairness,(vi) environmental and societal well-being, and (vii) accountability.
The second of these dimensionsis safety.
However, Sloane et al.
[62] argue that algorithmic audits are ill-deﬁned as the underlyingdeﬁnitions are vague.
The proposed work helps ﬁll that ill-deﬁnedness using a quantitative approach.
One may argue against this particular choice of quantiﬁcation, but it does start the community downthe path toward being more concrete in its deﬁnitions.
As with many other technologies, the proposed approach may be misused.
For example, the referencemodel may be chosen in a way that hides the safety concerns of the model being evaluated.
Transparentdocumentation and reporting with provenance guarantees can help avoid this kind of purposefuldeceit [63].
D Experiment Details and Additional Results
D.1 General Experiment Details
Data processing We use the training set of each dataset to train models and the test set as thebasis for evaluating maximum deviation, speciﬁcally as the set of centers for the (cid:96) ∞ balls in (2).
Continuous features are standardized and categorical features are one-hot encoded.
The (cid:96) ∞ norm iscomputed on the resulting normalized feature values.
Models We use scikit-learn [64] to train decision tree (DT), logistic regression (LR), and Random
Forest (RF) models.
The corresponding complexity parameters are the number of leaves for DT(parameter max_leaf_nodes ), the amount of (cid:96) 1 regularization for LR (inverse (cid:96) 1 penalty C ), and thenumber of estimators/trees for RF ( n_estimators ).
For additive models, we use Explainable Boosting Machines (EBM) from the InterpretML package [45] with zero interaction terms.
Smoothness iscontrolled by the max_bins parameter, the number of discretization bins for continuous features.
Deviation maximization In all cases, when the certiﬁcation set radius r = 0 , maximum deviationcan be computed simply by evaluating the models on the test set.
For the case where r > 0 , f is a DTor RF, and f 0 is a DT, Algorithm 1 is used (on a bipartite graph if f is a DT).
The cases where f is
LR or EBM fall under the generalized additive case of Section 4.2.
Given that f 0 is a DT, we use (7),(6) to determine the maximum deviation.
For r < ∞ when C is a union of (cid:96) ∞ balls, we maximizeseparately over each intersection between a ball and a leaf of f 0 and then take the maximum over theintersections.
Computation All experiments were run on CPU nodes with 64GB memory.
For decision treesand tree ensembles run times of Algorithm 1 were limited to 2 hours, after which the best availablebounds were used.
D.2 Home Mortgage Disclosure Act Dataset
Data source and pre-processing The data is made available by the US Consumer Finance Protection Bureau (CFPB) under the Home Mortgage Disclosure Act (HMDA).
We use the nationalsnapshot from year 2018 2 of “loan/application records,” which contain information on mortgageapplications and their outcomes.
According to their website, the CFPB has modiﬁed the data toprotect applicant and borrower privacy.
We processed the 2018 loan/application records as summarized below.
These steps were informed by
Gill et al.
[65] but not identical to theirs:
• Restrict to complete, submitted applications with ‘action_taken’ ≤ 3 (loan originated, application approved but not accepted by applicant, or application denied).
• Create a binary-valued target variable representing approval by binarizing ‘action_taken’
(originated or approved → 1 , denied → 0 ).
• Restrict to purchases of principal residences (the most consequential in terms of people’s
lives,
i
e., not reﬁnances or for investment) and single-family homes.• Restrict to loans that are not “special” in any way: conventional loans, ﬁrst mortgages, notmanufactured homes, no non-amortizing features, etc.
• Drop columns that are not applicable for site-built single-family homes.
• Drop columns that are not applicable or recorded until the approval/origination decision ismade.
For example, loan costs (points and fees) are not recorded until a loan is originated,the type of entity that purchases a loan does not apply unless the loan is originated, etc.
This
is in keeping with the mortgage approval scenario that we consider
• Drop all demographic columns to reﬂect laws that forbid lending decisions from explicitly
depending on applicant demographics.
[Figure]
• Drop geographical columns that have too many unique values (e.g., census tract).
• Drop rows that have null or “exempt” values in key features such as loan-to-value ratio,debt-to-income ratio, property value, income.
• Take a 10% sample of the records remaining after the above processing, to make experimentation less time- and resource-consuming.
• Split the subsampled dataset 80%–20% into training and test sets.
The dataset resulting from the above processing is imbalanced, with nearly 93% of mortgageapplications approved.
Thus in training models, we balance the classes by weighting, either using theclass_weight=‘balanced’ option in scikit-learn or deﬁning sample weights for the same purpose.
On the test set, we evaluate balanced accuracy instead of accuracy.
Reference model Figure 3 depicts the 8 -leaf DT reference model used in the experiments on the
HMDA dataset.
This DT has a test set balanced accuracy of 70 .
9% and area under the receiveroperating characteristic (AUC) of 0 .
The top-level split is based on debt-to-income ratio, ameasure often used in lending.
Other common mortgage measures such as loan-to-value ratio,property value, and whether a preapproval was requested (value 2 means no) also appear.
‘aus_1’=5and ‘aus_1’=6 refer to the automated underwriting system used to evaluate the application, withvalues 5 and 6 denoting “other” and “not applicable”.
LR and GAM models In Tables 2 and 3, we show the values of inverse (cid:96) 1 penalty C and max_binsthat were used for LR and GAM respectively, as well as statistics of the resulting classiﬁers.
Testset balanced accuracy and AUC increase and reach a plateau.
For LR, we take the (cid:96) 1 norm of thecoefﬁcients to be the main measure of smoothness as it depends on both the number of nonzerocoefﬁcients as well as their magnitudes, which both affect the extreme values attained in (6).
Sincethe LR balanced accuracy and AUC are not higher than those of the reference DT, we focus less on
LR in what follows.
[Table] [Table]
For EBM, based on Table 3, we select max_bins = 32 as a representative model with balancedaccuracy and AUC nearly equal to the maximum attainable values.
Plots for this EBM model areshown in Figure 4.
First note that whether a preapproval was requested (value 1 means yes) is quite predictive of ﬁnal approval.
The shape functions for the four continuous features debt-to-income ratio, loan-to-value ratio, property value, and income are mostly monotonic and agree with domain knowledge.
The log-odds of mortgage approval decrease as debt-to-income ratio and loan-to-value ratio increase, with abrupt drops around 50% for debt-to-income ratio and at 80% and 100% for loan-to ratio.
For property value and income, after a minimum value is reached, the shape
[Figure]
[Table]
[Figure]
[Table]
[Figure]
s i t y
[Figure]
c o r e
D e n s i t y
[Figure]
[Table]
Feature combinations that maximize deviation Table 4 shows feature values that yield the mostpositive difference between the predicted probabilities of the EBM f with max_bins = 32 and the8 -leaf DT f 0 .
This table corresponds to the second “Conﬂict between f , f 0 ” example in Section 5.
The 6 features that contribute most to the deviation are shown.
These contributions are determinedusing (8); since the maximum deviation occurs in one of the (cid:96) ∞ ball-leaf intersections and thisintersection is a Cartesian product, the feature-wise decomposition in (8) applies.
The contribution offeature j is then max x j ∈S j f j ( x j ) .
We take an average of the contributions over r to give a single ranking of features for all r .
The same method is used to determine feature contributions and choose the top 6 features for Table 1.
As mentioned in Section 5, all solutions in Table 4 have ‘preapproval’=1 and debt-to-income ratios> 52% that place them in leaf 2 in Figure 3.
The latter results in a low predicted probability ofapproval from f 0 while the former makes a large positive contribution to the probability from f (see
Figure 4).
The values of the other features also make increasingly larger positive contributions to fas r increases.
Loan-to-value ratio decreases, while co-applicant credit score type moves from type10 (no co-applicant, hence weaker application) to increasingly favorable score types (1, 7, 6, see
Figure 4)
; applicant credit score is similar.
This behavior is similar to the movement toward extremepoints seen in Table 1.
Dependence on model complexity Figure 5 shows the dependence of maximum deviation on thecomplexity of model f , quantiﬁed by the number of leaves for DTs, coefﬁcient (cid:96) 1 norm for LR,max_bins for EBM, and the number of estimators (trees) for RF.
The DT and RF cases demonstratethe methods in Section 4.3, speciﬁcally Algorithm 1, where in the RF case, the algorithm may onlyprovide bounds after a time limit of two hours.
The plots show that maximum deviation may ormay not increase with model complexity.
In Figure 5a, the deviation is small for a 10 -leaf DT andincreases rapidly.
Figures 5b and 5c indicate that maximum deviation is sensitive to the (cid:96) 1 norm of
LR models but not to the max_bins parameter of EBMs.
The latter may increase the resolution ofthe EBM shape functions but not their dynamic range.
Dependence on certiﬁcation set size Figure 6 shows the dependence of maximum deviation onthe certiﬁcation set radius r .
For LR and GAM, the maximum deviation is greater for r > 0 thanfor r = 0 , showing that evaluation on a ﬁnite test set may not be sufﬁcient and inﬁnite certiﬁcationsets (with r > 0 ) should be considered, especially to account for unexpected, out-of-distributiondeviations.
There are jumps at r = 1 because this is the radius that permits values of categoricalfeatures of test set points (the ball centers in (2)) to change to any other value.
In the case of GAM,
[Figure]
this jump is sufﬁcient for the deviation to equal that for
, the deviation remains constant
Running time Figure 7 shows the time required to compute the maximum deviation for LR and
GAM on the HMDA dataset.
These times were obtained using a single 2 .
GHz core of a server with64 GB of memory (only a small fraction of which was used) running Ubuntu 16.04 (64-bit).
Thetimes increase with the (cid:96) ∞ ball radius r because of the increasing number of ball-leaf intersectionsthat become non-empty and hence need to be evaluated.
The time for r = 0 is minimal because thiscase requires only model evaluation over the ﬁnite test set, as mentioned.
The jumps at r = 1 aredue again to the ability of categorical features to change values, leading to an increase in ball-leafintersections.
The ﬁlled-in regions show that there was little variation due to different (cid:96) 1 norms for
LR or max_bins for GAM.
This was most likely because of a vectorized implementation, whichoperates on all LR coefﬁcients or all GAM bins at once (i.e., without a for loop).
[Figure]
[Figure]
[Figure]
D.3 Adult Income dataset
We use the given partition of the Adult Income dataset into training and test sets.
Reference model Figure 8 depicts the 8 -leaf DT reference model used in the experiments onthe Adult Income dataset.
This DT has 85 .
0% accuracy on the test set.
The root node separatesindividuals based on whether the marital status is Married-civ-spouse .
The remaining splits dividethe population into those with high and low education, high and low capital gains, and high and lowcapital losses.
In particular, having high capital gains or losses is a good predictor of high income( > $50000 ).
LR and GAM models Tables 5 and 6 show the values of C and max_bins used for LR and GAMrespectively together with statistics of the resulting classiﬁers.
Based in part on Tables 5 and 6, weselect C = 0 .
01 and max_bins = 8 as representative models that remain simple and have accuraciesand AUCs not far from the maximum attainable.
Plots for these two models are shown in Figures 9and 10.
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
HoursPerWeek
[Table]
[Figure]
K 1 .
05 K - 1 .
22 K 1 .
22 K - 1 .
39 K 1 .
39 K - 1 .
57 K 1 .
57 K - 1 .
74 K 1 .
74 K - 1 .
92 K 1 .
92 K - 2 .
09 K 2 .
09 K - 2 .
27 K 2 .
27 K - 2 .
44 K 2 .
44 K - 2 .
61 K 2 .
61 K - 2 .
79 K 2 .
79 K - 2 .
96 K 2 .
96 K - 3 .
14 K 3 .
14 K - 3 .
31 K 3 .
31 K - 3 .
48 K 3 .
48 K - 3 .
66 K 3 .
66 K - 3 .
83 K 3 .
83 K - 4 .
01 K 4 .
01 K - 4 .
18 K 4 .
18 K - 4 .
36 K 0 10k20k30k
CapitalLoss
S c
o r e
D e n s i t y −
A m e r - I n d i a n - E
s k
i m o
A s i a n - P a c - I s l a n d e r B l a c k
O t h e r
W h
i t e10k20k
Race
S c
o r e
D e n s i t y
50 Number of leaf nodes 0.
m a x d e v i a t i o n
D
Decision Tree (a)    80 1 norm 0.
m a x d e v i a t i o n
D
LR, r = 0.
(b) 101010 3 max_bins 0.
m a x d e v i a t i o n D
GAM, r = 0.
(c)
Dependence on model complexity Figure 11 shows the dependence on model complexity for DT(number of leaves), LR (coefﬁcient (cid:96) 1 norm), and EBM ( max_bins ).
In Figure 11a, the maximumdeviation is 0 .
114 for trees with 9 and 10 leaves, and remains moderate up to 26 leaves, which isdifferent than in Figure 5a.
Similar to Figures 5b and 5c, (cid:96) 1 norm has a larger effect on maximumdeviation than max_bins (note the vertical scale in Figure 11c).
Dependence on certiﬁcation set size Figure 12a shows the dependence on the certiﬁcation setradius r for DT, LR, and GAM.
The patterns are similar to those in Figure 6: the deviations for LRand GAM increase from r = 0 and have jumps at r = 1 , while the deviation for DT remains constant.
One difference is that the LR curve in Figure 12a meets its r → ∞ asymptote (dashed line in ﬁgure),similar to GAM. [Figure]
Figure 12b shows the upper bound on the maximum deviation as a function of the certiﬁcation setsize for two RF models.
As the test set is large in this case, the deviations observed even for smallvalues of r are high and grow to reach the value of the full feature space quickly.
[Figure]
Relationships with accuracy and robust accuracy In Figure 13, we show maximum deviation asa function of test set accuracy for the DT, LR, and GAM models shown in Figure 11 (the LR and
GAM models are listed in Tables 5 and 6).
Broadly, the plots show two regimes: one where accuracyincreases and maximum deviation increases moderately or not at all, and one where accuracy stallswhile maximum deviation increases.
The latter is less desirable as it suggests increasing safety riskswithout a gain in accuracy.
The last branch of the DT curve actually decreases in accuracy, indicatingoverﬁtting, while maximum deviation is high.
We also consider the relationship of maximum deviation to robust accuracy .
Following Wong and
Kolter [17], robust loss for a pair ( x, y ) is deﬁned as the worst-case loss over an (cid:96) ∞ ball centered at
[Figure] [Figure]
the Adult Income dataset.
[Equation]
and robust accuracy is therefore 1 minus the average robust 0 - 1 loss over a dataset.
While Wong and
Kolter [17] focus on bounding robust loss for feedforward neural networks with ReLU activations,we ﬁnd that the results in Section 4.2 apply to computing robust loss (16) exactly for LR and GAMmodels.
Speciﬁcally, for 0 - 1 loss and (cid:96) ∞ balls, the separable optimization (8) applies, and the worstcase is obtained by minimizing f when the label y is positive and maximizing f when y is negative.
The resulting robust accuracy values for DT, LR and GAM are plotted in Figure 14 in a similarfashion as Figure 13.
Here we set (cid:15) = 0 .
1 and r = 0 .
1 as well in computing maximum deviation.
The DT plot shows maximum deviation increasing with model complexity while robust accuracyis stable up to a point.
In the subsequent regime, when there are a large number of leaves, modelrobustness reduces while deviation remains high.
The LR plot begins similarly to the one in Figure 13in that robust accuracy increases along with maximum deviation, but then it stalls and decreasesfor maximum deviation above 0 .
In the GAM plot, robust accuracy actually decreases with themax_bins parameter, i.e., the curve goes from right to left.
Breakdown by leaves of f
In Figures 15–18, we plot the maximum log-odds achieved by modelf ( max on RHS of (7)), the minimum log-odds achieved by f , and the reference model log-oddsg ( y 0 m ) over each leaf of the decision tree reference model in Figure 8.
Plots are on the log-oddsscale to show the deviations more clearly, including those that would be compressed by the nonlinearlogistic function g − 1 ( z )
= 1 / (1 + e − z ) .
Figures 15 and 17 show the dependence on the certiﬁcationset radius r while Figures 16 and 18 show dependence on the smoothness parameters for LR and
GAM.
These ﬁgures provide a more granular picture corresponding to the summary in Figure 11and support the trends seen there.
In Figures 15 and 17, there are jumps at r = 1 because this is thesmallest radius that permits the values of categorical features of test set points (the ball centers in (2))to change to any other value.
(Recall that categorical features are one-hot encoded into binary-valuedfeatures.)
In Figure 17, the GAM achieves the limiting deviations corresponding to r = ∞ ( C = X , dashed lines) no later than r = 1 . 2 .
In Figure 15, the LR model achieves the lower limit on log-oddsas soon as r > 1 but the upper limit is not achieved for most leaves.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Running time Figure 19 shows the time required to compute the maximum deviation for LR and
GAM on the Adult Income dataset.
The same observations apply as in Figure 7 earlier.
Primal bounds for RF The primal bound for max-deviation shown in Figure 2 for RF is updatedeach time the algorithm ﬁnds a K + 1 partite, i.e. has examined all trees in the Random Forest.
Themax-deviation computed for such a partite is a valid deviation.
To prove if its optimal, Algorithm1 needs to run to completion which may not be feasible.
Figure 2 shows that as the RF models get [Figure] larger (number of partites increase), it gets harder to ﬁnd primal solutions. [Figure]
Maximal cliques evaluated for DT, RF To investigate the effectiveness of pruning by bounds in
Algorithm 1, we investigate the number of times all the decision trees in the Random Forest haveto be processed.
This represents number of times the state could not be pruned and needed to beevaluated fully.
[Figure] Figure 20 shows two aspects at play.
(a) Pruning by bound is effective in restricting the search spacemore so for Random Forests than for decision trees, and (b) for larger graphs, more time is spent incomputing bounds in Eq.
[Figure]
li
[Figure]
u
[Figure]
s
[Figure]
v
[Figure]
l
[Figure]
Feature combinations that maximize deviation In Tables 7 and 8, we report feature values thatmaximize deviation over selected leaves of the DT reference model, for LR and GAM respectively.
For Table 7, we have chosen the minimum log-odds over leaf 6 (corresponding to Figure 15, leaf 6,blue curve), which is one of the two leaves m in (7) that maximize the deviation overall (the otherbeing leaf 8).
For Table 8, the minimum log-odds over leaf 12 is chosen (corresponding to Figure 17,leaf 12, blue curve) because this maximizes the deviation overall for most values of r .
The tablesshow the 6 features that contribute most to the minimum log-odds.
These contributions are againdetermined using (8) (with max replaced by min ); since the minimum log-odds occurs in one of the (cid:96) ∞ ball-leaf intersections and this intersection is a Cartesian product, the decomposition in (8)applies.
The contribution of feature j is then min x j ∈S j f j ( x j ) .
As in Tables 1 and 4, we take anaverage of the contributions over r to give a single ranking of features for all r .
[Table]
01 ) over leafof the decision tree reference model.
The 6 features that contribute most to the minimum are shown as a function of certiﬁcation set radius r .
As r increases, the predominant trend of the values of continuous features is toward extremes ofthe domain X , depending on the sign of the corresponding LR coefﬁcient w j or shape of the GAMfunction f j .
For example, EducationNum (education on an ordinal scale), hours per week, and agedecrease toward minimum values, while capital gain occupies the minimal interval permitted forleaf 12 (see Figure 8).
(These examples make sense since the log-odds of high income is beingminimized.)
This movement toward extremes is expected in the LR case because the functions w j x jare either increasing or decreasing, and it is also true for GAM if the function f
j is mainly increasingor decreasing.
The values sometimes change abruptly in the opposite direction, for example hoursper week in both Tables 7, 8, and age in the latter.
These abrupt changes are due to the minimumjumping from one ball in (2) to another as r increases, but the overall trend eventually prevails.
Forcategorical features, the trend is toward values that minimize f j ( x j ) , e.g., Never-married maritalstatus, Without-pay work class.
While the contribution of each of these features to minimizinglog-odds may be limited, together they do add up.
[Table]
White
[Table] [Table] [Table] [Table]
White
[Table] [Table]
State-gov [5095 5119]
White
[Table]
State-gov [5095 5119]
White
[Table]
A notable exception to the trend toward extremes is capital loss in Table 8.
This was discussed in the“Identiﬁcation of an artifact” example in Section 5.
Given the results in Tables 7 and
[Table]
one question that arises is whether the feature combinationsare indeed possible, if not the ones for r → ∞ , then at least for some ﬁnite value of r .
For the top features shown in the two tables, while some combinations may appear improbable (for example,
EducationNum = 1 and 1 hour per week), we submit that none appear impossible .
However, if oneconsiders features beyond the top 6, then some “impossible” combinations do occur (e.g., a femalehusband), although the contributions of these features to the minimum log-odds are much less.
Wetouch upon this issue in Appendix C.
The next question one might consider is the implication of these maximal deviations.
From Figure 8,it is seen that leaf 6 classiﬁes individuals with high capital gains as high income with high probability( 0 . 965 ).
Leaf 12 adds the attributes of married status and high education, and hence classiﬁes as highincome with even higher probability ( 0 . 996 ).
At the same time, the feature values in Tables 7 and 8,which minimize log-odds for LR and GAM, also make sense according to basic domain knowledge.
For example, few hours per week and young age are associated with lower income, as are Without-paywork class and Amer-Indian-Eskimo race in the United States.
When these conﬂicting associationsoccur in combination and the combination does not appear impossible, the question may be whichone prevails.
Such a question might be resolvable by a domain expert.
Alternatively, the disagreementbetween models f and f 0 on the extreme examples in Tables 7, 8 may be reason to be cautious aboutusing either of the models in these cases.
This might lead to a way of combining the models orabstaining from prediction altogether.
Lastly, the anomalously low region in the CapitalLoss functionidentiﬁed in Table 8 is a clear, concrete example where further investigation is warranted.
D.4 Lending Club dataset
This dataset consists of 2.26 million rows with 14 features on loans.
The target variable is whether aloan will be paid-off or defaulted on.
Features describe the terms of the loan, e.g. duration, grade,purpose, etc.
and borrower ﬁnancial information such as credit history and income.
For this casestudy, we consider a loan approval scenario using only information available at the time of application.
In particular, we exclude the feature ‘total_pymnt’ (total payment over time on the loan), whichbecomes known at essentially the same time as the target variable.
(When ‘total_pymnt’ is includedas a feature, the prediction task becomes easy and accuracies in the high 90% range are possible.)
[Figure]
Reference model Figure 21 depicts the 8 -leaf DT reference model for the Lending Club dataset.
Most of the splits partition the ‘sub_grade’ feature, which is a measure of the quality of the loan( 0 – 34 range, lower is better).
Node 3 differentiates between 60-month terms and 36-month terms(the only alternative), while nodes 4 and 7 split on ‘added_dti’ (added debt-to-income ratio), whichis the ratio between 12 months worth of the loan’s payment installments and the borrower’s annualincome.
While the structure of the DT agrees with domain knowledge (lower ‘sub_grade’ and lower‘added_dti’ correlate with higher repayment probability), the test set accuracy of 79 .
8% is no betterthan that of the trivial predictor that always returns the majority class of “paid off”.
The DT’s AUCof 0 . 689 however does indicate an improvement over the trivial predictor.
LR and GAM models Tables 9 and 10 show the statistics of the LR and GAM classiﬁers thatwere trained on the Lending Club data.
Similar to the DT reference model, the difference comparedto Tables 5, 6 for the Adult Income dataset is that the accuracies remain no better than that of thetrivial predictor, while the AUC does not show much increase either.
These statistics suggest thatthe prediction task is difﬁcult with the features available.
Figures 22 and 23 display plots for the
LR model with C = 0 .
01 and GAM with max_bins = 8 , which are again chosen as representativemodels.
The GAM in particular shows sensible monotonic behavior as functions of ‘sub_grade’,‘int_rate’ (interest rate), ‘dti’ (debt-to-income ratio), etc., despite the unimpressive accuracy.
[Table] [Table]
[Figure]
Lending
[Table]
[Figure]
[Table]
D e n s i t y
[Figure]
[Table]
[Figure]
[Table]
dti
S c
o r e
D e n s i t y
[Table]
[Figure]
[Table] [Table]
d i t _ c a r d d e b t _ c o n s o li d a t i o n e d u c a t i o n a l h
o m e _
i m p
r
o v e
m e n t
h
o u s e m a j o r _ p u r c
h a s e m e d i c a l m o v i n g o t
h e rr e n e w a b l e _ e n e r
g
y
s m a ll _ b u s i n e ss v a c a t i o n w e dd i n g50k100k150k purpose
S c
o r e
D e n s i t y
[Figure] [Figure]
[Figure]
Maximum deviation summary Figures 24–26 show maximum deviation as functions of certiﬁcation set radius r and model complexity parameters, in a similar manner as Figure 11 for the Adult
Income dataset.
The qualitative patterns are similar to before: increasing maximum deviation in allcases except with the number of RF estimators in Figure 26b, where the upper bound is stable around0 .
A major quantitative difference is that the maximum deviations for the GAM in Figure 25 aremuch lower than for the other models, in particular LR in Figure 24.
This is likely due to the factthat the GAM functions f j in Figure 23 are bounded while still being monotonic, unlike the linearfunctions w j x j in the LR model.
[Figure]
Breakdown by leaves of f
Figures 27–30 show a breakdown of the deviations for LR and GAMby leaves of the reference model, similar to Figures 15–18 and again on the log-odds scale.
Onedifference is that in Figure 27, the deviations for ﬁnite r do not come close to their r → ∞counterparts in most cases.
In Figure 29 however, the r → ∞ values are all attained when r is slightlygreater than 1 .
[Figure]
[Figure] [Figure]
l
[Figure]
l o g
[Figure]
o dd s
[Figure]
l o g - o dd
[Figure]
[Figure] [Figure]
l o g - o dd s
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
l o g - o dd s
[Figure] [Figure]
l
[Figure] [Figure]
[Figure] [Figure]
[Figure]
l o g - o dd
[Figure]
Running time, maximal cliques evaluated Figure 31 shows the time required to compute themaximum deviation for LR and GAM on the Lending Club dataset.
Figure 32 shows the number of
K + 1 -maximal cliques evaluated for DT and RF as well as the number of nodes in the graph.
Theobservations are the same as in Figures 19 and 20.
[Figure]
[Figure]
Feature combinations that maximize deviation Tables 11 and 12 present the feature values thatmaximize deviation for LR and GAM respectively.
For both tables, the minimum log-odds over leaf5 of the DT reference model is selected (corresponding to
because this choice maximizes the deviation overall for most values of r .
The previous trend toward extreme feature values that minimize log-odds also holds here as rincreases.
For example, debt-to-income ratios increase to outlier values above 100% , annual incomedrops to the minimum of $100 , the term changes to 60 months in Table 11 for r > 1 , and the purposechanges to small business, the category with the lowest log-odds in Figure 23.
The decrease in incomeand increases in debt-to-income ratios are qualitatively in accordance with each other.
However, thesequantities are related to each other by deterministic formulas (at least in theory) that also involve theinterest rate and installment amount.
It is not clear whether the values in Tables 11 and 12 violatethese relationships.
This may be an instance that could beneﬁt from constraints on possible featurecombinations, as brieﬂy mentioned in Appendix C.
[Table] [Table]
= 8 )