O N
THE
S
AFETY OF I NTERPRETABLE M ACHINE L EARNING : A M AXIMUM D EVIATION A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Interpretable and explainable machine learning has seen a recent surge of interest.
We posit that safety is a key reason behind the demand for explainability.
To explore this relationship, we propose a mathematical formulation for assessing the safety of supervised learning models based on their maximum deviation over a certiﬁcation set.
We then show that for interpretable models including decision trees, rule lists, generalized linear and additive models, the maximum deviation can be computed exactly and efﬁciently.
For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can still provide informative bounds.
For a broader class of piecewise Lipschitz functions, we repurpose results from the multi-armed bandit literature to show that interpretability produces tighter (regret) bounds on the maximum deviation compared with black box functions.
We perform experiments that quantify the dependence of the maximum deviation on model smoothness and certiﬁcation set size.
The experiments also illustrate how the solutions that maximize deviation can suggest safety risks.
I NTRODUCTION
Interpretable and explainable machine learning (ML) has seen a recent surge of interest because it is viewed as one of the key pillars in making models trustworthy, with implications on fairness, reliability, and safety (Varshney, 2019).
It is generally accepted that the ultimate measure of ML explainability is whether a human ﬁnds the explanations useful (Doshi-Velez & Kim, 2017; Dhurandhar et al., 2017).
However, less attention has been paid to deeper reasons behind the human desire for explainability.
In this paper, we posit an important reason is toward achieving safety and preventing unexpected harms (Varshney & Alemzadeh, 2017).
This reason is implicit in the dichotomy between directly interpretable models vs. post hoc explanations of black-box models.
Some argue that only directly interpretable models should be used in high-risk applications (Rudin, 2019).
The crux of this argument is that post hoc explanations leave a gap between the explanation and the model that is producing the predictions.
Thus, unusual data points may appear to be harmless based on the explanation, but truly cause havoc.
This argument however does not explicitly address the question: What does safety mean for such models, and how is it intertwined with interpretability?
Towards answering this question, we propose a mathematical deﬁnition for assessing the safety of supervised learning (i.e. predictive) models.
Viewing these models as functions mapping an input space to an output space, a key way in which these models can cause harm is through grossly unexpected outputs, corresponding to inputs that are poorly represented in training data.
Accordingly, we approach safety assessment for a model by determining its maximum deviation over a certiﬁcation set from the output of a reference model .
The idea of a certiﬁcation set is that it is a large subset of the input space and is intended to cover all conceivable inputs to the model.
The reference model could be a simple, well-understood model or an existing model that has been “tried and tested.”
These concepts are discussed further in Section 2.
In Section 4, we discuss the computation of the maximum deviation for different model classes and show how this is facilitated by interpretability.
For model classes regarded as interpretable, including trees, rule lists, generalized linear and additive models, the maximum deviation can be computed exactly and efﬁciently by exploiting the model structure.
For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can exploit their composition in terms of trees to provide anytime bounds on the maximum deviation.
The case of trees is also generalized in a different direction by considering a broader class of functions that are piecewise Lipschitz, which we argue cover many popular interpretable functions.
Here we show that the beneﬁt of interpretability is signiﬁcantly tighter regret bounds on the maximum deviation compared with black box functions, by appropriately repurposing results from the multi-armed bandit literature in this context.
On the other hand, it is less clear that post hoc explanations, which approximate a model locally (Ribeiro et al., 2016; Lundberg & Lee, 2017;
Dhurandhar et al., 2018) or globally (Buciluˇa et al., 2006;
Hinton et al., 2015), can help with evaluating the maximum deviation and hence safety.
We conduct experiments to illustrate the deviation maximization methods in Section 4 for decision trees, linear and additive models, and tree ensembles.
The results in Section 5 quantify how the maximum deviation increases as the size of the certiﬁcation set increases and as the smoothness of the models decreases.
For tree ensembles, we ﬁnd that the obtained upper bounds on the maximum deviation are informative, showing that the maximum deviation does not increase with the number of trees in the ensemble.
We also study the feature combinations that maximize deviation, which can shed light on the sources of extreme model outputs and guide further investigation.
Overall, our discussion suggests that a reason for preferring more interpretable models is that it is easier to assess them for unexpected and potentially unsafe outputs.
A SSESSING S AFETY T HROUGH M AXIMUM D EVIATION
We are given a supervised learning model f , which is a function mapping an input feature space X to an output space Y .
We wish to assess the safety of this model by ﬁnding its worst-case deviation from a given reference model f 0 : X (cid:55)→ Y .
To do this, we additionally require 1) a measure of deviation D :
Y × Y
(cid:55)→ R + , where R + is the set of non-negative reals, and 2) a certiﬁcation set
C ⊆ X over which the deviation is maximized.
Then the problem to be solved is
[Equation]
The deviation is worst-case because the maximization is over all x ∈ C ; further implications of this are discussed in Appendix C.
We view problem (1) as a means toward the goal of evaluating safety.
In particular, a large deviation value is not necessarily indicative of a safety risk, as two models may differ signiﬁcantly for valid reasons.
For example, one model may capture a useful pattern that the other does not.
What large deviation values do indicate, however, is a (possibly) sufﬁcient reason for further investigation.
Hence, the maximizing solutions in (1) (i.e., the arg max ) are of operational interest.
Below we further discuss some elements in this problem formulation.
Output space Y .
In the case of regression, Y is the set of reals R or an interval thereof.
In the case of binary classiﬁcation, while Y could be { 0 , 1 } or {− 1 , +1 } , these limit the possible deviations to binary values as well (“same” or “different”).
Thus to provide more informative results, we take Y to be the space of real-valued scores that are thresholded to produce a binary label.
For example, y could be a predicted probability in [0 , 1] or a log-odds ratio in R .
Similarly for multiclass classiﬁcation with M classes, Y ⊂ R M could be a M -dimensional space of real-valued scores.
Models that abstain can also be accommodated as noted in Appendix A.
Reference model f 0 .
The premise of the reference model is that it should be “safe” above all.
The simplest case mathematically is for f 0 to be a constant function representing a baseline value, for example zero.
More generally, f 0 may be a simple model that can be readily grasped by a human, may be validated against domain knowledge, or may be based on a small number of expert-selected features.
Such models are common in medical risk assessment, consumer ﬁnance, and predicting semiconductor yield.
By simple, we mean for example a linear model with 10 non-zero coefﬁcients or a decision tree with 10 leaves.
The reference model could also be an existing model that is not necessarily simple but has been extensively tested and deployed.
In this case, f could be a new version of the model, trained on more recent data or improved in some fashion, and we wish to evaluate its safety before deploying it in place of f 0 .
In this and more complex settings, f 0 may not be globally interpretable, but may be so in local regions.
The machinery developed in this work could be applied in these settings as well to assess the safety of f (more discussion in Appendix C).
Certiﬁcation set C .
The premise of the certiﬁcation set is that it contains all inputs that the model might conceivably be exposed to.
This may include inputs that are highly improbable but not physically or logically impossible (for example, a severely hypothermic human body temperature of 27°C).
Thus, while C might be based on the support set of a probability distribution or data sample, it does not depend on the likelihood of points within the support.
The set C may also be a strict superset of the training data domain.
For example, a model may have been trained on data for males, and we would now like to determine its worst-case behavior on an unseen population of females.
For tabular or lower-dimensional data, C might be the entire input space X .
For non-tabular or higher-dimensional data, the choice C
= X may be too unrepresentative because the manifold of realistic inputs is lower in dimension.
In this case, if we have a dataset { x i } ni =1 , one possibility is to use a union of (cid:96) p balls centered at x i ,
[Equation]
The set C is thus comprised of points somewhat close to the n observed examples x i , but the radius r does not have to be “small”.
In addition to determining the maximum deviation over the entire set C , maximum deviations over subsets of C (e.g., different age groups) may also be of interest.
For example, Appendix D.1 shows deviation values separately for leaves of a decision tree, which partition the input space.
R ELATED W ORK
Our work relates to a number of different technical directions.
Varshney & Alemzadeh (2017) and
Mohseni et al.
(2021) give qualitative accounts suggesting that directly interpretable models are an inherently safe design because humans can inspect them to ﬁnd spurious elements; in this paper, we attempt to make those qualitative suggestions more quantitative.
Furthermore, several other authors have highlighted safety as a goal for interpretability, but without much further development as done here (Otte, 2013; Doshi-Velez & Kim, 2017; Tomsett et al., 2018;
Gilpin et al., 2018; Rudin, 2019).
Moreover, there is no consensus on how to measure interpretability, which motivates the relationship explored in this paper between interpretability and the ease of evaluating safety.
In the area of ML veriﬁcation, robustness certiﬁcation methods aim to provide guarantees that the classiﬁcation remains constant within a radius (cid:15) of an input point, while output reachability is concerned with characterizing the set of outputs corresponding to a region of inputs (Wong & Kolter, 2018; Raghunathan et al., 2018; Huang et al., 2020).
Our problem of deviation maximization (1) is more closely related to output reachability.
The differences in our work are: 1) we consider two models, a model f to be assessed and a reference f 0 , and are interested in their difference as measured by deviation function D ; 2) our focus is global , over a comprehensive set C , rather than local to small neighborhoods around input points; 3) we study the role of interpretability in safety veriﬁcation.
Moreover, works in robust optimization applied to machine learning minimize the worst-case probability of error, but this worst case is over parameters of f rather than over individual values of x (Lanckriet et al., 2002).
Thomas et al.
(2019) present a framework where during the model training, a set of safety tests is speciﬁed in order to accept or reject the possible solution.
The speciﬁcation of these tests is left to the model designer but the goal of the proposed solution is to provide a reusable paradigm to support safety in ML solutions.
We build on related literature from model robustness and explainability areas that deals speciﬁcally with tree ensembles.
Kantchelian et al.
(2016) seek to ﬁnd the smallest perturbation of an input instance to ‘evade’ a classiﬁer using mixed-integer programming (MIP).
Optimization formulations are also explored by Parmentier & Vidal (2021) for the purposes of counterfactual explanations.
MIP approaches are computationally intensive however.
To address this Chen et al.
(2019) introduce graph based approaches for veriﬁcation on trees.
Their central idea, which we use, is to discretize veriﬁcation computations onto a graph constructed from the way leaves intersect.
The veriﬁcation problem is transformed to ﬁnding all maximum cliques.
Devos et al.
(2021) expand on this idea by providing anytime bounds by probing unexplored nodes.
Safety has become a critical issue in reinforcement learning (RL) with multiple works focusing on making RL policies safe (Amodei et al., 2016; Zhu et al., 2019; Inala et al., 2020; Rupprecht et al.,
There are two broad themes (Garc´ıa et al., 2015): (i) a safe and veriﬁable policy is learned at the outset by enforcing certain constraints, and (ii) post hoc methods are used to identify bad regimes or failure points of an existing policy.
Our current proposal is complementary to these works as we focus on the supervised learning setup viewed from the lens of interpretability.
Nonetheless, ramiﬁcations of our work in the RL context are brieﬂy discussed in Appendix C.
D EVIATION M AXIMIZATION FOR S PECIFIC M ODEL C LASSES
In this section, we discuss approaches to computing the maximum deviation (1) for f belonging to various model classes.
The beneﬁt of interpretable model structure is seen in different guises.
For decision trees, generalized linear and additive models in Sections 4.1 and 4.2, exact and efﬁcient computation is possible.
For tree ensembles in Section 4.3, their composition in terms of trees allows discrete optimization methods to provide anytime bounds.
For a general class of piecewise Lipschitz functions in Section 4.4, the application of multi-arm bandit results yields tighter regret bounds on the maximum deviation.
The results in Sections 4.1–4.4 also show that intuitive measures of model complexity, such as the number of leaves or pieces or smoothness of functions, have more precise interpretations as well in terms of the complexity of maximizing deviation.
On the other hand, for post hoc explanations in Appendix B.5, we believe that more development will be needed for them to help in evaluating maximum deviation.
To develop mathematical results and efﬁcient algorithms, we will sometimes assume that the reference model f 0 is from the same class as f .
We will also sometimes assume that the certiﬁcation set C and other sets are Cartesian products.
This means that C = (cid:81) dj
=1 C j , where for a continuous feature j , C j =
[ X j , X j ] is an interval, and for a categorical feature j , C j is a set of categories.
4. T REES
We begin with the case where f and f 0 are both decision trees.
A decision tree with L leaves partitions the input space X into L corresponding parts, which we also refer to as “leaves”.
We consider only non-oblique trees.
In this case, each leaf is described by a conjunction of conditions on individual features and is therefore a Cartesian product as deﬁned above.
With L l ⊂ X denoting the l th leaf and y l ∈ Y the output value assigned to it, tree f is described by the function
[Equation]
and similarly for tree f 0 with leaves L 0 m and outputs y 0 m , m = 1 , . . .
, L 0 .
Rule lists can be seen as one-sided trees (Yang et al., 2017;
Angelino et al., 2018)
and are subsumed in this discussion.
The partitioning of X by decision trees and their piecewise-constant nature simplify the computation of the maximum deviation (1).
Speciﬁcally, the maximization can be restricted to pairs of leaves ( l, m ) for which the intersection L l ∩L
m ∩C is non-empty.
The intersection of two leaves L l ∩L
m is another Cartesian product, and we assume that it is tractable to determine whether C intersects a given Cartesian product (see examples in Appendix B.1).
For visual representation and later use in Section 4.3, it is useful to deﬁne a bipartite graph, with L nodes representing the leaves L l of f on one side and L 0 nodes representing the leaves L 0 m of fon the other.
Deﬁne the edge set E = { ( l, m ) :
L l ∩ L 0 m ∩ C (cid:54) = ∅} ; clearly |E| ≤ L 0 L .
Then
[Equation]
We summarize the complexity of deviation maximization for decision trees as follows.
Proposition 1.
Let f and f 0 be decision trees as in (3) with L and L 0 leaves respectively, and E be the bipartite edge set of leaf intersections deﬁned above.
Then the maximum deviation (1) can be computed with |E| evaluations as shown in (4) .
4. L INEAR AND ADDITIVE MODELS
In this subsection, we assume that f is a generalized additive model (GAM) given by
[Equation]
where each f j is an arbitrary function of feature x j .
In the case where f j ( x j ) =
w j x j for all continuous features x j , where w j is a real coefﬁcient, (5) is a generalized linear model (GLM).
We discuss the treatment of categorical features in Appendix B.2.
The invertible link function g : R (cid:55)→ R is furthermore assumed to be monotonically increasing.
This assumption is satisﬁed by common
GAM link functions: identity, logit ( g ( y ) = log( y/ (1 − y )) ), and logarithmic.
Equation (5) implies that Y ⊂ R and the deviation D ( y, y 0 ) is a function of two scalars y and y 0 .
For this scalar case, we make the following intuitively reasonable assumption throughout the subsection.
Assumption 1.
For y, y 0 ∈ Y ⊆ R , 1) D ( y, y 0 )
=
whenever y = y 0 ; 2) D
( y, y 0 ) is monotonically non-decreasing in y for y ≥ y
and non-increasing in y for y ≤ y 0 .
Our approach is to exploit the additive form of (5) by reducing problem (1) to the optimization
[Equation]
for different choices of S ⊂ X
and where minimization is obtained by negating all f j .
We discuss below how this can be done for two types of reference model f 0 : decision tree (which includes the constant case L 0 = 1 ) and additive.
For the ﬁrst case, we prove the following result in Appendix B.2:
Proposition 2.
Let f be a GAM as in (5) and S be a subset of X where f 0 ( x ) ≡ y 0 is constant.
Then if Assumption 1 holds,
[Equation]

   .
Tree-structured f 0 .
Since f 0 is piecewise constant over its leaves L 0 m , m = 1 , . . .
, L 0 , we take S to be the intersection of C with each L 0 m in turn and apply Proposition 2.
The overall maximum is then obtained as the maximum over the leaves,
[Equation]
This reduces (1) to solving 2 L 0 instances of (6).
Additive f 0 .
For this case, we make the additional assumption that the link function g in (5) is the identity function, as well as Assumption 2 below.
The implication of these assumptions is discussed in Appendix B.2.
Assumption 2.
D ( y, y 0 )
=
D ( y − y 0 ) is a function only of the difference y − y 0 .
Then f 0 ( x ) = (cid:80) dj
=1 f 0 j ( x j ) and the difference f ( x ) − f 0 ( x ) is also additive.
Using Assumptions 2, 1 and a similar argument as in the proof of Proposition 2, the maximum deviation is again obtained by maximizing and minimizing an additive function, resulting in two instances of (6) with S
= C :
[Equation]
Computational complexity of (6) .
For the case of nonlinear additive f , we additionally assume that C is a Cartesian product.
It follows that S =
(cid:81)
dj
=1 S j is a Cartesian product (see Appendix B. for the brief justiﬁcation) and (6) separates into one-dimensional optimizations over S j ,
[Equation]
The computational complexity of (8) is thus (cid:80) dj
=1 C j , where C j is the complexity of the j th onedimensional optimization.
We discuss different cases of C j in Appendix B.2; the important point is that the overall complexity is linear in d .
In the GLM case where (cid:80) dj =1 f j ( x j ) =
w T x , problem (6) is simpler and it is less important that C be a Cartesian product.
In particular, if C is a convex set, so too is S (again see Appendix B.2 for justiﬁcation).
Hence (6) is a convex optimization problem.
4. T REE ENSEMBLES
We now extend the idea used for single decision trees in Section 4.1 to tree ensembles.
This class covers several popular methods such as Random Forests and Gradient Boosted Trees.
It can also cover rule ensembles (Friedman & Popescu, 2008; Dembczy´nski et al., 2010) as a special case, as explained in Appendix B.3.
We assume f is a tree ensemble consisting of K trees and f 0 is a single decision tree.
Let L l k denote the l th leaf of the k th tree in f for l = 1 , . . .
, L k , and L 0 m be the m th leaf f 0 , for m = 1 , . . .
, L 0 .
Correspondingly let y l k and y 0 m denote the prediction values associated with each leaf.
Deﬁne a graph G ( V , E ) , where there is a vertex for each leaf in f and f 0 , i.e.
[Equation]
Construct an edge for each overlapping pair of leaves in V , i.e.
[Equation]
This graph is a K + 1 -partite graph as leaves within an individual tree do not intersect and are an independent set.
Denote M to be the adjacency matrix of G .
Following Chen et al.
(2019), a maximum clique S of size K
+ 1 on such a graph provides a discrete region in the feature space with a computable deviation.
A clique is a subset of nodes all connected to each other; a maximum clique is one that cannot be expanded further by adding a node.
The model predictions y c and y
c can be ensembled from leaves in S .
Denote D ( S ) to be the deviation computed from the clique
S .
Maximizing over all such cliques solves (1).
However, complete enumeration is expensive, so informative bounds, either using the merge procedure in Chen et al.
(2019) or the heuristic function in Devos et al.
(2021) can be used.
We use the latter which exploits the K + 1 -partite structure of G .
Speciﬁcally, we adapt the anytime bounds of Devos et al.
(2021) as follows.
At each step of the enumeration procedure, an intermediate clique S contains selected leaves from trees in [1 , . . .
, k ] and unexplored trees in [ k + 1 , . .
, K + 1] .
For each unexplored tree, we select a valid candidate leaf that maximizes deviation, i.e.
[Equation]
Using these worst-case leaves, a heuristic function
[Equation]
(12) provides an upper (dual) bound.
In practice, this dual bound is tight and therefore very useful during the search procedure to prune the search space.
Each K + 1 clique provides a primal bound, so the search can be terminated early before examining all trees if the dual bound is less than the primal bound.
We adapt the search procedure from Mirghorbani & Krokhmal (2013) to include the pruning arguments.
Appendix B.3 presents the full algorithm.
Starting with an empty clique, the procedure adds a single node from each tree to create an intermediate clique.
If the size of the clique is K +the primal bound is updated.
Otherwise, the dual bound is computed.
A node compatibility vector is used to keep track of all feasible additions.
When the search is terminated at any step, the maximum deviation is bounded by ( D lb , D ub ) .
The algorithm works for the entire feature space.
When the certiﬁcation set C is a union of balls as in Eq.
(2), some additional considerations are needed.
First, we can disregard leaves that do not intersect with C during the graph construction phase.
A validation step to ensure that the leaves of a clique all intersect with the same ball in C is also needed.
4. P IECEWISE L IPSCHITZ F UNCTIONS
We saw the beneﬁts of having speciﬁc (deterministic) interpretable functions as well as their extensions in the context of safety.
Now consider a richer class of functions that may also be randomized with ﬁnite variance.
In this case let f and f 0 denote the mean values of the learned and reference functions respectively.
We consider the case where each function is either interpretable or black box, where the latter implies that query access is the only realistic way of probing the model.
This leads to three cases where either both functions are black box or interpretable, or one is black box.
What we care about in all these cases 1 is to ﬁnd the maximum (and minimum) of a function ∆( x ) =
f ( x ) − f 0 ( x ) .
Let us consider ﬁnding only the maximum of ∆ as the other case is symmetric.
Given that f and f 0 can be random functions ∆ is also a random function and if ∆ is black box a standard way to optimize it is either using Bayesian Optimization (BO) (Auer, 2002) or tree search type bandit methods (Bubeck et al., 2011; Carpentier & Valko, 2015).We repurpose some of the results from this latter literature in our context showcasing the beneﬁt of interpretability from a safety standpoint.
To do this we ﬁrst deﬁne relevant terms.
Deﬁnition 1 (Simple Regret (Bubeck et al., 2011)) .
If f ∗C denotes the optimal value of the function f on the certiﬁcation set C , then the simple regret r C q after querying the f function q times and obtaining a solution x q is given by, r C q ( f ) =
f ∗C − f ( x q ) .
Deﬁnition 2 (Order β c-Lipschitz) .
Given a (normalized) metric (cid:96) a function f is c-Lipschitz continuous of order β >
if for any two inputs x , y and for c > 0 we have, | f ( x ) − f ( y ) | ≤ c · (cid:96) ( x, y ) β .
Deﬁnition 3 (Near optimality dimension (Bubeck et al., 2011)) .
If N ( C , (cid:96), (cid:15) ) is the maximum number of (cid:15) radius balls one can ﬁt in C given the metric
(cid:96) and C (cid:15) =
{ x ∈ C|
f ( x ) ≥ f ∗C − (cid:15) } , then for c > 0 the c-near optimality dimension is given by, υ = max (cid:16) lim sup (cid:15) → 0 ln N ( C c(cid:15) ,(cid:96),(cid:15) )
ln( (cid:15) − 1 ) , 0 (cid:17) .
Intuitively, simple regret measures the deviation between our current best and the optimal solution.
The Lipschitz condition bounds the rate of change of the function.
Near optimality dimension measures the set size for which the function has close to optimal values.
The lower the value of υ , the easier it is to ﬁnd the optimum.
We now deﬁne what it means to have an interpretable function.
Assumption 3 (Characterizing an Interpretable Function) .
If a function f is interpretable, then we can (easily) ﬁnd 1 ≤ m (cid:28) n partitions {C (1) , ..., C ( m ) } of the certiﬁcation set C such that the function f ( i ) =
{ f ( x ) | x ∈ C ( i ) } ∀ i ∈ { 1 , ..., m } in each partition is c-Lipschitz of order β .
Note that the (interpretable) function overall does not have to be c-Lipschitz of bounded order, rather only in the partitions .
This assumption is motivated by observing different interpretable functions.
For example, in the case of decision trees the m partitions could be its leaves, where typically the function is a constant in each leaf ( c = 0 ).
For rule lists as well a ﬁxed prediction is usually made by each rule.
For a linear function one could consider the entire input space (i.e. m = 1 ), where for bounded slope α the function would also satisfy our assumption ( c = α and β = 1 ).
Examples of models that are not piecewise constant or globally Lipschitz are oblique decision trees (Murthy et al., 1994), regression trees with linear functions in the leaves, and functional trees.
Moreover, m is likely to be small so that the overall model is interpretable (viz.
shallow trees or small rules).
With the above deﬁnitions and Assumption 3 we now provide the simple regret for the function ∆ .
Both black box models: If both f and f 0 are black box then it seems no gains could be made in estimating the maximum of ∆ over standard results in bandit literature.
Hence, using Hierarchical Optimistic Optimization (HOO) with assumptions such as C being compact and ∆ being weakly Lipschitz (Bubeck
et al
2011) with near optimality dimension
υ the simple regret after q queries is:
[Equation]
Both interpretable models: If both f and f 0 are interpretable, then for each function based on Assumption 3 we can ﬁnd m 1 and m 0 partitions of C respectively where the functions are cand c 0 -Lipschitz of order β 1 and β 0 respectively.
Now if we take non-empty intersections of these partitions where we could have a maximum of m 1 m 0 partitions, the function ∆ in these partitions would be c = 2 max( c 0 , c 1 ) -Lipschitz of order β = min( β 0 , β 1 ) as stated next (proof in appendix).
Proposition 3.
If functions h 0 and h 1 are c 0 and c 1 Lipschitz of order β 0 and β 1 respectively, then the function h
= h
− h 1 is c-Lipschtiz of order β , where c = 2 max( c 0 , c 1 ) and β = min( β 0 , β 1 ) .
Given that ∆ is smooth in these partitions with underestimated smoothness of order β , the simple regret after q i queries in the i th partition C ( i ) with near optimality dimension υ i based on HOO is: r C ( i ) q i (
[Equation]
) ≤ O
[Equation]
, where υ i ≤ dβ .
If we divide the overall query budget q across the π ≤ m 0 m 1 non-empty partitions equally, then the bound will be scaled by π 1 υi +2 when expressed as a function of q .
Moreover, the regret for the entire C can then be bounded by the maximum regret across these partitions leading to the following result:
[Equation]
Notice that for a model to be interpretable m 0 and m 1 are likely to be small (i.e. shallow trees or small rule lists or linear model where m = 1 ) leading to a “smallish” π and υ can be much >> dβ in case 1.
Hence, interpretability signiﬁcantly reduces the regret in estimating the maximum deviation.
Black box and interpretable model: Making no further assumptions on the black box model and assuming ∆ satisﬁes properties mentioned in case 1, the simple regret has the same behavior as equation 13.
This is expected as the black box model could be highly non-smooth.
E XPERIMENTS
We present a case study on the UCI Adult Income dataset (Dua & Graff, 2017), using its given partition into training and test sets.
Additional datasets are presented in Appendix D. For the reference model f 0 , an 8 -leaf decision tree (DT) is ﬁt (using scikit-learn’s max leaf nodes parameter) on the training data.
This DT has 85 .
0% accuracy on the test set and is easy for a human to understand and validate (see Figure 3).
We take the deviation function D to be the absolute difference between predicted probabilities of the positive (high-income) class.
For the certiﬁcation set C , we consider a union of (cid:96) ∞
balls (2) centered at test set instances ( n = 16281 ).
The (cid:96) ∞ norm is computed on normalized feature values, where continuous features are standardized and categorical features are one-hot encoded.
The case r = 0 yields a ﬁnite set consisting only of the test set, while r → ∞ corresponds to C being the entire domain X .
5. T REES , LINEAR AND ADDITIVE MODELS
We ﬁrst consider models for which the maximum deviation can be computed exactly: decision trees, generalized linear and additive models.
For the ﬁrst two classes, we use scikit-learn (Pedregosa et al., 2011) to train DTs of varying numbers of leaves ( max leaf nodes ) and logistic regression (LR) models with varying amounts of (cid:96) 1 regularization.
For additive models, we use Explainable
Boosting Machines (EBM) from the InterpretML package (Nori et al., 2019) with zero interaction terms.
Smoothness is controlled by the max bins parameter, the number of discretization bins for continuous features.
Statistics and plots of the resulting models can be found in Appendix D.1.
[Figure]
m a x d e v i a t i o n D
GAM, r = 0.
To evaluate the maximum deviation for DTs, Algorithm 1 is used on a bipartite graph.
For the latter two cases where f is generalized additive, f 0 is a DT, and radius r > 0 , we use (7), (6) ( r = 0 is handled simply by evaluating the models on the test set).
For r < ∞ when C is a union of balls, we maximize separately over each intersection between a ball and a leaf of f 0 , and take their maximum.
Figure 1 shows a summary of the maximum deviation D as a function of certiﬁcation set radius r and each model’s smoothness parameter; Appendix D.1 has breakdowns by leaves of f 0 .
The ﬁrst observation is that the deviation can be large even between simple models, namely the 8 -leaf reference DT f 0 , a 50 -leaf DT f , a sparse LR model (16 nonzeros, (cid:96) 1 norm = 7 . 4 ), and a smooth
GAM (see Figure 5 for plots of f j ).
At the same time, the interpretability of these models allows the deviation to be computed for a range of inﬁnite certiﬁcation sets with r > 0 .
We can thus quantify the increase in deviation as r increases, as seen in the left of Figure 1.
For LR and GAM, r only has to be slightly larger than 1 for the deviation to equal that for r = ∞ ( C = X , dashed lines in ﬁgure), whereas for DT, the deviation at r = 0 is already maximal.
The right panels show that the deviation also increases as model smoothness decreases (number of leaves for DT, (cid:96) 1 norm for LR, max bins for GAM).
The initial increase is rapid for DT and LR.
Appendix D.1 reports on the complexity of computing D (running time, number of cliques evaluated).
Studying the solutions that maximize deviation (the arg max ) can bring attention to unanticipated extremes in model output, which may rise to the level of safety risks.
Appendix D.1 discusses in detail some maximizing solutions corresponding to Figure 1.
A common source of large deviations is the extrapolation of linear and monotonic functions (more generally, unbounded or poorly bounded functions) to extreme points of the certiﬁcation set, or even parts thereof.
At these extreme points, the model f and reference f 0 are in conﬂict, which may be reason for caution or call for domain expert intervention.
In the GAM case, the arg max also highlights the presence of an unintuitive extreme value in one of the functions f j , which may be an artifact warranting further investigation.
5. T REE E NSEMBLES
[Figure]
Next, we demonstrate the methods in Section 4.3 on a
Random Forest model, where the number of estimators is varied.
A time limit of two hours is imposed for the search procedure for each run.
There are two sets of experiments, one where the set C is assumed to be the entire domain, and another where C is a union of balls.
In Figure 2a we observe that max deviations can be large even for simpler models with fewer estimators.
The upper bound on max deviation reduces as the number of estimators increases.
The larger number of estimators increases averaging and may serve to make the model smoother.
The primal lower bound here is the maximum D for complete K + 1 -max-cliques.
Due to the size of underlying graphs, fewer K + 1 -max-cliques are evaluated and the lower bound is weak.
For safety evaluation, i.e. worst case behavior, the lower bound is of less consequence.
For the same setting, we now consider the case where C is a union of balls of the test set and the radius r is varied.
Figure 2b shows the upper bound on safety as a function of the certiﬁcation set size for two RF models.
As the test set is large in this case, the deviations observed even for small values of r are high and grow to reach the value of the full feature space quickly.
C ONCLUSION
We have considered the relationship between interpretability and safety in supervised learning through two main contributions: First, the proposal of maximum deviation as a means toward assessing safety, and second, discussion of approaches to computing maximum deviation and how these are simpliﬁed by interpretable model structure.
We believe that there is much more to explore in this relationship.
Appendix C provides further discussion of several topics.
E THICS D ISCUSSION
The ICLR Code of Ethics requires contributions to avoid harm and contribute to society.
The safety of machine learning systems is a fundamental factor in achieving these goals, and has been called out by the European Commission’s regulatory framework (High Level Expert Group on Artiﬁcial
Intelligence, 2020).
The commission states seven key dimensions to be evaluated and audited by a cross-disciplinary team: (i) human agency and oversight, (ii) technical robustness and safety, (iii) privacy and data governance, (iv) transparency, (v) diversity, non-discrimination and fairness, (vi) environmental and societal well-being, and (vii) accountability.
The second of these dimensions is safety.
However, Sloane et al.
(2021) argue that algorithmic audits are ill-deﬁned as the underlying deﬁnitions are vague.
The proposed work helps ﬁll that ill-deﬁnedness using a quantitative approach.
One may argue against this particular choice of quantiﬁcation, but it does start the community down the path toward being more concrete in its deﬁnitions.
As with many other technologies, the proposed approach may be misused.
For example, the reference model may be chosen in a way that hides the safety concerns of the model being evaluated.
Transparent documentation and reporting with provenance guarantees can help avoid this kind of purposeful deceit (Arnold et al., 2019).
R EPRODUCIBILITY D ISCUSSION
To promote reproducibility, all algorithms and mathematical derivations in the paper are presented with a large amount of detail in the main body and appendix.
All datasets used in the experiments are publicly-available.
All parameters and their settings are fully described.
A A DDITIONAL P ROBLEM F ORMULATION D ETAILS
Models that abstain The formulation in Section 2 can also accommodate models that abstain from predicting (and possibly defer to a human expert or other fallback system).
If f ( x ) = ∅ , representing abstention, then we may set D ( ∅ , y 0 ) =
d for any y 0 ∈ Y , where d > 0 is an intermediate value less than the maximum value that D can take (Bartlett & Wegkamp, 2008).
The value d might also be less than a “typically bad” value for D , to reward the model for abstaining when it is uncertain.
B A DDITIONAL D ETAILS ON D EVIATION M AXIMIZATION FOR S PECIFIC M ODEL C LASSES B. T REES
Rule lists A rule list is a nested sequence of IF-THEN-ELSE statements, where the IF condition usually involves a single feature and the THEN consequent is an output value.
As discussed in Yang et al.
Angelino et al.
(2018), such rule lists are one-sided trees and are thus subsumed in the discussion of Section 4.1.
The number of leaves in the equivalent tree is equal to the number of rules in the list (including the last default rule).
Intersection of C with a Cartesian product If C = (cid:81) dj
=1 C j is also a Cartesian product, then determining whether the intersection is non-empty amounts to checking whether all of the coordinatewise intersections with C j , j = 1 , . . .
, d , are non-empty.
If C is not a Cartesian product but is a union of (cid:96) ∞ balls (which are Cartesian products), then the intersection is non-empty if the intersection with any one ball is non-empty.
Additive reference model For the case where f is a decision tree and f 0 is a generalized additive model, if the deviation function is symmetric, D ( y, y 0 )
=
D ( y 0 , y ) , then this case is covered in
Section 4.2.
B. L INEAR AND A DDITIVE M ODELS
Categorical features A function f j ( x j ) of a categorical feature x j can be represented in two ways, depending on whether f is a GAM or a GLM.
In the GAM case, we may use the native representation in which x j takes values in a ﬁnite set X j of categories.
In the GLM case, x j is onehot encoded into multiple binary-valued features x jk , one for each category k .
Then any function f j can be represented as a linear function,
[Equation]
where w jk is the value of f j for category k .
Implication of Assumption
The second condition implies that the deviation increases or stays the same as y moves away from y 0 in either direction.
Proof of Proposition 2.
Let x ∈ S and S ( x ) = (cid:80)
dj
=1 f j ( x j ) .
Under Assumption 1.1, if S ( x ) =
g ( y 0 ) , then
[Equation]
As S ( x ) increases from g ( y 0 ) , f ( x ) also increases because g − 1 is an increasing function, and
D ( f ( x ) , y 0 ) increases or stays the same due to Assumption 1.2.
Similarly, as S ( x ) decreases from g ( y 0 ) , f ( x ) decreases, and D ( f ( x ) , y 0 ) again increases or stays the same.
It follows that to maximize D ( f ( x ) , y 0 ) , it sufﬁces to separately maximize and minimize S ( x ) , compute the resulting values of D ( f ( x ) , y 0 ) , and take the larger of the two.
This yields the result.
Implication of Assumption 2 and identity link function g
These two assumptions imply that the deviation is measured on the difference between f and f 0 in the space in which they are additive.
For example, if f and f 0 are logistic regression models predicting the probability of belonging to one of the classes, the difference is taken in the log-odds (logit) domain.
It is left to future work to determine other assumptions under which problem (1) is tractable when f and f 0 are both additive.
Cartesian product C implies Cartesian product S
In the cases of constant and additive f 0 ,
S = C .
In the decision tree case, since each leaf is a Cartesian product L 0 m =
(cid:81)
dj
=1 R mj , the intersections S =
L 0 m ∩ C are also Cartesian products (cid:81)
dj
=1 S j where S j = R mj ∩ C j .
One-dimensional optimization complexities C j
For discrete-valued x j , C j is proportional to the number of allowed values |S j | .
For continuous x j , it is common to use spline functions or tree ensembles as f j in constructing GAMs.
In the former case, C j is proportional to the number of knots.
In the latter, the tree ensemble can be converted to a piecewise constant function and C j is then proportional to the number of pieces.
Lastly in the case where f j ( x j ) =
w j x j is linear and
S j =
[ X j , X j ] is an interval, C j
= O (1) because it sufﬁces to evaluate the two endpoints.
Convex S
If C is a convex set, then in the cases of constant and additive f 0 , S
= C is also convex.
In the case of tree-structured f 0 , S =
L 0 m ∩ C and each leaf L 0 m can be represented as a convex set, with interval constraints on continuous features and set membership constraints on categorical features.
The latter can be represented as x jk = 0 constraints on the one-hot encoding (see “Categorical features” paragraph above) for non-allowed categories k .
Hence S is also convex.
As a speciﬁc example, suppose that S is the product of independent constraints on each categorical feature and an (cid:96) p norm constraint on the continuous features jointly.
The maximization over each categorical feature has complexity C j = |S j | as noted above, while the maximization of w T x over continuous features lying in an (cid:96) p ball has closed-form solutions for the common cases p = 1 , 2 , ∞ .
B. T REE E NSEMBLES
The full algorithm for clique search from Section 4.3 is presented in Algorithm 1.
It uses Z as a node compatibility vector to keep track of valid leaves and B a set of trees/partites not yet covered by the maximum clique.
The algorithm starts with and empty clique S and anytime bounds as 0 .
It starts the search with the smallest tree to limit the search space.
This is typically f 0 .
Each leaf is added to the intermediate clique S in turn (Line 6).
A stronger primal bound can be achieved if the traversal is ordered in a meaningful way.
In particular, starting with nodes with the highest heuristic function value H ( S ) aids the algorithm to focus on better areas of the search space.
If the size of the clique is K +1 the primal bound is updated.
Otherwise, the dual bound is computed.
If the node is promising, the algorithm recurses to the next level.
When the search is terminated at any step, the maximum deviation is bounded by ( D lb , D ub ) .
Rule ensembles Similar to the tree ensembles considered in Section 4.3, a rule ensemble is a linear combination of conjunctive rules, where the antecedent is a conjunction of conditions on individual features, and the consequent takes a real value if the antecedent is true and zero otherwise.
They are produced by algorithms such as SLIPPER (Cohen & Singer, 1999), that of R¨uckert &
Kramer (2006), RuleFit (Friedman & Popescu, 2008), ENDER (Dembczy´nski et al., 2010) and have also been referred to as generalized linear rule models (Wei et al., 2019).
A rule ensemble can be converted into a tree ensemble by converting each conjunctive rule into an IF-THEN-ELSE rule list, which is a one-sided tree (see Appendix B.1).
Speciﬁcally, the conditions in the conjunction are taken in any order, each condition is negated to become an IF condition, and the THEN consequents are all output values of zero.
The ﬁnal ELSE consequent, which is reached if all the IF conditions are false (and hence the original rule holds), returns the output value of the original rule.
The number of leaves in the resulting tree equals the number of conditions in the conjunction plus one.
Require: M adjacency matrix, H heuristic function 1: Z
[ i ] = 1 ∀ i ∈ V , B = { 1 , 2 , . .
, K + 1 } , S = ∅ (cid:46)
All nodes valid, all trees uncovered 2: Q = E NUMERATE ( Z, B, S ) 3: Initialize:
D lb = 0 , D ub =(cid:46)
Anytime bounds 4: function E NUMERATE
( Z, B, S ) 5: t = arg max b {| Z b | (cid:12)(cid:12) b ∈ B } (cid:46)
Uncovered tree with fewest valid nodes 6: for i in Z t do 7:
Z [ i ] =
(cid:46)
Mark node as incompatiable 8:
S = S ∪ { i } (cid:46) Add to candidate clique 9: if | S | = K + 1 then : (cid:46) Is it a max clique?
D lb = max ( D lb , D ( S ))
(cid:46) Update primal bound 11:
Q = Q ∪ S (cid:46)
Add to set of max cliques 12:
S =
S \ { i } (cid:46) Backtrack 13: else 14:
Z t +1 =
Z t ∧ M ( i ) (cid:46) Update valid nodes 15:
B = B \ { t } (cid:46) Update uncovered trees 16:
D ub = max ( D ub , H ( S ))
(cid:46) Update dual bound 17: if D ub > D lb then : (cid:46)
Pruning by bound 18:
E NUMERATE ( Z t +1 , B, S )
(cid:46)
Recurse to next level 19: end if 20:
S =
S \ { i } (cid:46) Backtrack 21:
B = B ∪ { t } 22: end if 23: end for
24: end function B. P IECEWISE L IPSCHITZ F UNCTIONS
Proof of Proposition 3.
Consider two inputs x and y then,
[Equation]
[Equation]
where, c = 2 max( c 0 , c 1 ) and β = min( β 0 , β 1 ) .
Other choices for D ( ., . )
: The results assumed D ( ., . ) to be the identity function, where ∆ =
D ( f 0 , f ) .
This choice of function clearly satisﬁes assumptions 1 and 2.
Again consistent with these assumptions we look at some other choices for D ( ., . ) .
If D ( ., . ) were an afﬁne function with a positive scaling such as D ( y 0 , y )
=
α ( y 0 − y )
+ b where α > 0 , then our result in equationwould be unchanged as only the Lipschitz constant of ∆ would change, but not its (underestimated) order.
If the function were a polynomial or exponential however, no such guarantees can be made and we would be back to case 1.
B. P OST H OC E XPLANATIONS FOR O THER M ODEL C LASSES
For model classes beyond the ones discussed in Section 4, it is less clear whether there exist reasonably tractable algorithms that guarantee exact computation of or bounds on the maximum deviation.
In this case, it is natural to ask whether post hoc explanations for the model can help.
One way in which this could occur is if the post hoc explanation approximates the model f by a simpler model
ˆ f
and if the deviation function D satisﬁes the triangle inequality
D ( f ( x ) , f 0 ( x )) ≤ D ( f ( x ) , ˆ f ( x ))
+ D ( ˆ f ( x ) , f 0 ( x )) .
Then the maximum deviation in (1) would be bounded as
[Equation]
While we may choose ˆ f to be interpretable so that the rightmost maximization is tractable, the middle maximization asks for a uniform bound on the deviation between f and ˆ f , i.e, the ﬁdelity of ˆ f .
We are not aware of a post hoc explanation method that provides such a guarantee.
Indeed, in general, the middle maximization might not be any easier than the left-hand one that we set out to bound.
A (practical) possibility may be to perform quantile regression (Koenker, 2005) for a large enough quantile to learn ˆ f , as opposed to minimizing expected error as is typically done.
This may be an interesting direction to explore in the future as quantile regression algorithms are available for varied model classes including linear models, tree ensembles (Meinshausen, 2006) and even neural networks (Petneh´azi, 2019).
More investigation is needed into whether quantile regression methods can provide approximate guarantees on the middle term in (15).
C F URTHER D ISCUSSION
Worst-case approach The formulation of (1) as the worst case over a certiﬁcation set represents a deliberate choice to depend as little as possible on a probability distribution or a dataset sampled from one.
As stated in Section 2, Certiﬁcation Set paragraph, C can depend at most on (an expanded version of) the support set of a distribution.
The reason for this choice is because safety is an outof-distribution notion: harmful outputs often arise precisely because they were not anticipated in the data.
The trade-off inherent in this choice is that the maximum deviation may be more conservative than needed.
The high maximum deviation values in e.g. Figures 1 and 2 may reﬂect this.
Given deﬁnition (1) as a starting point in this paper, future work could consider variations that depend more on a distribution and are thus less conservative, but may also offer a weaker safety guarantee.
Choice of reference model
The proposed deﬁnition of maximum deviation (1) depends on the choice of reference model f 0 .
Different choices will lead to different deviation values and, perhaps more importantly, different combinations of features that maximize the deviation.
We have discussed possible choices in Section 2, and the results in Section 4 indicate that, as with the assessed model f , interpretable forms for f 0 can ease the computation of maximum deviation.
Beyond these guidelines, it is up to ML practitioners and domain experts to decide on appropriate reference models for their application (and there may be beneﬁt to considering more than one).
We mention an additional concern with the reference model in the Ethics Discussion.
For some real applications it may be difﬁcult to come up with a globally interpretable reference model.
But speciﬁc to particular scenarios it may be possible.
For instance, it might be difﬁcult to provide general rules for how to drive a car, but in speciﬁc scenarios such as there being an obstacle in front, one can suggest that you stop or turn, which is a simple rule.
So our machinery could potentially be applied at a local level where the reference model is interpretable in that locality.
This might help in “spot checking” a deployed model and estimating its safety by computing these maximum deviations in scrupulously selected (challenging) scenarios.
Impossible inputs in certiﬁcation set Mathematically simple sets such as Cartesian products and (cid:96) p balls permit simpler algorithms for optimizing functions over them.
Accordingly, these sets have been the focus of not only the present work but also the related literature on ML veriﬁcation and adversarial robustness.
However, they may not serve to exclude inputs that are physically or logically impossible from the certiﬁcation set C , and thus, the resulting maximum deviation values may be too large and conservative.
Here it is important to distinguish between impossible inputs and those that are merely implausible (i.e., with low probability).
Techniques for capturing implausibility have been proposed for contrastive/counterfactual explanations (Dhurandhar et al., 2018; Luss et al., 2021), whereas we expect the set of impossible inputs to be smaller and more constrained.
As a simple example from the Adult Income dataset, if we agree that a wife/husband is deﬁned to be of female/male gender (regardless of the gender of the spouse), then the cross combinations male-wife and female-husband cannot occur.
Future work can consider the representation and handling of such constraints.
White-box vs. grey-box models In this paper, we have assumed full “white-box” access to both f and f 0 , namely complete knowledge of their structure and parameters.
Interesting questions may arise when this assumption is relaxed to different “grey-box” possibilities.
For example, one could further investigate the third case in Section 4.4, where one of f, f 0 is black-box and the other is white-box interpretable.
There may exist assumptions that we have not identiﬁed that would improve the query complexity compared to generic black-box optimization.
Other interpretability-safety relationships This paper has focused on one relationship between the interpretability of a model and the safety of its outputs.
It has not addressed other ways in which interpretability/explainability can affect the risk of a model (in the plain English sense, not the expectation of a loss function).
For example, in regulated industries such as consumer ﬁnance, not providing explanations or providing inadequate ones can lead to legal, ﬁnancial, and reputational risks.
On the other hand, providing explanations is associated with its own risks (Weller, 2019).
These include the leakage of personal information or model information (intellectual property), an increase in appeals of decisions for the decision-making entity, and strategic manipulation of attributes (i.e. “gaming”) by individuals to gain more favorable outcomes.
Applicability to RL settings In RL, if one views the actions as labels and state representation as features, one can build a tree, albeit likely a deep/wide one, to represent exactly the RL policy, where the probability distribution over the actions can be viewed as the class distribution in a normal supervised setting.
Rolling up the states, creating leaves with multiple states, and simply averaging the probabilities for each action would yield smaller trees that approximate the policy.
Our work lays a foundation where in principle we can also compare f and f 0 that are policies using such tree representations.
This may be related to a popular global explainability method (Bastani et al., 2018) that samples policies and builds trees to explain them.
D A DDITIONAL E XPERIMENTAL R ESULTS D. A DULT I NCOME DATASET
Reference model Figure 3 depicts the 8 -leaf DT reference model used in the experiments on the
Adult Income dataset.
The root node separates individuals based on whether the marital status is Married-civ-spouse .
The remaining splits divide the population into those with high and low education, high and low capital gains, and high and low capital losses.
In particular, having high capital gains or losses is a good predictor of high income ( > $50000 )
[Table]
LR and GAM models In Tables 1 and 2, we show the values of inverse (cid:96) 1 penalty C and max bins that were used for LR and GAM respectively, as well as statistics of the resulting classiﬁers.
Test set accuracy and area under the receiver operating characteristic (AUC) increase and reach a plateau.
For LR, we take the (cid:96) 1 norm of the coefﬁcients to be the main measure of smoothness as it depends on both the number of nonzero coefﬁcients as well as their magnitudes, which both
[Figure] [Figure]
affect the extreme values attained in (6).
Based in part on Tables 1 and 2, we select C = 0 .
01 and max bins = 8 as representative models that remain simple and have accuracies and AUCs not far from the maximum attainable.
Plots for these two models are shown in Figures 4 and 5.
Maximum deviation for decision trees with small numbers of leaves In Figure 6, we expand upon the second panel in Figure 1 by focusing on decision trees with fewer than 50 leaves (plotted with linear spacing).
The maximum deviation is 0 .
114 for trees with 9 and 10 leaves, and remains moderate up to 26 leaves.
Relationships with accuracy and robust accuracy In Figure 7, we show maximum deviation as a function of test set accuracy for the DT, LR, and GAM models shown in the right three panels of Figure 1 (the LR and GAM models are listed in Tables 1 and 2).
Broadly, the plots show two
[Figure]
regimes: one where accuracy increases and maximum deviation increases moderately or not at all, and one where accuracy stalls while maximum deviation increases.
The latter is less desirable as it suggests increasing safety risks without a gain in accuracy.
The last branch of the DT curve actually decreases in accuracy, indicating overﬁtting, while maximum deviation is high.
We also consider the relationship of maximum deviation to robust accuracy .
Following Wong &
Kolter (2018), robust loss for a pair ( x, y ) is deﬁned as the worst-case loss over an (cid:96) ∞ ball centered at x ,
[Figure]
[Equation]
(16) and robust accuracy is therefore 1 minus the average robust 0 - 1 loss over a dataset.
While Wong &
Kolter (2018) focus on bounding robust loss for feedforward neural networks with ReLU activations, we ﬁnd that the results in Section 4.2 apply to computing robust loss (16) exactly for LR and GAM models.
Speciﬁcally, for 0 - 1 loss and (cid:96) ∞ balls, the separable optimization (8) applies, and the worst case is obtained by minimizing f when the label y is positive and maximizing f when y is negative.
The resulting robust accuracy values for DT, LR and GAM are plotted in Figure 8 in a similar fashion as Figure 7.
Here we set (cid:15) = 0 .
1 and r = 0 .
1 as well in computing maximum deviation.
The DT plot shows maximum deviation increasing with model complexity while robust accuracy is stable up to a point.
In the subsequent regime, when there are a large number of leaves, model robustness reduces while deviation remains high.
The LR plot begins similarly to the one in Figure 7 in that robust accuracy increases along with maximum deviation, but then it stalls and decreases for maximum deviation above 0 .
In the GAM plot, robust accuracy actually decreases with the max bins parameter, i.e., the curve goes from right to left.
Breakdown by leaves of f
In Figures 9–12, we plot the maximum log-odds achieved by model f ( max on RHS of (7)), the minimum log-odds achieved by f , and the reference model log-odds g ( y 0 m ) over each leaf of the decision tree reference model in Figure 3.
Plots are on the log-odds scale to show the deviations more clearly, including those that would be compressed by the nonlinear logistic function g − 1 ( z )
= 1 / (1 + e − z ) .
Figures 9 and 11 show the dependence on the certiﬁcation set radius r while Figures 10 and 12 show dependence on the smoothness parameters for LR and
GAM.
These ﬁgures provide a more granular picture corresponding to the summary in Figure 1 and support the trends seen there.
In Figures 9 and 11, there are jumps at r = 1 because this is the smallest radius that permits the values of categorical features of test set points (the ball centers in (2)) to change to any other value.
(Recall that categorical features are one-hot encoded into binaryvalued features.)
In Figure 11, the GAM achieves the limiting deviations corresponding to r = ∞ ( C = X , dashed lines) no later than r = 1 . 2 .
In Figure 9, the LR model achieves the lower limit on log-odds as soon as r > 1 but the upper limit is not achieved for most leaves.
[Table]
[Table] [Table]
[Figure]
[Table]
05k10k15k
HoursPerWeek c o r e
D e n s i t y
[Table]
[Figure]
K 1 .
05 K - 1 .2 K 1 .
22 K - 1 .9 K 1 .
39 K - 1 .7 K 1 .
57 K - 1 .4 K 1 .
74 K - 1 .2 K 1 .
92 K - 2 .9 K 2 .
09 K - 2 .7 K 2 .
27 K - 2 .4 K 2 .
44 K - 2 .1 K 2 .
61 K - 2 .9 K 2 .
79 K - 2 .
K 2 .
96 K - 3 .4 K 3 .
14 K - 3 .1 K 3 .
31 K - 3 .8 K 3 .
[Figure]
[Figure] [Figure]
[Figure] [Figure]
83 K - 4 .
[Figure]
[Figure] [Figure]
[Figure] [Figure]
CapitalLoss
S c
o r e
D e n s i t y −
A m e r - I n d i a n - E
s k
i m o
A
[Figure]
i
[Figure]
n
[Figure]
P
[Figure]
c
[Figure]
I
[Figure]
a n
[Figure]
e r
[Figure]
l
[Figure]
c
[Figure]
O t h e r
W h
i t e10k20k
Race
[Figure] [Figure]
[Figure] [Figure]
v i a t i
[Figure]
n D ( r = 0 . 1 ) [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
Running time Figure 13 shows the time required to compute the maximum deviation for LR and
GAM on the Adult Income dataset.
These times were obtained using a single 2 .
GHz core of a server with 64 GB of memory (only a small fraction of which was used) running Ubuntu 16.
(64-bit).
The times increase with the (cid:96) ∞ ball radius r because of the increasing number of ball-leaf intersections that become non-empty and hence need to be evaluated.
The time for r = 0 is minimal because this case requires only model evaluation over the ﬁnite test set, as mentioned.
The jumps at r = 1 are due again to the ability of categorical features to change values, leading to an increase in ball-leaf intersections.
The ﬁlled-in regions show that there was little variation due to different (cid:96)norms for LR or max bins for GAM.
This was most likely because of a vectorized implementation, which operates on all LR coefﬁcients or all GAM bins at once (i.e., without a for loop).
Maximal cliques evaluated for DT, RF To investigate the effectiveness of pruning by bounds in
Algorithm 1, we investigate the number of times all the decision trees in the Random Forest have to be processed.
This represents number of times the state could not be pruned and needed to be evaluated fully.
Figure 14 shows two aspects at play.
(a) Pruning by bound is effective in restricting the search space more so for Random Forests than for decision trees, and (b) for larger graphs, more time is spent in computing bounds in Eq.
[Figure]
K +
[Figure]
m a x c li q u e s e v a l u a t e d
[Figure]
Feature combinations that maximize deviation In Tables 3 and 4, we report feature values that maximize deviation over selected leaves of the DT reference model, for LR and GAM respectively.
For Table 3, we have chosen the minimum log-odds over leaf 6 (corresponding to Figure 9, leaf 6, blue curve), which is one of the two leaves m in (7) that maximize the deviation overall (the other being leaf 8).
For Table 4, the minimum log-odds over leaf 12 is chosen (corresponding to
Figure 11, leaf 12, blue curve) because this maximizes the deviation overall for most values of r .
The tables show the 6 features that contribute most to the minimum log-odds.
These contributions are determined using (8) (with max replaced by min ); since the minimum log-odds occurs in one of the (cid:96) ∞ ball-leaf intersections and this intersection is a Cartesian product, the decomposition in (8) applies.
The contribution of feature j is then min x j ∈S j f j ( x j ) .
To account for the multiple r values, we take an average of the contributions over r .
[Table]
As r increases, the predominant trend of the values of continuous features is toward extremes of the domain X , depending on the sign of the corresponding LR coefﬁcient w j or shape of the GAM function f j .
For example, EducationNum (education on an ordinal scale), hours per week, and age decrease toward minimum values, while capital gain occupies the minimal interval permitted for leaf 12 (see Figure 3).
(These examples make sense since the log-odds of high income is being minimized.)
This movement toward extremes is expected in the LR case because the functions w j x j are either increasing or decreasing, and it is also true for GAM if the function f
j is mainly increasing or decreasing.
The values sometimes change abruptly in the opposite direction, for example hours per week in both Tables 3, 4, and age in the latter.
These abrupt changes are due to the minimum jumping from one ball in (2) to another as r increases, but the overall trend eventually prevails.
For categorical features, the trend is toward values that minimize f j ( x j ) , e.g., Never-married marital status, Without-pay work class.
While the contribution of each of these features to minimizing logodds may be limited, together they do add up.
[Table]
A notable exception to the trend toward extremes is capital loss in Table 4.
For this feature, the values settle into the intermediate interval [1598 , 1759] .
The plot of the CapitalLoss function in
Figure 5 shows that this interval corresponds to a curiously low value of the function.
This low region may be an artifact since it seems anomalous compared to the rest of the CapitalLoss function, and since individuals who report capital losses on their income tax returns to offset capital gains usually have high income (hence high log-odds).
Given the results in Tables 3 and 4, one question that arises is whether the feature combinations are indeed possible, if not the ones for r → ∞ , then at least for some ﬁnite value of r .
For the top features shown in the two tables, while some combinations may appear improbable (for example,
EducationNum = 1 and 1 hour per week), we submit that none appear impossible .
However, if one considers features beyond the top 6, then some “impossible” combinations do occur (e.g., a female husband), although the contributions of these features to the minimum log-odds are much less.
We touch upon this issue again in Appendix C.
The next question one might consider is the implication of these maximal deviations.
From Figure 3, it is seen that leaf 6 classiﬁes individuals with high capital gains as high income with high probability ( 0 . 965 ).
Leaf 12 adds the attributes of married status and high education, and hence classiﬁes as high income with even higher probability ( 0 . 996 ).
At the same time, the feature values in Tablesand 4, which minimize log-odds for LR and GAM, also make sense according to basic domain knowledge.
For example, few hours per week and young age are associated with lower income, as are Without-pay work class and Amer-Indian-Eskimo race in the United States.
When these conﬂicting associations occur in combination and the combination does not appear impossible, the question may be which one prevails.
Such a question might be resolvable by a domain expert.
Alternatively, the disagreement between models f and f 0 on the extreme examples in Tables 3,may be reason to be cautious about using either of the models in these cases.
This might lead to a way of combining the models or abstaining from prediction altogether.
Lastly, the anomalously low region in the CapitalLoss function identiﬁed in Table 4 is a clear, concrete example where further investigation is warranted.
D. L ENDING C LUB DATASET
This dataset consists of 2.26 million rows with 14 features on loans.
The target variable is whether a loan will be paid-off or defaulted on.
Features describe the terms of the loan, e.g. duration, grade, purpose, etc.
and borrower ﬁnancial information such as credit history and income.
For this case study, we consider a loan approval scenario using only information available at the time of application.
In particular, we exclude the feature ‘total pymnt’ (total payment over time on the loan), which becomes known at essentially the same time as the target variable.
(When ‘total pymnt’ is included as a feature, the prediction task becomes easy and accuracies in the high 90% range are possible.)
[Figure]
Reference model Figure 15 depicts the 8 -leaf DT reference model for the Lending Club dataset.
Most of the splits partition the ‘sub grade’ feature, which is a measure of the quality of the loan ( 0 – 34 range, lower is better).
Node 3 differentiates between 60-month terms and 36-month terms (the only alternative), while nodes 4 and 7 split on ‘added dti’ (added debt-to-income ratio), which is the ratio between 12 months worth of the loan’s payment installments and the borrower’s annual income.
While the structure of the DT agrees with domain knowledge (lower ‘sub grade’ and lower ‘added dti’ correlate with higher repayment probability), the test set accuracy of 79 .
8% is no better than that of the trivial predictor that always returns the majority class of “paid off”.
The DT’s AUC of 0 .
689 however does indicate an improvement over the trivial predictor.
LR and GAM models Tables 5 and 6 show the statistics of the LR and GAM classiﬁers that were trained on the Lending Club data.
Similar to the DT reference model, the difference compared to
Tables 1, 2 for the Adult Income dataset is that the accuracies remain no better than that of the trivial predictor, while the AUC does not show much increase either.
These statistics suggest that the prediction task is difﬁcult with the features available.
Figures 16 and 17 display plots for the
LR model with C = 0 .
01 and GAM with max bins = 8 , which are again chosen as representative
[Table] [Table]
models.
The GAM in particular shows sensible monotonic behavior as functions of ‘sub grade’, ‘int rate’ (interest rate), ‘dti’ (debt-to-income ratio), etc., despite the unimpressive accuracy.
[Figure]
revol_util funded_amnt credit_history purpose=debt_consolidation added_dti int_rate installment purpose=
small_business dti annual_inc term= 36 months sub_grade intercept
[Table]
[Figure]
[Table]
[Figure]
[Table]
int_rate
[Figure]
c o r e
D e n
[Table]
i t y
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
100k200k dti
S c
o r e
D e n s i t y
[Figure]
[Table]
inc
S c
o r e
D e n s i t y −0.
c a r c r e d i t _ c a r d d e b t _ c o n s o li d a t
i o n e d u c a t i o n a l h o m e _ i m p r o v e m e n t h
o u s e m a j o r _ p u r c
h a s e m e d i c a l m o v i n g o t
h e rr e n e w a b l e _ e n e r
g
y
s m a ll _ b u s i n e ss v a c a t i o n w e dd i n g50k100k150k purpose
S c
o r e
D e n s i t y
[Figure] [Figure]
[Figure]
s i n g l e D
e
c
i s i o n T r
[Figure]
Maximum deviation summary Figures 18–20 show maximum deviation as functions of certiﬁcation set radius r and model complexity parameters, in a similar manner as Figures 1 and 2 for the
Adult Income dataset.
The qualitative patterns are similar to before: increasing maximum deviation in all cases except with the number of RF estimators in Figure 20b, where the upper bound is stable around 0 .
A major quantitative difference is that the maximum deviations for the GAM in Figure 19 are much lower than for the other models, in particular LR in Figure 18.
This is likely due to
[Figure]
the fact that the GAM functions f j in Figure 17 are bounded while still being monotonic, unlike the linear functions w j x j in the LR model.
Breakdown by leaves of f
Figures 21–24 show a breakdown of the deviations for LR and GAM by leaves of the reference model, similar to Figures 9–12 and again on the log-odds scale.
One difference is that in Figure 21, the deviations for ﬁnite r do not come close to their r → ∞ counterparts in most cases.
In Figure 23 however, the r → ∞ values are all attained when r is slightly greater than
[Figure] [Figure]
dd
[Figure] [Figure]
l
[Figure] [Figure]
l o g
[Figure]
o
[Figure]
l
[Figure] [Figure]
l o g
[Figure]
dd
[Figure]
l
[Figure] [Figure]
[Figure] [Figure]
Running time, maximal cliques evaluated Figure 25 shows the time required to compute the maximum deviation for LR and GAM on the Lending Club dataset.
Figure 26 shows the number of
K + 1 -maximal cliques evaluated for DT and RF as well as the number of nodes in the graph.
The observations are the same as in Figures 13 and 14.
[Figure]
[Figure]
Feature combinations that maximize deviation Tables 7 and 8 present the feature values that maximize deviation for LR and GAM respectively.
For both tables, the minimum log-odds over leaf 5 of the DT reference model is selected (corresponding to because this choice maximizes the deviation overall for most values of r .
The previous trend toward extreme feature values that minimize log-odds also holds here as r increases.
For example, debt-to-income ratios increase to outlier values above 100% , annual income drops to the minimum of $100 , the term changes to 60 months in Table 7 for r > 1 , and the purpose changes to small business, the category with the lowest log-odds in Figure 17.
The decrease in income and increases in debt-to-income ratios are qualitatively in accordance with each other.
However, these quantities are related to each other by deterministic formulas (at least in theory) that also involve the interest rate and installment amount.
It is not clear whether the values in Tablesand 8 violate these relationships.
This may be an instance that could beneﬁt from constraints on possible feature combinations, as brieﬂy mentioned in Appendix C.
[Table] [Table]