Characterizing Anomalies with Explainable Classifiers
Abstract
As machine learning techniques are increasingly used to make societal-scale decisions, model performance issues stemming from data-drift can result in costly consequences.
While methods exist to quantify data-drift, a further classification of drifted points into groups of similarly anomalous points can be helpful for practitioners as a means to combating drift (e g. by providing context about how/where in the data pipeline shift might be introduced).
We show how such characterization is possible by making use of tools from the model explainability literature.
We also show how simple rules can be extracted to generate database queries for anomalous data and detect anomalous data in the future
Introduction
Fielded machine learning systems help corporations such as banks and technology firms, as well as governmental ad other institutions, make key decisions on a daily basis.
However, these systems are often used and trusted despite the fact that input covariates may be distributed differently than they were during training.
Demand-forecasting algorithms may be thrown off by new consumer practices that began during the COVID-19 pandemic [1].
Fraud-detection algorithms may gradually become ineffective as agents that intend to commit fraud develop more sophisticated strategies [2, 3].
While various techniques 1 exist to measure the extent to which the distribution has shifted, such measurements alone do not give practictioners intuition about what characterizes shifted points.
These characterizations can be especially difficult to construct and intuit when the correlations between covariates shift as opposed to the marginal distributions: metrics that track univariate drift for each feature are among the most interpretable, but in these cases they would fall short.
We propose a method based on tools from the explainable classification literature to (i) measure the total multivariate covariate shift present, (ii) identify and characterize clusters of similarly anomalous points, and (iii) construct simple rules that practitioners may use to isolate anomalous data for further exploratory data analysis.
Our approach fundamentally builds on the idea of discriminative distance [5], wherein the statistical distance between two distributions is thought of in terms of the performance of a classifier that aims to tell the two distributions apart.
If the classifier is able to distinguish between the two distributions easily, then we think of the two distributions as being far apart and vice-versa.
Our approach takes this idea a step further by making use of the classifier itself (by means of tools from the model explainability literature) to characterize and identify data points that are anomalous for similar reasons.
While we evaluate our approach on tabular datasets, in principle, our approach can be implemented on any form of data (e.g. images, text) for which tractable classification algorithms with local additive explanations exist.
[Table]
To obtain the drifted sets we first randomly split into a train and test set, where the test set comprises 25% of the data and the methods described in
Related Work
Isolation Forests and Statistical Divergences Isolation forests [6] are a common technique used to identify anomalies in datasets.
They function using the underlying intuition that anomalous datapoints are likely to be separated in data-space from the non-anomalous points.
However, for detecting anomalies in model covariates, isolation forests may not obtain good performance relative to discriminative approaches as they do not make use of the knowledge model designers have about the explicit partitioning of data as either training or test data.
Statistical divergences such as the
KL-Divergence are also often used to detect data drift, but due to issues that arise from data sparsity in higher dimensions, they are typically only applied to compute univariate drift.
Discriminative Distance and Domain Classification
Our work can be viewed as an extension of discriminative distance/domain classification approaches [5, 7, 4].
These approaches measure the statistical distance between two distributions in terms of the performance of an auxiliary classifier that aims to distinguish between samples from the two distributions.
We build on these approaches to further use the learned model to characterize anomalies.
Local Additive Explanations Given a model f : R d → R , a local additive explanation method ϕ assigns to each point x ∈ R d in the dataset and explanation vector ϕ ( x ) ∈ R d that satisfies ϕ
+ (cid:80) i ϕ ( x ) i
=
f ( x ) for some constant ϕ 0 .
The components ϕ ( x ) i are intended to denote the contribution of the value of feature i to the final model prediction.
Examples of popular local additive explanation methods include SHAP [8] and LIME [9].
We use SHAP in our approach to identify similarly anomalous groups of points.
High-Dimensional Visualization To visualize and qualitatively understand the performance of our approach
, we make use of nonlinear dimensionality reduction tools to project high-dimensional point clouds into two dimensions.
We principally make use of an approach known as UMAP [10], which assumes a Riemannian structure on the data manifold to make good visualizations.
Other popular tools such as TSNE [11] exist for this task as well.
Synthetic Data Generation
Before describing our method in full, we first outline the synthetic testing framework that we used to evaluate and visualize our approach.
To make our testing setup as realistic as possible, we developed a method that takes as input a tabular dataset, and splits the dataset into a training dataset and a drifted test set.
By inputting different real-world datasets, we are able to test our approach on a variety of realistic data manifolds.
To construct a synthetically drifted test set from some original test set with numerical columns
N ∈ R m × n and categorical columns C ∈ Object m × c , we first randomly select a number G to determine the number of anomalous groups we will construct.
For each g ∈ [ G ] , we randomly select some disjoint subset R g ⊂ [ m ] of rows to belong to group g , and subsequently randomly select a (small) subset of numerical features F numg ⊂ [ n ] of size s num and a similar subset of categorical features F catg ⊂ [ c ] of size s cat to perturb for each element in the group.
To build a rich class of non-linear possible numerical data drifts, we first generate a random groupspecific affine transformation T numg : R s num → R s num , and repeatedly apply it to build the drifted numerical columns N ′ ∈
R m × n .
As any smooth non-linear transformation can be approximated by the resulting polynomial operator, we are able to tractably sample realistic data drifts.
Formally, for some k ∈ N , we let N ′
[ R g , F numg ] :=
T numg ◦ · · · ◦ T numg [Equation] times (cid:0) N
[ R g , F numg ] (cid:1) and subsequently let the remainder of N ′ be unchanged (i.e. N ′
[ R g , F numg ] :=
N [ R g , F numg ] , N ′ (cid:104)(cid:84) g ∈ [ G ] R g , : (cid:105) :=
N (cid:104)(cid:84) g ∈ [ G ] R g , : (cid:105) ).
We define T numg ( Z ) :
= ( I s num ×
s num + ϵ g ) Z + S g where ϵ g ∈ R s num ×
s num and S g ∈ R s num
× s num are randomly generated Gaussian noise.
To shift the categorical features, we first select an element α i at random from the respective support of each column i ∈ F catg to be shifted.
We then “lock” the feature values for each row in group g to be the corresponding α i .
Formally, we let the shifted categorical columns be given by C ′ ∈ Object m × c , and let C ′
[ R g , F catg ] :=   α i. . .
α i scat .
α i. . .
α i scat 
 .
As before, we let the remainder of C ′ be unchanged from C : C ′
[ R g , F catg ] :=
C [ R g , F catg ] and C ′ (cid:104)(cid:84) g ∈ [ G ] R g , : (cid:105) :
=
C (cid:104)(cid:84) g ∈ [ G ] R g , : (cid:105) .
Running Examples We use our data generator to generate synthetically drifted test data for two datasets, which we use as running examples for the remainder of the paper.
Details regarding the datasets and the anomalous groups generated can be found in Table 1.
Identifying and Characterizing Anomalies
We now illustrate the function of our method on our two running examples.
In the Default on Credit
Card Clients (DCCC) example, the underlying dataset and the two anomalous groups are relatively larger (constituting a net ≈ 30% of the test dataset), and they have been perturbed by a greater extent.
In the Statlog example, the underlying dataset is 30 times smaller, and the anomalous data makes up just 10% of the test data.
Ultimately, we will show that in both cases, it is indeed possible to approximately recover all of the information contained in Table 1 given just the training and drifted test set alone.
Measuring Drift We first describe how to quantify the net distributional shift that has occurred in a way that takes into account changes not only in marginal distributions, but also in correlations.
While there are a few well-known approaches to do this (see Section 2), we take a discriminative distance approach.
More specifically, we train a random forest classifier f to distinguish between training and test instances.
We visualize the train and test sets, as well as the predicted score generated by the random forest on a holdout set.
In the Default on Credit Card Clients example, the two anomalous groups can be clearly seen, and indeed the random forest is able to clearly identify those points as anomalous.
While it is unclear exactly where the anomalous points are located in the Statlog example, the random forest score does seem to increase in regions where more orange test points are visible.
We propose to measure the distributional shift through the AUC 2 of f on a holdout set.
If the AUC is close to 0 .
5 , then the net shift is negligible, and if the AUC is close to 1 , then the shift is extreme.
Indeed, the AUCs as reported in Figure 1 correspond with our intuition regarding the extent of the shift.
Identifying Anomalous Groups We now show how with a local additive explanation method, we can use the classifier f to approximately recover the information contained in Table 1.
We make use of the SHAP package [8, 15] to generate these explanations for our random forest classifier.
When identifying groups of points that are similarly anomalous, it is important to note that such points may be located far from each other in data-space.
For example, if a group of Americans and Europeans adopt the same fraud strategy, this anomalous group might be represented in data-space as two blobs that are spaced apart due to differences in location.
Indeed, in the Statlog example, we know thatpoints were shifted in the same way, but their locations are not concentrated together.
We can already make some insights about the features that were shifted by looking at the SHAP summary plots (see
Figure 5 in Appendix A).
To identify groups of similarly anomalous points, we make use of local additive explanations.
For each point in data-space, we get a corresponding vector of explanations with the same dimensionality, such that the sum of the explanations is equal to predicted score (as returned by f ).
If two points are close in explanation-space, we know that their predicted anomaly score is similar, since the sum of the components must be similar.
Furthermore, the reason why the points are similarly anomalous must also similar, since the explanation vectors are similar.
Thus, by clustering points in explanation-space, we can identify the anomalous groups!
We use DBSCAN [16] for clustering, and use Kneedle [17] for hyperparameter tuning as first introduced in [18] (see Figure 6 for details).
Clustering is done in high-dimensional space – not in the UMAP visualization.
See Figure 2 for a description of the anomaly scores over explanation space.
By looking at the SHAP signatures of the identified anomalous blobs, we can approximately recover
F numg and F catg .
From the clustering, we estimate the sizes of the anomalous DCCC clusters to be 960 and 1560 , and the size of the Statlog cluster to be 22 .
The performance of our approach on the Statlog example is particularly impressive as the size of the anomalous group and the magnitude of the shift is quite small (indeed, the shift is practically imperceptible in the UMAP projection).
See Figure 8 for more details.
In addition to these synthetically drifted datasets, we have also tested our approach on the
Shifts dataset [19]: a real-world shifted dataset containing seasonal weather data.
Building Simple Rules Finally, using the Skope-Rules package [20], we can extract simple rules that approximately isolate anomalous clusters.
These rules {+(e.g., “ PAY_2 > 5 . 
 and PAY_4 ≤ 2 . 5 ” in the DCCC case)+} can be used as database queries that practitioners can use for exploratory data analysis, or as a means for detecting future anomalous instances.
See Figure 4 for more information. Cluster
Rule
Precision/Recall
DCCC
{+PAY_2 > 5 . 
+}
and PAY_4 ≤ 2 .1 /
DCCC
BILL_AMT5 > 884308 .1 /
Statlog
Guarantors > 1 . 5 and Dur._in_Cur_Addr.
≤ 1 .0 .