Top 5 Model Interpretability Libraries for Python
Tags: python, machinelearning, datascience, deeplearning
Published: 2021-01-14 15:14:48

Welcome to the post about the Top 5 Python ML Model Interpretability libraries! You ask yourself how we selected the libraries? Well, we took them from our [Best of Machine Learning with Python list](https://github.com/ml-tooling/best-of-ml-python#model-interpretability).  All libraries on this best-of list are automatically ranked by a quality score based on a variety of metrics, such as GitHub stars, code activity, used license and other factors. You can find more details in the [best-of-generator repo](https://github.com/best-of-lists/best-of-generator).

Model interpretability - also known as model explainability - is an important field of machine learning that helps, for instance, with debugging models or detecting issues around biases, or overall increases trust in the results. The overall goal of deploying models to production is to make decisions or build upon them, which can only work when you trust them in the first place!

---

> We recently started a newsletter and blog post series with focus on machine learning and the Python ecosystem. In this newsletter, we usually give a high-level overview of interesting libraries from our [best-of list](https://github.com/ml-tooling/best-of-ml-python) and [projects](https://github.com/ml-tooling/best-of-ml-python) we work on to support your machine learning efforts! Feel free to subscribe to stay updated 🙂

[![Discover the best-of machine learning tooling to improve your productivity.](https://dev-to-uploads.s3.amazonaws.com/i/wnqkp8xev047b5bjqskh.png)](https://mltooling.substack.com)

---

The intention of this post is to highlight some of the most popular libraries in the field based on our metric and give a crisp overview to make it hopefully easy for you to gain some high-level understanding! If you want to try them out, we can recommend our [ml-workspace container](https://github.com/ml-tooling/ml-workspace/tree/master/resources) to do so, as it comes pre-installed with all but the last of the libraries in the list!

So, that is enough of an introduction, here comes the list of our top 5 interpretability libraries.

## [1. SHAP](https://github.com/slundberg/shap)

The first library on our list is SHAP and rightly so with an impressive number of 11.4k stars on GitHub and active maintenance with over 200 commits in December alone. SHAP, which is an acronym for SHapley Additive exPlanations, is a model-agnostic explainer library that follows a game theory approach to assign importance values to features. You can find the original paper from 2017 that appeared in NIPS on [arxiv](https://arxiv.org/abs/1705.07874). 

As the idea behind Shapley Values is already nicely explained by others, for example in this neat [book chapter](https://christophm.github.io/interpretable-ml-book/shapley.html) or this well-written [blog post](https://towardsdatascience.com/shap-explained-the-way-i-wish-someone-explained-it-to-me-ab81cc69ef30), we keep it high-level: it distributes the contribution of a feature related to all combinations of other features. Based on this approach, the library allows to explain individual predictions but also can easily give a global overview.

The theory and math behind it is not so easy to explain, but luckily the library hides all of this from the user. For example, the following image

![SHAP explanation visualization](https://dev-to-uploads.s3.amazonaws.com/i/3f4wlm89t69gy9qawpp9.png)
*- [source](https://raw.githubusercontent.com/slundberg/shap/master/docs/artwork/boston_instance.png)*

can be generated with this code snippet

```python
import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
shap.force_plot(
    explainer.expected_value,
    shap_values[1, :],
    X.iloc[1, :],
)
```

where `model` is the model to be explained and `X` is the input data. As SHAP is model-agnostic, it works with all kinds of models. 

As you can imagine, the computation of the various combinations for many features can be very expensive and so the authors provide a fast C++ implementation for tree models as explained [here](https://github.com/slundberg/shap#tree-ensemble-example-with-treeexplainer-xgboostlightgbmcatboostscikit-learnpyspark-models). 

The GitHub page contains nice examples and tutorial notebooks, so it should be quite easy to start with it!

---

## [2. Lime](https://github.com/marcotcr/lime)

The library Lime, short for *Local interpretable model-agnostic explanations*, follows second on our list with an impressive number of 8.3k stars, last activity 21 days ago, and some nice [tutorials](https://github.com/marcotcr/lime#tutorials-and-api) and [API definition](https://lime-ml.readthedocs.io/en/latest/).

Lime is able to explain tabular data classifiers and text classifiers independent of the actual model. As the authors of SHAP, the authors of Lime came from the University of Washington - well done Washington to have 2 libraries in the top 5 👏 - and you can find their original paper from 2016 also on [arxiv](https://arxiv.org/abs/1602.04938). 

Lime is a concrete implementation of so-called *local surrogate models*, which focus on explaining individual predictions of black box models. To do so, Lime generates new data around the data point you want to have an explanation for and trains a linear, interpretable classifier.

![Lime local linear model](https://dev-to-uploads.s3.amazonaws.com/i/0usevkqqtok81flzk8nr.png)
*- [source](https://github.com/marcotcr/lime#what-are-explanations)*

You can then see what features were responsible for the local model's classification (in this example, it shows which words in the text contributed to the text belonging to the class *atheism*).

![Class feature contribution](https://dev-to-uploads.s3.amazonaws.com/i/uqh3uwincvypeh6xei49.png)
*- [source](https://marcotcr.github.io/lime/tutorials/Lime%20-%20basic%20usage%2C%20two%20class%20case.html)*

In contrast to SHAP, it does not look like Lime offers an API to conveniently generate a global explanation or at least we missed this. However, by generating multiple explanations for different examples, you could gain an understanding of the original model's behavior.

---

## [3. ELI5](https://github.com/TeamHG-Memex/eli5/blob/master/docs/source/index.rst)

The Python library ELI5 provides a unified API for debugging and explaining machine learning classifiers. The term ELI5 can be found [in the urban dictionary](https://www.urbandictionary.com/define.php?term=ELI5) and is slang for *Explain Like I am 5.* For black-box models it offers an implementation of the Lime algorithm for texts. Besides that, it offers support for [scikit-learn](https://eli5.readthedocs.io/en/latest/libraries/sklearn.html#library-scikit-learn), [XGBoost](https://eli5.readthedocs.io/en/latest/libraries/xgboost.html#library-xgboost), [LightGBM](https://eli5.readthedocs.io/en/latest/libraries/lightgbm.html#library-lightgbm), [CatBoost](https://eli5.readthedocs.io/en/latest/libraries/catboost.html#library-catboost), [lightning](https://eli5.readthedocs.io/en/latest/libraries/lightning.html#library-lightning), [sklearn-crfsuite](https://eli5.readthedocs.io/en/latest/libraries/sklearn_crfsuite.html#library-sklearn-crfsuite), and [Keras](https://eli5.readthedocs.io/en/latest/libraries/keras.html#library-keras). [Check out their documentation](https://eli5.read) for examples and an API reference. 

The following image shows how an explanation by ELI5 looks like for a text classification, where it shows which words contribute how much to a specific class.

![ELI5 text classification explanation visualization](https://dev-to-uploads.s3.amazonaws.com/i/as75ak0kzithk29dpexx.png)
*- [source](https://eli5.readthedocs.io/en/latest/tutorials/sklearn-text.html#baseline-model)*

It is generated by this code

```python
import eli5

eli5.show_weights(
    clf, vec=vec, top=10, target_names=target_names
)
```

where `clf` is, for example, an [sklearn LogisticRegressionCV](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn-linear-model-logisticregressioncv), `vec` is a vectorizer, for example an [sklearn CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) that transform texts into a token count matrix, `top` is the number of features to show per class in the image, and `target_names` tells *eli5* the actual labels of the classification based on the input data.

---

## [4. pyLDAvis](https://github.com/bmabey/pyLDAvis)

As the name already suggests, pyLDAvis focuses on [LDA topic models](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation). It can be used to interactively visualize them within [Jupyter Notebooks](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwid3pmStY3uAhWnyYUKHRi8DrUQFjAAegQIARAD&url=https%3A%2F%2Fipython.org%2Fnotebook.html&usg=AOvVaw2VRRFs60Sx4Mh2FsoDzD45). [The documentation](https://pyldavis.readthedocs.io/en/latest/readme.html#usage) contains example videos and also an overview notebook, though compared with the other libraries the API could have been documented a little bit better. The original paper from 2014 explains the methodology and can be found [here](https://nlp.stanford.edu/events/illvi2014/papers/sievert-illvi2014.pdf).

The documentation states that this lib is agnostic to how the model was trained, as you only need the *topic-term distribution*, *document-topic distributions*, and *basic corpus information*. For [sklearn](https://scikit-learn.org/stable/), [gensim](https://github.com/RaRe-Technologies/gensim), and [graphlab](https://github.com/apple/turicreate/) it even has convenient methods. For example, the following code utilizing the gensim helper function

```python
import pyLDAvis.gensim

pyLDAvis.gensim.prepare(lda, corpus, dictionary)
```

would output an interactive graphic which is displayed in the following image. The params `lda` is a gensim lda model, `corpus` is a [gensim matrix market corpus](https://radimrehurek.com/gensim/corpora/mmcorpus.html#module-gensim.corpora.mmcorpus), and `dictionary` is a gensim dictionary ([see their docs for the complete example](https://nbviewer.jupyter.org/github/bmabey/pyLDAvis/blob/master/notebooks/pyLDAvis_overview.ipynb#Making-the-common-case-easy---Gensim-and-others!)). That's it!

![pyLDAvis visualization](https://dev-to-uploads.s3.amazonaws.com/i/inkzm458x0bse1eupz5w.png)
*- Screenshot taken from executed tutorial code*

In the interactive visualization, you can click on the different topics, see which words belong to them, change the weighting parameter and such things!

---

## [5. InterpretML](https://github.com/interpretml/interpret)

InterpretML is in Alpha Release but already made it into the top 5 and is under active development; even one of our [Pull Requests on GitHub](https://github.com/interpretml/interpret/pull/192) we created during writing this post was merged within 7 hours 🥳! It is a library from Microsoft and has two functionalities: first, training glassbox models that are interpretable and, second, explaining blackbox models. Their paper can be found [here](https://arxiv.org/pdf/1909.09223.pdf). [In their Supported Techniques](https://github.com/interpretml/interpret#supported-techniques) section they mention the techniques the library brings with it, for example Decision Tree and SHAP Kernel Explainer as a glassbox model or blackbox explainer, respectively. As you can see, in the interpretability library world, the same techniques and libraries show up and are cross-referenced or used from each other.

The library is supposed to work for shedding light on individual predictions as well as making the global behavior understandable, which is also reflected in the API. With following lines of code, you can train one of their glassbox models and get some visualization for local and global interpretability

```python
from interpret.glassbox import ExplainableBoostingClassifier
from interpret import show

# train
ebm = ExplainableBoostingClassifier()
ebm.fit(X_train, y_train)

# global
ebm_global = ebm.explain_global()
show(ebm_global)

# local
ebm_local = ebm.explain_local(X_test, y_test)
show(ebm_local)
```

You can even pass an array of explanations to their `show` function to get a comparison visualization.

![InterpretML explanation comparison](https://dev-to-uploads.s3.amazonaws.com/i/y90pbqc0hlwffukd3vkk.png)
*- [source](https://github.com/interpretml/interpret#train-a-glassbox-model)*

The documentation links some more example notebooks for the other techniques, too, as well as tutorial videos or blog posts which are interesting to check out for those of you who want to go more into breadth or depth!

---

We hope you enjoyed the read and learned something about the most popular libraries that allow you to interpret and explain your models 💡 


If you liked this overview, have a look at the other [Python Machine Learning Libraries](https://github.com/ml-tooling/best-of-ml-python) we ranked in our best-of list. For updates on trending projects and detailed library comparisons, follow us on [Twitter](https://twitter.com/mltooling) or subscribe to our [newsletter](https://mltooling.substack.com/subscribe).

And, of course, we are very happy about all kinds of feedback! Leave a comment or get in touch with us via [team@mltooling.org](mailto:team@mltooling.org) or via [Twitter @mltooling](https://twitter.com/mltooling).

With ❤️ from Berlin!