Trends in XAI tools & research at NeurIPS 2021
Published: 2022-02-04 10:00:55

---
title: Trends in XAI tools & research at NeurIPS 2021
published: true
date: 2022-02-04 07:21:34 UTC
tags: 
canonical_url: https://dataroots.io/research/contributions/trends-in-xai-part-2-tools-research
---

eXplainable AI or XAI is crucial to ensure stakeholder and public trust, as well as reliability, particularly in high-stake contexts where AI decisions can impact lives.

Open-source contributors, researchers & companies are stepping up their game by providing ever-more ambitious and inventive methods to ensure interpretable and transparent AI. As a consequence, XAI methods are sprouting up like mushrooms, meaning that the decision on which method to use is becoming increasingly complicated.

With great new open-source initiatives and many more start-ups making it their founding mission to cover some or multiple aspects of model & data reliability, explainability and monitoring, combined with the increasing awareness of biases and gaps in existing explainability methods, it can be somewhat overwhelming to address explainability and interpretation in a high-stakes, high-paced business context.

Rather than loop you through existing XAI methods, we want to take you on an interactive tour of what's yet to come. Here, we'd like to show you **what's been published, what's been hyped, and what's actually relatively easy to start using in your projects.**

💡

For an overview of existing XAI methods, take a look at [our introduction to explainability](https://dataroots.io/research/contributions/machine-learning-explainability-an-introduction-to-cracking-open-the-black-box) instead. Our data strategy unit is actively developing materials to support you in your journey to making AI more explainable, trustworthy and transparent.

## Human-in-the-loop XAI

At NeurIPS 2021 alone, we saw a huge number of promising, novel application methods relying on learning from supervision_,_ and especially interactive learning with humans in the loop, to tackle existing shortcomings in XAI.

### Human-advisable reinforcement learning with visual feedback

**[EXPAND](https://arxiv.org/pdf/2006.14804.pdf)** is a significantly more powerful technique to integrate human feedback for data augmentation compared to other SOTA explanatory interactive learning methods. EXPAND achieves this by leveraging a powerful symbolic lingua franca (shared vocabulary) to bridge the human-AI communication gap: Instead of numerical label feedback, humans provide visual feedback by drawing bounding boxes on salient areas of the image which allows the AI agent to integrate human knowledge.

![](https://github.com/GuanSuns/Simple-Human-in-the-Loop-ML-Interface/raw/master/img/example.gif)
_Source_

At the current time, reinforcement learning agents are very costly to train. EXPAND enables the integration of visual rather than numerical feedback which is much more informative and thus sample-effective, enabling us to train RL agents faster, better and cheaper.

### Data quality & interactive learning

Faulty labels are a pervasive issue that can significantly impact the ultimate performance and useability of AI-driven solutions.

![](https://dataroots.ghost.io/content/images/2022/02/image-10.png)
_Examples of different types of label errors in popular image datasets. Source_

[**CINCER**](https://github.com/abonte/cincer) **,** is a new data cleaning method which re-examines label noise using sequential learning and a human-in-the-loop to optimize existing label cleaning methods and cover some of their drawbacks.

![](https://dataroots.ghost.io/content/images/2022/02/image-4.png)
_CINCER identified the left-most image as a "suspicious" example, i.e. likely to be mislabeled. Based on a similar counter-example, which is correctly annotated as 4, we should clearly adjust our annotation._

Specifically, CINCER selects mislabeled samples and presents it to a human for rellabelling. It's core mechanism is centered around identifying "suspicious" samples and selecting an appropriate counterexample which should maximally clarify that either the suspicious sample, or the counterexample, is mislabelled.

![](https://dataroots.ghost.io/content/images/2022/02/image-5.png)
_If the counter-example proposed by CINCER is annotated as a 7, then the current sample must also be a 7. Because the counter-example is clearly mis-annotated, CINCER provides a contrastive and maximally informative counterexample, in comparison to other methods._

The authors of CINCER described well what the method does better than its competitors: [1]

> CINCER identifies the reasons behind the model’s skepticism and asks the supervisor to double-check them too. Like most interactive approaches, there is a risk that CINCER annoys the user by asking an excessive number of questions. This is mitigated by querying the user only when the model is confident enough in its own predictions and by selecting influential counter-examples that have a high chance to improve the model upon relabeling, thus reducing the future chance of pointless queries.

💡

While CINCER is a promising method, it's not industry-ready. For a recent but mature solution to noisy labels, look at [cleanlab](https://github.com/cleanlab/cleanlab): this nifty package wraps around many existing solutions to take into account noisy labels, or allows you to identify and improve label issues. Unfortunately, you can't benefit from interactive learning with human feedback with this tool.

## Novel methods in post-hoc XAI

These approaches lend themselves well to nearly any application domain, data modality or model type. However, their generalizability means their novel contribution to XAI is usually more limited to transparency, and at best interpretability, for all stakeholders, and _explainability_ only to developers and domain experts.

### [InputIBA](https://github.com/camp-explain-ai/inputiba) - Information Bottleneck Attribution revised

![](https://github.com/CAMP-eXplain-AI/InputIBA/raw/master/resources/demo.gif)
_Input image of a piggy bank, and an image mask that represents the features/pixels that contributed most significantly to the prediction of a piggy bank. Source_

Feature attributions indicate how much each feature in your current data instance contributed to the final prediction. In the image domain, that can translate to showing the extent to which each pixel contributes.

Feature attribution methods were already extensively applied in the past, but InputIBA uses a novel approach based on information bottleneck attribution (IBA) that enables a much more robust approach in the image domain:

![](https://github.com/CAMP-eXplain-AI/InputIBA/raw/master/resources/example_results.jpg)
_A comparison of InputIBA compared to SOTA feature explanation techniques in the image domain. Source_

As well as in the language domain, for example by identifying informative words & symbols, enabling increased human interpretability:

![](https://github.com/CAMP-eXplain-AI/InputIBA/raw/master/resources/NLP_example.PNG)
_A comparison with other feature explanation techniques in the language domain, including the highly influential LIME method. Source_

### [SimplEx](https://github.com/jonathancrabbe/simplex?ref=pythonrepo.com) - Leveraging latent explanations of black-box models using examples

SimplEx elegantly combines feature explanation techniques and example-based techniques, tying together two powerful XAI frameworks.

💡

_The most common approach to provide example-based explanations for a wide variety of models mirrors feature importance methods. The idea is to **complement the model prediction by attributing a score to each training example**. This score reflects the importance of each training example for the model to issue its prediction._ [5]

In the image below, our model classified the left-most image as a three. On the right, we can see what examples in our corpus (a representative dataset composed by the human user) were most important for the model prediction. The scores represent the extent of the importance. What's different, compared to traditional example-based methods, is that each image also shows the importance of each pixel (from red to green, with dark-green representing most important).

![](https://dataroots.ghost.io/content/images/2022/02/image-11.png)
_An example of SimplEx in the image domain. Source_

In a more traditional machine learning context utilizing tabular data, SimplEx both provides individual examples from its corpus, and provides the importance of each feature in that example.

![](https://dataroots.ghost.io/content/images/2022/02/image-12.png)
_An example of SimplEx with tabular data in a medical context. Source_

What does SimplEx provide compared to its contemporary counterparts? What makes it so revolutionary? We cannot put it any more succintly or appropriately than the authors themselves [5]:

> **(1) SimplEx gives the user freedom to choose the corpus of examples whom with the model’s predictions are decomposed. (...) there is no need for this corpus of examples to be equal to the model’s training set.**   
>   
> This is particularly interesting for two reasons:   
> **(a) the training set of a model is not always accessible**   
> **(b) the user might want explanations in terms of examples that make sense for them**.   
> **For instance, a doctor might want to understand the predictions of a risk model in terms of patients they know**.   
>   
> **(2) The decompositions of SimplEx are valid, both in latent and output space. (**...) **SimplEx offer significantly more precision and robustness than previous methods (...)**   
>   
> **(3) SimplEx details the role played by each feature in the corpus mixture.** (...) **This creates a bridge between two research directions that have mostly developed independently: feature importance and example-based explanations**

💡

Particularly neat about SimplEx, is that the repository is well-documented and (relatively) easy to start using in an industry context. [Go check their code out!](https://github.com/JonathanCrabbe/Simplex)

## Uncertainty quantification of XAI methods

Existing explainability methods are being rightfully re-examined and improved, especially using uncertainty quantification. In industry, feature importance and similar metrics (typically based on Shapley or LIME) are frequently used uncritically to assess model reliability and establish a high-level interpretation that can be communicated to stakeholders. By allowing developers and users to assess the uncertainty of such methods, we can, to some extent, mitigate their shortcomings.

#### Uncertainty estimation for feature importance methods such as LIME or Shapley

In many cases **the uncertainty of existing feature importance methods were not or incorrectly reported, which may inflate confidence in a model that provides erroneous predictions**.

In **Reliable Post hoc Explanations: Modeling Uncertainty in Explainability** [6], this issue is being tackled. Instead of providing a point estimate of the feature importance, a distribution is estimated instead:

![](https://github.com/dylan-slack/Modeling-Uncertainty-Local-Explainability/raw/main/data/posteriors_fig_1.png)
_An example of feature importance distributions in tabular data. Source_

Distributional reasoning about feature importances opens up much more advanced approaches or at the very least rightfully portrays the inherent uncertainty in estimating feature importances, and hopefully provides some protection from using them uncritically for making important decisions.

#### Uncertainty estimation for neural networks by bootstrapping

**NeuBoots [7]** is a foundational contribution that may be **used for bootstrapping neural networks and as such deriving uncertainty**  **metrics** enabling the usage of conventional statistical uncertainty measures (e.g. condifence intervals), or to **improve performance by relying on the uncertainty estimation of individual predictions.**

Bootstrapping, in a nutshell, means that we fit a model multiple times, while re-sampling the data in a statistically rigorous manner to induce and assess variability in our model parameters and/or predictions.

In statistics, bootstrapping is a reliable approach for estimating statistical parameters. Unfortunately, in the context of deep learning, bootstrapping is oftentimes to computationally complex. Bootstrapping neural networks can be approximated by using ensemble methods, but NeuBoots is more efficient:

![](https://dataroots.ghost.io/content/images/2022/02/image-13.png)
_Source_

NeuBoots is comparable to a standard bootstrap at estimating confidence intervals in the context of neural networks:

![](https://dataroots.ghost.io/content/images/2022/02/image-14.png)
_(a) Frequentist coverage rate of the 95% confidence bands; (b) Curve fitting with different nonlinear funtions. 95% confidence band of the regression mean from NeuBoots; (c) the standard bootstrap. Each red dashed line indicates the mean, and the blue dotted lines show the true regression function. Source _

(And of course, it is much more efficient. For the above comparison, the authors had to train 1000 neural networks, whereas the NeuBoots algorithm evaluates the network only once.)

NeuBoots can be efficiently used for **prediction calibration.**

💡

Prediction calibration refers to ...

**For example, in image classification, the confidence estimates of neural networks are often not very representative, but using NeuBoots the confidence is much more representative.**

In conclusion, NeuBoots is a computationally efficient, scaleable approach to bootstrapping and uncertainty estimation that can be applied to many different types of neural networks. There's many more interesting functionalities in the context of XAI enabled by the power of uncertainty estimation, but for a complete overview, we recommend you have a look at [the paper.](https://arxiv.org/abs/2010.01051)

* * *

**Want to know more about these new approaches? Did we miss an important method\*?** Let us know and we may just dedicate a post to it!

**Confused about what method to use for your usecase?** [Contact us to see if we can help you out.](https://dataroots.io/)

**Stay tuned for more posts to come:** **[subscribe to our RSS feed](https://dataroots.io/ss.xml) and our [LinkedIn page](https://be.linkedin.com/company/dataroots).**

At [dataroots research](https://dataroots.io/research/) we'll be exploring what methods are crucial for enabling the continuity of services in the face of upcoming AI regulations, and translating academic output to industry-ready XAI tooling to accelerate integration into your business processes.

### References

[1] **CINCER:** Teso, S., Bontempelli, A., Giunchiglia, F., & Passerini, A. (2021). Interactive Label Cleaning with Example-based Explanations. arXiv preprint [arXiv:2106.03922](https://arxiv.org/pdf/2106.03922.pdf).

[2] **InputIBA:** Zhang, Y., Khakzar, A., Li, Y., Farshad, A., Kim, S. T., & Navab, N. (2021). Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information. _Advances in Neural Information Processing Systems_, _34_. arXiv preprint [arXiv:2110.01471](https://arxiv.org/abs/2110.01471)

[3] **InputIBA - Open Review:** [https://openreview.net/forum?id=HglgPZAYhcG](https://openreview.net/forum?id=HglgPZAYhcG)

[4] **Simplex - Open Review:** [https://openreview.net/forum?id=PIcuKeiWvj-](https://openreview.net/forum?id=PIcuKeiWvj-)

[5] **SimplEx:** Crabbé, J., Qian, Z., Imrie, F., & van der Schaar, M. (2021). Explaining Latent Representations with a Corpus of Examples. _Advances in Neural Information Processing Systems_, _34_. arXiv prepint [arXiv:2110.15355](https://arxiv.org/pdf/2110.15355.pdf)

[6] Slack, D., Hilgard, A., Singh, S., & Lakkaraju, H. (2021). **Reliable post hoc explanations** : **Modeling uncertainty in explainability**. _Advances in Neural Information Processing Systems_, _34_. arXiv preprint [arXiv:2008.05030](https://arxiv.org/abs/2008.05030)

[7] **NeuBoots:** Shin, M., Cho, H., Min, H. S., & Lim, S. (2021). Neural Bootstrapper. _Advances in Neural Information Processing Systems_, _34_. arXiv preprint [arXiv:2010.01051](https://arxiv.org/abs/2010.01051)