COCKATIEL : CO ntinuous Concept ran K ed AT tribution with Interpretable EL ements for explaining neural net classifiers on NLP (2023)

Abstract: Transformer architectures are complex and their use in NLP, while it has engendered many successes, makes their interpretability or explainability challenging. Recent debates have shown that attention maps and attribution methods are unreliable (Pruthi et al., 2019; Brunner et al., 2019). In this paper, we present some of their limitations and introduce COCKATIEL, which successfully addresses some of them. COCKATIEL is a novel, post-hoc, concept-based, model-agnostic XAI technique that generates meaningful explanations from the last layer of a neural net model trained on an NLP classification task by using Non-Negative Matrix Factorization (NMF) to discover the concepts the model leverages to make predictions and by exploiting a Sensitivity Analysis to estimate accurately the importance of each of these concepts for the model. It does so without compromising the accuracy of the underlying model or requiring a new one to be trained. We conduct experiments in single and multi-aspect sentiment analysis tasks and we show COCKATIEL’s superior ability to discover concepts that align with humans’ on Transformer models without any supervision, we objectively verify the faithfulness of its explanations through fidelity metrics, and we showcase its ability to provide meaningful explanations in two different datasets. Our code is freely available: https://github.com/fanny-jourdan/cockatiel

## introduction nlp models have undeniably gotten increasingly more complex since the introduction of the transformer architecture @xcite figure 1 : an illustration of cockatiel. given some sentences of imdb reviews, cockatiel (i) identifies concepts for prediction, (ii) ranks them, and (iii) gives the most important elements for each concept (to help us interpret the concept). @xcite . this trend, which is also occurring in the domain of computer vision, has brought about a need for understanding how these models make their predictions. the presence of bias in these models could indeed be prejudicial in applications where the user's lives are at stake @xcite . humans should be able to comprehend the reasons behind the model's decisions if these models are to gain general acceptance. also, companies need to ensure that they are deploying algorithms which are free of harmful biases and that the explanations that they are obligated to issue are easily understandable by employees and end-users alike @xcite . intelligibility by humans has then become a key topic in explainable ai systems. as ai systems become more sophisticated and are deployed in increasingly complex environments, the ability to provide clear and concise explanations of their decisions becomes more pressing. researchers have proposed multiple solutions to address this challenge. the most straightforward approach analyzes how each part of the input influences the model's prediction. there are different ways of doing this, through perturbation @xcite or by leveraging the gradients inside the neural network @xcite . however, these approaches suffer from being vulnerable to adversarial manipulation @xcite , from only performing partial input recovery @xcite and from a general lack of stability with respect to the input @xcite . another research path for transformer models harnesses the information in the attention maps of the transformers' layers to understand how the elements in the input relate to the output, implying that the attention mechanism is inherently interpretable. in spite of a number of supporters initially for this approach, there has been a recent wave of detractors of attentionbased explanations @xcite @xcite . more in line with our proposal work, researchers in the field of rationalization have proposed specific architectures to extract excerpts from whole inputs and predict a model's output based on these rationales @xcite @xcite @xcite . these rationales can be seen as explanations that are sufficiently high-level to be easily understood by humans. however, they require to train an entirely new model. only one rationale can also be found per input text, when there might intuitively be several predictions for a given prediction. finally, these approaches use architectures that have mostly been left behind since the introduction of the transformer architecture, due to their inferior predictive capabilities. in line with the project of generating explanations that are meaningful to humans, concept-based explainable ai (xai) has lately advanced the sate of the art. the pioneer method tcav @xcite goes beyond widespread attribution methods to create high-level explanations based on handpicked concepts. more recently, @xcite has extended this technique to discover automatically pertinent concepts inside the network's activation space and to find the parts of the input space that most align with each concept. still, it has only been applied to convolutional architectures for image classification tasks. in this paper, we present cockatiel, a novel technique for generating reliable and meaningful explanations for nlp neural architectures for classification problems. it extends craft @xcite and our contributions can be summarized as follows: ## cockatiel in this section, we describe cockatiel, our concept-based xai technique for nlp models to generate human-understandable explanations. it has three main components: (i) it uses non-negative matrix factorization (nmf) to discover the concepts that the neural network under study leverages to make predictions; (ii) it exploiting sensitivity analysis to estimate accurately the importance of each of these concepts for the model; and (iii) it uses a black-box explainability technique to generate instance-wise explanations at a per-word and per-clause level. fig. 2 presents a schematic outline of cockatiel. notation in a supervised learning framework, we assume that a neural network model f : x @xmath0. we denote by (x 1 , ..., x n ) ∈ x n a set of n input texts and (y 1 , ..., y n ) ∈ y n their associated labels. we consider f to be a composition of h, the last embedding of x (i.e. the last layer of the feature extractor model), and c, the classification function, f (x) = c • h(x) with h(x) ⊆ r p . cockatiel will factorize h through nmf, so we require h to be non-negative -i.e. h(x) ≥ 0 ∀@xmath1. this constraint is typically verified when the last layer has an activation function such that σ(x) ≥ 0, which is the case in (but it's not limited to) layers or blocks using relu. ## experimental evaluation for all of our results, we fine-tuned roberta @xcite based models on each dataset. we ensured the non-negativity of at least one layer of the model by adding a relu activation after the first layer of the 1-hidden-layer, dense mlp of the classification head. for the qualitative analysis, we also tested cockatiel's performance on bidirectional lstm models trained from scratch. more details about the implementations are left in appendix b. ## conclusion in this paper, we revisited concept-based explainability techniques and presented cockatiel, a post-hoc, model agnostic method capable of generating meaningful and faithful explanations for nlp models trained on classification tasks. the method has three parts: (i) a concept part, using non-negative matrix factorization to discover the figure 6 : concepts generated with @xmath2. the colored elements are those important for the concept of the corresponding color (calculated with part (iii) of our method). the more colorful the element, the more important it is for the concept (continuously). we have selected the 3 most important concepts for each label (see fig. 3 ). the name of the concept is chosen manually in view of the important elements corresponding to the concepts. we measured cockatiel's ability to discover concepts that align with those humans and obtained better scores than state-of-the-art methods. we demonstrated that in addition to generating meaningful concepts for humans, these explanations are faithful to the models. finally, we gave some qualitative examples of explanations for different models to understand the method "in practice".