Latent Concept-based Explanation of NLP Models (2024)

Abstract: Interpreting and understanding the predictions made by deep learning models poses a formidable challenge due to their inherently opaque nature. Many previous efforts aimed at explaining these predictions rely on input features, specifically, the words within NLP models. However, such explanations are often less informative due to the discrete nature of these words and their lack of contextual verbosity. To address this limitation, we introduce the Latent Concept Attribution method (LACOAT), which generates explanations for predictions based on latent concepts. Our foundational intuition is that a word can exhibit multiple facets, contingent upon the context in which it is used. Therefore, given a word in context, the latent space derived from our training process reflects a specific facet of that word. LACOAT functions by mapping the representations of salient input words into the training latent space, allowing it to provide latent context-based explanations of the prediction.

## introduction the opacity of deep neural network (dnn) models is a major challenge in ensuring a safe and trustworthy ai system. extensive research works have attempted to interpret and explain these models. one major line of work strives to understand and explain the prediction of a neural network model using the attribution of input words to prediction @xcite . however, the explanation based solely on input words is less informative due to the discrete nature of words and the lack of contextual verbosity. a word consists of multifaceted aspects, such as semantic, morphological, and syntactic roles in a sentence. consider the word "trump" in figure 1 . it has several facets such as a verb, a verb with specific semantics, and a named entity representing a figure 1 : an example of various facets of word "trump" certain aspect such as tower names, family names, etc. we argue that given various contexts of a word in the training data, the model learns these diverse facets during training. given a test instance, depending on the context a word appears, the model uses a particular facet of the input words in making the prediction. the explanation based on salient words alone does not reflect the facets of the word the model has used in the prediction and results in a less informed explanation. @xcite showed that the latent space of dnns represents the multifaceted aspects of words learned during training. the clustering of training data contextualized representations provides access to these multifaceted concepts, hereafter referred to as latent concepts. given an input word in context at test time, we hypothesize that the alignment of its contextualized representation to a latent concept represents the facet of the word being used by the model for that particular input. we further hypothesize that this latent concept serves as a correct and enriched explanation of the input word. to this end, we propose the latent concept attribution (lacoat) method that generates an explanation of a model's prediction using the latent concepts. lacoat discovers latent concepts of every layer of the model by clustering contextualized representations of words in the training corpus. given a test instance, it identifies the most salient input representations of every layer with respect to the prediction and dynamically maps them to the latent concepts of the training data. the shortlisted latent concepts serve as an explanation of the prediction. lastly, lacoat integrates a plausibility module that generates a human-friendly explanation of the latent concept-based explanation. lacoat is a local explanation method that provides an explanation of a single test instance. the reliance on the training data latent space makes the explanation reliable and further reflects on the quality of learning of the model and the training data. we perform qualitative and quantitative evaluation of lacoat using four classification tasks across four pre-trained models. lacoat generates an enriched explanation that is useful in understanding the model's reasoning for a prediction. we also conduct a human evaluation to measure the utility of lacoat with a human-in-the-loop. moreover, we measure the faithfulness of the most salient latent concept to the prediction using representation manipulation and show that it alters the prediction up to 46% of the time. ## methodology lacoat consists of the following four modules: • the first module, conceptdiscoverer, discovers latent concepts of a model given a corpus. consider a sentiment classification dataset and a sentiment classification model as an example. lacoat works as follows: conceptdiscoverer takes the training dataset and the model as input and outputs the model's latent concepts. at test time, given an input sentence, predictionattributor identifies the most salient input representations related to the prediction. conceptmapper maps these salient input representations to the training data's latent concepts and provides them as an explanation for the prediction. plausifyer takes the test sentence along with its concept-based explanation to generate a human-friendly and insightful interpretation of the prediction. consider m as the dnn model being interpreted, which has l layers, each of size h. let -→ z w i be the contextual representation of a word w i in an input sentence w 1 , w 2 , ..., w i , .... the representation can belong to any layer in the model, and lacoat will generate explanations with respect to that layer. ## experimental setup data we use parts-of-speech (pos) tagging, toxicity classification (toxicity), sentiment classification (sentiment), and natural language inference (nli) tasks for our experiments. pos is a sequence labeling task, while the other tasks are sequence classification tasks. we use the penn treebank dataset @xcite for pos, the jigsaw toxicity dataset @xcite for toxicity, the eraser movie reviews dataset @xcite for sentiment, and the mnli dataset @xcite for the nli task. appendix a provides information about each dataset. ## evaluation we perform a qualitative evaluation, a human evaluation and a module-level evaluation of lacoat to measure its correctness and efficacy. we find consistent results across all tasks and models. due to space limitation, we mainly present the results of pos and sentiment using the bert and roberta models in the main paper. the full set of results are presented in apps. f, g, h. ## related work the explainability methods can be approached by local explanations and global explanations targeting post-hoc analysis or introducing interpretability in the architecture @xcite @xcite @xcite @xcite @xcite @xcite . @xcite provides a survey of explainability methods in nlp. lacoat is a local explanation method providing post-hoc explanations given an input instance. previous work attempted to explain and interpret nlp models using human-defined concepts @xcite and concepts extracted from hidden representations @xcite @xcite . @xcite ; @xcite worked on the global explanation based on a surrogate model. we provide local explanations and we ensure the faithfulness of latent concepts by extracting them directly from the hidden representation without any supervised training. rajani et al. (2020) used k-nearest neighbors of the training data to identify erroneous correlations and misclassified instances. @xcite ; @xcite analyzed latent concepts in their ability to represent linguistic knowledge. our conceptdiscoverer module is motivated by them. however, we propose a method to explain a model's prediction using latent concepts. ## conclusion we presented lacoat that provides a faithful and human-friendly explanation of a model's prediction. the qualitative evaluation and human evaluation showed that lacoat explanations are insightful in explaining a correct prediction, in highlighting a wrong prediction and in comparing the explanations of models. the reliance on training data latent space enables interpreting how knowledge is structured in the network. similarly, it enables the study of the evolution of predictions across layers. lacoat promises human-in-the-loop in the decision-making process and is a step towards trust in ai.