N x P lain: A Web-based Tool for Discovery of Latent Concepts (2023)

Abstract: The proliferation of deep neural networks in various domains has seen an increased need for the interpretability of these models, especially in scenarios where fairness and trust are as important as model performance. A lot of independent work is being carried out to: i) analyze what linguistic and non-linguistic knowledge is learned within these models, and ii) highlight the salient parts of the input. We present NxPlain, a web-app that provides an explanation of a model’s prediction using latent concepts. NxPlain discovers latent concepts learned in a deep NLP model, provides an interpretation of the knowledge learned in the model, and explains its predictions based on the used concepts. The application allows users to browse through the latent concepts in an intuitive order, letting them efficiently scan through the most salient concepts with a global corpus-level view and a local sentence-level view. Our tool is useful for debugging, unraveling model bias, and for highlighting spurious correlations in a model. A hosted demo is available here: https://nxplain.qcri.org

## introduction interpretation of deep neural networks (dnns) has gained a lot of attention in recent years, especially in nlp, where state-of-the-art models are being widely deployed and used in practice. work done in interpretation can be broadly classified into two branches: i) representation analysis and ii) attribution analysis. the former attempts to understand what knowledge is learned within the representation @xcite and the latter is focused on how the model predicts the output @xcite @xcite . 2 a drawback of the methods in representation analysis is that it does not gauge whether the model uses what it has learned in making a prediction. on the other hand, the drawback of attribution analysis is that their explanations are limited to discrete units (e.g. words, some specific piece of the network), and the abstract nuances behind these discrete units are lost in the explanation, resulting in an inadequate or implausible explanation. some efforts have been made in trying to connect representation and attribution analysis @xcite . in this work, we present nxplain, a web-app that provides a holistic view by combining representation and attribution analysis. more specifically, we discover latent concepts in the model using the latent concept analysis @xcite and connect these concepts to specific predictions using integrated gradients @xcite , a model and input saliency method. nxplain allows the users to: ## pipeline components the nxplain application provides an easy interface to analyze the latent knowledge learned within a deep nlp model, as well as connect these latent concepts to specific predictions. in order to do this, the pipeline in the backend relies on three key components proposed by recent literature: i) concept discovery, ii) concept alignment, and iii) attribution analysis. ## frontend views the goal of nxplain is to provide an easy method for users to extract and analyze latent knowledge learned within a deep nlp model and connect them to the prediction. the frontend helps achieve this goal by providing a intuitive yet powerful gui that can be used to interact with a model's latent concepts and predictions. the user can upload a model and a corpus that they want to analyze. the computational queue of the application discovers latent concepts and aligns them using the components mentioned in section 3. the user can then use the frontend, where they can switch between three major views: overall view: this view presents a high-level overview of the concepts learned by the model. specifically, we can see i) the number of concepts learned, ii) statistics on the concepts aligned with the human-fined concepts, iii) a summary of the size distribution of these concepts, iv) and salient concepts in the data and model. figure 3 shows a sample overview page for a sentiment analysis model. ## conclusion we presented nxplain, a web-app for connecting concept analysis with model prediction. the application bridges representation analysis and attribution analysis to better explain the models' predictions, and provides a intuitive, yet powerful graphical interface to explore the knowledge learned by a model, and also to pinpoint the knowledge used in specific predictions. in the future, we plan to enable human-in-the-loop to enhance concept alignment, as well as incorporate feedback into the explanation system. a hosted version of the application can be accessed at https://nxplain.qcri.org .