Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models (2023)

Abstract: Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), but their lack of interpretability has been a major concern. Current methods for interpreting LLMs are post hoc, applied after inference time, and have limitations such as their focus on low-level features and lack of explainability at higher-level text units. In this work, we introduce proto-lm, a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings during the fine-tuning stage while maintaining competitive performance. Our method’s applicability and interpretability are demonstrated through experiments on a wide range of NLP tasks, and our results indicate a new possibility of creating interpretable models without sacrificing performance. This novel approach to interpretability in LLMs can pave the way for more interpretable models without the need to sacrifice performance. We release our code at https://github.com/yx131/proto-lm.

## introduction in recent years, large language models (llms) have significantly improved results on a wide range of natural language processing (nlp) tasks. however, despite their state-of-the-art performance, llms, such as bert @xcite , roberta @xcite and bart @xcite , are not easily interpretable. interpretability is a crucial aspect of language models, especially llms, as it enables trust and adoption in various domains. to address this issue, there is a growing interest in improving model interpretability for llms and neural models in general. current interpretation methods have several limitations, such as requiring a surrogate model to be built @xcite or being applied post-hoc to each instance, separate * co-corresponding authors. from the original decision-making process of the model @xcite @xcite . these limitations add extra computational complexity to the explanation process and can result in approximate explanations that are unfaithful foot_0 to the original model's decision-making process @xcite . finally, current interpretability methods focus on attributing importance to different words in the input and do not explain the model's decision at the sentence or sample level. @xcite @xcite . to address these challenges, we propose a novel framework that utilizes a prototypical network to learn an interpretable prototypical embedding layer on top of a fine-tuned llm, which can be trained end-to-end for a downstream task. our framework utilizes trainable parameters, called prototypes, to both perform the downstream task by capturing important features from the training dataset and provide explanations for the model's decisions via projection onto the most influential training examples. additionally, we utilize a token-level attention layer before the prototypical layer to select relevant parts within each input text, allowing our model to attribute importance to individual words like existing interpretability methods. our framework, named proto-lm, achieves competitive results on a wide range of nlp tasks and offers interpretability while addressing the previously mentioned limitations of current interpretability methods. fig. 1 shows our proposed proto-lm framework applied to a multi-classification example from the sst5 dataset @xcite . as the figure illustrates, the model's decision-making process is simple, transparent, and accurate as it classifies the input as "very positive" based on its highest figure 1 : illustration of the inherently interpretable decision-making process of the proto-lm model for an example from the sst5 (5-class fine-grained sentiment analysis) dataset. the figure highlights the words identified by token-level attention, with a stronger shade indicating a higher attention score. proto-lm correctly predicts the input by relating it to prototypes of the "very positive" class. the decision process of proto-lm is transparent, and no separate interpretability method is required. similarity to prototypes from the "very positive" class and the input's distance (lack of similarity) to "negative" and "very negative" prototypes. a key advantage of our framework is its inherent interpretability. our prototypical network does not require an additional surrogate model for explanation, and we can observe the exact contribution of each prototype to the final output, resulting in faithful explanations. additionally, our framework offers interpretability at both the token and sample level, allowing for the identification of important words in the input text as well as the attribution of importance to impactful samples in the training data. in this work, we make the following contributions: ## proto-lm the architecture of our proposed framework, proto-lm, is shown in figure 2 . we utilize a pretrained llm as the underlying language model to encode the input and build a prototypical layer on top of it to provide interpretable prototypes. the goal of the learned prototypes is to capture features representative enough of each class so that they can be fine-tuned for the downstream classification task. in the classification process, similarities between the encoded input and the learned prototypes are fed through a fully-connected layer to produce logits. we formalize this below. ## performance experiments as we do not want interpretability to come at the cost of performance, we first conduct experiments to assess the modeling capability of proto-lm. we implement proto-lm using bert (base-uncased and large-uncased), roberta (base and large) and bart-large as the base llm encoders and train the token-level attention module, prototypical layer and classification head as described in §2. we evaluate the predictive accuracy of proto-lm on 5 tasks (sst-2, qnli, mnli, wnli, rte) from the glue dataset @xcite , imdb sentiment analysis @xcite and sst-5 @xcite . for baselines, we use the same fine-tuned versions of the llms with a classification head. we tune all hyperparameters using the respective validation data in each dataset. we present the mean performance as well as the standard deviation over 5 runs under their respective optimal configurations of hyperparameters (cf. app.a) and compare them to their baselines in ## evaluating the interpretability of proto-lm through the usage of its prototypical space, proto-lm is a white-box, inherently interpretable model. but just how well do the explanations provided by proto-lm satisfy common desiderata in interpretability? we conduct experiments to try to answer that question in this section. specifically, we evaluate the inherent interpretability provided by proto-lm via measures of faithfulness (deyoung et al., 2019; jacovi and goldberg, 2020) and simulatability @xcite . ## conclusion we introduce proto-lm, a white-box framework designed to offer inherent interpretability for language model learning (llms) through interpretable prototypes. these prototypes not only explain model decisions but also serve as feature extractors for downstream tasks. our experimental results indicate that proto-lm delivers competitive performance across a range of natural language processing (nlp) tasks and exhibits robustness under various hyperparameter settings. the interpretability of proto-lm is evaluated, and our findings show that proto-lm delivers faithful explanations that can effectively assist humans in understanding and predicting model decisions.