Using Captum to Explain Generative Language Models (2023)

Abstract: Captum is a comprehensive library for model explainability in PyTorch, offering a range of methods from the interpretability literature to enhance users’ understanding of PyTorch models. In this paper, we introduce new features in Captum that are specifically designed to analyze the behavior of generative language models. We provide an overview of the available functionalities and example applications of their potential for understanding learned associations within generative language models.

## introduction model interpretability and explainability have become significantly more important as machine learning models are used in critical domains such as healthcare and law. it is insufficient to simply make a prediction through a black-box model and important to better understand why the model made a particular decision. interest in large language models (llms) has also grown exponentially in the past few years with the release of increasingly large and more powerful models such as @xcite . a lack of explainability continues to exist despite larger models, and with the use of these models expanding to more and more use-cases, it is increasingly important to have access to tooling providing model explanations. captum is an open-source model explainability library for pytorch providing a variety of generic interpretability methods proposed in recent literature such as integrated gradients, lime, deeplift, tracin, tcav and more @xcite . in this work, we discuss newly open-sourced functionalities in captum v0.7 to easily apply explainability methods to large generative language models, such as gpt-3. ## attribution methods one important class of explainability methods is attribution or feature importance methods, which output a score corresponding to each input feature's contribution or impact to a model's final output. formally, given a function f : r @xmath0, where @xmath1, an attribution method is defined as a function ϕ : f × r @xmath2. each element in the attribution output corresponds to a score of the contribution of corresponding feature @xmath3, where d denotes the set of all feature indices @xmath4, 2, ..., d}. many attribution methods also require a baseline or reference input @xmath5. we utilize the notation x s to denote selecting the feature values with indices from the set s ⊆ d and the remaining indices from b. formally, the value of feature i in x s is (x s ) @xmath6, where i is the indicator function. in this section, we provide background context on attribution methods available in captum. these methods can be categorized broadly into (i) perturbation-based methods, which utilize repeated evaluations of a black-box model on perturbed inputs to estimate attribution scores, and (ii) gradient-based methods, which utilize backpropagated gradient information to estimate attribution scores. perturbation-based methods do not require access to model weights, while gradientbased models do. ## language model attribution in captum v0. ## applications in this section, we discuss two applications of the attribution methods described above in different contexts. these applications provide additional transparency as well as contribute to a better understanding of a model's learned associations and robustness. ## conclusion in this work, we introduce new features in the open source library captum that are specifically designed to analyze the behavior of generative llms. we provide an overview of the available functionalities and example applications of their potential in understanding learned associations and evaluating effectiveness of few-shot prompts within generative llms. we demonstrate examples for using perturbation and gradient-based attribution methods with captum which highlight the api's flexibility on aspects such as feature definition, baseline choice and masking. this flexibility in structure allows users to generalize to a variety of cases, simplifying their ability to conduct explainability experiments on generative llms. in the future, we plan to expand our api for additional automation in baseline and feature selection as well as incorporate other categories of interpretability techniques for language models. runtime performance optimization of attribution algorithms is another area of research that could be beneficial for the oss community.