P rompt E xplainer: Explaining Language Models through Prompt-based Learning (2024)

Abstract: Pretrained language models have become workhorses for various natural language processing (NLP) tasks, sparking a growing demand for enhanced interpretability and transparency. However, prevailing explanation methods, such as attention-based and gradient-based strategies, largely rely on linear approximations, potentially causing inaccuracies such as accentuating irrelevant input tokens. To mitigate the issue, we develop PromptExplainer, a novel method for explaining language models through prompt-based learning. PromptExplainer aligns the explanation process with the masked language modeling (MLM) task of pretrained language models and leverages the prompt-based learning framework for explanation generation. It disentangles token representations into the explainable embedding space using the MLM head and extracts discriminative features with a verbalizer to generate class-dependent explanations. Extensive experiments demonstrate that PromptExplainer significantly outperforms state-of-the-art explanation methods.

## introduction recently, pretrained language models @xcite @xcite have achieved remarkable success across a wide range of nlp tasks, such as text classification, question answering and machine translation. however, the inherent complexity of these models, often characterized by billions of parameters @xcite and high nonlinearities, makes these models notably opaque and their predictions elusive to users @xcite . explaining language models is receiving significant attention due to the growing demand for facilitating accountability, transparency, trustworthiness, bias detection and ethical considerations @xcite @xcite . ## related work existing approaches to explaining language models can be classified into attention-based, gradientbased, and perturbation-based methods. the generated explanations fall into either the classdependent category (specific to each class label) or the class-agnostic (only based on the input and model) category. in attention-based methods, utilizing vanilla attention weights in attention modules to interpret model decisions @xcite is a straightforward approach. however, this method's reliability and effectiveness diminish when applied to transformer architectures @xcite , commonly used in language models @xcite @xcite . to capture transformers' intricate nonlinearities, attention rollout (abnar and zuidema, 2020) linearly combines attention weights across layers. additionally, attention flow @xcite views attention propagation as a max-flow problem in the pairwise attention graph. typically, attention-based explanations are considered to be class-agnostic. gradient-based methods employ backpropagation gradients to determine the significance of each token. the integrated gradient @xcite and input gradients @xcite have been proven effective in various models and domains. another approach, termed as generic attention explainability (gae) @xcite , integrates attention gradients along with gradients from other network components. it is worth noting that layer-wise relevance propagation (lrp) @xcite has also been used to measure the relative significance of each token @xcite . @xcite discovers that lrp could encounter difficulties in identifying the input feature contributions in transformers due to the intricate atten-tionheads and layernorm. to address the problem, they modify the current propagation rule to adhere to the conservation rule, which mandates that scores assigned to input variables and forming the explanation must sum up to the network's output. lrp-xai is the sota in delivering the most effective class-dependent explanations. a few perturbation-based methods have been proposed, which utilize the input reductions @xcite to determine the most relevant parts of the input by observing changes in model confidence or shapley values (lundberg and lee, 2017). contrastive explanations @xcite @xcite , which focus on identifying the causal factors influencing a model's output choice between two alternatives, have emerged in the last two years. it is a different task so we do not compare the contrastive methods to our proposed approach. ## conclusion in this paper, we present promptexplainer, a method for explaining language models through prompt-based learning. our approach aligns the interpreting process with the mlm objective and leverages the mlm head to disentangle token representations, creating an explainable feature space. we then utilize the verbalizer to extract discriminative features to generate explanations. extensive experiments demonstrate the superior performance of promptexplainer. in future work, we intend to extend the core concept of promptexplainer, which involves leveraging the lm head to provide explanations for model decisions, to llms such as gtpx @xcite .