Improving LLM Attributions with Randomized Path-Integration (2024)

Abstract: We present Randomized Path-Integration (RPI) - a path-integration method for explaining language models via randomization of the integration path over the attention information in the model. RPI employs integration on internal attention scores and their gradients along a randomized path, which is dynamically established between a baseline representation and the attention scores of the model. The inherent randomness in the integration path originates from modeling the baseline representation as a randomly drawn tensor from a Gaussian diffusion process. As a consequence, RPI generates diverse baselines, yielding a set of candidate attribution maps. This set facilitates the selection of the most effective attribution map based on the specific metric at hand. We present an extensive evaluation, encompassing 11 explanation methods and 5 language models, including the Llama2 and Mistral models. Our results demonstrate that RPI outperforms latest state-of-the-art methods across 4 datasets and 5 evaluation metrics.

## introduction recent advancements in ai research have impacted numerous application domains, fueling innovation and progress in user modeling and personalization @xcite @xcite @xcite @xcite @xcite , natural language processing (nlp) @xcite @xcite @xcite @xcite @xcite , computer vision @xcite @xcite @xcite , and sound synthesis @xcite @xcite . within the field of nlp, the advent of transformers @xcite @xcite have ushered in a new era of complex language model (lm) architectures @xcite @xcite . these models, with their expanding size and complexity, have become integral components across diverse applications. this surge in model capacity and complexity has intensified the need for a profound understanding of the decision-making processes intrinsic to lms. however, the inherent complexity often shrouds the transparency of these models' predictions, prompting the development of explainability methods to unveil the contributing factors influencing their outputs. consequently, a multitude of explanation methods has been devised @xcite @xcite @xcite . simultaneously, with the evolution of explanation methods, there has been a proliferation of various explanation metrics designed for the quantitative assessment of explainability methods @xcite . however, achieving consensus within the literature on the ultimate explanation metric remains elusive, as each metric offers unique insights into different facets of explanation quality. acknowledging the diversity in explanation metrics, we introduce randomized path-integration (rpi), a method designed to elucidate predictions made by lms through the integration of internal attention scores and their gradients along a random path. the introduction of randomness into the integration path stems from the representation of the baseline as a random tensor drawn from a specified baseline distribution. as a result, rpi produces a variety of baselines from this distribution, forming a pool of candidate attribution maps. the versatility afforded by multiple baselines facilitates the selection of the most effective attribution map, contingent upon the evaluation metric under consideration. through extensive evaluation spanning a variety of explanations methods, lms, and datasets, rpi exhibits superior performance compared to current state-of-the-art methods across multiple explanation metrics. ## related work explainable ai includes a wide array of methods, all aimed at improving the understanding of decisions made by deep learning models across multiple application domains @xcite @xcite @xcite @xcite @xcite @xcite @xcite . perturbation-based techniques such as lime @xcite and shap (lundberg and lee, 2017) introduce perturbations to individual inputs or neurons and observe their consequential effects on subsequent neurons in the network. relevance-decomposition methods look at the network's representation as vectors, each exerting distinct influences on the model's predictions. globenc @xcite and alti @xcite incorporate local-level decomposition, aggregating resulting vector norms using rollout @xcite to construct global-level explanations. recently, de-compx @xcite was introduced, providing state-of-the-art results by constructing decomposed token representations and sequentially propagating them through the model without interlayer mixing. gradient-based methods rely on the gradients of the model's prediction score concerning the input tokens. basic gradient methods, such as vanilla gradients @xcite and gradien-txinput @xcite , operate on the principle of gradient computation. integrated gradients (ig) @xcite is a path-integration method, computing gradients on interpolated points along a straight line between the data and an uninformative baseline. additional approaches such as deeplift @xcite and gradientshap (lundberg and lee, 2017) can be viewed as approximations of ig. other pathintegration methods, such as discretized integrated gradient (dig) @xcite , treat the integration path differently. dig replaces the continuous straight path with a discretized one, where interpolated points are words. another innovative path-integration approach comes in the form of sequential integrated gradients (sig) @xcite . sig addresses concerns about altered sentence meaning by computing the importance of each word while keeping other words fixed and creating interpolations between the baseline and the word of interest. notably, sig has demonstrated superior performance when compared to several existing methods, including dig, ig, and gradi-entshap. rpi complements this line of path-integration methods by modeling the baseline representation as a random tensor sampled from a gaussian diffusion process @xcite , and integrating the attention scores and their gradients along a randomized path between a randomly drawn baseline and the attention scores to form an attribution map. by baseline resampling from the baseline distribution, rpi establishes a pool of candidate attribution maps that facilitates the selection of the most effective attribution map based on the metric under consideration. our evaluation indicates that this unique feature of rpi leads to state-of-the-art results and can be further incorporated to existing path-integration to enhance their performance. ## randomized path-integration let v denote the vocabulary, and let @xmath0, where each u @xmath1. we further define x @xmath2, ..., x u k ] as a 2d tensor in r d×k accommodating k embeddings, with each x u @xmath3. in the context of this work, our emphasis lies in the domain of classification tasks. hence, we introduce a model denoted as f , designed to take an input x u and yield an output f (x u ) ∈ [0, 1] c . here, f i (x u ) denotes the probability assigned to class i, and c signifies the total number of available classes. the model f can assume the form of a lm utilizing a classification head, producing probabilities for each class within the specific task. this process involves either finetuning the lm or utilizing it as a foundational component for the transfer learning of a specific task. recent lm architectures, often referred to as large language models (llms), are predominantly decoder-based, tasked with completing sequences of tokens in the output @xcite . although these llms can be finetuned using classification heads, a more prevalent approach is classification via completion @xcite . in this method, the llm is prompted with the specific task, either in a few-shot or zero-shot scenario, and instructed to generate a token that signifies the correct class. for instance, in sentiment analysis, if the relevant classes are communicated to the llm (via prompt) as 'positive' and 'negative', logit scores for the tokens 'positive' and 'negative' are computed, and softmax is applied to obtain the probabilities associated with each class. it is noteworthy that this approach can be implemented either before or after finetuning the llm on the relevant dataset in a completion manner. in this work, we conduct experiments in various settings, including finetuning with classification heads on top of the llm as well as classification via completion (decoder-based models). ## experimental setup and results all experiments were conducted on a highperformance nvidia dgx server, equipped with 8 a100 gpus, utilizing the pytorch platform. ## conclusion this work responds to the diversity inherent in explanation metrics, recognizing that different metrics may promote different attribution maps. rpi effectively addresses this challenge by introducing randomness into the integration path through random baseline sampling, thereby generating a pool of candidate attribution maps. the adaptability provided by multiple baselines enables rpi to select the most effective attribution map tailored to the specific evaluation metric. through an extensive evaluation encompassing 11 explanation methods, 5 language models, and 4 datasets, our work establishes the superiority of rpi over current state-of-the-art methods across a diverse range of explanation metrics. these findings highlight the effectiveness of rpi as a machinery for explaining lms.