I v RA : A Framework to Enhance Attention-Based Explanations for Language Models with Interpretability-Driven Training (2024)

Abstract: Attention has long served as a foundational technique for generating explanations. With the recent developments made in Explainable AI (XAI), the multi-faceted nature of interpretability has become more apparent. Can attention, as an explanation method, be adapted to meet the diverse needs that our expanded understanding of interpretability demands? In this work, we aim to address this question by introducing IvRA, a framework designed to directly train a language model’s attention distribution through regularization to produce attribution explanations that align with interpretability criteria such as simulatability, faithfulness, and consistency. Our extensive experimental analysis demonstrates that IvRA outperforms existing methods in guiding language models to generate explanations that are simulatable, faithful, and consistent, in tandem with their predictions. Furthermore, we perform ablation studies to verify the robustness of IvRA across various experimental settings and to shed light on the interactions among different interpretability criteria.

## introduction the rapid adoption of language models @xcite @xcite in recent years has sparked an escalating interest in enhancing model interpretability. this has given rise to the burgeoning field of explainable ai (xai), which has devised various methods to increase model interpretability @xcite @xcite . however, an universal definition for the term "interpretability" remains elusive in the research community @xcite . interpretability assessment has primarily leaned on criteria tailored for different purposes that fall under the broad umbrella of the term "interpretability". some of the most popular criteria are simulatability @xcite , faithfulness @xcite , and consistency @xcite . simulatability measures whether a model's behavior is comprehensible enough for a human or another ml model to predict its outputs on unseen data, aligning with the objective of conveying the model's underlying mechanics to humans. faithfulness measures the extent to which an explanation reflects the actual decision-making process of the model. consistency assesses the explanation method's stability across varying input data, favoring explanations that remain similar for similar inputs and reflect input changes that lead to inconsistent outputs. the utility of attention for generating saliency explanations is widely recognized @xcite @xcite , notwithstanding initial doubts regarding the faithfulness and consistency of attention mechanisms @xcite . past works @xcite that have benchmarked existing attention-based text attribution methods along interpretability criteria such as simulatability, faithfulness and consistency do not explore the possibility of directly training attention distributions to become more interpretable with regard to a criterion. on the other hand, works that do train their explanations to become more interpretable via some criterion either only focus on a small subset of criteria @xcite @xcite and/or do not use attention as a technique @xcite , instead relying on a separate model as rationale extractor. in this work, we focus on developing an attentionbased explanation framework that enables a language model (lm) to produce explanations that align more closely with interpretability criteria. we summarize our contribution below: this paper introduces a novel framework-interpretability via regularized attention i and r * i denoting the output logit and explanations of s, respectively. for comprehensiveness and sufficiency ( §2.2.2), t k i denotes tokens recognized by ri with attention scores in the top k% of tokens. for consistency ( §2.2.3), xj represents another example within the dataset. ## experiments in order to evaluate our framework's effectiveness at producing simulatable, faithful and consistent explanations, we train three transformer-based language models with ivra: electra (clark, 2020), llama-2-7b @xcite , and gpt-2 (medium) @xcite ) on three nlp tasks: sentiment classification, entailment inference and question-answering, with the following datasets, respectively: imdb @xcite , snli @xcite , and squad @xcite . in the main paper, we present results of ivra using electra as the base language model, with further results using llama-2 and gpt-2 provided in i. in §3.1, 3.2, 3.3 we report results obtained when training models for each of the interpretability criteria separately. we then delve into mixed-criteria training in §3.4 and examine the ivra's effect on downstream accuracy in §3.5. in order to assess the relative effectiveness of ivra compared to other explanation methods, we conduct experiments using other methods on the same datasets and compare the extent to which each interpretability objective is achieved. we report the mean and standard error values from 5 runs for each experiment setting. the explanation methods that were employed in our experiments are: ## conclusion we introduce ivra, a paramerterized attention module for directly training a lm's attention distribution to produce explanations that align with interpretability criteria. we test ivra's effectiveness at producing explanations that are simulatable, faithful (comprehensive and sufficient) and consistent using multiple lms and on multiple nlp tasks. we perform ablation experiments to reveal insights on the interplay between different interpretability criteria and to assess ivra's influence on downstream accuracy. our findings demonstrate that ivra's attention-based explanations is robust under various settings and empowers lms to generate explanations that better align with interpretability criteria.