Explaining Language Model Predictions with High-Impact Concepts (2024)

Abstract: To encourage fairness and transparency, there exists an urgent demand for deriving reliable explanations for large language models (LLMs). One promising solution is concept-based explanations, i.e., human-understandable concepts from internal representations. However, due to the compositional nature of languages, current methods mostly discover correlational explanations instead of causal features. Therefore, we propose a novel framework to provide impact-aware explanations for users to understand the LLM’s behavior, which are robust to feature changes and influential to the model’s predictions. Specifically, we extract predictive high-level features (concepts) from the model’s hidden layer activations. Then, we innovatively optimize for features whose existence causes the output predictions to change substantially. Extensive experiments on real and synthetic tasks demonstrate that our method achieves superior results on predictive impact, explainability, and faithfulness compared to the baselines, especially for LLMs.

## introduction over the past few years, large language models (llms) have achieved tremendous progress, leading them to be widely applied in sensitive applications such as personalized recommendation bots and recruitment. however, explainable ai (xai) has not witnessed the same progress, making it difficult to understand llms' opaque decision processes @xcite . therefore, many users are still reluctant to adopt llms in high-stake applications due to transparency and privacy concerns. in this work, we aim to increase user trust and encourage transparency by deriving explanations that allow humans to better predict the model outcomes. to understand what happens inside an llm, previous studies @xcite show that dense vector representations in high layers of a language model tend to capture semantic meanings that are useful for solving the underlying task. however, such vector representations are not understandable to humans. to solve it, concept-based explanations attempt to map the hidden activation space to human-understandable features. for example, @xcite provides the concept bottleneck model, which first predicts an intermediate set of human-specific concepts, then uses them to predict the target. as illustrated by purple boxes in fig. 1 , for the movie review classification task, conceptbased explanations are semantically meaningful word clusters @xcite corresponding to abstract features such as "acting" and "directing". however, existing concept-based methods do not consider of the explanation impact on output predictions, leading to inferior explanations. by impact, we mean the causal effect of removing a feature on output predictions @xcite . as @xcite points out, these non-impact-aware methods derive correlational explanations that cannot answer questions about decision-making under alternative situations and are thus unreliable. an example is illustrated in fig. 1 . due to the conventional expression "hot mess", the word "hot" often co-occurs with "mess", which is usually used to classify nega-tive sentiment. traditional concept-based methods that do not consider impact may falsely use the correlational feature "weather" (i.e., "hot") to explain why the model classifies something as negative. however, excluding the "weather" concept does not cause the output prediction to change at all, resulting in zero impact (red line). thus, low-impact explanations such as "weather" are less valid as users cannot utilize them to consistently predict the model's behaviors when a feature changes. to tackle this bottleneck and incorporate impact into traditional concept-based models, in this work, we propose high-impact concepts (hi-concept), a complete concept explanation framework with causal impact optimization ( §3.2). specifically, we design a causal loss objective, stemming from the treatment effects in the causality literature @xcite . moreover, previous causality evaluations @xcite primarily focused on assessing the causal effect via local (i.e., instance-level) change and removal intervention (i.e., eliminating words/concepts from the source), leading to potentially biased evaluation results. to this end, we further propose a novel global (i.e., model-level) accuracy change metric and insertion operation to effectively diagnose the causality measurement ( §3.4). as a result, our method can consistently prioritize more influential features (green line in fig. 1 ) while disregarding correlational ones. extensive experiments with multiple language models, both established and newly proposed evaluation metrics, and rigorous human studies fully validate the effectiveness of hi-concept in finding high-impact concepts compared to baselines, especially for llms. our contributions are summarized as follows 1 : • to alleviate the problem of correlational explanations, we propose hi-concept, a framework for deriving explanatory features with high impacts by innovatively optimizing a causal objective. ## preliminaries we first introduce what concept-based explanations are, what properties they should satisfy, and our key baseline, concept bottleneck models. ## methodology then, we propose hi-concept, which aims to fill the current research gap on explanatory impact.