Variational Language Concepts for Interpreting Foundation Language Models (2024)

Abstract: Foundation Language Models (FLMs) such as BERT and its variants have achieved remarkable success in natural language processing. To date, the interpretability of FLMs has primarily relied on the attention weights in their self-attention layers. However, these attention weights only provide word-level interpretations, failing to capture higher-level structures, and are therefore lacking in readability and intuitiveness. To address this challenge, we first provide a formal definition of *conceptual interpretation* and then propose a variational Bayesian framework, dubbed VAriational Language Concept (VALC), to go beyond word-level interpretations and provide concept-level interpretations. Our theoretical analysis shows that our VALC finds the optimal language concepts to interpret FLM predictions. Empirical results on several real-world datasets show that our method can successfully provide conceptual interpretation for FLMs.

## introduction foundation language models (flms) such as bert @xcite and its variants @xcite @xcite have achieved remarkable success in natural language processing. these flms are usually large attention-based neural networks that follow a pretrain-finetune paradigm, where models are first pretrained on large datasets and then finetuned for a specific task. as with any machine learning models, interpretability in flms has always been a desideratum, especially in decision-critical applications (e.g., healthcare). to date, flms' interpretability has primarily relied on the attention weights in self-attention layers. however, these attention weights only provide document-level dataset-level word-level sentence 1: he playfully chided the senate's little bitty tax relief plan. sentence 2: we don't need a little bitty tax relief plan. concept 24: politics secretary congress capital gov concept 20: government speaker seat delegation suspect concept 27: names scott shelley mary task: decide whether sentence 1 paraphrases sentence 2. label: true prediction: true a document consists of two sentences. the task is to decide whether 'sentence 1' paraphrases 'sentence 2'. left: dataset-level concepts for mrpc dataset with 3 concepts and their nearest word embeddings. middle: document-level concept strength, showing that this document is mostly related to concept 20 and concept 24. right: word-level concepts, where the flm correctly predicts the label to be 'true', and valc interprets that this is because the both sentences consist of words with concept 24, i.e., politics. in this paper, we aim to go beyond word-level attention and interpret flm predictions at the concept level. such higher-level semantic interpretations are complementary to word-level importance scores and often more readable and intuitive. for example, as shown in fig. 1 , valc interprets the flm with the following multi-level concepts (details in appendix f): drawing inspiration from hierarchical bayesian deep learning @xcite @xcite , the core of our idea is to treat a flm's contextual word embeddings (and their corresponding attention weights) as observed variables and build a probabilistic generative model to automatically infer the higher-level semantic structures (e.g., concepts or topics) from these embeddings and attention weights, thereby interpreting the flm's predictions at the concept level. our valc is compatible with any attention-based flms and can work as an conceptual interpreter, which explains the flm predictions at multiple levels with theoretical guarantees. our contributions are as follows: usually they are first pretrained on large datasets in a self-supervised manner and then finetuned for a specific downstream task. bert @xcite is a pioneering flm that has shown impressive performance across multiple downstream tasks. following bert, there have been variants @xcite @xcite @xcite that design different self-supervised learning objectives or training schemes to achieve better performance. while flms offer attention weights for interpreting predictions at the word level, these interpretations lack readability and intuitiveness because they fail to capture higher-level semantic structures. interpretation methods for flms. existing conceptual interpretation methods for flms typically rely on topic models @xcite @xcite @xcite @xcite and prototypical part networks @xcite . there has been recent work that employs deep neural networks to learn topic models more efficiently @xcite @xcite , using techniques such as amortized variational inference. there is also work that improves upon traditional topic models by either leveraging word similarity as a regularizer for topic-word distributions @xcite or including word embeddings into the generative process @xcite @xcite . there is also work that builds topic models upon embeddings from flms @xcite @xcite @xcite . however, these methods often rely on a pipeline involving dimensionality reduction and basic clustering, which is not end-to-end, leading to potential information loss between flm embeddings and clustering outcomes. this can result in unfaithful interpretations for the underlying flm. additionally, they typically generate interpretations at a single level (e.g., document level), lacking a multi-level conceptual structure. beyond topic models, attribution-based approaches such as lime @xcite and shap (lundberg and lee, 2017) assign importance to input features to explain predictions. concept bottleneck models (cbms) @xcite @xcite @xcite @xcite offer interpretations by learning conceptual activation and then performing classifications on these concepts, while inherent models @xcite @xcite focus on model redesign/re-training for interpretability. however, these approaches often require extra supervision or re-training, making them unsuitable for our setting. in contrast, our method is inherently multi-level and end-to-end, models concepts across dataset, document, and word levels, and produces faithful post-hoc interpretations for any models based on flms with theoretical guarantees. ## methods in this section, we formalize the definition of conceptual interpretation, and describe our proposed valc for conceptual interpretation of flms. ## theoretical analysis in this section, we provide theoretical guarantees of valc on the four properties in definition 3.1. multi-level structure. as shown in alg. 1, valc (1) learns the dataset-level interpretation {ω k } k @xmath0, (2) infers the distribution of document-level interpretation θ m for document m, i.e., q(θ m |γ m ) (parameterized by γ m ), and (3) infers the posterior distribution of word-level concept index, i.e., q(z mj |ϕ mj ), parameterized by ϕ mj . such three-level interpretations correspond to property (1) in definition 3.1. normalization. the learned variational distribution q(θ m |γ m ) (described in eq. 1) is a dirichlet distribution; therefore we have k @xmath1. the update of ϕ mj (eq. 5) is naturally constrained by k @xmath2. additivity. valc is able to perform concept editing, i.e, add/subtract the learned concept activation µ k from flms via the following quadratic programming (qp) problem (@xmath3: given learned concepts {(µ k , σ k )} k @xmath4, valc obtains this qp's optimal solution x * ∈ r k and add/subtract any concept k from arbitrary flm embedding e m by: e m ← e @xmath5. alg. 2 summarizes this concept editing process; one can also replace e mj with the cls embedding c m for document-level editing (details in appendix d). ## conclusion we address the challenge of multi-level interpretations for flm predictions by defining conceptual interpretation and introducing valc, the first method to infer such interpretations effectively. empirical results are promising, and theoretical analysis confirms that valc reliably produces optimal conceptual interpretations by our definition.