Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models (2024)

Abstract: Recently, retrieval augmentation and tool augmentation have demonstrated a remarkable capability to expand the internal memory boundaries of language models (LMs) by providing external context. However, internal memory and external context inevitably clash, leading to knowledge conflicts within LMs. In this paper, we aim to interpret the mechanism of knowledge conflicts through the lens of information flow, and then mitigate conflicts by precise interventions at the pivotal point. We find there are some attention heads with opposite effects in the later layers, where memory heads can recall knowledge from internal memory, and context heads can retrieve knowledge from external context. Moreover, we reveal that the pivotal point at which knowledge conflicts emerge in LMs is the integration of inconsistent information flows by memory heads and context heads. Inspired by the insights, we propose a novel method called Pruning Head via PatH PatcHing (PH3), which can efficiently mitigate knowledge conflicts by pruning conflicting attention heads without updating model parameters. PH3 can flexibly control eight LMs to use internal memory ( \uparrow 44.0%) or external context ( \uparrow 38.5%). Moreover, PH3 can also improve the performance of LMs on open-domain QA tasks. We also conduct extensive experiments to demonstrate the cross-model, cross-relation, and cross-format generalization of our method. Our code is publicly available at https://github.com/jinzhuoran/MConflict/.

## introduction language models (lms) @xcite @xcite have memorized a substantial amount of factual knowledge during pre-training, and stored the knowledge within their parameters as internal memory (i.e., parametric knowledge) @xcite . during the inference phase, lms rely on their internal memory to understand and generate text. however, the internal memory may be limited or outdated, making lms prone to producing factually incorrect content. to alleviate the problem, one promising solution is to employ additional retrievers or tools to augment lms by providing external context (i.e., non-parametric knowledge). nevertheless, internal memory and external context can often contradict each other, which is known as knowledge conflicts @xcite @xcite . recent works have mainly investigated the behavior and preference of lms, attempting to determine whether these models are more inclined towards internal memory or external context when faced with knowledge conflicts. however, there is a limited understanding of the underlying mechanism of knowledge conflicts. insights into the mechanism will facilitate precise interventions at the pivotal point to mitigate knowledge conflicts, which can not only empower lms to more reliably adhere to internal memory (e.g., ignoring misleading external context) but also enhance faithfulness in generating text based on external context (e.g., correcting outdated memory). in this paper, we reveal that the pivotal point at which knowledge conflicts emerge in lms is the integration of inconsistent information flows by various attention heads in later layers. to investigate this, we consider a simple factual recall task (i.e., subject attribute prediction) inspired by the work of @xcite . as illustrated in figure 1 , given the question (i.e., "what is the capital of france?") and the conflicting external context (i.e., "the capital of france is rome."), the model can either use internal memory (i.e., "paris") or external context (i.e., "rome") to predict the subject's attribute. following this, we present a set of "topdown" analyses to locate the pivotal point where conflicts emerge and to identify the model components that are significant in knowledge conflicts, which primarily involves the following three steps: the capital of is rome . what is the capital of france ? france rome paris : yes the france rome paris context relation context subject context attribute question relation question subject ffn mha context head (1) (1) deep semantic shallow semantic (3) memory extraction (3) context extraction (2) (2) relation propagation paris rome subject propagation pivotal point of knowledge conflict memory head step 1: we start by answering the first question "what function do model components serve in knowledge conflicts?". we knock out the activations to examine the functionality of multi-head attention (mha) blocks and feed-forward network (ffn) blocks. we find that ffns enrich the semantic information of input elements in early layers, while mhas play an important role in passing information to the last token in later layers; step 2: based on this, the second question naturally arises, namely "when and where do mhas pass information to the last token?". we investigate the mhas by knocking out the attention weights from the last token to other input elements. results reveal that the question information is first propagated to the last token, and then the last token extracts attribute information from the subject and the attribute in the context; step 3: inspired by this, we aim to answer the final question "how do mhas extract attribute information under knowledge conflicts?". we find that some attention heads in late mhas play opposite roles, where memory heads can recall attributes from internal memory, and context heads can retrieve attributes from external context. according to our findings, the mechanism by which lms use both internal memory and external context can be summarized as three stages in figure 1: (1) enriching semantic information; inspired by our insights into knowledge conflicts, we propose a minimally-invasive control method called pruning head via path patching (ph3), which can efficiently mitigate knowledge conflicts by intervening on attention heads without updating model parameters. first, we use the path patching @xcite technique to localize important memory heads and context heads. our method can avoid the noise interference of other heads, enabling a more accurate calculation of the importance score for the target head. then, we perform structured pruning on those negative attention heads to mitigate conflicts. in this way, our method can flexibly control lms to use internal memory or external context. experimental results on the world capital dataset show that our method can not only reliably and consistently increase the average internal memory usage rate of eight lms by 44.0% (from 49.7% to 93.7%) but also increase the external context usage rate by 38.5% (from 50.3% to 88.8%). ph3 also enables lms to generate answers more faithfully according to retrieved passages in open-domain qa tasks. we conduct extensive experiments to demonstrate the cross-model (e.g., from gpt series to llama2 series), cross-relation (e.g., from world capital to official language), and cross-format (e.g., from triple format to document format) generalization. our contributions are summarized as follows: ## background in this work, we mainly focus on the autoregressive transformer-based language models. given a sequence of input tokens @xmath0, • • • , x n ], the lm g first embeds each token x i into a vector x 0 @xmath1, over a vocabulary v. the input embeddings are processed by l transformer layers. each layer consists of an mha and an ffn. formally, the hidden state x ℓ i of token x i at layer ℓ is calculated as: mha. a mha block consists of m attention heads, which are capable of aggregating global information from the hidden states @xcite ). an individual attention head h in layer ℓ consists of three learnable matrices, @xmath2. ffn. a ffn block can work as a key-value memory to store factual knowledge @xcite , enriching the hidden states of token i: ## interpreting knowledge conflicts we utilize a "top-down" analysis approach to locate the pivotal point where conflicts emerge and to identify the model components that are significant in knowledge conflicts. we start by examining the functionality of model components by knocking out activations, and reveal that mhas in the middle and late layers play a crucial role in passing information to the last token ( §4.1). then, we further investigate mhas by knocking out the attention weights. we find the question information is first passed to the last token, then the last token extracts information from the subject and the attribute in the context ( §4.2). last, we discover that some attention heads in later mhas play opposite roles in conflicts, where memory heads can recall knowledge from internal memory, and context heads can retrieve knowledge from external context ( §4.3). ## mitigating knowledge conflicts building on the above insights, we propose a novel method called pruning head via path patching (ph3) to efficiently mitigate knowledge conflicts by intervening on attention heads without the need to update model parameters ( §5.1). then, we conduct extensive experiments to show that our method can flexibly control lms to use internal memory or external context ( §5.2). moreover, we analyze the generalization capability of our method ( §5.3). ## conclusion in this paper, we perform an exploration into the mechanism of interpreting knowledge conflicts and reveal that memory and context heads in later layers can cause knowledge conflicts when merging inconsistent information flows. based on our insights, we propose a novel method called pruning head via path patching (ph3), which can mitigate knowledge conflicts by pruning those conflicting attention heads. we prove that ph3 can flexibly control lms to use internal memory or external context. we also demonstrate the cross-model, cross-relation, and cross-format generalization.