Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons (2024)

Abstract: In this paper, we investigate whether Large Language Models (LLMs) actively recall or retrieve their internal repositories of factual knowledge when faced with reasoning tasks. Through an analysis of LLMs’ internal factual recall at each reasoning step via Knowledge Neurons, we reveal that LLMs fail to harness the critical factual associations under certain circumstances. Instead, they tend to opt for alternative, shortcut-like pathways to answer reasoning questions. By manually manipulating the recall process of parametric knowledge in LLMs, we demonstrate that enhancing this recall process directly improves reasoning performance whereas suppressing it leads to notable degradation. Furthermore, we assess the effect of Chain-of-Thought (CoT) prompting, a powerful technique for addressing complex reasoning tasks. Our findings indicate that CoT can intensify the recall of factual knowledge by encouraging LLMs to engage in orderly and reliable reasoning. Furthermore, we explored how contextual conflicts affect the retrieval of facts during the reasoning process to gain a comprehensive understanding of the factual recall behaviors of LLMs. Code and data will be available soon.

## introduction recent advancements in large language models have underscored their exceptional reasoning prowess with natural language understanding across a broad spectrum of tasks @xcite @xcite . however, amidst these achievements, a specific form of reasoning has been somewhat overlooked and insufficiently investigated: reasoning tasks that require the utilization of internal factual knowledge associations. for instance, when presented with a 2-hop question such as "who is the chairperson of the manufacturer of the holden caprice?" in figure 1 , llms must first identify that the manufacturer of the holden caprice is general motors, and subsequently retrieve the name of general motors' chairperson from their internal knowledge, also referred to as parametric knowledge @xcite . previous work has shown factual knowledge emerges in both gpt @xcite and bert models @xcite . unlike mathematical @xcite and logical reasoning @xcite , factual reasoning heavily relies on the factual knowledge encoded within llms, acquired through extensive pretraining on vast corpora, rather than on userinputted premises. at the same time, it differs from commonsense reasoning @xcite , which taps into general knowledge acquired through dynamic training to foster a holistic understanding of the world, instead of emphasizing specific factual information. intuitively, it is reasonable to expect llms to harness extensive parametric knowledge to tackle reasoning tasks. yet, an important question emerges: how effectively can llms actually retrieve and utilize their internal knowledge for reasoning purposes? delving into this question is crucial for several reasons. first, efficient use of parametric knowledge may significantly reduce reliance on external data sources, thereby lowering operational costs of data retrieval and api usage. second, this dynamic capability allows the knowledge within llms to flow and interconnect @xcite , showcasing these models as organic entities rather than static information repositories @xcite . from a practical perspective, the accurate retrieval and application of parametric knowledge lead to more reliable and interpretable reasoning, enhancing their utility and trustworthiness in real-world applications. transformer-based language models have accumulated substantial knowledge through extensive pretraining @xcite . a significant body of recent research has focused on the factuality issues of llms @xcite . one stream of this research has concentrated on pinpointing the locations within these models' architectures where factual knowledge is stored and encoded @xcite @xcite @xcite . simultaneously, there has been a concerted effort to understand the mechanism by which this knowledge is accessed during the inference phase @xcite . another line of work discusses the balance of the retrieved knowledge and its parametric counterparts @xcite @xcite . however, the majority of these studies have either been confined to elementary retrieval tasks, such as recalling a single fact object o from a given triplet (s, r, o), or have not delved into the intricacies of factual knowledge recall and utilization in more advanced challenges, particularly within complex reasoning scenarios. our work addresses these limitations by examining the inner dynamics of factual recall within llms during the two-hop factual reasoning process, providing fresh insights into the behavior of factual recall in reasoning and highlighting avenues for enhancing the robustness and reliability of reasoning through more sophisticated knowledge utilization strategies. in this work, we investigate the harness of internal knowledge for reasoning through the lens of knowledge neurons (kns). we focus on the basic setting of factual reasoning involving the composition of two facts (for example, "who is the chairperson of the manufacturer of holden caprice?" in figure 1 ). to achieve this, we carefully craft twohop reasoning questions dataset that seamlessly integrates with the kn technique. we assess the level of factual recall at each reasoning step by introducing a novel metric, kn scores. we examine kn scores under three conditions of two-hop reasoning: no cot, zero-shot cot, and few-shot cot, unveiling the pitfalls existing in the reasoning process and the enhancement effect of cot @xcite . then we conduct targeted interventions on kns to enhance or suppress the factual retrieval process, finding the contributing impact on reasoning performance. furthermore, we provide a detailed analysis of factual shortcuts @xcite @xcite , potentially caused by redundant information stored in models' parameters within llms used for reasoning. finally, we explore how the presence of knowledge conflict outside llms influences the factual recall process. our findings can be summarized as follows: ## tfrkn: two-hop factual reasoning for knowledge neurons to investigate the behavior of factual recall in reasoning tasks for llms, we have developed a specialized dataset for knowledge neurons called twohop factual reasoning for knowledge neurons, tfrkn. dataset construction our dataset consists of two-hop factual questions, where each question involves two facts that are connected by an intermediate entity. llms are more likely to recall triplets related to popular entities @xcite . therefore, for entity selection, we use the cumulative pageview count over the past 12 months as a metric and select the top 500 popular entities from wikidata @xcite based on this criterion. two-hop fact triplets are then extracted from sub-graphs consisting solely of a set of manually selected relations and entities. to identify kns for each-hop fact, we reformulate each fact triplet into more than five varied natural questions using chatgpt (appendix a). the tfrkn dataset encompasses 4,550 distinct instances covering 213 unique relational combinations with a sample instance shown in ## diagnose the pitfalls of factual recall in reasoning in the realm of two-hop factual reasoning, an optimal and dependable reasoning trajectory is a multihop reasoning approach @xcite . this process requires identifying the bridge entity first and then using it to solve the second hop question, necessitating that llms recall the relevant fact at each hop step by step, culminating in the formulation of the correct answers. in this section, we investigate whether llms faithfully retrieve factual knowledge at each hop when undertaking reasoning tasks. the vertical axis shows the depth of layers, while the horizontal axis shows the neuron index in the ffn's intermediate layers. it is evident that kns are distributed in the middle and final layers. ## analysis of shortcuts in this section, we investigate whether successful two-hop reasoning implies the successful recall of factual knowledge. in other words, we examine whether accurate reasoning outcomes stem from a thorough grounding in multi-hop knowledge reasoning or are facilitated by alternative shortcuts. 6.1 experiment setup we investigate the utilization of individual fact triplets in correctly answered two-hop questions by analyzing the kn scores for each triplet. we compare these scores with those observed during single-hop queries to establish a threshold, denoted as τ , which serves as a benchmark for identifying the effective use of facts in the reasoning process. if the activation level of kns falls significantly below this threshold in comparison to singlehop queries, this indicates an under-utilization of the corresponding fact. conversely, if it exceeds the threshold, the fact is considered adequately utilized. using this criterion, we classified the correctly answered questions into four distinct categories: (1) ft: unsuccessful recall of the first-hop fact but successful second-hop recall; (2) tf: successful first-hop recall but unsuccessful second-hop recall; ## impact of contextual conflict the capacity of utilizing internal factual knowledge is contingent not solely upon the intrinsic properties of llms, but is also significantly influenced by the context within which they operate. this section elucidates how the presence of knowledge conflicts within a given context can impact the mechanisms of the retrieval process during reasoning. ## related work multi-hop reasoning multi-hop reasoning poses a significant challenge for llms. several studies have endeavored to address this challenge through the development of more faithful reasoning techniques @xcite @xcite . one such approach is cot, which stimulates llms to produce deductive intermediate steps, fostering a step-by-step analytical process @xcite . another line of research is focused on visualizing the implicit logical structures within llms from the perspective of mechanistic interpretability @xcite . for example, a recent study by @xcite recovers the reasoning tree from models's attention patterns using mechanisticprobe. cot mechanism a large body of literature is dedicated to the theoretical and empirical exploration of the mechanism underlying cot @xcite @xcite @xcite . some research endeavors to delve into a reverse-engineering analysis of cot prompting, uncovering the intricate information pathways that facilitate the generation of responses @xcite . however, the majority of these studies concentrate on the rationales produced by cot and have largely overlooked the broader implications for factual retrieval processes. in our current work, we complement this aspect and present compelling evidence that cot significantly bolsters the internal recall of factual information. ## conclusions this paper aims to provide a comprehensive understanding of factual recall behaviors for llms. we find that a considerable portion of reasoning failures are due to retrieval failures. manually enhancing the internal recall within llms can improve reasoning performance. for llms, they not only rely on multi-hop reasoning but also rely on other inference ways in llms such as shortcuts. cot can significantly stimulate llms to recall more facts by compelling models to engage in step-bystep thinking, diminishing the possibilities of taking shortcuts. the knowledge conflict existing in context could improve the confidence of parametric knowledge, therefore enhancing the internal recall.