Knowledge Mechanisms in Large Language Models: A Survey and Perspective (2024)

Abstract: Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI. This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. Knowledge utilization delves into the mechanism of memorization, comprehension and application, and creation. Knowledge evolution focuses on the dynamic progression of knowledge within individual and group LLMs. Moreover, we discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address. We hope this work can help understand knowledge in LLMs and provide insights for future research.

## introduction knowledge is the cornerstone of intelligence and the continuation of civilization, furnishing us with foundational principles and guidance for navigating complex problems and emerging challenges @xcite . throughout the extensive history of evolution, we have dedicated our lives to cultivating more advanced intelligence by utilizing acquired knowledge and exploring the frontiers of unknown knowledge @xcite . as we know, large language models (llms) are renowned for encapsulating extensive parametric knowledge @xcite @xcite @xcite @xcite @xcite , achieving unprecedented progress in application. however, the knowledge mechanisms in llms for learning, storage, utilization, and evolution still remain mysterious @xcite figure 1 : the analysis framework of knowledge mechanism within neural models includes knowledge evolution and utilization. dark knowledge denotes knowledge unknown to human or model (machine). we investigate the mechanisms of knowledge utilization (right) in llms during a specific period of their evolution (left). the knowledge limitations identified through mechanisms analysis will inspire subsequent evolution (left). @xcite . extensive works aim to demystify various types of knowledge in llms through knowledge neurons @xcite and circuits @xcite @xcite , yet these efforts, scattered across various tasks, await comprehensive review and analysis. this paper pioneeringly reviews the mechanism across the whole knowledge life cycle (as shown in fig 1 ). we also propose a novel taxonomy for knowledge mechanisms in llms, as illustrated in fig 4 , which encompasses knowledge utilization at a specific time and knowledge evolution across all periods of llms 1 . specifically, we introduce preliminaries of this field ( §a) and review the knowledge utilization mechanism from a new perspective ( §2), delve into the fundamental principles for knowledge evolution ( §3). then, we investigate how to construct more efficient and trustworthy llms from the perspective of knowledge mechanism ( §e). later, we discuss open questions about the knowledge llms have and have not acquired ( §4). finally, we also provide some future directions ( §g) and tools for knowledge mechanism analysis ( §d). our contributions are as follows: comparison with existing surveys previous interpretability surveys typically aim to investigate various methods for explaining the roles of different components within llms from the global and local taxonomy @xcite @xcite @xcite @xcite . in contrast, this paper focuses on knowledge in llms. hence, our taxonomy, oriented from target knowledge in llms, reviews how knowledge is acquired, stored, utilized, and subsequently evolves. additionally, previous taxonomy mostly explore the explainability during the inference stage (a specific period), while ignoring knowledge acquisition during the pre-training stage and evolution during the post-training stage @xcite @xcite @xcite . our taxonomy aims to explore the dynamic evolution across all periods from naivety to sophistication in both individual and group llms. in contrast to the most similar survey @xcite ) that introduces knowledge life cycle, our work focuses on the underlying mechanisms at each stage. generally, this paper may help us to explore and manipulate advanced knowledge in llms, examine current limitations through the history of knowledge evolution, and inspire more efficient and trustworthy architecture and learning strategy for future models from knowledge mechanism perspective. note that most hypotheses in this paper are derived from transformer-based llms. we also validate the generalizability of these hypotheses across other architectural models and then propose universality intelligence in §c. ## knowledge utilization in llms knowledge is an awareness of facts, a form of familiarity, awareness, understanding, or acquaintance @xcite @xcite . it often involves the possession of information learned through experience and can be understood as a cognitive success or an epistemic contact with reality. we also introduce some preliminary information in §a, which includes the definition of knowledge in llms and knowledge analysis methods. then, inspried by bloom's taxonomy of cognition levels @xcite @xcite , we categorize knowledge representation and utilization within llms into three levels (as shown in fig 2 ): memorization, comprehension and application, and creation foot_1 . note that these mechanistic analyses are implemented via methods in §a.4. we further evaluate the applicability, advantages, and limitations of different methods in §b.1. ## knowledge evolution in llms knowledge in llms should evolve with changes in the external environment. therefore, we introduce the dynamic intelligence hypothesis for knowledge evolution in individuals and groups. hypothesis 5: dynamic intelligence conflict and integration coexist in the dynamic knowledge evolution of llms. ## discussion in this section, we discuss some open questions and seek to explore their essence and underlying principles. specifically, we discuss what knowledge llms have learned in §4.1, examine the fragility of the learned knowledge in application in §4.2, analyze the dark knowledge not yet learned by machines or humans in §4.3, and explore how llms can expand the boundaries of unknown knowledge from interdisciplinary perspectives §f. ## conclusion in this paper, we propose a novel knowledge mechanism analysis taxonomy and review knowledge evolution. we further explore the knowledge llms have learned, assess its fragility, and analyze unknown dark knowledge. as for future works, we discuss some promising directions, including parametric vs. non-parametric knowledge ( §g.1), embodied intelligence ( §g.2), and domain llms ( §g.3). we hope these insights may inspire some promising directions for future research and shed light on more powerful and trustworthy models.