Time Course M ech I nterp: Analyzing the Evolution of Components and Knowledge in Large Language Models (2025)

Abstract: Understanding how large language models (LLMs) acquire and store factual knowledge is crucial for enhancing their interpretability, reliability, and efficiency. In this work, we analyze the evolution of factual knowledge representation in the OLMo-7B model by tracking the roles of its Attention Heads and Feed Forward Networks (FFNs) over training. We classify these components into four roles—general, entity, relation-answer, and fact-answer specific—and examine their stability and transitions. Our results show that LLMs initially depend on broad, general-purpose components, which later specialize as training progresses. Once the model reliably predicts answers, some components are repurposed, suggesting an adaptive learning process. Notably, answer-specific attention heads display the highest turnover, whereas FFNs remain stable, continually refining stored knowledge. These insights offer a mechanistic view of knowledge formation in LLMs and have implications for model pruning, optimization, and transparency.

## introduction large language models (llms) are trained on vast datasets, including sources like wikipedia, from which they acquire a wide range of factual knowledge. as a result, these models can provide informed answers when queried about facts. to uncover the mechanisms that enable such factual responses, mechanistic interpretability (mi) methods @xcite are employed. mi aims to reverse-engineer neural networks by translating their internal processes into human-understandable algorithms and concepts, and has made great progress in explaining how 1 https://github.com/adhakimi/time-course-mechinterp transformer-based llms process and store information. a key approach within mi is circuit analysis @xcite @xcite , which isolates minimal computational subgraphs comprising essential components like attention heads and ffns that reproduce a model's behavior on a given task. prior work on factual recall has focused on localizing knowledge within transformer parameters @xcite @xcite @xcite and on behavioral analyses that trace the emergence of linguistic and reasoning capabilities during pretraining @xcite @xcite @xcite @xcite . while these studies have advanced our understanding of factual knowledge from both internal and external perspectives, they have not systematically examined how the components of a factual recall circuit evolve over training. in this work, we bridge this gap by performing a time-course mechanistic interpretability analysis of factual knowledge representation in olmo-7b @xcite by tracing information flow routes and classifying component roles over 40 training snapshots (see figure 1 ). specifically, we investigate: to achieve this, we trace information flow routes @xcite in olmo-7b using interpretability tools, analyze component dynamics, and classify components according to their roles within factual recall circuits across different training snapshots. we make the following contributions: ## factual knowledge probing over time in this section, we describe our approach to probing factual knowledge (fig. 1 ). we first introduce our dataset (sec. 3.1), then detail the olmo-7b training snapshots used (sec. 3.2), and finally assess snapshot performance via accuracy (sec. 3.3). ## related work this section reviews prior work on mechanistic interpretability, model behavior evolution, and how transformers store and retrieve factual knowledge. while past research has deepened our understanding of fully trained models, less focus has been given to how these mechanisms evolve during training -a gap this work addresses. ## discussion & conclusion our time-course mechanistic interpretability study of olmo-7b uncovers several core patterns in how factual knowledge circuits form and evolve during pretraining: