Analyzing Memorization in Large Language Models through the Lens of Model Attribution (2025)

Abstract: Large Language Models (LLMs) are prevalent in modern applications but often memorize training data, leading to privacy breaches and copyright issues. Existing research has mainly focused on post-hoc analyses—such as extracting memorized content or developing memorization metrics—without exploring the underlying architectural factors that contribute to memorization. In this work, we investigate memorization from an architectural lens by analyzing how attention modules at different layers impact its memorization and generalization performance. Using attribution techniques, we systematically intervene in the LLM’s architecture by bypassing attention modules at specific blocks while keeping other components like layer normalization and MLP transformations intact. We provide theorems analyzing our intervention mechanism from a mathematical view, bounding the difference in layer outputs with and without our attributions. Our theoretical and empirical analyses reveal that attention modules in deeper transformer blocks are primarily responsible for memorization, whereas earlier blocks are crucial for the model’s generalization and reasoning capabilities. We validate our findings through comprehensive experiments on different LLM families (Pythia and GPT-Neo) and five benchmark datasets. Our insights offer a practical approach to mitigate memorization in LLMs while preserving their performance, contributing to safer and more ethical deployment in real-world applications.

## introduction large language models (llms) have become ubiquitous in modern applications, powering everything from conversational agents to advanced data analysis tools. however, these models often memorize parts of their training data, causing serious concerns such as privacy and copyright infringements @xcite @xcite , hindering its employment in high-stakes applications. prior works approach memorization from a posthoc perspective, where they predominantly focus on methods to extract memorized samples from trained llms @xcite or propose metrics to quantify memorization @xcite . while some works also explore machine unlearning techniques to remove specific memorized content after training @xcite , they do not delve into the fundamental model mechanisms that contribute to memorization. there has been little to no exploration of understanding memorization from a fundamental architectural viewpoint. in particular, systematically analyzing which layers and modules within transformer architectures contribute to memorization. we argue that identifying these components is essential for developing strategies to mitigate memorization without compromising the model's generalization capabilities. present work. in this work, our goal is to understand the trade-off between reasoning and memorization within llms by leveraging model attribution techniques. in particular, we perform systematic architectural interventions by bypassing (or "short-circuiting") the attention modules at targeted blocks of the transformer while preserving other modules such as layer normalization and mlp operations, allowing us to isolate the impact of specific attention modules on both memorization and generalization. we support our experimental findings with theoretical analysis of the impact of bypassing attention modules at different blocks. our theorems introduce bounds on the difference in model outputs with standard attention and shortcircuited attention. our theoretical insights and empirical results indicate that short-circuiting attention modules in earlier blocks leads to significant differences in the output vectors compared to the original model and these differences disrupt the model's internal representations, directly hindering its performance and affecting generalization capabilities. in contrast, bypassing attention modules in deeper blocks results in smaller differences in the output vectors, where these changes are sufficient to prevent the model from generating memorized content verbatim and do not significantly impair its ability to generalize from the data. our empirical analysis across two different model families-pythia and gpt-neo-and four benchmark datasets align with the theoretical findings and demonstrate that intervening in the attention modules of deeper layers mitigates memorization and preserves their generalization capabilities. our contributions can be summarized as follows: ## related work memorization and extraction of training data. works on image classifiers argue that a certain degree of memorization is inevitable during generalization @xcite . this phenomenon is even more pronounced in llms with huge parameter counts. llms have been shown to memorize and reproduce verbatim samples from their training corpus @xcite . various works have studied this phenomenon, from the angle of model scale @xcite , and data deduplication @xcite . recent work @xcite has studied memorization in llms with a deeper emphasis on semantics and other factors contributing to memorization. however, very few works explore the architectural aspect of memorization in large transformers. @xcite shows that while memorization cannot be localized to any part of the transformer, predictable gradient patterns are observed. in our work, we show via an attention-bypassing technique that certain blocks of the transformer can be identified as pertinent for memorization, while not boosting the generalization capabilities of the model. ## methodology our goal is to study the phenomenon of memorization in llms from the perspective of model attribution. previous literature @xcite @xcite has shown the efficacy of pruning both width and depth of transformers while maintaining downstream performance. this motivates us to study the effect of similar interventions on the memorization characteristics of llms. in particular, we wish to isolate the contribution of individual model components to memorization. the attention mechanism is fundamental to transformer models, enabling them to understand context and manage long-range dependencies @xcite . by dynamically weighing the significance of different words relative to each other, transformers capture nuanced meanings and relationships within data. for these reasons, we posit that the major contribution of memorization in llms stems from attention, and focus our analysis around it. we develop a mechanism to "short-circuit" the contribution of the attention mechanism in various blocks of a trained model during, and use it to conduct our experiments. ## experiments using our model attribution method described in sec. 3, we now study the effect of 'short-circuiting' the attention mechanism in various blocks of an llm on its memorization of the training dataset. ## conclusion our study explores how attention modules at different depths contribute to memorization in llms. by systematically bypassing attention modules at targeted blocks while preserving other components, we isolate their impact on memorization and generalization. our theoretical analysis showed that bypassing attention in earlier layers disrupts output representations, harming performance and generalization, while interventions in deeper layers prevent memorized content generation without impairing generalization abilities. empirical experiments with pythia and gpt-neo models aligned with these predictions, indicating that deeper attention blocks play a critical role in memorization phenomena. this work enhances understanding of how transformer components influence memorization and offers a practical approach to address ethical concerns like privacy violations. by reducing memorization without degrading model perfor-mance, we contribute to developing more ethical and reliable llms suitable for real-world applications. our targeted intervention strategy opens avenues for future research into architectural modifications that balance memorization and generalization, advancing responsible artificial intelligence technologies.