Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data (2025)

Abstract: This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers. The models are trained using synthetic text datasets derived from the Systematized Nomenclature of Medicine (SNOMED) knowledge graph: triplets, representing static connections, and sequences, simulating complex relation patterns. The results show that embedding size is the primary determinant of learning speed and capacity, while additional layers provide limited benefits and may hinder performance on simpler datasets. Activation functions play a crucial role, and Softmax demonstrates greater stability and capacity. Furthermore, increasing the complexity of the data set seems to improve the final memorization. These insights improve our understanding of transformer memory mechanisms and provide a framework for optimizing model design with structured real-world data.

## introduction transformer-based large language models (llms) have revolutionized natural language processing, excelling at tasks ranging from text generation and translation to question answering and summarization. despite these advances, a fundamental understanding of how these models store and recall information, particularly factual or structured knowledge, remains limited. clarifying these mechanisms is crucial for optimizing model performance and enabling efficient, real-world deployment. one impactful example is healthcare, where transformer-based models could assist clinicians through wearable devices such as smart glasses or watches @xcite @xcite . due to privacy and reliability, the preferred system would be a local on-edge, requiring minimal computation but with the capacity to memorize all relevant facts in the specific healthcare area. recent theoretical and empirical studies have sought to quantify the memorization capacity of transformers. @xcite introduced mathematical bounds for memory capacity, demonstrating that transformers could memorize o(d + n + √ nn ) parameters, where d, n, n correspond to embedding dimensions, dataset size, and model size, respectively. additionally, @xcite proved, that õ( √ nn ) parameters are not only sufficient, but also necessary for some types of transformers. @xcite extended this work by analyzing the effects of multi-head attention on memorization, revealing the interplay between architectural components and the model's ability to store and recall information. the experiments in @xcite used randomly generated sequences of numbers to evaluate the memorization capabilities of the transformer models on unstructured data. most capacity studies use synthetic datasets because accurate capacity measurement becomes very difficult in the case of uncontrolled free text content. the experiments reported in the current paper use sequential data generated from the knowledge graph, which, while controlled, has some of the hierarchical and relational complexity of realworld text content. more specifically, small-scale decoder-only transformer models @xcite were trained to memorize structured sentences derived from the systematized nomenclature of medicine (snomed) knowledge graph (kg) @xcite , a comprehensive medical ontology, which encodes semantic relationships between medical concepts, offering a rich dataset to explore memory mechanisms under realistic conditions. exact memorization of selected relations would be critical, for example, in the healthcare use cases described above. our aim is not to generalize to all llms or domains, but rather to offer a practical, reproducible framework for measuring memorization on realistic kg data. the relative task simplicity is by design: more complex or less-controlled tasks would conflate memorization with generalization, making it difficult to draw clear, interpretable conclusions about model capacity. to measure the memorization of the transformer models, the maximum attainable capacity (mac) method was used. it evaluates the practical limit of samples a model can retain when trained on a large dataset. our approach leverages structured datasets consisting of static triplets and longer sequences simulating graph traversal paths, capturing relationship patterns between concepts. these datasets allowed us to empirically analyze how model architecture, training configurations, dataset size, and complexity influence training dynamics and final memorization performance. this work serves as a proof-of-concept, showing that structured data in the real world can evaluate memorization in practice. firstly, we introduce a reproducible pipeline for converting large ontologies into tokenized datasets suitable for memorization studies. secondly, we evaluate how transformers' architecture influences capacity, building on prior theoretical insights. lastly, we highlight cases where models fail to memorize all samples despite sufficient capacity, motivating future studies into training dynamics and error patterns. our findings do not aim to establish universal scaling laws or generalization behavior but to provide a reproducible framework for studying memory-limited models under realistic constraints. ## discussion this study examined how decoder-only transformer models memorize structured data derived from a real-world medical ontology. our focus was not on generalization, but on a controlled analysis of memorization, presenting a proof-of-concept framework that bridges theoretical insights and practical evaluation. the complete snomed kg contains more than a million relations, integrating diverse fields of medicine (e.g., substances, diseases, and anatom-ical structures). however, in mobile applications, e.g. small transformers in smart glasses or smartwatches, models must efficiently retain only targeted subsets of information. for example, smart glasses for a cardiac surgeon or a smartwatch with a personal dietary coach might require a domainspecific llm that memorizes about 10 to 100,000 items. as discussed in @xcite ; @xcite , isolating memorization is a valid objective that reveals how much a transformer can reliably store under different architectural configurations. our methodology reflects this: we analyze how dataset characteristics and architectural choices affect convergence and memorization, independent of generalization ability or test-time reasoning. to ensure clear capacity measurement, we deliberately focused on tasks where ground-truth memorization can be unambiguously defined. increasing complexity would blur the line between memorization and generalization, making interpretation less fair and direct. ## conclusions this study investigated how transformer architecture and dataset structure influence memorization capacity, introducing a practical framework for evaluating memorization on real-world data, such as the snomed knowledge graph. key findings show that embedding size and activation function have more impact than depth, while larger datasets improved memorization but required longer training. triplets performed well in simpler models, whereas sequences excelled but introduced fluctuations. challenges remain in efficiency, layerspecific contributions, and generalization, necessitating further research on scalability, compression, and architecture optimization. for practical use of small transformers in medical smart devices, models must efficiently store specialized knowledge while maintaining computational feasibility. future work should explore longer sequences, adaptive memory compression, and layer-wise analysis to enhance structured knowledge memorization in practical deployments.