Position Paper: M e M o: Towards Language Models with Associative Memory Mechanisms (2025)

Abstract: Memorization is a fundamental ability of Transformer-based Large Language Models, achieved through learning. In this position/theory paper, we propose a paradigm shift by designing an architecture to memorize text directly, bearing in mind the principle that memorization precedes learning. We introduce MeMo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. By design, MeMo offers transparency and the possibility of model editing, including forgetting texts. We experimented with the MeMo architecture, showing the memorization power of the one-layer and the multi-layer configurations.

## introduction transformer-based large language models achieve unrivaled performance in language modeling by learning to capture and represent complex sequential dependencies from statistical patterns through extensive training phases that iteratively refine their weights to best approximate natural language. this has triggered significant interest in gaining a better understanding of the inner workings of these models, focusing on how these models generalize and capture structure between similar samples in terms of syntactic dependencies @xcite , compositional relations @xcite concerning the quantity @xcite and quality @xcite of training data. besides generalization, a key component of transformers' success is the ability to memorize data while learning @xcite . indeed, earlier work investigated this other side of learning. while @xcite ; @xcite demonstrated evidence of memorization, @xcite ; @xcite studied how the internal components lead to memorization, and @xcite estimated the boundary between generalization and memorization, providing an estimation on their storage capacity. memorization is not inherently a drawback in language models because it plays a crucial role in handling factual knowledge, which is important for question answering, summarization, or information retrieval. this factual recall relies on a delicate balance. while generalization helps capture patterns and unseen relationships in data, memorization ensures that models retain critical and exact information when required. recent research has highlighted that memorization capability can be effectively harnessed using concepts rooted in associative memories @xcite -a system designed to link inputs to specific outputs and offers a structured and transparent way to store and retrieve information. by leveraging associative memory mechanisms, researchers have proposed strategies to post-edit llms @xcite , enabling control over what is memorized, how it is stored, and how it is accessed, enhancing their reliability in fact-based tasks. in this position/theory paper, we propose a paradigm shift by designing language models based on a different principle: memorization proceeds learning. by using associative memories, we build memo, a novel architecture for language modeling that explicitly memorizes sequences of tokens in layered associative memories. memo leverages correlation matrix memories @xcite , the concept that tokens and sequences of tokens can be represented as random vectors @xcite , and the johnson-lindestrauss transform to embed larger vectors in smaller spaces by preserving their distances @xcite . by design, memo offers transparency and the possibility of model editing, including forgetting texts. we experimented memo, showing the memorization power of single and multi-layer architecture. ## preliminaries and background representing words or tokens in small random vectors is the first important step in building language models with neural network architectures. using random vectors is a standard technique. indeed, random vectors are used in random indexing @xcite in information retrieval to reduce the document vector space and in distributed representations for neural networks as a convenient way to determine a set of vectors to represent sets of different tokens @xcite or structures @xcite @xcite . moreover, random vectors are used to initialize weight matrices in any language-oriented application in neural networks, including the initialization of transformers @xcite to build large language models from scratch. multivariate gaussian random vectors have the important property of being able to generate sets e of nearly orthogonal unitary vectors that can form an approximate base of the space r n in a smaller space r d @xcite . each token t is then represented with a distinct vector in @xmath0, and the two following properties hold with a probability larger than 1δ: using these vectors with their properties, it is possible to represent a bag-of-tokens b in a single vector t b offering the operation that approximately counts the number of times a token is in b. the vector t b is obtained by summing up vectors representing tokens in b and, then, the counting operation is: correlation matrix memories (cmms) @xcite are a powerful tool to store key-value (k i , v i ) pairs in distributed memories as the sum of outer products of the vectors representing the keys k i and vectors representing the values v i : these cmms have been generally defined on onehot representations @xcite and, eventually, reduced afterwards @xcite . then, to retrieve the value associated with a key, the matrix c should be multiplied with k t j . as vectors k i are one-hot vectors, the following property holds: k t j @xmath1, we use the correlated form: to make cmms practical, in memo, we use these memories along with the multivariate gaussian vectors to represent keys and values. hence, the generic property of this associative matrices is k t j c ≈ e t j @xmath2. the idea behind correlation matrix memories has often been used to explain that feed-forward matrices are where transformer architectures store most information @xcite . in memo, cmms become the cornerstone for defining a novel approach to building language models. johnson-lindestrauss transform @xcite , derived by using the johnson-lindestrauss lemma (jll) @xcite , guarantees that it exists a linear transformation t d×n that transforms a substantial subset v of vectors in a bigger space r n in vectors in a smaller space r d by preserving their distance with an approximation ϵ with high probability. then, given two vectors a and b in v , the following property is guaranteed: 𝐶𝑀𝑀 = 𝑊 𝑉 𝑖𝑛 𝑊 𝑉 𝑡ℎ𝑒 𝑊 𝑉 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑊 𝑉 𝑎𝑛𝑑 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑇 𝑊 𝑉 ℎ𝑒 𝑊 𝑉 𝑒𝑛𝑟𝑜𝑙𝑙𝑒𝑑 𝑊 𝑉 𝑖𝑛 𝑊 𝑉 𝑡ℎ𝑒 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑇 𝑑 × 6 6 × 𝑑 𝑊 𝑉 𝑒𝑛𝑟𝑜𝑙𝑙𝑒𝑑 𝑊 𝑉 𝑖𝑛 𝑊 𝑉 𝑡ℎ𝑒 𝑊 𝑉 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑎𝑛𝑑 𝑇 𝑊 𝑉 𝑎𝑛𝑑 𝑊 𝑉 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑊 𝑉 𝑡𝑒𝑎𝑐ℎ𝑖𝑛𝑔 𝑊 𝑉 𝑑𝑖𝑝𝑙𝑜𝑚𝑎 𝑝𝑟𝑜𝑔𝑟𝑎𝑚 𝑇 a) … … 𝑖𝑛 𝑇 𝑊𝑉 𝑇 𝑡ℎ𝑒 𝑇 𝑊𝑉 𝑇 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑇 𝑊𝑉 𝑇 𝑎𝑛𝑑 𝑇 𝑊𝑉 𝑇 𝑑 × 6 … 𝐶𝑀𝑀 ≅ 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑇 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑇 𝑎𝑛𝑑 𝑇 𝑝𝑟𝑜𝑔𝑟𝑎𝑚 𝑇 … 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑇 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑇 𝑎𝑛𝑑 𝑇 𝑝𝑟𝑜𝑔𝑟𝑎𝑚 𝑇 … [0 0 1.00 … 0] = 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑇 b) 𝑖𝑛 𝑇 𝑊𝑉 𝑇 𝑡ℎ𝑒 𝑇 𝑊𝑉 𝑇 𝑚𝑎𝑡ℎ𝑒𝑚𝑎𝑡𝑖𝑐𝑠 𝑇 𝑊𝑉 𝑇 𝑜𝑟 𝑇 𝑊𝑉 𝑇 [0 0 0.75 … 0] 𝑝ℎ𝑦𝑠𝑖𝑐𝑠 𝑇 the jll with the demonstration in @xcite shows that it is possible to build this matrix t with high probability by using multivariate gaussian vectors as transformation rows. jlt matrices are the last ingredient of our new model, as we need to transpose sequences of tokens in their representations in the target r d space. ## memo: language models with multi-layer correlation matrix memories building on correlation matrix memories, on multi-variate gaussian vectors to represent tokens and token sequences, and on johnson-lindestrauss transforms, we present here memo 1 , a way to build language models that memorize texts in a clear, transparent way. we first present how to build a language model with a single cmm (sec. ## experimental investigation in this section, we experiment with the memorization capacity of memo with a single layer and with multiple layers. ## conclusion and future work memorization is a key component of transformerbased large language models. hence, in this paper, we proposed to shift the paradigm by designing language models based on memorization. we then presented memo as a novel way to build language models using correlation matrix memories stacked in layers. experimental evaluation has shown that memo-like architecture can memorize sequences of tokens. by using memorization, memo-like architectures are transparent and editable by design and opens back the possibility to include explicit knowledge modeling in neural network language models. indeed, memo can help leverage traditional linguistic studies in this era, where transformer-based large language models are obtaining unprecedented performance. with memo, we could control how linguistic knowledge is used to generalize examples, we could embed transformation rules, and we could represent knowledge graphs and linguistic ontologies. in other words, memo gives back control to knowledge experts, linguists, and nlp practitioners with the aim of reducing data hungriness of large language models.