\infty -former: Infinite Memory Transformer (2022)

Abstract: Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the \infty -former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the \infty -former’s attention complexity becomes independent of the context length, trading off memory length with precision.In order to control where precision is more important, \infty -former maintains “sticky memories,” being able to model arbitrarily long contexts while keeping the computation budget fixed.Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the \infty -former’s ability to retain information from long sequences.

## introduction when reading or writing a document, it is important to keep in memory the information previously read or written. humans have a remarkable ability to remember long-term context, keeping in memory the relevant details @xcite . recently, transformer-based language models have achieved impressive results by increasing the context size @xcite @xcite @xcite . however, while humans process information sequentially, updating their memories continuously, and recurrent neural networks (rnns) update a single memory vector during time, transformers do not -they exhaustively query every representation associated to the past events. thus, the amount of computation they need to perform grows with the length of the context, and, consequently, transformers have computational limitations about how much information can fit into memory. for example, a vanilla transformer requires quadratic time to process an input sequence and linear time to attend to the context when generating every new word. several variations have been proposed to address this problem @xcite . some propose using sparse attention mechanisms, either with data-dependent patterns @xcite @xcite @xcite or data-independent patterns @xcite @xcite , reducing the self-attention complexity @xcite @xcite , and caching past representations in a memory @xcite . these models are able to reduce the attention complexity, and, consequently, to scale up to longer contexts. however, as their complexity still depends on the context length, they cannot deal with unbounded context. in this paper, we propose the ∞-former (infinite former; fig. 1 ): a transformer model extended with an unbounded long-term memory (ltm), which allows the model to attend to arbitrarily long contexts. the key for making the ltm unbounded is a continuous-space attention framework @xcite which trades off the number of information units that fit into memory (basis functions) with the granularity of their representations. in this framework, the input sequence is represented as a continuous signal, expressed as a linear combination of n radial basis functions (rbfs). by doing this, the ∞-former's attention complexity is o(l 2 + l × n ) while the vanilla transformer's is o(l × (l + l ltm )), where l and l ltm correspond to the transformer input size and the long-term memory length, respectively (details in §3.1.1). thus, this representation comes with two significant advantages: (i) the context can be represented using a number of basis functions n smaller than the number of tokens, reducing the attention computational cost; and (ii) n can be fixed, making it possible to represent unbounded context in memory, as described in §3.2 and fig. 2 , without increasing its attention complexity. the price, of course, is a loss in resolution: using a smaller number of basis functions leads to lower precision when representing the input sequence as a continuous signal, as shown in fig. 3 . to mitigate the problem of losing resolution, we introduce the concept of "sticky memories" ( §3.3), in which we attribute a larger space in the ltm's signal to regions of the memory more frequently accessed. this creates a notion of "permanence" in the ltm, allowing the model to better capture long contexts without losing the relevant information, which takes inspiration from long-term potentiation and plasticity in the brain @xcite . to sum up, our contributions are the following: ## infinite memory transformer to allow the model to access long-range context, we propose extending the vanilla transformer with a continuous ltm, which stores the input embeddings and hidden states of the previous steps. we also consider the possibility of having two memories: the ltm and a short-term memory (stm), which consists in an extension of the transformer's hidden states and is attended to by the transformer's self-attention, as in the transformer-xl @xcite . a diagram of the model is shown in fig. 1 . ## experiments to understand if the ∞-former is able to model long contexts, we first performed experiments on a synthetic task, which consists of sorting tokens by their frequencies in a long sequence ( §4.1). then, we performed experiments on language modeling ( §4.2) and document grounded dialogue generation ( §4.3) by fine-tuning a pre-trained language model. foot_2 ## conclusions in this paper, we proposed the ∞-former: a transformer extended with an unbounded long-term memory. by using a continuous-space attention framework, its attention complexity is independent of the context's length, which allows the model to attend to arbitrarily long contexts while keeping a fixed computation budget. by updating the memory taking into account past usage, the model keeps "sticky memories", enforcing the persistence of relevant information in memory over time. experiments on a sorting synthetic task show that ∞former scales up to long sequences, maintaining a high accuracy. experiments on language modeling and document grounded dialogue generation by fine-tuning a pre-trained language model have shown improvements across several metrics.