Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-Context Models (2024)

Abstract: Numerous recent works target to extend effective context length for language models and various methods, tasks and benchmarks exist to measure model’s effective memory length. However, through thorough investigations, we find limitations for currently existing evaluations on model’s memory. We provide an extensive survey for limitations in this work and propose a new method called forgetting curve to measure the memorization capability of long-context models. We show that forgetting curve has the advantage of being robust to the tested corpus and the experimental settings, of not relying on prompt and can be applied to any model size. We apply our forgetting curve to a large variety of models involving both transformer and RNN/SSM based architectures. Our measurement provides empirical evidence for the effectiveness of transformer extension techniques while raises questions for the effective length of RNN/SSM based models. We also examine the difference between our measurement and existing benchmarks as well as popular metrics for various models.

## introduction arguably driven by large language model (llm) practical needs, numerous works today aim to extend llm contexts. the approaches range from modifying existing transformer architectures @xcite @xcite , training existing llms to extend their contexts @xcite @xcite , to more recent modelling based on fundamental architectural changes using rnns or hybrid architectures @xcite @xcite , etc. all of the green, blue, and red areas respectively indicate finegrained memory where the model achieves 99% token replication accuracy (except for very short sequences), coarse-grained memory where copy accuracy surpasses lm accuracy, and the amnesia area where the model completely ignores the prefix. the most commonly used metric for evaluating a model's natural language generation ability is perplexity (ppl). it is generally assumed that a superior long-range perplexity suggests a more effective utilization of long context. however, recent findings indicate that a lower perplexity for long context does not necessarily lead to improved performance in downstream long context applications @xcite . this raises questions about the validity of using perplexity as an indicator of a model's long-context memorization capability. 1 alternative metrics have been proposed to evaluate a model's long-term memory, including "needle in a haystack" @xcite and its derived metrics @xcite . however, these metrics also present their own limitations. for instance, "needle in a haystack" is highly sensitive to the prompt used, foot_1 and such sensitivity is attributed more to the varying instruction-following capabilities across different models, rather than differences in their memory capabilities. other metrics @xcite @xcite , which involve toy tasks, face challenges in generalizing to real-world settings and are susceptible to severe overfitting. we survey a significant number of popular metrics and benchmarks which measure model long-term memory in this work. to address these limitations and provide a reliable measure of a model's long-term memory, we introduce a new method called forgetting curve in this paper. our method also begins with a language model trained on a real-world corpus. we then exploit the model's emerging copy capability, assessing whether the trained model can replicate its prefix as part of the completion process. in order to account for instances where the token prediction relies solely on the model's natural language modelling (lm) capabilities instead of long-term memorization, we additionally plot the lm accuracy curve. figure 1 illustrates our method, demonstrating the progression of a memory pattern from fine-grained to coarse-grained memory, and ultimately to complete memory absence. the paper is organized as the following. we start with a critical evaluation of the limitations inherent in the metrics and tasks currently employed to measure a model's long-term memorization capability (section 2). following this, we detail the construction of the forgetting curve, highlighting its dual function in visualizing memory and enabling statistical calculation of memory length. we also establish the reliability of our method in assessing a model's memorization capability (section 3). using our forgetting curve, we comprehensively examine the long-range memorization capabilities of various current models and architectures (section 4). we demonstrate that the forgetting curves effectively illustrate a model's memorization capability through visualizations, revealing varying patterns across different architectures and models. we also validate certain context extension techniques, and raise questions about some architectural claims. finally in analysis (section 5), we show that forgetting curves can be applied to models of different sizes under different settings. compared to existing effective length measurements @xcite or perplexity, the forgetting curve provides a self-contained, memory-focused measurement. ## limitations of long-range memory measures limited memory usage (lmu) the first limitation that we identify in the long-range measurement literature is limited memory usage (lmu), where the measurement or metric does not have a strong relationship with model's capability to leverage long context. perplexity is a surprising example of lmu. recent works @xcite @xcite typically demonstrate their model's long-range capability by showing a stably low perplexity given long contexts. however, @xcite recently found that lower perplexity does not imply an improved downstream performance on long-context tasks. in section 5, we further demonstrate, using forgetting curves, that some models can improve long-range perplexity without enhancing long-range memory. lmu also manifests in popular long-range measurement tasks such as "needle in a haystack"(gkamradt, 2023) which assesses the capability to retrieve a specific piece of information from lengthy distracting texts. the memory usage is clearly limited since the model can theoretically achieve great performance by focusing only on the 'needle' and ignoring all the long context. methods derived from "needle in a haystack" like counting stars @xcite alleviate this drawback but still inherit such limitations. prompt required (pr) the second limitation in existing evaluation methods is their strong dependency on the exact prompts during evaluation. for instance, in the claude 2.1 experiments @xcite , a single prompt change dramatically improved the model's ability to find the 'needle', boosting claude 2.1's task success rate from 27% to 98% in the original evaluation. such significant dependency on prompts raises the following two issues. firstly, it makes the comparisons between models extremely difficult due to the large performance variance across different prompts. 3 secondly, it is challenging to apply these metrics to unaligned models and smaller ones due to their limited capability to follow instructions to find the 'needle'. this implies that new architectures need to be scaled up and aligned before these metrics can be applied, preventing rapid and low-cost experimental validation of a model's long-context capability. although latest benchmarks such as ruler @xcite and streameval @xcite can force a model to produce answers through carefully designed prompts, this approach is still cumbersome and model-specific. ## the forgetting curve in this work, we introduce a novel method called "forgetting curve" to measure a model's inherent long-context memorization capability. we show that the forgetting curve effectively mitigates the previously listed limitations and remains robust under various experimental settings. this makes it an ideal method for characterizing and visualizing a model's long-context memorization capability. ## analysis & discussion forgetting curve can be applied to a small model and is robust to data leakage. we want to see whether the forgetting curve can be used to measure model memory for small models. to this end, we train an 83m parameter model using the llama architecture on the pg-19 training dataset with a training context size of 1024. further details are provided in appendix d.1. we then plot its forgetting curves, measured using both the pg-19 test and training datasets. figure 4 shows the corresponding forgetting curves for the llama-83m model. according to our fine memory and coarse memory definition presented in the previous section, the model does not have fine memory but clearly exhibit coarse memory slightly beyond half of the training context size. we remark that the measurement is achieved on such small and arguably under-trained models because our measurement could separate between the memory capability and natural language understanding capability. in the contrary, it is not possible to measure memory using popular metrics such as "needle in a haystack" as the model has not yet acquired understanding at this stage. we finally remark the closeness between forgetting curves measured on pg-19 test set and those measured on the training set. obviously, both measurements point to very close coarse memory length. this shows that the forgetting curve is robust to data leakage as the forgetting curve greatly focuses on the difference between copy accuracy and lm accuracy. ## conclusion in this work, we identify the limitations present in existing long-sequence language modeling metrics, tasks and benchmarks. to address these limitations, we introduce a new method called forgetting curve, which provides a flexible and robust measure to visualize llm's long-range memory capabilities. using forgetting curve, we study the memory capabilities of fourteen open-source models with claimed context sizes ranging from 4k to 1m. our findings empirically validate the effectiveness of existing transformer context extension methods, while raising questions for rnn/ssm architectures. forgetting curve further reveals no direct correlation between memory capacity and perplexity. compared to existing ways to measure longcontext memory, forgetting curves mainly differs in trying to effectively decouple the model's longterm memory and language understanding ability. this provides a novel perspective for evaluating language models at all sizes, potentially guiding future research in model's long-term memory.