Evaluating the Long-Term Memory of Large Language Models (2025)

Abstract: In applications such as dialogue systems, personalized recommendations, and personal assistants, large language models (LLMs) need to retain and utilize historical information over the long term to provide more accurate and consistent responses. Although long-term memory capability is crucial, recent studies have not thoroughly investigated the memory performance of large language models in long-term tasks. To address this gap, we introduce the Long-term Chronological Conversations (LOCCO) dataset and conduct a quantitative evaluation of the long-term memory capabilities of large language models. Experimental results demonstrate that large language models can retain past interaction information to a certain extent, but their memory decays over time. While rehearsal strategies can enhance memory persistence, excessive rehearsal is not an effective memory strategy for large models, unlike in smaller models. Additionally, the models exhibit memory preferences across different categories of information. Our study not only provides a new framework and dataset for evaluating the long-term memory capabilities of large language models but also offers important references for future enhancements of their memory persistence.

## introduction in recent years, large language models (llms) have been widely applied across various fields, driving technological advancements. in many practical applications, such as personal assistants @xcite , personalized recommendations @xcite , and dialogue systems @xcite , models need to retain and utilize past information over the long term to provide more accurate and consistent responses. although long-context strategies @xcite and retrieval-augmented generation techniques @xcite have improved llms' memory in handling long-term tasks, these text-based memory methods face significant limitations in terms of token count, computational cost, and inference time @xcite . in contrast, parameter-based memory stores information by adjusting the model's internal parameters, meaning that this information is an inherent part of the model, better reflecting the concept of memory within the model itself. while prior work has demonstrated the memory performance of llms in related domains @xcite , their memory performance in long-term tasks remains underexplored. considering that human-machine dialogue is a crucial application of llms, memory plays a key role. evaluating llms' performance in long-term dialogue tasks can indirectly reflect their long-term memory capabilities @xcite . to this end, we propose a pipeline for constructing long-term dialogue data: long conversation generation (locogen), an automated dialogue generation pipeline based on llms. we use locogen to build two dialogue datasets focused on evaluating llms' long-term memory capabilities-long-term chronological conversations (locco) and its extended version locco-l. locco contains 100 users' long-term dialogues with a chatbot, totaling 3080 interactions, simulating the application scenario of llms as chatbots. locco-l provides even longer conversational sequences with more iterations, offering additional test cases for evaluating memory persistence across extended timelines. previous research has predominantly assessed memory by evaluating the extent to which models fit the training data, employing identical task formats during both the training and evaluation phases @xcite @xcite . however, for llms, the memory process should represent an organic integration of training data, rather than mere rote memorization of its paradigms. inspired by @xcite , we examine llms' memory through dialogue question-answering tasks. in our experimental setup, the model does not learn how to utilize the conversation to perform the q&a tasks during the memory formation process. therefore, when the model can accurately answer questions using information from the conversation, it indicates that the model has genuinely retained the conversational information. this demonstrates an organic and interactive memory process. additionally, metrics like rouge @xcite and bleu @xcite have limited accuracy in open-domain dialogues, so we trained a consistency model to replace existing automated metrics for assessing response accuracy. experiments on open-source llms show that they possess a certain degree of memory capability in long-term tasks, able to recall historical interaction information such as names, places, and specific events, and use this information to answer questions. however, llms face limitations in handling dialogues over long time spans, gradually forgetting historical dialogues. to enhance memory persistence, we employed a rehearsal strategy from continual learning. the results demonstrate that, unlike in smaller models, excessive rehearsal is not an effective memory strategy. our contributions are as follows: ## conclusion to explore the long-term memory of llms, we developed an automated pipeline, locogen, for constructing long-term dialogue data and created the locco dataset, which includes long-term dialogue data between 100 users and a chatbot, along with qa pairs to evaluate model memory. we additionally constructed the locco-l dataset to provide extended test cases for future research on longterm memory mechanisms. experiments show that llms can remember historical interaction information with users to some extent, but this memory gradually weakens over time, especially when dealing with very long time spans. we also revealed that models have preferences when remembering different categories of information, providing a new direction for future research on how to balance and optimize memory capabilities for different types of information. additionally, we found that repetition strategies can effectively improve the persistence of model memory. our research not only provides new methods and datasets for evaluating the long-term memory capabilities of llms but also offers important references and insights for future improvements in the persistence and accuracy of model memory. future work can further explore improvements in model architecture and training methods to better support long-term memory retention and application.