M em B ench: Towards More Comprehensive Evaluation on the Memory of LLM -based Agents (2025)

Abstract: Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench .

## introduction in recent years, large language models (llms) have demonstrated remarkable capabilities in processing natural languages and performing complex tasks across various domains @xcite . however, vanilla llms typically operate in static scenarios, without interacting with external environments, thereby limiting their potential advancement toward artificial general intelligence (agi). to address this limitation, many recent works propose llm-based agents with extra modules besides the foundation models, enabling them to interact with environments with autonomous learning and dynamic adaptation @xcite . among them, the memory module serves as an essential foundation for saving critical information and accumulating experiences. it empowers llm-based agents to better meet the demands of dynamic tasks, as well as evolve within their environments continuously @xcite . some previous studies evaluate the memory capability of llm-based agents in a subjective way, which adopts human evaluators or llms to score the memory process @xcite . other studies focus on the evaluation in an indirect way @xcite . they measure the task performances of agents conditional on different memory mechanisms, where a better memory mechanism generally leads to better performances. recently, some studies introduce longterm dialogue datasets, which can be used to evaluate the long-term memory capabilities of llmbased agents objectively @xcite . however, previous works have some limitations on evaluating the memory capability of llm-based agents. first of all, most of them provide insufficient evaluation of the different levels of memory capabilities, which primarily focus on factual memory while neglecting reflective memory. here, we define the factual memory as a low-level type of memory that involves information that is explicitly provided. in contrast, the reflective memory stands a higher level, which is not explicitly stated but can be implicitly reflected. for example, a user's taste preferences represent reflective memory, while their preference for specific dishes is factual memory. second, most of them are limited to participation scenarios, where the agent interacts with the user from a first-person perspective. however, in the agent's daily usage, there are also observation scenarios, where the agent observes and records the user's messages from a third-person perspective. moreover, most of them are just focusing on the effectiveness of memory mechanisms without considering their efficiency and capacity, which is also significant in real-world applications. to address these limitations, we propose a more comprehensive dataset and benchmark to evaluate the memory capability of llm-based agents. the major features of our dataset and benchmark are presented as follows: multi-scenario dataset. to evaluate the agent's memory capabilities across different scenarios, our dataset includes data from two common usage scenarios. the first is the participation scenario, where the agent interacts with the user. the second is the observation scenario, where the agent is assumed as the role of an observer and required to record the information provided by the user. ## related works in recent years, llm-based agents have been widely applied in many fields, such as recommendation @xcite , finance @xcite and personal assistants @xcite , because of their great capabilities in solving complex tasks and interactive scenarios @xcite . among the various abilities of agents to solve problems, memory is one of the most important, which is responsible to store observed information and recall relevant experiences, in order to support llm inference @xcite . the evaluation on the memory capability of llm-based agents is a critical problem for developing advanced memory. previous datasets used for memory evaluation mainly come from dialogue datasets designed to evaluate chat assistants, focusing on assessing the factual memory capabilities of the assistant. lo-como @xcite constructs longterm conversations with llm-expanded personalized descriptions and temporal event graphs, creating conversations for multiple evaluation tasks. longmemeval @xcite builds a userassistant interaction dataset with attribute ontology and timestamped history. @xcite adapts personachat @xcite through translation and role-playing, with annotated personalization usage and partial information visibility. @xcite propose a bilingual benchmark in english and chinese, covering comprehensive tasks like q&a, summarization, and code completion. @xcite provide a dataset designed to evaluate long-context language models across diverse domains and input lengths. perltqa @xcite generates character profiles and events with chatgpt and wikipedia, creating profiles, events, and qa pairs after manual validation. as we have shown in compared with previous works, our work is the first study that emphasizes reflective memory, puts forward observation scenarios, adopts evaluation methods that are better suited to the agent's memory process, with more comprehensive metrics. ## benchmark in this section, we create a benchmark based on our dataset to evaluate the memory capabilities of llmbased personal agents. to better evaluate the upper bounds of the agent's memory mechanism capabilities, we also utilize the news dataset (dataguy and amoako, 2022) to generate a large amount of dialogues and messages serving as noise memory content that is irrelevant to the questions. we ensures that the content of noise data does not contain factual conflicts with memory messages, or dialogues in our evaluation dataset. it also allows us to control the difficulty of the evaluation by adjusting the proportion of noise data. ## conclusion this paper provides a more comprehensive and scalable dataset for evaluating llm-based agent's memory mechanisms. it includes a dataset with multi-scenarios (both participation and observation), and multi-level memory content include re-flective memory and factual memory. based on this dataset, we constructed a time-aware evaluation framework that simulates the daily interactions between users and agents with multi-metric include accuracy, recall, capacity and temporal efficiency. we evaluate the performance of seven common memory mechanisms in agents on our benchmark.