Evaluating Very Long-Term Conversational Memory of LLM Agents (2024)

Abstract: Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 600 turns and 16K tokens on avg., over up to 32 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.

## introduction despite recent advancements in dialogue models based on llms for extended contexts @xcite , as well as the integration of retrieval augmented generation (rag) figure 1 : an example in locomo. dialogs are steered by the speakers' personas and corresponding events e.g., joanna's responses are consistent with her pet allergies. for nate, the event got a new dog is followed by a playdate with neighbor's dog, showcasing long-term memory. multimodal dialog is enabled with image-sharing and image-response behaviors. ## generative pipeline for locomo an overview of our generative pipeline for lo-como is shown in figure 3 . we create two virtual agents, named l 1 and l 2 , each initialized with a llm m (i.e., gpt-3.5-turbo). to start, unique persona statements p are assigned to each agent l i , ensuring the integration of distinct personalities into their dialogues ( §3.1). to mirror real-life experiences, we create a temporal event graph g for each agent, which illustrates a realistic sequence of life events ( §3.2). the llm agent architecture @xcite is utilized for each agent l i , enabling them to effectively memorize and reflect conversation history into ongoing dialogues ( §3.3). further, each agent l i can share coherent images, thereby enhancing the multi-modal dialogue aspect. finally, human annotators are tasked with manually filtering and refining the generated data ( §3.4). ## locomo evaluation benchmark based on the dialogues generated in section 3, we introduce an evaluation benchmark (see figure 2 ) composed of three tasks to assess the accuracy of long-term memory. see statistics of the dataset and evaluation benchmark in ## experimental setup for the question-answering and event summarization tasks, we replace images in locomo with their captions @xcite , and use state-ofart llms to reason over text-only dialogues interleaved with image captions. we use images directly for the multimodal dialog generation task only. see additional details in appendix c. question answering. we evaluate three types of models: (1) base llms operating with constrained context lengths where earlier dialogues are omitted i.e., mistral-7b-instruct-v0.2 (jiang et al., 2023), llama-2-70b-chat @xcite , and llama-3-70b-instruct foot_4 ; (2) long-context llms with an extended context window i.e., gpt-3.5-turbo foot_5 , gpt-4-turbo foot_6 , gemini-1.0-pro @xcite and claude-3-sonnet foot_7 ; (3) retrieval-augmented generation (rag) involves retrieving relevant context from a database of dialog history, observations (assertions about speakers; see §3.3, figure 9 ), or session-level summaries (see §3.3, figure 8 ). we employ dragon @xcite as retriever and gpt-3.5-turbo as reader. event summarization. we present experiments using base and long-context setups from the question-answering task, but refrain from including rag since summarization requires a comprehensive understanding of the entire dialogue, rather than just retrieving a specific portion. we implement incremental summarization i.e., iteratively create a summary of a preceding sessions and then use that summary as a basis to summarize the subsequent sessions @xcite . ## experimental results we evaluate and analyze the comprehensive performance of all baseline methods for question answering ( §6.1), event graph summarization ( §6.2), and multi-modal dialogue generation ( §6.3). ## conclusion we develop a human-machine pipeline to collect locomo, a dataset of 10 high-quality very long conversations, each encompassing 600 turns and 16k tokens on avg., over up to 32 sessions, and propose an evaluation framework consisting of three tasks that evaluate models' proficiency in long conversations. our experiments show that llms struggle to comprehend long-term narratives within the dialog and fail to draw temporal and causal connections between events discussed by speakers.