MAD ial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation (2025)

Abstract: Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory injection, emotion support (ES) proficiency, and intimacy, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy.

## introduction long-term memory is crucial for achieving humanlike communication in scenarios that demand sustained interactions, such as personal companionship, psychological counselling, and personal as-sistant tasks @xcite . powerful large language models (llm) such as llama @xcite , claude @xcite , gpt4 @xcite and @xcite have demonstrated a remarkable ability to understand and generate coherent responses @xcite . however, their lack of long-term memory hinders their application in dialogue systems for natural conversations. therefore, researchers built diverse memory-augmented dialogue systems (mads) that maintain contextual understanding, and ensure meaningful interactions over time @xcite @xcite . there are two types of mads: (1) memory encoded in parameters or retrieving memory via learned adaptive residual side-network @xcite . (2) dialogue histories as memory and retrieve the memory based on semantic similarities @xcite . the first of which is non-scalable due to the expensive cost with a growing user number, while all types of mads are using evaluation schemes that are limited to question-answering tasks with language modelling assessment, instead of evaluating the realistic multi-turn dialogues @xcite . additionally, existing mads rely solely on passive memory retrieval triggered by users' mention, which significantly deviates from the proactive nature of human accessing memory. according to the two-stage theory in cognitive science, human memory recall involves a memory search (or generation) process followed by a recognition (or decision) process @xcite . an example is shown in appendix i. memory recall can also be triggered by states rather than dialogue utterance, where "state" encompasses an individual's surroundings, and mental and physical states when the memory was constructed @xcite . for example, emotional states can trigger memory recall, highlighting the role of long-term memory in emotion regulation, instead of only serving as the common ground between individuals for conversation @xcite @xcite . long-term memory is a core component for emotion regulation @xcite . indeed, by recalling a happy memory, people can mitigate the effect of a negative mood state @xcite . however, current mads only leverage memory as common ground in conversations, while overlooking the potential for emotional support that requires proactive memory recall. to approach a practical and comprehensive evaluation of mads, we introduce memory-augmented dialogue benchmark (madial-bench) to evaluate generated responses with human-centred aspects justified by cognition science and psychology theories. madial-bench represents memories from emotions, scenes and other cues as states. the two-stage theory is adopted for separate assessment in madial-bench to imitate the human memory recall procedure. the dataset enables the evaluation of two recall mechanisms: passive recall, searching memories when the user mentions a specific event or item, which provides common ground in conversation; and proactive recall, retrieving memories based on user emotion or scene as per psychological strategies, to achieve interpersonal emotional regulation. the dialogues in the benchmark are refined by humans with high emotional intelligence (ei) to achieve the goal. we also designed a set of guidelines based on emo-tional supporting (es) theories @xcite to measure the aspect of es proficiency. it is the first multi-recall and multi-stage memoryaware emotional dialogue generation benchmark. the paper highlights the evaluation of the retrieval accuracy, memory recognition & injection ability, and comprehensive assessment of generated responses, e.g. language style, es proficiency and intimacy. after extensive experiments on llms, we discovered memory injection improves intimacy between mads and users. there is still a gap between the most advanced models and human performance. our main contributions are three-fold: • we construct a delicate memory-augmented dialogue benchmark, called madial-bench. it is the first multi-recall multi-stage memory augmented dialogue system benchmark guided by cognitive science theory. ## related works memory recall is an information retrieval process from people's past experience, which also acquaints them with past events @xcite . two main relevant theories on the recall process are the two-stage theory and the theory of encoding specificity. they are essential to explain the underlying assumption and mechanisms in the process of memory recall @xcite . ## madial-bench we simulated long-term dialogues between two users (a boy named bart and a girl named lisa) and a virtual assistant with memory information at various time points, in the form of event summaries. we evaluate the dialogue between children for their simple expression and direct emotional exposure. by sampling different topics and scenarios, we generated 171 historical memories and 160 dialogues in 1474 turns. the final madial-bench was created through prompt-based generation using gpt-4, followed by multiple manual refinement and revision rounds. ## experiments in this section, we detail the evaluation setups of top models, embedding models for memory retrieval, and llms for memory-injected response generation in the task of memory recognition and response generation. the response evaluation includes the use of both human evaluation measures over rich aspects and commonly used automatic evaluation approaches. ## results and discussion in this section, we discuss the performance of models in each task and aspect. the surprising relations between aspects are also dug out. later on, automatic evaluation results also show their disability in assessing llm outputs in this benchmark.