P er LTQA : A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Fusion in Question Answering (2024)

Abstract: In conversational AI, effectively employing long-term memory improves personalized and consistent response generation. Existing work only concentrated on a single type of long-term memory, such as preferences, dialogue history, or social relationships, overlooking their interaction in real-world contexts. To this end, inspired by the concept of semantic memory and episodic memory from cognitive psychology, we create a new and more comprehensive Chinese dataset, coined as PerLTQA, in which world knowledge, profiles, social relationships, events, and dialogues are considered to leverage the interaction between different types of long-term memory for question answering (QA) in conversation. Further, based on PerLTQA, we propose a novel framework for memory integration in QA, consisting of three subtasks: Memory Classification , Memory Retrieval , and Memory Fusion , which provides a comprehensive paradigm for memory modeling, enabling consistent and personalized memory utilization. This essentially allows the exploitation of more accurate memory information for better responses in QA. We evaluate this framework using five LLMs and three retrievers. Experimental results demonstrate the importance of personal long-term memory in the QA task

## introduction long-term memory is a crucial element in conversational communication, facilitate the consistent and personalized response generation @xcite . previous studies, as shown in ## related work the long-term memory differentiation is mirrored in the datasets like @xcite @xcite . in the realm of question answering, natural-qa @xcite and coqa @xcite both target wikipedia-based knowledge, exemplifying the use of world knowledge as semantic memory. within dialogue tasks @xcite , msc @xcite and dulemon @xcite consider dialogues as episodic memory. memory-bank @xcite introduces a bilingual dataset using gpt-4 to summarize dialogues and personal data, effectively simulating episodic memory in multi-turn dialogues. however, existing datasets @xcite lack comprehensive coverage of both memory types with detailed annotations on social relationships and events, highlighting a research gap for llms in personal long-term memory fusion. efficient retrieval methods for external mem-profile (semantic memory) ... events (episodic memory) dialogue (episodic memory) social relationships (semantic memory) name: 王小明 (wang xiaoming) sex: 男 (male) nickname: 明明 (mingming) age: 28 occupation: 软件工程师 (software engineer) hobby: 摄影，打篮球 (photography, basketball) appearance: 短发，戴眼镜 (short hair, wearing glasses) education background: 本科毕业于计算机 科学与工程 (undergraduate computer science and technology major graduation) … supporting character: 王小红 wang xiaohong description: 王小明的姐姐 (wang xiaoming's sister, 36 years old, is a doctor) relationship: 姐姐(sister) … summary: 探索大峡谷 (explore the grand canyon) topic: 家庭旅行 (family trip) characters: 王小红，王小明 ( wang xiaohong, wang xiaoming ) time: 2022年5月12日 (may 12, 2022) content: 王小明和姐姐王小红打 算去美国亚利桑那州探索大峡谷。 (wang xiaoming and his sister wang xiaohong decided to explore the grand canyon in arizona, usa) … date: 2022-05-18, dialogues: : 你上次和王小红去亚利桑那州探索大峡谷感 觉怎么样？(how did you and wang xiaohong go to the grand canyon in arizona last time? ) : 非常棒 (very nice!) : 你们拍了什么好看的照片吗？(did you take any good photos? ) :我用相机拍下了北卡尼诺峡谷的照片，姐姐 也用手机记录了很多美好的瞬间。(i took photos of north canino canyon with my camera, and my sister also recorded many beautiful moments with her phone.) … step 2: generate profiles from seed data step 3: generate relationships for characters from step 2 step 4: generate events for characters from step 1, 2, 3 step 5: generate dialogues between assistant and characters from step 4 step 1: collect seed data step 6: validation names, hobbies, occupation, education backgrounds, topics ory in dialogue system fall into two main categories: sparse retrieval method like bm25 @xcite and vector-based retrieval method like dpr @xcite , contriever @xcite . the use of retrieval-augmented generation (rag) is increasingly enhancing retrieval tasks within large language models (llms). within this framework, fine-tuned embeddings are employed for text similarity searches, such as re-plug @xcite , openai embeddings foot_0 . this integration helps generate context-aware responses that consider personal memory, thereby improving the interaction quality in systems like those documented in recent studies @xcite and platforms like langchain foot_1 and lla-maindex 4 . with the aim of integrating the memories recovered in the responses, llms provide the consistent response generation method based on prompts @xcite @xcite @xcite . in dialogue systems, this approach incorporates memory directly into prompts, generating tailored responses that reflect individual memory @xcite @xcite . ## dataset collection we detail the creation of the perltqa dataset, which involves collecting perlt memories and generating and annotating perlt qa pairs. using an in-context technique, we build a memory database that encompasses profiles, social relationships, world knowledge, events, and dialogues. we then semi-automatically annotate components of memory-based q&a, including questions, answers, reference memories, and memory anchors that connect answers to their respective memories. ## conclusion our study introduces the perltqa dataset, which includes a memory database and memory-based question-answer pairs, covering personal longterm memory such as profiles, social relationships, events, and dialogues, categorized into semantic and episodic types. we outline three subtasks-memory classification, retrieval, and fusion-and report baseline experiments involving five large language models (llms) and three retrievers. our findings indicate that bert-based classifiers excel at categorizing memory types compared to other llms. additionally, we observe significant variances among llms in producing accurate memory-based responses. we also discover that enhancing personalization and consistency in responses requires integrating the unique characteristics of various memory types with those of different retrieval models. future research should focus on refining retrieval models to better manage complex memory structures and on minimizing irrelevant noise in the context, thus improving the quality of responses generated by llms.