Memories for Virtual AI Characters (2023)

Abstract: In this paper, we present a system for augmenting virtual AI characters with long-term memory, enabling them to remember facts about themselves, their world, and past experiences. We propose a memory-creation pipeline that converts raw text into condensed memories and a memory-retrieval system that utilizes these memories to generate character responses. Using a fact-checking pipeline based on GPT-4, our evaluation demonstrates that the character responses are grounded in the retrieved memories and maintain factual accuracy. We discuss the implications of our system for creating engaging and consistent virtual characters and highlight areas for future research, including large language model (LLM) guardrailing and virtual character personality development.

## introduction memory is arguably one of the keys to personality, influencing how we think and respond to external stimuli @xcite . it is also pivotal for maintaining a coherent and engaging conversation @xcite . such characteristics of memory hold considerable implications for the field of natural language processing (nlp), particularly for ai chatbots, which most recently have emerged as a significant form of human-computer interaction (brown et al.; @xcite . recently, chat applications utilizing large language models (llms) have incorporated a type of short-term memory into their design, namely the chat history, which is included in the context window of the llm @xcite . however, the integration of long-term memory into these systems is not yet standard. one reason is that without supporting systems, long-term memory can only be achieved by frequently retraining the underlying llm, which is impractical due to prohibitive computational costs and associated emissions @xcite . most ai chatbots today are designed to function as assistants, parsing user inputs as commands or providing answers to queries, often relying on the support of information-retrieval systems (adamopoulou and moussiades). 1 however, envisioning a slight shift in this objective-using llms to generate responses for virtual ai characterspresents new challenges and opportunities. the incorporation of long-term memory becomes critical in this context. to ensure that a virtual character appears human-like to a user, it must accurately remember facts about its universe and recall past interactions. achieving this could elevate the user experience to a new level, creating a dynamic, immersive experience that adapts to the user @xcite . this paper presents a system to augment virtual ai characters with persistent, long-term memories. at a high level, we achieve this by generating character responses with an llm, augmenting the prompts with relevant memories-condensed versions of character experiences, knowledge, or descriptions. 2 figure 1 shows how memories can influence character responses. our exploration spans relevant related work, and we provide a detailed explanation of our system, encompassing retrieval systems, potential llm steering, memory creation, and forgetting mechanisms. we evaluate our system based on two characters: sherlock holmes, a well-known character present in the llm's training data and implicitly represented by the model's weights, and amina osei, a newly created character with no such representation. the evaluation focuses on the alignment between the character responses and the retrieved memories and uses a semi-automated fact-checking pipeline that leverages gpt-4. when answering, characters have access to a list of relevant memories, a description of themselves, and the chat history. section 3 explains the proposed system in detail. this paper makes several contributions. first, it demonstrates how llms can be substantially enhanced by integrating them into a more extensive system. second, it introduces a process of creating memories from unstructured text, simplifying the character-creation process. third, it outlines partitioning memories into multiple "buckets," offering greater flexibility and potential for dynamic story progression. fourth, it proposes a new querygeneration approach resulting in targeted retrieval of relevant memories. furthermore, it presents a forgetting model that mimics human memory patterns and allows customization of character memory traits. ## related work the proliferation of natural language (nl) interfaces in computer systems, including systems such as siri, alexa, and google assistant, has opened new avenues for intuitive interaction. typically, these interfaces focus on question answering, command parsing, and named entity recognition, assisting users in a variety of tasks @xcite . another strand of research centers around developing conversational chatbots capable of realistic user interactions. applications range from in-game non-player-characters (npcs) @xcite to virtual companions, as popularized in various science-fiction movies. 3 our work intersects these 3 popular movies featuring ai companions include her (2013) and blade runner 2049 @xcite . examples of busi-two streams of research, aiming to create virtual characters capable of recalling factual information about themselves and their past experiences, effectively giving them long-term memory. ## proposed system the overarching idea of the proposed system is to generate character responses by prompting a general-purpose llm with a dynamically created prompt based on the chat context and relevant memories. this section explains how different system components work together to create this prompt and how interactions are distilled into memories. figure 2 explains the system on a high level. a detailed description of the system's most central components and stages follows. note: for all of our core system components described in this section, we use gpt-3.5-turbo (brown et al.) as the base llm. foot_2 ## future work a promising extension to our current system is a "response routing" stage. this feature, not yet fully realized, could augment control over character responses. it aims to maintain character integrity by ensuring responses stay within their memory bounds during improvisation and thus forms a critical component of our theoretical model. response routing would decide the type of re-sponse to be generated. it could consider the chat history, the created query, and retrieved memories, determining which of n response generation strategies to use. it could generate normal responses (like explained in section 3.5) or evasive responses depending on memory relevance. we propose separating decision-making from response generation due to the reasoning limitations of current llms. for example, a simple conditional prompt such as, "if the memories are relevant, create a normal response. otherwise, create a response in which the character is clueless about the topic" has proven unreliable in our tests, sometimes leading to hallucinated responses if no relevant memories were retrieved. we further plan to investigate how memories could influence character personalities over time. future work will also improve the evaluation methodology, including a human evaluation. we are also making ongoing efforts to increase the naturalness of the interactions with the virtual characters. currently, the main problem is that they are purely reactive to user input. the interaction follows a strict pattern of questionanswer-question-answer, where the user has to keep the conversation going. future work will investigate how we can give virtual characters the ability to take initiative. they should be able to independently write messages to the user, potentially following some plan or narrative goal, as has been explored in previous works for other systems (lehman and leite; leite et al.; paetzel-prüsmann and kennedy). ## conclusion this paper introduced a system for crafting memory-enhanced virtual ai characters that can interact intelligently. these virtual characters are grounded in their universe by being able to remember facts about themselves, their world, and past experiences. the presented system, consisting of multiple knowledge sources, a vector search, and an llm, is designed to be flexible, making it possible to create interactive characters by giving them access to memories that can be created from any textual descriptions about them. we designed the character memories to be human-like, being strengthened if used and forgotten when not, making characters feel authentic. our evaluation, powered by a semiautomated fact-checking pipeline, showcased that the character responses were strongly grounded in the retrieved memories. we anticipate this work to spur further investigations into the integration of memory systems within virtual characters and how to further guardrail llms to avoid hallucinations. as a result, we envision a future where these characters can perform convincingly within their narrative bounds while maintaining the capacity to improvise intelligently.