Mixed-Session Conversation with Egocentric Memory (2024)

Abstract: Recently introduced dialogue systems have demonstrated high usability. However, they still fall short of reflecting real-world conversation scenarios. Current dialogue systems exhibit an inability to replicate the dynamic, continuous, long-term interactions involving multiple partners. This shortfall arises because there have been limited efforts to account for both aspects of real-world dialogues: deeply layered interactions over the long-term dialogue and widely expanded conversation networks involving multiple participants. As the effort to incorporate these aspects combined, we introduce Mixed-Session Conversation, a dialogue system designed to construct conversations with various partners in a multi-session dialogue setup. We propose a new dataset called MiSC to implement this system. The dialogue episodes of MiSC consist of 6 consecutive sessions, with four speakers (one main speaker and three partners) appearing in each episode. Also, we propose a new dialogue model with a novel memory management mechanism, called Egocentric Memory Enhanced Mixed-Session Conversation Agent (EMMA). EMMA collects and retains memories from the main speaker’s perspective during conversations with partners, enabling seamless continuity in subsequent interactions. Extensive human evaluations validate that the dialogues in MiSC demonstrate a seamless conversational flow, even when conversation partners change in each session. EMMA trained with MiSC is also evaluated to maintain high memorability without contradiction throughout the entire conversation.

## introduction dialogue systems have been evolving along two dimensions: depth, for supporting long-term interactions, and width, for accommodating a greater 1 our dataset/code are publicly available at https:// mixed-session.github.io/ number of conversation partners. multi-session conversations @xcite @xcite have been proposed as an instance of such long-term dialogue systems retaining dialogue context across consecutive sessions. expanding the network of conversation partners, in the other dimension, includes multi-party conversations @xcite @xcite @xcite @xcite @xcite . it expands the scope of interactions by increasing the number of conversation partners engaged in a dialogue session. however, in the real-world, conversations occur within complex contexts that are both lengthy and deeply layered, involving a wide range of people. therefore, focusing on either of the two dimensions would not fully capture these dynamics. given this significance, there have been surprisingly few efforts to advance dialogue systems in both directions. to expand the boundaries of those dialogue systems, we introduce mixed-session conversa-tion. unlike multi-session conversations, where a speaker engages with one fixed partner across all sessions, the main speaker in mixed-session conversation encounters multiple partners in a mixed order of sessions. this approach is thus referred to as mixed-session. specifically, mixed-session conversation consists of multiple dialogue sessions, during which several speakers, including a main speaker, interact dynamically over time. the main speaker engages in conversations with different partners, one partner per session, focusing on a specific event. this setting enables a dialogue system to build a deep, layered context with each of its partners, thereby expanding and complicating the dynamics. session 1 egocentric memory session 2 doctor, i've been feeling quite anxious about the surgery tomorrow. can you talk me through what to expect? …… sarah and i have coordinated everything to ensure the best possible care for you. (referring memory △) we need to ensure that everyone on the team is aware of the crucial role emotional and psychological support plays in healthcare. our approach to mr. thompson's case should reflect our dedication to his overall well-being. (figure 1 ). misc comprises 8.5k episodes, with each episode consisting of 6 sessions (a total of 51k sessions). in each episode, four speakers participate, with one main speaker involved in all sessions and each of the other three speakers participating as a conversation partner. to enable the main speaker to retain all contexts across sessions and partners, we introduce a new memory managing system called egocentric memory. egocentric memory keeps memory about each partner from the main speaker's perspective, enabling accurate recall to align the events with each partner without contradiction. we actualize mixed-session conversa-tion through a novel dialogue model, named egocentric memory enhanced mixed-session conversation agent (emma). trained on misc, emma ensures seamless continuity during interactions between speakers leveraging egocentric memory. as the session progresses, the memory of each speaker is newly added or updated; thereby, all memory can be retained without losing information about the previous sessions and partners. through extensive human evaluation, the quality of misc and conversations generated from emma are verified to have high qualities. to be specific, misc exhibits high consistency and coherence throughout the episode, retaining accurate memory of each partner from the main speaker's perspective, even with conversation partners changing with each session. conversations from emma demonstrate high humanness, engagingness, and memorability. our contributions in this study are: ## misc we introduce mixed-session conversation, a novel dialogue setting that advances along both depth and width dimensions simultaneously. unlike previous systems, mixed-session conver-sation allows the main speaker to engage with different sub-speakers as conversation partners in each session. to implement this conversation system, we propose a new dataset called misc. misc comprises 8.5k episodes, each consisting of 6 sessions, totaling 51k sessions in all. in each episode, four speakers are involved, with one acting as the main speaker and the others as sub-speakers. we construct the dialogue dataset in a sequential manner, starting with the collection of topics and progressing to the generation of conversations. we use llms to build the dataset through elaborately designed methodologies (please refer to figure 2 for an overview of the process). ## emma we propose a novel dialogue model called emma. emma collects memories for each conversation partner from its own perspective in every session, ensuring seamless continuity in subsequent sessions. emma consists of two parts: (1) the dialogue module; (2) the retrieval module. an overview of emma's architecture is illustrated in figure 3 . ## experiments evaluating open-domain conversations poses a significant challenge. while metrics such as ppl, rouge @xcite , and bleu @xcite offer quantitative measures, they often fail to capture the contextual intricacies, emotional tone, and level of engagement within conversations. consequently, recent research in open-domain conversation increasingly leans towards human evaluation as the standard method @xcite @xcite @xcite @xcite . by employing human judgment, researchers can better assess the nuanced qualities of conversational systems, ensuring a more comprehensive understanding of their performance. given the significance of assessing the conversational flow in both misc and emma, we apply human evaluation as a quality verification method. we use misc to train emma, for more training details, please refer to appendix e. ## results in this section, we explain the evaluation results of misc and emma. please refer to section 5 for specific evaluation settings. conversation quality. as evidenced by the evaluation results, the high scores of memory links facilitate seamless tracking and utilization of memory, thereby enhancing the effectiveness of interactions between the main speaker and partners across the entire conversation. our egocentric memory seeks to enhance memory management by streamlining the process and maximizing the collaborative potential between speakers and partners, ultimately leading to more cohesive and diverse conversations. emma performance. the human evaluation results for 0.2k episodes generated through interactions among four instances of emma are presented in memory dynamics. comparison with other methods. emma possesses the ability to engage in conversations across multiple sessions, seamlessly adapting its dialogue to accommodate different partners. this adaptability is made possible through egocentric memory, which allows emma to manage memories tailored to each specific partner rather than storing generic memories. different from other memory management approaches, our emma stores and manages memories separately for each partner. this personalized approach ensures that emma can recognize changes in conversation partners and maintain consistency in dialogue, preventing the potential for misunderstandings or inconsistencies that might arise from a more generalized memory system. to verify this, we compare the existing multisession conversation models msc 2.7b @xcite , rebot @xcite , and emma. to examine whether the models can recognize a change in partners in subsequent sessions, we conduct a live chat, as shown in ## conclusion we introduce mixed-session conversation, a new dialogue system designed to incorporate long-term interactions and accommodate a wide range of speakers. mixed-session conversa-tion allows a main speaker to engage with different partners across multiple sessions, enabling the dialogue system to cover a more wide-layered context. unlike multi-session conversations with a fixed partner, mixed-session conversation feature interactions with various partners in mixed order. we also propose a new dataset called misc to implement mixed-session conversation. we develop emma, a new dialogue model trained via misc. emma collects memories for each partner with egocentric memory and utilizes them in subsequent sessions to maintain seamless continuity. extensive human evaluation demonstrates dialogues in misc maintain a natural flow across sessions even when the conversation partner changes. emma exhibits high memorability and engagingness in conversations by actively utilizing egocentric memory.