History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System (2022)

Abstract: With the evolution of pre-trained language models, current open-domain dialogue systems have achieved great progress in conducting one-session conversations. In contrast, Multi-Session Conversation (MSC), which consists of multiple sessions over a long term with the same user, is under-investigated. In this paper, we propose History-Aware Hierarchical Transformer (HAHT) for multi-session open-domain dialogue. HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context and generate well-informed and context-relevant responses. Specifically, HAHT first encodes history conversation sessions hierarchically into a history memory. Then, HAHT leverages historical information to facilitate the understanding of the current conversation context by encoding the history memory together with the current context with attention-based mechanisms. Finally, to explicitly utilize historical information, HAHT uses a history-aware response generator that switches between a generic vocabulary and a history-aware vocabulary. Experimental results on a large-scale MSC dataset suggest that the proposed HAHT model consistently outperforms baseline models. Human evaluation results support that HAHT generates more human-like, context-relevant, and history-relevant responses than baseline models.

## introduction open-domain dialogue systems, also known as chatbots, are designed to converse with and engage users on any topic with the aim of establishing, maintaining, and strengthening long-term relationships @xcite . recently, open-domain dialogue systems built based on large-scale generative pre-trained models @xcite @xcite have substantially improved the performance of chatbots. however, most existing chatbots are designed to interact with users in a single conversation session. when the current session ends, the chatbot forgets its contents and will commence a new independent session with the same user next time. when previously discussed topics reemerge, such chatbots often appear ignorant and fail to reengage users appropriately. the apparent forgetfulness limits the chatbots' ability to establish and maintain long-term relationships with users. we argue that, to better engage users in multisession conversations (mscs), a chatbot should maintain a long-term memory of historical contexts, which allows the chatbot to reengage the user appropriately when similar contexts reemerge. by learning from historical conversations, the chatbot should gradually refine its understanding of and deepen its relationship with the user. figure 1 shows an example of a two-session conversation between a user and a chatbot. in the second session, the chatbot infers that sonny is a cat and generates the response based on the history information that sonny likes watching tv with the user. history-aware chatbots will be able to generate more well-informed and context-relevant re-sponses, which can help to elicit long-term commitments and develop emotional attachments from users to sustain close relationships over time. to this end, we propose the history-aware hierarchical transformer (haht) for multi-session opendomain dialogue systems, which can effectively leverage history conversations to conduct more engaging mscs. haht maintains a long-term memory to store historical conversational contexts, which is updated when a new session is conducted. based on the long-term memory and the context in the current session, relevant tokens in historical contexts are selected to adapt the current response. specifically, as the number of tokens in a conversation utterance and the number of turns in a conversation are usually not very long foot_0 , we first encode the history conversation hierarchically into the history memory using transformer @xcite . the history memory serves as a high-level representation of history conversations. secondly, as history conversations usually can facilitate the understanding of the current conversation context, we design a history-aware context encoder. the context encoder encodes conversation context, considering both history conversations and the current conversation, by adopting the transformer attention over the history memory and current conversation context. then, the context encoder also updates the history memory based on the current conversation context. finally, we design a history-aware decoder to fuse learned history information into the response generation process. the history-aware decoder can switch between two strategies, i.e., generating a word from the generic vocabulary or directly copying a word from history conversations. experimental results on the large-scale facebook msc dataset show that the proposed haht model outperforms previous multi-session opendomain dialogue systems in various evaluation metrics. human evaluation results support that haht generates more readable, context-relevant, and history-relevant responses than baseline models. in addition, the ablation study confirms that both the hierarchical encoding of history conversations and the history-aware decoder contribute greatly to haht's performance on mscs and help it leverage historical information more effectively. their methods need to retrieve a very large portion of history conversations to achieve better results than the standard transformer. in addition, these models still need to concatenate the retrieved raw history conversation text with the current conversation context, yielding concatenations that are still much longer than the 128 token truncation lengths. therefore, the incorporation of historical contexts in these methods is still limited by the short token truncation lengths of pre-trained models. ## the proposed method in general, a multi-session conversation (msc) consists of a current conversation session and several history conversation sessions that happen before the current one, all between the same two interlocutors. a multi-session open-domain dialogue system aims to generate natural, well-informed, and context-relevant responses to the user's utterances based on all history conversation sessions and the current conversation context. formally, we denote the msc dataset d by a list of n conversations in the format of (h, x, y). here, @xmath0, x 2 , • • • , x nx } denotes n x context utterances of the current conversation session. @xmath1, h 2 , • • • , h m } denotes m history conversation sessions, where ## experimental settings in this section, we introduce the experimental dataset, evaluation metrics, baseline methods, and model settings. ## experimental results this section presents the experimental results of the automatic evaluation, human evaluation, evaluation on session openings, ablation study, and case study. ## conclusion in this work, we propose the history-aware hierarchical transformer (haht) model for multisession open-domain dialogue systems. the proposed haht model maintains a history memory by hierarchically encoding the history conversation sessions. after that, haht uses attention-based encoding to encode the current conversation context together with the history memory and updates the history memory with the current context. in order to explicitly leverage historical information in the responses, haht is designed with a history-aware response generator which can switch between a generic vocabulary and a history-aware vocabulary. experimental results obtained under both normal and session opening msc settings demonstrate that history conversation sessions [session 1] ... user: