Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation (2023)

Abstract: Knowing how to end and resume conversations over time is a natural part of communication, allowing for discussions to span weeks, months, or years. The duration of gaps between conversations dictates which topics are relevant and which questions to ask, and dialogue systems which do not explicitly model time may generate responses that are unnatural. In this work we explore the idea of making dialogue models aware of time, and present GapChat, a multi-session dialogue dataset in which the time between each session varies. While the dataset is constructed in real-time, progress on events in speakers’ lives is simulated in order to create realistic dialogues occurring across a long timespan. We expose time information to the model and compare different representations of time and event progress. In human evaluation we show that time-aware models perform better in metrics that judge the relevance of the chosen topics and the information gained from the conversation.

## introduction as language models scale to unprecedented sizes, so too has their ability to generate reasonable and fluent responses to human dialogue. however, one limitation of large language models (llms) towards creating human-like dialogue lies in the fundamental distinction that human cognition is embodied, while llms are not. any considerations which stem from the human physical experience may not be well-represented explicitly in the training data dialogues of llms, as reiterating information known to both parties would violate the gricean maxim of quantity @xcite , and may not be effectively utilized during generation. in this paper we focus on one aspect of embodied experience -time -and the role that awareness of time plays in shaping realistic dialogue in multisession conversations. we argue that an awareness of the passage of time between conversations allows human speakers to more accurately gauge which topics and questions will result in new information, leading to conversations which are more informative and appear more natural. for example (figure 1 ), a dissertation is a time-consuming endeavor, so if given a two-hour gap between conversations, it is unlikely that significant progress has been made. in the context of potential information gain, asking for a progress update may be less useful than asking about an event with more expected progress within that time frame, such as a lunch or a meeting. to enable dialogue models to behave similarly, we propose incorporating an awareness of time as it pertains to (1) how much time has passed since the previous conversation, (2) what events were previously discussed, and (3) what progress of each event is expected over that duration. previous work introduced the concept of multisession chat (msc) and dataset for the task where the discussion between two speakers is divided with gaps between sessions @xcite . however, gaps between sessions are relatively short (1-7 hours, or 1-7 days), and event topics are derived from basic persona attributes (i have six cats) which prevents any discussion of long-term events or event progress. moreover, annotators vary between sessions, which may have further reduced consistency. to remedy these issues and support long-term msc research, we present gapchat, a dataset which extends the msc dataset with additional sessions foot_0 . however, while chat in msc is openended, conversations in gapchat are based on simulated timelines. this design choice allows for two participants to create a realistic and consistent conversation about long-term events, with gaps between sessions on the scale of days, weeks, or months, in a comparatively shorter period of time. we explore multiple ways of making dialogue models time-aware, using information such as the ongoing events of speakers, the expected duration of the events, and the duration of time gap between sessions. our contributions are as follows: (1) the creation of a new msc-style dataset that simulates explicit gaps in communication, across various time-scales, to enable research into long-term msc dialogue generation, (2) the exploration of different time-aware dialogue models, which incorporate time information into their contexts to enable varying degrees of time-based reasoning about discourse topics, and (3) we demonstrate via human evaluations of generated dialogues that the inclusion of time information improves the naturalness, informativeness, and relevance of conversations. ## gapchat: a time-aware msc dataset the msc dataset contains gaps between sessions and their durations are annotated. however, the gaps were short, and the annotators were changed between sessions. they were not instructed to pay attention to the relation between topics, events and the duration between gaps. t hus temporal information in msc has been shown to have a minimal effect on model performance @xcite . however, we hypothesize that the use of a dataset with discussions focusing on more realistic longterm events may have a more significant impact session 1 session 2 session 3 session 4 session 5 build a website (3 wks) 2 weeks 1 month 3 months 6 months learning spanish (6 months) move to a new apartment (2 wks) take a basic cookery course (2 wks) have a short trip (2 days) watch a movie (2 hrs) argentina won the championship of world cup a huge avalanche happened in austria ftx went bankrupt on model performance. to test this hypothesis, we create a new conversation dataset, gapchat. we follow the general settings in msc and similarly collect dialogue data via crowdsourcing. thus gapchat can also be used as additional data for any task utilizing msc. however, unlike msc, conversations in gapchat are not completely open-ended, but are grounded to the hypothetical lives of the two annotators, who are each given a procedurally generated timeline of events in their lives. this notion of a simulated life provides speakers with updates to events in their lives which are realistic with respect to the time which is said to have passed since the last chat session. sessions are then scheduled to take place randomly at various positions along the timeline, which may occur before, during, or after any particular event. ## modeling time we propose a method of modeling time in multisession dialogue systems. as shown in figure 4 , this consists of three procedures: 1) extracting on- i'm good. i'm just busy with my doctorate thesis... hey, how are you doing? ... prompt: extract events in the conversation ... extracted events: b: writing doctorate thesis, book reading event. a b progress label: b: writing doctorate thesis -> no significant progress. book reading event -> finished. i'll need to join a book reading event today. b prompt: estimate the duration ... how was the book reading event? a yeah, it was great. people shared a lot of interesting books. you should also go there next time ! b dialogue history: new session: extracting ongoing events computing events progress rag model time-aware dialogue generation going events, 2) computing events progress, and 3) time-aware dialogue generation. ## experiments models are separated into two groups, timeunaware models and time-aware models. human evaluation is conducted to compare all models against rag (ft). ## results & analysis 6.1 main findings results of the human evaluation are shown in between different types of time information, we find that progress labels contribute most to the enhancement of informativeness, but schedule information has relatively minor impact on the informativeness. this may be attributable to the more direct manner in which this approach provides information to the model, as the progress towards an event represents the outcome of a timebased reasoning process that is performed outside of the model. using the progress representation would closely align with a desired discourse action (e.g., do not select events with labels of "no significant progress"), whereas the schedule representation requires the model to perform additional reasoning. ## conclusion this work is a study of the role that temporal reasoning plays in shaping dialogues over multiple sessions, and the extent to which temporal information is useful when integrated into existing dialogue systems. we demonstrate via human evaluations that time-aware models generate text with improved naturalness, informativeness, and relevance, resulting better topic selection and improved text generation quality overall. the results emphasize the importance for future dialogue models to consider "time-awareness" as an important factor in achieving natural conversations. additionally, our analysis shows that incorporating time-aware information also enhances the performance of existing llms such as chatgpt.