Towards Effective Long Conversation Generation with Dynamic Topic Tracking and Recommendation (2024)

Abstract: During conversations, the human flow of thoughts may result in topic shifts and evolution. In open-domain dialogue systems, it is crucial to track the topics discussed and recommend relevant topics to be included in responses to have effective conversations. Furthermore, topic evolution is needed to prevent stagnation as conversation length increases. Existing open-domain dialogue systems do not pay sufficient attention to topic evolution and shifting, resulting in performance degradation due to ineffective responses as conversation length increases. To address the shortcomings of existing approaches, we propose EvolvConv. EvolvConv conducts real-time conversation topic and user preference tracking and utilizes the tracking information to evolve and shift topics depending on conversation status. We conduct extensive experiments to validate the topic evolving and shifting capabilities of EvolvConv as conversation length increases. Un-referenced evaluation metric UniEval compare EvolvConv with the baselines. Experimental results show that EvolvConv maintains a smooth conversation flow without abruptly shifting topics; the probability of topic shifting ranges between 5%-8% throughout the conversation. EvolvConv recommends 4.77% more novel topics than the baselines, and the topic evolution follows balanced topic groupings. Furthermore, we conduct user surveys to test the practical viability of EvolvConv. User survey results reveal that responses generated by EvolvConv are preferred 47.8% of the time compared to the baselines and comes second to real human responses.

## introduction the responses in open-domain dialogue systems are designed to align with the intentions of human users @xcite . due to the human flow of thoughts, human intentions and requirements evolve as the conversation progresses @xcite . therefore, topic evolving and shifting is necessary for dialogue systems to maintain a long and engaging conversation with users. recently, @xcite proposed a clustering system with a self-training autoencoder to detect indomain topics in an unsupervised manner, and @xcite proposed uncertainty and diversitybased sampling techniques to identify topics of interest from extracted topics efficiently. these recent works focus on identifying and reusing the topics discussed in the conversation with no scope for evolving the topics. @xcite adapted text generation models to generate responses that bridge the new topic to the topic of the previous conversation turn. this approach has scope for topic evolvement; however, it is very restrictive since it can only handle one-turn topic transitions and requires the next response as input. none of these approaches explicitly model user preferences. in this study, we overcome the shortcomings of previous studies by proposing evolvconv, which conducts dynamic topic tracking and user preference analysis for better topic evolving and shifting. specifically, evolvconv includes a topic-tracking module that captures implicit and explicit conversational cues, extracts discussed topics from the conversation, and user preferences for the topics. a graph structure is used to efficiently store the extracted topics, their relationships, and user preferences that serve as a conversation history tracker. evolvconv takes advantage of the graph structure to retrieve potential topics and user preferences that can be part of generated responses. however, these retrieved topics are already discussed in the conversation; therefore, evolvconv includes a topic recommender module that recommends novel topics aligning with the retrieved topics for better topic evolving and shifting. finally, a response generation module generates responses with the recommended topics that align with user preferences. figure 1 compares the conversation experience of evolvconv with the baselines. the conversation demonstrates that evolvconv can evolve topics and smoothly shift between related topics, keeping the conversation interesting and the user engaged. due to our proposed novel architecture, existing datasets cannot be used for training evolvconv. therefore, we propose new datasets to train modules of evolvconv. we train evolvconv on our proposed new datasets and evaluate on benchmark datasets to test the topic-shifting and evolving capabilities of evolvconv. specifically, we evaluate (1) the topic-shifting probability, (2) how well the topic evolves in responses generated, and (3) user preference modeling for long conversations. the experimental results show that evolvconv balances topic-shifting and evolving better than the baselines. specifically, evolvconv does not hastily shift topics during initial turns in the conversation, and topic shifting is done based on an understanding of user requirements as the conversation progresses. similarly, evolvconv provides sufficient discussion time for each topic and then smoothly evolves to new topics. we conduct user surveys to analyze the user preference modeling capabilities of evolvconv. the survey reveals that the responses generated by evolvconv are pre-ferred by the users for long conversations. overall, the experimental results confirm that the dynamic topic tracking and recommendation capabilities of evolvconv result in effective long conversation generation. to summarize, the following are the key contributions of this work. conversation topic and user preference tracking: understanding the topics of the conversation and user preferences for the topics can help generate effective and relevant responses. unsupervised studies in conversation topic extraction in open domain dialogue (odd) propose augmenting temporal relationship information between responses with tf-idf-based vector space model (adams @xcite or applying latent dirichlet allocation (lda) model for topic extraction @xcite . earlier supervised approaches trained logistic classifiers, support vector machines, and gated recurrent units @xcite to extract topics. recently, @xcite proposed a multi-agent ai system that follows question question-answering approach to query gpt-4 to extract topics in the task-oriented dialogue (tod) setting. @xcite proposed an unsupervised dialogue segmentation algorithm to split the dialogue passage into topic-concentrated fragments for dialogue comprehension. these studies do not focus on user preferences for the topics. several approaches @xcite @xcite in conversational recommender systems focus on understanding user preferences for items. these approaches interact with users by asking questions about their preference for items and processing user feedback to learn their preferences. to learn user preferences, @xcite uses gating modules, @xcite uses stochastic gradient variational bayesian (sgvb) estimator, @xcite propose clustering algorithm to cluster users with similar preferences. @xcite employ representation learning. @xcite propose reformulating user preferences as instruction tuning. we do not consider user feedback in this work; therefore, these approaches cannot be applied. more related to our work, @xcite trains llms to learn personalized post representation and construct a general user profile from the user's historical responses. similarly, @xcite trains llms by exploring the conditional relations underneath each post-response pair of the user to learn an implicit user profile from dialogue history. in this study, we design instructions to query tinyllama2 @xcite to extract topic and user preferences directly from conversation utterances following the odd setting. our proposed evolvconv does not restrict the topic search space and does not require any additional feedback or external knowledge. summarization and response generation for long conversations: current models, including large language models (llms), struggle to understand long conversation contexts, hindering the generation of responses for long conversations. to overcome this problem, several studies summarize long conversation texts since conversations always contain redundant texts, which make a limited contribution to the overall meaning @xcite . some approaches @xcite partition long conversations into fine-grained segments of equal size and apply compression-based language modeling techniques to compress the text. while others follow topic modeling techniques utilizing the topic shifts in the conversation for summarization. @xcite use key points in the paragraph to decode each sub-summary using a leader-writer network, @xcite improved by fixing the type of key points considered and using an mrc-based method to fetch segments. @xcite al. (2021) implicitly modeled topics through token-level salient correspondences. liu et al. (2021) modeled conversation utterances at the section level to ensure coherence in forming topic segments. chen and yang (2020) used multi-view attention to summarize, considering the topic view and stage view. these approaches do not utilize conversation summarization to generate responses. different from the above studies, han et al. (2024) proposes to capture the topic structure of the conversation as a seq2seq task and leverage it to guide the generation of the summary. zhong et al. (2022a) use llms as multi-level refiners to extract the most valuable tokens from dialogue history and leverage data from similar users to generate personalized responses. this study does not summarize the conversation; instead, we extract the conversation topics and user preferences from conversation utterances and utilize them to recommend novel topics to include in the generated responses. our proposed novel pipeline generates effective long conversations through smooth topic shifting and evolution. ## methodology given a conversation history c containing m conversation utterances, our goal is to generate a response r that best engages the user to continue the conversation. to achieve this goal, we propose evolvconv foot_0 , a multi-step framework incorporating topic shifting and evolution in response generation. given the conversation history c, we first extract the discussed explicit and implicit topics and user preferences for each topic from each conversation utterance. while extracting topics, we consider different levels of topic granularity to ensure a proper understanding of user preferences. for example, certain users may like soccer, while others may like a specific team or player. the user preferences take values in {n o, y es, u nknown}, representing dislike, like, and no explicit preference for each extracted topic of different granularity. we store the extracted conversation topics and user preferences as a graph. to enable effective topic shifting and evolution and prevent repetition of conversation topics, the topics relevant to current conversation utterances are extracted from the graph and provided as input to the recommender module that recommends topics to include in generated responses. the recommended topics can be novel, aligning with current conversation utterances and extracted user preferences. finally, the response generator generates responses incorporating recommended topics that align with the conversation history c. figure 2 provides an overview of evolvconv. ## experiments to test the topic shifting and evolving capabilities of evolvconv, we evaluated its performance on several benchmark datasets. we provide additional evaluation studies in appendix a. ## conclusion this work proposes evolvconv, a multi-step model that utilizes dynamic topic tracking and recommendation to perform topic shifting and evolution for effective long conversation generation. instead of storing the entire conversation history, evolvconv only stores topics and corresponding user preferences as a graph. then, the graph is utilized to retrieve tpp, which form the input to the recommender module that is responsible for topic shifting and evolution in the responses. finally, the response generator generates responses incorporating recommended topics and aligning with the conversation flow. through extensive experiments, we demonstrate the topic-shifting and evolving capabilities of evolvconv for long conversations, including the ability to model user preferences effectively.