Sequential Topic Selection Model with Latent Variable for Topic-Grounded Dialogue (2022)

Abstract: Recently, topic-grounded dialogue system has attracted significant attention due to its effectiveness in predicting the next topic to yield better responses via the historical context and given topic sequence. However, almost all existing topic prediction solutions focus on only the current conversation and corresponding topic sequence to predict the next conversation topic, without exploiting other topic-guided conversations which may contain relevant topic-transitions to current conversation. To address the problem, in this paper we propose a novel approach, named Sequential Global Topic Attention (SGTA) to exploit topic transition over all conversations in a subtle way for better modeling post-to-response topic-transition and guiding the response generation to the current conversation. Specifically, we introduce a latent space modeled as a Multivariate Skew-Normal distribution with hybrid kernel functions to flexibly integrate the global-level information with sequence-level information, and predict the topic based on the distribution sampling results. We also leverage a topic-aware prior-posterior approach for secondary selection of predicted topics, which is utilized to optimize the response generation task. Extensive experiments demonstrate that our model outperforms competitive baselines on prediction and generation tasks.

## introduction dialog systems have been widely used in a variety of applications. recent efforts in dialogue systems aim at improving the diversity of agent responses @xcite @xcite and endowing agents with the ability to exploit knowledge, express empathy and retain personality @xcite @xcite . however, in many real-world scenarios (e.g., conversa-tional recommendation, shopping guide and psychological counseling), conversational agents need to proactively steer the conversation by smoothly transforming the conversation topic into a specified one. therefore, topic-grounded controllable conversation has recently attracted extensive attention. indeed, topic-grounded controllable dialogue is a task to obtain informative responses through a series of predicted topics and the given context. recently, topic-grounded conversation works mainly focus on modeling the post-to-response topic transition for predicting the next topic to guide the response generation. many deep learning based approaches are proposed for the task, which utilize topic similarity information as well as additional knowledge information to model dynamic topic transitions @xcite @xcite @xcite . these approaches have achieved encouraging results, but they still face the issues as follows. first, some of these methods infer the next turn topic only with current turn topic embedding @xcite @xcite . however, a topic may be inherently associated with several previous topics. thus they may suffer from the inability of sufficiently modeling the sequential topic-transitions into the topic embeddings as ignoring the inherent topic dependencies over historical topic sequence. second, almost all previous approaches model the topic transferring information only over the current topic sequence while neglecting the useful topic-transition patterns from other sequences, since frequently co-occurring topics over all topic-sequences are more likely to related to each other. therefore, fully exploring such information is conceptually advantageous to accurately modeling the topic transition for better infer the next topic. in this paper, we propose a transformer-based sequential modeling approach, named sequential global topic attention (sgta), to fully exploit topic-co-occurrences over all topic sequences for better modeling the post-to-response topic transitions and accurately predicting the next topic to the current conversation. specifically, sgta consists of three key elements: topic sequence s, global co-occurrence matrix c and latent variable z. the relationships among these elements are elaborated with the graphical model in figure 1 . specifically, we propose a new latent space based on the value variation of the transformer and use it to model the contextual relation within current topic sequence. the latent space is modeled as a multivariate skew-normal (msn) distribution @xcite due to the flexibility of its covariance parameter to integrate multiple information and the specificity of its shape parameter to control the skewness of the distribution. the covariance parameter σ is designed via task-specific kernel functions for measuring the similarity of pair-wise sequences based on the topic representations. the shape parameter α is used to represent the unique relative relationships of different topics in a sequence. these two parameters are constructed efficiently using current sequence information and other sequences' global information. by sampling the msn distribution on transformers, we provide a reparameterization of the msn distribution to enable amortized inference over the latent space. based on the sampling results, the model simulates the topic transition relationships in the sequence for predicting the next topic in current conversation. to properly use predicted words and address fluency and controllability issues in generating responses, we leverage a response-based posterior topic distribution to instruct the model to generate responses based on the predicted topics. the prior and posterior ideas are widely used in response generation tasks @xcite , which is widely-adapted for avoiding to learn the same conversation patterns to each utterance. concretely, we utilize the kl-div loss between the prior and posterior distributions as part of the overall loss, which allows the model to better adapt to response generation by converging the transition scope of the semantic space. we summarize the contributions of this work as: topic transition conversation recently, several studies attempted to build agents that can actively guide conversations by introducing designated target keyword. tang et al. (2019) proposed a nextturn keyword predictor and a rule-based keyword selection strategy to solve the topic transition problem, given the conversation history and the target keyword. qin et al. (2020) improved (tang et al., 2019) by exploiting the semantic knowledge relation among candidate keywords to achieve smooth keyword transition. very closely, zhong et al. (2021) leverages commonsense knowledge for keyword transition prediction through gnnbased models. inspired by this category of work for controlled topic conversations, liu et al. (2020) first introduce goals in conversational recommen-bert sequence embedding position embedding current topic sequence entire topic sequences global co-occurrence matrix location covariance shape sequential modeling module multivariate skew-normal distribution z context response prior c posterior c concat sample approximate ffn prediction module decoder generation module kl-div loss {𝑡 𝑎 1 , 𝑡 𝑎 2 … , 𝑡 𝑎 𝑙 } {𝑡 𝑥 1 , 𝑡 𝑥 2 … , 𝑡 𝑥 𝑘 } topic word 𝜉 σ 𝛼 utterance 𝑡 𝑎 𝑙 +1 dation to enhance the controllability of the conversation process, and @xcite utilizes pre-trained language model(prlm) to capture the given current topic sequence to guide conversational recommendation. our work follows the task definition in @xcite , mainly focusing on topic transition and next-turn topic prediction problem in multi-turn dialogues, given the topic sequence. however, all previous approaches only model the topic-transition information on the current sequence. in contrast, our work learns topic transition information across all sequences to enhance the topic transfer modeling of the current sequence. topic-aware response generation most of early studies on topic-related dialogue fall into two categories, i.e., implicit topic-based @xcite @xcite and explicit topic-based @xcite @xcite . the former aims to model contexts at multiple semantic levels (i.e., topic, style, etc) to capture the dynamic conversation topic flow while ignoring the informativeness of responses. the latter generates appropriate utterances on the conversation utterances and given topic information, which relies heavily on manually predefined topic sequences. due to the excellent performance of the prlms on the dialog generation task, @xcite and @xcite choose to encode topic information and context by the prlm to obtain informative and fluent responses. however, it may suffer from topic-noise problem (i.e., raw topic fitting and similar topic selection) when generating responses. in contrast, our work leverages several topics to a prior distribution, using posterior information on already known topic to guide the generation of prior distribution that affects the generation process. ## methodology we propose a novel model for topic-guided dialogue. figure 2 presents the architecture of our model, which comprises three main components: 1) global sequential topic attention layer; 2) point-wise feed-forward network prediction layer; 3) prior and posterior response generation layer. we next present the three components and parameter modeling in detail. ## conclusion in this paper, we focus on topic-grounded controllable dialogue tasks. we propose a new approach named sgta. it models the latent space as a msn distribution utilizing global information, intra-sequence semantic and position information, which allows the model to better integrate the relationships between information and make topic predictions based on the results of distribution sampling. we also utilize a prior-posterior distribution approach to generate a new topic response. extensive experiments on tg-redial dataset show that sgta achieves state-of-the-art performance on prediction and generation tasks.