Multi-Turn Target-Guided Topic Prediction with M onte C arlo Tree Search (2021)

Abstract: This paper concerns the problem of topic prediction in target-guided conversation, which requires the system to proactively and naturally guide the topic thread of the conversation, ending up with achieving a designated target subject. Existing studies usually resolve the task with a sequence of single-turn topic prediction. Greedy decision is made at each turn since it is impossible to explore the topics in future turns under the single-turn topic prediction mechanism. As a result, these methods often suffer from generating sub-optimal topic threads. In this paper, we formulate the target-guided conversation as a problem of multi-turn topic prediction and model it under the framework of Markov decision process (MDP). To alleviate the problem of generating sub-optimal topic thread, Monte Carlo tree search (MCTS) is employed to improve the topic prediction by conducting long-term planning. At online topic prediction, given a target and a start utterance, our proposed MM-TP (MCTS-enhanced MDP for Topic Prediction) firstly performs MCTS to enhance the policy for predicting the topic for each turn. Then, two retrieval models are respectively used to generate the responses of the agent and the user. Quantitative evaluation and qualitative study showed that MM-TP significantly improved the state-of-the-art baselines.

## introduction although impressive efforts have been made to integrate background knowledge into the conversation systems @xcite @xcite , existing open-domain conversation systems still suffer from creating generic response @xcite and struggle to perform engaging conversations @xcite . moreover, there exists a strong demand in real-world applications to integrate the goals and strategies into the open-domain conversation systems, to make them achieve some specific goals such as recommending an item or accomplishing nursing goals. faced with these problems, target-guided open-domain conversation @xcite has attracted increasing research attentions. different from traditional open-domain conversation, target-guided open-domain conversation requires the system to proactively and naturally guide the conversational thread, and end up with recommending a target item or mentioning a target word. existing studies @xcite @xcite usually resolve the task with a sequential of single-turn topic predictions and response generations. at each turn, the model firstly selects a topic from the candidate topic set based on the history context, and then retrieves response according to the selected topic. since the single-turn topic prediction mechanism has no ability to plan the topics in the future turns, greedy decision has to be made at each turn. as a result, these methods usually suffer from generating sub-optimal topic threads. figure 1 illustrates an example conversation between user and the kernel agent @xcite , which utilizes single-turn topic prediction model to select topics. at the third turn, the sub-optimal topic "music" was selected. though it is strongly relevant to the topic "listen" in the second turn, it is irrelevant to the final target topic "job". the example verified that the greedy decisions in the single-turn topic prediction cannot naturally guided the conversation to achieve the target. to deal with the issue, we propose to formulate target-guided conversation as a multi-turn topic prediction problem, and model it with markov decision process (mdp). in the mdp, the environment is responsible for collecting the conversational history as the states, and the conversational system agent is responsible for selecting action as topic for each turn. inspired by the reinforcement learning method of alphago zero @xcite , we utilize monte carlo tree search (mcts) to make a long-term planning by considering the topics in the future turns and then generate topic for the current turn. given a pre-defined target topic and a randomly selected start utterance, the proposed model, referred to as mm-tp (mcts-enhanced mdp for topic prediction), iteratively generates topic sequence and guides the conversation to achieve the target topic. at each turn, mcts is firstly utilized to enhance the raw policy and predict the topic of this turn. two retrieval models are then respectively employed to generate the responses of the agent and the user. in this way, the problem of generating sub-optimal topic threads could be alleviated by the mcts at a certain extent. we conducted experiments on two popular target-guided open-domain conversation benchmarks. quantitative results show that mm-tp outperformed the state-of-the-art baselines by achieving the target more accurately and providing more smooth topic transition. qualitative study also show that our mm-tp improved baseline methods by making long-term planning of the topics. the major contributions of the paper are three-fold: as shown in figure 2 , a multi-turn target-guided open-domain conversation system starts with randomly selecting a specific target topic and the start utterance (step 1) by the simulator. the user generates an appropriate response (step 2). then, the system repeats several conversational turns before achieving the ends. at each turn, the system first accesses to conversational history utterances and predicts a topic (step 3) satisfying both transition smoothness and target achievement. then the agent and user generate responses respectively according to the predicted topic (step 4 and step 2). during the conversation, the target word is only presented to the agent and is unknown to user. the system consists of two components which are topic prediction module and response generation module. formally, let's use a and x to denote the sets of candidate target topics and responses, respectively. following the practices in @xcite , each target topic @xmath0.e., an entity name or a common noun), and the candidate utterance set x is derived from the personachat corpus @xcite . suppose that the agent e starts a conversation (1st turn) with utterance x e 1 and its target topic is a * . the user retrieval model g u generates a response x u 1 . then, at each turn @xmath1, • • • , m}, the topic prediction module takes previous utterance context ## related work existing research of dialogue system can be broadly concluded as two categories, which are task-oriented dialogue systems and open-domain dialogue systems. task-oriented dialogue system aims to accomplish some pre-defined goals @xcite , conduct negotiation @xcite or perform symmetric collaborations @xcite . open-domain dialogue systems are designed to chat naturally with human and aiming to provide reasonable responses. previous work make efforts to improve response generation by developing novel neural networks and training on large-scale corpus @xcite @xcite . although the promising progresses have been achieved, these chat-oriented dialogue systems still struggle to a set of limitations such as dull or inconsistent responses @xcite . due to these limitations, a novel task named target-guided open-domain conversaion was proposed, which requires the system to proactively and naturally guide the topic thread by integrating goals and strategies. @xcite for the first time introduced this task and employed a simple target-guided strategy to attain smooth topic transition by turn-level supervised learning. @xcite further improved this work by capturing semantic or factual knowledge relations among candidate topics through a dynamic knowledge routing network. however, both these methods employ single-turn supervised learning to predict the topic of each turn according to the human annotated topic sequence. moreover, they only consider existing context and overlook the long-term planning of topics in next several turns. monte carlo tree search (mcts) enhanced mdp was firstly proposed in games @xcite @xcite and has been applied in other fields such as diverse ranking @xcite , name entity recognition @xcite and task-oriented conversation @xcite . in this paper, we apply mcts in open-domain conversation to generate topic sequence which is utilized to guided the conversation thread to achieve the target. ## conclusion in this paper, we formulate the target-guided conversation as a multi-turn topic prediction problem, and propose a novel approach called mm-tp to resolve this task. mm-tp formalizes the multi-turn topic prediction as sequential decision prediction problem, and models it with mdp. mcts is used to improve the raw policy by making a long-term planning of topics in next several turns and then selecting a topic for the current turn. the model parameters are learned by reinforcement learning. experimental results demonstrate that mm-tp outperformed existing baseline systems in terms of both the successful rate of achieving target and the topic transition smoothness.