T op KG : Target-oriented Dialog via Global Planning on Knowledge Graph (2022)

Abstract: Target-oriented dialog aims to reach a global target through multi-turn conversation. The key to the task is the global planning towards the target, which flexibly guides the dialog concerning the context. However, existing target-oriented dialog works take a local and greedy strategy for response generation, where global planning is absent. In this work, we propose global planning for target-oriented dialog on a commonsense knowledge graph (KG). We design a global reinforcement learning with the planned paths to flexibly adjust the local response generation model towards the global target. We also propose a KG-based method to collect target-oriented samples automatically from the chit-chat corpus for model training. Experiments show that our method can reach the target with a higher success rate, fewer turns, and more coherent responses.

## introduction human-like dialog agents have three types of approaches: open-domain @xcite , task-oriented @xcite @xcite , and target-oriented dialog @xcite @xcite . the open-domain dialog only requires the dialog generation to be fluent and context coherent. in contrast, typical taskoriented dialog further completes a specific task by understanding users' intention and collecting the required information of predefined sub-tasks of the intention. however, as a more challenging task, target-oriented dialog aims to achieve a global target that often can not be clearly defined as subtasks. the dialog agents are required to lead the conversation to the target flexibly, and the process is excepted to be coherent, effective, and successful. due to its purpose and flexibility, target-oriented dialog agents have a broad-based demand, e.g., conversational recommendation @xcite ; kang * corresponding author. to reach a target topic effectively and coherently in dialog, existing approaches primarily represent the topic as keywords and adopt a two-stage architecture, i.e., predicting a next-turn keyword and keyword-augmented response retrieval @xcite . in this direction, @xcite further introduces reinforcement learning with "target similarity" rewards to target-oriented dialog learning. however, the target-oriented dialog is a typical knowledge-rich task. although dialog context can support the semantic concern of dialog generation, it is not quite effective to model the knowledgedriven process in the target-oriented dialog. to involve global knowledge, @xcite and @xcite incorporate a dialog graph into the target-oriented dialog and @xcite uses the external commonsense kg (conceptnet @xcite ) to improve the performance. although existing target-oriented dialog works have demonstrated practical approaches in selfsimulation test, there is still some open issues: (1) lack of multi-turn target-oriented dialog corpus for training and benchmarks. most existing targetoriented corpus are prepared for next-turn local target (e.g., otters @xcite ), or adopt chit-chat corpora and randomly select a keyword in the next-turn utterance as the local target, (2) lack of global planning of dialog process. although the latest works use a global target to guide every turn of response generation, they adopt a short-sighted and greedy strategy instead of global planning to optimize the process towards the global target. to this end, we propose target-oriented dialog with global planning on knowledge graph (topkg), which effectively supports the target-oriented process by global reasoning on kg concerning the dialog context. specifically, to address the first data issue, we automatically select a new dataset named target-guided convai (tgconv) from the chit-chat corpus convai2 @xcite . we select target-oriented samples from con-vai2 by identifying the dialog utterances containing a go-through entity sequence that aligns with the kg path. furthermore, we distinguish the selected dialog samples according to whether the global target is easy to reach or not to verify the performance of topkg in dealing with hard global target-oriented cases. for instance, the sample in the left part of figure1 is target-oriented because the keywords in this dialog are connected (direct or low-order connected) in a commonsense kg, which embodies a smooth transition towards global target words. to address the second issue, we first improve the existing one-turn target-oriented response generation, trained in a supervised fashion to predict a next-turn keyword and generate a fluent and coherent response with the predicted keyword. using the improved one-turn model as local-model, we further introduce a reinforcement learning based global-model to effectively guide the local-model towards a global target with global planning on kg. specially, the global-model adjusts the nextturn keyword selection of the local-model to follow the global planning path on kg and reward the keyword-based response generation with success in reaching the global target. our main contributions are as follows: the dataset can be downloaded in data folders from https://github.com/yyyyyyzt/topkgchat 2 related work target-oriented dialogue systems. current targetoriented dialog studies can generally be divided into local-target oriented and global-target oriented methods. local-target oriented methods @xcite pays attention to the next-turn target. for example, @xcite proposes a hierarchical policy model to plan and generate responses of different levels where the high-level policy plans a topic. however, the low-level policy plans responses that are coherent to this topic instead of approaching it. global-target oriented methods @xcite uses global target to guide every turn of response generation. these methods propose a keyword predictor to determine the next-turn keyword to talk about and produce a response relevant to the determined keyword. however, they adopt a short-sighted and greedy strategy instead of explicit planning to optimize the process towards the global target. kg-grounded dialogue systems. leveraging background information for dialogue system improvement is a well-researched topic, especially in target-oriented settings. some work uses structured knowledge, dkrn @xcite incorporates a dialog graph, and ckc @xcite uses the conceptnet to improve the performance. for how to utilize kg, classical methods are divided into using full path @xcite and using flexible path fragments @xcite . these models enjoy rich knowledge augmentation since short kg paths relating to the context are encoded, but they lack the ability to plan on kg. another set of works focuses on grounds in unstructured knowledge @xcite , which can also be divided into independent sentences and documents. this unstructured knowledge is more challenging to use than kg. ## conclusion and future work we propose effectively guiding the target-oriented dialog towards a global target with global planning on kg. we first design a novel method to automatically select target-oriented samples from the chit-chat corpus by identifying kg reasoning paths throughout the dialog. we train a reinforcement learning model with a selected high-quality corpus that can guide a gpt-2 based response generation model to reach a global target word by global planning on conceptnet. automatic and human evaluations show that our method exceeds the baselines from both local and global views, and global planning provides a significant contribution. we will explore to balance the coherence and number of turns in global planning in future work. two situations. one is that in the face of some uncommon words, even if there is a correct path, the generative model can only generate sentences with similar semantics. the other is that the path is very long and it is difficult to generate replies that contain relevant words. start: dye hard target: volley context a: hi i dye my hair 4 times a year. b: do you live in the sky ? i do . in the clouds. topkg a: i dye it dark brown b: do you also like the martant? it is my favorite. a: yes i like it because it is on the ground b: do you have a favorite field? ....(can not reached) path: dye-mordant-ground-volley start: mowing hard target: ethicist context a: i also enjoy lawn mowing on sunny days . b: i enjoy green living and just put solar panels on roof topkg a: that's awesome. solar roofs are awesome. b: yes they're. are you good at math a: i am excellent at it. mornings mornings are like night. b: i like the night a: it is my favorite part b: what is your favorite part about it? ....(can not mention related words and can not reached) path: mowing-morning-forenoon--part-particular-ethicist