Conversational Graph Grounded Policy Learning for Open-Domain Conversation Generation (2020)

Abstract: To address the challenge of policy learning in open-domain multi-turn conversation, we propose to represent prior information about dialog transitions as a graph and learn a graph grounded dialog policy, aimed at fostering a more coherent and controllable dialog. To this end, we first construct a conversational graph (CG) from dialog corpora, in which there are vertices to represent “what to say” and “how to say”, and edges to represent natural transition between a message (the last utterance in a dialog context) and its response. We then present a novel CG grounded policy learning framework that conducts dialog flow planning by graph traversal, which learns to identify a what-vertex and a how-vertex from the CG at each turn to guide response generation. In this way, we effectively leverage the CG to facilitate policy learning as follows: (1) it enables more effective long-term reward design, (2) it provides high-quality candidate actions, and (3) it gives us more control over the policy. Results on two benchmark corpora demonstrate the effectiveness of this framework.

## introduction how to effectively learn dialog strategies is an enduring challenge for open-domain multi-turn conversation generation. to address this challenge, previous works investigate word-level policy models that simultaneously learn dialog policy and language generation from dialog corpora @xcite . but these word-level policy models often lead to a degeneration issue where the utterances become ungrammatical or repetitive @xcite . to alleviate this issue, utterance-level policy models have been proposed to decouple policy learning from response generation, and they focus on how to incorporate take care of yourself when doing a very hard work. ## related work policy learning for chitchat generation to address the degeneration issue of word-level policy models @xcite , previous works decouple policy learning from response generation, and then use utterance-level latent variables @xcite or keywords @xcite as rl actions to guide response generation. in this work, we investigate how to use prior dialog-transition information to facilitate dialog policy learning. knowledge aware conversation generation there are growing interests in leveraging knowledge bases for generation of more informative responses @xcite @xcite @xcite @xcite . in this work, we employ a dialog-modeling oriented graph built from dialog corpora, instead of a external knowledge base, in order to facilitate multi-turn policy learning, instead of dialog informativeness improvement. specifically, we are motivated by @xcite . the method in @xcite has the issue of cross-domain transfer since it relies on labor-intensive knowledge graph grounded multiturn dialog datasets for model training. compared with them, our conversational graph is automatically built from dialog datasets, which introduces very low cost for training data construction. furthermore, we decouple conversation modeling into two parts: "what to say" modeling and "how to say" modeling. it is reasonable to only adjust the "what-" part when transfer to different domains which further reduces the domain transfer cost. ## our approach the overview of cg-policy is presented in figure 2 . given a user message, to obtain candidate actions, the nlu module attempts to retrieve contextually relevant subgraphs from cg. the state/action module maintains candidate actions, history keywords that selected by policy at previous turns or mentioned by user, and the message. the policy module learns to select a response keyword and a responding mechanism from the above subgraphs. the nlg module first encodes the message into a representation using a message encoder and the selected mechanism, and then employs a seq2bf model 2 @xcite to produce a response 2 it decodes a response starting from the input keyword, and generates the remaining previous and future words subsequently. in this way, the keyword will appear in the response. with the above representation and the selected keyword as input. the models used in cg construction/policy/nlg/reward are trained separately. ## conclusion in this paper we present a novel graph grounded policy learning framework for open-domain multiturn conversation, which can effectively leverage prior information about dialog transitions to foster a more coherent and controllable dialog. experimental results demonstrate the effectiveness of this framework in terms of local appropriateness, global coherence and dialog-target success rate. in the future, we will investigate how to extend the cg to support hierarchical topic management in conversational systems. b： 晚饭吃了么 [did you have dinner?] u： 我想吃排骨了，晚上炖排骨吃吧 [i want to have spareribs. let's stew spareribs in the evening.] b： 我请你吃猪蹄吧，我也要吃 [i'll treat you to pig hooves. i'll eat them, too.] u： 炸猪蹄很好吃啊 [fried pig hooves are delicious.] b： 这是什么类型啊，你吃过么？[what kind of food is it? have you eaten it?] u： 我还是吃炸酱面吧，最近减肥 [i'd better eat fried sauce noodles. i've lost weight recently.] b： 正宗了，我也想吃 [it's authentic. i want to eat it, too.] u： 我想跟你一起跳舞 [i want to dance with you.] b： 好萌啊，好可爱，好喜欢 [how budding! you are lovely and i like it.] u： 我想起我家那条狗了，泰迪狗，给你看看照片 [i think of my dog, teddy. show you the picture.] b： 我想养狗了，我也要养一只 [i want to have a dog. i want to have one, too.] u： 我想回家吃狗肉 [i want to go home and eat dog meat.] b： 可怜的泰迪啊，哈哈哈 [poor teddy, haha] u： 我不是要回家吃我的泰迪！[i'm not going home to eat my teddy!] b： 昨天去菜市场买了一大袋土豆，累死我了 [yesterday i went to the vegetable market and bought a big bag of potatoes. i am exhausted.] u： 话说楼下新开了个超市 [i heard that there is a new supermarket downstairs.] b： 我去了， 不打折了，哈哈 [i've been there. no discount yet, haha.] u： 里面的水果新鲜么 [is the fruit fresh?] b： 我都没看出是有多难吃 [it's untasty, even more than i expected.] u： 刚开业就不打折,真抠门,我去淘宝买吧 [it's really stingy that have no discount just after opening. i'll go to taobao to buy fruit.] b： 在哪里买啊，低价代购？求地址 [which shop? substitute purchase at low price? give me the link address.] u： 很多店铺吧, 搜一下就好了 [a lot of shops can be your choice. just search them.] b： 万能的淘宝网店，天猫要火啊 [taobao shops are really omnipotent, tianmao is becoming more and more famous] u： 双十一要到了，马云赚钱好开心 [the double eleventh shopping festival is coming. ma yun is happy to earn lots of money.] b： 我去年双十一购买了5件商品 [i bought five items last year in the double eleventh shopping festival.] u： 美死了,哈哈,好可爱,哈哈 [haha, you are lovely, haha] b： 你也很可爱 [you're lovely, too] u： 我想买一只萨摩耶，好可爱 [i want to buy a samoa. it's cute.] the red words are keywords. we translate the original chinese utterances into english.