Policy-Driven Neural Response Generation for Knowledge-Grounded Dialog Systems (2020)

Abstract: Open-domain dialog systems aim to generate relevant, informative and engaging responses. In this paper, we propose using a dialog policy to plan the content and style of target, open domain responses in the form of an action plan, which includes knowledge sentences related to the dialog context, targeted dialog acts, topic information, etc. For training, the attributes within the action plan are obtained by automatically annotating the publicly released Topical-Chat dataset. We condition neural response generators on the action plan which is then realized as target utterances at the turn and sentence levels. We also investigate different dialog policy models to predict an action plan given the dialog context. Through automated and human evaluation, we measure the appropriateness of the generated responses and check if the generation models indeed learn to realize the given action plans. We demonstrate that a basic dialog policy that operates at the sentence level generates better responses in comparison to turn level generation as well as baseline models with no action plan. Additionally the basic dialog policy has the added benefit of controllability.

## introduction open-domain dialog systems have typically been modeled using end-to-end approaches, more specifically encoder-decoder architectures @xcite @xcite . these seq2seq models are commonly trained on a maximum likelihood objective, which leads to repetitive and uninformative responses @xcite . as seen in figure 1 , candidate a is a typical generic response given the dialog context. in order to deal with this problem, previous work proposed grounding generated responses on knowledge sentences related to the ... speaker 1: right. teams do all kinds of things to bother the competition. i've heard of teams having heated benches in the winter for themselves but not for the visitors. speaker 2: i would hate a cold bench. then again, i wouldn't want to be some place that cold or watching football. ## dialog policy our proposed pd-nrg approach has two parts: a dialog policy that determines the action plan based on the dialog context, and a response generation model that takes the action plan and the dialog context as input to generate a response. the dialog policy has components that predict the individual elements of the action plan: knowledge selection and dialog act planning. knowledge selection determines the knowledge to be integrated in the response by finding sentences from a knowledge document corpus that are relevant to the dialog context. dialog act (da) planning determines the style of the response in the form of das to be realized. we have two forms of da planning methods: knowledge-dependent da planning and knowledge-independent da planning. figure 2 depicts the architecture of pd-nrg. @xcite . ## policy-driven response generation as shown in figure 2 , at a given turn in the dialog context, the goal of the response generator is to realize the action plan output by the dialog policy. our proposed models generate the next turn based on the action plan at a sentence-level, in a sequential manner as opposed to at a turn-level. as shown in figure 3a when decoding each sentence of the next turn, the dialog context d j as well as the previous sentences generated for the next turn till that iteration are used as input. algorithm 1 shows the process for sentence-level generation. as seen in the algorithm all the attributes within the ap are jointly taken in as input. to jointly condition on the action plan, each attribute is concatenated to the dialog history as shown in figure 3c . in the training process each dialog act label is a separate token in the vocabulary and has its own embedding vector which is initialized randomly and learned during training. to train our model we represent the knowledge sentence and topic label with the pretrained embeddings from the gpt model whose vocabulary is bpe tokenized. finally, the use-knowledge flag decides whether or not to include the knowledge embeddings as part of the input. in some of our experiments, we also include the dialog acts for the past turns by concatenating each turn in the dialog history with its respective acts. ## conclusions in this work, we propose a policy-driven neural response generation approach for knowledge grounded open-domain dialog systems. we estimate an action plan that consists of a set of attributes that control the content and style of the generated responses at the turn and sentence levels. we investigate both manual and machine learning based policies. through human evaluation, we empirically demonstrate that a basic dialog policy that does sentence level generation outperforms turn level generation, as well as knowledge-grounded response generation baselines. furthermore, the generated responses realize their respective action plans. this allows builders of dialog systems control over the model's responses allowing for more consistent user experiences. our future work includes investigation of better approaches for learning such dialog policy models along with adding other attributes such as sentiment.