Semantic Guidance of Dialogue Generation with Reinforcement Learning (2020)

Abstract: Neural encoder-decoder models have shown promising performance for human-computer dialogue systems over the past few years. However, due to the maximum-likelihood objective for the decoder, the generated responses are often universal and safe to the point that they lack meaningful information and are no longer relevant to the post. To address this, in this paper, we propose semantic guidance using reinforcement learning to ensure that the generated responses indeed include the given or predicted semantics and that these semantics do not appear repeatedly in the response. Synsets, which comprise sets of manually defined synonyms, are used as the form of assigned semantics. For a given/assigned/predicted synset, only one of its synonyms should appear in the generated response; this constitutes a simple but effective semantic-control mechanism. We conduct both quantitative and qualitative evaluations, which show that the generated responses are not only higher-quality but also reflect the assigned semantic controls.

## introduction dialogue generation systems with adequate artificial intelligence responses hold great potential for practical use. a decent human-computer dialogue system should generate coherent and informative responses based on human-provided posts @xcite . sequence-to-sequence models @xcite with long-short term memory @xcite or gated recurrent networks @xcite have demonstrated profound improvements in open-domain dialogue systems @xcite @xcite @xcite . however, these models often generate overly generic responses @xcite that are independent of the given posts due to the maximum-likelihood-estimation-based objectives. to improve the variety of the responses, recent studies usually use semantically conditioned lstm, relying on additional semantic indicators such as keywords to guide the decoding process. however, keywords typically appear repeatedly in generated utterances with this strategy. to address this, @xcite propose a special gate mechanism to reduce the influence of the keywords. however, since this design does not directly address the concern in the objectives, repeated keywords still often remain a problem in practice; we confirm this is in our experiments. to address this issue, in this paper, we introduce the semantically controlled and recorded lstm (scr-lstm) cell, which provides semantic guidance via reinforcement learning (rl) as well as a recording mechanism that records the existence of the desired semantics to ensure that the generated responses indeed include the given or predicted semantics; also, the desired semantics are not to appear repeatedly in the response. for the form of the assigned semantics we use synsets, which provide a more flexible semantic representation for practical use, and any lexical or knowledge taxonomy can be used to serve this role. for a given/assigned/predicted synset, only one of its covering synonyms should appear in the generated response. in addition, when synsets are used to semantically control the generated responses, the responses may indeed show the assigned semantics, but the responses could be not diverse enough, or the relation to the given posts may be tenuous, because the major goal of the model is to meet the semantic constraints. therefore, we add a conditional seqgan @xcite to assure that the generated responses are similar to true human responses and are related to the given posts while specifying semantics to avoid dull or repetitive responses. as with conventional gan @xcite , our conditional seqgan comprises a generator and a discriminator; however, with the proposed discriminator we seek to not only distinguish machine-generated utterances from humangenerated utterances but also distinguish postindependent from post-dependent utterances. the resulting additional seqgan architecture generates responses that are more related to the posts. ## conclusion in this work, to develop an effective semantic control mechanism, we propose the scr-lstm model with reinforcement learning to ensure that the desired semantics appear once and do not repeat. we also present a conditional seqgan to help generate more coherent and informative responses. results from both human and automatic evaluations show that the proposed models outperform other baselines and achieve the lowest repetition and absence percentages of the assigned synsets in the generated responses, proving that the proposed approach indeed produces high-quality responses under the desired semantic control. also, we prove that seq-gan is an essential part of enabling the model to generate more diverse and coherent responses. the proposed model leverages synsets to serve as the semantic guidance. to investigate the feasibility of our model in practical use, in this work, the assigned synsets are simply fetched from posts. however, the selection or prediction of the desired