TSDG : Content-aware Neural Response Generation with Two-stage Decoding Process (2020)

Abstract: Neural response generative models have achieved remarkable progress in recent years but tend to yield irrelevant and uninformative responses. One of the reasons is that encoder-decoder based models always use a single decoder to generate a complete response at a stroke. This tends to generate high-frequency function words with less semantic information rather than low-frequency content words with more semantic information. To address this issue, we propose a content-aware model with two-stage decoding process named Two-stage Dialogue Generation (TSDG). We separate the decoding process of content words and function words so that content words can be generated independently without the interference of function words. Experimental results on two datasets indicate that our model significantly outperforms several competitive generative models in terms of automatic and human evaluation.

## introduction with the development of deep learning, the opendomain neural response generation has achieved remarkable progress @xcite @xcite in resent years. at present, most of generative models are based on encoder-decoder framework @xcite . in the decoding process, these models always use a single decoder to generate the final response at a stroke in a left-to-right manner. however, we find it hard for these methods to model the dependency of semantic between post and response which causes irrelevant and uninformative responses. we analyze this problem from the perspective of linguistics as following. in linguistics, there are two different types of words to form a sentence, namely content words (words which have substantive lexical content) and function words (words which essentially serve to make grammatical properties) @xcite . for the response "i am going to read an interesting book." in figure 1 , content words "read, interesting, book" give us the most important semantic information which establishes the semantic dependency with the post, while function words "i, am, going, to, an" are used to stitch content words together. highquality content words are a critical component of a relevant and informative response. although function words are small in numbers (less than 0.04% of our vocabulary), they account for over half of the words used in our daily speech @xcite . therefore, function words are always highfrequency relative to the content words. in vanilla encoder-decoder models, these models always use a single decoder to generate a complete response at a stroke. when the decoder generates content words and function words at a stroke, it tends to generate high-frequency function words with less semantic information rather than lowfrequency content words with more semantic information. since function words have very little substantive meaning, they not only are redundant for understanding semantic dependency, but also make the dependency sparse. therefore, generating content words and function words at a stroke makes it difficult to learn the semantic dependency between the post and response. to address the aforementioned issue, we propose a novel content-aware tsdg model with a two-stage decoding process. as shown in figure 1 , the key idea is to separate the decoding process of content words and function words so that content words can be generated independently without the interference of function words. in the first decoding stage, we use the first decoder to focus on generating a content word sequence according to the post. in the second decoding stage, we use the second decoder to expand the content word sequence to a complete and fluent response. through this stage, our model gets the final fluent response including more relevant and informative content words. our contributions in this paper are two-fold: ## related work open-domain conversation has long attracted the attention of researchers. the generative models have shown great potential in terms of flexibility, which has aroused a research hotspot. most of generative models are based upon encoder-decoder framework @xcite . however, the traditional encoder-decoder models tend to generate short and uninformative responses, which are known as "safe responses" @xcite . lots of models have been proposed to solve this issue: (1) modifying the objective function to penalize the generation probability of the safe response @xcite . (2) generating from latent variables to increase the diversity of response @xcite . (3) using additional topic content @xcite . (4) content-introducing methods @xcite . ( 5 ) knowledge-based methods @xcite . @xcite take commonsense knowledge into account to facilitate conversation understanding. in the decoding process, these models always use a single decoder to generate the final response at a stroke in a left-to-right manner. ## model the architecture of the proposed tsdg model is illustrated in figue 2. it consists of an encoding process and a two-stage decoding process. given a post @xmath0, u 2 ..., u i as input, our model first uses a self-attention encoder (sae) to encode them into a hidden vector. then, the first decoding stage decodes this hidden vector into a content word sequence @xmath1, c 2 , ..., c k without the influence of function word. finally, the second decoding stage expands the content word sequence into a complete response @xmath2, r 2 , ..., r j . ## conclusion in this paper, we analyze the limitation of the current generative models in the decoding process. to address this, we propose a content-aware neural response generative model with a two-stage decoding process. evaluation results on two datasets indicate that our model can generate more appropriate content words and significantly outperform several competitive models in terms of automatic and human evaluation. there still is some room for improvement. we will refine our model from two decoding stages independently in the future.