Focus-Constrained Attention Mechanism for CVAE -based Response Generation (2020)

Abstract: To model diverse responses for a given post, one promising way is to introduce a latent variable into Seq2Seq models. The latent variable is supposed to capture the discourse-level information and encourage the informativeness of target responses. However, such discourse-level information is often too coarse for the decoder to be utilized. To tackle it, our idea is to transform the coarse-grained discourse-level information into fine-grained word-level information. Specifically, we firstly measure the semantic concentration of corresponding target response on the post words by introducing a fine-grained focus signal. Then, we propose a focus-constrained attention mechanism to take full advantage of focus in well aligning the input to the target response. The experimental results demonstrate that by exploiting the fine-grained signal, our model can generate more diverse and informative responses compared with several state-of-the-art models.

## introduction nowadays, developing intelligent open-domain conversational systems has become an active research field @xcite . compared with rule-based and retrieval-based methods, neural generative models have attracted increasing attention because they do not need extensive feature engineering and have achieved promising results recently with largescale conversational data @xcite @xcite . typically, neural generative models are trained to learn the post-response mappings based on the seq2seq architecture using maximum likelihood (mle) training objective. this kind of objective induces the model to treat the post-response relationship as one-to-one mappings. however, the conversations in the real world often embodies one-to-many relationships, where a post is often associated with multiple valid responses @xcite . due to this discrepancy, standard seq2seq models tend to generate high-frequency but trivial responses such as "i don't know" or "i'm ok" @xcite . to address this issue, one promising research line resorts to conditional variational autoencoder (cvae), which introduces a latent variable to seq2seq models through variational learning. the latent variable is supposed to capture the discourselevel semantics of target response and in turn encourage the response informativeness. recent literature along this line attempted to improve the model performance by putting extra control on the latent variable @xcite @xcite . despite the control, these methods still relied on the discourse-level latent variable, which is too coarse for the decoders to mine sufficient guiding signals at each generation step. as a result, these variational models are observed to ignore the latent variable @xcite @xcite and to generate semantically irrelevant or grammatically disfluent responses @xcite . in this paper, we propose a novel cvae-based model, which exploits fine-grained word-level information for diverse response generation. firstly, we transform the discourse-level information into word-level signals, i.e., focus. by attending the latent variable to the post words, the focus weight measures the response's correlation with the post 1 words. the higher the weight, the semantics is more likely to concentrate on the corresponding word. to utilize the focus, we develop a focus-constrained attention mechanism which better aligns the post words with the response according to the fine-grained signals. in this way, the model is able to produce a semantically different response directed by a different focus. our contributions can be summarized as three folds: 1) we propose a novel cvae-based model for diverse response generation, by directing the decoder with fine-grained information. 2) we introduce focus to represent the fine-grained information, and propose a focus-constrained attention mechanism to make full use of it. 3). experimental results demonstrate our model outperforms several state-of-the-art models in terms of response's diversity as well as appropriateness. ## related work the attention mechanism @xcite has become a widely-used component for seq2seq @xcite to model short-text conversation @xcite @xcite . although promising results have been achieved, attention-based seq2seq models still tend to generate generic and trivial responses @xcite . there have been many approaches attempted to address this problem. @xcite reranked the n-best generated responses based on maximum mutual information (mmi). @xcite adopted segement-level reranking to encourage diversity during early decoding steps. however, these reranking-based methods only introduce a few variants of decoded words. another group of researches attempted to encourage diversity by incorporating extra information. @xcite injected topic words and @xcite introduced a cue word based on point-wise mutual information (pmi) into generation models. @xcite grounded on knowledge bases to provide factual information for the decoder. however, it is difficult to ensure these external information are always appropriate to the conversation context. another line of research introduced a set of latent responding mechanisms and generated responses based on a selected mechanism. @xcite learned the post-response mappings as a mixture of the mechanisms, but it is questionable that they only relied on one single mechanism when generating responses given a new post. @xcite adopted posterior selection to build one-toone mapping relationship between the mechanisms and target responses. since the target response is missing during testing, it is hard to ensure a satisfactory generated response by a randomly picked mechanism. our work centers in the research line of conditional response generation through variational learning @xcite . however, the variational methods inevitably suffer from bypassing the latent variable and generating disfluent responses. @xcite combined cvae with dialog acts to learn meaningful latent variable, however the discourse-level dialog act is hard to be captured from short conversation. @xcite introduced gaussian mixture prior network, but it is hard to determine the number of mixtures and the optimization is complicated. @xcite assumed the response generation is driven by a single word, and connected each latent variable with words in the vocabulary. nevertheless, the difficulty is how to target the driving word for a specific post-response pair. more importantly, all of these methods rely on the coarse-grained discourse-level information, which might be insufficient in leading to a satisfactory response. notably, our work induces the response generation with focus, a fine-grained feature extracted from the discourse-level latent variable. compared with the variational attention that models the alignment as latent variable @xcite , we are mainly inspired by the idea of coverage vector @xcite to dynamically adjust the attention based on the attention history and the proposed focus. the difference is that @xcite addressed the under/over translation problem and the decoder in their work pays equal attention to the source words. in contrast, our work constrains the decoder to align the decoding attention with the fine-grained focus to generate diverse responses. ## conclusion in this paper, we identify the insufficiency of discourse-level latent variable in response generation. to address this, we develop a novel cvaebased model, which exploits a fine-grained wordlevel feature to generate diverse responses. on a real-world benchmarking dataset, we demonstrate that our proposed model is able to fully leverage the fine-grained feature, and generate better responses as compared to several sota models. based on the ablation studies, we verify the contribution of each proposal in our method and highlight the significance of fine-grained signal in response generation.