Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems (2021)

Abstract: Human dialogue contains evolving concepts, and speakers naturally associate multiple concepts to compose a response. However, current dialogue models with the seq2seq framework lack the ability to effectively manage concept transitions and can hardly introduce multiple concepts to responses in a sequential decoding manner. To facilitate a controllable and coherent dialogue, in this work, we devise a concept-guided non-autoregressive model (CG-nAR) for open-domain dialogue generation. The proposed model comprises a multi-concept planning module that learns to identify multiple associated concepts from a concept graph and a customized Insertion Transformer that performs concept-guided non-autoregressive generation to complete a response. The experimental results on two public datasets show that CG-nAR can produce diverse and coherent responses, outperforming state-of-the-art baselines in both automatic and human evaluations with substantially faster inference speed.

## introduction creating a "human-like" dialogue system is one of the important goals of artificial intelligence. recently, due to the rapid advancements in natural language generation (nlg) techniques, datadriven approaches have attracted lots of research interest and have achieved impressive progress in producing fluent dialogue responses @xcite @xcite . however, such seq2seq models tend to degenerate generic or off-topic responses @xcite ). an effective way to address this issue is to leverage external knowledge @xcite or topic information @xcite , which are integrated as additional semantic representations to improve dialogue informativeness. although promising results have been obtained by equipping dialogue models with external knowledge, the development of dialogue discourse still has its own challenge: human dialogue generally evolves around a number of concepts that might frequently shift in a dialogue flow @xcite . the lack of concept management strategies might lead to incoherent dialogue due to the loosely connected concepts. to address this problem, recent studies have combined concept planning with response generation to form a more coherent and controllable dialogue @xcite @xcite . most of these approaches incorporate concepts into responses in an implicit manner, which cannot guarantee the appearance of a concept in a response. compared with dialogue concepts, a large proportion of chit-chat words are common and usually have a high word frequency and are relatively overoptimized in language models @xcite . consequently, conventional seq2seq generators are more "familiar" with these generic words than those requiring concept management, which prevents introducing certain concepts to the response with sequential decoding (either greedily or with beam search) @xcite . moreover, speakers naturally associate multiple concepts to proactively convey diverse information, e.g., action, entity, and emotion (see figure 1 ). unfortunately, most existing methods can only retrieve one concept for each utterance @xcite . another line of approaches attempt to explicitly integrate concepts into responses and generate the remaining words in both directions @xcite ), but they also fail to deal with multiple concepts. in this paper, we devise a concept-guided nonautoregressive model (cg-nar) to facilitate dialogue coherence by explicitly introducing multiple concepts into dialogue responses. specifically, following @xcite , a concept graph is constructed based on the dialogue data, where the vertices represent concepts, and edges represent concept transitions between utterances. based on the concept graph, we introduce a novel multiconcept planning module that learns to manage concept transitions in a dialogue flow. it recurrently reads historical concepts and dialogue context to attentively select multiple concepts in the proper order, which reflects the transition and arrangement of target concepts. then, we customize an insertion transformer @xcite by initializing the selected concepts as a partial response for subsequent non-autoregressive generation. the remaining words of a response are generated in parallel, aiming to foster a fast and controllable decoding process. we conducted experiments on persona-chat @xcite and weibo @xcite . the results of automatic and human evaluations show that cg-nar achieves better performance in terms of response diversity and dialogue coherence. we also show that the inference time of our model is much faster than conventional seq2seq models. all our codes and datasets are publicly available. 1 our contributions to the field are three-fold: 1) we design a concept-guided non-autoregressive strategy that can successfully integrate multiple concepts into responses for a controllable decoding process. 2) the proposed multi-concept planning module effectively manages multi-concept transitions and remedies the problem of dialogue incoherence. 3) comprehensive studies on two datasets show the effectiveness of our method in terms of response quality and decoding efficiency. ## methodology the overall framework of cg-nar is shown in figure 2 . based on a concept graph that represents candidate concept transitions, a multi-concept planning module is designed to select and arrange appropriate target concepts from the contextually related subgraphs, which is conditioned on the previous concept flow and the dialogue context. then, we input the selected concepts as a partial response into an insertion transformer @xcite to parallelly generate the remaining words.