Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure (2022)

Abstract: With the availability of massive general-domain dialogue data, pre-trained dialogue generation appears to be super appealing to transfer knowledge from the general domain to downstream applications. In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability. This paper proposes a novel dialogue generation model with a latent structure that is easily transferable from the general domain to downstream tasks in a lightweight and transparent way. Experiments on two benchmarks validate the effectiveness of the proposed model. Thanks to the transferable latent structure, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model with about 22% parameters particularly delivers a 5x speedup in running time compared with the strongest baseline. Moreover, the proposed model is explainable by interpreting the discrete latent variables.

## introduction conversation between humans and machines has long been a goal of artificial intelligence @xcite . building an open-domain dialogue system with data-driven techniques has gotten a lot of attention in the ai and nlp fields in recent years, thanks to breakthroughs in deep learning @xcite @xcite . in particular, with the availability of massive human dialogue data (e.g., the reddit comments) on social media @xcite , pre-trained dialogue generation appears to be super appealing to alleviate potential discrepancies between general domain and downstream applications @xcite @xcite . the common idea behind the pre-trained dialogue generation can be highlighted as a two-step pipeline: a) it firstly trains a deep neural model on massive general-domain dialogue data, b) and then transfers the model into downstream tasks via finetuning or zero-shot learning. under this pipeline, the transferability is mainly obtained by fitting a large model with millions of parameters on massive data in an exhaustive way. consequently, the downsides in existing works are obvious: their running is inefficient and their outputs are difficult to explain. this paper thereby aims to build a pre-trained dialogue model which is easily transferable from the general domain to downstream tasks in a lightweight and transparent way. to this end, we propose a novel dialogue model with a latent structure consisting of several latent variables. by using some self-supervised tasks to endow its latent variables with some prior properties during training, the latent structure makes the knowledge better transferable across different domains. specifically, we first propose to incorporate the transformer architecture with a discrete conversation flow. given a dialogue session, our model will sequentially infer the discrete state for each utterance which provides essential hints for future states and has an effect on the generation of the associated utterance. we further propose a method to disentangle the context-sensitive information from the conversation flow, which is achieved by two disentangled latent variables to capture the contextsensitive information (e.g., topic and persona) and the context-independent information (e.g., dialogue logic for each utterance) respectively. through tailor-designed self-supervised tasks, the contextsensitive latent variable is able to capture the holistic information of a dialogue session while the context-independent variable is supposed to reflect the dynamic flow of dialogue in each utterance. meanwhile, the model is optimized with variational inference by maximizing the evidence lower bound of the likelihood. we conduct experiments with two multi-turn dialogue generation benchmarks, including dailydialog @xcite and convai2 @xcite . thanks to the transferable latent structure, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model including about 22% -66% parameters particularly delivers a 2x -30x speedup in running time. moreover, the proposed model is explainable by visualizing the discrete latent variables. our contributions in the paper are three-fold: (1) we present a context-free dialogue structure that captures the prior knowledge about state transition in a large-scale dialogue corpus. furthermore, with the help of this dialogue structure, our model outperforms the state-of-the-art dialogue pre-training method with much fewer parameters. (2) we propose a disentangled structure learning framework to induce a context-free dialogue structure that enjoys better transferability and interpretability. (3) we empirically verify the effectiveness and efficiency of the proposed model on two benchmarks. ## related work the success of neural networks in machine translation promotes early research on end-to-end opendomain dialogue generation @xcite @xcite . various adaptations to the vanilla encoder-decoder architecture have been built to model the structure of dialogue contexts @xcite @xcite ; improve response diversity @xcite @xcite ; introduce external knowledge @xcite ; and control response qualities @xcite @xcite @xcite . large-scale pre-training for open-domain dialogue generation has recently become promising as a way to bridge the gap between conversation with existing systems and conversation with humans. inspired by the successfulness of gpt-2 @xcite , @xcite propose to train the transformer models on a very large dialogue dataset to generate informative text. @xcite further use discrete latent variables to address the one-to-many mapping problem in opendomain dialogue. despite prior successes, the di-alogue context is simply concatenated as a long sequence, which may fail to capture the discourselevel coherence among utterances. to this end, @xcite and @xcite introduce more self-supervision objectives to capture the discourselevel coherence and the dynamic information flow respectively. the concept of dialogue structure has proven useful in modeling the complicated relationships between utterances. in the field of task-oriented dialogue, @xcite propose a discrete variational recurrent neural network (dvrnn) to learn the dialogue structure through unsupervised learning; @xcite further propose to enhance prior work with a structured attention mechanism; and @xcite propose a conversational graph to represent deterministic dialogue structure, where nodes and edges represent the utterance and context information, respectively. in the field of open-domain dialogue, @xcite construct a large dialogue structure graph with around 1.6 million vertices to cover a wide range of topics. this work introduces a disentangled structure learning framework, which can induce a transferable substructure and an interpretable dialogue substructure, to incorporate the structural bias in dialogue pre-training. thanks to the tailor-designed selfsupervised tasks, our latent structure is more general than the dialogue structure in existing work. ## conclusion we propose a novel dialogue model with structural bias which is explainable to humans and easily transferable to general dialogue tasks. empirical experiments on two benchmark datasets indicate that our model with only 22% parameters outperforms the strongest baseline dialoflow in both decoding speed and response quality measured by automatic and human evaluations. we further show that the learned latent structure enjoys superior transferability and interpretability compared to the conventional methods.