S ide C ontrol: Controlled Open-domain Dialogue Generation via Additive Side Networks (2021)

Abstract: Transformer-based pre-trained language models boost the performance of open-domain dialogue systems. Prior works leverage Transformer-based pre-trained language models to generate texts with desired attributes in two general approaches: (1) gradient-based methods: updating all latent representations of pre-trained models with gradients from attribute models; (2) weighted-decoding methods: re-ranking beam candidates from pre-trained models with attribute functions. However, gradient-based methods lead to high computation cost and can easily get overfitted on small training sets, while weighted-decoding methods are inherently constrained by the low-variance high-bias pre-trained model. In this work, we propose a novel approach to control the generation of Transformer-based pre-trained language models: the SideControl framework, which leverages a novel control attributes loss to incorporate useful control signals, and is shown to perform well with very limited training samples. We evaluate our proposed method on two benchmark open-domain dialogue datasets, and results show that the SideControl framework has better controllability, higher generation quality and better sample-efficiency than existing gradient-based and weighted-decoding baselines.

## introduction with the advance of transformer-based pre-trained language models @xcite @xcite , many dialogue systems @xcite @xcite have shown promising performance in challenging open-domain conversations with humans. however, for controlled dialogue generation, prior works mainly focus on building lstm-based class-conditional generative model on specific datasets with task-specific design on model architecture @xcite @xcite or policy learning strategy @xcite @xcite . in this work, we explore effective method for controlled generation on transformerbased dialogue systems, with the goal of adding controllability functionality into state-of-the-art transformer-based dialogue systems with lower computation cost, less training data and more flexible control mechanism. prior works on controlled text generation for transformer-based pre-trained language models can be categorized into two general approaches: (1) gradient-based methods and (2) weighted-decoding methods. the gradient-based methods @xcite @xcite propose a plug-and-play language model following p(x|a) ∝ p(a|x)p(x), which plugs an attribute model p(a|x) with a pre-trained language model p(x) to control generation. the gradients from p(a|x) are used to guide the latent representations of pre-trained models encoding more control attribute information. the weighted-decoding methods @xcite @xcite modify the sampling weights with attribute functions in beam search at each decoding timestep to control generation. essentially, the attribute functions are used to re-rank the original beam candidates generated by the pre-trained language models. the main idea of both gradient-based methods and weighted decoding methods is the flexibility: users can design any attribute models or functions for different controlled generation tasks and apply the attribute model or function to any state-of-the-art pre-trained language models for generating high quality texts. however, weighted decoding methods @xcite @xcite (a) sidenet for knowledge document control (b) sidenet for semantic label control figure 1 : general architecture of the sidecontrol framework. @xcite are limited by the low-variance high-biased pre-trained language models, since they do not update the pre-trained language models. if the pre-trained model yields commonly observed words rather than target attribute words in the beam candidates list, it is difficult for the attribute functions to re-rank and find the target words during generation. although gradient-based methods @xcite @xcite do not have this limitation since they update the latent representations of pre-trained models during inference, the gradient propagation at each decoding timestep involves heavy computation, which results in slow response speed to users. in addition, the controllability performance of gradient-based methods relies on the attribute model. if the attribute model gets overfitted on a small training set, the gradient from this attribute model will just lead to meaningless updates. to build an effective and efficient controlled open-domain dialogue system, we propose the sidecontrol framework, which treats the pretrained lanaguage model as a feature extractor and train light-weight side networks to encode complementary information from control attributes. in addition, we introduce a novel control attributes loss to guide the side network during training. as shown in figure 1 , the final output representation is a mixture of a base representation from the pretrained language model and a side representation from the side network. the mixture coefficient α is learned during training, and is used to balance the prior knowledge from the base network and the task-specific control attributes signals from the side network. from the encoding perspective, the side-control framework not only can be applied to any pre-trained language models, but also supports diverse format attributes control (e.g. dialogue act, external knowledge document). from the decoding perspective, the sidecontrol framework has low computation cost, since it directly samples from its optimized class-conditional language model p(x|a) without additionally updating latent representations during generation. from the sample-efficiency perspective, the sidecontrol framework achieves good performance with a few thousand training samples by leveraging the control loss. we summarize the contributions of this work as follows: ## sidenet for controlled generation firstly, we introduce the sidecontrol framework in subsection 2.1, which presents the general idea of using a small side network to coordinate the generation process based on large-scale pre-trained language models @xcite @xcite . then we provide two realizations of side networks for two types of control attributes: (1) external knowledge document in subsection 2.2, (2) semantic label in subsection 2.3. ## conclusions in this work, we propose a new method for controlled dialogue generation: adding a small side network to incorporate useful control signals into the pre-trained language models. we design control attributes loss to teach the side network learning useful control signals. empirical experiments show that our method is effective even with 100 ∼ 1000 training samples. besides, our side network supports diverse forms of attributes control and can be flexibly applied to any pre-trained language models, which extends its possible application to other general controlled text generation tasks.