S tyle DGPT : Stylized Response Generation with Pre-trained Language Models (2020)

Abstract: Generating responses following a desired style has great potentials to extend applications of open-domain dialogue systems, yet is refrained by lacking of parallel data for training. In this work, we explore the challenging task with pre-trained language models that have brought breakthrough to various natural language tasks. To this end, we introduce a KL loss and a style classifier to the fine-tuning step in order to steer response generation towards the target style in both a word-level and a sentence-level. Comprehensive empirical studies with two public datasets indicate that our model can significantly outperform state-of-the-art methods in terms of both style consistency and contextual coherence.

## introduction with advances in neural machine learning @xcite @xcite and availability of huge amount of human conversations on social media, there has been significant progress on building open-domain dialogue systems with natural language generation techniques. though neural generative models are notorious for replying with bland responses @xcite , some very recent work demonstrates that response generation models learned with pre-training techniques @xcite can effectively overcome the deficiency suffered by previous models and are capable of having smooth conversations with humans through reasonable and specific replies @xcite . the compelling performance exhibited by the pre-trained dialogue models encourages us to explore more difficult yet important problems in conversational ai. in this work, we study stylized response generation, that is responses provided by a * corresponding author model should not only be coherent with the conversation contexts, but also be consistent with a designated style. such research could facilitate developers to customize their dialogue systems in terms of response styles, and thus broaden applications of the systems, from a social companion @xcite or a virtual assistant @xcite to a variety of vertical scenarios such as customer service (requiring a polite style), virtual characters in games (requiring specific personas), assistants in specific domains (requiring domain knowledge), etc. normally, a target style is specified by a non-conversational corpus (e.g., novels, news, blogs, etc.) apart from the paired dialogue corpus @xcite @xcite . thus, the major challenge of the task lies in the scarcity of paired data for learning the correspondence between conversation contexts and proper responses in the desired style, which is a key factor in success of the neural dialogue models developed so far. as a result, it is very likely that a response either digresses from the context of the current dialogue @xcite , or loses fidelity to the target style @xcite . we consider addressing the challenge by taking advantage of the large scale pre-trained language models. the basic idea is that deep neural language models learned from huge amount of text, such as gpt-2 @xcite and dialogpt @xcite , have packed enough style knowledge into their parameters @xcite , and thus by simply steering the distribution in decoding towards the desired style, we can obtain both contextual coherence and style consistency. following the idea, we build a response generation model on top of a pre-trained language model and devise both a word-level loss and a sentence-level loss to fine-tune the pre-trained model towards the target style. the word-level loss regularizes the likeli-hood of response generation with a kl divergence term between the probability of dialogues and the probability of stylized language estimated by finetuning a pre-trained language model on the style corpus, while the sentence-level loss maximizes the likelihood of a response given by the pre-trained response generation model being classified as a sentence matching the target style. we employ a gumbel trick to overcome the obstacle in backpropagation due to the discrete nature of natural language when optimizing the sentence-level loss. the final response is selected by a sample-and-rank strategy to further enhance relevance regarding to the dialogue context and fidelity regarding to the target style. we name our model styledgpt standing for "stylized dialogpt". empirical studies are conducted on two tasks: arxiv-style response generation and holmes-style response generation with the data shared in @xcite , where responses in the style of scientific papers and the style of sherlock holmes novels are pursued respectively for a given context. besides the style intensity used in @xcite , we further examine style consistency from both a lexical perspective and a syntactic perspective with two new metrics. evaluation results on both automatic metrics and human judgment indicate that our model can significantly outperform state-of-the-art methods. the code is available at https://github. com/tobeyyang/styledgpt . our contributions are three-fold: (1) proposal of tackling the problem of stylized response generation with pre-trained language models; (2) proposal of a word-level objective and a sentence-level objective in fine-tuning of a pre-trained language model for the task; and (3) empirical verification of the effectiveness of the proposed method on public datasets. stylized text generation has attracted broad interest in recent years, especially the style transfer, which aims to alter one or more attributes of text while preserving the content. a prevalent idea of unsupervised style transfer is learning to separate "content" and "style" of text and manipulate the style to induce transfer at inference time @xcite @xcite . however, some works show that the disentanglement cannot be met and is not necessary, and leverage techniques like reconstruction and back-translation introduced in unsupervised machine translation @xcite , transformer @xcite to achieve unsupervised style transfer. different from style transfer, stylized response generation requires that the response is coherent with its context and the content can be varied. @xcite first train a basic model on a large-scale dialogue corpus and then fine-tune the model with a small stylized corpus. @xcite propose three weakly-supervised methods to generate polite responses using non-parallel data. @xcite build a structured latent space sharing between conversation modeling and style transfer. however, limited by the sparsity of the latent space, it is difficult to balance the style and contextual coherence while sampling in the neighborhood of the latent code of context at inference time. pretraining methods have led remarkable success in various nlp tasks which demonstrates its great capabilities in language understanding and text generation @xcite @xcite @xcite @xcite . recently, the pretraining methods have also been used to tackle the key challenges in dialogue systems such as context representation @xcite , response selection (henderson and su, 2019), knowledge-grounded response generation @xcite and personalized response generation @xcite . in particular, the large-scale pre-trained open-domain dialogue systems @xcite ) make a large step towards human-like chatbot against previous works which rely on complex frameworks developed over many years. on this basis, we propose to study the open-domain stylized response generation with pre-trained models in this work. ## approach we employ dialogpt @xcite as the general response generation model p (y |x), and try to bias p (y |x) towards the language distribution p (s) estimated from d style in fine-tuning. below, we first briefly review the openai gpt-2 @xcite and dialogpt, which serve as the backbone of our model. then, we introduce two learning objectives from both a word perspective and a sentence perspective to interpolate style s into response generation.