Plug-and-Play Conversational Models (2020)

Abstract: There has been considerable progress made towards conversational models that generate coherent and fluent responses; however, this often involves training large language models on large dialogue datasets, such as Reddit. These large conversational models provide little control over the generated responses, and this control is further limited in the absence of annotated conversational datasets for attribute specific generation that can be used for fine-tuning the model. In this paper, we first propose and evaluate plug-and-play methods for controllable response generation, which does not require dialogue specific datasets and does not rely on fine-tuning a large model. While effective, the decoding procedure induces considerable computational overhead, rendering the conversational model unsuitable for interactive usage. To overcome this, we introduce an approach that does not require further computation at decoding time, while also does not require any fine-tuning of a large language model. We demonstrate, through extensive automatic and human evaluation, a high degree of control over the generated conversational responses with regard to multiple desired attributes, while being fluent.

## introduction large pre-trained language models @xcite @xcite have greatly improved the state-of-the-art in many downstream tasks. these language models are trained using the simple log-likelihood objective over large amounts of unlabeled data (e.g., wikipedia articles). this approach results in large powerful language models that produce coherent text and can be with human annotators in the loop to generate contuining text with positive sentiment. both of these approaches require learning/fine-tuning all of the models' parameters, and new desired attributes cannot be easily incorporated into the generation once the models have been trained. other approaches that do not alter the language model, but modify the decoding procedure for controlled generation include 1) re-weighting the output distribution using discriminators @xcite or bag of words @xcite @xcite , and 2) perturbing the models activation with an attribute model (pplm) @xcite . these approaches, instead, are plug-and-play methods in that they can be used on top of any existing pre-trained language model. these methods, do not modify or train the parameters of the original models and they can achieve comparable performance to finetuning methods @xcite . weighted decoding is generally difficult to tune because it can easily generate unrelated responses when the weight is not properly set @xcite . on the other hand, @xcite incurs a high computational cost during the decoding stage, which is problematic for online systems such as dialogue systems. open-domain conversational systems are a special case of language models where the prefix is the dialogue history and the continuation is a humanlike response @xcite . recently, large pre-training language models trained on unlabeled human-to-human conversation (i.e. reddit) @xcite @xcite have shown excellent performance in modelling human responses. similarly, the output of large pre-trained conversational models cannot be directly controlled without having to re-train/finetune the model from scratch, which is practically inconvenient and sometimes impossible since few or no-conversational datasets exist for certain attributes or styles. on the other hand, plug-and-play methods are a viable solution since they do not require dialogue specific datasets, and they can be computed online on top of existing pre-trained models. a major drawback however is the high computational cost @xcite at decoding time. this is acceptable for language models, where generating paragraphs or stories can be done offline, but it is problematic for online systems such as con-versational models. in this paper, we explore the approach from @xcite (pplm) in large pre-trained dialogue models for controlling the style and topic of the responses without finetuning on any dialogue specific dataset. moreover, to cope with the computational cost at the decoding time, we propose to generate style/topic consistent responses with pplm @xcite and then use it to optimize residual adapters @xcite for directly learning how to steer the original distribution towards the selected attribute. with our extensive automatic and human evaluation, we empirically demonstrate that plug-andplay methods are effective in controlling the response while being computationally efficient. to summarize, our key contributions are: open-domain conversational models generating human-like responses involves overcoming a variety of challenges such as personalization @xcite @xcite @xcite , knowledge grounding @xcite @xcite @xcite , emotions @xcite @xcite @xcite , diversity @xcite @xcite and so on. in terms of controlled dialogue generation, @xcite stud-ied of conditional generative models @xcite and weighted decoding @xcite in controlling models trained on personachat. @xcite concluded that controlling specificity, relatedness, and repetition increase human-engagement, motivating us to extend the controllabitly to styles and topics. in this paper, we focus on these two since large pre-trained models can already achieve a high humanness score @xcite @xcite . controlled text generation recent methods for controlled generation include fine-tuning models using supervised learning @xcite , reinforcement learning @xcite , adversarial training @xcite , by pre-training models with control codes @xcite @xcite , and other various approaches @xcite @xcite . alternatively, weight decoding using both bag-of-words @xcite @xcite and discriminators @xcite , does not require any fine-tuning. similarly, @xcite propose the plugand-play language model (pplm) to control the generation of a pre-trained language model, e.g., gpt2 @xcite , both in terms of style and topic of the generated text. finally, residual adapters @xcite has been used to learn multiple language generation tasks @xcite without fine-tuning the original models' parameters. concurrently to our work, smith et al. (2020) compare the performance and tradeoffs of three existing controllable language generation methods on 200 possible styles. ## methodology a dialogue consists of one or more alternating turns between two speakers. we define the dialogue history at turn t as d @xmath0, s 1 , . . . , u t } where u t is the user utterance and s t is the system response. for simplicity, we overload d t to denote the concatenation of sequences across turns with a special token separating the turns. in this paper, we model the dialogue responses using a transformer @xcite -based language model (lm) by using the dialogue history d t as a prefix and then generating the continuation s t in an auto-regressive manner @xcite . causal language modeling let us denote the concatenation of d t and s t as the sequence of tokens @xmath1, . . . , x n }, then we can compute the language model distribution using the chain rule of probability @xcite as: following the notation of @xcite , we define the transformer decoding process in a recursive manner. let us define the matrix h t as the key-value pairs from the dialogue history past, i.e., h @xmath2 and then x t+1 is sampled from the distribution p t+1 = softmax(w o t+1 ), where w is a linear transformation that maps the hidden state of the last layer o t+1 to a vector of vocabulary size. this efficient transformer implementation @xcite leverages the cached memories to generate x t+1 without recomputing h t . ## experimental setup in this section, we conduct extensive experiments on the proposed methodology using both automatic and human-evaluation. differently from pplm @xcite , where a set of predefined prefixes are used to trigger the generation, in our experiments we use 100 conversations (adiwardana et al., 2020) for generating 1100 possible prefixes (i.e., moving window of size two). these open-domain generic dialogues serve as a prefix to trigger the responses rather than fine-tuning. in all our experiments, we use dialogpt @xcite medium, a large pre-trained model trained on 147 million multi-turn dialogues from reddit, spanning from 2005 to 2017. importantly, the proposed methodology is model agnostic, and thus it can be applied to any other large pre-trained model such as meena @xcite and blender-bot @xcite . since plug-and-play adapters use the generated responses from pplm, we randomly split the prefixes with 80% for learning the adapter perturbation and the remaining 20% for the final automatic and human evaluation. this is done to have a fair comparison between other baselines and adapters (see appedix a for more details). ## results in this section, we evaluate the proposed methodology to answer three research questions: 1) is it possible to use plug-and-play methods for controlling the output of a large pre-trained conversational model? if so, 2) what are the most effective plug-and-play methods?, and 3) how difficult is to control the response generation given various attributes? to answer the first two questions, we rely on both automatic and human evaluation. ## analysis in this section, we evaluate the difficulty of controlling the response generation for a given attribute. to do so, we analyse the behaviour of pplm over two opposite styles (i.e., positive and negative) and then we conduct a qualitative evaluation over the generated responses. ## conclusion we explore plug-and-play methods for controlling the response generation of large pre-trained con-versational models in a light-weight manner while being effective. with extensive automatic and human evaluations, we show that pplm is able to generate fluent and attribute consistent responses. further, to overcome the significant computational overhead introduced by pplm at decoding, we optimize a tiny residual adapter for each attribute based on a few synthetic responses generated using pplm. the resulting model does not require further computation at decoding time, and outperforms pplm both in terms of fluency and attribute consistency.