DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation (2020)

Abstract: We present a large, tunable neural conversational response generation model, DIALOGPT (dialogue generative pre-trained transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.

## introduction we introduce dialogpt, a tunable gigawordscale neural network model for generation of conversational reponses, trained on reddit data. recent advances in large-scale pre-training using transformer-based architectures @xcite @xcite have achieved great empirical success. openai's gpt-2 @xcite , for example, has demonstrated that transformer models trained on very large datasets can capture long-term dependencies in textual data and generate text that is fluent, lexically diverse, and rich in content. such models have the capacity to capture textual data with fine granularity and produce output with a high-resolution that closely emulates real-world text written by humans. dialogpt extends gpt-2 to address the challenges of conversational neural response genera-tion. neural response generation is a subcategory of text-generation that shares the objective of generating natural-looking text (distinct from any training instance) that is relevant to the prompt. modelling conversations, however, presents distinct challenges in that human dialogue, which encapsulates the possibly competing goals of two participants, is intrinsically more diverse in the range of potential responses @xcite @xcite . it thus poses a greater one-to-many problem than is typical in other text generation tasks such as neural machine translation, text summarization and paraphrasing. human conversations are also generally more informal, noisy, and, when in the form of textual chat, often contain informal abbreviations or syntactic/lexical errors. most open-domain neural response generation systems suffer from content or style inconsistency @xcite @xcite , lack of long-term contextual information @xcite , and blandness @xcite @xcite . while these issues can be alleviated by modelling strategies specifically designed to boost information content, a transformer-based architecture like gpt-2 @xcite , which uses a multi-layer self-attentive mechanism to allow fully-connected cross-attention to the full context in a computationally efficient manner, seems like a natural choice for exploring a more general solution. transformer models, for example, allow long-term dependency information to be better be preserved across time @xcite , thereby improving content consistency. they also have higher model capacity due to their deep structure (up to 48 layers in gpt-2) and are more effective in leveraging large-scale datasets (more than 100 million training instances) than rnn-based approaches @xcite . like gpt-2, dialogpt is formulated as an autoregressive (ar) language model, and uses the multi-layer transformer as model architecture. unlike gpt-2, however, dialogpt is trained on large-scale dialogue pairs/sessions extracted from reddit discussion chains. our assumption is that this should enable dialogpt to capture the joint distribution of p (target, source) in conversational flow with finer granularity. in practice, this is what we observe: sentences generated by dialogpt are diverse and contain information specific to the source prompt, analogous what gpt-2 generates for continuous text. we have evaluated the pre-trained model on a public benchmark dataset (dstc-7), and a new 6k multireference test dataset extracted from reddit postings. dialogpt achieves state-of-the-art results in both automatic and human evaluation, lifting performance to near-human response quality. we have released the source code and a pre-trained model to facilitate future research. 1 . our model can be easily leveraged and adapted to new dialogue datasets, especially datasets with few training examples. the dialogpt package also contains an open-source training pipeline (data extraction/preparation and model training/evaluation) built upon the huggingface pytorch transformer (huggingface, 2019). 2 ## dataset the dataset is extracted from comment chains scraped from reddit spanning from 2005 till 2017. reddit discussions can be naturally expanded as tree-structured reply chains, since a thread replying to one thread forms the root node of subsequent threads. we extract each path from the root node to the leaf node as a training instance containing multiple turns of dialogue. we filter the data by removing the instances where (1) there is a url in source or target, (2) where the target contains word repetitions of at least three words, (3) where the response does not contain at least one of the top-50 most frequent english words (e.g., "the", "of", "a"), since this probably indicates it might not be an english sentence, (4) where the response contains special markers such as "[" or "]", as this could be markup 1 github: https://github.com/microsoft/ dialogpt ; blog: https://aka.ms/dialogpt 2 our model is also available over hugging face transformers. https://huggingface.co/microsoft/ dialogpt-medium language, (5) where source and target sequences together are longer than 200 words, (6) where the target contains offensive language, identified by phrase matching against a large blocklist. we also excluded a large number of subreddits that had been identified as likely to contain offensive content. in addition, we aggressively filtered out blandness, e.g., removing instances where the responses contained 90% of tri-grams that have been seen more than 1000 times. often uninformative, such responses account for about 1% of the data. after filtering, the dataset comprises 147,116,725 dialogue instances, in total 1.8 billion words. ## related work there are several open-sourced toolkits for largescale pre-trained transformer models. huggingface conv-ai transfer learning repository @xcite ) contains the code for training conversational ai systems with transfer learning based on the gpt-2 transformer language model, which achieves the state-of-the-art performance on convai-2 dialogue competition. dlgnet @xcite is a large transformer model trained on dialogue dataset and achieves good performance in multi-turn dialogue generation. allennlp @xcite is developed as a toolkit for many natural language processing tasks, including the large-scale pre-trained bi-lstm sentence representation learning framework elmo @xcite . texar @xcite focuses on text generation including style transferring and controllable generation. it includes reinforcement learning capabilities along with its sequence modelling tools. deeppavlov @xcite is a popular framework focusing on task-oriented dialogue. this public repository contains several demos and pre-trained models for question answering and sentiment classification. icecaps @xcite is a response generation toolkit with techniques such as grounding on personalities or external knowledge and multi-task training. the convai2 challenge @xcite has a focus on personalized conversations. parlai @xcite is another library for developing task-oriented dialogue systems. it contains pre-trained models for knowledge-grounded chatbot trained with crowdsourced data. the text-to-text transformer @xcite unifies multiple text modeling tasks, and achieves the state-of-the-art results in various natural language generation and understanding benchmarks. ## limitations and risks dialogpt is released as a model only; the onus of decoder implementation resides with the user. despite our efforts to minimize the amount of overtly offensive data prior to training, di-alogpt retains the potential to generate output that may trigger offense. output may reflect gender and other historical biases implicit in the data. responses generated using this model may exhibit a propensity to express agreement with propositions that are unethical, biased or offensive (or the ## conclusion we have released an open-domain pre-trained model, dialogpt, trained on massive real-world reddit dataset. the package consists of a distributed training pipeline and several pre-trained models that can be fine-tuned to obtain a conversation model on a moderately-sized customized dataset in few hours. dialogpt is fully opensourced and easy to deploy, allowing users to ex-tend the pre-trained conversational system to bootstrap training using various datasets. it serves as a building block to novel applications and methodologies. detection and control of toxic output will be a major focus of future investigation. we will investigate leveraging reinforcement learning to further improve the relevance of the generated responses and prevent the model from generating egregious responses.