Large Scale Multi-Actor Generative Dialog Modeling (2020)

Abstract: Non-goal oriented dialog agents (i.e. chatbots) aim to produce varying and engaging conversations with a user; however, they typically exhibit either inconsistent personality across conversations or the average personality of all users. This paper addresses these issues by controlling an agent’s persona upon generation via conditioning on prior conversations of a target actor. In doing so, we are able to utilize more abstract patterns within a person’s speech and better emulate them in generated responses. This work introduces the Generative Conversation Control model, an augmented and fine-tuned GPT-2 language model that conditions on past reference conversations to probabilistically model multi-turn conversations in the actor’s persona. We introduce an accompanying data collection procedure to obtain 10.3M conversations from 6 months worth of Reddit comments. We demonstrate that scaling model sizes from 117M to 8.3B parameters yields an improvement from 23.14 to 13.14 perplexity on 1.7M held out Reddit conversations. Increasing model scale yielded similar improvements in human evaluations that measure preference of model samples to the held out target distribution in terms of realism (31% increased to 37% preference), style matching (37% to 42%), grammar and content quality (29% to 42%), and conversation coherency (32% to 40%). We find that conditionally modeling past conversations improves perplexity by 0.47 in automatic evaluations. Through human trials we identify positive trends between conditional modeling and style matching and outline steps to further improve persona control.

## introduction modeling dialog agents, otherwise known as chatbots, has been a longstanding goal within artificial intelligence research. historically, approaches to this task can be divided into one of the two categories: retrieval and generative. the former is posed as a search problem where an appropriate response for a conversation is selected from a large set of candidate replies, whereas the latter autoregressively samples a reply, thereby potentially creating a response that the model may not have seen before. the flexibility and creativity afforded by not prespecifying every possible response is a major draw for generative based approaches. in recent years, advances in neural methods have shown promise in effectively modeling this task. early progress first demonstrated potential with recurrent network based models capable of holding simple conversations @xcite . further architecture optimizations and tweaks improved user experiences; however, they largely experienced issues with the agent exhibiting an inconsistent personality and producing uninteresting comments @xcite . some works have attempted to alleviate this through conditioning on various factors of the conversation through methods such as sentiment or speaker embeddings @xcite , but the added data annotation makes these methods not scale well to the gargantuan amounts of data needed to train larger models. a persona-based conversation task was introduced by @xcite where a set of reddit comments and their replies were accompanied by brief descriptions or factoids about the speakers, such as their hobbies and interests. recent works @xcite yellow highlights indicate where the model appropriately transferred a part of style between references and generations for speaker b (e.g. 'i' v. 'i'), boxed words indicate a transfer of content for speaker b, and green highlights indicate consistent style for turns from speaker a (e.g. '3ds' v. '3ds'). emulating this abstract notion of style requires large amount of data and sufficiently powerful model. we propose a data collection procedure that heuristically scrapes user data and comment threads from reddit 1 to produce conversations that vary widely in content, speakers, and reference histories to condition on. this work also introduces the generative conversation con-trol (gcc) model, an augmented and fine-tuned gpt-2 language model. we take advantage of 1 https://reddit.com/ large transformers' ability to model long contexts and dependencies, and successfully model multiturn and multi-actor conversations that are significantly longer (up to 15 turns) than most prior work. we find that scaling model sizes from 117m to 8.3b parameters yields an improvement from 23.14 to 13.14 perplexity on 1.7m held out reddit conversations. similar improvements from model scaling are found in human evaluations that measure sample preference to the held out target distribution in terms of realism (31% increased to 37% preference), style matching (37% to 42%), grammar and content quality (29% to 42%), and conversation coherency (32% to 40%). to summarize, our contributions in this work are three-fold: i we introduce a new conversational task and demonstrate added value over traditional conversation modeling through both better control and response generation. ii we document the creation of a large, multiturn, multi-actor conversational dataset and the techniques used to clean it and extract conversations and reference material for style. iii we demonstrate that by increasing model size from 117m to 8.3b parameters, human evaluations measuring preference of model gener-ated samples over held out target distribution increase with respect to realism, style matching, grammar, and conversation coherency. automatic evaluations also showcase similar trends with the largest model leading to significantly lower perplexities. ## conversation modeling let c represent a multi-turn conversation of variable-length, and let x j represent a single turn that contains a variable-amount of tokens. mathematically, this is represented as @xmath0, . . . , x |c| ), with x @xmath1,1 , . . . , x j,|x j | ). every token, in every turn, belongs to the same fixed vocabulary (i.e. ## data in order to sufficiently train a model to be able to autoregressively generate turns in a conversation conditioned on prior conversations, we require an ample amount of diverse examples accompanied with plenty of reference material. a suitable source of data for this purpose can be found from comments made on reddit posts. thanks to a publicly available archive on pushshift.io, comments are processed from reddit ranging from october of 2018 to march of 2019 for training, and april of 2019 for validation. the techniques described in this section can naturally be extended to the full range of reddit data spanning as far back as 2005; however, we choose to focus on just the 6 months in question for the sake of tractability. ## model all models proposed stem from the gpt-2 model architecture as their base design @xcite . the class of models will be defined as generative conversation control models, gcc we experiment with the number of layers, l, the hidden size, h, and the number of attention heads, a in the gpt-2 model architecture. modeling conversations with gcc requires three steps: (i) identify a speaker to emulate and obtain their reference history consisting of comments they made on other reddit posts, (ii) input the reference history and conversation turns into the model, and (iii) retrieve estimated next-token probabilities only associated with turns in the conversation spoken by the target speaker. ## experiments it is known that for the language modeling validation perplexity measures using teacher forcing is not the best evaluation of generative capabilities, even if there is correlation between the two. however, it is a commonly used metric for language modeling, and can be parallelized and computed inexpensively without the need for autoregressive sampling of output text. with that in mind, two sets of evaluations were done, the first of which being an architecture search using automatic evaluation with validation perplexity and the second being a qualitative study using amazon's mechanical turk 3 to assess generated samples. ## conclusions when a large conversational model is trained on a diverse collection of multi-turn conversations, it is able to generate quality conversations that are engaging, coherent, and plausibly human. furthermore, when conditioned on prior conversations, the model is able to utilize a speaker's personality when choosing how to reply in a conversation to allow for greater control and more diverse responses. in the future, we aim to leverage these pre-trained models to advance sota on downstream conversational tasks, such as knowledge-grounded conversations or question answering. recent advancements in learnable information retrieval systems could select contextually relevant references to further strengthen the quality of generated dialogue.