Recipes for Building an Open-Domain Chatbot (2021)

Abstract: Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we highlight other ingredients. Good conversation requires blended skills: providing engaging talking points, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models.

## introduction this work provides recipes for building opendomain chatbots that perform well, i.e., to study and identify which methods work well together in order to maximize human evaluations. it has been shown across the field of nlp @xcite and in conversational agents in particular @xcite @xcite ) that pre-training on large corpora is important. beyond simply scaling models, the two main takeaways from our study are that: (1) large improvements can be made by fine-tuning on data that emphasizes desirable conversational skills. we select tasks that make the model focus on personality and engagingness, knowledge, and empathy, achieving large gains by using the recently introduced blended skill talk (bst) set-up @xcite , which targets those aspects by providing training data and initial conversational context (personas and topics). small models using bst can match or outperform larger models that do not. (2) two models with the same perplexity but different decoding algorithms can give vastly different results, e.g. we show that the length of the bot's utterances are crucial to human judgments of quality. we show, contrary to previous work which reports that beam search is inferior to sampling @xcite , that careful choice of search hyperparameters can give strong results, e.g., constraining the minimum beam length gives a crucial control of the bland versus spicy spectrum of responses. in human evaluations our best model outperforms both dialoggpt @xcite and the state of the art meena chatbot @xcite , the latter in a pairwise comparison 75% to 25% in terms of engagingness, and by 65% to 35% in terms of humanness (both statistically significant, two-tailed binomial test, @xmath0.01). while the performance of our bot at first sight is very good, we do not believe we are yet close to solving the problem of open-domain conversation. we thus discuss limitations of our models, and initial attempts to solve them. in particular, our models still display: a lack of in-depth knowledge if sufficiently interrogated; a tendency to stick to simpler language; and a tendency to repeat oftused phrases. we show how unlikelihood training @xcite and @xcite mechanisms are potential avenues for fixing these problems; however, our initial experiments with these methods are inconclusive. we thus discuss future possibilities for alleviating these problems, as well as methods for evaluating them. finally, we believe releasing models is the most reliable way to enable full insight into their capabilities. we thus make publicly available our large-scale, state of the art open-domain conversational agent, including code to fine-tune it, the model weights, and code to evaluate it, so that our setup is reproducible. ## related work the area of open-domain dialogue has made significant progress recently with end-to-end neural approaches. the convai2 competition at neurips 2018 featured large pre-trained transformers for the top two winning teams @xcite . in particular, @xcite pre-trained via the method of @xcite using the bookscorpus dataset, resulting in the best perplexities and f1 scores. since then, results have improved further with the advent of larger, improved pre-training @xcite . in general this extends beyond con-vai2 to many open-domain dialogue datasets, such as daily dialogue and cornell movies @xcite , and also when multi-tasking across many of these datasets, as we also do here @xcite . a particular large-scale model of note that we compare to in this work is meena @xcite , a 2.6b parameter transformer-based model trained on 341 gb of text, that was shown to be superior to variants of dialogpt @xcite , mitsuku foot_0 , cleverbot foot_1 , and xiaoice @xcite . we also compare directly to dialogpt, which is a 345m parameter model trained based 147m multi-turn dialogues extracted from reddit discussion threads. the evaluation metric used for meena in @xcite was ssa, the average of sensibleness and specificity, as judged by human raters either in static or interactive setups, which is shown to highly correlate with asking raters how "humanlike" the model is. we note however that the authors themselves state it may not capture all aspects of such a test, e.g. might not measure empathy. we additionally note that neither meena's model, the static "mini turing benchmark" used in the paper, nor the phrasing of the ssa evaluation question provided to annotators was released, making certain comparisons difficult. further, the human-bot conversations were conducted by employees and were not blind to the model type (in the logs they say phrases such as "hi meena!"). dialogpt in contrast is a publicly available open source model, and evaluations can be easily conducted and reproduced. in this work we employ unbiased crowdworkers with reproducible experiments, and use acute-eval @xcite (described more in section 4) to directly ask the humanness question, rather than a proxy. further, we also report results on engagingness as a main metric, because this measures more closely whether a human will be interested in talking to our bots. ## evaluation methods while we employ and report automatic metrics, our main evaluation involves the acute-eval procedure @xcite , whereby evaluators are asked to make pairwise evaluations of complete dialogues. we consider two evaluation questions, derived from @xcite : (1) engagingness question: "who would you prefer to talk to for a long conversation?"; (2) humanness question: "which speaker sounds more human?". nevertheless, full human evaluations are time consuming and costly, requiring humans to spend time conducting conversations with bots as well as scoring them. as an alternative, we use the selfchat procedure from @xcite for some of our modeling and hyperparameter choices where the full acute-eval would end up too costly, and only use the full human-bot chat evaluation at the final stage. in this work we use the bst-setting to perform self-chats, i.e. models are given the personas, topics and previous utterances to initiate the conversation, see appendix e.2 and figure a.2. note that when using deterministic methods such as beam decoding, this prevents the models from generating the same conversation repeatedly. ## discussion in this work we aimed to study and identify which methods work well together to maximize human evaluations, and identified a few key factors (scale, fine-tuning tasks, and decoding choices) that when blended together give state of the art performance. while our methods have achieved higher human ratings of engagingness and humanness, our models still have numerous issues. firstly, even our best models still make mistakes: they i) contradict or repeat themselves on occasion, ii) tend to repeat the same phrases in separate conversations, and iii) hallucinate knowledge as seen in other generative systems @xcite . each of these faults naturally leads to future research directions; we made some attempt here using unlikelihood @xcite and conditioning on knowledge @xcite , but more needs to be done. as the human evaluations are on short dialogues (14 turns), longer conversations would likely make these issues appear much worse, and expose the limitations of the transformer architectures stemming from their limited dialogue history. a number of recent architectures attempt to incorporate longer memory, and that is also a fruitful direction, although evaluation is more challenging as long conversations have to be collected, and evaluated. an alternative is to seed the conversation with a topic or otherwise provide instructions to the human speaker during evaluation to give the conversation a certain focus, which would more deeply probe the skills of the bot. on the modeling side, longer conversations could also make the choice of context material provided to the bot more salient. besides helping with consistency, the persona and topic that are given as initial context in blended skill talk can help models introduce interesting talking points in the conversation. however, they would need to be far more detailed for longer or repeated conversations to help the models be consistent and avoid repetition, and in our experimental setup did not affect evaluations strongly. for deployment of a chatbot, being well-behaved remains a significant challenge. in particular, we expect bots to have more integrity than the average human (or to even be faultless), but they have much less understanding of what they are saying than humans. recent work mitigating toxicity @xcite and gender bias in dialogue generation @xcite is encouraging, but much work remains to be done. making models publicly available could help the community work together on understanding and fixing these issues. the work of @xcite showed that there is a correlation between human evaluation and perplexity, given a fixed decoding scheme. we argue that while this is important, other factors are also at play and cannot be ignored: (1) the choice of training data is paramount, as shown by our pushshift.io reddit (pre-training) vs. blended skill talk experiments; and (2) decoding algorithms make large differences for the same fixed perplexity model (sec. h.2). we find that while our 2.7b parameter model gives large gains over our 90m parameter model, our largest 9.4b model does not have a clear win in human evaluations over our 2.7b model, despite having lower perplexity. this is in line with previous observations, e.g., dialogue competitions are not always won by the model with the lowest perplexity @xcite , and models that take a small hit in perplexity but provide gains at decoding time can give far improved results @xcite . further refining and understanding these ingredients, and how they help to build the recipe as a whole, remain important directions. @xcite for retrieval encodes global features of the context using multiple representations (codes), which are attended to by each possible candidate response. this final attention mechanism gives improved performance over a single global vector representation, while being tractable to compute.