The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2020)

Abstract: We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pre-trained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge.

## introduction one of the goals of ai is to build a seeing, talking agent that can discuss, reason, empathize, and provide advice -in short a system that can perform natural communication displaying many of the properties expected when speaking to a human partner. ideally, it should be able to be knowledgeable and personable, expert and engaging, serious or humorous -depending on the situation. it should be capable of answering questions, asking questions, responding to statements, having its own persona, and grounding the dialogue with external information and images. while no single task exists that can train an agent or measure its ability on all of these axes at once, a number of distinct large-scale datasets targeting subsets of these skills have recently become available. we thus assemble these disparate tasks to form a single challenge: dodecadialogue, consisting of 12 subtasks. each contains both training data to build the skills we desire for our agent, and validation and test sets to measure our agent's ability at that skill. the overall goal is a single agent that can display all these skills. as some of the subtasks have very large datasets, e.g. 2.2 billion utterances, they can possibly help the agent with other skills too. we thus build a model capable of training and multi-tasking on all these sources. we employ a transformer-based architecture @xcite which accepts an image, external textual information and dialogue history as input, and generates a response for a given dialogue turn. practically, by pre-training on the largest of the subtasks and then multi-tasking on all them, we can obtain state-of-the-art results compared to existing independently reported performance on all 10 of the 12 subtasks that have previous comparable results. we hence set a strong baseline for this challenge. while many existing approaches use large-scale pre-training on general text corpora, we show that using dialogue datasets instead, which are more closely linked to the desired agent's goals, is a strong alternative. however, many challenges remain. while multitasking performs well, and has clear benefits, as shown in other works @xcite , when compared to fine-tuning of the same system we do obtain typically small losses. zeroshot transfer to left-out tasks is also demanding for current approaches. we analyze these aspects, along with our model's ability to ground on external knowledge and images in conjunction with the dialogue context, the impact of decoding algorithms, analysis of the weighting of tasks during multi-tasking as well as cross-task transfer ability in order to shed light and make progress on this challenging topic. a s k q u e s t i o n s a n s w e r q u e s t i o n s r e s p o n d t o s t a t e m e n t s p e r s o n a g r o u n d i n g k n o w l e d g e g r o u n d i n g s i t u a t i o n g r o u n d i n g i m a g e g r o u n d i n g resp. name train valid test # turns length ## the dodecadialogue task the dodecadialogue task is intended to assemble important aspects of an engaging conversational agent into a single collection, where each subtask covers some of those goals. such an agent should be able to get to know you when you first talk to it (convai2), discuss everyday topics (dailydialog, pushshift.io reddit, twitter, cornell movie), speak knowledgeably at depth (wizard of wikipedia, ubuntu) and answer questions on such topics (eli5). it must be able to handle situated conversations and demonstrate empathy (empathetic dialog, light) . it can also discuss images, as this is a vital part of human connection (image chat, igc). we note that all of the provided subtasks are in english. the overall statistics of the subtasks are given in ## experiments task training we employ the parlai framework @xcite for training on single tasks and for multi-tasking, as many of the tasks are already implemented there, along with a (multitask) training and evaluation framework for such models. pre-training as pushshift.io reddit and (to some extent) twitter are much larger than our other tasks, we try pre-training the seq2seq module of our image+seq2seq networks with those datasets, before multi-tasking on all of the tasks, or for evaluating single task fine-tuning. model c o n v a i 2 w i z . o f w i k i p e d i a e m p a t h e t i c d i a l o g reddit 18.3 15.3 14.4 reddit+convai2 11.4 14.2 14.7 reddit+wiz. of wikipedia 16.3 8.7 14.0 reddit+empathetic dialog 17.9 15.3 11.3 multi-tasking all 4 tasks 11.6 8.7 11.2 a comment conditioned on the full thread leading up to the comment. comments containing urls or that were under 5 characters in length were removed from the corpus, as were all child comments. comments were truncated to 1024 bpe tokens. the model was trained with a batch size of 3072 sequences for approximately 3m updates using a learning rate of 5e-4, and an inverse square root scheduler. this took approximately two weeks using 64 nvidia v100s. we note that our transformer pre-training only includes text, while our image encoder was pre-trained separately in previous work @xcite . learning how to combine these sources occurs during fine-tuning. it is important to note that, while compute-heavy, pre-training was conducted exactly once, and all of the subsequent fine-tuning is significantly faster to run. transfer performance between tasks we first perform a preliminary study on a subset of the tasks: reddit, convai2, wizard of wikipedia and empathetic dialogues, and report the transfer ability of training on some of them, and testing on all of them (using the validation set), reporting perplexity. the results are reported in comparison of pre-training + fine-tuning strategies across all 12 tasks, we compare several pre-training strategies: using bert, no pretraining at all, only initializing via fasttext @xcite , and using twitter and pushshift.io reddit pre-training with our image+seq2seq architecture. for each variant we tune the learning rate, layers, number of heads and embedding size, with less pre-training typically requiring smaller capacity models. we then only fine-tune on a single task in these experiments, and report perplexity for that task alone, over all 12 tasks. the results are given in the results show a clear reduction in perplexity with more pre-training, as expected. this is most easily seen by the dodecascore (last row) that is the mean perplexity over all 12 tasks, which decreases from 49.5 (from scratch models) down to 17.1 with pushshift.io reddit pre-training. fasttext (45.7) and twitter (35.6) initializations help, but nowhere near as much. bert fares better, but still is clearly worse than pushshift.io reddit pre-training. the hypothesis here is that pushshift.io reddit yields much more effective transfer as it is a dialogue task like our others, whereas non-dialogue corpora such as wikipedia are not. this was previously observed for retrieval models in @xcite . note that we do not report results for the image dialogue tasks for bert as that architecture does not deal with images. finally, as pushshift.io reddit is so effective, we also compare to pushshift.io reddit training only, with no fine-tuning at all across all tasks, similar to our initial study in image and knowledge grounding some of our tasks involve grounding on knowledge or images. to show such grounding helps, we report results with and without grounding on those tasks in multi-task results next, we perform multitask training across all tasks, which is our ultimate goal in order to obtain an open-domain conversational agent. we optimize over the same set of hyperparameters as before, including multi-tasking weights for tasks, where one samples during training with differing probabilities, and we choose the best model by performing early stopping on the average performance across all tasks. in this way, we treat all 12 tasks as a single task, and thus during test time it is the model's responsibility to understand how to respond from the context (image/dialogue) itself. in the end we did not obtain clear improvements beyond pre-training with pushshift.io reddit and then equally sampling from all tasks. we report that final model's validation performance in terms of perplexity in multi-task followed by fine-tuning as also performed in @xcite ; @xcite we can try to train in a multi-task manner on all tasks, before fine-tuning on a single task, and build a separate model performing this procedure for all tasks, in an attempt to improve single task results further. using this approach, one is free to perform hyperparameter search differently for each task. here, we found that applying relative task up-weighting during multi-tasking training made a clear difference to the final quality of the fine-tuned target task model, see decoding strategies so far, we have only been measuring perplexity, but we are actually interested in generation, which requires us to decode. we consider several standard approaches: greedy, beam search (with beam size, and minimum and maximum output length foot_1 hyperparameters), beam search with beam blocking (blocking n-grams, we use @xmath0. we show the effect of these choices in final systems the final test performance for our best multi-task and fine-tuned (via multi-task followed by fine-tuning) systems are reported in comparison to existing systems we compare to existing state-of-the-art results previously published for each task. results are given in human evaluation in addition to automatic metrics, we perform human evaluation on two of the tasks to assess the abilities of our all tasks mt conversational agent: the knowledge grounding task wizard of wikipedia (wow) and the image grounding task image chat. we follow the same evaluation protocols as in @xcite ; @xcite , comparing our method to the existing approaches referenced in the results, given in figure 1 , show our method outperforming the existing state of the art generative models on all three comparisons: image chat, wow seen topics and wow unseen topics. all three results are statistically significant (binomial test, @xmath1.05). additional details and results breakdown are given in appendix section b. example outputs we show some example outputs of our multi-task model for some of the tasks in appendix c. our model is able to leverage images, knowledge, and given personality attributes to produce engaging dialogue with a large amount of variety, depending on the situation. leave-one-out zero-shot performance last, but not least, we evaluate the performance of a multi-task model at zero-shot transfer to a new dialogue task. this is performed by training on all but one of the tasks, and reporting performance on the left out one, repeating this experiment for all tasks. our best performing models in that regard are reported in ## discussion we have introduced the dodecadialogue task, and provide strong baseline results leveraging multimodal image+seq2seq transformers trained across all tasks. the goal of introducing this task is not just as another challenge dataset, but to further motivate building and evaluating conversational agents capable of multiple skills -one of the core goals of ai. we believe current systems are closer to that goal than ever before -but we also still have a long way to go. recently reported results show systems can be reasonably competitive compared to humans in particular domains for short conversations @xcite . this work tries to bridge the gap to avoid agents with niche skills, to move towards evaluating an open-domain set of skills. still, despite leveraging 12 tasks, there are many skills not included in our set. for example, longer conversations involving memory @xcite , or mixing open-domain conversation with task oriented goals. future work should consider adding these tasks to the ones used here, while continuing the quest for improved models.