Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills (2020)

Abstract: Being engaging, knowledgeable, and empathetic are all desirable general qualities in a conversational agent. Previous work has introduced tasks and datasets that aim to help agents to learn those qualities in isolation and gauge how well they can express them. But rather than being specialized in one single quality, a good open-domain conversational agent should be able to seamlessly blend them all into one cohesive conversational flow. In this work, we investigate several ways to combine models trained towards isolated capabilities, ranging from simple model aggregation schemes that require minimal additional training, to various forms of multi-task training that encompass several skills at all training stages. We further propose a new dataset, BlendedSkillTalk, to analyze how these capabilities would mesh together in a natural conversation, and compare the performance of different architectures and training schemes. Our experiments show that multi-tasking over several tasks that focus on particular capabilities results in better blended conversation performance compared to models trained on a single skill, and that both unified or two-stage approaches perform well if they are constructed to avoid unwanted bias in skill selection or are fine-tuned on our new task.

## introduction a good open-domain conversational agent should have a well-rounded set of skills 1 and qualities that allow it to seamlessly blend listening with empathy, providing knowledgeable responses, and talking about various topics from everyday life to their favorite hobbies or latest challenges. recent research has made solid strides towards gauging and improving performance of opendomain conversational agents along specific axes such as how knowledgeable they are @xcite @xcite , how well they can display empathy @xcite or talk about their personal background @xcite . however it remains unclear whether models optimized for performance along one of these axes can retain the learned skill while blending it with other desirable skills, or how to best conduct simultaneous training of multiple skills. in this work, we compare several ways to combine tasks designed to evaluate and improve a single conversational skill, ranging from multi-task training over several datasets to training a top-level classifier to play the role of a dialogue manager and query the most appropriate single-skill pretrained model for a response. in order to evaluate those methods, we propose a new englishlanguage dataset, blendedskilltalk, that blends several skills into a single conversation, and use it to evaluate methods with both automated metrics and human crowdsourced ratings across different axes. our experiments show that existing single-skill tasks can effectively be combined to obtain a model that blends all skills into a single conversational agent if care is taken to make the dialogue agent avoid unwanted biases when selecting the skill, or if fine-tuning on blended data, or both. we propose methods that compare those competing approaches, and provide a detailed analysis of their successes and failures. ## related work while most commercial dialogue systems rely on hand-coded narrow skills (e.g., see @xcite ; @xcite ), typically focusing on separate task-oriented features such as alarm setting, calendar entries, etc., we are interested in models that display various qualities in opendomain dialogue. further, we focus on skills that can be learned end-to-end, as end-to-end learning affords the promise of better generalization to unseen domains. recent promising conversational models have leveraged very large conversation-like data such as datasets extracted from reddit and made available by a third party on pushshift.io @xcite @xcite . these large-scale datasets are very useful in providing vast amounts of conversational material that allow for reproducible research and comparison with prior work, however the qualities of resulting conversational agents are dependent on the qualities present in the source conversations. given how online conversations can turn toxic and lack empathy, indiscriminate pretraining on such corpora is unlikely to spontaneously endow a conversational agent with desirable qualities such as avoiding toxic responses @xcite or demonstrating empathy @xcite or knowledge @xcite . this has led the community to propose tasks and datasets focusing specifically on some trait or skill. in this work, we examine how to combine three such traits that each have a corresponding task and dataset: demonstrating an ability to talk about oneself and get to know your partner, as captured by the convai2 dataset, an extension of the personachat dataset @xcite ; being knowledgeable and discussing a topic in depth, as measured through the wizard of wikipedia task @xcite ; and demonstrating empathy and being able to talk about emotional personal situations, as measured by the empatheticdialogues benchmark proposed in @xcite . the convai2 dataset comprises more than 140k utterances of crowdsourced conversations between paired workers getting to know each other. each worker was assigned a persona consisting of a few sentences such as "i have a pet hamster," which had separately been crowdsourced. the wizard of wikipedia (wow) task aims to explore conversation informed by expert knowledge from wikipedia, and provides about 194k utterances of conversations on about 1,250 topics. the em-patheticdialogues (ed) dataset consists in about 50k utterances between a speaker who is talking about an emotional situation, and a listener who is tasked to respond in an empathetic manner, acknowledging the other person's feelings. in addition to being associated with easy-to-use datasets, these three skills benefit from being clearly defined and separate in scope. focusing on blending only three skills keeps data collection, ablations, and analyses manageable while already presenting a challenge for models, and it helps narrow down the most promising approaches for blending a greater number of skills. ## blending skills in a conversation a model separately trained on a variety of skills might be able to do well on each of them in isolation, but still struggle to seamlessly blend them over the course of a single conversation where it has to navigate whether a given utterance calls for informative knowledge or empathy, for example. it must learn to switch between skills, each time incorporating previous dialogue context which may contain utterances from either partner relating to multiple skills, and on some turns may have to blend skills into a single response. ## experiments in section 4.1, we introduce the automated metrics and human evaluations that we use to measure and compare model performance. section 4.2 discusses how adding personas and topic strings during multi-task training de-biases the selection of retrieval candidates from across our three skillbased tasks. sections 4.3 and 4.4 detail the performance of our models using automated metrics on single-skill and blendedskilltalk benchmarks, respectively, and section 4.5 compares the performance of the models on human evaluation: in all three cases, models trained on all three skills generally outperform those trained on individual skills. ## discussion and conclusion this paper focuses on the goal of creating an open-domain conversational agent that can display many skills, and blend them in a seamless and engaging way. we have shown several ways to leverage previous work focusing on individual conversational skills, either by combining trained singleskill models in a two-stage way, by re-using the datasets for simultaneous multi-task training, and by fine-tuning on the overall blended task. we compared the performance of these schemes on blendedskilltalk, a new english-language dataset blending three conversation skills in balanced proportions (demonstrating knowledge, empathy, or ability to talk about oneself). we showed that multiple multi-task approaches can be effective on this task, however careful construction of the training scheme is important to mitigate biases when blending and selecting skills, while fine-tuning on the overall blended task improves models further. one natural extension would be to generalize these findings to other skills than the three addressed here, such as humor/wit, eloquence, image commenting, etc. this would in principle be straightforward to do as long as these additional skills have a corresponding "single-skill" dataset to train on and are sufficiently distinguishable from each other.