B ots T alk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets (2022)

Abstract: To build open-domain chatbots that are able to use diverse communicative skills, we propose a novel framework BotsTalk, where multiple agents grounded to the specific target skills participate in a conversation to automatically annotate multi-skill dialogues. We further present Blended Skill BotsTalk (BSBT), a large-scale multi-skill dialogue dataset comprising 300K conversations. Through extensive experiments, we demonstrate that our dataset can be effective for multi-skill dialogue systems which require an understanding of skill blending as well as skill grounding. Our code and data are available at https://github.com/convei-lab/BotsTalk.

## introduction a considerable progress has been made towards open-domain chatbots with different desirable qualities in conversations. each of these models is capable of being specialized in one communicative skill, i.e., skill grounding. a number of distinct largescale datasets targeting a specific conversational skill have recently become available. convai2 @xcite is provided for research work that aims to endow chatbots with personas @xcite , enabling chatbots to talk about themselves. wizard of wikipedia (wow) @xcite is a popular option for recent studies @xcite @xcite ) that focus on knowledgeable conversational agents discussing topics in depth. empathetic dialogues (ed) @xcite is also commonly used to embody empathy in dialogue systems @xcite . most of such skill-grounded datasets are designed to improve a single skill, and thus effective when models are asked to demonstrate the targeted conversational skill. benefiting from the advances of these conversational agents, recent research focuses on another aspect of open-domain chatbots: the ability to blend various conversational skills into one cohesive flow in a seamless manner, i.e., skill blending. a good open-domain chatbot should be able to weave multiple behaviors and skills in a single conversation, so that it enables to deal with different users and situations appropriately @xcite . towards this goal, there is a need to construct a multi-skill dialogue dataset, which consists of multi-turn dialogues that exhibit multiple skills. while @xcite propose a crowdsourced dataset blended skill talk (bst) of 5k conversations as a reliable benchmark for measuring dialogue systems' ability at the blended objective, it is not sufficient to build a multi-skill chatbot due to its limited scale. scaling up crowdsourcing is not feasible, as it requires labor intensive manual annotation and verification. instead, automatic curation shows promising results on large-scale dialogue generation @xcite . in this paper, we aim to generate a large-scale multi-skill dialogue dataset without additional costs or human efforts. to this end, we introduce an automatic data curation approach named botstalk, where multiple dialogue agents grounded to individual skills engage in the conversation to blend all skills together. based on this framework, we create blended skill botstalk (bsbt), a large-scale multi-skill dialogue dataset of 300k conversations blended and grounded with a number of skills derived from convai2, wow, and ed. our experiments demonstrate that by using our dataset dialogue models successfully yield large performance gains in skill blending while maintaining competitive performance in skill grounding. furthermore, we validate the quality of bsbt dataset by human evaluation, showing our machine-sourced conversations are consistently preferred over crowdsourced ones from bst by human judges across all metrics. ## problem formulation in this section, we formulate the problem of multiskill dialogue annotation and desirable characteristics for the dialogue dataset as a training resource. ## botstalk framework we now present botstalk, a novel framework that automatically annotates multi-skill dialogues based on multiple single-skill dialogue datasets. the focus of our framework is to mimic a natural conversation by featuring both skill blending and grounding within a dialogue episode. figure 1 illustrates three main phases of the framework. implementation details are provided in appendix c. ## conclusion to build multi-skill chatbots, we construct a largescale dialogue dataset bsbt through automatic botstalk framework. we validate its efficacy as training resource by experiments and analyses.