D ialog S tudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI (2024)

Abstract: Despite advancements in conversational AI, language models encounter challenges to handle diverse conversational tasks, and existing dialogue dataset collections often lack diversity and comprehensiveness. To tackle these issues, we introduce DialogStudio: the largest and most diverse collection of dialogue datasets, unified under a consistent format while preserving their original information. Our collection encompasses data from open-domain dialogues, task-oriented dialogues, natural language understanding, conversational recommendation, dialogue summarization, and knowledge-grounded dialogues, making it an incredibly rich and diverse resource for dialogue research and model training.To further enhance the utility of DialogStudio, we identify the licenses for each dataset, design external knowledge and domain-aware prompts for selected dialogues to facilitate instruction-aware fine-tuning. To improve transparency and support dataset and task-based research, as well as language model pre-training, all datasets, licenses, codes, and models associated with DialogStudio will be made publicly accessible.

## introduction recent years have seen remarkable progress in conversational ai, primarily driven by the advent of approaches and language models @xcite @xcite . despite the advancements, these models could fall short when handling various tasks in a conversation due to the lack of comprehensive and diverse training data. current dialogue datasets @xcite are typically limited in size and taskspecific, which thus results in suboptimal ability in task-oriented model performance. additionally, the lack of dataset standardization impedes model generalizability. a few recent works @xcite @xcite have introduced a large collection of datasets, which includes diverse tasks based on public datasets. for instance, flant5 @xcite presents the flan collections with a wide array of datasets and tasks. despite this breadth, the coverage of dialogue datasets within the flan collection remains notably sparse, featuring only about ten datasets. although opt @xcite have incorporated collections with several dialogue datasets, these collections remain inaccessible to the public. in contract, efforts like instructdial @xcite and @xcite consist of more dialogue datasets, but they lack diversity and comprehensiveness. for instance, parlai mainly includes open-domain dialogue datasets, which are exclusively accessible through their platform. other collections @xcite @xcite often distill single dataset from chatgpt or process datasets into a sequence-to-sequence format to support language model training, featuring only input-output pairs such as dialogue context and system response. however, previous collections often overlook other crucial dialogue information, constraining their utility for research on individual datasets, tasks, and broader applications. to overcome the aforementioned challenges, we introduce dialogstudio, the most comprehensive and diverse collection of publicly available dialogue datasets, unified under a consistent format. by aggregating dialogues from various sources, dialogstudio promotes holistic analysis and the development of models adaptable to a variety of conversational scenarios. the collection spans an extensive range of domains, aspects, and tasks, and it is inclusive of several categories: open-domain dialogues, task-oriented dialogues, natural language understanding, conversational recommendation, dialogue summarization, and knowledge-grounded dialogues. thus, it can provide support for research in both individual dialogue tasks and largescale language pre-training. dialogstudio stands out not only for its comprehensive coverage but also for its accessibility. it offers easy access with a unified format and documents. a straightforward load dataset() command through huggingface allows users to seamlessly interact with the collection, and we have included documentation for each dataset to enhance usability. we anticipate that this collection will enable comprehensive and standardized training and evaluations of dialogue models, fostering fair comparisons and propelling further advancements in conversational ai. furthermore, we identify dialogue domains, design external knowledge for available dialogues and create tailored prompts for selected datasets accordingly. leveraging these datasets from di-alogstudio, we have constructed instruction-aware models, with capacities ranging from 770m to 3b parameters. these models have the ability to handle various external knowledge and are adept at both response generation and general tasks, demonstrating the benefits of dialogstudio. the main contributions of this paper are as follows: ## datasets unification and access we collect and process a wide range of datasets, involving different domains, types, and tasks. since these datasets originally contain various information and format, we propose a unification strategy to process all the datasets such that they can be loaded in the same data loader. ## experiments in this section, we present the pre-training details, methodologies, and metrics used to assess the performance of our dialogstudio model. the evaluation process aims to measure the model's ability to both solve task-oriented dialogues and understand general prompt-based instruction. ## conclusion in this study, we have introduced dialogstudio, a comprehensive collection that aggregates more than 80 diverse dialogue datasets while preserving their original information. this aggregation not only represents a significant leap towards consolidating dialogues from varied sources but also offers a rich tapestry of conversational patterns, intents, and structures, capturing the nuances and richness of human interaction. utilizing dialogstudio, we developed corresponding models, demonstrating superior performance in both zero-shot and few-shot learning scenarios. in the spirit of open research and advancing the field, we are committed to releasing dialogstudio to the broader research community.