C onv L ab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format (2023)

Abstract: Task-oriented dialogue (TOD) systems function as digital assistants, guiding users through various tasks such as booking flights or finding restaurants. Existing toolkits for building TOD systems often fall short in delivering comprehensive arrays of data, model, and experimental environments with a user-friendly experience. We introduce ConvLab-3: a multifaceted dialogue system toolkit crafted to bridge this gap. Our unified data format simplifies the integration of diverse datasets and models, significantly reducing complexity and cost for studying generalization and transfer. Enhanced with robust reinforcement learning (RL) tools, featuring a streamlined training process, in-depth evaluation tools, and a selection of user simulators, ConvLab-3 supports the rapid development and evaluation of robust dialogue policies. Through an extensive study, we demonstrate the efficacy of transfer learning and RL and showcase that ConvLab-3 is not only a powerful tool for seasoned researchers but also an accessible platform for newcomers.

## introduction task-oriented dialogue (tod) systems converse with their users in natural language to help them fulfil a task, such as booking a flight or finding a restaurant. unlike chit-chat dialogues, a critical aspect of these systems is that they are grounded in an ontology that contains domains, slots, and values which describe the dialogue task, i.e. user goal, as well as including domain-specific databases. there are two distinct capabilities that tod systems need to exhibit. they need to track the state of the dialogue and based on that decide on the next action to take in order to steer the conversation towards fulfilling the user's goal @xcite . the architecture of tod systems typically adopts a modular approach, often encompassing components like dialogue state trackers and policies, and may include language understanding or generation units, as depicted in figure 1 . the complexity of a tod system necessitates a toolkit with advanced, easily integrable modules allowing for straightforward training, evaluation, and combination. the vast amount of possible user behaviours and tasks that a tod system might assist with necessitates the study of generalization and transfer towards new users and datasets. while many datasets for studying task-oriented dialogue have been proposed @xcite @xcite @xcite @xcite , the various dialogue, ontology and database formats hinder researchers from validating their models on unseen data. in this work we propose a unified format to bridge the gap between different tod datasets and models and provide a unified training and evaluation framework that accelerates the study of generalization capabilities. once a dataset is transformed into the unified format, it can be immediately used by supported models. similarly, once a model supports the unified format, it can access all supported datasets. this feature reduces the cost of adapting m models to n datasets from m × n to m + n . the dialogue policy, as the decision-making component of a tod system, is pivotal to the success or failure of a dialogue task. it is typically optimized using reinforcement learning (rl), necessitating additional components such as algorithms, figure 1 : convlab-3: the unified format serves as a bridge, connecting diverse datasets and dialogue models. it streamlines the integration of various tod modules, including supervised learning, evaluation, and a wide array of essential evaluation metrics, thanks to the unified data loader and evaluator. these modules can be incorporated, either in the agent or user simulator, through a configuration file, defining the environment for interactive evaluation and reinforcement learning. convlab-3 is especially useful for practitioners seeking to construct a dialogue system without extensive expertise. additionally, it provides a fast, convenient, and dependable platform for both novice and experienced researchers to conduct experiments. in particular, it enables: (1) researchers to perform experiments across a variety of datasets, (2) developers to construct an dialogue system using custom datasets, and (3) community contributors to consistently add models and datasets. in summary, our contributions are: ## related work while rasa @xcite , nemo @xcite and dialoguestudio @xcite provide unified data formats, they do not have rl tools or user simulators for interactive training and evaluation of dialogue systems. parlai @xcite includes a reward attribute in their unified format, but without accessible rl tools. pydial @xcite and the predecessors of convlab-3 @xcite provide reinforcement learning toolkits, however they lack a unified format and thus the possibility to study generalization across datasets. moreover, pydial and previous versions of convlab do not provide multiple data-driven user simulators and their training evaluation provides no tools for in-depth analysis. in addition, none of the above toolkits provide a sufficient set of state-of-the-art models for the different components in a tod system. ## unified format in our unified format, a dataset consists of (1) an ontology that defines the annotation schema, (2) dialogues with transformed annotations, and (3) a database that links to external knowledge sources (see figure 1 ). typically converting the formats of different datasets is not straightforward, hindering format adaptation of existing and new corpora. however, in convlab-3 we provide detailed guidelines and scripts that make the process of format adaptation straightforward and error-free. convlab-3 offers a large number of datasets in the unified format as shown in moreover, as shown in listing 1, we provide utility functions to process the unified datasets, such as delexicalization, splitting data for few-shot learning, and loading data for specific tasks. based on the unified format, evaluations of common tasks across models and corpora are standardized, which facilitates comparability. more details of already supported datasets and tasks can be found in appendix a and b, respectively. ## integrated models convlab-3 provides a wide array of standard and state-of-the-art models covering all modules in a tod system. this allows straightforward plug-andplay experimentation when developing a specific module, as well as building tod systems easily on custom datasets. a model is considered integrated once it implements the corresponding module interface and supports processing datasets in the unified format. besides existing models in convlab-2 @xcite , we integrate new transformer-based models supporting the unified data format, including @xcite and trippy @xcite for dialogue state tracking (dst), ddpt @xcite and lava @xcite for policy learning, sc-gpt @xcite for natural language generation (nlg), and soloist @xcite with t5 as backbone model @xcite for end-to-end modeling (end2end). we also integrate multiple powerful data-driven user simulators (us): tus @xcite ) that outputs user dialogue acts, gentus @xcite ) that outputs both user dialogue acts and response, and emous @xcite that additionally outputs emotions. in addition, we apply text-generation models to solve the tasks of tod modules (see appendix c. we provide a range of models built upon t5 @xcite , covering nlu, dst, nlg, etc. we also provide an interface to instruct large language models (llms) such as chatgpt and llama @xcite to serve as different modules such as user simulators, nlu, dst, nlg, etc. see @xcite for an example of how chatgpt can be instructed to serve as a dst model. all integrated models are shown in appendix b. ## reinforcement learning toolkit the difficulty of building a comprehensive tod toolkit lies in the fact that it needs to support not only supervised but also reinforcement learning. as shown in figure 1 , this includes functionalities to build configurable agents and user simulators consisting of different modules, an evaluator to provide reward signals, and analysis tools to evaluate the training process and rl algorithms. convlab-3 supports the straightforward combination of components with an easy-to-use configuration file, including the definition of the interactive environment given by the choice of user policy and its components, see appendix g for an example. the dialogue policy module obtains the semantic information of the dst (and nlu) as input and produces a list of atomic actions [(domain 1 , intent 1 , slot 1 ), ...] as output, e.g. [(hotel, inform, phone), (hotel, inform, addr)], which results in a large action space due to the high number of possible atomic actions and their combinations. as the input is on semantic level while the policy network expects vectorized input, convlab-3 provides a vectoriser class that acts as communication module between semantic and vector representation. we treat the vectoriser as an additional pipeline module, which allows straightforward investigation of different vectorization strategies in a plug-and-play fashion. moreover, policy networks can be used off-theshelf while only the vectoriser needs to be adapted. convlab-3 provides a base vectoriser class that can be easily adapted, as well as common vectorization strategies. in addition, we add the possi-bility for masking certain actions as inultes et al. @xcite . this allows controllability of the policy output and facilitates learning during rl due to reduction of the large action space. moreover, in addition to the on-policy rl algorithms rein-force @xcite and ppo @xcite , which are already implemented in convlab-2, we provide the state-of-the-art continual rl model ddpt together with state-of-the-art algorithms vtrace @xcite and clear @xcite for off-policy (sutton and barto, 2018) and continual rl @xcite , respectively. ## supervised learning experiments conducting supervised learning experiments on multiple tod datasets is convenient with the unified data format. we believe this feature will encourage researchers to build general dialogue models that perform well on various data as well as to investigate knowledge transfer. in these experiments, we demonstrate the ease of evaluating a model's knowledge transfer abilities using our unified format. initially, we pre-train all models on the schema-guided dialogue (sgd) @xcite and taskmaster-1&2&3 @xcite datasets jointly. these models are then finetuned on multiwoz 2.1 @xcite in fulldata or low-resource settings. to configure these different training setups, one only needs to make a few changes to the unified dataloader parameters, as depicted in listing 1. for low-resource finetuning, we set the data ratios of both training and validation set to 1% and 10%. in the low-resource setting, we observe that pre-training is beneficial, as evidenced in ## reinforcement learning experiments convlab-3 supports a convenient way to run rl training and evaluation supported by the unified format and availability of multiple user simulators. to showcase this, we run transfer learning experiments as well as experiments with multiple user simulators. ## conclusion in this paper, we present the dialogue system toolkit convlab-3, which puts a large number of datasets under one umbrella through our proposed unified data format. the usage of the unified format facilitates comparability and significantly reduces the implementation cost required for conducting experiments on multiple datasets. in addition, we provide recent powerful models for all components of a dialogue system and provide a convenient rl toolkit which enables researchers to easily build, train, analyze and evaluate dialogue systems. we showcase the advantages of the unified format and rl toolkit in a large number of experiments, ranging from pre-training to rl training. the release of convlab-3 supports the community in developing the next generation of task-oriented dialogue systems.