C onv L ab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems (2020)

Abstract: We present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of ConvLab, ConvLab-2 inherits ConvLab’s framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. The analysis tool presents rich statistics and summarizes common mistakes from simulated dialogues, which facilitates error analysis and system improvement. The interactive tool provides an user interface that allows developers to diagnose an assembled dialogue system by interacting with the system and modifying the output of each system component.

## introduction task-oriented dialogue systems are gaining increasing attention in recent years, resulting in a number of datasets @xcite @xcite and a wide variety of models @xcite @xcite @xcite . however, very few opensource toolkits provide full support to assembling an end-to-end dialogue system with state-of-the-art models, evaluating the performance in an end-toend fashion, and analyzing the bottleneck both qualitatively and quantitatively. to fill the gap, we have developed convlab-2 based on our previous dialogue system platform convlab @xcite . convlab-2 inherits its predecessor's framework and extend it by integrating many recently proposed state-of-the-art dialogue models. in addition, two powerful tools, namely the analysis tool and the interactive tool, are provided for in-depth error analysis. convlab-2 will be the development platform for multi-domain task-oriented dialog challenge ii track in the 9th dialog system technology challenge (dstc9) foot_0 . as shown in figure 1 , there are many approaches to building a task-oriented dialogue system, ranging from pipeline methods with multiple components to fully end-to-end models. previous toolkits focus on either end-to-end models @xcite or one specific component such as dialogue policy (pol) @xcite , while the others toolkits that are designed for developers @xcite do not have state-of-the-art models integrated. convlab @xcite is the first toolkit that provides various powerful models for all dialogue components and allows researchers to quickly assemble a complete dialogue system (using a set of recipes). convlab-2 inherits the flexible framework of con-vlab and imports recently proposed models that achieve state-of-the-art performance. in addition, convlab-2 supports several large-scale dialogue datasets including camrest676 @xcite , multiwoz @xcite , dealorn-odeal @xcite , and crosswoz @xcite . to support end-to-end evaluation, convlab-2 provides user simulators for automatic evaluation and integrates amazon mechanical turk for human evaluation, similar to convlab. moreover, it provides an analysis tool and a human-machine interactive tool for diagnosing a dialogue system. researchers can perform quantitative analysis using the analysis tool. it presents useful statistics extracted from the conversations between the user simulator and the dialogue system. this information helps reveal the weakness of the system and signifies the direction for further improvement. with the interactive tool, researchers can perform qualitative analysis by deploying their dialogue systems and conversing with the systems via the webpage. during the conversation, the intermediate output of each component in a pipeline system, such as the user dialogue acts and belief state, are presented on the webpage. in this way, the performance of the system can be examined, and the prediction errors of those components can be corrected manually, which helps the developers identify the bottleneck component. the interactive tool can also be used to collect real-time humanmachine dialogues and user feedback for further system improvement. ## code and resources convlab-2 is publicly available on https:// github.com/thu-coai/convlab-2 . resources such as datasets, trained models, tutorials, and demo video are also released. we will keep track of new datasets and state-of-the-art models. contributions from the community are always welcome. ## conclusion we present convlab-2, an open-source toolkit for building, evaluating, and diagnosing a taskoriented dialogue system. based on convlab @xcite , convlab-2 integrates more powerful models, supports more datasets, and develops an analysis tool and an interactive tool for comprehensive end-to-end evaluation. for demonstration, we give an example of using convlab-2 to build, evaluate, and diagnose a system on the multiwoz dataset. we hope that convlab-2 is instrumental in promoting the research on task-oriented dialogue.