CRSL ab: An Open-Source Toolkit for Building Conversational Recommender System (2021)

Abstract: In recent years, conversational recommender systems (CRSs) have drawn a wide attention in the research community, which focus on providing high-quality recommendations to users via natural language conversations. However, due to diverse scenarios and data formats, existing studies on CRSs lack unified and standardized implementation or comparison. To tackle this challenge, we release an open-source toolkit CRSLab, which provides a unified and extensible framework with highly-decoupled modules to develop CRSs. Based on this framework, we collect 6 commonly used human-annotated CRS datasets and implement 19 models that include advanced techniques such as graph neural networks and pre-training models. Besides, our toolkit provides a series of automatic evaluation protocols and a human-machine interaction interface to evaluate and compare different CRS methods. The project and documents are released at https://github.com/RUCAIBox/CRSLab .

## introduction recent years have witnessed remarkable progress in recommender systems, which aim to present items (e.g., products or movies) of potential interests to users based on their preferences @xcite . traditional recommender systems mainly leverage the user historical behavior data (e.g., click or purchase) to estimate user preferences implicitly. recently, with the rapid development of human-machine conversation techniques @xcite @xcite , conversational recommender systems (crss) @xcite @xcite is gaining increasing attention in recent years. it relies on multi-turn natural language conversations to clarify explicit user preferences and generate more appropriate recommendations. to build an effective crs, researchers have proposed several datasets @xcite @xcite and models @xcite @xcite . however, due to their diverse scenarios (e.g., e-commerce or movie recommendation) and data formats (e.g., historical utterances or interacted items), it is challenging for users to quickly set up reasonable baseline systems and compare their performances. to alleviate the above issues, we have developed crslab, an open-source crs toolkit for research purpose. in crslab, we offer a unified and extensible framework to develop crss. based on this toolkit, users are able to quickly train and evaluate crs models via a few lines of code, and easily design new crs models using the provided interfaces. to implement the overall framework, we design six highly-decoupled modules (e.g., data module and model module), each module provides clear interfaces for specific functions. besides, we encapsulate useful procedures and common functions shared by different modules for users to add new datasets or develop new models using our toolkit. based on the framework, we integrate comprehensive benchmark datasets and models in crslab. so far, we have incorporated 6 commonly used human-annotated datasets and implemented 19 models, including advanced techniques such as graph neural networks (gnn) @xcite and pre-training models @xcite . to support these models, we perform necessary preprocessing (e.g., entity linking and word segmentation) on included datasets, and release the processed version. be-sides, crslab supports both configuration files and command-line instructions, which facilitate running, comparing and testing models on different datasets. furthermore, crslab provides a series of automatic evaluation protocols for testing and comparing different methods, covering commonly used metrics in existing works. it makes our work useful for the standardization of evaluation protocols for conversational recommendations. in addition, crslab provides a human-machine interactive interface to perform quantitative analysis, which is helpful for users to deploy their systems and converse with the systems via graphical interfaces. our contributions are as follows: ## background as aforementioned, the various scenarios and input formats in earlier works lead to inconvenience when applying existing crs models on different datasets. by surveying previous crs works @xcite @xcite , we summarize two basic tasks and an auxiliary task, namely recommendation task, conversation task and policy task. given the dialog context (i.e., historical utterances) and other useful side information (e.g., user historical behaviors and knowledge graphs), the recommendation sub-task is defined as predicting user-preferred items (e.g., movies or products), and the conversation sub-task is to generate a proper response for conversing with the user. in existing works, the recommendation and the conversation sub-tasks are considered as the primary objective of the crs. as a complementary sub-task, policy sub-tasks are proposed by recent works @xcite @xcite to better control the overall crs. it usually focuses on selecting appropriate system actions (e.g., recommendation or conversation) or tracking dialog states (topic prediction or goal tracking) to proactively guide the conversation. ## crslab the overall framework of our toolkit crslab is presented in figure 1 . the configuration module provides a flexible interface for users to easily set up the experiment environment (e.g., datasets, models and hyperparameters). the data, model and evaluation modules are built upon the configuration module, which forms the core part of our toolkit. the bottom part is the utility module, providing auxiliary functions and interfaces for reuse in other modules (e.g., layers and scheduler). in the following part, we briefly present the design of the above modules. more details can be found in the toolkit documents. ## system demonstration in this section, we show how to use our crslab with code examples. we detail the usage descriptions in two parts, namely running an existing crss in our toolkit and implementing a new crs based on the interfaces provided in our toolkit. ## conclusion in this paper, we released a toolkit called crslab, which is the first open-source conversational recommender systems (crss) toolkit for research purpose. in crslab, we offered a unified and extensible framework with highly-decoupled modules to develop crss. based on this framework, we have incorporated 6 datasets and implemented 19 models in our toolkit. besides, we also provided extensive automatic evaluation pro-tocols and a human-machine interactive interface in crslab, to help evaluate and compare different crss. for demonstration, we illustrated how to run or implement a crs using our toolkit. with this toolkit, we expect to help users quickly run existing crss, ease the development of new crss, and set up a benchmark framework for the research of crss. in the future, we will make continuous efforts to add more datasets and models. we will also consider adding more utilities for improving the usage of our toolkit, such as result visualization and algorithm debugging.