B ot SIM : An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems (2022)

Abstract: We present BotSIM, a data-efficient end-to-end Bot SIMulation framework for commercial task-oriented dialog (TOD) systems. BotSIM consists of three major components: 1) a Generator that can infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing; 2) an agenda-based dialog user Simulator (ABUS) to simulate conversations with the dialog agents; 3) a Remediator to analyze the simulated conversations, visualize the bot health reports and provide actionable remediation suggestions for bot troubleshooting and improvement. We demonstrate BotSIM’s effectiveness in end-to-end evaluation, remediation and multi-intent dialog generation via case studies on two commercial bot platforms. BotSIM’s “generation-simulation-remediation” paradigm accelerates the end-to-end bot evaluation and iteration process by: 1) reducing manual test cases creation efforts; 2) enabling a holistic gauge of the bot in terms of NLU and end-to-end performance via extensive dialog simulation; 3) improving the bot troubleshooting process with actionable suggestions. A demo of our system can be found at https://tinyurl.com/mryu74cd and a demo video at https://youtu.be/qLPJm6_UOKY .

## introduction the typical dialog system development cycle consists of dialog design, pre-deployment testing, deployment, performance monitoring, model improvement and iteration. as in any production software system, effective and comprehensive testing at all stages is of paramount importance. unfortunately, evaluating and troubleshooting production tod systems is still a largely manual process requiring large amount of human conversations with the systems. this process is time-consuming, expensive, and inevitably fails to capture the breadth * work done at salesforce research. figure 1 : botsim overview including the generator, simulator, and remediator. the dotted (optional) paths from users can be used for bot performance monitoring: they can provide production chat logs or manually crafted utterances when creating evaluation goals. to address these limitations, we present botsim, a bot simulation environment for data-efficient end-to-end commercial bot evaluation, remediation via multi-intent dialog generation and agenda-based dialog user simulation @xcite . botsim consists of three major modules, namely generator, simulator, and remediator (figure 1 ). we use a pretrained sequence-to-sequence t5 model @xcite in the generator to simulate lexical and syntactic variations in user queries via paraphrasing. the generator is also responsible to generate various templates needed by the simulator. to make botsim more platform-and task-agnostic, we adopt dialogact level abus to simulate conversations with bots via apis. the dialog acts are automatically inferred by the generator via a unified interface to convert bot designs of different platforms to a universal graph representation. the graph has all dialogs as nodes and their transitions as edges. through graph traversal, botsim offers a principled and scalable approach to generating and exploring multi-intent conversations. not only can the conversation path generation greatly increase evaluation coverage for troubleshooting dialog errors caused by faulty designs (e.g., unexpected dialog loops), it is also valuable for bot design improvements. the remediator summarizes bots' health status in a dashboard for easy comprehension. it also analyzes the simulated conversations to identify any issues and further provides actionable suggestions to remedy them. botsim's "generation-simulation-remediation" paradigm can significantly accelerate bot development and evaluation, reducing human efforts, cost and time-to-market. our contributions include: there are two main categories of dialog user simulators, namely the agenda-based user simulator (abus) @xcite @xcite @xcite and recent neural-based user simulator (nus) @xcite @xcite @xcite . since botsim is designed to support commercial bot evaluation and remediation, we focus on the review of the testing capacities offered by some existing bot platforms rather than the simulators. recently, abus is also used in amazon's alexa conversation @xcite for training an end-to-end dialog agent, which is also beyond the scope of our discussion. ## botsim botsim system overview is shown in figure 1 . ## conclusion we presented botsim, a modular end-to-end bot simulation framework for multi-intent dialog generation and evaluation of commercial tod systems via agenda-based dialog user simulation. our case studies show that botsim can save substantial manual effort in bot evaluation, troubleshooting and improvement. botsim can be easily extended to support new platforms by implementing a set of well-defined parser functions to convert bot messages to dialog acts. we are in the midst of opensourcing the codes including the web app. we also plan to support more platforms as future work.