Rethinking Task-Oriented Dialogue Systems: From Complex Modularity to Zero-Shot Autonomous Agent (2024)

Abstract: Task-oriented dialogue (TOD) systems are predominantly designed to be composed of several functional modules (e.g. dialogue state tracker, dialogue policy, natural language generation) whether they are pipeline or end-to-end architectures. However, this modular design not only heavily relies on massive fully-annotated data, but also suffers from many intrinsic drawbacks, such as serious error accumulation, poor generalization ability, high customization cost, and low fault tolerance rate. In this paper, we rethink the architecture of the task-oriented dialogue systems and propose a novel fully zero-shot autonomous TOD agent, named AutoTOD, where all the delicate modules in traditional TOD systems are deprecated and all it needs is a general-purpose instruction-following language model (e.g. GPT-4). AutoTOD only leverages a simple instruction schema consisting of the description of tasks and external APIs, and can autonomously decide what to do at each dialogue turn, including asking for information, calling APIs, summarizing API results, and correcting previous mistakes. Moreover, we propose a simulation-based evaluation framework to better validate the abilities of TOD models in real-life scenarios. Extensive experiments conducted on the MultiWOZ and SGD datasets show the superior task completion ability and flexible language skills of AutoTOD.

## introduction task-oriented dialogue (tod) systems have gained increasing attention in recent years both in the research community and the industry @xcite . they are designed to help users complete specific tasks through natural language interactions, such as querying flight tickets and booking restaurant tables @xcite ., natural language understanding dialogue state tracking dialogue policy learning natural language generation causal language model user utterance dialgoue state db results system action response db external api 1 external api 2 external api 3 instruction following lm dialogue insturction 2020). traditional tod systems are mostly designed as a pipeline of several separate modules, including natural language understanding, dialogue state tracker, dialogue policy, and natural language generation @xcite . these modules are trained separately and work one by one to generate the dialogue response to the user @xcite . later, end-to-end tod systems emerged where the separate modules are combined and built on a single pretrained language model @xcite . thus the whole system can be trained end-to-end with annotated task dialogues. examples of these two kinds of tod systems are shown in @xcite . nevertheless, both the pipeline and end-to-end models are essentially in the same modular architecture. the classic modular tod framework has many intrinsic drawbacks in it. firstly, the modules are connected sequentially. any mistake in a module will propagate to the subsequent ones. although there are later works to merge some adjacent modules, the error propagation problem has not been fundamentally solved @xcite . secondly, the training of these systems requires a large number of fully-annotated task dialogues. at the same time, the capabilities of such tod systems are also severely limited to the training data, which makes it difficult for them to extend to new dialogue scenarios @xcite . thirdly, when encountering dialogue tasks in new forms, the module architecture usually needs to be redesigned, which makes building and maintaining a long-running tod system a challenging and costly endeavour @xcite . finally, the fault tolerance ability of the systems is quite poor. when facing inappropriate input utterances or misunderstandings in previous turns, the systems struggle to lead the dialogue back to the correct path @xcite . to tackle the above problems, in this paper, we rethink the architecture of the task-based dialogue systems and propose a novel fully zero-shot autonomous tod agent, named autotod. autotod gives up all the delicate functional modules in the traditional pipeline and end-to-end tod models, and the only thing it needs is a general-propose instruction-following language model, e.g. @xcite and llama 2 @xcite (shown in figure 1 (c) ). we propose an instruction schema to tell the base language model what tasks it will deal with and how it should do. the instruction schema is simple enough to easily apply to various dialogue tasks, thus the generalization ability is greatly improved and the customization cost is greatly reduced. autotod can intelligently and autonomously decide to call an external api with proper parameters and summarize the api results into the final responses, which greatly flexes the rigid process of state tracking and database querying in traditional tod models. to the best of our knowledge, autotod is the first tod system that completely deprecates the traditional fragile modular design and has the real zero-shot capability. to better evaluate the performance of auto-tod, we also propose a simulation-based evaluation framework to validate the abilities of tod models in real-life scenarios. we use an instructionfollowing language model to act as the user simulator, which has the task goals in its prompts and tries to achieve the goals via talking with the dialogue system. the dialogue system has no access to the user goals and can only do its best to complete the user's requests. then, we propose a novel tod evaluation approach to validate the system's ability to complete user goals where another instruction-following language model is used to extract the key information that the system provides to the user from the dialogues. by comparing with the goals for the user simulator, it's easy to conclude whether the tod system completes the user goals successfully. we conduct extensive experiments on the multiwoz @xcite and sgd @xcite datasets and evaluate the system performance by both our proposed simulation-based framework and the traditional tod evaluation approach @xcite . the results show the superior task completion ability and fluent language skills of autotod. furthermore, autotod demonstrates great robustness when facing various dialogue scenarios. ## the autotod agent autotod is a fully zero-shot autonomous taskoriented dialogue (tod) agent where all the delicate modules in traditional tod systems are deprecated and all it needs is a general-purpose instruction-following language model (e.g. gpt-4). the ability of autotod comes from two aspects: the language knowledge from the base language model, and the dialogue ability from the instruction schema we proposed for the dialogue tasks. the instruction schema is composed of three parts: scenario description, task information, and output format (a brief example is shown in figure 2 ). the detailed introduction of each part is given below. ## the simulation-based evaluation framework the autotod agent communicates with the user in an end-to-end manner where no dialogue state or system action is generated as in the traditional tod systems. besides, autotod directly outputs the natural language responses rather than the delexicalized utterances without real slot values. however, the classical tod evaluation approach @xcite used by almost all previous tod systems is deeply coupled with the traditional modular tod system architecture, and can not be used for autotod directly. therefore, we propose a new simulation-based evaluation framework that has no assumption about the architecture of the tod system. the framework we proposed consists of a user simulator and a dialogue evaluator. the detailed introduction is presented below. ## conclusion in this paper, we propose autotod, a novel fully zero-shot autonomous agent for task-oriented dialogues. autotod deprecates all the delicate modules in traditional tod systems and only uses an instruction-following language model to autonomously call external apis and communicate with the user, which greatly reduces the construction cost and improves the generalization ability. we also propose a simulation-based evaluation framework to evaluate tod systems in more real scenarios. extensive experiments demonstrate the superior task completion ability and flexible language skills of autotod.