SGP - TOD : Building Task Bots Effortlessly via Schema-Guided LLM Prompting (2023)

Abstract: Building and maintaining end-to-end task bots using minimal human effort is a long-standing challenge in dialog research. In this work, we introduce SGP-TOD, Schema-Guided Prompting for building Task-Oriented Dialog systems effortlessly based on large language models (LLMs). Utilizing the predefined task schema, i.e., belief instruction and dialog policy, we instruct fixed LLMs to generate appropriate responses on novel tasks, without the need for training data. Specifically, SGP-TOD comprises three components: an LLM for interacting with users, a Dialog State Tracking (DST) Prompter to aid the LLM in tracking dialog states with the given belief instruction, and a Policy Prompter to direct the LLM to generate proper responses adhering to the provided dialog policy. Experimental results on Multiwoz, RADDLE, and STAR datasets show that our training-free strategy, SGP-TOD, yields state-of-the-art (SOTA) zero-shot performance, significantly surpassing the few-shot approaches. In a domain-extension setting, SGP-TOD aptly adapts to new functionalities by merely adding supplementary schema rules. We make our code and data publicly available.

## introduction building task-oriented dialog (tod) systems has been a long-standing challenge in artificial intelligence. the prevailing approach for creating task bots @xcite @xcite is to fine-tune pre-trained language models (plms), such as t5 @xcite and gpt-2 @xcite . despite their great success, developing and maintaining such task bots generally requires adequate annotated data and extensive fine-tuning/re-training. recently, large language models (llms), such as chatgpt @xcite and gpt-4 (openai, 1 https://github.com/zhangxy-2019/sgp-tod user: i am looking for a restaurant with moderate price range serving modern european food. llm dialog policy database system: i have at least 2 choices, do you have a preferred area? i need a restaurant that serves [value_food] food. request price range yes no preferred area? recommend restaurant request area yes preferred price range? no dst prompter restaurant @xmath0, golden wok, etc.; @xmath1, chinese, british, etc.; @xmath2, cheap, moderate, expensive; @xmath3... belief instruction task schema policy prompter sql: select * from restaurant where @xmath4; @xmath5: restaurant two match action: restaurant (inform (choices), request (area)) 2023), have revolutionized natural language processing (nlp) applications @xcite , owing to their remarkable conversational skills @xcite , instruction-following abilities @xcite and zero-shot generalization capabilities @xcite . this raises a research question: can llms be effectively utilized for building task bots with minimum human effort? a contemporary study @xcite explores the potential of llms for rapidly building task bots via few-shot prompting, a.k.a. in-context learning (icl) paradigm @xcite . though demonstrably effective, the icl performance is highly influenced by the quality of the in-context exemplars @xcite @xcite , as they struggle to provide comprehensive information for dialog task completion. in this work, we introduce symbolic knowledge @xcite , i.e., the task schema into llms, for creating task bots. task schema @xcite encompasses a concise symbolic representation of a task, supplying llms with a comprehensive blueprint. it comprises (i) task-specific ontology containing all slots and their appropriate values @xcite ; and (ii) a dialog flow explicitly outlining fundamental interaction patterns @xcite . specifically, we propose sgp-tod (as depicted in figure 1 ), a schema-guided prompting method for rapidly building task bots. we integrate the predefined task schema and dialog context into prompts through the use of two specifically-designed prompters, namely a dst prompter and a policy prompter. utilizing these prompters, we adeptly guide fixed llms to track dialog states, retrieve database entries, and generate appropriate responses for novel tasks in a zero-shot manner, without the need for additional training or fine-tuning. by incorporating task-specific symbolic knowledge into llms, sgp-tod provides knowledge-based, coherent and human-like responses. moreover, this trainingfree design empowers developers to flexibly prototype dialog systems on new tasks, while seamlessly extending system functionalities through modifying the task schema. we perform empirical automatic evaluations on two multi-domain datasets, namely, multiwoz 2.0 and 2.2 @xcite , as well as two single-domain/task datasets, raddle @xcite and star @xcite , within zero-shot scenarios. additionally, we complement these assessments with interactive human evaluations. the results indicate that sgp-tod, employing merely task schema devoid of any training or fine-tuning, substantially boosts the sota zero-shot results, markedly outperforming few-shot prompting/fine-tuning methods, and even attaining competitive results cf. full-shot finetuning approaches. in a domain-extension context, sgp-tod proficiently adapts to new functionalities by simply adding a handful of schema rules without necessitating further data collection, significantly exceeding the few-shot prompting/finetuning methods reinforced by machine teaching @xcite . in summary, our contributions are three-fold: zero-shot task-oriented dialog modeling. zero-shot generalization is an essential yet challenging task in tod research. a comprehensive study is shown in appendix a. in this paper, we focus on zero-shot end-to-end dialog modeling, including policy management and dialog generation. the works by @xcite and @xcite utilize ontology and response templates to train dialog models, enabling the discovery of shared dialog policies between the source and target domains. to enable broader adaptation to diverse dialog policies, mosig et al. (2020); @xcite implement task-specific policy skeletons, training dialog models to adhere to novel policies. furthermore, @xcite employs a neural language model (lm) for tracking dialog states and user actions using slot and action descriptions; subsequently, a policy program is deployed to facilitate an lm in generating system actions and responses. despite the effectiveness of previous approaches, they still require ample fine-tuning and copious annotated dialog corpora on source or heterogeneous domains/tasks. a concurrent study to ours is @xcite , which employs a prompting strategy -ig-tod (instruction-guided tod) to guide frozen llms in generating suitable responses. specifically, ig-tod first tracks belief states by utilizing slot descriptions as prompts, then retrieves database entries, and generates responses. our sgp-tod differs in that: (i) we employ slot names and value examples, rather than slot descriptions, as prompts to facilitate frozen llms in generating belief states, thereby reducing human effort; (ii) we offer a policy skeleton to guide llms in producing appropriate responses. in addition, experimental results indicate that sgp-tod substantially outperforms ig-tod. leveraging llms for dialog tasks. llms @xcite have exhibited unparalleled mastery of natural language understanding, reasoning and generation @xcite . three primary research directions have obtained substantial success in numerous dialog tasks by utilizing llms. (i) few-shot prompting @xcite has showcased remarkable performance in intent classification @xcite , semantic parsing @xcite , dialog state tracking @xcite , and response generation @xcite . (ii) li et al. (2022); mehri et al. (2022); dai et al. (2023) employ llms for data augmentation, i.e., generating synthetic task-oriented dialogs to train smaller models for inference. (iii) recently, several studies endeavor to support llms in specialized tasks by incorporating external knowledge. peng et al. (2023) advocates for enhancing llms' responses with external knowledge and automated feedback to reduce hallucination. liang et al. ( 2023 ) suggests connecting llms with millions of apis to accomplish diverse tasks. different from the aforementioned works, we aim to employ llms in building task bots in a zero-shot manner using pre-defined task schema. ## conclusion we present sgp-tod, a schema-guided prompting strategy aimed at the expeditious construction of end-to-end task bots, relying exclusively on llms and the corresponding task schema. employing the symbolic knowledge -task schema, sgp-tod guides fixed llms to generate suitable responses for novel tasks in a zero-shot fashion. empirical findings on four well-studied datasets reveal that sgp-tod attains remarkable sota zero-shot performance, using both automatic and human evaluations. for future work, we plan to explore the use of sgp-tod to develop personalized chatbots by utilizing pertinent task schema.