Are Large Language Models All You Need for Task-Oriented Dialogue? (2023)

Abstract: Instruction-finetuned large language models (LLMs) gained a huge popularity recently, thanks to their ability to interact with users through conversation. In this work, we aim to evaluate their ability to complete multi-turn tasks and interact with external databases in the context of established task-oriented dialogue benchmarks. We show that in explicit belief state tracking, LLMs underperform compared to specialized task-specific models. Nevertheless, they show some ability to guide the dialogue to a successful ending through their generated responses if they are provided with correct slot values. Furthermore, this ability improves with few-shot in-domain examples.

## introduction large language models (llms) have transformed the nlp field, showing outstanding performance across many nlp benchmarks such as winograd challenge @xcite or glue @xcite . recently, instruction finetuning of llms proved to be able to align the model outputs with human preferences @xcite and improved the llms' communication capabilities substantially. state-of-theart llms are not only good at understanding user needs but also capable of providing relevant answers. consequently, we see many chatbot applications both inside and outside academia (chat-gpt foot_0 , claude foot_1 , sparrow foot_2 ) which build upon the raw power of instruction-finetuned llms. given the millions of daily interactions with these chatbots, it appears that the models are able to handle users' needs to their satisfaction, at least to some extent. however, these chatbots are tuned using unstructured open-domain conversations. the then, an initial prompt is constructed and an llm is asked to provide the current dialogue state. based on that, we retrieve database results and construct another prompt. finally, we ask the llm to provide the response. i am looking for a five-star hotel in the north llm state tracker db context store context encoder retrieved examples user input history retrieved examples user input history state database (0) (1) (2) llm generator we've got 4 hotels available. (3) llm dom. detection few-shot zero-shot conversation pipeline (see figure 1 ) and evaluate its performance with respect to commonly used task-oriented metrics such as joint goal accuracy, slot f1, and dialogue success @xcite . our pipeline resembles other approaches based on lms @xcite , using state tracking and response generation as two main, separate steps, while keeping the role of a dialogue policy implicit. however, instead of finetuning lms, it intentionally relies almost exclusively on the usage of pretrained llms as-is, so we can test their out-of-the-box capabilities. the dialogue context and domain description are introduced to the model only by including them in the input prompt. in the zero-shot setting, the model receives a domain description only; in the few-shot setting, it additionally uses a few retrieved examples (see section 3 for details). in our experiments, we find that llms are not very good at state tracking and their performance falls behind the state-of-the-art. however, if provided with correct belief states, some of them yield interesting response generation performance, comparable to earlier finetuned state-of-the-art models. to our knowledge, our zero-shot experiments establish a state-of-the-art result in unsupervised tod modeling on the multiwoz and schema-guided datasets @xcite . while there may be room for improvement through prompt engineering, our results aim to show the out-of-the-box llm capabilities. we plan to release our experimental code on github. 4 4 https://github.com/vojtsek/to-llm-bot 2 related work large language models the transformer architecture @xcite enabled the training of large and capable language models. the research on their few-shot and zero-shot abilities dates back to the gpt-2 and gpt-3 models @xcite , which are scaled versions of the transformer decoder. many followed this path of training large transformer decoders @xcite , yielding models of up to hundreds of billions parameters in size @xcite . other models leverage the whole original (encoder-decoder) transformer architecture @xcite . recent research focuses on improving the training of moderate-sized architectures to broaden access to highly capable llms @xcite . instruction tuning the idea of using reinforcement learning techniques to align model-based agents better with users' intents was pioneered in game agent development @xcite and later explored for training language models @xcite . although these techniques proved to be quite effective, the process is still very demanding in terms of collecting feedback from users. consequently, several datasets were proposed @xcite @xcite that collected millions of instructions-based tasks in natural language and can be applied to finetune llms using reinforcement learning. lm-based tod modeling task-oriented dialogue modeling with pretrained lms was introduced by @xcite and @xcite , who followed text-based state encoding and two-stage generation proposed by @xcite : an lm is first used to decode a structured belief state, represented as text. the belief state is then used to retrieve database information and the lm is called once more to generate a response, conditioned on the belief state and retrieved information. several improvements to the basic setup were proposed, such as contrastive state training @xcite or using belief state differences @xcite . others proposed a combination of generative models with retrieval-based approaches @xcite @xcite . all described works finetune lms on in-domain data, which is in contrast with the pure in-context learning approach that we apply. few-shot dialogue modeling one of the first neural models focusing on learning dialogue from a few in-domain examples was the hybrid code networks @xcite , a trainable system based on recurrent neural networks, with partially handcrafted components. another approach was proposed by @xcite , who used latent action representations to enable the transfer of domain knowledge. latent actions were also used by @xcite and @xcite . more recent approaches leverage the capabilities of pretrained transformer lms @xcite . @xcite used llms and in-context learning to perform belief state tracking, formulating the task as an sql query generation. unlike our work, they did not use instruction-tuned models and omitted database retrieval and response generation. ## method we introduce our method step-by-step. an overall description of the proposed pipeline is shown in figure 2 . the system consists of a pretrained llm and an (optional) context store in a vector database. three llm calls are performed in each dialogue turn, with specific prompts (see section 3.1). first, the llm performs domain detection and state tracking (section 3.2). the updated belief state informs a database query, whose results are used in the subsequent llm-based response generation step (section 3.3). in the few-shot setting, the context store is used to store a limited number of examples from the training set, which are retrieved based on similarity with the conversation context and included in llm prompts (see section 3.4). prompt definition: capture values from a conversation about hotels. capture pair "entity:value" separated by colon and no spaces in between. separate the "entity:value" pairs by hyphens values that should be captured are: output: pricerange:"cheap" ## experimental setup to obtain a broad overview of the current llms' capabilities, we compare several models, spanning different numbers of trainable parameters and different training methods. we also experiment with four variants of the base setup, using either zeroshot or few-shot operations and using either predicted or oracle belief states. ## model analysis 6.1 human evaluation we employed 6 annotators with a background in linguistics and nlp and let them interact with the two strongest models in terms of automatic metrics: chatgpt and tk-instruct. the annotators were given randomly selected goals from the mul-tiwoz 2.2 dataset and a minimal set of essential instructions on how to proceed. we present the results in ## conclusion & future work we present an experimental evaluation of instruction-tuned llms applied to the established task of task-oriented dialogue modeling, with five llms evaluated on two datasets. we find that llms are not performing well in terms of belief state tracking, even when provided with in-context fewshot examples. however, there is some potential to improve through prompt tuning and output parsing robust to irregularities. if provided with a correct belief state, the models can interact with the user successfully, provide useful information and fulfill the user's needs. while the performance does not match the supervised state of the art, it is important to note that these models were not finetuned on in-domain data and work with just a domain description or a few examples (which again improve performance). therefore, carefully picking representative ex-amples and combining the llm with an in-domain belief tracker can be a viable choice for a taskoriented dialogue pipeline. interestingly, in the human interactive evaluation, both chatgpt and tk-instruct outperformed the expectations set by automatic metrics. this shows certain flexibility and ability to correct their own mistakes on the part of llms, and further demonstrates that single-turn evaluation is too rigid and does not show the whole picture @xcite . in future work, we want to focus on addressing the prompt-recoverable errors while maintaining the ability to use model-independent prompts and easily swap models. we also aim to find a more effective method of relevant example selection.