C hat GPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity? (2023)

Abstract: Recent research on dialog state tracking (DST) focuses on methods that allow few- and zero-shot transfer to new domains or schemas. However, performance gains heavily depend on aggressive data augmentation and fine-tuning of ever larger language model based architectures. In contrast, general purpose language models, trained on large amounts of diverse data, hold the promise of solving any kind of task without task-specific training. We present preliminary experimental results on the ChatGPT research preview, showing that ChatGPT achieves state-of-the-art performance in zero-shot DST. Despite our findings, we argue that properties inherent to general purpose models limit their ability to replace specialized systems. We further theorize that the in-context learning capabilities of such models will likely become powerful tools to support the development of dedicated dialog state trackers and enable dynamic methods.

## introduction dialogue state tracking (dst) is a critical component for task-oriented dialogue systems. its purpose is to extract and track user's goals throughout a conversation @xcite . dst is challenging due to the infinite possibilities of user/agent conversations, and because services and schemas/apis that dialogue systems interface are subject to constant change @xcite . although traditional approaches achieve high accuracy when operating on a pre-defined set of concepts called an ontology @xcite @xcite , ongoing research explores transfer to new domains with little to no additional learning @xcite using ontology independent architectures to allow seamless adaptation to out-of-ontology concepts. many strategies for zero-shot transfer to unseen domains have been proposed. @xcite treat dst as a question answering (qa) task by leveraging data augmentation. @xcite propose dst by relying on schema descriptions while @xcite utilize natural language descriptions to facilitate zero-shot transfer. @xcite and @xcite suggest learning from non-dialogue qa data which are available in large amounts to improve generalization. @xcite harness large synthesized data based on abstract dialogue models. however, none of these techniques are ideal solutions. fine-tuning is challenging due to computational costs, risk of over-fitting and the need for expensive @xcite task-specific data. cross-task transfer still requires curated data and careful consideration of suitable learning tasks. data augmentation requires high level task knowledge and an adequate synthesizing strategy. a new generation of large language models (llms) @xcite @xcite comes with the promise to be equipped to solve any task without task-specific fine-tuning, but solely with world knowledge they acquired during self-training on massive amounts of data. such llms have been shown to perform remarkably well on in-context learning (icl), where only a natural language prompt and examples are provided to condition the generation process, achieving significant improvements over fine-tuned approaches in few-shot setups @xcite . chatgpt (ope-nai, 2022) -trained using human feedback and reinforcement learning -is the most recent of such models and single-handedly solves an array of challenging natural language processing (nlp) tasks with super-human capabilities, all through a natural language dialogue interface. in this work, we aim to answer the question: does chatgpt solve the problem of zero-shot dst? we show that crafting intuitive natural language prompts is sufficient to achieve state-of-the-art performance with chatgpt, exceeding conventional, engineering-heavy approaches to zero-shot dst by a large margin. however, despite our findings, we argue that properties inherent to general purpose models inhibit their ability to simply replace specialized systems. we speculate that while in the foreseeable future general purpose models may not become holistic solutions to complex problems, they will provide ample opportunities to empower specialized systems to go beyond their pre-defined scopes, enable on-the-fly extensibility and generation of high quality training data by zero-shot synthesizing or automatic labeling. ## zero-shot dst with chatgpt our investigative approach to zero-shot dst with chatgpt differs considerably from related works. we decode dialogue state updates with a general purpose model, without undergoing any parameter updates. consequently, we neither employ data augmentation nor cross-task transfer learning. instead, we solely rely on the general capacities of chatgpt as an aligned dialogue agent. we take a most rigorous approach to zero-shot transfer where we do not allow the provision of any examples, nor of a formal task definition. instead, we only permit natural language explanations of what the model is supposed to do. this sets our investigation apart from the closely related ic-dst @xcite . in zero-shot dst, the set of slots s relevant during inference and the set of slots s ′ seen during training of the model x θ with parameters θ are disjoint, i.e., s ∩ s ′ = ∅. further, it may be s ′ = ∅, in which case θ is not specifically tuned towards solving dst. this is precisely the case for chat-gpt in our setup. our approach to zero-shot dst with chatgpt is formalized as follows. let a 1 =p ⊕ "system":m 1 ⊕ "user":u 1 , a @xmath0:m t ⊕ "user":u t , ∀@xmath1, t ], where p is the task description which provides the model with instructions for how to process a dialogue between a system m and a user u . a 1 is the initial prompt to chatgpt. a @xmath2, only containing a single turn-pair of the dialogue of interest. chatgpt is particularly suitable for this strategy due to its chat based interface. chatgpt generates its next output b t conditioned on the current prompt a t-1 , as well as all preceding user queries and system responses of the same chat. the dialogue state update ds t can be found in b t , but may not be directly interpretable as such due to the diversity in the output surface forms. thus, we require a normalization operation ds @xmath3. in contrast to @xcite , we do not condition b t on ds t . this renders the task even more challenging, as chat-gpt is forced to solve complex subtasks such as coreference resolution -the case where a newly encountered slot refers to the value of another slot -solely given the initial prompt and its own latent dialogue state given the dialogue history. ## experiments at the time of conducting our experiments, chat-gpt is a proprietary research preview accessible for free via a web interface foot_0 . we used the jan 9 version of the model. we use a regular expression term to extract all parts that are json formatted. we form ds t by accumulating all predicted updates up to turn t. evaluation. we evaluate on the 1000 dialogues of the multiwoz 2.1 @xcite et al. ( 2020 ). we process each dialogue once and refrain from using chatgpt's regeneration feature. prompt. we imposed restrictions that the taskdefining prompt p be intuitive natural language and provides no formal schema. the crafting process involves simple trial-and-error on fewer than 10 held-out dialogues from the multiwoz training set. the design process was guided by the intention to imitate the behavior of a triple copy strategy (trippy) dst @xcite . p consists of three parts. first, a list of names for detectable informable slots along with natural language descriptions. the slot names help us extract a ds t that is compatible with the dataset's labels. second, a sparse list of slots that are categorical, along with their value candidates for (1) aiding normalization of values that are expected to show high variability in expression, and (2) modeling boolean slots. third, an informal task description. foot_2 4.1 chatgpt vs. supervised sota comparing chatgpt's performance to state-of-theart supervised approaches that achieve close to 60% jga is not a fair fight foot_3 , and yet we observe an impressive 31.5% zero-shot jga. this result is double-edged; on the one hand it is evidence that chatgpt is capable of dst foot_4 , and on the other hand is no match for specialized systems. the comparison to trippy, a sota supervised model, allows us a more fine-grained analysis. in figure 1 , slot filling performance is broken down into value types. we observed that chatgpt underperforms in non-trivial cases, namely refer, where a newly encountered slot refers to the value of another slot, and inform, where a slot-value was mentioned by the system and confirmed by the user. chatgpt shows slight underperformance for boolean slots. remarkably, performance for values that are extracted directly from user utterances -the most relevant category in terms of frequency - is exceeding the strong supervised baseline. lastly, chatgpt has a clear advantage in the underrepresented and therefore notoriously difficult dontcare cases, where a user is indifferent about a particular value for a slot. ## conclusion this work is the first to investigate chatgpt's capacities for zero-shot dst. despite remarkable preliminary results that we achieved, we identified limitations rooted in inherent properties of general purpose models, preventing these to become holistic solutions to complex nlp problems without further research. we discussed opportunities provided by chatgpt and similar models to advance the development of specialized systems. with our insights and discussion, we hope to stimulate research in similar directions.