In-Context Learning for Few-Shot Dialogue State Tracking (2022)

Abstract: Collecting and annotating task-oriented dialogues is time-consuming and costly. Thus, zero and few shot learning for dialogue tasks presents an exciting opportunity. In this work, we propose an in-context (IC) learning framework for zero-shot and few-shot learning dialogue state tracking (DST), where a large pretrained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates. This approach is more flexible and scalable than prior DST work when adapting to new domains and scenarios. To better leverage a tabular domain description in the LM prompt, we reformulate DST into a text-to-SQL problem. We also propose a novel approach to retrieve annotated dialogues as exemplars. Empirical results on MultiWOZ show that our method IC-DST substantially outperforms previous fine-tuned state-of-the-art models in few-shot settings. In addition, we test IC-DST in zero-shot settings, in which the model only takes a fixed task instruction as input, finding that it outperforms previous zero-shot methods by a large margin.

## introduction dialogue state tracking (dst) is an important module in many task-oriented dialogue systems. the goal of this module is to extract users' intentions at each turn in the dialogue as represented in the slot values of a predefined schema. collecting and annotating turn-level dialogue states is notoriously hard and expensive @xcite . also, in commercial applications, it is common to extend the schema and incorporate new domains. thus, it is important to develop dst learning strategies that are flexible and scalable, in addition to requiring less data. previous studies have explored zero/few-shot dst, but with some limitations. most few-shot methods are based on finetuning pretrained language models (wu et al., 2020; li et al., 2021; su 1 our code : https://github.com/yushi-hu/ic-dst turn t-1 dialogue state (predicted) restaurant-area: center, restaurant-food: catalan turn t system: i am sorry, there are no catalan dining in the city center. would you like a different cuisine or area? user: actually, can you try anything that serves french food? current test turn dialogue retriever labeled few-shot examples system: there are no restaurants matching your criteria. would you like a different style of food? user: how about brazilian food? state changes (sql format): select * from restaurant where @xmath0: west, restaurant-food: irish turn k retrieved labeled dialogue turns inference lm predicted state changes (sql format): select * from restaurant where @xmath1: state changes: restaurant-food: french dialogue state: restaurant-area: center, restaurant-food: french (1) (2) (4) (3) figure 1 : illustration of dst task and ic-dst approach. the task is to track the slot values associated with a user request up to the current turn (dialogue state). in fewshot settings, given a test turn (1), ic-dst first retrieves a few most similar turns from the labeled dialogues as examples (2). the task schema (not shown in the figure ), examples, and the test dialogue turn are concatenated in the prompt to a lm (e.g. gpt3) (3) to produce the current turn dialogue state changes as a sql query (4). @xcite @xcite . such systems are less flexible, since they need to be retrained when new slots or domains are added, and finetuning large lms is computationally expensive. most zero-shot methods have involved domain-transfer approaches @xcite , which have not yielded good performance. to address the above challenges, we propose the ic-dst model to solve the dst problem with the in-context learning (icl) paradigm @xcite , in which a large language model makes predictions based on the task instruction and/or examples in the prompt. in few-shot settings, the prompt contains exemplars that are retrieved from a small set of labeled training data. a motivation behind this framework is that it requires no finetuning (i.e., no parameter updates), which makes systems flexible in that they can handle queries in a new domain via the exemplar retrieval process without re-training. this enables developers to quickly prototype systems in new domains and rapidly leverage new collected data. icl has been used successfully in semantic parsing @xcite @xcite , especially in few-shot scenarios. however, these studies focus on sentence-level tasks. icl has been explored for dst @xcite , but the performance fell short of pretraining and domain-transfer approaches to few/zero-shot learning. dst involves long, two-party dialogue histories with grounding in a structured ontology. we believe these challenges cause the poor icl performance on dst tasks in previous work. to address these challenges, we explore incontext learning with three novel contributions. first, we reformulate dst as a text-to-sql task, including a tabular description of the ontology in the prompt. this is a better match to the knowledgegrounded scenario, and it takes advantage of large language models pretrained with code: codex @xcite , gpt-neo @xcite , and codegen @xcite . second, we use the dialogue state in representing context, rather than the full conversation history, which is more efficient and better suited to domain changes. lastly, in the few-shot scenario, we propose a new approach to learning a similarity score for selecting in-context examples that is trained to match similarity based on dialogue state changes. the ic-dst approach, which incorporates these advances, achieves a new state of the art on multiwoz fewshot settings, i.e. when using 1-10% training data. we also substantially improve the zero-shot state of the art by 10-30% absolute accuracy on each domain. a further contribution is an extensive analysis demonstrating impact from each innovation. in summary, our work makes the following contributions: ## analysis to better understand the effectiveness of our proposed methods, we provide detailed analysis in this section. all ablation experiments are conducted on 100 random multiwoz 2.4 development set dialogues in the 5% few-shot setting. representation of dialogue context dialogue states vs. state changes we also explore the benefit of using state change vs. full dialogue state labels (c t vs. y t ). note that example labels also act as the basis for the similarity objective in training the retriever. a likely reason is that state changes contain fewer slots compared with dialogue states, making for an easier prediction task for the lm. also, there are many more example turns with the same state changes, compared with examples with the same dialogue states, making it easier for the retriever to fetch good examples in few-shot scenario. we further compare these two kinds of example labels by investigating the jga on each turn, illustrated in figure 5 . because states have increasing numbers of slots with more turns, jga (red and yellow lines) of the full state decreases for later turns. however, all turns have a relatively small number of state changes, so the state change jga (blue line) remains high throughout the dialogue. as a result, the jga of the full state benefits from using predicted state change updates (red line, performance dependence on the retriever effect of dst as text-to-sql ## conclusion we successfully apply in-context learning for dialogue state tracking by introducing a new approach to representing dialogue context, a novel objective for retriever training, and by reformulating dst into a text-to-sql task. on multiwoz, our system achieves a new state of the art in both few-shot and zero-shot settings. our analyses show that each innovation benefits performance. we also study in detail the contribution of each design decision. future work may apply this in-context learning framework to a wider range of dialogue tasks.