D i STRICT : Dialogue State Tracking with Retriever Driven In-Context Tuning (2023)

Abstract: Dialogue State Tracking (DST), a key component of task-oriented conversation systems, represents user intentions by determining the values of pre-defined slots in an ongoing dialogue. Existing approaches use hand-crafted templates and additional slot information to fine-tune and prompt large pre-trained language models and elicit slot values from the dialogue context. Significant manual effort and domain knowledge is required to design effective prompts, limiting the generalizability of these approaches to new domains and tasks. In this work, we propose DiSTRICT, a generalizable in-context tuning approach for DST that retrieves highly relevant training examples for a given dialogue to fine-tune the model without any hand-crafted templates. Experiments with the MultiWOZ benchmark datasets show that DiSTRICT outperforms existing approaches in various zero-shot and few-shot settings using a much smaller model, thereby providing an important advantage for real-world deployments that often have limited resource availability.

## introduction task-oriented dialogue systems are increasingly used to enable users to perform tasks through multiturn conversations in various domains such as travel reservations, banking transactions, or online shopping. dialogue state tracking (dst) is a critical component of these systems that tracks user requirements by determining key information at each turn in the dialogue @xcite . given a predefined schema of task parameters (i.e., slots), dst models identify and represent the dialogue state as pairs of slots and their corresponding values as shown in figure 1 . in real-world deployments, new task domains and slots are frequently added to improve user functionality. hence, periodic updates to the models may be required, even when the new domains offer little to no dialogue data for training. to address • restaurant-day: july 4 figure 1 : an example of multi-domain dialogue state tracking (dst) in a conversation to book a hotel stay and reserve a table at a restaurant. recent advances in dst have leveraged pretrained language models (lms) to elicit slot values from dialogues using primarily two methodsfine-tuning and in-context learning. however, both methods suffer from several shortcomings -fine-tuning lms -most existing approaches condition lms for dst by fine-tuning their parameters using prompts derived from historical dialogue data. however, they typically rely on handcrafted prompt templates that include slot-specific questions, value based functions, or even text-tocode snippets @xcite . in addition to the manual effort required, these templates are customized to specific domains and slots, and hence have low generalizability to new domains. some approaches also extend finetuning to first train models on additional natural language tasks using external datasets which is expensive and requires access to significantly more data. they also often rely on additional information from the schema such as slot descriptions and task instructions @xcite . however, real-world datasets may not always have this necessary information, which again limits their generalizability. in-context learning -lms have shown remarkable performance through in-context learning of new tasks @xcite , where a raw lm (i.e. pre-trained lm without fine-tuning on taskspecific data) is prompted during inference using input-output task examples to condition the generated output. for dst, approaches have leveraged this to craft prompts containing examples of dialogue history and slot values @xcite . however, similar to fine-tuning approaches, these prompt examples are hand-crafted, customized, and require significant effort. additionally, as a result of using raw lms, these approaches have to rely on extremely large models limiting their practical use. importantly, it has been shown that prompting raw lms without fine-tuning is often oversensitive to example choices and instruction wording @xcite , and can demonstrate undesirable biases that significantly reduce performance @xcite . in this work, we address these challenges and present district, a generalizable approach for dialogue state tracking that fine-tunes a lm using relevant examples (i.e., in-context tuning). for a given input dialogue and slot to be tracked, district retrieves semantically similar dialogues and slots from available historical data in zero-shot and fewshot settings, and concatenates them into a prompt with no hand-crafted template requirements. we first fine-tune the language model using in-context examples and input dialogues from the training set, and subsequently perform similar inference on test inputs as shown in figure 2 . specifically, we make the following contributions -• to the best of our knowledge, district is the first dst approach to use in-context tuning by fine-tuning a lm with in-context examples. ## related work dialogue state tracking (dst) is a critical yet challenging task for task-oriented dialogue systems @xcite , and several multi-domain benchmark conversation datasets have been proposed @xcite @xcite to evaluate research efforts. a majority of state-of-the-art approaches finetune language models using hand-crafted templates containing descriptions or questions related to the dialogue slots. @xcite used slot descriptions and examples of slot values to create templates while @xcite and @xcite provided different types of manually annotated slot descriptions to the model. @xcite extended this by also including task instructions and other constraints. in contrast, our approach does not require hand-crafted templates for fine-tuning and is hence more easily generalizable. another set of approaches aim to improve zeroshot performance by exploiting external knowledge and datasets from other natural language tasks before fine-tuning a model for dst. for instance, @xcite ; @xcite ; @xcite pre-train models on reading comprehension data, @xcite reformulate dst as a dialogue summarization task with external annotated data, and @xcite use semantic analysis and named entity recognition to identify slots. in contrast, our approach does not require any extra datasets or training efforts . in-context learning (icl) for dst has been explored as part of a larger set of few-shot generative tasks @xcite , but the lack of a task-specific prompt design resulted in low performance. @xcite and @xcite achieved improved performance using extremely large models, where the former reformulated dst as a text-to-sql task by using semantic matching to identify relevant examples that were subsequently crafted into sql queries, and the latter manually created example dialogues containing combinations of slots in the schema. recent efforts have shown that the shortcomings of icl @xcite can be overcome through in-context tuning of lms. gu-user: i need a reservation for an indian restaurant for 2 people at 12:45 in-context examples ( single-turn dialogues from training set) system: i have successfully made a reservation at the royal spice test input in-context retriever (few/fullshot) in-context retriever (zero-shot) query slot: restaurant-people : example #1: user: book me a restaurant table for 8 on monday at 19:30 system: the booking has been completed slot: restaurant-people: 8 example #2: …. dialog context, query slot example #1: user: i need a hotel in nyc for 4 people for 2 nights from thursday system: this is your confirmation number slot: hotel-people: 4 example #2: …. dialog context, query slot fine-tuned language model fine-tuned language model restaurant-people: 2 predicted dialog state prompt prompt access to example dialogs from same domain search for semantically similar dialogs and slots from other domains dialog context restaurant-people: 2 figure 2: overview of district. given an input dialogue and slot to be tracked, high-relevance examples are first identified by the retriever and used to create the prompt. in (1) zero-shot settings, the retriever must search for semantically similar dialogue contexts and slots from other domains, while in (2) few/full-shot settings, the retriever additionally has access to some example dialogues from the same domain that could also contain the query slot. rurangan et al. (2020) and liu et al. (2022) demonstrate the improved performance of models finetuned with examples compared to icl over a variety of language tasks. in this work, we leverage this concept specifically for dst. ## approach we first present the background and some definitions for dialogue state tracking before describing our approach. ## conclusion we present district, a novel approach for dialogue state tracking using in-context tuning of language models (lms). for an input dialogue instance and slots, district retrieves the most relevant examples from the training data through semantic matching, and uses these examples as part of the input to the lm to obtain the dialogue state. the fully automated prompt construction, with-out requiring hand-crafted templates or additional schema information, overcomes drawbacks of prior dst approaches and also reflects the high generalizability of district to new task domains and slots. our experiments show that district outperforms existing baselines in different zero-shot and few-shot experiments despite using a smaller and lower-resource model. we also demonstrate the effectiveness of our semantic-search based retriever for the dst task and highlight several tradeoffs between model performance and resource requirements that impact real-world use. as part of future work, we intend to improve robustness to dialogue quality and distribution-drifts.