A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding (2025)

Abstract: Dialogue State Tracking (DST) is crucial for understanding user needs and executing appropriate system actions in task-oriented dialogues. Majority of existing DST methods are designed to work within predefined ontologies and assume the availability of gold domain labels, struggling with adapting to new slots values. While Large Language Models (LLMs)-based systems show promising zero-shot DST performance, they either require extensive computational resources or they underperform existing fully-trained systems, limiting their practicality. To address these limitations, we propose a zero-shot, open-vocabulary system that integrates domain classification and DST in a single pipeline. Our approach includes reformulating DST as a question-answering task for less capable models and employing self-refining prompts for more adaptable ones. Our system does not rely on fixed slot values defined in the ontology allowing the system to adapt dynamically. We compare our approach with existing SOTA, and show that it provides up to 20% better Joint Goal Accuracy (JGA) over previous methods on datasets like MultiWOZ 2.1, with up to 90% fewer requests to the LLM API.

## introduction dialogue state tracking (dst) is a critical component of task-oriented dialogue systems, designed to extract and maintain users' goals throughout a conversation @xcite . the challenge of dst lies in the infinite possibilities of user/agent conversations and the constant evolution of services, schemes, and apis that dialogue systems interface with @xcite . while traditional approaches demonstrate reasonable performance within predefined ontologies @xcite , current research is exploring various strategies for domain transfer. these strategies include adaptation to unseen domains @xcite , leveraging non-dialogue qa data to enhance generalization @xcite , and framing dst as a question-answering problem using natural language descriptions to enable zero-shot transfer @xcite . however, these approaches still require training on seen domains and closely adhere to domain ontologies. a new generation of large language models (llms) such as @xcite , llama 2 @xcite and gemini 1.0 @xcite promise the ability to solve tasks without task-specific fine-tuning, relying instead on the extensive world knowledge acquired from training on vast amount of data. these llms have shown remarkable capabilities in in-context learning (icl), where the model generates responses based on a natural language prompt and a few examples, achieving significant advancements over fine-tuned methods in few-shot scenarios. researchers have begun to apply llms with icl techniques to address the dst challenge @xcite @xcite ), yet they have not surpassed state-of-the-art (sota) supervised methods, or lacked practicality in terms of number of queries to be executed for every single turn, and are highly dependent on the fixed ontology. furthermore, the majority of these works @xcite use gold domain labels, usually skipping the domain classification phase, which is nontrivial. to address these challenges, we introduce a zero-shot, resource-efficient, and open-vocabulary pipeline system for task-oriented dialogue understanding. our pipeline starts with domain classification, a crucial phase often overlooked in existing approaches, followed by two complementary approaches for dst. first, we propose dstas-qa, which transforms dst into a multiple-choice qa problem @xcite , providing a strong adaptation for smaller or less capable llms. second, inspired by the recent success of self-refining/correcting prompts @xcite , we propose dst-as-srp-for the first time in literature-that considers llm as a black-box dialogue state tracker, and cast the problem into a well-structured prompt, i.e., self-refined prompt (srp). unlike ontologybased approaches that need to process all possible slot value pairs @xcite within the ontology, open-vocabulary approaches only use the generic slot definition (which sometimes is common knowledge, e.g., hotel name, restaurant area) and generate/extract the values directly from the dialogue. we then perform a series of experiments with the most recent open-source (e.g., llama3) and proprietary language models (e.g., gpt-4-turbo) on common dst datasets such as multiwoz and sgd (see §5). we further measure the effect of having access to gold domains (i.e., with/without domain classification), and ontology; and compare our approaches with various sota methods (both fully trained and zero-shot) on fair settings (e.g., by using the same llm). we show that dst-as-srp achieves new state-of-the-art results with up to 90% fewer requests to the llm api(compared to a previous sota that prompts llms for each slot @xcite ), improving the strict joint goal accuracy (jga) score by 20%, 3%, and 2% on the multiwoz 2.1, multiwoz 2.4, and sgd datasets, respectively, while still being constrained to open-vocabulary and zero-shot settings unlike current sota methods. 2 finally, we show that the improvements are not simply due to using a larger language model but the dst-as-srp technique in a controlled setup. ## methodology an overview of our methodology is given in fig. 1 . our pipeline begins by identifying the active domain for each turn. for domain classification, the active turn and all preceding turns in the dialogue history are passed to the model, which determines the domain using a specific prompt tailored to the language model (see § 3.2). next, we extract the values of the selected domain slots using two approaches: question answering (see § 3.3) and selfrefined prompting (see § 3.4). ## conclusion dst approach # llm api requests all slots a 449.7 turn domains' slots b 120.2 all slots in one prompt c 7.4 dst-as-qa 16.4 dst-as-srp 13.1 a track all slots from all domains each in a single llm request b track slots from active domains and each in a single llm request c track slots from all domains but in a single llm request