S3- DST : Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLM s (2024)

Abstract: Traditional Dialogue State Tracking (DST) has focused on tracking preferences and intents in conversations centered around specific tasks (e.g. booking services). These conventional systems assume a relatively restricted conversation flow in which each turn gradually offers new information. However, advancements in Large Language Models (LLMs) have ushered in more versatile open-domain chat systems in which extended dialogue sessions encompassing numerous tasks and topics are common—in turn requiring new conversational tracking tools in order to successfully orchestrate such systems. Addressing these challenges, we introduce a novel approach combining dialogue segmentation and state tracking within open-domain dialogues, tailored for zero-shot applications appropriate to a true open-domain dialogue system. Our proposed method S3-DST employs a unique structured prompting technique and *Pre-Analytical Recollection*, a novel grounding mechanism we designed for improving long context tracking. Tested on proprietary anonymized open-domain dialogue datasets as well as publicly available DST and segmentation datasets, S3-DST consistently outperforms the state-of-the-art, showcasing its effectiveness and adaptability state tracking in the next wave of LLM-based chat systems. We also release S3-DST annotations with GPT-4 on a curated subset of LMSYS-Chat-1M to be used as a testbed to fuel research in this direction.

## introduction the advent of open-domain large language model (llm)-based chat systems like chatgpt and bing chat has ushered in a new age of dialogue systems. previously, dialogue systems were relatively constrained in their scope and abilities, typically confined to either narrow task-oriented conversations or social chitchat @xcite . by contrast, llm-based chat systems are remarkable because they can converse fluidly with users over a seemingly infinite range of topics, and can accomplish many user tasks out-of-the-box that previously required specialized systems, like code generation, question answering, and more. while llm-based chat systems have significantly changed the landscape of human-ai dialogue, the enhanced fluency and unrestricted nature also significantly increase the challenges associated with the analysis and tagging of dialogues, due to their closer resemblance to fluid real-world conversation. conventional chat agents designed for specific tasks interact with users in a predictable sequence, extracting required necessary information (state values) turn-by-turn to properly serve user requests @xcite . however, the traditional method of turn-by-turn dialogue state tracking (dst), although effective for previous domain-specific tra-ditional chatbots, does not adequately address the complexities of real-world, open-domain dialogues. these conversations often involve a considerable amount of back-and-forth interaction between the user and agent, and the context may shift several times within a single dialogue, moving between unrelated topics and/or intents. this dynamic is illustrated in figure 1 , where open-domain dialogues facilitated by llm-based chat systems enable users to pursue various intents within a single conversation, thereby creating distinct topical and/or goal-oriented segments. as depicted, certain segments necessitate multiple exchanges between the user and the agent to accurately determine the user's intent, which then informs the orchestration layer and the selection of relevant apis (e.g. web search, image or code generation, etc.) to satisfy the user's request. a turn-by-turn tracking scheme is thus insufficient to accurately capture dialogue state changes across multi-turn segments. in order to tackle this, we propose to track both segments and states in open-domain dialogue: segmentation helps us identify boundaries that mark the start and end of contextually cohesive conversation "units," whereas states are the intent variables of interest we wish to track, applied per segment. beyond bringing dst into the era of opendomain conversation and llms by joint tracking of segments and states, we introduce llm-based solutions for open-domain dst. assuming a zeroshot setting for dialogue tagging, which is realistic due to the cost of labeling, we introduce s3-dst, a structured prompting approach for opendomain dst. within s3-dst we propose a novel pre-analytical recollection (par) prompting strategy that grounds each output state prediction on the content of the corresponding dialogue turn, thereby helping the llm track long dialogue context without forgetting or hallucination. we evaluate s3-dst on a fully anonymized open-domain dialogue dataset collected from microsoft's bing chat system, alongside public dst and segmentation benchmarks. foot_0 s3-dst outperforms comparable baselines significantly across all benchmarks, indicating its potential as an initial framework for advancing research in open-domain dialogue modeling. moreover, we release s3-dst annotations with gpt-4 on a curated subset of lmsys-chat-1m @xcite , which we believe will be an excellent starting point for understanding and benchmarking open-domain dst. ## problem definition informally, the goal of traditional dst is to predict the dialogue state y t given a sequence of user and agent utterance turns c @xmath0, a 1 , . . . , u t , a t ]. 2 the state y t consists of a set of slot-value pairs, where slots correspond to intent attributes in a particular application domain (e.g., "restaurantname", "hotel-address") and values correspond to predefined categorical options or unconstrained text @xcite . however, as we have previously discussed, a single open-domain conversation will often consist of multiple potentially unrelated intents across a variety of topics. indeed, according to a preliminary analysis on 10k anonymized bing chat conversations, we estimate that over 50% of conversations display multiple user intents and over 90% of conversations contain discussion of multiple topics. therefore, we propose to merge dialogue segmentation, which aims to find contextually cohesive "units" of dialogue within a larger conversation, with dialogue state tracking. in particular, we perform state tracking at the segment level, where the goal is to label each segment with the slots and values of interest, such that multiple segments within a conversation may have diverging or conflicting state values, reflecting the true variety of open-domain chat. in the rest of this section, we define segmentation and state, and finally formalize the joint task. ## prompting strategies as discussed previously, real-world dialogues often span multiple topics and intents. previous studies @xcite aimed at disassociating individual dialogue turns and processing them one by one for tracking dialogue state changes, which worked reasonably well in task-oriented dialogues within predefined narrow domains. however, real-world dialogues commonly require multiple turns to adequately comprehend contextual nuances, which is a challenge because transformers still struggle when processing lengthy input contexts, particularly in the middle @xcite . to address these difficulties, we propose a novel turn-by-turn prompting technique that gives structure to inputs and outputs while accurately preserving the context in the process. we discuss our approach below and summarize it in figure 2 : or open text). to provide a flexible yet structured format to the llm's output, we propose to instruct it to generate outputs in a hierarchical xml format. we see xml as advantageous because it provides code-like structure to the dst task, which has been shown to greatly improve performance compared to plain-text outputs, while still being extensible and flexible compared to more rigid output formats like sql @xcite . our approach uses an xml format in which each turn from 1 to t comprises an xml @xmath1...</t{id}> and several nested xml tags within it. the labels of these nested tags (e.g. <segment_start>...</segment_-start>, <intent>...</intent>, @xmath2...</@xmath3, and each value between opening and closing tags represent the model's inferred value. this strategy is beneficial for two reasons: (i) due to a bounded and well-defined structured formatting, generated outputs are more likely to be aligned with labeling instructions than free-form texts; and (ii) well-formed structured output formats are easier to parse, thus reducing postprocessing requirements. ## experiments we conduct comprehensive evaluations across multiple datasets. we primarily evaluate our approach on fully anonymized bing chat logs annotated by domain experts. additionally, we evaluate s3-dst on the standard task-oriented dst and segmentation tasks using public benchmark datasets multi-woz @xcite and dialseg711 @xcite respectively. finally, we also construct a curated set of conversations from the recently open-sourced lmsys-chat-1m @xcite dataset, which is a similar open-domain human-llm dialogue dataset. we apply s3-dst with gpt-4 and open-source the results, which we believe can be highly useful as a starting point for open-domain dialogue analysis and benchmarking. we discuss the lmsys-chat-split dataset in more detail in appendix d. a detailed description of the other datasets is provided below, alongside the dataset statistics in ## discussion and conclusion llm-based chat systems have broadened the horizons of human-ai conversation, warranting new methods for tracking user intentions. therefore, we bring dialogue state tracking in the realm of opendomain dialogue systems by jointly tracking topically coherent segments and state intent variables per segment. since this requires the assumption of a zero-shot setting due to the impracticality of annotation across all disciplines, we propose s3-dst, which structures the prompt in an xml format and leverages our proposed grounding mechanism (par) for long context tracking. across extensive experiments on proprietary and public datasets, s3-dst shows large performance gains over state-ofthe-art zero-shot techniques in dialogue state tracking and segmentation approaches. in the future, as llm-based chat systems become more prevalent, we expect dialogue systems research to shift further toward understanding and modeling opendomain dialogue. in this respect, we aim to further study and develop techniques for extended context preservation, to improve grounding in dst alongside other important dialogue modeling tasks.