Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable performance in zero-shot dialogue state tracking (DST), reducing the need for task-specific training. However, conventional DST benchmarks primarily focus on structured user-agent conversations, failing to capture the complexities of real-world multi-user interactions. In this study, we assess the robustness of LLMs in multi-user DST while minimizing dataset construction costs. Inspired by recent advances in LLM-based data annotation, we extend an existing DST dataset by generating utterances of a second user based on speech act theory. Our methodology systematically incorporates a second user’s utterances into conversations, enabling a controlled evaluation of LLMs in multi-user settings. Experimental results reveal a significant performance drop compared to single-user DST, highlighting the limitations of current LLMs in extracting and tracking dialogue states amidst multiple speakers. Our findings emphasize the need for future research to enhance LLMs for multi-user DST scenarios, paving the way for more realistic and robust DST models.

## introduction large language models (llms) have achieved remarkable success across various natural language processing tasks, ranging from translation to mathematical problem-solving @xcite @xcite @xcite . their effectiveness has also been demonstrated in dialogue state tracking (dst), a task that predicts a user's goal based on conversations between the user and an agent @xcite . traditional approaches to dst typically rely on training sequence-tosequence models @xcite @xcite . however, recent * equal contribution as co-first author. studies suggest that llms can achieve competitive performance in dst through zero-shot inference, eliminating the need for task-specific training @xcite . despite these advances, conventional dst benchmarks primarily feature structured user-agent dialogues, limiting their applicability to more complex real-world scenarios. for instance, multiwoz @xcite , one of the most widely used dst datasets, contains only single-user conversations. in contrast, real-world interactions often involve multiple users collaboratively engaging in decision-making before interacting with an agent. for example, a group of friends discussing restaurant options might first debate their cuisine preferences and locations before making a request to a booking system (e.g., "i prefer korean food." → "is there a good korean restaurant in suzhou?"). while previous studies introduced the concept of multi-user dst, they approached it by aggregating users' utterances and performing dst on the aggregated input as if it were a single-user setup @xcite . this highlights the need to assess dst models in genuinely multi-user scenarios to better evaluate their robustness and generalizability. in this work, we conduct a case study to assess the performance of llms in multi-user dst, with minimal dataset construction costs. inspired by recent efforts to employ llms for data annotation, we extend existing dst datasets using state-ofthe-art llms @xcite . specifically, we generate additional user utterances that interleave between the original user-agent exchanges. our approach comprises two main steps: (1) identifying the appropriate type of interjections based on speech act theory @xcite , and (2) prompting an llm to generate and validate the second user's utterances accordingly. figure 1 presents an example of a dialogue with utterances of the second user, based on our suggested framework. we compare the performance of llms in cases where the second user utterance is present or absent throughout the experiment. experimental results on our extended dataset show a substantial performance drop compared to single-user dst. this indicates that current llms struggle to correctly extract and track dialogue states in conversations with multiple users, underscoring limitations in their generalizability. our findings highlight the need for future research to develop dst methods and prompt-engineering strategies optimized for multi-user settings. our primary contributions are as follows: ## conclusion we presented a systematic investigation of multiuser dst, a task that has been largely overlooked in traditional dst research. by extending an ex-isting dataset through llm-based generation of additional user 2 utterances, we revealed that contemporary llms face notable performance drops in multi-user dst scenarios, even when the underlying user request remains unchanged. our findings underscore the importance of focusing on multi-user dialogue and highlight the challenges in maintaining accurate slot tracking when multiple speakers contribute to the conversation. future work could explore more advanced prompt-engineering strategies or model designs that disentangle each speaker's goal. in real-world applications, multiple people often share or refine a collective plan, such as making group travel arrangements or ordering meals. improving llmbased dst for these more realistic, collaborative settings will be crucial for developing robust, universally applicable dialogue systems.