Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants (2024)

Abstract: Conversational systems must be robust to user interactions that naturally exhibit diverse conversational traits. Capturing and simulating these diverse traits coherently and efficiently presents a complex challenge. This paper introduces Multi-Trait Adaptive Decoding (mTAD), a method that generates diverse user profiles at decoding-time by sampling from various trait-specific Language Models (LMs). mTAD provides an adaptive and scalable approach to user simulation, enabling the creation of multiple user profiles without the need for additional fine-tuning. By analyzing real-world dialogues from the Conversational Task Assistant (CTA) domain, we identify key conversational traits and developed a framework to generate profile-aware dialogues that enhance conversational diversity. Experimental results validate the effectiveness of our approach in modeling single-traits using specialized LMs, which can capture less common patterns, even in out-of-domain tasks. Furthermore, the results demonstrate that mTAD is a robust and flexible framework for combining diverse user simulators.

## introduction thoroughly testing a conversational system with real users is a costly and time consuming process. part of this cost lies in the myriad of conversational traits, reflecting user's behaviors, knowledge, and goals, which when diverse have shown to help improve performance of conversational systems @xcite . user simulators have been a successful approach to model real user conversational traits and discover errors and limitations in conversational systems @xcite @xcite . prior research investigated user simulators across diverse settings, including task-oriented dialogues @xcite @xcite and recommendation @xcite @xcite simulating a user conversational profile entails a combination of diverse conversational traits, as shown in to identify these traits, we analyzed real-world dialogues in the novel conversational task assistant (cta) domain @xcite and extracted the most relevant traits. in cta scenarios, users actively engage in completing manual tasks (e.g., baking a cake) with the system's assistance, fostering mixed-initiative dialogues that pose unique user modeling challenges. in summary, one of the core contributions of this paper is the multi-trait adaptive decoding method (mtad), which allows for the combination of lms to simulate multiple different user profiles at decoding time, removing the need for combinatory data or additional fine-tuning. the second contribution is the introduction of a set of user conversational traits at dialogue-level and utterancelevel, derived from real-world cta data collected during the alexa taskbot challenge @xcite . experimental results support the need for specialized lms to accurately represent each trait. additionally, the results show the scalability and performance of mtad, which can flexibly simulate users with arbitrary combinations of traits, without user profile-specific training. user simulators initial works @xcite used rule-based agendas to model user actions. more recently, model-based approaches @xcite , in specific with llms @xcite @xcite have been used. additionally, diverse user simulators have shown to improve system performance @xcite . we believe more attention should be given to modeling specific user traits, which we address by modeling user profiles as combinations of specialized lms. model combination user simulators must exhibit diverse conversational patterns to effectively evaluate conversational systems. model merging techniques @xcite @xcite combine models at a weight level, enhancing their capabilities across various nlp tasks. alternatively, mixture-of-experts approaches @xcite @xcite use a trainable router in the transformer @xcite to integrate information from multiple models. in a different vein @xcite uses a classifier to adjust token probabilities, while @xcite applies a classifier to determine when to activate the decoding process of different models. in this work, we propose mtad, a controllable and adaptable method that combines token distributions from various models at decoding time without requiring additional fine-tuning. ## adaptable multi-trait user simulators end-to-end simulators @xcite may overlook less common styles and language subtleties due to smoothing or potential forgetting @xcite , while being limited w.r.t. generalization to novel traits. therefore, we consider a trait-oriented model-based approach, in which specialized trait simulators are created and flexibly combined at decoding time. ## user simulation in conversational task assistants (ctas) conversational task assistants (ctas) guide users through tasks such as cooking or diy @xcite . this setting raises a number of challenges @xcite , that are particularly well addressed by mtad: dialogues have mixed-initiative, users follow a task plan with the aim of completing a manual task, ask explorative questions, and engage in chit-chat as shown in ## conclusions this paper addresses the challenge of simulating diverse user traits and effectively combining them in a conversational setting. using real-world data to identify user traits, we developed a framework to generate profile-aware conversations, being one of the first to deliver user simulators for the conversational task assistance (cta) setting. our results demonstrate the need for specialized simulators for each trait (sts), which produce a more accurate trait adherence when compared to jointly learning all traits with a single model. the latter converges to highly smoothed trait distributions that tend to overlook subtle trait characteristics. with sts, trait-specific characteristics are better preserved, including in unseen domains with different tasks, highlighting its generalizability. to flexibly combine diverse user traits and profiles without extra fine-tuning, we proposed multi-trait adaptive decoding (mtad). the results show that mtad effectively combines multiple user traits. this shows that our framework provides an adaptable and controllable approach to create user simulators that can accommodate a wide range of profiles and tasks.