TOAD : Task-Oriented Automatic Dialogs with Diverse Response Styles (2024)

Abstract: In light of recent advances in large language models (LLMs), the expectations for the next generation of virtual assistants include enhanced naturalness and adaptability across diverse usage scenarios. However, the creation of high-quality annotated data for Task-Oriented Dialog (TOD) is recognized to be slow and costly. To address these challenges, we introduce Task-Oriented Automatic Dialogs (TOAD), a novel and scalable TOD dataset along with its automatic generation pipeline. The TOAD dataset simulates realistic app context interaction and provide a variety of system response style options. Two aspects of system response styles are considered, verbosity level and users’ expression mirroring. We benchmark TOAD on two response generation tasks, and the results show that modeling more verbose responses or responses without user expression mirroring is more challenging.

## introduction task-oriented dialog (tod) stands as a fundamental task of machine intelligence @xcite , involving goal-driven conversations between a human and a system to achieve specific tasks. it has been used in training virtual assistant systems for various real-world applications, including mobile phones (hoy, 2018), virtual reality devices (vr) @xcite , smart home devices @xcite , online shopping assistants @xcite , and trip booking helpers @xcite . advancements in developing tod systems face a challenge posed by conflicting requirements @xcite : the demand for extensive datasets clashes with the substantial time and financial investments (months if not years) required for data collection @xcite . looking "lv + m": done. "lv + no m": removed. (default) ## data generation pipeline the toad pipeline, depicted in fig. 2 , comprises three key stages for the automated creation of dialogues. the initial stage involves the generation of device context, simulating users' personas and their device app contexts. in the second stage, essential conversation attributes, including conversational content, flow, and language phenomena, are determined and incorporated into a dialog plot. the final stage involves the realization of the dialog plot into dialog utterances by a dialog generator. this generator simulates user speaking habits and provides a spectrum of system response styles. ## system response style the primary evaluation criterion for today's virtual assistants is their accuracy in responding and assisting users in achieving their goals. the next generation of virtual assistants is expected to exhibit increased naturalness and adaptability to various usage scenarios, which means the systems should be able to respond with different styles. toad stands as the first work in investigating the dimensions of style that virtual assistants should consider. we aim to open the discussion on optimal response styles to encourage future exploration. in this work, our focus centers on two main aspects: verbosity level and mirroring. ## dataset statistics the current toad dataset covers 11 service domains with 37,678 turns (18,839 user turn and 113,034 system response variations, 131,873 turns in total) in 8,087 dialogues. it's worth highlighting that as toad operates as an automatic generation pipeline, both datasize and covered service can be scaled up. additionally, the creation of a new service schema typically requires only 15 minutes. across the 8087 dialogs, 40.5% are multi-intent compound, 20.9% are compositional and 38.6% are single intent. the distributions of the number of turns and word counts in utterances are shown in fig. 4(i ) and (ii) respectively. as for local phenomena, 19.0% dialog are self-revision, 15.1% have complex referral, 3.3% have both and 62.6% have no local phenomena. note that local phenomena can appear in both single and multi-intent dialog. fig. 4 (iii) illustrates the service distribution which is relatively uniform for single intent and compound dialogs. however, in compositional dialog, due to the joint sample requirement, certain services may have less matchable slots. scaling up the supporting services in the schema can potentially expand the joint service sample pool. as mentioned above, toad does not have a fixed set of plot actions. fig. 4 (iv) displays the distribution of actions whose occurrences are greater than 3%. based on the selection rules for the default response style, 61.7% of the system responses are default high verbosity, 28.8% and 9.5% are mid and low verbosity. the average word counts for across different verbosity styles. all results are evaluated on the predictions generated by the oracle language model setup on the test set. we consistently noting that the performance on mirroring responses is better than non-mirroring ones, as expected, given that mirroring allows the system to replicate user expressions directly from the dialog history, making it easier to predict. we also observe that as verbosity increases, performance tends to decrease. this decline is likely attributed to lower verbosity resulting in shorter utterances with a reduced vocabulary, making them easier to model. another observation worth noticing is that, even for non-mirroring options, having access to dialog history can yield improved modeling performance, as evident when comparing the left-most column of tab. 3 with the second left-most column of tab.4. ## conclusion in conclusion, our study explores the naturalness and adaptiveness of system responses for the next generation of tod virtual assistants. we introduce toad, a dataset designed to train tod systems for diverse verbosity levels, mirroring styles, and realistic app context interactions. additionally, we present a cost-effective and scalable automatic data generation pipeline as a practical alternative to traditional human annotations. by addressing those critical gaps, we aim for toad to inspire future exploration in modeling and analyzing system response styles.