Multi-User M ulti WOZ : Task-Oriented Dialogues among Multiple Users (2023)

Abstract: While most task-oriented dialogues assume conversations between the agent and one user at a time, dialogue systems are increasingly expected to communicate with multiple users simultaneously who make decisions collaboratively. To facilitate development of such systems, we release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent. To collect this dataset, each user utterance from MultiWOZ 2.2 was replaced with a small chat between two users that is semantically and pragmatically consistent with the original user utterance, thus resulting in the same dialogue state and system response. These dialogues reflect interesting dynamics of collaborative decision-making in task-oriented scenarios, e.g., social chatter and deliberation. Supported by this data, we propose the novel task of multi-user contextual query rewriting: to rewrite a task-oriented chat between two users as a concise task-oriented query that retains only task-relevant information and that is directly consumable by the dialogue system. We demonstrate that in multi-user dialogues, using predicted rewrites substantially improves dialogue state tracking without modifying existing dialogue systems that are trained for single-user dialogues. Further, this method surpasses training a medium-sized model directly on multi-user dialogues and generalizes to unseen domains.

## introduction voice assistants like amazon alexa and google assistant are widespread, and users often interact with them in multiparty settings, such as playing games and making decisions with family members @xcite . however, most dialogue systems are designed to support only single-user dialogues, i.e., the agent expects to converse with one user at a time via a succinct command that contains all and only necessary information for conducting a task. by contrast, multi-user task-oriented dialogues are significantly richer, containing deliberation and social chatter, and poses additional challenges like separating out task-relevant information from social and sensitive information related to user privacy. the main bottleneck for a first step into supporting multi-user task-oriented dialogues is a lack of proper datasets, as most (if not all) existing datasets for task-oriented dialogues are single-user. to overcome this limitation and facilitate future research, we build and release a dataset of multi-user task-oriented dialogues and propose the novel task of multi-user contextual query rewriting. our dataset multi-user multiwoz is an extension of multiwoz 2.2 @xcite to multi-user dialogues (figure 1 , §3 ). multiwoz 2.2 is one of the largest and most popular single-user task-oriented dialogues. the guiding principle of our data collection is to extend each user utterance in multiwoz 2.2 to a chat between two users making decisions together (multi-user chat henceforth) that leads to the same dialogue state as the source utterance. this allows us to (1) re-use system acts and responses annotated in the source single-user dialogue and (2) train a query rewriting model that converts a multi-user chat to its source utterance (as the ground-truth rewrite) so that output rewrites can be consumed by dialogue systems that expect single-user utterances. compared to existing related datasets, which are either single-user @xcite @xcite or lack dialogue state annotations @xcite , our dialogues reflect interesting dynamics of collaborative decision-making in task-oriented conversations, such as questions to elicit slot values, social chatter, and deliberation. empowered by this dataset, we propose the novel task of multi-user contextual query rewriting: to rewrite a multi-user chat as a single request that is concise and contains all and only task-relevant information ( §5). this task is important because (1) it can bridge the gap between multi-user chats and a dialogue system trained for single-user dialogues without replacing the entire dialogue system, and (2) it alleviates users' privacy concerns by processing multi-user chats on device and sending the rewrites to the system server with only taskrelevant information. we demonstrate the accuracy of baseline models on the rewriting task and discuss the main challenges. further, we verify that model-predicted rewrites are helpful for dialogue state tracking for dialogue systems that are trained on single-user dialogues, by substantially outperforming a baseline that simply concatenates utterances in a multi-user chat and treats it as a "single" utterance. this task also benefits unseen domains. our contributions are twofold: our work is closely related to three fields of nlp: task-oriented dialogues, contextual query rewriting, and dialogue summarization. task-oriented dialogues: existing datasets of task-oriented dialogues @xcite @xcite @xcite are, to our knowledge, all single-user conversations. by contrast, our dataset consists of task-oriented dialogues where two users are making decisions together with the help of an agent. since input to a dialogue system is a chat between users rather than a single user utterance, such dialogues pose challenges to dialogue systems trained via traditional algorithms for dialogue state tracking @xcite . while multiparty dialogue datasets exist @xcite , they are not task-oriented and thus missing important information for training task-oriented dialogue systems. our multi-user dataset is task-oriented by nature and annotated with relevant information, such as dialogue states and system acts and responses. contextual query rewriting: contextual query rewriting refers to decontextualizing a user query to a self-contained one that contains necessary contextual information for the agent to serve the user's request @xcite . this mostly involves anaphora resolution and ellipsis resolution, and sequence-to-sequence models (e.g., gpt) have performed well @xcite . our work is related in that the source utterance of each mutli-user chat can be seen as a decontextualized rewrite of the chat that the agent can process without parsing the whole chat. while existing work and datasets for contextual query rewriting focus on converting one user request at a time @xcite @xcite , our task rewrites a chat between two users (hence, multi-user contextual query rewriting). this task is more challenging as it often goes beyond anaphora/ellipsis resolution; for instance, it also involves resolving deliberation between users, and user intents and slots may be spread out across multiple utterances in a chat. our dataset provides training and evaluation resources for this task. dialogue summarization: our dataset is also related to dialogue summarization in that each multi-user chat can be seen as a short task-oriented dialogue between two users and its rewrite as a summary of the dialogue. existing datasets for dialogue summarization are not well-suited for summarizing multi-user task-oriented dialogues. one of the main reasons is that they have different goals for summarization. for instance, many datasets @xcite @xcite @xcite @xcite aim to summarize diverse perspectives of speakers rather than succinct task-oriented information @xcite . our dataset focuses on summaries where deliberations are resolved and only task-relevant information (e.g., user intents and slots) is retained. while some datasets aim to summarize task-oriented dialogues from customer services @xcite @xcite , they are not designed to summarize chats between users; rather, they summarize a chat between user and agent, which has different dynamics than collaborative decision-making between users. ## data collection in building a dataset of multi-user task-oriented dialogues, our principle is to extend an existing dataset of single-user task-oriented dialogues, such that each user utterance in that dataset is expanded to a chat between two users (multi-user chat) that leads to the same dialogue state. two main benefits to this approach are: (1) we can reuse system acts and responses in the original dataset without annotating them anew, and (2) pairs of a multi-user chat (in the collected data) and its source utterance (in the original data) can be used to train a query rewriting model that rewrites a multi-user chat as a concise query (and vice versa). to that end, we use multiwoz 2.2 @xcite as our basis foot_1 . it consists of task-oriented dialogues between one user and one agent, where the agent helps the user with booking and finding information for eight services (hotel, attraction, restaurant, train, taxi, bus, police, hospital). we expand each of the first four user utterances foot_2 in each dialogue to an imaginary chat between two users making collaborative decisions. we describe our pilot study, data collection protocol, validation process, and data statistics. ## dialogue quality assessment we verify that the collected dialogues have high quality and are realistic to happen in the real world. adapting the dialogue quality assessment in @xcite to multi-user dialogues, we evaluated each dialogue in six aspects: scale: 1 (always says something that abruptly for realism, our data and original multiwoz data showed no statistically significant difference (mann-whitney u test), meaning evaluators judge our dialogues to be as likely to occur in reality as the well-established multiwoz. our data was rated higher than the pilot data (p-@xmath0.03), suggesting that collecting multi-user task-oriented di-alogues in a pairwise setting is difficult. it can produce less realistic dialogues than our protocol, mainly because laypeople are not good at creating a collaborative scenario and holding a relevant dialogue in interactive settings, unlike single-user settings like multiwoz and wizard of wikipedia @xcite . regarding the other criteria, original multiwoz was rated slightly higher than our data (by 0.2 points) for naturalness, coherence, and consistency with p-@xmath1.05. this is expected because it is naturally difficult for multi-user dialogues to achieve the same level of coherence and consistency as single-user dialogues. compared to the pilot data, our data showed no statistically significant difference for any criteria (except for realism), suggesting our protocol produces similar quality to a pairwise setting. even compared to well-established multi-party dialogue datasets, our dialogues are scored higher than dailydialog for interestingness and topicalchat for coherence. ## multi-user contextual query rewriting we propose the novel task of multi-user contextual query rewriting, which rewrites a task-oriented chat between users as a concise query. our main goal in creating this task is for this query rewriting to serve as the first module of a language understanding pipeline. the main advantage of such a module is that it can be plugged into an existing dialogue system and convert a multi-user chat to a concise self-contained utterance, enabling multi-user dialogues with minimal modifications to the dialogue system. further, this module can filter out taskirrelevant information (e.g., personal information and social chatter) on device before sending the rewrites to the system server, thus enhancing user privacy. ## conclusion we release the multi-user multiwoz dataset, containing task-oriented dialogues between two users and one agent. the dialogues reflect important characteristics of collaborative decision-making in task-oriented settings, such as slot elicitation, social chatter, and deliberations. this dataset also enables the task of multi-user contextual query rewriting, which aims to rewrite a multi-user chat as a concise query that contains task-relevant information. we demonstrated that this task improves dialogue state tracking for a dialogue system trained for singleuser dialogues both in-domain and cross-domain. this result is promising because this task can easily be plugged into a language understanding pipeline, processing multi-user dialogues with little modification to the dialogue system. further, this task can be conducted on device, filtering out task-irrelevant information in multi-user chats and sending only task-relevant information to the system server for further processing, which enhances user privacy. our work assumes that a dialogue system starts dialogue state tracking only after a multi-user chat ends. this approach is common in practical dialogue systems, where systems start processing user inputs only when the user signals their utterance is directed at the device (e.g., by using a wake word). for multi-user task-oriented dialogues, an alternative approach is to track dialogue states after each utterance in multi-user chats; that is, a dialogue state is updated as soon as a user finishes speaking to the other user or the system. while this approach is theoretically plausible and allows dialogue systems to proactively intervene in the middle of a multi-user chat, it requires substantial effort to annotate a dialogue state for each user turn in multi-user chats. by contrast, query rewrites in our dataset are a byproduct of our data collection and thus do not require annotation effort. nevertheless, it is an interesting question which method is more effective between turn-level dialogue state tracking and contextual query rewriting (as in our work). we leave this to future work.