Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings (2022)

Abstract: Conversational Task Assistants (CTAs) are conversational agents whose goal is to help humans perform real-world tasks. CTAs can help in exploring available tasks, answering task-specific questions and guiding users through step-by-step instructions. In this work, we present Wizard of Tasks, the first corpus of such conversations in two domains: Cooking and Home Improvement. We crowd-sourced a total of 549 conversations (18,077 utterances) with an asynchronous Wizard-of-Oz setup, relying on recipes from WholeFoods Market for the cooking domain, and WikiHow articles for the home improvement domain. We present a detailed data analysis and show that the collected data can be a valuable and challenging resource for CTAs in two tasks: Intent Classification (IC) and Abstractive Question Answering (AQA). While on IC we acquired a high performing model (>85% F1), on AQA the performance is far from being satisfactory (~27% BertScore-F1), suggesting that more work is needed to solve the task of low-resource AQA.

## introduction in recent years, the way to access web information has changed from using keyword-and semanticsbased search @xcite , to question answering (qa) @xcite and conversational agents (cas) @xcite . cas have evolved to support different types of interaction and information: there are cas for chatting @xcite and cas that let users interact with existing information systems to accomplish specific tasks, e.g., booking a restaurant, as in task-oriented dialogues @xcite . a specific type of information humans are looking for is how to perform tasks, e.g., cooking a dish or fixing a household problem. the web contains articles accompanied by images or videos with step-by-step instructions to perform variegate tasks. existing cas can be used mainly to browse tasks or to answer specific questions. however, they fail in providing a comprehensive natural conversation that includes search, context-aware qa, step-by-step instructions, and multi-modal information sharing. the alexa prize taskbot foot_0 challenge @xcite is a research challenge sponsored by amazon to foster research on ctas to assist humans in executing real-world tasks. the targeted tasks require multiple steps and decisions, including multi-modal (voice and visual) user experiences. the challenge includes two domains: cooking, i.e., guiding people in preparing recipes; and home improvement, i.e., guiding people through common household do-it-yourself tasks such as painting a wall or pruning trees. as shown in figure 1 , ctas should support qa capabilities on web sources as well as selected recipes or articles, dialog management to support step-by-step instruction navigation, and multi-modal interaction. in this paper, we present wizard of tasks foot_1 (wot), the first dataset for ctas. we collected a total of 549 conversations with ∼ 18k utterances in two domains with a wizard of oz (woz) crowd-sourcing setting, where one worker is willing to execute a task, while another worker has the relevant knowledge to perform it and guides the first towards its execution. we adopted an asynchronous strategy to collect conversations, so that neither worker needs to wait for the other to respond and can multi-task better, enabling faster data collection. we present a detailed analysis of the dataset as well as experiments in two tasks: intent classification (ic) and abstractive question answering (aqa). we show that a transformerbased ic model can @xmath0. in contrast, the performance on aqa is still far from satisfactory (∼27% bertscore-f1), suggesting the need for better qa models for low-resource settings. in the rest of the paper, section 2 discusses related work. section 3 presents our data collection strategy. in sections 4 and 5 we discuss the data analysis and experimental evaluation. finally, in section 6, we state our conclusions. ## related work there is a large body of work focusing on the development of training corpora for conversational agents. in the following section, we summarize the work done to collect conversations for task-oriented and open-domain cas. ## crowd-sourcing a cta dataset we designed a crowd-sourcing task to create a dataset suitable for a cta to assist users in completing complex tasks requiring multiple steps and reasoning. we collected conversations for two target domains 3 : i) cooking, i.e., assisting users in performing recipes and ii) home improvement (diy), i.e., assisting users in performing tasks to improve their home. ## wizard of tasks analysis in this section, we present the analysis of the wizard of tasks data. dataset statistics are presented in section 4.1, followed by a linguistic analysis in section 4.2. foot_7 ## experiments and results in this section, we experiment with two tasks: intent classification and abstractive question answering, to demonstrate the value of the wizard of tasks dataset for improving existing ctas. ## conclusions in this paper, we presented wizard of tasks, a novel dataset for conversations in the area of conversational task assistants. this dataset is inspired by the alexa prize taskbot challenge, where the participants have to create a dialog agent on top of the alexa platform to help users in performing real world tasks in two domains: cooking and home improvement. to the best of our knowledge, our task-oriented conversational dataset is the first of its kind in these two domains. we discussed our dataset collection asynchronous strategy in a crowdsourcing setting. we reported multiple analyses of the collected data as well as initial experimental evaluations on two tasks: intent classification and abstractive question answering. despite the small size of the collected data, we hope it can foster research in the novel area of ctas. in the future, we plan to expand the dataset to additional domains.