Grounded Complex Task Segmentation for Conversational Assistants (2023)

Abstract: Following complex instructions in conversational assistants can be quite daunting due to the shorter attention and memory spans when compared to reading the same instructions. Hence, when conversational assistants walk users through the steps of complex tasks, there is a need to structure the task into manageable pieces of information of the right length and complexity. In this paper, we tackle the recipes domain and convert reading structured instructions into conversational structured ones. We annotated the structure of instructions according to a conversational scenario, which provided insights into what is expected in this setting. To computationally model the conversational step’s characteristics, we tested various Transformer-based architectures, showing that a token-based approach delivers the best results. A further user study showed that users tend to favor steps of manageable complexity and length, and that the proposed methodology can improve the original web-based instructional text. Specifically, 86% of the evaluated tasks were improved from a conversational suitability point of view.

## introduction voice-based assistants can guide users through everyday complex tasks, such as cooking, crafts, and home repairs. these conversational assistants need to understand the users' intention, find a specific recipe, and communicate it in a structured and wellpaced manner. supporting this type of task-guiding interaction is a recent topic @xcite @xcite , where conversational assistants must work hand-inhand with users in order to guide them throughout the task execution. we argue that most instructional texts found online are structured in a non-optimal way for conversational assistants, due to the inherent differences between screen and voice-based interfaces. recipes are a great example where the decomposition of the recipe's text into dialogue-suited steps is critical (example in appendix a) -as @xcite observed, the user is dividing attention through various and possibly parallel actions. hence, and following previous knowledge @xcite , we aim for steps that are structured and presented to the user in ordered pieces of information, while dosing complexity, with the aim of achieving an efficient task completion. to tackle this new problem, we part ways with topic-based segmentation methods @xcite @xcite and propose a novel human-focused methodology to convert reading-structured instructions into conversational ones. figure 1 offers a clear view of the differences between the original web-based recipes and their dialogue-suited counterparts. the distribution of linguistic characteristics such as length, verbs (which cover actions), and nouns (covering ingredients, tools, etc.), confirms that dialog-suited instructions should avoid overwhelming users' short-term memory @xcite . our proposed methodology starts with the curation of a corpus which we call convrecipes, where online recipes are segmented into recipes with steps more suited to a conversational agent. moreover, we identify the key traits of a conversational step. an example of this can be seen in figure 2 . this example shows the need for models that can tackle our task, in specific, in "step 2" of the task, where it is noticeable that the step would be very difficult to follow in a conversational assistant due to the long sentence and the inherent complexity of its actions. to tackle this task computationally, we propose the dialogue-task segmenter transformer (dts-transformer), which follows state-of-the-art approaches in text-segmentation @xcite @xcite and adopts a transformer-based backbone @xcite . distinct from previous work, we follow a tokenlevel approach which by modeling steps' text at a finer granularity, is capable of better modeling the inherent structural characteristics of conversational tasks. note that, we did not follow generative approaches and ground our task segmentation task on the recipes' original text. we do this to avoid the risk of introducing hallucinations or mistakes in step-by-step procedures @xcite . finally, we validated the proposed methodology with automatic experiments, and, more interestingly, with a user evaluation. we observed that the best dts model, a t5-3b encoder backbone, trained on the proposed convrecipes corpus, was able to improve the conversational structure of 86% of the evaluated tasks. this evidences both the conversational characteristics of the convrecipes corpus and the effectiveness of the model's approach to the grounded conversational task segmentation task. next, we will relate our contributions to previous corpus and methods. in section 3, we carefully detail the proposed methodology. experimental validation and user evaluations are presented in section 4, and we conclude with the final takeaways and future work. methods and models. initial works for text segmentation were mostly based on statistical and unsupervised approaches, such as texttiling @xcite and c99 @xcite . after these, supervised neural methods, particularly with the use of rnns were utilized. in @xcite , a cnn is used to generate sentence embeddings in conjunction with an lstm to keep sequential information. @xcite also presents an rnnbased model with an additional pointing mechanism and in @xcite it is used a hierarchical bi-lstm model. currently, the state-of-the-art is based on supervised transformer-based approaches @xcite @xcite . in @xcite , cross-segment and hierarchical models are proposed, where predictions are made based on consecutive segments or sentencebased representations of the segments. @xcite presented a hierarchical approach combining sentence and cross-segment embeddings. @xcite proposed a hierarchical bilstm to complement @xcite sentence representations, aided by a coherence-related auxiliary task. some approaches such as @xcite , tackle task structuring as a generation task, where an end-to-end pipeline is proposed to generate day-to-day tasks. in a dialogue setting, @xcite applied a bert model for transcriptbased meetings segmentation @xcite by calculating the similarity between segment embeddings given by a pretrained model. given the particular intricacies of conversational-suited task structuring, while we also adopt a transformer backbone, we propose a task segmentation model that makes decisions at a token-level being able to consider the global task's structure. ## structuring conversational tasks our hypothesis is that the recipe instructions found online are not suited for conversational assistants, motivating both the task and the dataset collection efforts. to convert instructions from a reading structure into a conversationally structured format, we followed a human-focused methodology. first, we collected task instructions and ran a user study to curate them as conversational instructions. second, we ask users to annotate the relevance of various conversational instructions traits. third, we analyzed the linguistic characteristics of reading instructions compared to conversational task instructions. finally, we modeled conversationalsteps computationally with various transformerbased @xcite architectures. ## experiments in this section, we demonstrate how the proposed framework tackles the challenge of structuring task instructions in a conversational setting. experimental validation was done with both automatic metrics and human evaluation. ## conclusions in this paper, we proposed a methodology to tackle the problem of converting web/reading structured instructions into conversationally structured ones, using a task-grounded segmentation by considering the original task's steps. in summary, the key contributions are as follows: convrecipes corpus. this corpus enables a better understanding of the problem. its analysis showed that instructional text as it is presented online is not optimal for a conversational setting.