GRILLB ot: A multi-modal conversational agent for complex real-world tasks (2022)

Abstract: We present GRILLBot, an open-source multi-modal task-oriented voice assistant to help users perform complex tasks, focusing on the domains of cooking and home improvement. GRILLBot curates and leverages web information extraction to build coverage over a broad range of tasks for which a user can receive guidance. To represent each task, we propose TaskGraphs as a dynamic graph unifying steps, requirements, and curated domain knowledge enabling contextual question answering, and detailed explanations. Multi-modal elements play a key role in GRILLBot both helping the user navigate through the task and enriching the experience with helpful videos and images that are automatically linked throughout the task. We leverage a contextual neural semantic parser to enable flexible navigation when interacting with the system by jointly encoding stateful information with the conversation history. GRILLBot enables dynamic and adaptable task planning and assistance for complex tasks by combining elements of task representations that incorporate text and structure, combined with neural models for search, question answering, and dialogue state management. GRILLBot competed in the Alexa prize TaskBot Challenge as one of the finalists.

## introduction we present grillbot, a task-oriented multimodal conversational assistant developed during the 2021/2022 alexa prize taskbot challenge @xcite . grillbot aims to be an open research platform for complex tasks and supports flexible graph-based task representations, contextual semantic parsing, and incorporates image and video content for clarity and instruction. we release the core components of the system as oat foot_0 (open assistant toolkit). grillbot is still deployed throughout the united states with users able to invoke the bot by issuing the command "hey alexa, assist me" to their voice-only or screened alexa device. our system provides open-ended assistance focusing in the domains of cooking and home improvement. it guides the user through all phases of the task, from performing preference elicitation to guiding a user to a relevant task from large task corpora, (i.e. "making a new york-style pizza" or "how to paint a wall") and then proceeds to assist in executing the task in an engaging way. its capabilities include question answering, task-oriented chit-chat, and instructional video content. grillbot is part of the first generation of assistants (ipek et al.) that leverage screen-enabled conversational devices for complex real-world tasks. these tasks are extensive, with some taking over an hour. as a result, a performant system requires long-term state tracking with capabilities to adapt to a changing environment. it achieves this by introducing a new novel task structure, a taskgraph, that captures the actions and information dependencies to guide the user through a complex task. taskgraphs are enriched offline with content from information extraction, knowledge-based content, and multimedia images and videos. traditional task-oriented dialogue systems @xcite take a slot-filling approach to deriving system actions. academic datasets such as multiwoz @xcite capture slot-value pairs from the user utterances within a constrained set of domains enabling data-driven neural models. @xcite extend this traditional representation towards semantic parsing with dataflow graphs while constrained to the domain of events booking in the smcalflow dataset. the neural decision parser in grillbot similarly generates code but focused on all aspects of a conversation from navigation to task search and question answering. other challenges such as dstc11 @xcite attempt this fully featured taskoriented experience, yet only do so in a virtual setting. grillbot stands apart as a system required to engage with real-world users in their environment and assist in complex tasks for cooking and home improvement. ## system overview the system uses a micro-service architecture with a centralized orchestrator that defines the system behavior. we use a phase-based policy to transition from searching for a suitable task (i.e. planning phase) to guiding users in performing a task (i.e. execution phase). the orchestrator is the central process that directs and receives all the information from other microservices. specifically, these child components are called functionalities and provide the necessary tools required by policies during conversations. the main components inside functionalities are: neural decision parser, task searcher and question answering. we discuss the neural decision parser in section 4. the task searcher leverages our collection of taskgraphs to find candidate tasks. our approach is based on a combination of traditional sparse & dense retrieval and neural re-ranking @xcite . the question answering system provides extra task information, handles user questions, and provides chit-chat elements. it uses a collection of qa systems across six categories. ## taskgraphs a taskgraph is a new graph-based representation based on a directed acyclic graph that encodes the actions and information dependencies that the system needs to enable complex dialogue flows. information is represented with heterogeneous nodes, each with a specific role: combining all these nodes, we can obtain adaptive interactions where system-initiative allows the system to adapt to the user's needs. figure 3 shows how taskgraphs can be leveraged using a task-oriented conversation helping a user cooking "creamy zucchini pasta." ## neural decision parser for example, if a user asks "can you go back to the first step?", the neural decision parser would generate a parameterized parse step_select(1). contextual semantic parsing as a dsl figure 4 shows our state transition domain specific language (dsl) that captures all system actions. this dsl outlines a parameterized global command set that is understood throughout the system to derive what actions or external apis should be called, and what the response utterance should be. this flexible navigation allows for a complex conversation design that leverages taskgraphs. model we use a single t5 large model @xcite to generate an agent action based on the taskgraph and conversational content. using a pre-trained language model allows advanced language capabilities to be leveraged across all system parts, including coreference resolution, search parameterization, setting timers, and state prediction. we train the neural decision parser by annotating simulated conversations with the appropriate function calls and associated arguments. our annotated training data comprises 1,200 turns across various conversation stages and includes taskgraph and conversational context. through user studies and system comparisons, we find that this approach achieves strong performance, and allows flexible task navigation. ## conclusion this demo presents grillbot, a newly developed alexa prize taskbot system for complex realworld tasks with rich multimodal capability. it demonstrates multiple novel components including taskgraphs to manage long complex tasks that are automatically enriched with offline process to add multimodal image and instructional video content. it also shows key elements of the system that make it engaging, including its flexible neural decision parser that performs contextual semantic parsing as parametrized code generation. the result is a demonstration of a new research platform designed from the ground-up around around flexible cloud micro-services and large-scale neural language models.