Interactive Task Learning from GUI -Grounded Natural Language Instructions and Demonstrations (2020)

Abstract: We show SUGILITE, an intelligent task automation agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations, using the graphical user interfaces (GUIs) of third-party mobile apps. This system provides several interesting features: (1) it allows users to teach new task procedures and concepts through verbal instructions together with demonstration of the steps of a script using GUIs; (2) it supports users in clarifying their intents for demonstrated actions using GUI-grounded verbal instructions; (3) it infers parameters of tasks and their possible values in utterances using the hierarchical structures of the underlying app GUIs; and (4) it generalizes taught concepts to different contexts and task domains. We describe the architecture of the SUGILITE system, explain the design and implementation of its key features, and show a prototype in the form of a conversational assistant on Android.

## introduction interactive task learning (itl) is an emerging research topic that focuses on enabling task automation agents to learn new tasks and their corresponding relevant concepts through natural interaction with human users @xcite . this topic is also known as end user development (eud) for task automation @xcite . work in this domain includes both physical agents (e.g., robots) that learn tasks that might involve sensing and manipulating objects in the real world @xcite , as well as software agents that learn how to perform tasks through software interfaces @xcite @xcite . this paper focuses on the latter category. a particularly useful application of itl is for conversational virtual assistants (e.g., apple siri, google assistant, amazon alexa). these systems have been widely adopted by end users to perform tasks in a variety of domains through natural language conversation. however, a key limitation of these systems is that their task fulfillment and language understanding capabilities are limited to a small set of pre-programmed tasks @xcite . this limited support is not adequate for the diverse "long-tail" of user needs and preferences @xcite . although some software agents provide apis to enable thirdparty developers to develop new "skills" for them, this requires significant programming expertise and relevant apis, and therefore is not usable by the vast majority of end users. natural language instructions play a key role in some itl systems for virtual assistants, because this modality represents an natural way for humans to teach new tasks (often to other humans), and has a low learning barrier compared to existing textual or visual programming languages for task automation. some prior systems @xcite @xcite @xcite relied solely natural language instruction, while others @xcite @xcite also used demonstrations of direct manipulations to supplement the natural language instructions. we surveyed the prior work, and identified the following five key design challenges: ## system overview this section explains how sugilite learns new tasks and concepts from the multi-modal interactive instructions from the users. the user starts with speaking a command. the command can describe either an action (e.g., "check the weather") or an automation rule with a condition (e.g., "if it is hot, order a cup of iced cappuccino"). suppose that the agent has no prior knowledge in any of the involved task domains, then it will recursively resolve the unknown concepts and procedures used in the command. although it does not know these concepts, it can recognize the structure of the command (e.g., conditional), and parse each part of the command into the corresponding typed resolve functions, as shown in figure 1 . sug-ilite uses a grammar-based executable semantic parsing architecture @xcite ; therefore its conversation flow operates on the recursive execution of the resolve functions. since the resolve functions are typed, the agent can generate prompts based on their types (e.g., "how do i tell whether. . . " for resolvebool and "how do i find out the value for. . . " for resolvevalue). when the sugilite agent reaches the resolve function for a value query or a procedure, it asks the users if they can demonstrate them. the users can then demonstrate how they would normally look up the value, or perform the procedure manually with existing mobile apps on the phone by direct manipulation (figure 2a ). for any ambiguous demonstrated action, the user verbally explains the intent behind the action through multi-turn conversations with the help from an interaction proxy overlay that guides the user to focus on providing more effective input (see figure 2bcde , more details in section 3.2). when the user demonstrates a value query (e.g., finding out the value of the temperature), sugilite highlights the gui elements showing values with the compatible types (see figure 3 ) to assist the user in finding the appropriate gui element during the demonstration. all user-instructed value concepts, boolean concepts, and procedures automatically get generalized by sugilite. the procedures are parameterized so that they can be reused with different parameter values in the future. for example, for utterance 8 in figure 1 , the user does not need to demonstrate again since the system can invoke the newlylearned order starbucks function with a different parameter value (details in section 3.3). the learned concepts and value queries are also generalized so that the system recognizes the different definitions of concepts like "hot" and value queries like "temperature" in different contexts (details in section 3.4). ## evaluation we conducted several lab user studies to evaluate the usability, efficiency and effectiveness of sug-ilite. the results of these study showed that end users without significant programming expertise were able to successfully teach the agent the procedures of performing common tasks (e.g., ordering pizza, requesting uber, checking sports score, ordering coffee) @xcite , conditional rules for triggering the tasks @xcite , and concepts relevant to the tasks (e.g., the weather is hot, the traffic is heavy) @xcite using sug-ilite. the users were also able to clarify their intents when ambiguities arise @xcite . most of our participants found sugilite easy and natural to use @xcite @xcite . efficiency wise, teaching a task usually took the user 3-6 times longer than how long it took to perform the task manually in our studies @xcite , which indicates that teaching a task using sug-ilite can save time for many repetitive tasks. ## conclusion we described sugilite, a task automation agent that can learn new tasks and relevant concepts interactively from users through their gui-grounded natural language instructions and demonstrations. this system provides capabilities such as intent clarification, task parameterization, and concept generalization. sugilite shows the promise of using app guis for grounding natural language instructions, and the effectiveness of resolving unknown concepts, ambiguities, and vagueness in natural language instructions using a mixed-initiative multi-modal approach.