Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context (2020)

Abstract: We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that all considered interpretations are consistent with executable actions in any context, thus simplifying the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition without any linguistic annotation.

## introduction people routinely perform repetitive web-based tasks, involving sequences of clicking and typing actions. these include activities such as forwarding emails, booking flight tickets, ordering pizza, etc. these activities largely consist of small sequences of actions in an environment with restricted semantics, and are potentially amenable to automation. in this work, we explore whether an ai agent can be taught such tasks through natural language explanations and a single demonstration by a user (as one might teach such a task to a human assistant). here, we explore learning very simple tasks from the mini world-of-bits framework using natural language explanations and a single demonstration of the task from the perspective of language understanding, this involves challenges such as converting instructional language to actions, resolving ambiguities through pragmatics, and learning script-like behavior. the web domain is rich in textual, structural and spatial features, allowing for exploration of multiple types of grounding behavior including spatial and visual language understanding, as well as reasoning over semi-structured data. also, despite its richness, the tasks involved usually do not require much background knowledge. from a practical perspective, teachable ai assistants can change the way people interact with computers. today's conversational assistants such as alexa or cortana act on a small number of preprogrammed language commands (e.g., "what is the weather going to be like?"). however, they cannot be taught new functionalities important to a user (as in figure 1 ). enabling users to teach computers personalized procedures through explained demonstrations can make conversational ai systems fundamentally more useful. in section 2, we situate our work in the broader body of work on grounded semantic parsing and learning from language. section 3 summarizes our framework and dataset. in section 4, we describe our approach in detail. here, we investigate a new paradigm for interpreting language in grounded contexts. instead of mapping statements to logical forms that then execute in a context as in traditional semantic parsing, the method considers the set of possible typing and clicking actions in a context, identifies features of corresponding web elements and their relationships with other elements on the webpage, and aligns these to natural language explanations through a generative model. section 5 describes the empirical evaluation. our contributions are: interactive learning from language: several frameworks have leveraged natural language supervision to learn new tasks, starting with early work on the shrldu system @xcite and interactive task learning @xcite . in particular, several reinforcement learning approaches have been explored in text-based environments for learning strategies, following instruction manuals, game playing, etc. @xcite @xcite . these approaches leverage the ability to explore and interact with the environment to learning policies that lead to favourable outcomes. this is different from our goal here, where the agent needs to learn from a single explained demonstration of a task, and no interactivity with the environment is assumed. some recent approaches have shown language explanations to be effective for learning realistic tasks including relation extraction, concept learning and question answering @xcite @xcite . in terms of the goal and problem formulation, our approach extends multiples lines of previous work. @xcite 's work is similar to ours in motivation in learning user-specified recipes, but has no aspects of grounding or demonstrations. @xcite explore interactive parser training through language games in context of block-world environments. @xcite explore mapping natural language to specific elements on complex and realistic web-pages, although not in context of learning from demonstrations. our framework directly extends previous work on learning web-based tasks from the mini word-of-bits framework using multiple demonstrations and exploration of the environment @xcite . in particular, our dsl extends the constraint language defined in @xcite to explore learning from explained demonstrations instead. ## framework and dataset we build on the mini world-of-bits (miniwob) framework @xcite , a collection of webbased tasks initially proposed as a testbed for reinforcement learning agents. the tasks vary in difficulty in terms of the number of actions required, variability between instances of the task, and types of reasoning involved (including clicking specified buttons, forwarding emails and playing tic-tac-toe). see the top half of figure 2 for an example of a task. each task consists of a task description (yellow box), and an interactive web interface. while previous methods have focused on learning sequential decision making to complete these tasks through a mixture of exploration (the framework provides simulators, where correctly completing a task yields a reward) and behavior cloning (by observing multiple demonstrations from human users); our focus is on learning to complete these tasks in a one-shot sense (without any exploration). this is because the one-shot case is a much more realistic scenario for learning web-based procedures from a teacher. in practical situations (where there are no simulators), it would not be feasible for an ai agent to learn to book flights by booking multiple incorrect tickets, or manage a user's email by sending multiple incorrect emails. on the other hand, a paradigm where the agent attempts to generalize from a single demonstration and explanations can be feasible for many more of such scenarios. task: forward an email click on the segment that mentions maureen click on the button name "forward" at the bottom of the page type in the word 'amata' in front of the row tagged 'to' click on the arrow button at the top of the page ## learning from explanations and demonstrations our approach for learning web-based tasks, which we call led -for learning from explained demondemonstrations (led). we prefer logical forms (l) that are both consistent with the user demonstration (d) in the context (c), and relevant to the user's explanations (x). figure 4 illustrates this for a toy-example, where the context consists of a web-page with three elements, the demonstration consists of a single action, and a corresponding explanation is provided. based on the observed demonstration (that elem3 was clicked), it is hard to infer the reason behind clicking it. multiple logical forms in the dsl can be consistent with clicking elem3 in this context. e.g., it is at the top of the page, its color is blue, etc. however, these interpretations would not justify the provided explanation as those logical forms are not relevant to the explanation. modeling relevance between logical forms and explanations can help identify the reasoning behind user demonstrations. this framing diverges from traditional semantic parsing, where statements x are mapped to logi-cal forms l (e.g., database queries), which are then are executed against a context c (e.g., a knowledge base) to get a denotation (corresponds here with a demonstration) d. i.e., @xmath0. in this modeltheoretic view of semantics, parsed logical forms are not informed by the environmental context until execution. in comparison, led roots logical forms in the observed context, and thus pragmatic consistency is ensured by design. 2 we maximize the log-likelihood of observing the explanations given the demonstration in a grounded context: here, the first term corresponds to scoring relevance between logical forms and explanations (modeled using a semantic parsing model). the second term enforces consistency between candidate logical forms and the demonstration in the context, and can be deterministically evaluated. as we see in section 4.2, consistency is enforced by temperature-based annealing during training. ## experiments we next discuss led's empirical performance. ## conclusion our work here is a step in the direction of teachable ai agents that can learn new behavior from conversational interactions with ordinary users. in terms of technique, our bottom-up approach to generating logical forms ensures consistency between interpretations and the ambient context during search. conversely, this would be complicated in domains with rich composition and nesting in logical forms, which go beyond simple features and relations. e.g., "click the third email from jeanette", and where modeling inverse semantics is infeasible. here, we posed the learning of web-based tasks as similar to instruction-following problem, with no aspect of interactivity or exploration of the environment. in future work, the possibility of learning from a mix of explanations, exploration and a limited budget of interaction with the environment can be explored. also, language grounding models that incorporate richer alignments between explanations and demonstrations can lead to more effective learning. since led only requires tokenization as pre-processing, it can possibly extend to low resource scenarios. in terms of problem framing, interactive use-cases that enable the agent to ask questions when it is confused may also be realistic. future work can also explore curriculum learning in this domain, by first learning simpler tasks, which can be compositionally invoked in explanations for complex tasks.