Grounding Open-Domain Instructions to Automate Web Support Tasks (2021)

Abstract: Grounding natural language instructions on the web to perform previously unseen tasks enables accessibility and automation. We introduce a task and dataset to train AI agents from open-domain, step-by-step instructions originally written for people. We build RUSS (Rapid Universal Support Service) to tackle this problem. RUSS consists of two models: First, a BERT-LSTM with pointers parses instructions to WebLang, a domain-specific language we design for grounding natural language on the web. Then, a grounding model retrieves the unique IDs of any webpage elements requested in the WebLang. RUSS may interact with the user through a dialogue (e.g. ask for an address) or execute a web operation (e.g. click a button) inside the web runtime. To augment training, we synthesize natural language instructions mapped to WebLang. Our dataset consists of 80 different customer service problems from help websites, with a total of 741 step-by-step instructions and their corresponding actions. RUSS achieves 76.7% end-to-end accuracy predicting agent actions from single instructions. It outperforms state-of-the-art models that directly map instructions to actions without WebLang. Our user study shows that RUSS is preferred by actual users over web navigation.

## introduction grounding natural language is a key to building robots and ai agents @xcite that interact seamlessly with people. besides grounding tasks visually @xcite , future ai agents must be able to ground language and execute actions on the web. we build a general-purpose, interactive agent to master tasks from open-domain natural language instructions on websites. we focus on the service domain for tasks such as redeeming a gift card, 3 2 1. ## task and model given a set of natural language instructions @xmath0, . . . , i n ) and a starting web page, our task is to construct an agent that follows the instructions through a series of action @xmath1, . . . , a n ). actions include web navigation and end-user inter-action in order to obtain necessary information. surveying online customer service tasks, 6 action operations were identified as necessary for agents: open a url page, enter text, click on buttons, say something to the user, read the results to the user, and ask user for some information. details are described in russ is trained to execute tasks by grounding natural language instructions on the web. the modular design of russ, with separate semantic parser and grounding model, is motivated by the high cost of training data acquisition, and the ability to improve each component independently. we first describe thingtalk, then the three components of russ: the semantic parser model, the grounding model, and the runtime. ## datasets this paper contributes two detasets, the russ evaluation dataset with real-world instructions and the russ synthetic dataset for training semantic parsers. ## conclusion russ demonstrates how a semantic parser and grounding model can be used to perform unseen web tasks from natural language instructions. by achieving 76.7% accuracy on the russ evaluation dataset, we show how a modular semantic parsing approach can outperform end-to-end neural models on this task, and demonstrate how humans interact with russ-like systems in the user study. like many datasets in nlp, we believe extensive research is still required to go from russ's 76.6% overall accuracy on the evaluation dataset to 100%. as seen in the russ evaluation dataset introduces a set of real instructions for grounding language to executable actions on the web to evaluate future research in this direction, including training semantic parsers to new targets using real-world instructions and neural models for grounding formal language representations on the web. our work provides the task, technical foundation, and user research for developing open-domain web agents like russ.