W eb WISE : Unlocking Web Interface Control for LLM s via Sequential Exploration (2024)

Abstract: This paper investigates using Large Language Models (LLMs) to automatically perform web software tasks using click, scroll, and text in- put operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered Document Object Model (DOM) elements as observations and performs tasks step-by-step, sequentially generating small programs based on the current observations. We use in-context learning, either benefiting from a single manually provided example, or an automatically generated example based on a successful zero-shot trial. We evaluate our proposed method on the MiniWob++ benchmark. With only one in-context example, our WebWISE method using gpt-3.5-turbo achieves similar or better performance than other methods that require many demonstrations or trials.

## introduction a major goal of ai is to develop intelligent agents that interact with their environments to perform tasks. this goal is often explored in the context of physical environments. our work explores performing software tasks with the long-term aim of creating agents that work using software designed for humans. software tasks are valuable by themselves -much of the work we do is on computers -and also offer highly controllable and repeatable tasks that have many of the same challenges as physical tasks, such as manipulable environments, goals that require long sequences of actions, and need for exploration. prior works to control software have used reinforcement learning (rl), requiring many demonstrations and scored trials to learn simple interaction tasks. instead, we use large language models (llms) to generate actions (e.g., * equal contribution click and enter text) based on environment observations (dom elements) in web software. the goal is to complete a web software task, given a natural language instruction and an api for observing and interacting with the environment. in this setting, our observations are dom elements, indicating the layout and state of buttons, text, and other displayed elements. actions include clicking an element, scrolling the mouse wheel, and entering text in a text box. the experiments are performed on the miniwob++ benchmark @xcite , which consists of randomized simple tasks that involve menu navigation, text entry, and/or clicking buttons and other interactive elements. llms are commonly used to generate code, e.g., codex @xcite , and their use has recently been explored to complete ai tasks given an api and a small number of in-context examples, e.g., visprog @xcite , vipergpt @xcite , progprompt @xcite , code4struct @xcite , and vi-struct @xcite . our application of controlling software differs in two key ways: an additional challenge is how to train llms to better control software. this study limits the focus to in-context learning. one option is to manually create examples of programs that satisfy instructions, but this requires some level of expertise and experimentation to be effective. observing that the llms occasionally produce successful programs for specific tasks sometimes (rather than always or never), we propose using these successes as a form of context. this way, llms can create its own context using only a success indicator. in this paper, our main contribution is to investigate the use of llms to control web software, particularly: ## methods this section describes our problem formulation and progressive layers to our approach: based on instruction alone, incorporating filtered dom elements as observations, acting step-by-step, and auto-generating context. figure 1 and algorithm 1 show an overview of our full approach, which we call webwise. algorithm 1 webwise function procedure webwise(input task description l, api a, in-context example @xmath0, . . . , e m , program generator π, dom elements d t ) initialize success state s as false, iteration count t as 0, maximum iterations as t while @xmath1, d t , a, e) ▷ generate program @xmath2, d t ) ▷ execute program d t+1 = getdom () ▷ update dom elements after execution @xmath3.1 problem formulation each task involves a virtual agent assigned to carry out high-level natural language instructions (task descriptions) denoted as l. the agent must interact with the environment, which is initially represented by a set of observable dom elements d t . a program generator π is employed to generate program x @xmath4, d t ). the execution engine ϕ then applies the generated program through ϕ(x t , d t ), resulting in success indicator s and updated dom elements d t+1 , representing the environment's next state. a maximum number of iterations t is set to limit the total number of iterations that the llm can generate programs. ## experiments and results all approaches (single-step, multi-step and auto-context generation) are evaluated on 48 tasks derived from the miniwob++ dataset @xcite . these tasks are chosen to span a range of complexities. additionally, a variant of the singlestep method, which excludes any dom elements, is evaluated and labeled as the 'instruction only' method. the conventional single-step process is labeled as 'instruction+ filtered dom' approach. we evaluate our methods using gpt-3.5-turbo and llama 2 7b. however, we primarily focus on gpt-3.5-turbo as the model demonstrates significantly better performance. ## conclusion our work presents an initial exploration into using large language models (llms) to generate programs that interact with web interfaces. our experiments indicate: filtered dom elements are effective forms of observation; the step-by-step action and observation is more effective than single-step generation; and automatically generated in-context examples from successful trials can boost success rates for many tasks.