Hierarchical Prompting Assists Large Language Model on Web Navigation (2023)

Abstract: Large language models (LLMs) struggle on processing complicated observations in interactive decision making. To alleviate this issue, we propose a simple hierarchical prompting approach. Diverging from previous prompting approaches that always put the full observation (a web page) to the prompt, we propose to first construct an action-aware observation which is more condensed and relevant with a dedicated Summarizer prompt. The Actor prompt then predicts the next action based on the summarized history. While our method has broad applicability, we particularly demonstrate its efficacy in the complex domain of web navigation where a full observation often contains redundant and irrelevant information. Our approach outperforms the previous state-of-the-art prompting mechanism with the same LLM by 6.2% on task success rate, demonstrating its potential on interactive decision making tasks with long observation traces.

## introduction in our daily lives, we often encounter tasks such as household duties @xcite and web navigation @xcite that necessitate interactive and sequential decisionmaking. these tasks require us to take actions (e.g., entering a search query) based on both the state of the environment and the specific objectives (e.g., buy a shirt). there has been growing interest in automating these decision-making tasks using natural language commands @xcite . large language models demonstrate their potential on performing interactive decision making tasks, owing to their capacity for encoding vast amounts of real-world knowledge. one common approach for utilizing llms in these tasks is [button] citrus @xcite [button] mango [button] size [button] travel set (4-pack) @xcite [button] 3 ounce (pack of 1) @xcite [button] 3-ounce (2-pack) @xcite [button] family 16oz [button] scent [button] citrus @xcite [button] mango @xcite size [button] 3 ounce (pack of 1) @xcite [button] 3-ounce (2-pack) @xcite .. a t-1 figure 1 : ash prompting. in each round, the sum-marizer will produce a condensed observation given raw observation and the actor's previous action. the actor will produce the next action based on the more concise summarized observation and the interaction history. the right side shows the raw observation and the summarized observation for the instruction "find me a small bottle of fruit deodorant". we apply ash on web navigation with a special focus on purchasing products on a real-world scale e-commerce simulator @xcite , since it is the only relevant existing dataset with a complex observation space to the best of our knowledge. our ash significantly outperforms the previous state-of-the-art approach @xcite ) by 6.8% on task success rate (29% relative gain). these results show the promise of ash in addressing the challenges of comprehensive history and complicated states in real-world interactive decision making tasks. ## the web navigation task in this section, we formally define the task of web navigation. given a web environment e, initial state s 0 with observation o 0 (typically a web page), the goal of an agent is to reach a specific environment state s * defined by natural language u by performing a sequence of actions. at time t, the agent needs to decide the action a t based on interaction history up to t (h @xmath0, a 1 , o 1 , . . . , a t-1 , o t }). once a stop action is predicted, the score of the task is measured by comparing the final state ŝ with s * with a similarity function f (ŝ, s * ) which ranges from zero to one. ## ash prompting as summarized in figure 1 , ash comprises of a stateless summarizer that takes an action a t-1 and observation o t and then outputs a condensed observation o ′ t that is semantically comprehensible by the actor. then, the stateful actor takes updated history h ′ @xmath1, a 1 , o ′ 2 , a 2 , . . . , a t-1 , o ′ t } different scenarios. in figure 1 , the instruction requests a small bottle of deodorant with a fruit scent. the current product includes suitable configurations that meet this requirement. however, there are unnecessary details like product descriptions and distracting options such as large-volume bottles. the summarizer filters out these irrelevant contents, retaining only the relevant information for further processing. (2) an actor that takes the trajectory history (i.e., observations and actions) and produce the next action. the hierarchical modularized design of ash effectively reduce the heavy reasoning burden by avoiding dumping all information to a single prompt. the summarizer of ash could effectively remove irrelevant information, and provide a more concise observation representation. this representation aligns with the demonstration in the context for the actor to make more precise predictions. we apply ash on web navigation with a special focus on purchasing products on a real-world scale e-commerce simulator @xcite , since it is the only relevant existing dataset with a complex observation space to the best of our knowledge. our ash significantly outperforms the previous state-of-the-art approach @xcite ) by 6.8% on task success rate (29% relative gain). these results show the promise of ash in addressing the challenges of comprehensive history and complicated states in real-world interactive decision making tasks. ## the web navigation task in this section, we formally define the task of web navigation. given a web environment e, initial state s 0 with observation o 0 (typically a web page), the goal of an agent is to reach a specific environment state s ⇤ defined by natural language u by performing a sequence of actions. at time t, the agent needs to decide the action a t based on interaction history up to t (h @xmath2, a 1 , o 1 , . . . , a t 1 , o t }). once a stop action is predicted, the score of the task is measured by comparing the final state ŝ with s ⇤ with a similarity function f (ŝ, s ⇤ ) which ranges from zero to one. ## ash prompting as summarized in figure 1 , ash comprises of a stateless summarizer that takes an action a t 1 and observation o t and then outputs a condensed observation o 0 t that is semantically comprehensible action: click @xcite instruction: find me a 3 ounce bottle of of bright citrus deodorant observation: scent [button] assorted scents @xcite [button] bright citrus [button] size @xcite travel set (4-pack) @xcite [button] 3 ounce (pack of 1) @xcite [button] 3-ounce (2-pack) @xcite ... information: let us think step-by-step. first, i need to let the actor know if this is the correct product. then, i need to filter out options that does not match to make it easier for the actor to choose from scent @xcite bright citrus [button] size @xcite bright citrus @xcite [button] 3 ounce (pack of 1) @xcite [button] 3-ounce (2-pack) [button] by the actor. then, the stateful actor takes updated history h 0 @xmath3, a 1 , o 0 2 , a 2 , . . . , a t 1 , o 0 t } and predicts a t . mathematically, ash decomposes the probability of generating a t to two components. ## experiment dataset we test ash on @xcite , a real-world scale e-commerce benchmark where the agent interacts with an amazonstyle website to purchase products described by some nl descriptions (e.g., "find me black ereader with price lower than 270.00 dollars"). in this benchmark, the agent can perform three actions: (1) search[...] with a text-based query (2) click[...] any interactable web element (3) think[...] the current situation and make further movements. the think action, which does not change the state, is adopted from re-act @xcite . webshop presents a significant challenge for intelligent agents with its large-scale and diverse product catalog, consisting of 1,181,436 unique products, each with its own distinct features, resulting in a highly complex and varied state space. we used text rich setting of webshop on the full product space plus, and sampled a fixed 500 trajectory for experiments. 2 2 model and hyperparameters we use code-davinci-002 @xcite for both ash and the reproduction of react. 3 3 we use greedy decoding and max token of 500. we terminate a trajectory after 20 steps (the average number of steps is 6.5), or if the agent issues an invalid action (e.g., click a button that does not exist) for five consecutive steps. ## results the main result is given in performance vs. trajectory length one key challenge in interactive decision making is to handle long-horizon tasks due to the difficulty of keeping track of long histories. to analyze this problem, we plot the average score with respect to the length of the trajectories in figure 4 . we observed that ash consistently yields a higher average score than react across different trajectory lengths. the performance gap is particularly large on examples with more than 11 steps. while react only achieves an 2022) on how to summarize the current state (in green). importantly, we design specialized examples tailored to different scenarios to encourage versatile reasoning. for example, as in figure 2 , the instruction outlines how to process a product page. specifically, the summarizer is tasked to verify the desirability of the product, followed by filtering out any irrelevant options. more instructions on improving search query, compressing search results and others can be found in appendix c.1. in this way, the summarized states can be more customized towards the current scenario. ## experiment dataset we test ash on @xcite , a real-world scale e-commerce benchmark where the agent interacts with an amazonstyle website to purchase products described by some nl descriptions (e.g., "find me black ereader with price lower than 270 dollars"). in this benchmark, the agent can perform three actions: (1) search[...] with a text-based query (2) click[...] any interactable web element (3) think[...] the current situation and make further movements. the think action, which does not change the state, is adopted from react @xcite . webshop is challenging for intelligent agents due to its large-scale and diverse product catalog, consisting of 1, 181, 436 unique products, each with its own distinct features, resulting in a highly complex and varied state space. 2 task score calculation we follow @xcite to calculate the task scores. if a trajectory ends with buying a product, it will be assigned a score between zero and one, denoting the percentage of required attributes satisfied by the product. a score of 1.0 indicates a successful trajectory, while a 0.0 marks a failure. however, as a special case, if a trajectory either produces k consecutive invalid actions or exceeds the max length, it will be terminated and assigned a score of 0.0. ## results the main result is given in ## conclusion we introduce ash, a hierarchical method tailored for sequential decision-making in web navigation using llms. tested on the webshop benchmark, ash surpassed the strong prompting method re-act in average score and success rate. notably, ash excels in complex tasks with lengthy trajectories. it reduces llms' hallucinations and mitigates the need of explicit reasoning.