Grounded Language Agent for Product Search via Intelligent Web Interactions (2024)

Abstract: Recent research has focused on developing agents powered by large language models (LLMs) to accomplish complex high-level user intents. However, employing LLMs with billions of parameters (e.g., GPT-4) may incur substantial costs on top of handcrafting extensive prompts. To address this, we introduce a Grounded Language Agent for Intelligent Web Interactions, named GLAINTEL. GLAINTEL employs Flan-T5 as its backbone and is flexible in training in various settings: unsupervised learning, supervised learning, and unsupervised domain adaptation. Specifically, we tackle both the challenge of learning without human demonstrations and the opportunity to leverage human demonstrations effectively when those are available. Additionally, we explore unsupervised domain adaptation for cases where demonstrations are limited to a specific domain. Experimental evaluations across diverse setups demonstrate the effectiveness of GLAINTEL in unsupervised settings, outperforming in-context learning-based approaches that employ larger models with up to 540 billion parameters. Surprisingly, behavioral cloning-based methods that straightforwardly use human demonstrations do not outperform unsupervised variants of GLAINTEL. Additionally, we show that combining human demonstrations with reinforcement learning-based training yields results comparable to methods utilizing GPT-4. The code is available at: https://github.com/MultifacetedNLP/Web-Agents-Unsupervised

## introduction large language models (llms) have demonstrated their proficiency in diverse tasks such as text classification, information extraction, and question answering @xcite @xcite @xcite . similarly, reinforcement learning (rl) has evolved as a powerful paradigm for training intelligent agents to navigate complex environments @xcite @xcite . moreover, recent research highlights the capabilities of agents powered by llms. for example, agents utilizing gpt-4 can explore the virtual world in minecraft, acquire a diverse set of composable skills, and exhibit exceptional proficiency in playing the game @xcite . the exceptional amount of world knowledge, often derived from vast text datasets, opens up possibilities for developing llm-assisted intelligent web navigation agents capable of navigating and interacting with web pages akin to humans. despite their remarkable capabilities, off-theshelf pre-trained llms face challenges in grounding and aligning themselves in interactive web environments @xcite . this limitation hampers their functional competence without additional customization. additionally, employing llms with billion-scale parameters, such as gpt-4, may incur substantial costs on top of handcrafting extensive prompts. on the other hand, training smaller llms (e.g., flan-t5) as agents can be challenging. for instance, consider a real-world product search scenario, where effective query formulation requires the agent to operate over a huge action space (i.e., language vocabulary), and navigating through diverse web pages poses additional challenges that need strategic exploration due to the presence of different actions on each page (i.e., dynamic action space). this complexity prevents the straightforward utilization of an action head on top of llm. moreover, the challenge extends to preserving long-term memory capabilities, which are crucial for comparing items or backtracking during the search process. in this work, we introduce glaintel, a grounded language agent designed for intelligent web interactions. given a user's intent specifying a product requirement, glaintel formulates queries, navigates diverse web pages, and goal: i am looking for a queen sized bed that is black, and price lower than 140.00 dollars. result page observation: [user instruction]:i am looking for a queen sized … [button] back to search [text] page 1 (total results: 50) [button] @xmath0.0 [link] b09k46kxgr [heading] queen size upholstered platform bed … … … agent dynamic action space: click on [button] back to search click on [button] @xmath1, customize, and purchase the desired product. glaintel uses the open-source flan-t5 language model (i.e., 780 million parameters) as its backbone and can be flexibly trained in various scenarios: unsupervised, supervised, and unsupervised domain adaptation settings. specifically, we address the following research questions. ## related work fine-tuning llms with rl and human feedback. fine-tuning llms with human feedback and reinforcement learning has been studied extensively. @xcite developed the webgpt by fine-tuning the gpt-3 model using behavior cloning and rejection sampling. moreover, instructgpt @xcite was developed using the three-step approach: supervised fine-tuning, reward model training, and reinforcement learning via ppo with the help of the trained reward model. additionally, the authors in @xcite fine-tuned a model that may choose a human-preferred summary, they used this model as a reward function to fine-tune a summarization policy using rl. foundation models for decision making. foundation models possess robust decision-making capabilities, rendering them invaluable across various downstream tasks. for instance, recent works @xcite showcase the application of foundation models in the robotics domain. moreover, works @xcite @xcite utilize foundation models to intelligently navigate android applications. additionally, the foundation models have been utilized in gaming contexts (, fair; @xcite @xcite @xcite . web navigation. many benchmarks and datasets exist for the training and assessment of web agents @xcite @xcite @xcite . researchers have consequently proposed diverse web agents and tested their performance on these benchmarks. the miniwob++ benchmark is among these benchmarks on which different methods have been applied. for example, @xcite employed a combination of reinforcement learning and behavioral cloning, @xcite utilized supervised training on an instruction-finetuned llm, @xcite introduced workflowguided exploration (wge), and @xcite trained dqn agents (qweb network and inet network). additionally, the mind2web benchmark introduced the mindact model, synergizing the strength of small and large llms @xcite . additionally, a visual language model named cogagent was utilized for the benchmark @xcite . @xcite presented agenttuning as another notable approach to tackle the mind2web benchmark. furthermore, considering the webshop benchmark, various methodologies have been proposed that use in-context learning @xcite @xcite , supervised learning @xcite , and rl @xcite . nonetheless, no work has clearly outlined the impact of human demonstrations and the optimal utilization of available demonstration data. furthermore, uda remains underexplored. ## conclusion we introduce glaintel, a flexible agent designed for training across diverse product search scenarios, accommodating situations with limited or no human demonstrations for supervision. we also investigate the optimal utilization of demonstration data, showing that straightforward supervised learning approaches, like behavior cloning, do not yield superior results when using human demonstration data. through extensive experimental evaluations in the webshop environment, we highlight the crucial role of the unsupervised training phase employing the ppo algorithm. when combined with supervised learning, this approach achieved results comparable to methods utilizing gpt-4. additionally, we explore an underexplored scenario where demonstration data is confined to a single domain, we employ uda techniques to accommodate novel domains. we also present evaluations on a real website, ebay, to showcase the applicability of glaintel in the real world.