On the Multi-turn Instruction Following for Conversational Web Agents (2024)

Abstract: Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks. Despite these advancements, the potential for LLM-powered agents to effectively engage with sequential user instructions in real-world scenarios has not been fully explored. In this work, we introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment, supported by a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To tackle the limited context length of LLMs and the context-dependency issue of the conversational tasks, we further propose a novel framework, named self-reflective memory-augmented planning (Self-MAP), which employs memory utilization and self-reflection techniques. Extensive experiments are conducted to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the proposed method.

## introduction a longstanding objective in artificial intelligence is to develop ai agents @xcite that can execute complex tasks, thereby minimizing human effort in routine activities. with the advent of large language models (llms), llm-powered agents @xcite showcase exceptional planning capabilities in performing multi-turn interactions with diverse environments, which contribute to various realworld problem-solving. as shown in figure 1 (a), the web agent @xcite @xcite instruction: book a wwe ticket for price range between @xmath0100. yao et al., 2022) is designed to interpret the states of a webpage and execute a series of actions using keyboard and mouse inputs. its purpose is to accomplish the tasks defined in natural language, such as booking tickets, through multi-turn interactions with the web-grounded environment. despite the proficiency in executing each individual instruction, the capability of interacting with multi-turn user instructions remains underexplored, which is crucial for applying llmpowered agents onto real-world applications. as the example shown in figure 1(c ), during a conversational web navigation session, users tend to request follow-up or co-referencing instructions without repeating previous information. they may also provide a succinct or brief instruction, which is similar to other conversation problems. motivated by recent efforts @xcite @xcite on the investigation of conversational capabilities in the interactions with human users for llms, we propose a novel task, named conversational web naviga-tion. it requires the multi-turn interaction capabilities with both users and environment. in particular, we introduce a new dataset, named multi-turn mind2web (mt-mind2web). mt-mind2web is constructed by using the single-turn interactions from mind2web @xcite , an expertannotated web navigation dataset, as the guidance to construct conversation sessions. ## method we introduce the self-map framework, which combines memory-augmented planning with selfreflection. the overview of self-map is presented in figure 3 , consisting of three main components: memory, reflection, and planning modules. ## conclusions to investigate the capability of web agents to follow instructions over multiple turns, we introduce the mt-mind2web dataset for conversational web navigation, which requires complex, multi-turn interactions with both users and the web environment. to overcome the underlying challenges, such as the restricted context length of llms and their dependency on conversational context, we present a novel framework named self-reflective memory-augmented planning (self-map), which utilizes memory augmentation and self-reflection techniques. we rigorously evaluate the mt-mind2web dataset against extensive baselines, conducting thorough analyses across various domains. our experimental findings demonstrate the effectiveness of our proposed approach.