BASES : Large-scale Web Search User Simulation with Large Language Model based Agents (2024)

Abstract: Due to the excellent capacities of large language models (LLMs), it becomes feasible to develop LLM-based agents for reliable user simulation. Considering the scarcity and limit (e.g., privacy issues) of real user data, in this paper, we conduct large-scale user simulations for the web search scenario to improve the analysis and modeling of user search behavior. Specially, we propose BASES, a novel user simulation framework with LLM-based agents, designed to facilitate comprehensive simulations of web search user behaviors. Our simulation framework can generate unique user profiles at scale, which subsequently leads to diverse search behaviors. To demonstrate the effectiveness of BASES, we conduct evaluation experiments based on two human benchmarks in both Chinese and English, demonstrating that BASES can effectively simulate large-scale human-like search behaviors. To further accommodate the research on web search, we develop WARRIORS, a new large-scale dataset encompassing web search user behaviors, including both Chinese and English versions, which can greatly bolster research in the field of information retrieval.

## introduction web search is a typical information-seeking scenario, where a user issues a query, retrieves the web pages from the search engine, and subsequently selects and browses through the pages of interest. the core to the success of search systems lies in the accurate understanding and modeling of user behavior @xcite . thus, by analyzing large-scale user behavior under web search scenarios, we can enhance the understanding of user's information needs, thereby facilitating the development of more effective search systems @xcite @xcite . however, existing research works typically rely on real-user experiments @xcite , which can be costly to scale up. moreover, concerns regarding the quality and integrity of collected user data emerge as notable impediments, potentially undermining the precision of the analysis @xcite . for example, web search users usually do not maintain a user profile, or the profile is incomplete or noisy. in addition, these experiments encompass ethical considerations, such as user privacy issues. as a result, user behavior simulation emerges as a promising direction to facilitate user behavior analysis and modeling @xcite . recent years have witnessed the unprecedented success of large language models (llms) @xcite @xcite . with superb model capabilities, llms are capable of comprehending complex instructions and executing actions as autonomous agents @xcite . prior studies have explored llm-based agents in various fields, such as recommender systems @xcite and dialogue systems @xcite . in contrast with prior studies, we focus on a quintessential user interaction scenario of web search and utilize llm-based agents to simulate web search user behaviors. despite that this idea is intuitive, there exist several major challenges to simulate the behaviors of web search users. firstly, each user is an independent individual, it is difficult to ensure that each user has a unique and reasonable profile, particularly at a large scale. secondly, predicting web search user behaviors with precision and personalization is challenging. to address these issues, we propose a novel framework bases for large-scale web search user simulation with llm-based agents. first, we design user profile attributes tailored to the characteristics of web search users with reference to the related study @xcite . each user profile encapsulates two basic categories of attribute types, static and dynamic, and a total of eight attribute types are allocated within the two categories. to construct unique profiles for largescale simulated users, we propose a synergistic synthesis method, which predefines a comprehensive range of potential attribute values combining manual definition with the cooperation of gpt-4, ensuring both efficiency and diversity for user profile generation. subsequently, llm-based agents tailored with distinct user profiles are employed to conduct user simulations through the proposed query and click behavior prompting strategies in the search engine, thereby generating precise and personalized user behaviors. to verify the effectiveness of the bases framework, we first show the high anthropomorphism in search behaviors of the llm-based agents by quantitative analysis, and conduct the manual evaluation on the simulated behavior data to demonstrate the personalization degree. following these foundational assessments, we adopt bases within both chinese and english scenarios on two typical information retrieval (ir) tasks, including session search and click prediction. we find that models trained by user behavior data generated by bases achieve improvements of up to 13% in ndcg@1 compared to other behavior datasets trained models, while utilizing less than one percent or one thousandth of their training data volume. moreover, bases also shows the considerable potential in the adaption of low-resource scenarios, bring up to 7% improvements in ndcg@1 with multiple adapting strategies. finally, to accommodate the progressively envolving search patterns of users, we systematically collect the simulated data and construct a new dataset warriors of web search user behaviors with 100,000 users' search sessions for both chinese and english versions. the key contributions are summarized as follows: ## large-scale user simulation for web search in this section, we propose a novel framework for large-scale user simulation in web search scenarios based on llm-based agents, called bases, with specially designed user profile construction strategies and tailored simulation process. we also discuss the superiority and feasibility of the simulation framework by evaluating the human-agent consistency. ## effectiveness verification in this section, we investigate the effectiveness of our proposed bases framework on classical information retrieval (ir) tasks using evaluation benchmarks constructed with real user data. ## the warriors dataset previous experiments verified the effectiveness of the bases framework. to this end, we construct warriors, a large-scale simulation dataset of web search user behaviors based on bases. ## conclusion in this study, we introduced bases, a new framework designed for the simulation of large-scale web search user behaviors with llm-based agents. we designed specific strategies to construct largescale user profiles, which can subsequently lead to human-like search behavior data in our frame-work. through extensive experiments, we not only demonstrated the effectiveness of bases in simulating authentic and diverse user behaviors, but also highlighted its potential in improving information retrieval tasks, especially in low-resource scenarios. furthermore, to facilitate the related research, we develop the warriors dataset, available in both chinese and english versions, which comprises a large collection of simulated web search user behavior data by bases. we believe that this work can provide new perspectives to investigate the search behaviors of web users, thereby contributing to the advancement of search technologies and user experience optimization. for future work, we will enrich the behavior types of simulated users, and also consider applying the bases framework to other fields like sociology analysis.