Personal Large Language Model Agents: A Case Study on Tailored Travel Planning (2024)

Abstract: Large Language Models (LLMs) have made significant progress, becoming more autonomous and capable of handling real-world tasks through their access to tools, various planning strategies, and memory, referred to as LLM agents. One emerging area of focus is customizing these models to cater to individual user preferences, thereby shaping them into personal LLM agents. This work investigates how the user model, which encapsulates user-related information, preferences, and personal concepts, influences an LLM agent’s planning and reasoning capabilities. We introduce a personalized version of TravelPlanner, called TravelPlanner+, and establish baselines for personal LLM agents. Our evaluation strategy contains an LLM-as-a-Judge component, which provides further in-depth insights into the decision-making process of a personal LLM agent by comparing generic and personal plans. Our findings reveal that while generic plans perform robustly, personal plans show marked improvement in relevance and suitability, with preference rates up to 74.4% on validation and 87.3% on the test set. These results highlight the potential of personal LLM agents to significantly enhance user satisfaction.

## introduction ai agents are computational entities that perceive their surroundings, plan, and take actions using tools to complete a task @xcite . due to the emergent capabilities of large language models (llms), augmenting llms with reasoning capabilities and tools enables them to act as ai agents @xcite , i.e., llm agents. to provide personalized solutions to users, agents need to know their profiles, personal preferences, concepts and understand their personal queries. personal concepts can be anything specific to the * these authors contributed equally to this work. in this study, we explore the impact of users' models on agent's decision-making and planning processes to create personalized solutions for users. different environments exist to evaluate the capabilities of llm agents @xcite . however, none currently incorporate personal user information. we drew inspiration from the travelplanner benchmark @xcite , designed to generate a travel plan based on the user's text-based query. travelplanner provides a rich and complex environment to test the efficacy of the llm agents. like other agent-based benchmarks, the travelplanner benchmark offers a generic environment where no personal information or characteristics are provided for the customer. to investigate the effectiveness of the llms as a personal agent, in this study, we provide : to evaluate the capabilities of llm-based agents, various benchmarks @xcite have been developed across different categories, including code @xcite , game @xcite , and web @xcite @xcite . we specifically focused on web-based environments and decided to adapt travelplanner because it has a wider range of uses for personalization among the general population. additionally, travelplanner presents significant challenges due to its multi-constraints and long-term planning. to make things even more complex, we introduced a personalization element to fully expose and address these challenges. ## personal llm agent: travelplanner+ we present the travelplanner+ to evaluate the effectiveness of llm agents in generating personalized travel itineraries based on the user model and the reference information, referred to as sole planning. this setting eliminates the need for tool calls, as agents no longer need to gather information from scratch using tools. this provides an opportunity to evaluate agents' planning skills solely @xcite . this reference information comprises detailed and essential data provided directly to the agents, including restaurants, accommodations, and attractions in the specified cities in the query. in this work, we develop distinct user models to benchmark the effectiveness of open-and closedsource llm agents and evaluate agents' planning skills in generating personalized plans. the benchmarking process involves using queries from trav-elplanner's validation and test splits and four planning strategies to craft multi-day (three-, five-, or seven-day) itineraries tailored for each user. additionally, we curate personal queries to evaluate the performance of llm agents in crafting plans that align with specific personal profiles and concepts. ## conclusions our work introduced travelplanner+, a personalized version of the travelplanner benchmark. we created user models to integrate into the decisionmaking process of the llm agent. this was the first study entirely devoted to personal llm agents. our design decisions, such as the size of the open-source models and the non-parametric approach to personalization, were made to facilitate the on-device deployment of llm agents, providing a privacy-preserving solution by design. as demonstrated by the llm-as-a-judge, our evaluation framework clarified the quality of individual plans, which were previously obscured by generic performance indicators.