T rip T ailor: A Real-World Benchmark for Personalized Travel Planning (2025)

Abstract: The continuous evolution and enhanced reasoning capabilities of large language models (LLMs) have elevated their role in complex tasks, notably in travel planning, where demand for personalized, high-quality itineraries is rising. However, current benchmarks often rely on unrealistic simulated data, failing to reflect the differences between LLM-generated and real-world itineraries. Existing evaluation metrics, which primarily emphasize constraints, fall short of providing a comprehensive assessment of the overall quality of travel plans. To address these limitations, we introduce TripTailor, a benchmark designed specifically for personalized travel planning in real-world scenarios. This dataset features an extensive collection of over 500,000 real-world points of interest (POIs) and nearly 4,000 diverse travel itineraries, complete with detailed information, providing a more authentic evaluation framework. Experiments show that fewer than 10% of the itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. Moreover, we identify several critical challenges in travel planning, including the feasibility, rationality, and personalized customization of the proposed solutions. We hope that TripTailor will drive the development of travel planning agents capable of understanding and meeting user needs while generating practical itineraries.

## introduction the field of artificial intelligence has seen remarkable advancements in recent years, particularly with the evolution of large language models (llms) @xcite @xcite . these sophisticated models have enhanced reasoning capabilities and a remarkable ability to generate human-like text, making them invaluable across various applications. one such application is in travel planning, where ai-powered tools are revolutionizing how individuals and organizations organize their trips @xcite . while llms hold significant potential for travel planning, current systems mainly rely on numerous rule combinations and human intervention. achieving fully autonomous planning agents that can generate feasible, rational and personalized itineraries remains a considerable challenge. in a benchmark test for domestic travel planning in the u.s. called travelplanner @xcite , even the most advanced model at the time, gpt-4, achieved a mere 0.6% success rate when adhering to all constraints. although this initial finding was disappointing, subsequent research rapidly advanced the field. for instance, @xcite introduced the llm-modulo framework for travel planning, which iteratively combines llms with a series of external verifiers, increasing the success rate to 20.6%. meanwhile, @xcite developed a strategy that integrates llm-based and algorithm-based planning methods, substantially raising the success rate to 97%, effectively addressing the challenge. while methods employing formal verification tools significantly enhance llms' ability to manage complex constraints, does this mean that travel itineraries produced by such llms can compete with those carefully designed by humans in real-world scenarios? to accurately assess this, we need to perform a comparative analysis of different plans related to each user query. however, existing benchmarks for evaluating travel itinerary, such as travelplanner @xcite and chinatravel @xcite , exhibit certain limitations in terms of authenticity and data scale. travelplanner relies primarily on simulated data for its evaluations, making it difficult to reflect actual conditions. in contrast, while ## conclusion we introduce triptailor, a benchmark specifically designed to evaluate the performance of agents in real-world travel planning scenarios. by collecting extensive real-world pois and travel itineraries, and adopting a comparative evaluation approach, we effectively address the issues of limited authenticity, incomplete evaluation, and poor scalability present in previous benchmarks. experiments reveal that even state-of-the-art models struggle to generate feasible, rational, and personalized travel itineraries. we hope that our work provides valuable insights for future research and advances the development of smarter travel planning agents.