Can LLM s Truly Plan? A Comprehensive Evaluation of Planning Capabilities (2025)

Abstract: The existing assessments of planning capabilities of large language models (LLMs) remain largely limited to single-language or specific representation formats. To address this gap, we introduce the Multi-Plan benchmark comprising 204 multilingual and multi-format travel planning scenarios. In experimental results obtained with state-of-the-art LLMs, the Multi-Plan benchmark effectively highlights the performance disparities among models, notably showing superior results for reasoning-specialized models. Interestingly, language differences exhibited minimal impact, whereas mathematically structured representations significantly improved planning accuracy for most models, underscoring the crucial role of the input format. These findings enhance our understanding of planning abilities of LLMs, offer valuable insights for future research, and emphasize the need for more sophisticated AI evaluation methods. This dataset is publicly available at http://huggingface.co/datasets/Bllossom/Multi-Plan.

## introduction large language models (llms) have demonstrated near-human performance in various tasks, including translation and summarization @xcite . however, their potential and limitations in human-specific tasks like planning remain underexplored @xcite . planning ability is crucial for human-level intelligence yet remains challenging for llms, as it requires understanding multilayered constraints and sequential reasoning @xcite . current evaluations of llm planning capabilities follow two paradigms: classical and practical. classical planning focuses on generating abstract and formalized action sequences for the * corresponding authors. to this end, we propose multi-plan, a benchmark designed to comprehensively assess llm planning capabilities in multilingual and multi-format contexts. multi-plan consists of 204 travel planning scenarios, each including planning requests and corresponding correct plans expressed in korean, english, and mathematical formats. all scenarios were manually constructed by human annotators to ensure consistency and reliability. additionally, we conducted in-depth evaluations of planning accuracy for the latest llms developed by leading ai enterprises, shedding light on current challenges and future development directions. the key contributions of this study include: ## multi-plan we propose multi-plan, a benchmark designed to comprehensively evaluate the planning capabilities of llms across multilingual and multiformat contexts. this benchmark refers to natural plan @xcite , but unlike natural plan, which exclusively relies on english, our benchmark encompasses korean, english, and mathematical expressions. ## conclusion this study introduced multi-plan, a comprehensive benchmark designed to evaluate llm planning capabilities across multilingual and multi-format contexts. by providing 204 travel planning scenarios in korean, english, and mathematical formats, multi-plan facilitates a broad and balanced assessment of llm planning abilities. evaluations using multi-plan yielded several key insights. first, multi-plan effectively distinguished model performance nuances, highlighting not only the overall superiority of reasoningspecialized models but also sensitively reflecting model architecture-specific characteristics. second, the negligible accuracy differences observed between two structurally distinct languages, korean and english, emphasized the maturity of multilingual processing capabilities in modern llms. ultimately, the study contributed to enhancing our understanding of the planning capabilities and limitations of contemporary llms and highlighted critical considerations for future research and development in sophisticated ai systems.