U rban LLM : Autonomous Urban Activity Planning and Management with Large Language Models (2024)

Abstract: Location-based services play an critical role in improving the quality of our daily lives. Despite the proliferation of numerous specialized AI models within spatio-temporal context of location-based services, these models struggle to autonomously tackle problems regarding complex urban planing and management. To bridge this gap, we introduce UrbanLLM, a fine-tuned large language model (LLM) designed to tackle diverse problems in urban scenarios. UrbanLLM functions as a problem- solver by decomposing urban-related queries into manageable sub-tasks, identifying suitable spatio-temporal AI models for each sub-task, and generating comprehensive responses to the given queries. Our experimental results indicate that UrbanLLM significantly outperforms other established LLMs, such as Llama and the GPT series, in handling problems concerning complex urban activity planning and management. UrbanLLM exhibits considerable potential in enhancing the effectiveness of solving problems in urban scenarios, reducing the workload and reliance for human experts.

## introduction location-based services are ubiquitous in urban spaces, supporting a range of scenarios from commuting assistance for travelers and daily activities for residents to event monitoring for city regulators. the diverse and substantial demand for location-based services has driven the development of specialized ai models tailored to specific tasks within spatio-temporal context. these tasks include (spatio-)temporal forecasting @xcite @xcite , imputation @xcite , and anomaly detection @xcite , as well as travel time estimation @xcite figure 1 : the process of solving a real-world problem in urban scenarios involving urban activity planning by human experts. @xcite , trajectory prediction @xcite , and poi recommendation @xcite , etc. despite the promising results of specialized ai models in addressing urban-related tasks with fixed input formats, these models are inadequate for handling complex queries in natural language that require strategic planning, reasoning, and collaboration among multiple specialized models across potentially various data modalities, such as gps coordinates, addresses, and traffic records. as illustrated in figure 1 , to answer the given query, human experts need to decompose it into several sub-tasks, select and employ an appropriate combination of models for each sub-task, and synthesize the response based on model outputs. to enhance user experiences in location-based services, it is critical to design an autonomous and effective method for solving urban-related problems, particularly regarding complex urban activity planning and management. large language models (llms), such as chat-gpt foot_0 , have attracted significant interest from both academia and industry @xcite . llms have been applied in diverse domains, such as commerce, finance, healthcare, and the geospatial domain @xcite @xcite . their exceptional capabilities in language comprehension and reasoning have positioned them as core modules in autonomous agents, where they function as problem solvers @xcite . however, conventional llms, as well as llm agents built on them, encounter several challenges specific to our target. first, conventional llms, despite their intrinsic ability in reasoning and comprehension, possess limited geospatial knowledge about tasks and ai models within spatio-temporal context in urban scenarios @xcite . second, agents such as autogpt foot_1 , agentgpt foot_2 , and babyagi foot_3 utilize tools for web search and code execution, which are not suited for location-based services. moreover, they are designed to refine solutions for individual tasks in a recursive way, lacking the ability to holistically analyze and decompose urban-related problems for multiple specialized models. third, while hug-ginggpt @xcite proposes to schedule text and image-related problems into sub-tasks and coordinate responses from corresponding models, it still relies on conventional llms as its backbones, thus inheriting their limitations in solving tasks in urban scenarios. to tackle these challenges, we propose urban-llm, a novel urban foundation model to effectively decompose queries into sub-tasks and schedule specialized ai models within spatio-temporal context for each sub-task, thereby autonomously solving complex problems in urban scenarios. the idea is to align the problem-solving process with the established paradigms of llms, and enhance llms with the capability for urban activity planning and management through targeted fine-tuning. in this process, llms are equipped to serve as a universal interface capable of handling diverse tasks for various urban scenarios and producing appropriate responses. specifically, llms are initially fine-tuned using a structured template with high-quality examples that focus on spatio-temporal task decomposition and scheduling, thus augmenting the reasoning capabilities for the problems in targeted urban scenarios. subsequently, the processing of new urbanrelated problems is conducted in three stages for inference phase. first, in the task analysis stage, a query of urban-related problems is effectively decomposed into a series of sub-tasks, each of which corresponds to a specific type of specialized ai models within spatio-temporal context. next, in the model matching stage, the most suitable model is selected for each sub-task from a pool of candidate specialized ai models based on their descriptions. finally, in the results generation stage, the selected models are executed to obtain output results, which are then organized into prompts to formulate a comprehensive response to the original query. by utilizing urbangpt, we facilitate the efficient and convenient resolution of urban-related problems without intensive manual efforts. our contributions are summarized as follows. ## related work large language models (llms), due to their powerful capability in reasoning and comprehension, are increasingly utilized to assist in specific urban applications. for example, trafficgpt @xcite employs chatgpt as a control agent to interact with various system components, such as databases, visualization and statistical tools, to perform basic analytical operations in traffic-related tasks. travelplanner @xcite assessed the performance of large language models (llms) functioning as travel planning agents for responding to travel-related queries under various constraints. the experimental results indicate that current llms, including gpt-4, achieved a success rate of only 0.6%. geogpt @xcite utilizes chatgpt to address similar analytical operations in geographical information systems (gis) domain. @xcite leverages llms to simulate roles such as planners and residents within a multi-agent framework to help urban land use and development planning. llmob @xcite introduces a framework that considers individual activity patterns for urban mobility data generation. however, these studies typically rely on original llms and possess limited domain knowledge, which restricts their effectiveness in addressing challenges regarding complex urban planning and management. to overcome such intrinsic limitations, various initiatives have deployed fine-tuned llms in targeted scenarios. for example, llmlight @xcite is fine-tuned based on llama-2 to improve decision-making and policies in traffic signal control. transgpt @xcite ) is trained on a corpus of examples from traffic prediction, public transportation, and autonomous driving, to address urban transportation tasks such as identifying traffic rules and signs. urbangpt @xcite aims to enhance spatio-temporal forecasting accuracy in a zero-shot setting by integrating a specialized decoder with an instruction-tuning paradigm. geollm @xcite enhances urban regional questions such as population density and home value by fine-tuning llms with specifically designed templates. moreover, several models focus more on text-related dimensions. plangpt @xcite is fine-tuned on a large corpus of urban planning regulations from numerous local governments in china, aimed at revising or generating texts for new regulations and evaluating planning documents. k2 @xcite learns additional geospatial knowledge from a collection of geoscience text training corpus, enhancing nlp tasks such as summarization and text classification, specifically for the geoscience domain. unlike all these studies, we are the first to equip llms with the capability to decompose queries regarding complex urban planning and management into manageable components that align with specialized ai models within spatio-temporal context, thereby autonomously tackling diverse urban-related problems. ## urbanllm our proposed urbanllm is structured into two phases: the learning phase and the inference phase. in the learning phase, we fine-tune urbanllm, employing meticulously crafted examples from our constructed self-instruct dataset. this dataset includes examples that contain reasoning hints, various types of backbone spatio-temporal ai models, diverse queries, and the corresponding decomposed sub-tasks. this fine-tuning process effectively enhances the comprehension and reasoning capabilities of urbanllm tailored to address our targeted objectives in urban scenarios. the inference phase consists of three stages: spatial-temporal analysis, model matching, and results generation. in the spatio-temporal analysis stage, urbanllm receives queries that are organized within the same prompt template used during the training stage with new queries, enabling it to effectively decompose the query into a series of types of spatio-temporal tasks, owing to enhancements achieved through fine-tuning. the model matching stage involves pairing each identified spatio-temporal task with suitable ai models and selecting the most appropriate one. finally, in the results generation stage, the selected spatiotemporal models are executed, and their outputs are formulated into a prompt to produce the response to the query. the overall process for the two phases is depcited in figure 2 . ## experiments and results discussion in this section, we compare urbanllm with several strong llm baselines, such as llama-3 and gpt-4o, to demonstrate the urban activity planning and management capability and superior performance of urbanllm after instruction fine-tuning. we then include a case study to visually illustrate the differences between urbanllm and gpt-4o. furthermore, we conduct an ablation study to test the contribution of each component in urbanllm. ## conclusions in this study, we introduce urbanllm, a fine-tuned llm developed to enhance the ability to perform autonomous urban activity planning management. after fine-tuning on a corpus of examples of problems in urban scenarios, urbanllm learns to decompose new queries into sub-tasks and identifies appropriate spatio-temporal ai models for each sub-task, thereby enhancing the accuracy of urban planning and the efficiency of management processes. operating through both the learning and the inference phase and three meticulously designed stages, spatio-temporal task analysis, model matching, and results generation, urbanllm functions as a pipeline to achieve the problem-solving process and produce the response to the given query. our experimental results demonstrate that urban-llm significantly outperforms other llm models, including llama-3 and the gpt-4o, in the context of urban activity planning and management tasks by a large margin. @xcite