Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances, Resources, and Future Directions (2025)

Abstract: Autonomous Driving Systems (ADSs) are revolutionizing transportation by reducing human intervention, improving operational efficiency, and enhancing safety. Large Language Models (LLMs), known for their exceptional planning and reasoning capabilities, have been integrated into ADSs to assist with driving decision-making. However, LLM-based single-agent ADSs face three major challenges: limited perception, insufficient collaboration, and high computational demands. To address these issues, recent advancements in LLM-based multi-agent ADSs have focused on improving inter-agent communication and cooperation. This paper provides a frontier survey of LLM-based multi-agent ADSs. We begin with a background introduction to related concepts, followed by a categorization of existing LLM-based approaches based on different agent interaction modes. We then discuss agent-human interactions in scenarios where LLM-based agents engage with humans. Finally, we summarize key applications, datasets, and challenges in this field to support future research (https://github.com/Yaozuwu/LLM-based_Multi-agent_ADS).

## introduction autonomous driving systems (adss) are redefining driving behaviors, reshaping global transportation networks, and driving a technological revolution @xcite . traditional adss primarily rely on data-driven approaches (as detailed in appendix a), focusing on system development while overlooking dynamic interactions with the environment. to enhance engagement with diverse and complex driving scenarios, agentic roles have been incorporated into adss @xcite using methods like reinforcement learning @xcite and active learning @xcite . despite notable progress, these methods struggle with "long-tail" scenarios, where rare but critical driving situations, such as sudden obstacles, pose significant challenges to model performance. furthermore, their "black-box" nature limits interpretability, making their decisions difficult to trust. llm-based single-agent adss help overcome the limitations of data-driven methods @xcite . pre-trained on vast, multi-domain datasets, llms excel in knowledge transfer and generalization @xcite , enabling strong performance in traffic scenarios under zero-shot settings, thus addressing the long-tail issue @xcite . moreover, techniques such as reinforcement learning from human feedback (rlhf) and chain-of-thought (cot) @xcite , enhance language-based interaction and logical reasoning, allowing llms to make human-like, realtime decisions while providing interpretable and trustworthy feedback across various driving conditions. for instance, drive-like-a-human @xcite builds a closed-loop system comprising environment, agent, memory, and expert modules. the agent interacts with the environment, reflects on expert feedback, and ultimately accumulates experience. for example, dilu @xcite replaces human experts with a reflection module and integrates an llm-based reasoning engine to enable continuous decision-making. agent-driver @xcite designs a tool library to collect environmental data and uses llms' cognitive memory and reasoning to improve planning. however, as shown in figure 1 , researchers have identified three critical limitations of llm-based single-agent adss in complex traffic environments: ❶ limited perception: llms can only respond to sensor inputs and lack predictive and generalization capabilities. as a result, llm-based singleagent adss cannot complement incomplete sensor information and thus miss critical information in at an intersection without traffic lights, an accident has occurred ahead, causing veh1 to be stuck. due to limited perception, veh1 is unable to assess the situation and cannot proceed. veh2 intends to go straight, and veh3 wants to turn left. however, due to insufficient collaboration, they are also unable to navigate the intersection efficiently. furthermore, due to high computing demands, the lightweight agent on veh1 struggles to handle the complex driving scenario and has to rely on a more powerful cloud-based agent for assistance. to address these limitations, llm-based multiagent adss enable distinct agents to communicate and collaborate, improving safety and performance. first, llms enhance contextual awareness by allowing agents to share data, extend their perceptual range, and enhance the detection of occluded objects in complex environments @xcite . second, real-time coordination among llm-based agents mitigates insufficient collaboration, enabling joint decisions in tasks like lane merging and roundabout navigation, ultimately leading to safer and more efficient driving operations @xcite . third, llms optimize computational efficiency by distributing tasks across agents, reducing individual load and enabling realtime processing in resource-limited systems. as llm capabilities continue to advance, they are playing an increasingly significant role in ads as intelligent driving assistants. several reviews have focused on two primary aspects: i) the integration of llms in data-driven methods @xcite and ii) the applications of specific llm types, such as vision-based @xcite and multimodal-based @xcite ) models in ads. however, no comprehensive survey has systematically examined the emerging field of llm-based multi-agent adss. this gap motivates us to provide a comprehensive review that consolidates existing knowledge and offers insights to guide future research and the development of advanced adss. in this study, we present a comprehensive survey of llm-based multi-agent systems. specifically, section 2 introduces the core concepts, including agent environments and profiles, inter-agent interaction mechanisms, and agent-human interactions. section 3 provides a structured review of existing studies: multi-vehicle interaction, vehicleinfrastructure interaction, and vehicle-assistant interaction. as agent capabilities continue to grow, human-vehicle co-driving is emerging as the dominant autonomous driving paradigm, with human playing an increasingly vital role. humans collaborate with agents by providing guidance or supervising their behavior. therefore, we consider humans as special virtual agents and examine humanagent interactions in section 4. section 5 explores various applications, while section 6 compiles a comprehensive collection of public datasets and open-source resources. section 7 discusses existing challenges and future research directions. finally, section 8 concludes the study. ## llm-based agent-human interaction depending on the roles of human assume when interacting with agents, we classify current methods as: instructor paradigm and partnership paradigm. ## datasets and benchmark we organize recent open-source work to foster research on advanced adss. mainstream ads datasets are summarized in single-agent autonomous driving data. singleagent datasets are obtained from a single reference agent, which can be the ego vehicle or roadside infrastructure, using various sensors. mainstream singel-agent autonomous driving datasets like kitti @xcite , nuscenes @xcite , and waymo @xcite provide comprehensive multimodal sensor data, enabling researchers to develop and benchmark algorithms for multiple tasks such as object detection, object tracking, and object segmentation. in addition to these foundational datasets, newer ones like bdd-x @xcite , drivelm @xcite , and nuscenes-qa @xcite introduce action descriptions, detailed captions, and questionanswer pairs that can be used to interact with llms. combining language information with visual data can enrich semantic and contextual understanding, promote a deeper understanding of driving scenarios, and improve the safety and interaction capabilities of autonomous vehicles. ## challenges and future directions this section explores key open challenges and potential opportunities for future research. ❶ hallucination, safety & trustworthiness. hallucination refers to llms generating outputs that are factually incorrect or non-sensical @xcite . in complex driving scenarios, a single driving agent's hallucinations in an llm-based multiagent ads can be accepted and further propagated by other agents in the network via the inter-agent communication, potentially leading to serious accidents. detecting agent-level hallucinations and managing inter-agent information flow are key to improving system safety and trust @xcite . recent advances in spatiotemporal traffic analysis @xcite further support real-time condition assessment, improving vehicle-road interaction and overall safety of ads. ❷ legal, security & privacy. as agents autonomously exchange and process information within multi-agent ads, the distribution of legal liability between individual users and manufacturers becomes ambiguous, particularly in cases involving system failures or collisions. in addition, vulnerable communication methods and strict user privacy requirements place high demands on cryptographic protocols and data management. these interrelated concerns collectively represent critical directions for future research and regulatory initiatives. ❸ multi-modality ability. in current multi-agent systems, agents primarily use llms for scene understanding and decision-making. perception outputs are converted into text via manual prompts or interpreters, then processed by llms to generate decisions. this pipeline is limited by perception performance and may cause information loss @xcite . integrating language understanding with multimodal data fusion offers a promising direction for future multimodal multi-agent adss. ❹ real-world deployment & scalability. llmbased multi-agent ads can scale up by adding more agents to handle increasingly complex driving scenarios. however, more llm agents increase the demand for computing resources, while their interactions impose strict requirements on communication efficiency, which is critical for real-time decision-making @xcite . therefore, under limited computing resources, it is crucial to develop a system architecture that supports distributed computing and efficient communication, as well as agents capable of adapting to various realworld environments and tasks, to optimize multiagent ads within resource constraints. ❺ human-agent interaction. current multi-agent ads struggle to communicate intentions to human road users, relying on static signals inadequate for complex scenarios. developing llmpowered adaptive interfaces that generate contextappropriate, human-understandable communications while maintaining safety and trust presents a key deployment challenge @xcite . ## conclusion this paper systematically reviews llm-based multi-agent adss and traces their evolution from single-agent to multi-agent systems. we detail their core components, including agent environments and profiles, inter-agent interaction, and agent-human communication. existing studies are categorized by interaction types and applications. we further compile public datasets and open-source implementations, and discuss challenges and future directions. we hope this review will inspire nlp community to explore more practical and impactful applications in llm-based multi-agent ads.