Paper: DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models (ICLR-2024)

Reviewer comment (Writing):
The language could be clearer, less vague and heavily simplified to make the arguments easier to understand.

Author response:
**A1:** Thanks for your suggestion, we will polish the language in the paper in subsequent versions.

Reviewer comment (Experiments):
The evaluation against a single self-trained RL baseline makes it hard to estimate the performance. Since it seems there is no limit to the perception of the own agent, another fair comparison would be a simpler approach where a statistical or rule-based approach would have all information of all cars and drive. Without having a state-of-the-art RL method that is already optimized on highway-env it is hard to see if the performance gain comes from the proposed method or from the failure to adopt the RL method on the task.

Author response:
**A2:** We would like to clarify that our comparison method, GRAD (Graph Representation for Autonomous Driving) [1], is an RL-based approach that is **specifically optimized for autonomous driving scenarios and achieves SOTA performance on the highway-env environment.** GRAD creates a global scene representation that includes estimated future trajectories of other vehicles and outperforms the best social attention representation method in the highway-env environment. We will add more specific descriptions to GRAD within the experimental setup of the updated version.

Reviewer comment (Evaluation):
Conceptually it is hard to imagine right now how this is supposed to drive in real time. Is the video sped up or slowed down? It is a challenge of the last decade how to get convolutional networks fast enough to be usable in a car. The computation challenges are not discussed at all. What is the reaction time of this and is execution speed a bottleneck?

Author response:
**A3:** The videos are not rendered in real-time. The decision-making process of the DiLu framework takes about 5-10 seconds, which includes the time for LLM inference and API call delays. As an initial attempt at a knowledge-driven AD framework, our goal in this paper is to demonstrate the capabilities of DiLu, such as causal reasoning, experience accumulation, and reflection for correction, without focusing too much on the performance aspects of the framework at this stage. 

In fact, there have been some advancements in both academia and industry regarding compressing LLMs and enhancing their inference speeds, such as reducing inference costs through linear transformations [2,3], and deploying smaller models on edge devices like GLM-2B[4], and Phi-1[5]. These explorations indicate the potential for further reductions in LLM execution times.

Reviewer comment (Theory):
What are the more precise concepts of knowledge-driven human driving that inspire this?

Author response:
**A4:**  The concept of "knowledge-driven" introduced in our paper contrasts with "data-driven". We view knowledge as a lower-dimensional, generalized representation of real-world phenomena, encompassing human knowledge and summaries of past experiences. Current data-driven AD systems learn the mapping between inputs (mostly sensor data, environmental context) and outputs (perception, decisions, controls), which lacks a comprehensive understanding of the environment and limits their generalization ability. In contrast, a Knowledge-driven AD system seeks to understand the surrounding environment and learn the causal reasoning capability between inputs and outputs through the common sense and knowledge. It also has the potential to be more interpretable, providing explanations for its decisions and actions.

Reviewer comment (Writing):
“Instill knowledge-driven capabilities” sounds very vague. Methods that acquire experiences from real-world datasets cover all learning-based methods depending on what you mean with acquire. The abstract could be more concrete, as it's hard to take away anything apart from that a LLM seems to do decision making while following a continuous learning scheme.

Author response:
**A5:** Thank you for your suggestions. Based on your valuable comments, we have revised the last part of the abstract as follows:

> *To the best of our knowledge, we are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles. Through the proposed DiLu framework, LLM is strengthened to apply knowledge and to reason causally in autonomous driving.*

Reviewer comment (Writing):
The language is hard to follow and the citations do not seem to support the claims well. In the Introduction, the sentence "This phenomenon inevitably leads to the marginal performance of data-driven methods." is one example of a broad claim without enough support in citations. There are autonomous cars driving in cities today with vision algorithms which are data-driven. They do not show marginal performance. The citations for this claim are one work describing a methodology to categorize corner cases in three common sensor modalities, which is not very related, and the second citation "Chen et al. 2022" seems to be a catch-all "survey of surveys" which is a large list of autonomous driving surveys with some added, partially trivial, thoughts.

Author response:
**A6:** To address your concerns, we accessed two common benchmarks used to evaluate vision algorithms for autonomous driving: [nuScenes detection](https://www.nuscenes.org/object-detection?externalData=no&mapData=no&modalities=Camera) and [SemanticKITTI segmentation](https://codalab.lisn.upsaclay.fr/competitions/6280#results). We have compiled statistics of SOTA methods and metrics from these benchmarks, starting from the first quarter of 2019 to the present, recorded quarterly. The results are shown in the table below. We also draw a line chart to provide a more intuitive view of the performance trends: [Performance Trend Chart (click it!)](https://smms.app/image/7rSgZh64kJLmcfe).

|  | SemanticKITTI Lidar Segmentation |  | nuScenes 3D Detection |  |
| --- | --- | --- | --- | --- |
| Quarter |  Method | mIoU | Method | NDS |
| 2019Q1 |  |  | PointPillars | 0.453 |
| 2019Q2 | kechn | 0.525 | MEGVII | 0.633 |
| 2019Q3 | flopie2009 | 0.537 | MEGVII | 0.633 |
| 2019Q4 | hugues.thomas | 0.588 | MEGVII | 0.633 |
| 2020Q1 | LiDARSeg | 0.613 | MEGVII | 0.633 |
| 2020Q2 | Shuangjie | 0.637 | Noah CV Lab fusion | 0.689 |
| 2020Q3 | MIT-HAN-LAB | 0.67 | Noah CV Lab lidar | 0.697 |
| 2020Q4 | Cylinder3D | 0.689 | CenterPoint v2 | 0.713 |
| 2021Q1 | AF2S3Net | 0.708 | CenterPoint v2 | 0.713 |
| 2021Q4 | huanghui | 0.71 | Centerpoint-Fusion | 0.749 |
| 2022Q1 | huanghui | 0.71 | FusionVPE | 0.754 |
| 2022Q2 | Point-Voxel-KD | 0.734 | DeepInteraction-e | 0.762 |
| 2022Q4 | UniSeg | 0.752 | MMFusion-e | 0.770 |
| 2023Q1 | PointSeg | 0.753 | IEI-BEVFusion++ | 0.776 |
| 2023Q2 | TASeg | 0.76 | IEI-BEVFusion++ | 0.776 |
| 2023Q3 | TASeg | 0.76 | EA-LSS | 0.776 |

It can be seen that the increase in the SOTA metrics for both benchmarks has slowed since 2021. In particular, nuScenes has improved by less than 3 percentage points from 2022 to the present. Furthermore, neither benchmark has crossed the 0.8 threshold in its respective metric. We believe that **data-driven algorithms, especially vision methods, do face a problem of marginal performance gains.**

Reviewer comment (Writing):
Other examples where statements are too broad and hard to understand are: "Furthermore, this task is particularly formidable and expensive for autonomous driving systems due to the complex challenge of iterating diverse and unpredictable driving scenarios." What is this sentence supposed to say? The authors should heavily simplify their language to deliver their points clearer. Formidable and expensive can be understood in many different ways and distracts from the core argument the authors want to make.

Author response:
**A7:** Sorry for the confusion, this sentence wants to convey that current autonomous driving systems require significant human labor and financial resources to collect and annotate driving data to handle complex and varied real-world driving scenarios. This process is essential for training and enhancing autonomous driving systems, but it presents a significant investment in terms of time, human labor, and costs.

Reviewer comment (Theory):
Claims that the knowledge-based system is how a human drives cannot be supported by the current state of research and not by the citations in this paper. The paper would benefit from not making the claim that they imitate a system in humans but instead limiting themselves to saying they designed a framework to include an LLM in a continuous learning setting where it outperforms certain other approaches.

Author response:
**A8:** Thank you for your insight. We acknowledge your statement and will reduce the emphasis on imitating human behavior in the Introduction section of subsequent versions. Instead, we will focus more on designing the DiLu framework to enable the system to continuously accumulate experience and correct erroneous decisions made by LLMs, thereby continually improving performance.

Reviewer comment (Writing):
Please add enough details from Johnson et al. 2019 to understand the vector similarity on an idea level. Make the paper more self-complete.

Author response:
**A9:** Thank you for your feedback! We’ve revised the content of Section 3.2, the Memory Module, to include an introduction to similarity retrieval. 

> *At each decision frame, the agent receives a textual description of the driving scenario. Before making a decision, the current driving scenario is embedded into a vector, which serves as the memory key. This key is then clustered and searched to find the closest scenarios (Johnson et al., 2019) in the memory module and their corresponding reasoning processes, or memories. These recalled memories are provided to the agent in a few-shot format to assist in making accurate reasoning and decisions for the current scenario.*

Reviewer comment (Experiments):
What is the data the LLM is trained on? If the training data contains driving situations from several countries, how will it be ensured that it follows the appropriate traffic rules?

Author response:
**A10:**  The LLM used in our DiLu framework includes GPT-3.5 and GPT-4 from OpenAI, which were trained on a vast corpus of internet text. Precisely due to the training data containing information about traffic scenarios and regulations in different countries, it is non-trivial to naively use LLM for decision-making in autonomous driving without any adaptations.

DiLu framework addresses this issue through the use of a memory module and the few-shot approach. Specifically, the memory module stores experiences and knowledge applicable to the current country or scenario. When these related experiences are input into the large model in a few-shot manner, they also constrain the decision-making process to consider logic and rules relevant to the specific scenario.

Reviewer comment (Experiments):
What are the 5 human crafted experiences and why are they needed?

Author response:
**A11:** We’re sorry for overlooking that detail in our article. We’ve updated the content of Section 3.2, the Memory Module, to include the specifics you mentioned. 

> *The initialization of memory module is similar to a human attending driving school before hitting the road. We select a few scenarios and manually outline the correct reasoning and decision-making processes for these situations to form the initial memory. These memories instruct the agent on the correct decision-making process for driving.*

Reviewer comment (Presentation):
Figure 7 a) could be easily replaced by a table to save space.

Author response:
**A12:** Thanks, we'll think about revising it.

Reviewer comment (Experiments):
It is a bit unsatisfactory to have only a comparison against one baseline which was re-trained on this particular data. Is there no standard scenario on Highway-Env or another RL-based approach that was already applied to Highway-env to compare against? The reviewer could not find one themselves, so they don’t see this as a downside in their rating, but they think it would make the paper stronger if the authors could find a way to compare against more than one baseline.

Author response:
**A2:** We would like to clarify that our comparison method, GRAD (Graph Representation for Autonomous Driving) [1], is an RL-based approach that is **specifically optimized for autonomous driving scenarios and achieves SOTA performance on the highway-env environment.** GRAD creates a global scene representation that includes estimated future trajectories of other vehicles and outperforms the best social attention representation method in the highway-env environment. We will add more specific descriptions to GRAD within the experimental setup of the updated version.

Reviewer comment (Experiments):
One of the main proposed advantages is better generalization through instilling human knowledge-driven capabilities instead of a data-driven only approach. However, the experimental settings derived from HighwayEnv are too restrictive to help extrapolate how such LLM based reasoning would perform on diverse new scenes using retrieval + few shot prompting. While it is perfectly fine to work with restricted settings and smaller datasets for new research work, the bridge to answer the most interesting questions is too long.

Author response:
**A1:**  Although the experimental setup in highway-env has certain limitations, we have demonstrated the effectiveness of the DiLu framework through our experiments, including the accumulation of memory and the use of few-shot experiences, as well as its outstanding generalization and transformation capabilities compared to RL-based methods. We also validated the importance of the two types of memories in the Reflection module through ablation experiments. Furthermore, we supplemented our study with experiments on aggressive agents, as detailed in the response to Q8. 

We are excited about applying the knowledge-driven paradigm to autonomous driving, as this is our initial work. We will continue to explore how knowledge can be used to drive the entire autonomous driving pipeline (including perception, decision making, and planning) so that this paradigm can be applied in a more diverse and broader range of scenarios.

Reviewer comment (Experiments):
As mentioned in strengths section, providing directions and initial experiments on assisting planning stacks (instead of directly doing discrete action decision making) could provide a lot of value.

Author response:
**A2:** Your suggestion is insightful. Using LLMs for decision making and further trajectory planning in continuous space is promising. In fact, we have tried to enable DiLu to perform trajectory planning, but the performance has not been satisfactory. One reason for this is that LLMs have difficulty producing detailed temporal trajectories, which are a series of coordinate points. Also, the trajectories produced by LLMs may contain certain hallucinations, making these output trajectories potentially risky. We believe that using existing planning methods (such as sample-based or search-based methods) after the decision process is complete seems more feasible. This will be one of the directions we will explore in our future work.

Reviewer comment (Experiments):
The experiment settings used to demonstrate generalization are not too convincing. The number of lanes and traffic density is changed, but this is still an extremely similar environment where the retrieved few-shot scenarios could be nearly directly applicable.

Author response:
**A3:** The increase in the number of lanes and traffic density actually leads to changes in driving strategies. For example, increased traffic density tends to slow down the average speed of the traffic flow, requiring more cautious driving, while an increased number of lanes provides the ego vehicle with more lane-changing opportunities. As shown in Figure 7(a), these changes in the experimental settings have a significant impact on the performance of the RL-based method. In addition, to further validate the generalization capabilities of DiLu, we perform additional experiments in environments with aggressive agents (as detailed in the response to Q8).

Reviewer comment (Experiments):
Under the above setting, it is possible that with a large enough memory module the task reduces to simply copying the answer from one of the retrieved experiences. It would be good to see a baseline where the decision from one of the retrieved experiences is used as is (voting with mixture of experts or winner takes all).

Author response:
**A4:**  To address your concern To address your concern, we conducted additional tests using a winner-take-all baseline method with the following setup: We used a memory module containing 160 experiences (including those accumulated from the Highway-env and those obtained from the CitySim dataset). For each decision frame, we conduct a similarity query for the five most similar experiences from the memory module. If the decision outcomes of at least *three of the five* experiences were consistent, we used that outcome for the decision of the current frame. If fewer than three experiences had the same decision outcome, we still used a few-shot approach for the LLM-based decision. We conducted ten repeated experiments, and here are the results:

| Framework | Success rate (SR) | Avg. Success steps (SS) | Median Success steps (SS) |
| -- | -- | -- | -- |
| Winner takes all, with 160 memories | 60% | 22.5 | 30 |
| DiLu, with 40 memories | **90%** | **27.7** | **30** |

Despite having a knowledge base four times larger than that of DiLu, the winner-takes-all baseline *underperformed* compared to the DiLu method in terms of SR (Success Rate) and Avg. SS (Average Sequential Success). This is because the similar experiences retrieved from the memory module cannot be simply copied as answers. For instance, two very similar scenarios might require entirely different decisions due to minor differences in the acceleration of the car ahead, and directly copying answers could lead to dangerous situations. 

In contrast, the few-shot approach used by the DiLu framework leverages the common sense and causal reasoning capabilities inherent in LLMs. It learns the key knowledge and reasoning logic from few-shot experiences, thereby making more reasonable decisions.

Reviewer comment (Evaluation):
The metric movement with CitySim in Figure 7b and Table 1 correction row do not seem significant to make the corresponding claims?

Author response:
**A5:** Figure 7 (b) exhibits the transformation performance of DiLu using knowledge from real-world dataset (CitySim). It can be found that DiLu performs better with knowledge from CitySim dataset compared with one without any prior experiences (the righty-side boxes are higher than the gray dash-dot line). Also, when tested in a more congested traffic environment, the knowledge in CitySim obtains better performance (Max SS=30, right fleshcolor box) compared with the accumulated experience in the simulation domain (Max SS=23, left fleshcolor box).

Reviewer comment (Reproducibility):
Nit: The key frame sampling for successful experiences seems like an important detail that has not been explained.

Author response:
**A6:** We indeed overlooked the details of keyframe sampling in our text. In our experiments, we uniformly sample frames in the sequence at equal intervals. However, this is merely a simple setup for our experiments. A more effective approach would be to cluster the scene descriptions of these frames and then select a frame from each cluster for storage.

Reviewer comment (Novelty):
Minor nit: The claim for this being the first work addressing AV planning via leveraging LLMs might need to be revised with recent papers like GPT driver (depending on chronology).

Author response:
**A7:** We are also aware of recent works using LLMs for solving autonomous driving planning tasks, such as GPT-Driver[1], DriveGPT4[2], LanguageMPC[3], and Driving with LLMs[4]. However, all these works were published after October 1st, 2023, which is after the ICLR submission deadline (September 28th, 2023). To ensure accuracy and precision, we have revised the relevant statements in the Abstract and Introduction to uniformly state: "We are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles."

Reviewer comment (Experiments):
What kind of diverse interactions do we get from the Highway-env simulator? Would it be possible to evaluate the framework under more interactive / challenging conditions, especially wrt agent interactions? It would be interesting to see the generalization to intersections, interactions with peds, aggressive agents etc.

Author response:
**A8:** We agree with your point that interactions between agents present a more challenging experimental setup. In response to this, we have performed new experiments with different types of agents. The default behavior model of surrounding cars in Highway-env is [IDMVehicle](https://highway-env.farama.org/dynamics/vehicle/behavior/#highway_env.vehicle.behavior.IDMVehicle), and we added experiments using [AggressiveVehicle](https://highway-env.farama.org/dynamics/vehicle/behavior/#highway_env.vehicle.behavior.AggressiveVehicle) and [DefensiveVehicle](https://highway-env.farama.org/dynamics/vehicle/behavior/#highway_env.vehicle.behavior.DefensiveVehicle) behavior models. These vehicle types exhibit more aggressive lane changing and acceleration/deceleration behaviors, thus posing greater challenges to the ego car's decision-making. We conducted experiments with both DiLu and the RL-based approach (GRAD) in environments with these three types of vehicle behaviors, using a 5-shots setting, `lane-4-density-2`, and 40 experiences in the memory module.

| Environment | Success Rate (SR) | Avg. Success steps (SS) | Median Success steps (SS) |
| --- | --- | --- | --- | --- |
| IDM Vehicle, DiLu | **90%** | **29.2** | **30** |
| IDM Vehicle, GRAD | 70% | 26.1 | 29 |
| Aggressive Vehicle, DiLu | **80%** | **26.2**| **30** |
| Aggressive Vehicle, GRAD | 48% | 20.1 | 23.1 |
| Defensive Vehicle, DiLu | **70%** | **22.5** | **30** |
| Defensive Vehicle, GRAD | 40% | 19.7 | 21.5 |

The results show that changes in the behavior type of surrounding vehicles significantly decrease the success rate of the RL-based method (70%->40%), while DiLu's performance, although slightly reduced, is still able to successfully drive in over 70% of cases for a continuous 30 steps. This demonstrates the clear advantage of our proposed knowledge-driven framework under more interactive and challenging conditions. **We have uploaded the new experiment videos in the supplementary materials.**

Due to the limited rebuttal period and the limitations of Highway-env, we were unable to complete experiments involving interactions at intersections or with pedestrians. As one of our next steps, we are exploring alternatives to Highway-env and conducting experiments in more realistic and diverse simulators.

Reviewer comment (Experiments):
The correction experiences intuitively should provide a strong boost to performance since they are akin to hard example mining and injecting reasoning about the negative outcomes. However the corresponding results in Table 1 do not show strong improvements. Is it possible understudied and warrants more extensive experimentations?

Author response:
**A9:** We need to clarify that the last row of Table 1 presents the results of using **both success memory and correction memory simultaneously**. We provide the following table to remove any ambiguity:

| Methods | MIN | Q1 | Median | Q3 | MAX |
| --- | --- | --- | --- | --- | --- |
| baseline | 2.0 | 3.0 | 10.0 | 22.0 | 30.0 |
| baseline + success memory | 3.0 | 7.0 | 24.5 | 29.3 | 30.0 |
| baseline + correcction memory | 4 | 7.75 | 21.5 | 30 | 30 |
| **baseline + success & correcction memory** | **5** | **7.75** | **24.5** | **30** | **30** | 

The purpose of the ablation study and Table 1 is to show that both types of memory are useful. However, it also shows that there is some overlap in the knowledge contained in the success and correction memories. Therefore, the improvement brought about by the correction memory is not as significant as the reviewer might expect.

Reviewer comment (Experiments):
Complexity and Scalability: The integration of LLMs and the need for continuous updating and reflection may introduce complexity, potentially impacting the scalability of the system.

Author response:
**A1:** We understand your concerns about introducing LLMs potentially increasing system complexity and presenting scalability issues. In fact, the DiLu framework is designed to address these problems in several ways: we store driving experiences in the memory module, and each time only a small number of similar experiences are looked up for few-shot learning, limiting the dispersion of LLM responses and enhancing the ability of LLMs to solve AD tasks. The memory module can simultaneously contain driving experiences from different datasets for different traffic scenarios. Moreover, different LLMs can be integrated into the DiLu framework. The textually stored driving experiences/knowledge can be immediately input to new LLMs in the proposed few-shot approach, without the need to accumulate from scratch again.

Reviewer comment (Experiments):
Real-World Application: While the framework shows promise, the transition from controlled experiments to real-world application can be challenging, given the unpredictable nature of real-world environments.

Author response:
**A2:** Our paper, as an early work on leveraging human knowledge reasoning capabilities in autonomous driving, focuses more on validating the capabilities of the proposed DiLu framework, including using LLMs for driving decisions and accumulating driving experience through the memory module. How to actually apply large language models in autonomous vehicles, such as the reduction of inference costs through linear transformations [1,2] and the deployment of small models on edge devices (e.g. GLM-2B[3], DeciCoder[4], Phi-1[5]), is an ongoing exploration in the industry and is beyond the scope of this paper. Furthermore, given the unpredictable nature of the real world, it is precisely the common sense knowledge contained in LLMs that the driver agent needs to make decisions, rather than data-driven  methods.

Reviewer comment (Evaluation):
Dependence on LLMs: The framework's reliance on LLMs means that its performance is heavily dependent on the capabilities and limitations of these models.

Author response:
**A3:** This is precisely the advantage of our proposed DiLu framework over those methods that require fine-tuning of LLMs. DiLu uses out-of-the-box LLMs and is loosely coupled to them. As large models continuously evolve (e.g. from GPT-3.5 to GPT-4 to GPT-4 Vision), our framework can take advantage of the improved capabilities of these models without any modifications. At the same time, the driving experience previously accumulated in the memory module (represented in pure text) can be fed into new LLMs, eliminating the need to start accumulating from scratch.

Reviewer comment (Evaluation):
Evaluation thoroughness: The authors only evaluate the proposed methods with oversimplied metrics (collisions) and compared to a simple baseline (RL). The limitation of the evaluation poses a question mark on how such system actually performs in the real driving scenarios, compared to sota autonomous driving systems.

Author response:
**A5:** We understand the reviewer's concern about the thoroughness of the framework evaluation. We would like to clarify the following points:

1. The baseline algorithms we compared are not just simple reinforcement learning methods. GRAD, which stands for [Graph Representation for Autonomous Driving,](https://ml4ad.github.io/files/papers2022/A%20Graph%20Representation%20for%20Autonomous%20Driving.pdf) is an RL-based approach specifically designed for autonomous driving scenarios [6]. GRAD generates a global scene representation that includes estimated future trajectories of other vehicles. It outperforms the best-performing social attention representation methods in the highway environment.
2. Evaluation metrics in the experimental section include Success Rate (SR) and Sequential Success Steps (SS). We define "*a successfully completed task means that the ego vehicle navigates through traffic at a reasonable speed and without collisions*" (in Appendix A.1), which includes two metrics commonly used in closed-loop simulations[7]: No collision and Speed limit compliance. 

Through this comparative experiment, we aim to demonstrate the core advantage of the DiLu framework: its use of common sense knowledge and minimal experience accumulation to make reasonable decisions in autonomous driving. As mentioned in Section 4.3, the DiLu framework needs only 40 memories to demonstrate comparable performance to a reinforcement learning method trained on 600,000 episodes in training scenarios, and much higher generalization capability compared to data-driven methods.

Reviewer comment (Novelty):
While LLM-based agent systems have shown success in various embodied systems, the adaptation of it in the AV tasks is still unclear to the reviewer. AI agent system has shown prominent success in task planning for open world robotic tasks, but AV has a different setting (with different challenges). The motivation and advantages of using AI agent system for AV need to be elaborated more.

Author response:
**A4:** One of the motivations of our paper is that the preliminary results achieved by LLM-based agents in the field of embodied AI have shown us the potential for their application in AV tasks. As an initial attempt to apply LLMs in the field of autonomous driving, our paper proposes the novel DiLu framework. DiLu continuously accumulates knowledge in the driving domain through its memory module and improves the closed-loop decision performance with few-shot driving experiences. In addition, DiLu uses a reflection module to reflect and correct erroneous decisions, allowing the whole framework to continuously improve its performance by exploring the environment.

Reviewer comment (Evaluation):
The authors did not evaluate the proposed framework thoroughly enough (with only simple metrics and simple baselines). This further raises questions regarding how promising or what are the key advantages of using AI agent system in AV setting.

Author response:
**A5:** We understand the reviewer's concern about the thoroughness of the framework evaluation. We would like to clarify the following points:

1. The baseline algorithms we compared are not just simple reinforcement learning methods. GRAD, which stands for [Graph Representation for Autonomous Driving,](https://ml4ad.github.io/files/papers2022/A%20Graph%20Representation%20for%20Autonomous%20Driving.pdf) is an RL-based approach specifically designed for autonomous driving scenarios [6]. GRAD generates a global scene representation that includes estimated future trajectories of other vehicles. It outperforms the best-performing social attention representation methods in the highway environment.
2. Evaluation metrics in the experimental section include Success Rate (SR) and Sequential Success Steps (SS). We define "*a successfully completed task means that the ego vehicle navigates through traffic at a reasonable speed and without collisions*" (in Appendix A.1), which includes two metrics commonly used in closed-loop simulations[7]: No collision and Speed limit compliance. 

Through this comparative experiment, we aim to demonstrate the core advantage of the DiLu framework: its use of common sense knowledge and minimal experience accumulation to make reasonable decisions in autonomous driving. As mentioned in Section 4.3, the DiLu framework needs only 40 memories to demonstrate comparable performance to a reinforcement learning method trained on 600,000 episodes in training scenarios, and much higher generalization capability compared to data-driven methods.

Reviewer comment (Novelty):
The proposed AI agent follows a typical setup compared to the other existing works in robotics tasks. The authors should highlight more on the unique challenges and design choices tailored for the AV task.

Author response:
**A6:** Robotics tasks tend to have clearer objectives (such as grasping, navigation, etc.) and researchers are more concerned with completing the entire task. Most of the scenarios that robots are exposed to are closed, low-speed indoor scenarios. In contrast, the objectives of autonomous driving tasks are more ambiguous (just driving on the road) and require consideration of more complex performance metrics (including driving safety, passenger comfort, and compliance with traffic regulations). 

The unique challenges of Autonomous Vehicle (AV) tasks are mainly related to the fact that AVs operate in highly dynamic and open environments and encounter more unexpected edge cases. These challenges are difficult to address with traditional rule- or RL-based methods. The proposed DiLu, through its memory module, records past valuable experiences and leverages the common sense knowledge inherent in LLMs (i.e., traffic rules, how to properly interact with other participants) to achieve better generalization and continuous learning. We thank you for the insightful suggestion and will consider adding these explanations to the introduction to make the paper clearer and more readable.